Metrics v1.12.0
Collect metrics
There are multiple ways to collect a set of metrics to provide an overview of the health and performance of Edera and the infrastructure it runs on. As of v1.11, some metrics can be picked up through containerd as normal. Other metrics are exposed by in the Prometheus format, by default on http://127.0.0.1:3035/.
If an alternative address is preferred, it can be configured via the metrics-endpoint-address flag in daemon.toml.
Metrics can be viewed directly via their respective scrape endpoint or scraped by third-party tooling.
There are three endpoints exposing different categories of metrics from Edera:
System metrics
System-level metrics are hosted at /metrics and include the following:
Host metrics
Host metrics encapsulates metrics obtained from the host of Edera, also referred to as dom0.
| Metric | Description | Type | Labels |
|---|---|---|---|
host_cpu_usage_percent | CPU usage in percent, in the range 0-100. Under Xen, host refers to dom0 rather than machine resources. | gauge | cpu |
host_cpu_usage_seconds_total | CPU time used, in seconds, per cpu, per mode. Under Xen, host refers to dom0 rather than machine resources. | counter | cpu, mode |
host_memory_total_bytes | Total number of bytes of memory allocated to the host. Under Xen, host refers to dom0 rather than machine resources. Under KVM, equal to hypervisor_memory_total_bytes. | gauge | |
host_memory_free_bytes | Number of bytes of memory allocated to the host and currently unused. Excludes reclaimable page cache, so it’s lower than host_memory_available_bytes. Under Xen, host refers to dom0 rather than machine resources. | gauge | |
host_memory_available_bytes | Number of bytes of memory allocated to the host and available for new allocations without swapping. Under Xen, host refers to dom0 rather than machine resources. Under KVM, equal to hypervisor_memory_available_bytes. | gauge | |
host_memory_used_bytes | Number of bytes of memory allocated to the host and currently in use. Under Xen, host refers to dom0 rather than machine resources. | gauge | |
daemon_zones | Current number of zones, per state. | gauge | state |
Hypervisor metrics
Hypervisor metrics encapsulates machine-wide metrics obtained from the hypervisor, as opposed to dom0-scoped host metrics.
| Metric | Description | Type | Labels |
|---|---|---|---|
hypervisor_memory_total_bytes | Total number of bytes of memory in the machine’s hardware, as reported by the hypervisor. Under Xen, hypervisor refers to machine resources rather than dom0. Under KVM, equal to host_memory_total_bytes. | gauge | |
hypervisor_memory_available_bytes | Number of bytes of machine memory available to start new zones. Under Xen, hypervisor refers to machine resources rather than dom0. Under KVM, equal to host_memory_available_bytes. | gauge |
Health check metrics
Health check metrics encapsulates metrics which provide an insight into the health of Edera services.
| Metric | Description | Type | Labels |
|---|---|---|---|
health_check_total | Total number of health checks performed, labeled by service and result status (success or failure). | counter | service, status |
Zone metrics
Zone metrics are exposed in two flavors: Edera Zone metrics and Kubernetes metrics. Though the exposed metrics are the same, the filtering and labeling differ for their respective contexts.
Edera Zone metrics
Edera Zone metrics are hosted at /metrics/resource. Edera Zone metrics include metrics for all Zones, regardless how the Zone was created.
Zone metrics
Zone metrics encapsulates metrics obtained from an Edera Zone.
| Metric | Description | Type | Labels |
|---|---|---|---|
zone_cpu_usage_percent | CPU usage in percent, in the range 0-100. | gauge | cpu, zone_id |
zone_cpu_usage_seconds_total | CPU time used, in seconds, per cpu. | counter | cpu, zone_id |
zone_create_timestamp_milliseconds | Time when zone was created as a Unix timestamp with millisecond resolution. | gauge | zone_id |
zone_memory_total_bytes | Total number of bytes of memory allocated to the zone. | gauge | zone_id |
zone_memory_free_bytes | Number of bytes of memory allocated to the zone and currently unused. | gauge | zone_id |
zone_memory_used_bytes | Number of bytes of memory allocated to the zone and currently in use. | gauge | zone_id |
zone_ready_timestamp_milliseconds | Time when a zone is ready for use as a Unix timestamp with millisecond resolution. | gauge | zone_id |
zone_memory_pressure_state | Current memory pressure state per guest NUMA node (0=unknown, 1=ok, 2=soft, 3=hard, 4=critical). | gauge | zone_id, node |
zone_memory_pressure_events_total | Cumulative memory pressure transitions per guest NUMA node and cause, since the daemon began tracking the zone. | counter | zone_id, node, cause |
zone_memory_pressure_vmstat_pages | Most recent in-kernel vmstat reading observed on a memory pressure event, in guest pages, per guest NUMA node and vmstat field. Absent until the field has been read at least once; the reclaim path that triggered the event determines which fields are available, so free only updates on kswapd wakeups. | gauge | zone_id, node, field |
zone_memory_pressure_transition_timestamp_milliseconds | When the daemon last received a memory pressure transition for a guest NUMA node, as a unix timestamp with millisecond resolution. Stamped on receipt, since the guest’s own event clock shares no epoch with the host; use it to tell a current reading from a stale one, since the pressure state and vmstat series otherwise hold their last observed values. | gauge | zone_id, node |
zone_memory_target_bytes | Memory the zone is currently asked to balloon to, in bytes, from its active resource spec. Compare against zone_hypervisor_memory_total_bytes to detect a balloon driver that is not converging on the requested target. | gauge | zone_id |
zone_memory_min_bytes | Floor below which the resource enforcer will not drive the zone’s target, in bytes. A target sitting at this floor means no further memory can be reclaimed from the zone by policy. | gauge | zone_id |
zone_memory_adjustment_policy | The zone’s resource adjustment policy (0=unknown, 1=static, 2=dynamic). Only dynamic zones are ballooned by the resource enforcer. | gauge | zone_id |
Hypervisor metrics
Hypervisor metrics encapsulates metrics obtained from the hypervisor of an Edera zone.
| Metric | Description | Type | Labels |
|---|---|---|---|
zone_hypervisor_cpu_usage_seconds_total | CPU time used, in seconds, per zone. | counter | zone_id |
zone_hypervisor_memory_max_bytes | Maximum memory allowed, in bytes, per zone. | gauge | zone_id |
zone_hypervisor_memory_total_bytes | Total memory currently allocated, in bytes, per zone. Under Xen this is the hypervisor’s own accounting. Under KVM it is Cloud Hypervisor’s memory_actual_size, counting only virtio-mem blocks the guest has plugged and subtracting the balloon size the guest reports. Either can be compared against zone_memory_target_bytes to judge balloon convergence. If the KVM VMM API is unreachable, this falls back to the last size the daemon requested. | gauge | zone_id |
zone_hypervisor_memory_outstanding_bytes | Memory owed but not yet allocated, in bytes, per zone. | gauge | zone_id |
zone_hypervisor_vcpus_online | Number of virtual CPUs currently online, per zone. | gauge | zone_id |
Kubernetes Zone metrics
Kubernetes Zone metrics are hosted at /metrics/kubernetes. Kubernetes Zone metrics include metrics only for Zones that were launched by Kubernetes and include labeling that correlates the Zone to Kubernetes Namespaces and Pods.
Zone metrics
Zone metrics encapsulates metrics obtained from an Edera Zone.
| Metric | Description | Type | Labels |
|---|---|---|---|
zone_cpu_usage_percent | CPU usage in percent, in the range 0-100. | gauge | cpu, zone_id, namespace, pod |
zone_cpu_usage_seconds_total | CPU time used, in seconds, per cpu. | counter | cpu, zone_id, namespace, pod |
zone_create_timestamp_milliseconds | Time when zone was created as a Unix timestamp with millisecond resolution. | gauge | zone_id, namespace, pod |
zone_memory_total_bytes | Total number of bytes of memory allocated to the zone. | gauge | zone_id, namespace, pod |
zone_memory_free_bytes | Number of bytes of memory allocated to the zone and currently unused. | gauge | zone_id, namespace, pod |
zone_memory_used_bytes | Number of bytes of memory allocated to the zone and currently in use. | gauge | zone_id, namespace, pod |
zone_ready_timestamp_milliseconds | Time when a zone is ready for use as a Unix timestamp with millisecond resolution. | gauge | zone_id, namespace, pod |
zone_memory_pressure_state | Current memory pressure state per guest NUMA node (0=unknown, 1=ok, 2=soft, 3=hard, 4=critical). | gauge | zone_id, node, namespace, pod |
zone_memory_pressure_events_total | Cumulative memory pressure transitions per guest NUMA node and cause, since the daemon began tracking the zone. | counter | zone_id, node, cause, namespace, pod |
zone_memory_pressure_vmstat_pages | Most recent in-kernel vmstat reading observed on a memory pressure event, in guest pages, per guest NUMA node and vmstat field. Absent until the field has been read at least once; the reclaim path that triggered the event determines which fields are available, so free only updates on kswapd wakeups. | gauge | zone_id, node, field, namespace, pod |
zone_memory_pressure_transition_timestamp_milliseconds | When the daemon last received a memory pressure transition for a guest NUMA node, as a unix timestamp with millisecond resolution. Stamped on receipt, since the guest’s own event clock shares no epoch with the host; use it to tell a current reading from a stale one, since the pressure state and vmstat series otherwise hold their last observed values. | gauge | zone_id, node, namespace, pod |
zone_memory_target_bytes | Memory the zone is currently asked to balloon to, in bytes, from its active resource spec. Compare against zone_hypervisor_memory_total_bytes to detect a balloon driver that is not converging on the requested target. | gauge | zone_id, namespace, pod |
zone_memory_min_bytes | Floor below which the resource enforcer will not drive the zone’s target, in bytes. A target sitting at this floor means no further memory can be reclaimed from the zone by policy. | gauge | zone_id, namespace, pod |
zone_memory_adjustment_policy | The zone’s resource adjustment policy (0=unknown, 1=static, 2=dynamic). Only dynamic zones are ballooned by the resource enforcer. | gauge | zone_id, namespace, pod |
Zone /proc metrics
In addition, zone kernel-level metrics from the zone’s local /proc are scraped on intervals and exposed through the /proc/resources (and the aliased /proc/kubernetes) endpoint.
The kernel-level metrics are dynamically generated, in the form zone_PROCFILE_SNAKE-CASED-FIELDNAME, from the contents of the current zone kernel’s:
/proc/meminfo/proc/stat/proc/net/netstat/proc/diskstats
For instance, the zone’s kernel-level /proc/net/netstats InBcastPkts value would be dynamically exported on the Prometheus endpoint as:
zone_netstat_in_bcast_pkts{zone_id="75afee96-537d-426c-8a57-abdd514708e1",proto="ip_ext"} 23
All kernel fields from the above files are exported for each zone in this manner, too many to list here. For more details and complete stat lists, see the respective manpages for the above /proc files:
man proc_meminfoman proc_statman proc_netman proc_diskstats
Hypervisor metrics
Hypervisor metrics encapsulates metrics obtained from the hypervisor of an Edera zone.
| Metric | Description | Type | Labels |
|---|---|---|---|
zone_hypervisor_cpu_usage_seconds_total | CPU time used, in seconds, per zone. | counter | zone_id |
zone_hypervisor_memory_max_bytes | Maximum memory allowed, in bytes, per zone. | gauge | zone_id |
zone_hypervisor_memory_total_bytes | Total memory currently allocated, in bytes, per zone. Under Xen this is the hypervisor’s own accounting. Under KVM it is Cloud Hypervisor’s memory_actual_size, counting only virtio-mem blocks the guest has plugged and subtracting the balloon size the guest reports. Either can be compared against zone_memory_target_bytes to judge balloon convergence. If the KVM VMM API is unreachable, this falls back to the last size the daemon requested. | gauge | zone_id |
zone_hypervisor_memory_outstanding_bytes | Memory owed but not yet allocated, in bytes, per zone. | gauge | zone_id |
zone_hypervisor_vcpus_online | Number of virtual CPUs currently online, per zone. | gauge | zone_id |
Label examples
Exposed metrics may contain labels, which provides additional context, such as the zone a metric is being populated from, and allows for more detailed querying and filtering.
| Label | Description | Example |
|---|---|---|
cpu | Name of the CPU. | cpu="cpu1" or cpu="1" |
cause | Cause of a zone memory pressure transition. | cause="wakeup_kswapd" or cause="try_to_free_pages" |
field | Vmstat field name for a memory pressure snapshot reading. | field="free" or field="anon_mapped" |
namespace | Name of the Kubernetes namespace the zone is in. | namespace="default" |
node | Guest NUMA node ID within the zone. | node="0" |
pod | Kubernetes pod ID the zone is related to. | pod="nginx-deployment-7d6c54b4c6-g84cl" |
mode | Mode of the CPU. | mode="idle" or mode="user" |
service | Name of the Edera service. | service="protect-cri" or service="protect-daemon" |
state | State of the Edera zone. | state="ready" or state="failed" |
status | Status returned from the Edera service during a health check. | status="success" or status="failed" |
zone_id | ID of the Edera zone. | zone_id="7568edad-3a34-47e3-b605-36ede54a9aab" |