v1.10.0

4 min read

Release Notes

These release notes describe all changes since the previous minor release, v1.9.0, not since the most recent 1.9.x patch. Some of the fixes and enhancements listed below were also delivered in the v1.9.1 patch release; they are repeated here so this page is a complete record of what changed between 1.9.0 and 1.10.0.

New Features & Enhancements

Installer

  • KVM-based virtualization is in Early Access. The feature is supported but ships with minor known issues in this release (see Known Issues). KVM-based virtualization is opt-in, and may be enabled by passing the HYPERVISOR=kvm flag to the installer. KVM is more convenient, can use your existing kernel, and does not require installing Xen or modifying boot-level components, but has reduced performance compared to Xen.
  • The installer now automatically chooses the best available virtualization mechanism for Xen zones based on available hardware virtualization capabilities.
  • Removed the protect-orchestrator service from the install.

Core

  • Added zone suspend/restore capabilities to Xen zones, allowing zones to be suspended or resumed safely.
  • Added zone forking capabilities to Xen zones. This allows a new zone to be cheaply “snapshotted” from an existing suspended zone, using a copy-on-write mechanism and shared pages to achieve tight bin-packing and efficient memory consumption. This feature is currently only supported with Xen PVH zones, is currently in Early Access, and requires enabling the fork-v0 feature flag to use.
  • Full NUMA support. Edera now intelligently selects the optimal NUMA node count and selection strategy, and it is now possible to manually specify NUMA topology options for optimal performance, if desired.
  • Added support for issuing Edera control plane API commands from Edera-managed workloads. This requires enabling the control-api-idm-v0 feature flag for those workloads.
  • Added hundreds of zone-specific kernel metrics to the node-level /metrics/resource Prometheus endpoint. These expose all stats in each zone kernel’s:
    • /proc/meminfo
    • /proc/stat
    • /proc/net/netstat
    • /proc/net/snmp
    • /proc/diskstats
    • /proc/vmstat
  • Added live eBPF-based memory pressure event tracking and metrics to all zones. See the daemon configuration for scrape configuration and the metrics reference for more details.
  • Destroyed zone logs are now retained as tombstones so they remain available for post-mortem inspection.
  • The Prometheus /metrics endpoint previously hosted by protect-orchestrator is now hosted by protect-daemon. The default port and endpoint are unchanged, and previously exposed metrics are all still present.

Tools

  • protect-ctl zone|workload destroy now supports destroying multiple zones/workloads by selector match, for example protect-ctl zone destroy --selector status.state=failed. Currently, status is the only supported selector field.
  • protect-ctl zone launch now supports the --numa-nodes and --numa-strategy flags.
  • Added the protect-ctl zone suspend subcommand.
  • Added the protect-ctl zone resume subcommand.
  • Added the protect-ctl zone fork subcommand. Note that this command will not function unless the fork-v0 feature flag has been enabled.

Bug Fixes

Kubernetes

  • Fixed an issue where the read-only flag for K8S host mounts was not respected in all cases.
  • Fixed an issue where zones were not destroyed as eagerly as they could be when the kubelet invoked RemovePodSandbox/StopPodSandbox.
  • Fixed an issue where the wrong image cache ref could be handed back to the kubelet in some cases, causing pod scheduling to fail.
  • Fixed an issue where the Kubernetes SPDY proxy service would not properly restart if protect-cri was restarted via systemd.

Core

  • Fixed several IDM deadlocks and throughput bottlenecks under load, including a storm-induced irrecoverable deadlock, bottlenecks that capped packet throughput, and a self-deadlock in stale-stream cleanup, resulting in a 7.5x speedup in IDM throughput.
  • Reworked the zone IDM reader to use a per-backend event channel file descriptor with non-blocking polling.
  • Fixed Xen event channel issues, including retrying ioctls on EINTR and closing a WakeMap unbind/subscribe race.
  • Raised the protect-daemon file descriptor limit to 8192.
  • Fixed an issue where Xen physmap population could incorrectly fail in conditions where memory is heavily fragmented.
  • Fixed an issue where Edera’s dom0 memory resource monitoring was not accurately calculating the free pool size.
  • Fixed an issue where a malformed zone kernel image could cause protect-daemon to crash.
  • Fixed various OCI image parsing and handling bugs.

Tools

  • Fixed an issue where commands that stream zone/workload stdin would appear to hang until terminal input was received.

Known Issues

KVM support

  • PCI passthrough is not supported.
  • NUMA support is not enabled.
  • The Edera DRA driver does not function on KVM nodes.
  • KVM zones will not correctly respect custom MTU values set on host links.
  • KVM workloads are not resilient across protect-daemon restarts/crashes.

Upgrade notes

There are no known breaking changes in this release from the previous minor or patch release.

Last updated on