Speculative execution mitigations in Edera
Speculative execution is a performance optimization that is implemented in most modern processors. It allows for evaluating both sides of a branch in code before knowing which branch will actually be taken. Unfortunately, some implementations of this performance optimization have had security issues, specifically that the side effects of speculative execution can be made visible in certain cases, which may lead to the disclosure of confidential information. The Meltdown security advisory, targeting Intel processors, is an example of a severe speculative execution information leak.
To mitigate the risks of speculative execution, newer processors have implemented hardware-level features that act as effective speculative execution barriers. However, the availability of these hardware-level mitigations is inconsistent: AMD processors, for example, were never vulnerable to the Meltdown vulnerability due to design decisions made in their processors, while many Intel processors available today still require software-level mitigations to fully protect against Meltdown and similar speculative execution vulnerabilities.
Hardware vs. software mitigations
Hardware-based speculative execution mitigations are primarily implemented through hardware-specific features intended for this purpose. For example, while modern Intel processors still largely have a vulnerability to Meltdown, they also have a functional mitigation for Meltdown in the form of IBRS, which restricts how indirect branches can be speculated. Additionally, newer Intel processors have support for supervisor mode execution prevention (SMEP), which restricts the execution of code in user-mode memory when running in supervisor mode. The latter feature, when enabled, mitigates Meltdown entirely with no performance penalty, while IBRS imposes a mild performance penalty.
Software-based speculative execution mitigations, on the other hand, are largely implemented by leveraging features known to act as effective speculation barriers. For example, Linux’s KPTI and Xen’s XPTI (kernel page-table isolation and Xen page-table isolation, respectively) work by reconfiguring the MMU to hide kernel or user pages as appropriate, and then forcing a TLB flush. The combination results in speculative execution faulting when it tries to branch to inaccessible code. This is an effective mitigation against Meltdown, but modifying memory mappings and then flushing the TLB is a very expensive operation, and it happens on every context switch.
Accordingly, hardware-based mitigations should be preferred over software-based mitigations whenever possible.
Do speculative mitigations actually matter?
The largest area of risk for speculative execution vulnerabilities is multi-tenancy. If a tenant is processing sensitive data in the absence of speculation barriers, it may be possible for another tenant to observe some of that data through speculative side-channel attacks. Because of this, Edera Protect defaults to enabling all relevant speculation mitigations on a given compute platform, but these mitigations may be redundant depending on the nature of that compute platform.
For example, if Edera Protect is deployed on an already virtualized platform as a nested hypervisor, the platform itself may have speculations enabled that are redundant, such as clearing the return stack buffer on context switches. In these cases, the mitigation in Edera Protect can be disabled without any impact on security.
Another architectural decision that influences the risk of speculative execution vulnerabilities is whether the code running on a platform is fully trusted. If the code that can be run is controlled entirely by the platform, then the risk of speculative execution vulnerabilities is significantly reduced.
Finally, on platforms where the execution environment is backed by multiple physical CPU cores, speculative execution can be derisked by placing different tenants in different CPU pools, which prevents those tenants from attacking each other using speculative execution vulnerabilities. In these environments, speculative mitigations can be safely disabled as long as multiple tenants are not contending the same CPU cores at the same time. Additionally, disabling SMT (hyper-threading) can also prevent tenants from observing speculative execution side effects from other tenants and may be a lower-cost option than using speculative mitigations.
In essence, speculative mitigations are only necessary in environments where multiple tenants are sharing threads on the same CPU core. If this can be avoided through architectural design, then they can be disabled safely in their entirety. Otherwise, only a subset of mitigations may be necessary, depending on the nature of the workloads and how they are scheduled.
Primary speculative mitigations
There are two main avenues for mitigating speculative side-channel attacks in Edera Protect: XPTI or KPTI and disabling SMT. Edera recommends simply disabling SMT and turning off XPTI or KPTI for optimal performance. However, SMT is generally safe to leave enabled on AMD platforms (and XPTI or KPTI are not used on AMD platforms by default).
XPTI and KPTI
Xen page-table isolation and kernel page-table isolation are the classical mitigation strategies against Meltdown. They work together to protect the entire platform, first by isolating the hypervisor page tables from unauthorized kernel and user access, and second, to isolate each zone kernel’s page tables from unauthorized user access.
The page table isolation is accomplished by reconfiguring the MMU and flushing the translation buffer on every context switch, so it is very expensive. This is partially offset on some Intel platforms by using processor contexts (PCID) to reduce the overhead of reconfiguring and flushing the MMU.
On AMD platforms, these features are disabled by default as they are unnecessary.
To disable XPTI, use the xpti=off flag when installing the hypervisor. It is also possible to isolate the Xen page tables only from zone kernels by using xpti=dom0=on,domU=off instead, which can reduce the overhead slightly.
To disable KPTI for a zone, use the nopti kernel flag when launching a zone.
The worst-case performance overhead for these features working in parallel is approximately 50%.
SMT or hyper-threading disable
As discussed above, hyper-threading significantly increases the likelihood that a speculative execution exploit will be successful. It is possible to turn off hyper-threading, which ensures that multiple workloads will not share a CPU core at the same physical moment, thus making it considerably more difficult to observe speculative side channels for other tenants.
To disable SMT (hyper-threading), use the smt=off flag when installing the hypervisor.
SMT can provide a marginal performance improvement for workloads that are multi-threaded and contended by memory access latency (uncached memory access on Coffee Lake platforms is approximately 270 cycles per access), but can also cause performance degradation depending on the design of the workload. In general, Edera recommends disabling SMT over using XPTI or KPTI.
Secondary speculative mitigations
In addition to the mitigations listed above, there are several other microarchitectural-specific speculative mitigations that may impose a mild amount of overhead. However, measurements have indicated that these mitigations represent roughly 5% or less of the overall overhead imposed by speculative mitigations on Intel platforms, and approximately 3% or less overhead on AMD platforms. In general, Edera does not recommend disabling these mitigations as a result.
Indirect branch restricted speculation (IBRS)
IBRS is used to protect against speculative mispredictions of branch targets, which can result in the observation of data leakage in limited circumstances, such as the kernel leaking information into user space, or the hypervisor leaking information into either kernel or user space. IBRS works by preventing speculative execution across privileged domains by using a machine-specific control register to control when speculation should be restricted.
As a hardware-level mitigation, the cost of this feature is minimal in most cases. Some older Intel processors implement IBRS in microcode, however, which results in a minor performance overhead of approximately 0.5% on processes that are syscall-heavy. Because the overhead is minimal, Edera does not recommend disabling IBRS support when available.
To disable IBRS, use the spec-ctrl=ibrs=off flag when installing the hypervisor.
Indirect branch prediction barrier (IBPB)
IBPB is used to force the CPU to clear its branch prediction history when switching privilege domains. These barriers are intended to be used in conjunction with IBRS to provide full coverage for speculative leaks across privileged domains. In essence, IBPB and IBRS work together to solve the same problem from different angles.
As a hardware-level mitigation, the cost of this feature is more than IBRS alone since it requires the processor to fully flush its branch prediction history on every context switch. However, this overhead is only observable in syscall-heavy processes, and the overhead is only about 2% over baseline in benchmarks. Accordingly, Edera does not recommend disabling IBPB support when available.
To disable IBPB, use the spec-ctrl=ibpb=off flag when installing the hypervisor.
Speculative store bypass disable (SSBD)
In speculative execution, memory load and store operations are sometimes bypassed, which can leak information about the kernel or user state through timing analysis. The speculative store bypass disable mitigation is a hardware-level mitigation that disables bypassing memory store operations during speculation to ensure that speculative execution runs in constant time.
The cost of this feature is extremely minimal, with a measured overhead of approximately 0.08% on a syscall-heavy benchmark workload. Accordingly, Edera does not recommend disabling SSBD support when available.
To disable SSBD, use the spec-ctrl=ssbd=off flag when installing the hypervisor.
Level 1 data cache flushing (L1D flush)
In some cases, speculative execution may result in observable changes to the L1 data cache lines. Edera Protect defaults to flushing L1 data cache lines on context switches, which may, in some cases, have a performance penalty imposed due to L1 cache misses. In general, these cache misses impose a minimal impact when amortized over the entire workload’s execution time. Edera does not recommend disabling this mitigation.
To disable L1D cache flushing, use the spec-ctrl=l1d-flush=off flag when installing the hypervisor.
Summary
Speculative execution mitigations are an important part of maintaining strong isolation in multi-tenant environments. Edera Protect enables these mitigations by default to reduce risk and protect against known vulnerabilities like Meltdown.
However, not every environment requires the same level of protection. In cases where workloads are fully trusted, or tenants can be isolated at the CPU core level, some mitigations can be safely disabled to improve performance.
Edera Protect offers flexible options for tuning speculative mitigations based on the architecture, processor type, and workload characteristics of the deployment. When in doubt, Edera recommends using the default settings, which prioritize security with reasonable performance trade-offs.