kernel.git/arch/x86/kernel/fpu, branch linux-6.15.y

x86/pkeys: Simplify PKRU update in signal frame

2025-07-06T09:04:25Z

commit d1e420772cd1eb0afe5858619c73ce36f3e781a1 upstream. The signal delivery logic was modified to always set the PKRU bit in xregs_state->header->xfeatures by this commit: ae6012d72fa6 ("x86/pkeys: Ensure updated PKRU value is XRSTOR'd") However, the change derives the bitmask value using XGETBV(1), rather than simply updating the buffer that already holds the value. Thus, this approach induces an unnecessary dependency on XGETBV1 for PKRU handling. Eliminate the dependency by using the established helper function. Subsequently, remove the now-unused 'mask' argument. Signed-off-by: Chang S. Bae Signed-off-by: Ingo Molnar Cc: Andy Lutomirski Cc: Aruna Ramakrishna Cc: H. Peter Anvin Cc: Linus Torvalds Cc: Oleg Nesterov Cc: Tony W Wang-oc Cc: Dave Hansen Link: https://lore.kernel.org/r/20250416021720.12305-9-chang.seok.bae@intel.com Cc: Ben Hutchings Signed-off-by: Greg Kroah-Hartman

x86/fpu: Refactor xfeature bitmask update code for sigframe XSAVE

2025-07-06T09:04:25Z

commit 64e54461ab6e8524a8de4e63b7d1a3e4481b5cf3 upstream. Currently, saving register states in the signal frame, the legacy feature bits are always set in xregs_state->header->xfeatures. This code sequence can be generalized for reuse in similar cases. Refactor the logic to ensure a consistent approach across similar usages. Signed-off-by: Chang S. Bae Signed-off-by: Ingo Molnar Cc: Andy Lutomirski Cc: H. Peter Anvin Cc: Linus Torvalds Cc: Oleg Nesterov Link: https://lore.kernel.org/r/20250416021720.12305-8-chang.seok.bae@intel.com Cc: Ben Hutchings Signed-off-by: Greg Kroah-Hartman

x86/fpu: Update the outdated comment above fpstate_init_user()

2025-03-25T08:57:33Z

fpu_init_fpstate_user() was removed in: commit 582b01b6ab27 ("x86/fpu: Remove old KVM FPU interface"). Update that comment to accurately reflect the current state regarding its callers. Signed-off-by: Chao Gao Signed-off-by: Ingo Molnar Link: https://lore.kernel.org/r/20250324131931.2097905-1-chao.gao@intel.com

x86/fpu/xstate: Fix inconsistencies in guest FPU xfeatures

2025-03-17T22:52:31Z

Guest FPUs manage vCPU FPU states. They are allocated via fpu_alloc_guest_fpstate() and are resized in fpstate_realloc() when XFD features are enabled. Since the introduction of guest FPUs, there have been inconsistencies in the kernel buffer size and xfeatures: 1. fpu_alloc_guest_fpstate() uses fpu_user_cfg since its introduction. See: 69f6ed1d14c6 ("x86/fpu: Provide infrastructure for KVM FPU cleanup") 36487e6228c4 ("x86/fpu: Prepare guest FPU for dynamically enabled FPU features") 2. __fpstate_reset() references fpu_kernel_cfg to set storage attributes. 3. fpu->guest_perm uses fpu_kernel_cfg, affecting fpstate_realloc(). A recent commit in the tip:x86/fpu tree partially addressed the inconsistency between (1) and (3) by using fpu_kernel_cfg for size calculation in (1), but left fpu_guest->xfeatures and fpu_guest->perm still referencing fpu_user_cfg: https://lore.kernel.org/all/20250218141045.85201-1-stanspas@amazon.de/ 1937e18cc3cf ("x86/fpu: Fix guest FPU state buffer allocation size") The inconsistencies within fpu_alloc_guest_fpstate() and across the mentioned functions cause confusion. Fix them by using fpu_kernel_cfg consistently in fpu_alloc_guest_fpstate(), except for fields related to the UABI buffer. Referencing fpu_kernel_cfg won't impact functionalities, as: 1. fpu_guest->perm is overwritten shortly in fpu_init_guest_permissions() with fpstate->guest_perm, which already uses fpu_kernel_cfg. 2. fpu_guest->xfeatures is solely used to check if XFD features are enabled. Including supervisor xfeatures doesn't affect the check. Fixes: 36487e6228c4 ("x86/fpu: Prepare guest FPU for dynamically enabled FPU features") Suggested-by: Chang S. Bae Signed-off-by: Chao Gao Signed-off-by: Ingo Molnar Cc: Andy Lutomirski Cc: H. Peter Anvin Cc: Linus Torvalds Cc: Oleg Nesterov Cc: Dave Hansen Cc: Juergen Gross Cc: Stefano Stabellini Cc: Paolo Bonzini Cc: Vitaly Kuznetsov Cc: Sean Christopherson Cc: David Woodhouse Link: https://lore.kernel.org/r/20250317140613.1761633-1-chao.gao@intel.com

x86/fpu: Clarify the "xa" symbolic name used in the XSTATE* macros

2025-03-17T12:47:12Z

Tie together the %[xa] in the XSAVE/XRSTOR definitions with the respective usage in the asm macros so that it is perfectly clear. No functional changes. Signed-off-by: Borislav Petkov (AMD)

x86/fpu: Use XSAVE{,OPT,C,S} and XRSTOR{,S} mnemonics in xstate.h

2025-03-13T17:36:52Z

Current minimum required version of binutils is 2.25, which supports XSAVE{,OPT,C,S} and XRSTOR{,S} instruction mnemonics. Replace the byte-wise specification of XSAVE{,OPT,C,S} and XRSTOR{,S} with these proper mnemonics. No functional change intended. Signed-off-by: Uros Bizjak Signed-off-by: Ingo Molnar Cc: Andy Lutomirski Cc: Brian Gerst Cc: H. Peter Anvin Cc: Dave Hansen Cc: Linus Torvalds Link: https://lore.kernel.org/r/20250313130251.383204-1-ubizjak@gmail.com

x86/fpu: Improve crypto performance by making kernel-mode FPU reliably usable in softirqs

2025-03-06T11:44:09Z

Background: =========== Currently kernel-mode FPU is not always usable in softirq context on x86, since softirqs can nest inside a kernel-mode FPU section in task context, and nested use of kernel-mode FPU is not supported. Therefore, x86 SIMD-optimized code that can be called in softirq context has to sometimes fall back to non-SIMD code. There are two options for the fallback, both of which are pretty terrible: (a) Use a scalar fallback. This can be 10-100x slower than vectorized code because it cannot use specialized instructions like AES, SHA, or carryless multiplication. (b) Execute the request asynchronously using a kworker. In other words, use the "crypto SIMD helper" in crypto/simd.c. Currently most of the x86 en/decryption code (skcipher and aead algorithms) uses option (b), since this avoids the slow scalar fallback and it is easier to wire up. But option (b) is still really bad for its own reasons: - Punting the request to a kworker is bad for performance too. - It forces the algorithm to be marked as asynchronous (CRYPTO_ALG_ASYNC), preventing it from being used by crypto API users who request a synchronous algorithm. That's another huge performance problem, which is especially unfortunate for users who don't even do en/decryption in softirq context. - It makes all en/decryption operations take a detour through crypto/simd.c. That involves additional checks and an additional indirect call, which slow down en/decryption for *everyone*. Fortunately, the skcipher and aead APIs are only usable in task and softirq context in the first place. Thus, if kernel-mode FPU were to be reliably usable in softirq context, no fallback would be needed. Indeed, other architectures such as arm, arm64, and riscv have already done this. Changes implemented: ==================== Therefore, this patch updates x86 accordingly to reliably support kernel-mode FPU in softirqs. This is done by just disabling softirq processing in kernel-mode FPU sections (when hardirqs are not already disabled), as that prevents the nesting that was problematic. This will delay some softirqs slightly, but only ones that would have otherwise been nested inside a task context kernel-mode FPU section. Any such softirqs would have taken the slow fallback path before if they tried to do any en/decryption. Now these softirqs will just run at the end of the task context kernel-mode FPU section (since local_bh_enable() runs pending softirqs) and will no longer take the slow fallback path. Alternatives considered: ======================== - Make kernel-mode FPU sections fully preemptible. This would require growing task_struct by another struct fpstate which is more than 2K. - Make softirqs save/restore the kernel-mode FPU state to a per-CPU struct fpstate when nested use is detected. Somewhat interesting, but seems unnecessary when a simpler solution exists. Performance results: ==================== I did some benchmarks with AES-XTS encryption of 16-byte messages (which is unrealistically small, but this makes it easier to see the overhead of kernel-mode FPU...). The baseline was 384 MB/s. Removing the use of crypto/simd.c, which this work makes possible, increases it to 487 MB/s, a +27% improvement in throughput. CPU was AMD Ryzen 9 9950X (Zen 5). No debugging options were enabled. [ mingo: Prettified the changelog and added performance results. ] Signed-off-by: Eric Biggers Signed-off-by: Ingo Molnar Cc: Andy Lutomirski Cc: H. Peter Anvin Cc: Linus Torvalds Cc: Oleg Nesterov Cc: Uros Bizjak Link: https://lore.kernel.org/r/20250304204954.3901-1-ebiggers@kernel.org

x86/fpu/xstate: Simplify print_xstate_features()

2025-02-27T18:54:41Z

print_xstate_features() currently invokes print_xstate_feature() multiple times on separate lines, which can be simplified in a loop. print_xstate_feature() already checks the feature's enabled status and is only called within print_xstate_features(). Inline print_xstate_feature() and iterate over features in a loop to streamline the enabling message. No functional changes. Signed-off-by: Chang S. Bae Signed-off-by: Ingo Molnar Reviewed-by: Dave Hansen Link: https://lore.kernel.org/r/20250227184502.10288-2-chang.seok.bae@intel.com

x86/fpu: Refine and simplify the magic number check during signal return

2025-02-27T18:38:06Z

Before restoring xstate from the user space buffer, the kernel performs sanity checks on these magic numbers: magic1 in the software reserved area, and magic2 at the end of XSAVE region. The position of magic2 is calculated based on the xstate size derived from the user space buffer. But, the in-kernel record is directly available and reliable for this purpose. This reliance on user space data is also inconsistent with the recent fix in: d877550eaf2d ("x86/fpu: Stop relying on userspace for info to fault in xsave buffer") Simply use fpstate->user_size, and then get rid of unnecessary size-evaluation code. Signed-off-by: Chang S. Bae Signed-off-by: Ingo Molnar Cc: Andy Lutomirski Cc: H. Peter Anvin Cc: Linus Torvalds Cc: Oleg Nesterov Cc: Dave Hansen Link: https://lore.kernel.org/r/20241211014500.3738-1-chang.seok.bae@intel.com

x86/fpu: Fix guest FPU state buffer allocation size

2025-02-21T13:47:26Z

Ongoing work on an optimization to batch-preallocate vCPU state buffers for KVM revealed a mismatch between the allocation sizes used in fpu_alloc_guest_fpstate() and fpstate_realloc(). While the former allocates a buffer sized to fit the default set of XSAVE features in UABI form (as per fpu_user_cfg), the latter uses its ksize argument derived (for the requested set of features) in the same way as the sizes found in fpu_kernel_cfg, i.e. using the compacted in-kernel representation. The correct size to use for guest FPU state should indeed be the kernel one as seen in fpstate_realloc(). The original issue likely went unnoticed through a combination of UABI size typically being larger than or equal to kernel size, and/or both amounting to the same number of allocated 4K pages. Fixes: 69f6ed1d14c6 ("x86/fpu: Provide infrastructure for KVM FPU cleanup") Signed-off-by: Stanislav Spassov Signed-off-by: Ingo Molnar Link: https://lore.kernel.org/r/20250218141045.85201-1-stanspas@amazon.de