kernel.git/arch/x86/crypto, branch linux-6.10.y

crypto: x86/sha256 - Add parentheses around macros' single arguments

2024-10-10T10:00:13Z

[ Upstream commit 3363c460ef726ba693704dbcd73b7e7214ccc788 ] The macros FOUR_ROUNDS_AND_SCHED and DO_4ROUNDS rely on an unexpected/undocumented behavior of the GNU assembler, which might change in the future (https://sourceware.org/bugzilla/show_bug.cgi?id=32073). M (1) (2) // 1 arg !? Future: 2 args M 1 + 2 // 1 arg !? Future: 3 args M 1 2 // 2 args Add parentheses around the single arguments to support future GNU assembler and LLVM integrated assembler (when the IsOperator hack from the following link is dropped). Link: https://github.com/llvm/llvm-project/commit/055006475e22014b28a070db1bff41ca15f322f0 Signed-off-by: Fangrui Song Reviewed-by: Jan Beulich Signed-off-by: Herbert Xu Signed-off-by: Sasha Levin

crypto: x86/aes-xts - switch to new Intel CPU model defines

2024-05-22T09:10:48Z

New CPU #defines encode vendor and family as well as model. Signed-off-by: Tony Luck Signed-off-by: Borislav Petkov (AMD) Reviewed-by: Eric Biggers Acked-by: Herbert Xu Link: https://lore.kernel.org/r/20240520224620.9480-2-tony.luck@intel.com

crypto: x86/aes-gcm - simplify GCM hash subkey derivation

2024-04-26T09:26:10Z

Remove a redundant expansion of the AES key, and use rodata for zeroes. Also rename rfc4106_set_hash_subkey() to aes_gcm_derive_hash_subkey() because it's used for both versions of AES-GCM, not just RFC4106. Signed-off-by: Eric Biggers Signed-off-by: Herbert Xu

crypto: x86/aes-gcm - delete unused GCM assembly code

2024-04-26T09:26:10Z

Delete aesni_gcm_enc() and aesni_gcm_dec() because they are unused. Only the incremental AES-GCM functions (aesni_gcm_init(), aesni_gcm_enc_update(), aesni_gcm_finalize()) are actually used. This saves 17 KB of object code. Signed-off-by: Eric Biggers Signed-off-by: Herbert Xu

crypto: x86/aes-xts - simplify loop in xts_crypt_slowpath()

2024-04-26T09:26:10Z

Since the total length processed by the loop in xts_crypt_slowpath() is a multiple of AES_BLOCK_SIZE, just round the length down to AES_BLOCK_SIZE even on the last step. This doesn't change behavior, as the last step will process a multiple of AES_BLOCK_SIZE regardless. Signed-off-by: Eric Biggers Signed-off-by: Herbert Xu

crypto: x86/aes-xts - optimize size of instructions operating on lengths

2024-04-19T10:54:19Z

x86_64 has the "interesting" property that the instruction size is generally a bit shorter for instructions that operate on the 32-bit (or less) part of registers, or registers that are in the original set of 8. This patch adjusts the AES-XTS code to take advantage of that property by changing the LEN parameter from size_t to unsigned int (which is all that's needed and is what the non-AVX implementation uses) and using the %eax register for KEYLEN. This decreases the size of aes-xts-avx-x86_64.o by 1.2%. Note that changing the kmovq to kmovd was going to be needed anyway to make the AVX10/256 code really work on CPUs that don't support 512-bit vectors (since the AVX10 spec says that 64-bit opmask instructions will only be supported on processors that support 512-bit vectors). Signed-off-by: Eric Biggers Signed-off-by: Herbert Xu

crypto: x86/aes-xts - eliminate a few more instructions

2024-04-19T10:54:19Z

- For conditionally subtracting 16 from LEN when decrypting a message whose length isn't a multiple of 16, use the cmovnz instruction. - Fold the addition of 4*VL to LEN into the sub of VL or 16 from LEN. - Remove an unnecessary test instruction. This results in slightly shorter code, both source and binary. Signed-off-by: Eric Biggers Signed-off-by: Herbert Xu

crypto: x86/aes-xts - handle AES-128 and AES-192 more efficiently

2024-04-19T10:54:19Z

Decrease the amount of code specific to the different AES variants by "right-aligning" the sequence of round keys, and for AES-128 and AES-192 just skipping irrelevant rounds at the beginning. This shrinks the size of aes-xts-avx-x86_64.o by 13.3%, and it improves the efficiency of AES-128 and AES-192. The tradeoff is that for AES-256 some additional not-taken conditional jumps are now executed. But these are predicted well and are cheap on x86. Note that the ARMv8 CE based AES-XTS implementation uses a similar strategy to handle the different AES variants. Signed-off-by: Eric Biggers Signed-off-by: Herbert Xu

crypto: x86/aesni-xts - deduplicate aesni_xts_enc() and aesni_xts_dec()

2024-04-19T10:54:19Z

Since aesni_xts_enc() and aesni_xts_dec() are very similar, generate them from a macro that's passed an argument enc=1 or enc=0. This reduces the length of aesni-intel_asm.S by 112 lines while still producing the exact same object file in both 32-bit and 64-bit mode. Signed-off-by: Eric Biggers Reviewed-by: Ard Biesheuvel Signed-off-by: Herbert Xu

crypto: x86/aes-xts - handle CTS encryption more efficiently

2024-04-19T10:54:19Z

When encrypting a message whose length isn't a multiple of 16 bytes, encrypt the last full block in the main loop. This works because only decryption uses the last two tweaks in reverse order, not encryption. This improves the performance of decrypting messages whose length isn't a multiple of the AES block length, shrinks the size of aes-xts-avx-x86_64.o by 5.0%, and eliminates two instructions (a test and a not-taken conditional jump) when encrypting a message whose length *is* a multiple of the AES block length. While it's not super useful to optimize for ciphertext stealing given that it's rarely needed in practice, the other two benefits mentioned above make this optimization worthwhile. Signed-off-by: Eric Biggers Signed-off-by: Herbert Xu