summaryrefslogtreecommitdiffstats
path: root/lib/crypto
Commit message (Collapse)AuthorAgeFilesLines
* lib/crypto: blake2s: move hmac construction into wireguardJason A. Donenfeld2022-01-182-68/+0
| | | | | | | | | | | | | | | Basically nobody should use blake2s in an HMAC construction; it already has a keyed variant. But unfortunately for historical reasons, Noise, used by WireGuard, uses HKDF quite strictly, which means we have to use this. Because this really shouldn't be used by others, this commit moves it into wireguard's noise.c locally, so that kernels that aren't using WireGuard don't get this superfluous code baked in. On m68k systems, this shaves off ~314 bytes. Cc: Herbert Xu <herbert@gondor.apana.org.au> Tested-by: Geert Uytterhoeven <geert@linux-m68k.org> Acked-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* lib/crypto: add prompts back to crypto librariesJustin M. Forbes2022-01-181-5/+12
| | | | | | | | | | | | | | | | Commit 6048fdcc5f269 ("lib/crypto: blake2s: include as built-in") took away a number of prompt texts from other crypto libraries. This makes values flip from built-in to module when oldconfig runs, and causes problems when these crypto libs need to be built in for thingslike BIG_KEYS. Fixes: 6048fdcc5f269 ("lib/crypto: blake2s: include as built-in") Cc: Herbert Xu <herbert@gondor.apana.org.au> Cc: linux-crypto@vger.kernel.org Signed-off-by: Justin M. Forbes <jforbes@fedoraproject.org> [Jason: - moved menu into submenu of lib/ instead of root menu - fixed chacha sub-dependencies for CONFIG_CRYPTO] Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* lib/crypto: blake2s: include as built-inJason A. Donenfeld2022-01-074-29/+15
| | | | | | | | | | | | | | | | | | In preparation for using blake2s in the RNG, we change the way that it is wired-in to the build system. Instead of using ifdefs to select the right symbol, we use weak symbols. And because ARM doesn't need the generic implementation, we make the generic one default only if an arch library doesn't need it already, and then have arch libraries that do need it opt-in. So that the arch libraries can remain tristate rather than bool, we then split the shash part from the glue code. Acked-by: Herbert Xu <herbert@gondor.apana.org.au> Acked-by: Ard Biesheuvel <ardb@kernel.org> Acked-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Masahiro Yamada <masahiroy@kernel.org> Cc: linux-kbuild@vger.kernel.org Cc: linux-crypto@vger.kernel.org Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* crypto: sm4 - Do not change section of ck and sboxNathan Chancellor2021-09-171-2/+2
| | | | | | | | | | | | | | | | | | | | | | When building with clang and GNU as, there is a warning about ignored changed section attributes: /tmp/sm4-c916c8.s: Assembler messages: /tmp/sm4-c916c8.s:677: Warning: ignoring changed section attributes for .data..cacheline_aligned "static const" places the data in .rodata but __cacheline_aligned has the section attribute to place it in .data..cacheline_aligned, in addition to the aligned attribute. To keep the alignment but avoid attempting to change sections, use the ____cacheline_aligned attribute, which is just the aligned attribute. Fixes: 2b31277af577 ("crypto: sm4 - create SM4 library based on sm4 generic code") Link: https://github.com/ClangBuiltLinux/linux/issues/1441 Signed-off-by: Nathan Chancellor <nathan@kernel.org> Reviewed-by: Tianjia Zhang <tianjia.zhang@linux.alibaba.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: arm64/sm4-ce - Make dependent on sm4 library instead of sm4-genericTianjia Zhang2021-07-301-1/+1
| | | | | | | | | SM4 library is abstracted from sm4-generic algorithm, sm4-ce can depend on the SM4 library instead of sm4-generic, and some functions in sm4-generic do not need to be exported. Signed-off-by: Tianjia Zhang <tianjia.zhang@linux.alibaba.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: sm4 - create SM4 library based on sm4 generic codeTianjia Zhang2021-07-303-0/+182
| | | | | | | | | | | | | | | Take the existing small footprint and mostly time invariant C code and turn it into a SM4 library that can be used for non-performance critical, casual use of SM4, and as a fallback for, e.g., SIMD code that needs a secondary path that can be taken in contexts where the SIMD unit is off limits. Secondly, some codes have been optimized, such as unrolling small times loop, removing unnecessary memory shifts, exporting sbox, fk, ck arrays, and basic encryption and decryption functions. Signed-off-by: Tianjia Zhang <tianjia.zhang@linux.alibaba.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: lib - rename 'mod_init' & 'mod_exit' functions to be module-specificRandy Dunlap2021-07-163-12/+12
| | | | | | | | | | | | | | | | | Rename module_init & module_exit functions that are named "mod_init" and "mod_exit" so that they are unique in both the System.map file and in initcall_debug output instead of showing up as almost anonymous "mod_init". This is helpful for debugging and in determining how long certain module_init calls take to execute. Signed-off-by: Randy Dunlap <rdunlap@infradead.org> Cc: Herbert Xu <herbert@gondor.apana.org.au> Cc: "David S. Miller" <davem@davemloft.net> Cc: linux-crypto@vger.kernel.org Cc: Jason A. Donenfeld <Jason@zx2c4.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: doc - fix kernel-doc notation in chacha.c and af_alg.cRandy Dunlap2021-04-221-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fix function name in chacha.c kernel-doc comment to remove a warning. Convert af_alg.c to kernel-doc notation to eliminate many kernel-doc warnings. ../lib/crypto/chacha.c:77: warning: expecting prototype for chacha_block(). Prototype was for chacha_block_generic() instead chacha.c:104: warning: Excess function parameter 'out' description in 'hchacha_block_generic' af_alg.c:498: warning: Function parameter or member 'sk' not described in 'af_alg_alloc_tsgl' ../crypto/af_alg.c:539: warning: expecting prototype for aead_count_tsgl(). Prototype was for af_alg_count_tsgl() instead ../crypto/af_alg.c:596: warning: expecting prototype for aead_pull_tsgl(). Prototype was for af_alg_pull_tsgl() instead af_alg.c:663: warning: Function parameter or member 'areq' not described in 'af_alg_free_areq_sgls' af_alg.c:700: warning: Function parameter or member 'sk' not described in 'af_alg_wait_for_wmem' af_alg.c:700: warning: Function parameter or member 'flags' not described in 'af_alg_wait_for_wmem' af_alg.c:731: warning: Function parameter or member 'sk' not described in 'af_alg_wmem_wakeup' af_alg.c:757: warning: Function parameter or member 'sk' not described in 'af_alg_wait_for_data' af_alg.c:757: warning: Function parameter or member 'flags' not described in 'af_alg_wait_for_data' af_alg.c:757: warning: Function parameter or member 'min' not described in 'af_alg_wait_for_data' af_alg.c:796: warning: Function parameter or member 'sk' not described in 'af_alg_data_wakeup' af_alg.c:832: warning: Function parameter or member 'sock' not described in 'af_alg_sendmsg' af_alg.c:832: warning: Function parameter or member 'msg' not described in 'af_alg_sendmsg' af_alg.c:832: warning: Function parameter or member 'size' not described in 'af_alg_sendmsg' af_alg.c:832: warning: Function parameter or member 'ivsize' not described in 'af_alg_sendmsg' af_alg.c:985: warning: Function parameter or member 'sock' not described in 'af_alg_sendpage' af_alg.c:985: warning: Function parameter or member 'page' not described in 'af_alg_sendpage' af_alg.c:985: warning: Function parameter or member 'offset' not described in 'af_alg_sendpage' af_alg.c:985: warning: Function parameter or member 'size' not described in 'af_alg_sendpage' af_alg.c:985: warning: Function parameter or member 'flags' not described in 'af_alg_sendpage' af_alg.c:1040: warning: Function parameter or member 'areq' not described in 'af_alg_free_resources' af_alg.c:1059: warning: Function parameter or member '_req' not described in 'af_alg_async_cb' af_alg.c:1059: warning: Function parameter or member 'err' not described in 'af_alg_async_cb' af_alg.c:1083: warning: Function parameter or member 'file' not described in 'af_alg_poll' af_alg.c:1083: warning: Function parameter or member 'sock' not described in 'af_alg_poll' af_alg.c:1083: warning: Function parameter or member 'wait' not described in 'af_alg_poll' af_alg.c:1114: warning: Function parameter or member 'sk' not described in 'af_alg_alloc_areq' af_alg.c:1114: warning: Function parameter or member 'areqlen' not described in 'af_alg_alloc_areq' af_alg.c:1146: warning: Function parameter or member 'sk' not described in 'af_alg_get_rsgl' af_alg.c:1146: warning: Function parameter or member 'msg' not described in 'af_alg_get_rsgl' af_alg.c:1146: warning: Function parameter or member 'flags' not described in 'af_alg_get_rsgl' af_alg.c:1146: warning: Function parameter or member 'areq' not described in 'af_alg_get_rsgl' af_alg.c:1146: warning: Function parameter or member 'maxsize' not described in 'af_alg_get_rsgl' af_alg.c:1146: warning: Function parameter or member 'outlen' not described in 'af_alg_get_rsgl' Signed-off-by: Randy Dunlap <rdunlap@infradead.org> Cc: Herbert Xu <herbert@gondor.apana.org.au> Cc: "David S. Miller" <davem@davemloft.net> Cc: linux-crypto@vger.kernel.org Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: poly1305 - fix poly1305_core_setkey() declarationArnd Bergmann2021-04-023-3/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | gcc-11 points out a mismatch between the declaration and the definition of poly1305_core_setkey(): lib/crypto/poly1305-donna32.c:13:67: error: argument 2 of type ‘const u8[16]’ {aka ‘const unsigned char[16]’} with mismatched bound [-Werror=array-parameter=] 13 | void poly1305_core_setkey(struct poly1305_core_key *key, const u8 raw_key[16]) | ~~~~~~~~~^~~~~~~~~~~ In file included from lib/crypto/poly1305-donna32.c:11: include/crypto/internal/poly1305.h:21:68: note: previously declared as ‘const u8 *’ {aka ‘const unsigned char *’} 21 | void poly1305_core_setkey(struct poly1305_core_key *key, const u8 *raw_key); This is harmless in principle, as the calling conventions are the same, but the more specific prototype allows better type checking in the caller. Change the declaration to match the actual function definition. The poly1305_simd_init() is a bit suspicious here, as it previously had a 32-byte argument type, but looks like it needs to take the 16-byte POLY1305_BLOCK_SIZE array instead. Fixes: 1c08a104360f ("crypto: poly1305 - add new 32 and 64-bit generic versions") Signed-off-by: Arnd Bergmann <arnd@arndb.de> Reviewed-by: Ard Biesheuvel <ardb@kernel.org> Reviewed-by: Eric Biggers <ebiggers@google.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: lib/chacha20poly1305 - define empty module exit functionJason A. Donenfeld2021-01-221-0/+5
| | | | | | | | | | | | With no mod_exit function, users are unable to unload the module after use. I'm not aware of any reason why module unloading should be prohibited for this one, so this commit simply adds an empty exit function. Reported-and-tested-by: John Donnelly <john.p.donnelly@oracle.com> Acked-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: blake2s - move update and final logic to internal/blake2s.hEric Biggers2021-01-031-40/+8
| | | | | | | | | | | | Move most of blake2s_update() and blake2s_final() into new inline functions __blake2s_update() and __blake2s_final() in include/crypto/internal/blake2s.h so that this logic can be shared by the shash helper functions. This will avoid duplicating this logic between the library and shash implementations. Signed-off-by: Eric Biggers <ebiggers@google.com> Acked-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: lib/blake2s - Move selftest prototype into header fileHerbert Xu2020-12-042-3/+1
| | | | | | | This patch fixes a missing prototype warning on blake2s_selftest. Reported-by: kernel test robot <lkp@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: lib/curve25519 - Move selftest prototype into header fileHerbert Xu2020-11-201-2/+0
| | | | | | | | | This patch moves the curve25519_selftest into curve25519.h so we don't get a warning from gcc complaining about a missing prototype. Reported-by: kernel test robot <lkp@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: sha - split sha.h into sha1.h and sha2.hEric Biggers2020-11-201-1/+1
| | | | | | | | | | | | | | | | | | | | | | | Currently <crypto/sha.h> contains declarations for both SHA-1 and SHA-2, and <crypto/sha3.h> contains declarations for SHA-3. This organization is inconsistent, but more importantly SHA-1 is no longer considered to be cryptographically secure. So to the extent possible, SHA-1 shouldn't be grouped together with any of the other SHA versions, and usage of it should be phased out. Therefore, split <crypto/sha.h> into two headers <crypto/sha1.h> and <crypto/sha2.h>, and make everyone explicitly specify whether they want the declarations for SHA-1, SHA-2, or both. This avoids making the SHA-1 declarations visible to files that don't want anything to do with SHA-1. It also prepares for potentially moving sha1.h into a new insecure/ or dangerous/ directory. Signed-off-by: Eric Biggers <ebiggers@google.com> Acked-by: Ard Biesheuvel <ardb@kernel.org> Acked-by: Jason A. Donenfeld <Jason@zx2c4.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: lib/sha256 - Unroll LOAD and BLEND loopsArvind Sankar2020-10-301-4/+20
| | | | | | | | | | Unrolling the LOAD and BLEND loops improves performance by ~8% on x86_64 (tested on Broadwell Xeon) while not increasing code size too much. Signed-off-by: Arvind Sankar <nivedita@alum.mit.edu> Reviewed-by: Eric Biggers <ebiggers@google.com> Acked-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: lib/sha256 - Unroll SHA256 loop 8 times intead of 64Arvind Sankar2020-10-301-136/+38
| | | | | | | | | | | | This reduces code size substantially (on x86_64 with gcc-10 the size of sha256_update() goes from 7593 bytes to 1952 bytes including the new SHA256_K array), and on x86 is slightly faster than the full unroll (tested on Broadwell Xeon). Signed-off-by: Arvind Sankar <nivedita@alum.mit.edu> Reviewed-by: Eric Biggers <ebiggers@google.com> Acked-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: lib/sha256 - Clear W[] in sha256_update() instead of sha256_transform()Arvind Sankar2020-10-301-6/+5
| | | | | | | | | | | | | | The temporary W[] array is currently zeroed out once every call to sha256_transform(), i.e. once every 64 bytes of input data. Moving it to sha256_update() instead so that it is cleared only once per update can save about 2-3% of the total time taken to compute the digest, with a reasonable memset() implementation, and considerably more (~20%) with a bad one (eg the x86 purgatory currently uses a memset() coded in C). Signed-off-by: Arvind Sankar <nivedita@alum.mit.edu> Reviewed-by: Eric Biggers <ebiggers@google.com> Acked-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: lib/sha256 - Don't clear temporary variablesArvind Sankar2020-10-301-1/+0
| | | | | | | | | | | | | | | | | The assignments to clear a through h and t1/t2 are optimized out by the compiler because they are unused after the assignments. Clearing individual scalar variables is unlikely to be useful, as they may have been assigned to registers, and even if stack spilling was required, there may be compiler-generated temporaries that are impossible to clear in any case. So drop the clearing of a through h and t1/t2. Signed-off-by: Arvind Sankar <nivedita@alum.mit.edu> Reviewed-by: Eric Biggers <ebiggers@google.com> Acked-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: lib/sha256 - Use memzero_explicit() for clearing stateArvind Sankar2020-10-301-1/+1
| | | | | | | | | | | | | Without the barrier_data() inside memzero_explicit(), the compiler may optimize away the state-clearing if it can tell that the state is not used afterwards. At least in lib/crypto/sha256.c:__sha256_final(), the function can get inlined into sha256(), in which case the memset is optimized away. Signed-off-by: Arvind Sankar <nivedita@alum.mit.edu> Reviewed-by: Eric Biggers <ebiggers@google.com> Acked-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: lib/chacha20poly1305 - Set SG_MITER_ATOMIC unconditionallyHerbert Xu2020-09-251-3/+1
| | | | | | | | | | | | | | There is no reason for the chacha20poly1305 SG miter code to use kmap instead of kmap_atomic as the critical section doesn't sleep anyway. So we can simply get rid of the preemptible check and set SG_MITER_ATOMIC unconditionally. Even if we need to reenable preemption to lower latency we should be doing that by interrupting the SG miter walk rather than using kmap. Reported-by: Linus Torvalds <torvalds@linux-foundation.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: lib/sha256 - add sha256() functionEric Biggers2020-07-161-0/+10
| | | | | | | | | | | | Add a function sha256() which computes a SHA-256 digest in one step, combining sha256_init() + sha256_update() + sha256_final(). This is similar to how we also have blake2s(). Reviewed-by: Ard Biesheuvel <ardb@kernel.org> Tested-by: Hans de Goede <hdegoede@redhat.com> Signed-off-by: Eric Biggers <ebiggers@google.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: lib/chacha20poly1305 - Add missing function declarationHerbert Xu2020-07-161-2/+0
| | | | | | | This patch adds a declaration for chacha20poly1305_selftest to silence a sparse warning. Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: lib/sha1 - remove unnecessary includes of linux/cryptohash.hEric Biggers2020-05-081-1/+0
| | | | | | | | | | | | | | | <linux/cryptohash.h> sounds very generic and important, like it's the header to include if you're doing cryptographic hashing in the kernel. But actually it only includes the library implementation of the SHA-1 compression function (not even the full SHA-1). This should basically never be used anymore; SHA-1 is no longer considered secure, and there are much better ways to do cryptographic hashing in the kernel. Most files that include this header don't actually need it. So in preparation for removing it, remove all these unneeded includes of it. Signed-off-by: Eric Biggers <ebiggers@google.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: lib/sha256 - return voidEric Biggers2020-05-081-12/+8
| | | | | | | | | | | | The SHA-256 / SHA-224 library functions can't fail, so remove the useless return value. Also long as the declarations are being changed anyway, also fix some parameter names in the declarations to match the definitions. Signed-off-by: Eric Biggers <ebiggers@google.com> Reviewed-by: Jason A. Donenfeld <Jason@zx2c4.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: arm64/chacha - correctly walk through blocksJason A. Donenfeld2020-03-201-3/+8
| | | | | | | | | | | | | | | | | | | | Prior, passing in chunks of 2, 3, or 4, followed by any additional chunks would result in the chacha state counter getting out of sync, resulting in incorrect encryption/decryption, which is a pretty nasty crypto vuln: "why do images look weird on webpages?" WireGuard users never experienced this prior, because we have always, out of tree, used a different crypto library, until the recent Frankenzinc addition. This commit fixes the issue by advancing the pointers and state counter by the actual size processed. It also fixes up a bug in the (optional, costly) stride test that prevented it from running on arm64. Fixes: b3aad5bad26a ("crypto: arm64/chacha - expose arm64 ChaCha routine as library function") Reported-and-tested-by: Emil Renner Berthing <kernel@esmil.dk> Cc: Ard Biesheuvel <ardb@kernel.org> Cc: stable@vger.kernel.org # v5.5+ Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> Reviewed-by: Eric Biggers <ebiggers@google.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: chacha20poly1305 - prevent integer overflow on large inputJason A. Donenfeld2020-02-141-0/+3
| | | | | | | | | | | | | | | | | | This code assigns src_len (size_t) to sl (int), which causes problems when src_len is very large. Probably nobody in the kernel should be passing this much data to chacha20poly1305 all in one go anyway, so I don't think we need to change the algorithm or introduce larger types or anything. But we should at least error out early in this case and print a warning so that we get reports if this does happen and can look into why anybody is possibly passing it that much data or if they're accidently passing -1 or similar. Fixes: d95312a3ccc0 ("crypto: lib/chacha20poly1305 - reimplement crypt_from_sg() routine") Cc: Ard Biesheuvel <ardb@kernel.org> Cc: stable@vger.kernel.org # 5.5+ Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> Acked-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: chacha20poly1305 - add back missing test vectors and test chunkingJason A. Donenfeld2020-01-221-14/+1698
| | | | | | | | | | | | | | | | When this was originally ported, the 12-byte nonce vectors were left out to keep things simple. I agree that we don't need nor want a library interface for 12-byte nonces. But these test vectors were specially crafted to look at issues in the underlying primitives and related interactions. Therefore, we actually want to keep around all of the test vectors, and simply have a helper function to test them with. Secondly, the sglist-based chunking code in the library interface is rather complicated, so this adds a developer-only test for ensuring that all the book keeping is correct, across a wide array of possibilities. Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: curve25519 - Fix selftest build errorHerbert Xu2020-01-163-10/+30
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | If CRYPTO_CURVE25519 is y, CRYPTO_LIB_CURVE25519_GENERIC will be y, but CRYPTO_LIB_CURVE25519 may be set to m, this causes build errors: lib/crypto/curve25519-selftest.o: In function `curve25519': curve25519-selftest.c:(.text.unlikely+0xc): undefined reference to `curve25519_arch' lib/crypto/curve25519-selftest.o: In function `curve25519_selftest': curve25519-selftest.c:(.init.text+0x17e): undefined reference to `curve25519_base_arch' This is because the curve25519 self-test code is being controlled by the GENERIC option rather than the overall CURVE25519 option, as is the case with blake2s. To recap, the GENERIC and ARCH options for CURVE25519 are internal only and selected by users such as the Crypto API, or the externally visible CURVE25519 option which in turn is selected by wireguard. The self-test is specific to the the external CURVE25519 option and should not be enabled by the Crypto API. This patch fixes this by splitting the GENERIC module from the CURVE25519 module with the latter now containing just the self-test. Reported-by: Hulk Robot <hulkci@huawei.com> Fixes: aa127963f1ca ("crypto: lib/curve25519 - re-add selftests") Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au> Reviewed-by: Jason A. Donenfeld <Jason@zx2c4.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: x86/poly1305 - wire up faster implementations for kernelJason A. Donenfeld2020-01-161-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | These x86_64 vectorized implementations support AVX, AVX-2, and AVX512F. The AVX-512F implementation is disabled on Skylake, due to throttling, but it is quite fast on >= Cannonlake. On the left is cycle counts on a Core i7 6700HQ using the AVX-2 codepath, comparing this implementation ("new") to the implementation in the current crypto api ("old"). On the right are benchmarks on a Xeon Gold 5120 using the AVX-512 codepath. The new implementation is faster on all benchmarks. AVX-2 AVX-512 --------- ----------- size old new size old new ---- ---- ---- ---- ---- ---- 0 70 68 0 74 70 16 92 90 16 96 92 32 134 104 32 136 106 48 172 120 48 184 124 64 218 136 64 218 138 80 254 158 80 260 160 96 298 174 96 300 176 112 342 192 112 342 194 128 388 212 128 384 212 144 428 228 144 420 226 160 466 246 160 464 248 176 510 264 176 504 264 192 550 282 192 544 282 208 594 302 208 582 300 224 628 316 224 624 318 240 676 334 240 662 338 256 716 354 256 708 358 272 764 374 272 748 372 288 802 352 288 788 358 304 420 366 304 422 370 320 428 360 320 432 364 336 484 378 336 486 380 352 426 384 352 434 390 368 478 400 368 480 408 384 488 394 384 490 398 400 542 408 400 542 412 416 486 416 416 492 426 432 534 430 432 538 436 448 544 422 448 546 432 464 600 438 464 600 448 480 540 448 480 548 456 496 594 464 496 594 476 512 602 456 512 606 470 528 656 476 528 656 480 544 600 480 544 606 498 560 650 494 560 652 512 576 664 490 576 662 508 592 714 508 592 716 522 608 656 514 608 664 538 624 708 532 624 710 552 640 716 524 640 720 516 656 770 536 656 772 526 672 716 548 672 722 544 688 770 562 688 768 556 704 774 552 704 778 556 720 826 568 720 832 568 736 768 574 736 780 584 752 822 592 752 826 600 768 830 584 768 836 560 784 884 602 784 888 572 800 828 610 800 838 588 816 884 628 816 884 604 832 888 618 832 894 598 848 942 632 848 946 612 864 884 644 864 896 628 880 936 660 880 942 644 896 948 652 896 952 608 912 1000 664 912 1004 616 928 942 676 928 954 634 944 994 690 944 1000 646 960 1002 680 960 1008 646 976 1054 694 976 1062 658 992 1002 706 992 1012 674 1008 1052 720 1008 1058 690 This commit wires in the prior implementation from Andy, and makes the following changes to be suitable for kernel land. - Some cosmetic and structural changes, like renaming labels to .Lname, constants, and other Linux conventions, as well as making the code easy for us to maintain moving forward. - CPU feature checking is done in C by the glue code. - We avoid jumping into the middle of functions, to appease objtool, and instead parameterize shared code. - We maintain frame pointers so that stack traces make sense. - We remove the dependency on the perl xlate code, which transforms the output into things that assemblers we don't care about use. Importantly, none of our changes affect the arithmetic or core code, but just involve the differing environment of kernel space. Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> Signed-off-by: Samuel Neves <sneves@dei.uc.pt> Co-developed-by: Samuel Neves <sneves@dei.uc.pt> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: poly1305 - add new 32 and 64-bit generic versionsJason A. Donenfeld2020-01-164-163/+399
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | These two C implementations from Zinc -- a 32x32 one and a 64x64 one, depending on the platform -- come from Andrew Moon's public domain poly1305-donna portable code, modified for usage in the kernel. The precomputation in the 32-bit version and the use of 64x64 multiplies in the 64-bit version make these perform better than the code it replaces. Moon's code is also very widespread and has received many eyeballs of scrutiny. There's a bit of interference between the x86 implementation, which relies on internal details of the old scalar implementation. In the next commit, the x86 implementation will be replaced with a faster one that doesn't rely on this, so none of this matters much. But for now, to keep this passing the tests, we inline the bits of the old implementation that the x86 implementation relied on. Also, since we now support a slightly larger key space, via the union, some offsets had to be fixed up. Nonce calculation was folded in with the emit function, to take advantage of 64x64 arithmetic. However, Adiantum appeared to rely on no nonce handling in emit, so this path was conditionalized. We also introduced a new struct, poly1305_core_key, to represent the precise amount of space that particular implementation uses. Testing with kbench9000, depending on the CPU, the update function for the 32x32 version has been improved by 4%-7%, and for the 64x64 by 19%-30%. The 32x32 gains are small, but I think there's great value in having a parallel implementation to the 64x64 one so that the two can be compared side-by-side as nice stand-alone units. Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: lib/curve25519 - re-add selftestsJason A. Donenfeld2019-12-273-0/+1339
| | | | | | | | | | Somehow these were dropped when Zinc was being integrated, which is problematic, because testing the library interface for Curve25519 is important.. This commit simply adds them back and wires them in in the same way that the blake2s selftests are wired in. Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: lib/chacha20poly1305 - use chacha20_crypt()Eric Biggers2019-11-221-8/+8
| | | | | | | | | | Use chacha20_crypt() instead of chacha_crypt(), since it's not really appropriate for users of the ChaCha library API to be passing the number of rounds as an argument. Signed-off-by: Eric Biggers <ebiggers@google.com> Acked-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: lib/chacha20poly1305 - reimplement crypt_from_sg() routineArd Biesheuvel2019-11-172-0/+195
| | | | | | | | | | | | | | | | | | | | | | | | Reimplement the library routines to perform chacha20poly1305 en/decryption on scatterlists, without [ab]using the [deprecated] blkcipher interface, which is rather heavyweight and does things we don't really need. Instead, we use the sg_miter API in a novel and clever way, to iterate over the scatterlist in-place (i.e., source == destination, which is the only way this library is expected to be used). That way, we don't have to iterate over two scatterlists in parallel. Another optimization is that, instead of relying on the blkcipher walker to present the input in suitable chunks, we recognize that ChaCha is a streamcipher, and so we can simply deal with partial blocks by keeping a block of cipherstream on the stack and use crypto_xor() to mix it with the in/output. Finally, we omit the scatterwalk_and_copy() call if the last element of the scatterlist covers the MAC as well (which is the common case), avoiding the need to walk the scatterlist and kmap() the page twice. Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: chacha20poly1305 - import construction and selftest from ZincArd Biesheuvel2019-11-174-0/+7578
| | | | | | | | | | | | | | This incorporates the chacha20poly1305 from the Zinc library, retaining the library interface, but replacing the implementation with calls into the code that already existed in the kernel's crypto API. Note that this library API does not implement RFC7539 fully, given that it is limited to 64-bit nonces. (The 96-bit nonce version that was part of the selftest only has been removed, along with the 96-bit nonce test vectors that only tested the selftest but not the actual library itself) Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: lib/curve25519 - work around Clang stack spilling issueArd Biesheuvel2019-11-171-5/+5
| | | | | | | | | | | | | | | | | | Arnd reports that the 32-bit generic library code for Curve25119 ends up using an excessive amount of stack space when built with Clang: lib/crypto/curve25519-fiat32.c:756:6: error: stack frame size of 1384 bytes in function 'curve25519_generic' [-Werror,-Wframe-larger-than=] Let's give some hints to the compiler regarding which routines should not be inlined, to prevent it from running out of registers and spilling to the stack. The resulting code performs identically under both GCC and Clang, and makes the warning go away. Suggested-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: curve25519 - generic C library implementationsJason A. Donenfeld2019-11-175-0/+1707
| | | | | | | | | | | | | | | | | | | | | | | This contains two formally verified C implementations of the Curve25519 scalar multiplication function, one for 32-bit systems, and one for 64-bit systems whose compiler supports efficient 128-bit integer types. Not only are these implementations formally verified, but they are also the fastest available C implementations. They have been modified to be friendly to kernel space and to be generally less horrendous looking, but still an effort has been made to retain their formally verified characteristic, and so the C might look slightly unidiomatic. The 64-bit version comes from HACL*: https://github.com/project-everest/hacl-star The 32-bit version comes from Fiat: https://github.com/mit-plv/fiat-crypto Information: https://cr.yp.to/ecdh.html Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> [ardb: - move from lib/zinc to lib/crypto - replace .c #includes with Kconfig based object selection - drop simd handling and simplify support for per-arch versions ] Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: blake2s - generic C library implementation and selftestJason A. Donenfeld2019-11-175-0/+894
| | | | | | | | | | | | | | | | | | | | The C implementation was originally based on Samuel Neves' public domain reference implementation but has since been heavily modified for the kernel. We're able to do compile-time optimizations by moving some scaffolding around the final function into the header file. Information: https://blake2.net/ Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> Signed-off-by: Samuel Neves <sneves@dei.uc.pt> Co-developed-by: Samuel Neves <sneves@dei.uc.pt> [ardb: - move from lib/zinc to lib/crypto - remove simd handling - rewrote selftest for better coverage - use fixed digest length for blake2s_hmac() and rename to blake2s256_hmac() ] Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: mips/poly1305 - incorporate OpenSSL/CRYPTOGAMS optimized implementationArd Biesheuvel2019-11-171-0/+1
| | | | | | | | | | | | | | | | | | This is a straight import of the OpenSSL/CRYPTOGAMS Poly1305 implementation for MIPS authored by Andy Polyakov, a prior 64-bit only version of which has been contributed by him to the OpenSSL project. The file 'poly1305-mips.pl' is taken straight from this upstream GitHub repository [0] at commit d22ade312a7af958ec955620b0d241cf42c37feb, and already contains all the changes required to build it as part of a Linux kernel module. [0] https://github.com/dot-asm/cryptogams Co-developed-by: Andy Polyakov <appro@cryptogams.org> Signed-off-by: Andy Polyakov <appro@cryptogams.org> Co-developed-by: René van Dorst <opensource@vdorst.com> Signed-off-by: René van Dorst <opensource@vdorst.com> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: arm/poly1305 - incorporate OpenSSL/CRYPTOGAMS NEON implementationArd Biesheuvel2019-11-171-1/+1
| | | | | | | | | | | | | | | | This is a straight import of the OpenSSL/CRYPTOGAMS Poly1305 implementation for NEON authored by Andy Polyakov, and contributed by him to the OpenSSL project. The file 'poly1305-armv4.pl' is taken straight from this upstream GitHub repository [0] at commit ec55a08dc0244ce570c4fc7cade330c60798952f, and already contains all the changes required to build it as part of a Linux kernel module. [0] https://github.com/dot-asm/cryptogams Co-developed-by: Andy Polyakov <appro@cryptogams.org> Signed-off-by: Andy Polyakov <appro@cryptogams.org> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: arm64/poly1305 - incorporate OpenSSL/CRYPTOGAMS NEON implementationArd Biesheuvel2019-11-171-0/+1
| | | | | | | | | | | | | | | | This is a straight import of the OpenSSL/CRYPTOGAMS Poly1305 implementation for NEON authored by Andy Polyakov, and contributed by him to the OpenSSL project. The file 'poly1305-armv8.pl' is taken straight from this upstream GitHub repository [0] at commit ec55a08dc0244ce570c4fc7cade330c60798952f, and already contains all the changes required to build it as part of a Linux kernel module. [0] https://github.com/dot-asm/cryptogams Co-developed-by: Andy Polyakov <appro@cryptogams.org> Signed-off-by: Andy Polyakov <appro@cryptogams.org> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: x86/poly1305 - expose existing driver as poly1305 libraryArd Biesheuvel2019-11-171-0/+1
| | | | | | | | | Implement the arch init/update/final Poly1305 library routines in the accelerated SIMD driver for x86 so they are accessible to users of the Poly1305 library interface as well. Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: poly1305 - expose init/update/final library interfaceArd Biesheuvel2019-11-172-0/+100
| | | | | | | | | | | Expose the existing generic Poly1305 code via a init/update/final library interface so that callers are not required to go through the crypto API's shash abstraction to access it. At the same time, make some preparations so that the library implementation can be superseded by an accelerated arch-specific version in the future. Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: poly1305 - move core routines into a separate libraryArd Biesheuvel2019-11-173-0/+164
| | | | | | | | | | | | | | Move the core Poly1305 routines shared between the generic Poly1305 shash driver and the Adiantum and NHPoly1305 drivers into a separate library so that using just this pieces does not pull in the crypto API pieces of the generic Poly1305 routine. In a subsequent patch, we will augment this generic library with init/update/final routines so that Poyl1305 algorithm can be used directly without the need for using the crypto API's shash abstraction. Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: chacha - move existing library code into lib/cryptoArd Biesheuvel2019-11-174-0/+180
| | | | | | | | | | | | | | | | | | | | Currently, our generic ChaCha implementation consists of a permute function in lib/chacha.c that operates on the 64-byte ChaCha state directly [and which is always included into the core kernel since it is used by the /dev/random driver], and the crypto API plumbing to expose it as a skcipher. In order to support in-kernel users that need the ChaCha streamcipher but have no need [or tolerance] for going through the abstractions of the crypto API, let's expose the streamcipher bits via a library API as well, in a way that permits the implementation to be superseded by an architecture specific one if provided. So move the streamcipher code into a separate module in lib/crypto, and expose the init() and crypt() routines to users of the library. Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: lib - tidy up lib/crypto Kconfig and MakefileArd Biesheuvel2019-11-172-8/+23
| | | | | | | | In preparation of introducing a set of crypto library interfaces, tidy up the Makefile and split off the Kconfig symbols into a separate file. Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: sha256 - Remove sha256/224_init code duplicationHans de Goede2019-09-051-32/+0
| | | | | | | | | | | | | | | lib/crypto/sha256.c and include/crypto/sha256_base.h define 99% identical functions to init a sha256_state struct for sha224 or sha256 use. This commit moves the functions from lib/crypto/sha256.c to include/crypto/sha.h (making them static inline) and makes the sha224/256_base_init static inline functions from include/crypto/sha256_base.h wrappers around the now also static inline include/crypto/sha.h functions. Signed-off-by: Hans de Goede <hdegoede@redhat.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: sha256 - Merge crypto/sha256.h into crypto/sha.hHans de Goede2019-09-051-1/+1
| | | | | | | | | The generic sha256 implementation from lib/crypto/sha256.c uses data structs defined in crypto/sha.h, so lets move the function prototypes there too. Signed-off-by: Hans de Goede <hdegoede@redhat.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: sha256 - Add missing MODULE_LICENSE() to lib/crypto/sha256.cHans de Goede2019-08-301-0/+3
| | | | | | | | lib/crypto/sha256.c / lib/crypto/libsha256.o may end up being a module, so it needs a MODULE_LICENSE() line, add this. Signed-off-by: Hans de Goede <hdegoede@redhat.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: sha256 - Add sha224 support to sha256 library codeHans de Goede2019-08-221-2/+35
| | | | | | | | | | Add sha224 support to the lib/crypto/sha256 library code. This will allow us to replace both the sha256 and sha224 parts of crypto/sha256_generic.c when we remove the code duplication in further patches in this series. Suggested-by: Eric Biggers <ebiggers@kernel.org> Signed-off-by: Hans de Goede <hdegoede@redhat.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: sha256 - Make lib/crypto/sha256.c suitable for generic useHans de Goede2019-08-222-0/+7
| | | | | | | | | | | | | Before this commit lib/crypto/sha256.c has only been used in the s390 and x86 purgatory code, make it suitable for generic use: * Export interesting symbols * Add -D__DISABLE_EXPORTS to CFLAGS_sha256.o for purgatory builds to avoid the exports for the purgatory builds * Add to lib/crypto/Makefile and crypto/Kconfig Signed-off-by: Hans de Goede <hdegoede@redhat.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-20 12:32:19 +0000