summaryrefslogtreecommitdiffstats
path: root/drivers/crypto/intel
Commit message (Collapse)AuthorAgeFilesLines
* crypto: qat - adf_get_etr_base() helperGiovanni Cabiddu2024-04-023-6/+12
| | | | | | | | | | | Add and use the new helper function adf_get_etr_base() which retrieves the virtual address of the ring bar. This will be used extensively when adding support for Live Migration. Signed-off-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Xin Zeng <xin.zeng@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: iaa - Fix nr_cpus < nr_iaa caseTom Zanussi2024-03-221-3/+7
| | | | | | | | | | | | | | If nr_cpus < nr_iaa, the calculated cpus_per_iaa will be 0, which causes a divide-by-0 in rebalance_wq_table(). Make sure cpus_per_iaa is 1 in that case, and also in the nr_iaa == 0 case, even though cpus_per_iaa is never used if nr_iaa == 0, for paranoia. Cc: <stable@vger.kernel.org> # v6.8+ Reported-by: Jerry Snitselaar <jsnitsel@redhat.com> Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: iaa - fix the missing CRYPTO_ALG_ASYNC in cra_flagsBarry Song2024-03-081-0/+1
| | | | | | | | | Add the missing CRYPTO_ALG_ASYNC flag since intel iaa driver works asynchronously. Signed-off-by: Barry Song <v-songbaohua@oppo.com> Acked-by: Tom Zanussi <tom.zanussi@linux.intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: iaa - Fix comp/decomp delay statisticsTom Zanussi2024-03-013-32/+13
| | | | | | | | | | | | | | The comp/decomp delay statistics currently have no callers; somehow they were dropped during refactoring. There originally were also two sets, one for the async algorithm, the other for the synchronous version. Because the synchronous algorithm was dropped, one set should be removed. To keep it consistent with the rest of the stats, and since there's no ambiguity, remove the acomp/adecomp versions. Also add back the callers. Reported-by: Rex Zhang <rex.zhang@intel.com> Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: iaa - Fix async_disable descriptor leakTom Zanussi2024-03-011-2/+2
| | | | | | | | | | The disable_async paths of iaa_compress/decompress() don't free idxd descriptors in the async_disable case. Currently this only happens in the testcases where req->dst is set to null. Add a test to free them in those paths. Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - make ring to service map common for QAT GEN4Damian Muszynski2024-02-245-112/+90
| | | | | | | | | | The function get_ring_to_svc_map() is present in both 420xx and 4xxx drivers. Rework the logic to make it generic to GEN4 devices and move it to qat_common/adf_gen4_hw_data.c. Signed-off-by: Damian Muszynski <damian.muszynski@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - fix ring to service map for dcc in 420xxDamian Muszynski2024-02-241-0/+8
| | | | | | | | | | | | | | | | If a device is configured for data compression chaining (dcc), half of the engines are loaded with the symmetric crypto image and the rest are loaded with the compression image. However, in such configuration all rings can handle compression requests. Fix the ring to service mapping so that when a device is configured for dcc, the ring to service mapping reports that all rings in a bank can be used for compression. Fixes: fcf60f4bcf54 ("crypto: qat - add support for 420xx devices") Signed-off-by: Damian Muszynski <damian.muszynski@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - fix ring to service map for dcc in 4xxxDamian Muszynski2024-02-241-0/+8
| | | | | | | | | | | | | | | | If a device is configured for data compression chaining (dcc), half of the engines are loaded with the symmetric crypto image and the rest are loaded with the compression image. However, in such configuration all rings can handle compression requests. Fix the ring to service mapping so that when a device is configured for dcc, the ring to service mapping reports that all rings in a bank can be used for compression. Fixes: a238487f7965 ("crypto: qat - fix ring to service map for QAT GEN4") Signed-off-by: Damian Muszynski <damian.muszynski@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - fix comment structureAdam Guerin2024-02-241-2/+2
| | | | | | | | | | | | | Move comment description to the same line as the function name. This is to fix the following warning when compiling the QAT driver using the clang compiler with CC=clang W=2: drivers/crypto/intel/qat/qat_common/qat_crypto.c:108: warning: missing initial short description on line: * qat_crypto_vf_dev_config() Signed-off-by: Adam Guerin <adam.guerin@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - remove unnecessary description from commentAdam Guerin2024-02-243-6/+2
| | | | | | | | | | | | | | Remove extra description from comments as it is not required. This is to fix the following warning when compiling the QAT driver using the clang compiler with CC=clang W=2: drivers/crypto/intel/qat/qat_common/adf_dev_mgr.c:65: warning: contents before sections drivers/crypto/intel/qat/qat_common/adf_isr.c:380: warning: contents before sections drivers/crypto/intel/qat/qat_common/adf_vf_isr.c:298: warning: contents before sections Signed-off-by: Adam Guerin <adam.guerin@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - remove double initialization of valueAdam Guerin2024-02-241-4/+2
| | | | | | | | | | | | | | | | | | Remove double initialization of the reg variable. This is to fix the following warning when compiling the QAT driver using clang scan-build: drivers/crypto/intel/qat/qat_common/adf_gen4_ras.c:1010:6: warning: Value stored to 'reg' during its initialization is never read [deadcode.DeadStores] 1010 | u32 reg = ADF_CSR_RD(csr, ADF_GEN4_SSMCPPERR); | ^~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ drivers/crypto/intel/qat/qat_common/adf_gen4_ras.c:1109:6: warning: Value stored to 'reg' during its initialization is never read [deadcode.DeadStores] 1109 | u32 reg = ADF_CSR_RD(csr, ADF_GEN4_SER_ERR_SSMSH); | ^~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Fixes: 99b1c9826e48 ("crypto: qat - count QAT GEN4 errors") Signed-off-by: Adam Guerin <adam.guerin@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - avoid division by zeroAdam Guerin2024-02-241-0/+3
| | | | | | | | | | | | | | | | | Check if delta_us is not zero and return -EINVAL if it is. delta_us is unlikely to be zero as there is a sleep between the reads of the two timestamps. This is to fix the following warning when compiling the QAT driver using clang scan-build: drivers/crypto/intel/qat/qat_common/adf_clock.c:87:9: warning: Division by zero [core.DivideZero] 87 | temp = DIV_ROUND_CLOSEST_ULL(temp, delta_us); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Fixes: e2980ba57e79 ("crypto: qat - add measure clock frequency") Signed-off-by: Adam Guerin <adam.guerin@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - removed unused macro in adf_cnv_dbgfs.cAdam Guerin2024-02-241-1/+0
| | | | | | | | | | | | | | | This macro was added but never used, remove it. This is to fix the following warning when compiling the QAT driver using the clang compiler with CC=clang W=2: drivers/crypto/intel/qat/qat_common/adf_cnv_dbgfs.c:19:9: warning: macro is not used [-Wunused-macros] 19 | #define CNV_SLICE_ERR_MASK GENMASK(7, 0) | ^ Fixes: d807f0240c71 ("crypto: qat - add cnv_errors debugfs file") Signed-off-by: Adam Guerin <adam.guerin@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - remove unused macros in qat_comp_alg.cAdam Guerin2024-02-241-9/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As a result of the removal of qat_zlib_deflate, some defines where not removed. Remove them. This is to fix the following warning when compiling the QAT driver using the clang compiler with CC=clang W=2: drivers/crypto/intel/qat/qat_common/qat_comp_algs.c:21:9: warning: macro is not used [-Wunused-macros] 21 | #define QAT_RFC_1950_CM_OFFSET 4 | ^ drivers/crypto/intel/qat/qat_common/qat_comp_algs.c:16:9: warning: macro is not used [-Wunused-macros] 16 | #define QAT_RFC_1950_HDR_SIZE 2 | ^ drivers/crypto/intel/qat/qat_common/qat_comp_algs.c:17:9: warning: macro is not used [-Wunused-macros] 17 | #define QAT_RFC_1950_FOOTER_SIZE 4 | ^ drivers/crypto/intel/qat/qat_common/qat_comp_algs.c:22:9: warning: macro is not used [-Wunused-macros] 22 | #define QAT_RFC_1950_DICT_MASK 0x20 | ^ drivers/crypto/intel/qat/qat_common/qat_comp_algs.c:18:9: warning: macro is not used [-Wunused-macros] 18 | #define QAT_RFC_1950_CM_DEFLATE 8 | ^ drivers/crypto/intel/qat/qat_common/qat_comp_algs.c:20:9: warning: macro is not used [-Wunused-macros] 20 | #define QAT_RFC_1950_CM_MASK 0x0f | ^ drivers/crypto/intel/qat/qat_common/qat_comp_algs.c:23:9: warning: macro is not used [-Wunused-macros] 23 | #define QAT_RFC_1950_COMP_HDR 0x785e | ^ drivers/crypto/intel/qat/qat_common/qat_comp_algs.c:19:9: warning: macro is not used [-Wunused-macros] 19 | #define QAT_RFC_1950_CM_DEFLATE_CINFO_32K 7 | ^ Fixes: e9dd20e0e5f6 ("crypto: qat - Remove zlib-deflate") Signed-off-by: Adam Guerin <adam.guerin@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - uninitialized variable in adf_hb_error_inject_write()Dan Carpenter2024-02-241-5/+6
| | | | | | | | | | | | | | | | | | There are a few issues in this code. If *ppos is non-zero then the first part of the buffer is not initialized. We never initialize the last character of the buffer. The return is not checked so it's possible that none of the buffer is initialized. This is debugfs code which is root only and the impact of these bugs is very small. However, it's still worth fixing. To fix this: 1) Check that *ppos is zero. 2) Use copy_from_user() instead of simple_write_to_buffer(). 3) Explicitly add a NUL terminator. Fixes: e2b67859ab6e ("crypto: qat - add heartbeat error simulator") Signed-off-by: Dan Carpenter <dan.carpenter@linaro.org> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - resolve race condition during AER recoveryDamian Muszynski2024-02-171-6/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | During the PCI AER system's error recovery process, the kernel driver may encounter a race condition with freeing the reset_data structure's memory. If the device restart will take more than 10 seconds the function scheduling that restart will exit due to a timeout, and the reset_data structure will be freed. However, this data structure is used for completion notification after the restart is completed, which leads to a UAF bug. This results in a KFENCE bug notice. BUG: KFENCE: use-after-free read in adf_device_reset_worker+0x38/0xa0 [intel_qat] Use-after-free read at 0x00000000bc56fddf (in kfence-#142): adf_device_reset_worker+0x38/0xa0 [intel_qat] process_one_work+0x173/0x340 To resolve this race condition, the memory associated to the container of the work_struct is freed on the worker if the timeout expired, otherwise on the function that schedules the worker. The timeout detection can be done by checking if the caller is still waiting for completion or not by using completion_done() function. Fixes: d8cba25d2c68 ("crypto: qat - Intel(R) QAT driver framework") Cc: <stable@vger.kernel.org> Signed-off-by: Damian Muszynski <damian.muszynski@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - change SLAs cleanup flow at shutdownDamian Muszynski2024-02-171-1/+19
| | | | | | | | | | | | | | | | | | | | | | | | | The implementation of the Rate Limiting (RL) feature includes the cleanup of all SLAs during device shutdown. For each SLA, the firmware is notified of the removal through an admin message, the data structures that take into account the budgets are updated and the memory is freed. However, this explicit cleanup is not necessary as (1) the device is reset, and the firmware state is lost and (2) all RL data structures are freed anyway. In addition, if the device is unresponsive, for example after a PCI AER error is detected, the admin interface might not be available. This might slow down the shutdown sequence and cause a timeout in the recovery flows which in turn makes the driver believe that the device is not recoverable. Fix by replacing the explicit SLAs removal with just a free of the SLA data structures. Fixes: d9fb8408376e ("crypto: qat - add rate limiting feature to qat_4xxx") Cc: <stable@vger.kernel.org> Signed-off-by: Damian Muszynski <damian.muszynski@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - improve aer error reset handlingMun Chun Yep2024-02-091-1/+25
| | | | | | | | | | | | | | | | | | | | | Rework the AER reset and recovery flow to take into account root port integrated devices that gets reset between the error detected and the slot reset callbacks. In adf_error_detected() the devices is gracefully shut down. The worker threads are disabled, the error conditions are notified to listeners and through PFVF comms and finally the device is reset as part of adf_dev_down(). In adf_slot_reset(), the device is brought up again. If SRIOV VFs were enabled before reset, these are re-enabled and VFs are notified of restarting through PFVF comms. Signed-off-by: Mun Chun Yep <mun.chun.yep@intel.com> Reviewed-by: Ahsan Atta <ahsan.atta@intel.com> Reviewed-by: Markas Rapoportas <markas.rapoportas@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - limit heartbeat notificationsFurong Zhou2024-02-092-3/+17
| | | | | | | | | | | | | When the driver detects an heartbeat failure, it starts the recovery flow. Set a limit so that the number of events is limited in case the heartbeat status is read too frequently. Signed-off-by: Furong Zhou <furong.zhou@intel.com> Reviewed-by: Ahsan Atta <ahsan.atta@intel.com> Reviewed-by: Markas Rapoportas <markas.rapoportas@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Mun Chun Yep <mun.chun.yep@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - add auto reset on errorDamian Muszynski2024-02-094-1/+49
| | | | | | | | | | | | | | | | | Expose the `auto_reset` sysfs attribute to configure the driver to reset the device when a fatal error is detected. When auto reset is enabled, the driver resets the device when it detects either an heartbeat failure or a fatal error through an interrupt. This patch is based on earlier work done by Shashank Gupta. Signed-off-by: Damian Muszynski <damian.muszynski@intel.com> Reviewed-by: Ahsan Atta <ahsan.atta@intel.com> Reviewed-by: Markas Rapoportas <markas.rapoportas@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Mun Chun Yep <mun.chun.yep@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - add fatal error notificationMun Chun Yep2024-02-092-1/+9
| | | | | | | | | | | | | | | | | Notify a fatal error condition and optionally reset the device in the following cases: * if the device reports an uncorrectable fatal error through an interrupt * if the heartbeat feature detects that the device is not responding This patch is based on earlier work done by Shashank Gupta. Signed-off-by: Mun Chun Yep <mun.chun.yep@intel.com> Reviewed-by: Ahsan Atta <ahsan.atta@intel.com> Reviewed-by: Markas Rapoportas <markas.rapoportas@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - re-enable sriov after pf resetMun Chun Yep2024-02-094-4/+79
| | | | | | | | | | | | | | | | | | | | | | | When a Physical Function (PF) is reset, SR-IOV gets disabled, making the associated Virtual Functions (VFs) unavailable. Even after reset and using pci_restore_state, VFs remain uncreated because the numvfs still at 0. Therefore, it's necessary to reconfigure SR-IOV to re-enable VFs. This commit introduces the ADF_SRIOV_ENABLED configuration flag to cache the SR-IOV enablement state. SR-IOV is only re-enabled if it was previously configured. This commit also introduces a dedicated workqueue without `WQ_MEM_RECLAIM` flag for enabling SR-IOV during Heartbeat and CPM error resets, preventing workqueue flushing warning. This patch is based on earlier work done by Shashank Gupta. Signed-off-by: Mun Chun Yep <mun.chun.yep@intel.com> Reviewed-by: Ahsan Atta <ahsan.atta@intel.com> Reviewed-by: Markas Rapoportas <markas.rapoportas@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - update PFVF protocol for recoveryMun Chun Yep2024-02-098-2/+109
| | | | | | | | | | | | | | | | | | | | | | | | | Update the PFVF logic to handle restart and recovery. This adds the following functions: * adf_pf2vf_notify_fatal_error(): allows the PF to notify VFs that the device detected a fatal error and requires a reset. This sends to VF the event `ADF_PF2VF_MSGTYPE_FATAL_ERROR`. * adf_pf2vf_wait_for_restarting_complete(): allows the PF to wait for `ADF_VF2PF_MSGTYPE_RESTARTING_COMPLETE` events from active VFs before proceeding with a reset. * adf_pf2vf_notify_restarted(): enables the PF to notify VFs with an `ADF_PF2VF_MSGTYPE_RESTARTED` event after recovery, indicating that the device is back to normal. This prompts VF drivers switch back to use the accelerator for workload processing. These changes improve the communication and synchronization between PF and VF drivers during system restart and recovery processes. Signed-off-by: Mun Chun Yep <mun.chun.yep@intel.com> Reviewed-by: Ahsan Atta <ahsan.atta@intel.com> Reviewed-by: Markas Rapoportas <markas.rapoportas@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - disable arbitration before resetFurong Zhou2024-02-091-0/+8
| | | | | | | | | | | | | | | Disable arbitration to avoid new requests to be processed before resetting a device. This is needed so that new requests are not fetched when an error is detected. Signed-off-by: Furong Zhou <furong.zhou@intel.com> Reviewed-by: Ahsan Atta <ahsan.atta@intel.com> Reviewed-by: Markas Rapoportas <markas.rapoportas@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Mun Chun Yep <mun.chun.yep@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - add fatal error notify methodFurong Zhou2024-02-093-0/+45
| | | | | | | | | | | | | | | | | Add error notify method to report a fatal error event to all the subsystems registered. In addition expose an API, adf_notify_fatal_error(), that allows to trigger a fatal error notification asynchronously in the context of a workqueue. This will be invoked when a fatal error is detected by the ISR or through Heartbeat. Signed-off-by: Furong Zhou <furong.zhou@intel.com> Reviewed-by: Ahsan Atta <ahsan.atta@intel.com> Reviewed-by: Markas Rapoportas <markas.rapoportas@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Mun Chun Yep <mun.chun.yep@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - add heartbeat error simulatorDamian Muszynski2024-02-098-6/+188
| | | | | | | | | | | | | | | | | | | | | Add a mechanism that allows to inject a heartbeat error for testing purposes. A new attribute `inject_error` is added to debugfs for each QAT device. Upon a write on this attribute, the driver will inject an error on the device which can then be detected by the heartbeat feature. Errors are breaking the device functionality thus they require a device reset in order to be recovered. This functionality is not compiled by default, to enable it CRYPTO_DEV_QAT_ERROR_INJECTION must be set. Signed-off-by: Damian Muszynski <damian.muszynski@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Reviewed-by: Lucas Segarra Fernandez <lucas.segarra.fernandez@intel.com> Reviewed-by: Ahsan Atta <ahsan.atta@intel.com> Reviewed-by: Markas Rapoportas <markas.rapoportas@intel.com> Signed-off-by: Mun Chun Yep <mun.chun.yep@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - use kcalloc_node() instead of kzalloc_node()Erick Archer2024-01-261-1/+1
| | | | | | | | | | | | | | | | | | | | As noted in the "Deprecated Interfaces, Language Features, Attributes, and Conventions" documentation [1], size calculations (especially multiplication) should not be performed in memory allocator (or similar) function arguments due to the risk of them overflowing. This could lead to values wrapping around and a smaller allocation being made than the caller was expecting. Using those allocations could lead to linear overflows of heap memory and other misbehaviors. So, use the purpose specific kcalloc_node() function instead of the argument count * size in the kzalloc_node() function. Link: https://www.kernel.org/doc/html/next/process/deprecated.html#open-coded-arithmetic-in-allocator-arguments [1] Link: https://github.com/KSPP/linux/issues/162 Signed-off-by: Erick Archer <erick.archer@gmx.com> Reviewed-by: Gustavo A. R. Silva <gustavoars@kernel.org> Acked-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: iaa - Remove unnecessary debugfs_create_dir() error check in ↵Minjie Du2024-01-261-2/+0
| | | | | | | | | | | | | iaa_crypto_debugfs_init() This patch removes the debugfs_create_dir() error checking in iaa_crypto_debugfs_init(). Because the debugfs_create_dir() is developed in a way that the caller can safely handle the errors that occur during the creation of DebugFS nodes. Signed-off-by: Minjie Du <duminjie@vivo.com> Acked-by: Tom Zanussi <tom.zanussi@linux.intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: iaa - Remove header table codeTom Zanussi2024-01-263-131/+3
| | | | | | | | | | | | | | The header table and related code is currently unused - it was included and used for canned mode, but canned mode has been removed, so this code can be safely removed as well. This indirectly fixes a bug reported by Dan Carpenter. Reported-by: Dan Carpenter <dan.carpenter@linaro.org> Closes: https://lore.kernel.org/linux-crypto/b2e0bd974981291e16882686a2b9b1db3986abe4.camel@linux.intel.com/T/#m4403253d6a4347a925fab4fc1cdb4ef7c095fb86 Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com> Reviewed-by: Dave Jiang <dave.jiang@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - avoid memcpy() overflow warningArnd Bergmann2024-01-261-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | The use of array_size() leads gcc to assume the memcpy() can have a larger limit than actually possible, which triggers a string fortification warning: In file included from include/linux/string.h:296, from include/linux/bitmap.h:12, from include/linux/cpumask.h:12, from include/linux/sched.h:16, from include/linux/delay.h:23, from include/linux/iopoll.h:12, from drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.c:3: In function 'fortify_memcpy_chk', inlined from 'adf_gen4_init_thd2arb_map' at drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.c:401:3: include/linux/fortify-string.h:579:4: error: call to '__write_overflow_field' declared with attribute warning: detected write beyond size of field (1st parameter); maybe use struct_group()? [-Werror=attribute-warning] 579 | __write_overflow_field(p_size_field, size); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ include/linux/fortify-string.h:588:4: error: call to '__read_overflow2_field' declared with attribute warning: detected read beyond size of field (2nd parameter); maybe use struct_group()? [-Werror=attribute-warning] 588 | __read_overflow2_field(q_size_field, size); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Add an explicit range check to avoid this. Fixes: 5da6a2d5353e ("crypto: qat - generate dynamically arbiter mappings") Signed-off-by: Arnd Bergmann <arnd@arndb.de> Acked-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - fix arbiter mapping generation algorithm for QAT 402xxDamian Muszynski2024-01-261-0/+1
| | | | | | | | | | | | | | | | | | | | The commit "crypto: qat - generate dynamically arbiter mappings" introduced a regression on qat_402xx devices. This is reported when the driver probes the device, as indicated by the following error messages: 4xxx 0000:0b:00.0: enabling device (0140 -> 0142) 4xxx 0000:0b:00.0: Generate of the thread to arbiter map failed 4xxx 0000:0b:00.0: Direct firmware load for qat_402xx_mmp.bin failed with error -2 The root cause of this issue was the omission of a necessary function pointer required by the mapping algorithm during the implementation. Fix it by adding the missing function pointer. Fixes: 5da6a2d5353e ("crypto: qat - generate dynamically arbiter mappings") Signed-off-by: Damian Muszynski <damian.muszynski@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: iaa - Account for cpu-less numa nodesTom Zanussi2023-12-291-2/+13
| | | | | | | | | | | | | | | | | In some configurations e.g. systems with CXL, a numa node can have 0 cpus and cpumask_nth() will return a cpu value that doesn't exist, which will result in an attempt to add an entry to the wq table at a bad index. To fix this, when iterating the cpus for a node, skip any node that doesn't have cpus. Also, as a precaution, add a warning and bail if cpumask_nth() returns a nonexistent cpu. Reported-by: Zhang, Rex <rex.zhang@intel.com> Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - generate dynamically arbiter mappingsDamian Muszynski2023-12-295-112/+235
| | | | | | | | | | | | | | | | | The thread-to-arbiter mapping describes which arbiter can assign jobs to an acceleration engine thread. The existing mappings are functionally correct, but hardcoded and not optimized. Replace the static mappings with an algorithm that generates optimal mappings, based on the loaded configuration. The logic has been made common so that it can be shared between all QAT GEN4 devices. Signed-off-by: Damian Muszynski <damian.muszynski@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - add support for ring pair level telemetryLucas Segarra Fernandez2023-12-2910-5/+324
| | | | | | | | | | | | | | | | | Expose through debugfs ring pair telemetry data for QAT GEN4 devices. This allows to gather metrics about the PCIe channel and device TLB for a selected ring pair. It is possible to monitor maximum 4 ring pairs at the time per device. For details, refer to debugfs-driver-qat_telemetry in Documentation/ABI. This patch is based on earlier work done by Wojciech Ziemba. Signed-off-by: Lucas Segarra Fernandez <lucas.segarra.fernandez@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Reviewed-by: Damian Muszynski <damian.muszynski@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - add support for device telemetryLucas Segarra Fernandez2023-12-2912-0/+1236
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Expose through debugfs device telemetry data for QAT GEN4 devices. This allows to gather metrics about the performance and the utilization of a device. In particular, statistics on (1) the utilization of the PCIe channel, (2) address translation, when SVA is enabled and (3) the internal engines for crypto and data compression. If telemetry is supported by the firmware, the driver allocates a DMA region and a circular buffer. When telemetry is enabled, through the `control` attribute in debugfs, the driver sends to the firmware, via the admin interface, the `TL_START` command. This triggers the device to periodically gather telemetry data from hardware registers and write it into the DMA memory region. The device writes into the shared region every second. The driver, every 500ms, snapshots the DMA shared region into the circular buffer. This is then used to compute basic metric (min/max/average) on each counter, every time the `device_data` attribute is queried. Telemetry counters are exposed through debugfs in the folder /sys/kernel/debug/qat_<device>_<BDF>/telemetry. For details, refer to debugfs-driver-qat_telemetry in Documentation/ABI. This patch is based on earlier work done by Wojciech Ziemba. Signed-off-by: Lucas Segarra Fernandez <lucas.segarra.fernandez@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Reviewed-by: Damian Muszynski <damian.muszynski@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - add admin msgs for telemetryLucas Segarra Fernandez2023-12-293-0/+51
| | | | | | | | | | | | | | | | | | | | | | | | Extend the admin interface with two new public APIs to enable and disable the telemetry feature: adf_send_admin_tl_start() and adf_send_admin_tl_stop(). The first, sends to the firmware, through the ICP_QAT_FW_TL_START message, the IO address where the firmware will write telemetry metrics and a list of ring pairs (maximum 4) to be monitored. It returns the number of accelerators of each type supported by this hardware. After this message is sent, the firmware starts periodically reporting telemetry data using by writing into the dma buffer specified as input. The second, sends the admin message ICP_QAT_FW_TL_STOP which stops the reporting of telemetry data. This patch is based on earlier work done by Wojciech Ziemba. Signed-off-by: Lucas Segarra Fernandez <lucas.segarra.fernandez@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Reviewed-by: Damian Muszynski <damian.muszynski@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - include pci.h for GET_DEV()Lucas Segarra Fernandez2023-12-291-0/+1
| | | | | | | | | | | GET_DEV() macro expansion relies on struct pci_dev being defined. Include <linux/pci.h> at adf_accel_devices.h. Signed-off-by: Lucas Segarra Fernandez <lucas.segarra.fernandez@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Reviewed-by: Damian Muszynski <damian.muszynski@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: iaa - remove unneeded semicolonJiapeng Chong2023-12-291-1/+1
| | | | | | | | | | | | No functional modification involved. ./drivers/crypto/intel/iaa/iaa_crypto_main.c:979:2-3: Unneeded semicolon. Reported-by: Abaci Robot <abaci@linux.alibaba.com> Closes: https://bugzilla.openanolis.cn/show_bug.cgi?id=7772 Signed-off-by: Jiapeng Chong <jiapeng.chong@linux.alibaba.com> Acked-by: Tom Zanussi <tom.zanussi@linux.intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: iaa - Remove unneeded newline in update_max_adecomp_delay_ns()Tom Zanussi2023-12-291-1/+0
| | | | | | | | | | Remove a stray newline in update_max_adecomp_delay_ns(). Reported-by: Christophe JAILLET <christophe.jaillet@wanadoo.fr> Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com> Reviewed-by: Dave Jiang <dave.jiang@intel.com> Reviewed-by: Fenghua Yu <fenghua.yu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: iaa - Change desc->priv to 0Tom Zanussi2023-12-291-4/+4
| | | | | | | | | | | | | | | | | | In order for shared workqeues to work properly, desc->priv should be set to 0 rather than 1. The need for this is described in commit f5ccf55e1028 (dmaengine/idxd: Re-enable kernel workqueue under DMA API), so we need to make IAA consistent with IOMMU settings, otherwise we get: [ 141.948389] IOMMU: dmar15: Page request in Privilege Mode [ 141.948394] dmar15: Invalid page request: 2000026a100101 ffffb167 Dedicated workqueues ignore this field and are unaffected. Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com> Reviewed-by: Dave Jiang <dave.jiang@intel.com> Reviewed-by: Fenghua Yu <fenghua.yu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - add support for 420xx devicesJie Wang2023-12-2213-5/+849
| | | | | | | | | | | | | | | Add support for 420xx devices by including a new device driver that supports such devices, updates to the firmware loader and capabilities. Compared to 4xxx devices, 420xx devices have more acceleration engines (16 service engines and 1 admin) and support the wireless cipher algorithms ZUC and Snow 3G. Signed-off-by: Jie Wang <jie.wang@intel.com> Co-developed-by: Dong Xie <dong.xie@intel.com> Signed-off-by: Dong Xie <dong.xie@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - move fw config related structuresJie Wang2023-12-222-12/+19
| | | | | | | | | | | | | Relocate the structures adf_fw_objs and adf_fw_config from the file adf_4xxx_hw_data.c to the newly created adf_fw_config.h. These structures will be used by new device drivers. This does not introduce any functional change. Signed-off-by: Jie Wang <jie.wang@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - relocate portions of qat_4xxx codeJie Wang2023-12-228-484/+552
| | | | | | | | | | | | | | | | | Move logic that is common between QAT GEN4 accelerators to the qat_common folder. This includes addresses of CSRs, setters and configuration logic. When moved, functions and defines have been renamed from 4XXX to GEN4. Code specific to the device is moved to the file adf_gen4_hw_data.c. Code related to configuration is moved to the newly created adf_gen4_config.c. This does not introduce any functional change. Signed-off-by: Jie Wang <jie.wang@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - change signature of uof_get_num_objs()Jie Wang2023-12-223-3/+3
| | | | | | | | | | | | | Add accel_dev as parameter of the function uof_get_num_objs(). This is in preparation for the introduction of the QAT 420xx driver as it will allow to reconfigure the ae_mask when a configuration that does not require all AEs is loaded on the device. This does not introduce any functional change. Signed-off-by: Jie Wang <jie.wang@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: qat - relocate and rename get_service_enabled()Jie Wang2023-12-223-26/+34
| | | | | | | | | | | | | Move the function get_service_enabled() from adf_4xxx_hw_data.c to adf_cfg_services.c and rename it as adf_get_service_enabled(). This function is not specific to the 4xxx and will be used by other QAT drivers. This does not introduce any functional change. Signed-off-by: Jie Wang <jie.wang@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: iaa - Add IAA Compression Accelerator statsTom Zanussi2023-12-156-2/+427
| | | | | | | | | | | | | | | Add support for optional debugfs statistics support for the IAA Compression Accelerator. This is enabled by the kernel config item: CRYPTO_DEV_IAA_CRYPTO_STATS When enabled, the IAA crypto driver will generate statistics which can be accessed at /sys/kernel/debug/iaa-crypto/. See Documentation/driver-api/crypto/iax/iax-crypto.rst for details. Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: iaa - Add irq support for the crypto async interfaceTom Zanussi2023-12-152-2/+266
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The existing iaa crypto async support provides an implementation that satisfies the interface but does so in a synchronous manner - it fills and submits the IDXD descriptor and then waits for it to complete before returning. This isn't a problem at the moment, since all existing callers (e.g. zswap) wrap any asynchronous callees in a synchronous wrapper anyway. This change makes the iaa crypto async implementation truly asynchronous: it fills and submits the IDXD descriptor, then returns immediately with -EINPROGRESS. It also sets the descriptor's 'request completion irq' bit and sets up a callback with the IDXD driver which is called when the operation completes and the irq fires. The existing callers such as zswap use synchronous wrappers to deal with -EINPROGRESS and so work as expected without any changes. This mode can be enabled by writing 'async_irq' to the sync_mode iaa_crypto driver attribute: echo async_irq > /sys/bus/dsa/drivers/crypto/sync_mode Async mode without interrupts (caller must poll) can be enabled by writing 'async' to it: echo async > /sys/bus/dsa/drivers/crypto/sync_mode The default sync mode can be enabled by writing 'sync' to it: echo sync > /sys/bus/dsa/drivers/crypto/sync_mode The sync_mode value setting at the time the IAA algorithms are registered is captured in each algorithm's crypto_ctx and used for all compresses and decompresses when using a given algorithm. Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: iaa - Add support for deflate-iaa compression algorithmTom Zanussi2023-12-152-18/+1069
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch registers the deflate-iaa deflate compression algorithm and hooks it up to the IAA hardware using the 'fixed' compression mode introduced in the previous patch. Because the IAA hardware has a 4k history-window limitation, only buffers <= 4k, or that have been compressed using a <= 4k history window, are technically compliant with the deflate spec, which allows for a window of up to 32k. Because of this limitation, the IAA fixed mode deflate algorithm is given its own algorithm name, 'deflate-iaa'. With this change, the deflate-iaa crypto algorithm is registered and operational, and compression and decompression operations are fully enabled following the successful binding of the first IAA workqueue to the iaa_crypto sub-driver. when there are no IAA workqueues bound to the driver, the IAA crypto algorithm can be unregistered by removing the module. A new iaa_crypto 'verify_compress' driver attribute is also added, allowing the user to toggle compression verification. If set, each compress will be internally decompressed and the contents verified, returning error codes if unsuccessful. This can be toggled with 0/1: echo 0 > /sys/bus/dsa/drivers/crypto/verify_compress The default setting is '1' - verify all compresses. The verify_compress value setting at the time the algorithm is registered is captured in the algorithm's crypto_ctx and used for all compresses when using the algorithm. [ Based on work originally by George Powley, Jing Lin and Kyung Min Park ] Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: iaa - Add compression mode management along with fixed modeTom Zanussi2023-12-154-2/+504
| | | | | | | | | | | | | | | | | Define an in-kernel API for adding and removing compression modes, which can be used by kernel modules or other kernel code that implements IAA compression modes. Also add a separate file, iaa_crypto_comp_fixed.c, containing huffman tables generated for the IAA 'fixed' compression mode. Future compression modes can be added in a similar fashion. One or more crypto compression algorithms will be created for each compression mode, each of which can be selected as the compression algorithm to be used by a particular facility. Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
* crypto: iaa - Add per-cpu workqueue table with rebalancingTom Zanussi2023-12-152-0/+229
| | | | | | | | | | | | | | | | | | | | | | | | | | | The iaa compression/decompression algorithms in later patches need a way to retrieve an appropriate IAA workqueue depending on how close the associated IAA device is to the current cpu. For this purpose, add a per-cpu array of workqueues such that an appropriate workqueue can be retrieved by simply accessing the per-cpu array. Whenever a new workqueue is bound to or unbound from the iaa_crypto driver, the available workqueues are 'rebalanced' such that work submitted from a particular CPU is given to the most appropriate workqueue available. There currently isn't any way for the user to tweak the way this is done internally - if necessary, knobs can be added later for that purpose. Current best practice is to configure and bind at least one workqueue for each IAA device, but as long as there is at least one workqueue configured and bound to any IAA device in the system, the iaa_crypto driver will work, albeit most likely not as efficiently. [ Based on work originally by George Powley, Jing Lin and Kyung Min Park ] Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
generated by cgit v1.2.3 (git 2.25.1) at 2024-09-01 19:07:20 +0000