kernel.git/drivers/dma/dw/core.c, branch linux-6.1.y

dmaengine: dw: Add memory bus width verification

2024-09-04T11:25:01Z

[ Upstream commit d04b21bfa1c50a2ade4816cab6fdc91827b346b1 ] Currently in case of the DEV_TO_MEM or MEM_TO_DEV DMA transfers the memory data width (single transfer width) is determined based on the buffer length, buffer base address or DMA master-channel max address width capability. It isn't enough in case of the channel disabling prior the block transfer is finished. Here is what DW AHB DMA IP-core databook says regarding the port suspension (DMA-transfer pause) implementation in the controller: "When CTLx.SRC_TR_WIDTH < CTLx.DST_TR_WIDTH and the CFGx.CH_SUSP bit is high, the CFGx.FIFO_EMPTY is asserted once the contents of the FIFO do not permit a single word of CTLx.DST_TR_WIDTH to be formed. However, there may still be data in the channel FIFO, but not enough to form a single transfer of CTLx.DST_TR_WIDTH. In this scenario, once the channel is disabled, the remaining data in the channel FIFO is not transferred to the destination peripheral." So in case if the port gets to be suspended and then disabled it's possible to have the data silently discarded even though the controller reported that FIFO is empty and the CTLx.BLOCK_TS indicated the dropped data already received from the source device. This looks as if the data somehow got lost on a way from the peripheral device to memory and causes problems for instance in the DW APB UART driver, which pauses and disables the DMA-transfer as soon as the recv data timeout happens. Here is the way it looks: Memory <------- DMA FIFO <------ UART FIFO <---------------- UART DST_TR_WIDTH -+--------| | | | | | | No more data Current lvl -+--------| |---------+- DMA-burst lvl | | |---------+- Leftover data | | |---------+- SRC_TR_WIDTH -+--------+-------+---------+ In the example above: no more data is getting received over the UART port and BLOCK_TS is not even close to be fully received; some data is left in the UART FIFO, but not enough to perform a bursted DMA-xfer to the DMA FIFO; some data is left in the DMA FIFO, but not enough to be passed further to the system memory in a single transfer. In this situation the 8250 UART driver catches the recv timeout interrupt, pauses the DMA-transfer and terminates it completely, after which the IRQ handler manually fetches the leftover data from the UART FIFO into the recv-buffer. But since the DMA-channel has been disabled with the data left in the DMA FIFO, that data will be just discarded and the recv-buffer will have a gap of the "current lvl" size in the recv-buffer at the tail of the lately received data portion. So the data will be lost just due to the misconfigured DMA transfer. Note this is only relevant for the case of the transfer suspension and _disabling_. No problem will happen if the transfer will be re-enabled afterwards or the block transfer is fully completed. In the later case the "FIFO flush mode" will be executed at the transfer final stage in order to push out the data left in the DMA FIFO. In order to fix the denoted problem the DW AHB DMA-engine driver needs to make sure that the _bursted_ source transfer width is greater or equal to the single destination transfer (note the HW databook describes more strict constraint than actually required). Since the peripheral-device side is prescribed by the client driver logic, the memory-side can be only used for that. The solution can be easily implemented for the DEV_TO_MEM transfers just by adjusting the memory-channel address width. Sadly it's not that easy for the MEM_TO_DEV transfers since the mem-to-dma burst size is normally dynamically determined by the controller. So the only thing that can be done is to make sure that memory-side address width is greater than the peripheral device address width. Fixes: a09820043c9e ("dw_dmac: autoconfigure data_width or get it via platform data") Signed-off-by: Serge Semin Acked-by: Andy Shevchenko Link: https://lore.kernel.org/r/20240802075100.6475-3-fancer.lancer@gmail.com Signed-off-by: Vinod Koul Signed-off-by: Sasha Levin

dmaengine: dw: Add peripheral bus width verification

2024-09-04T11:25:00Z

[ Upstream commit b336268dde75cb09bd795cb24893d52152a9191f ] Currently the src_addr_width and dst_addr_width fields of the dma_slave_config structure are mapped to the CTLx.SRC_TR_WIDTH and CTLx.DST_TR_WIDTH fields of the peripheral bus side in order to have the properly aligned data passed to the target device. It's done just by converting the passed peripheral bus width to the encoded value using the __ffs() function. This implementation has several problematic sides: 1. __ffs() is undefined if no bit exist in the passed value. Thus if the specified addr-width is DMA_SLAVE_BUSWIDTH_UNDEFINED, __ffs() may return unexpected value depending on the platform-specific implementation. 2. DW AHB DMA-engine permits having the power-of-2 transfer width limited by the DMAH_Mk_HDATA_WIDTH IP-core synthesize parameter. Specifying bus-width out of that constraints scope will definitely cause unexpected result since the destination reg will be only partly touched than the client driver implied. Let's fix all of that by adding the peripheral bus width verification method and calling it in dwc_config() which is supposed to be executed before preparing any transfer. The new method will make sure that the passed source or destination address width is valid and if undefined then the driver will just fallback to the 1-byte width transfer. Fixes: 029a40e97d0d ("dmaengine: dw: provide DMA capabilities") Signed-off-by: Serge Semin Acked-by: Andy Shevchenko Link: https://lore.kernel.org/r/20240802075100.6475-2-fancer.lancer@gmail.com Signed-off-by: Vinod Koul Signed-off-by: Sasha Levin

dma:dw: remove reference to AVR32 architecture in core.c

2022-08-03T09:03:02Z

The AVR32 architecture does no longer exist in the Linux kernel, hence remove a reference to it in comments to avoid confusion. Signed-off-by: Hans-Christian Noren Egtvedt

dmaengine dw: Revert "dmaengine: dw: Enable runtime PM"

2021-02-08T12:06:12Z

This reverts commit 842067940a3e3fc008a60fee388e000219b32632. For some solutions e.g. sound/soc/intel/catpt, DW DMA is part of a compound device (in that very example, domains: ADSP, SSP0, SSP1, DMA0 and DMA1 are part of a single entity) rather than being a standalone one. Driver for said device may enlist DMA to transfer data during suspend or resume sequences. Manipulating RPM explicitly in dw's DMA request and release channel functions causes suspend() to also invoke resume() for the exact same device. Similar situation occurs for resume() sequence. Effectively renders device dysfunctional after first suspend() attempt. Revert the change to address the problem. Fixes: 842067940a3e ("dmaengine: dw: Enable runtime PM") Cc: Andy Shevchenko Signed-off-by: Cezary Rojewski Acked-by: Andy Shevchenko Link: https://lore.kernel.org/r/20210203191924.15706-1-cezary.rojewski@intel.com Signed-off-by: Vinod Koul

dmaengine: dw: Enable runtime PM

2020-11-09T11:49:20Z

When consumer requests channel power on the DMA controller device and otherwise on the freeing channel resources. Note, in some cases consumer acquires channel at the ->probe() stage and releases it at the ->remove() stage. It will mean that DMA controller device will be powered during all this time if there is no assist from hardware to idle it. The above mentioned cases should be investigated separately and individually. Signed-off-by: Andy Shevchenko Acked-by: Viresh Kumar Link: https://lore.kernel.org/r/20201103183938.64752-1-andriy.shevchenko@linux.intel.com Signed-off-by: Vinod Koul

dmaengine: dw: convert tasklets to use new tasklet_setup() API

2020-09-18T06:48:11Z

In preparation for unconditionally passing the struct tasklet_struct pointer to all tasklet callbacks, switch to using the new tasklet_setup() and from_tasklet() to pass the tasklet pointer explicitly. Signed-off-by: Romain Perier Signed-off-by: Allen Pais Link: https://lore.kernel.org/r/20200831103542.305571-6-allen.lkml@gmail.com Signed-off-by: Vinod Koul

dmaengine: dw: Add DMA-channels mask cell support

2020-08-17T06:28:31Z

DW DMA IP-core provides a way to synthesize the DMA controller with channels having different parameters like maximum burst-length, multi-block support, maximum data width, etc. Those parameters both explicitly and implicitly affect the channels performance. Since DMA slave devices might be very demanding to the DMA performance, let's provide a functionality for the slaves to be assigned with DW DMA channels, which performance according to the platform engineer fulfill their requirements. After this patch is applied it can be done by passing the mask of suitable DMA-channels either directly in the dw_dma_slave structure instance or as a fifth cell of the DMA DT-property. If mask is zero or not provided, then there is no limitation on the channels allocation. For instance Baikal-T1 SoC is equipped with a DW DMAC engine, which first two channels are synthesized with max burst length of 16, while the rest of the channels have been created with max-burst-len=4. It would seem that the first two channels must be faster than the others and should be more preferable for the time-critical DMA slave devices. In practice it turned out that the situation is quite the opposite. The channels with max-burst-len=4 demonstrated a better performance than the channels with max-burst-len=16 even when they both had been initialized with the same settings. The performance drop of the first two DMA-channels made them unsuitable for the DW APB SSI slave device. No matter what settings they are configured with, full-duplex SPI transfers occasionally experience the Rx FIFO overflow. It means that the DMA-engine doesn't keep up with incoming data pace even though the SPI-bus is enabled with speed of 25MHz while the DW DMA controller is clocked with 50MHz signal. There is no such problem has been noticed for the channels synthesized with max-burst-len=4. Signed-off-by: Serge Semin Link: https://lore.kernel.org/r/20200731200826.9292-6-Sergey.Semin@baikalelectronics.ru Signed-off-by: Vinod Koul

dmaengine: dw: Discard dlen from the dev-to-mem xfer width calculation

2020-08-17T06:28:31Z

Indeed in case of the DMA_DEV_TO_MEM DMA transfers it's enough to take the destination memory address and the destination master data width into account to calculate the CTLx.DST_TR_WIDTH setting of the memory peripheral. According to the DW DMAC IP-core Databook 2.18b (page 66, Example 5) at the and of a DMA transfer when the DMA-channel internal FIFO is left with data less than for a single destination burst transaction, the destination peripheral will enter the Single Transaction Region where the DW DMA controller can complete a block transfer to the destination using single transactions (non-burst transaction of CTLx.DST_TR_WIDTH bytes). If there is no enough data in the DMA-channel internal FIFO for even a single non-burst transaction of CTLx.DST_TR_WIDTH bytes, then the channel enters "FIFO flush mode". That mode is activated to empty the FIFO and flush the leftovers out to the memory peripheral. The flushing procedure is simple. The data is sent to the memory by means of a set of single transaction of CTLx.SRC_TR_WIDTH bytes. To sum up it's redundant to use the LLPs length to find out the CTLx.DST_TR_WIDTH parameter value, since each DMA transfer will be completed with the CTLx.SRC_TR_WIDTH bytes transaction if it is required. We suggest to remove the LLP entry length from the statement which calculates the memory peripheral DMA transaction width since it's redundant due to the feature described above. By doing so we'll improve the memory bus utilization and speed up the DMA-channel performance for DMA_DEV_TO_MEM DMA-transfers. Signed-off-by: Serge Semin Acked-by: Andy Shevchenko Link: https://lore.kernel.org/r/20200731200826.9292-4-Sergey.Semin@baikalelectronics.ru Signed-off-by: Vinod Koul

Merge branch 'for-linus' into fixes

2020-08-05T13:32:07Z

Signed-off-by: Vinod Koul Conflicts: drivers/dma/idxd/sysfs.c

dmaengine: dw: Initialize max_sg_burst capability

2020-07-27T09:00:55Z

Multi-block support provides a way to map the kernel-specific SG-table so the DW DMA device would handle it as a whole instead of handling the SG-list items or so called LLP block items one by one. So if true LLP list isn't supported by the DW DMA engine, then soft-LLP mode will be utilized to load and execute each LLP-block one by one. The soft-LLP mode of the DMA transactions execution might not work well for some DMA consumers like SPI due to its Tx and Rx buffers inter-dependency. Let's initialize the max_sg_burst DMA channels capability based on the nollp flag state. If it's true, no hardware accelerated LLP is available and max_sg_burst should be set with 1, which means that the DMA engine can handle only a single SG list entry at a time. If noLLP is set to false, then hardware accelerated LLP is supported and the DMA engine can handle infinite number of SG entries in a single DMA transaction. Signed-off-by: Serge Semin Reviewed-by: Andy Shevchenko Link: https://lore.kernel.org/r/20200723005848.31907-11-Sergey.Semin@baikalelectronics.ru Signed-off-by: Vinod Koul