From: John Feeney <jfeeney@redhat.com>
Date: Fri, 3 Sep 2010 21:12:07 -0400
Subject: [dma_v3] update I/O AT and DCA drivers
Message-id: <4C8164A7.1020203@redhat.com>
Patchwork-id: 28134
O-Subject: [RHEL5.6 PATCH 1/3] Update I/O AT and DCA drivers
Bugzilla: 568551
RH-Acked-by: David S. Miller <davem@redhat.com>
This patch contains the code that actually updates the I/O AT and DCA
drivers.
Note: drivers/dma_v3 took the place of drivers/dma due to kabi issues
in the previous update to I/O AT for RHEL5. So all patches were applied
to "_v3" files where applicable. Also, when updating "I/O AT drivers",
changes are made to the network code (net/ipv6/tcp_ipv6.c for example.)
Note: Some upstream patches used change drivers not currently in RHEL5,
such as other dma and crypto drivers. Obviously, adding drivers so as
they can be updated by this bz is beyond the scope of this bz thus
those modifications were ignored.
Upstream commits in chronological order:
1. [TCP]: Add missing I/O AT code to ipv6 side.
David S. Miller Fri, 26 Oct 2007
[TCP]: Add missing I/O AT code to ipv6 side.
net/ipv6/tcp_ipv6.c
commit: b4caea8aa8b4caeda2af6ef6b7f0d43c516815ee
2. drivers/dma/ioat_dma.c: drop code after return
Julia Lawall Sun, 14 Sep 2008
The break after the return serves no purpose.
drivers/dma_v3/ioat_dma.c
commit: 89f72a0633d1d4f28c4c5c8831ec814523d7671a
3. [2/4] I/OAT: fix dma_pin_iovec_pages() error handling
Maciej Sosnowski Fri, 7 Nov 2008
Error handling needs to be modified in dma_pin_iovec_pages().
drivers/dma_v3/iovlock_v3.c
commit: c2c0b4c5434c0a25f7f7796b29155d53805909f5
4. [1/4] I/OAT: fix channel resources free for not allocated channels
Maciej Sosnowski Fri, 7 Nov 2008
If the ioatdma driver is loaded but not used it does not allocate
descriptors. Before it frees channel resources it should first be sure
that they have been previously allocated.
drivers/dma_v3/ioat_dma.c
commit: c3d4f44f50b65b0b0290e357f8739cfb3f4bcaca
5. [3/4] I/OAT: fix async_tx.callback checking
Maciej Sosnowski Fri, 7 Nov 2008
async_tx.callback should be checked for the first not the last
descriptor in the chain.
drivers/dma_v3/ioat_dma.c
commit: 12ccea24e309d815d058cdc6ee8bf2c4b85f0c5f
6. [4/4] dca: fixup initialization dependency
Dan Williams Fri, 7 Nov 2008
Mark dca_init as a subsys_initcall since it needs to be ready to go
before dependent drivers start registering themselves.
drivers/dca/dca-core.c
commit: 1207e795568a368928dfd23d6817e47f2e8097e3
7. dmaengine: protect 'id' from concurrent registrations
Dan Williams Thu, 4 Dec 2008
There is a possibility to have two devices registered with the same id.
drivers/dma_v3/dmaengine_v3.c
commit: b0b42b16ff2b90f17bc1a4308366c9beba4b276e
8. dmaengine: remove dependency on async_tx
Dan Williams Tue, 6 Jan 2009
async_tx.ko is a consumer of dma channels. A circular dependency arises
if modules in drivers/dma rely on common code in async_tx.ko.
drivers/dma_v3/dmaengine_v3.c
include/linux/dmaengine.h
commit: 07f2211e4fbce6990722d78c4f04225da9c0e9cf
9. dmaengine: up-level reference counting to the module level
Dan Williams Tue, 6 Jan 2009
Simply, if a client wants any dmaengine channel then prevent all dmaengine
modules from being removed.
drivers/dma_v3/dmaengine_v3.c
include/linux/dmaengine.h
include/net/netdma.h
net/ipv4/tcp.c
commit: 6f49a57aa5a0c6d4e4e27c85f7af6c83325a12d1
10. dmaengine: centralize channel allocation, introduce dma_find_channel
Dan Williams Tue, 6 Jan 2009
Allowing multiple clients to each define their own channel allocation
scheme quickly leads to a pathological situation.
drivers/dma_v3/dmaengine_v3.c
include/linux/dmaengine.h
commit: bec085134e446577a983f17f57d642a88d1af53b
11. dmaengine: provide a common 'issue_pending_all' implementation
Dan Williams Tue, 6 Jan 2009
async_tx and net_dma each have open-coded versions of issue_pending_all,
so provide a common routine in dmaengine.
drivers/dma_v3/dmaengine_v3.c look at client_device_register
include/linux/dmaengine.h
net/core/dev.c
commit: 2ba05622b8b143b0c95968ba59bddfbd6d2f2559
12. dmaengine: introduce dma_request_channel and private channels
Dan Williams Tue, 6 Jan 2009
This interface is primarily for device-to-memory clients which need to
search for dma channels with platform-specific characteristics.
drivers/dma_v3/dmaengine_v3.c
include/linux/dmaengnine.h
commit: 59b5ec21446b9239d706ab237fb261d525b75e81
13. atmel-mci: convert to dma_request_channel and down-level dma_slave
Dan Williams Tue, 6 Jan 2009
dma_request_channel provides an exclusive channel, so we no longer need to
pass slave data through dmaengine.
drivers/dma_v3/dmaengine_v3.c
include/linux/dmaengine.h
commit: 74465b4ff9ac1da503025c0a0042e023bfa6505c
14. dmaengine: replace dma_async_client_register with dmaengine_get
Dan Williams Tue, 6 Jan 2009
Now that clients no longer need to be notified of channel arrival
dma_async_client_register can simply increment the dmaengine_ref_count.
drivers/dma_v3/dmaengine_v3.c
include/linux/dmaengine.h
net/core/dev.c
commit: 209b84a88fe81341b4d8d465acc4a67cb7c3feb3
15. dmaengine: kill struct dma_client and supporting infrastructure
Dan Williams Tue, 6 Jan 2009
All users have been converted to either the general-purpose allocator,
dma_find_channel, or dma_request_channel.
drivers/dma_v3/dmaengine_v3.c
drivers/dma_v3/ioat_dma_v3.c
include/linux/dmaengine.h Note: can't take out dma_client due to
kabi but can remove whatever had been previously added.
net/core/dev.c
commit: aa1e6f1a385eb2b04171ec841f3b760091e4a8ee
16. net_dma: convert to dma_find_channel
Dan Williams Tue, 6 Jan 2009
Use the general-purpose channel allocation provided by dmaengine.
include/linux/netdevice.h
include/net/netdma.h - could not change softnet_data due to kabi
net/core/dev.c
net/ipv4/tcp.c
net/ipv4/tcp_input.c
net/ipv4/tcp_ipv4.c
net/ipv6/tcp_ipv6.c
commit: f67b45999205164958de4ec0658d51fa4bee066d
17. dmaengine: remove 'bigref' infrastructure
Dan Williams Tue, 6 Jan 2009
Reference counting is done at the module level so clients need not worry
that a channel will leave while they are actively using dmaengine.
drivers/dma_v3/dmaengine_v3.c
include/linux/dmaengine.h - can't take out refcount or done due to kabi
commit: f27c580c3628d79b17f38976d842a6d7f3616e2e
18. dmaengine: kill enum dma_state_client
Dan Williams Tue, 6 Jan 2009
DMA_NAK is now useless. We can just use a bool instead.
drivers/dma_v3/dmaengine_v3.c
include/linux/dmaengine.h
commit: 7dd602510128d7a64b11ff3b7d4f30ac8e3946ce
19. dmaengine: add a release for dma class devices and dependent infrastructure
Dan Williams Tue, 6 Jan 2009
Resolves: WARNING: at drivers/base/core.c:122 device_release+0x4d/0x52()
Device 'dma0chan0' does not have a release() function, it is broken and
must be fixed.
drivers/dma_v3/dmaengine_v3.c
include/linux/dmaengine.h
commit: 41d5e59c1299f27983977bcfe3b360600996051c
20. dmaengine: use idr for registering dma device numbers
Dan Williams Tue, 6 Jan 2009
This brings some predictability to dma device numbers, i.e. an rmmod/insmod
cycle may now result in /sys/class/dma/dma0chan0 being restored rather than
/sys/class/dma/dma1chan0 appearing.
drivers/dma_v3/dmaengine_v3.c
include/linux/dmaengine.h
commit: 864498aaa9fef69ee166da023d12413a7776342d
21. dmaengine: advertise all channels on a device to dma_filter_fn
Dan Williams Tue, 6 Jan 2009
Allow dma_filter_fn routines to disambiguate multiple channels on a device
rather than assuming that all channels on a device are equal.
drivers/dma_v3/dmaengine_v3.c
commit: e2346677af86150c6083974585c131e8a2c3ddcc
22. dmaengine: bump initcall level to arch_initcall
Dan Williams Tue, 6 Jan 2009
There are dmaengine users that would like to register dma devices at
subsys_initcall time to ensure channels are available by device_initcall
time.
drivers/dca/dca-core.c
drivers/dma_v3/dmaengine_v3.c
commit: 652afc27b26859a0ea5f6db681d80b83d2c43cf8
23. dmatest: convert to dma_request_channel
Dan Williams Tue, 6 Jan 2009
Replace the client registration infrastructure with a custom loop to
poll for channels.
include/linux/dmaengine.h
commit: 33df8ca068123457db56c316946a3c0e4ef787d6
24. ioat: do not perform removal actions at shutdown
Dan Williams Tue, 6 Jan 2009
Unregistering services should only happen at "remove" time.
drivers/dma_v3/ioat.c
commit: 4fac7fa57cf8001be259688468c825f836daf739
25. net_dma: acquire/release dma channels on ifup/ifdown
Dan Williams Sun, 11 Jan 2009
The recent dmaengine rework removed the capability to remove dma device
driver modules while net_dma is active.
include/linux/dmaengine.h
net/core/dev.c
commit: 649274d993212e7c23c0cb734572c2311c200872
26. dmaengine: fix dependency chaining
Yuri Tikhonov Mon, 12 Jan 2009
The first 'next' in chain is still remaining set, regardless the fact,
that tx->next has been already submitted.
drivers/dma_v3/dmaengine_v3.c
commit: dd59b8537f6cb53ab863fafad86a5828f1e889a2
27. dmaengine: kill some dubious WARN_ONCEs
Dan Williams Mon, 19 Jan 2009
dma_find_channel and dma_issue_pending_all are good places to warn about
improper api usage.
drivers/dma_v3/dmaengine_v3.c
commit: 83436a0560e9ef8af2f0796264dde4bed1415359
28. dmaengine: dma_issue_pending_all == nop when CONFIG_DMA_ENGINE=n
Dan Williams Mon, 19 Jan 2009
The device list will always be empty in this configuration, so no need
to walk the list.
include/linux/dmaengine.h - could not remove prototype
commit: c50331e8be32eaba5e1949f98c70d50b891262db
29. dmaengine: add async_tx_clear_ack() macro
Guennadi Liakhovetski Mon, 19 Jan 2009
To complete the DMA_CTRL_ACK handling API add a async_tx_clear_ack() macro.
include/linux/dmaengine.h
commit: ef560682a97491f62ef538931a4861b57d66c52c
30. dca: redesign locks to fix deadlocks
Maciej Sosnowski Tue, 3 Feb 2009
Change spin_locks to irqsave to prevent dead-locks.
drivers/dca/dca-core.c
commit: eb4400e3a040b90a3ad805b01fcbc99a5f615c8f
31. net_dma: call dmaengine_get only if NET_DMA enabled
David S. Miller Sat, 7 Feb 2009
The commit 649274d993212e7c23c0cb734572c2311c200872 ("net_dma:
acquire/release dma channels on ifup/ifdown") added unconditional call
of dmaengine_get() to net_dma.
include/linux/dmaengine.h
net/core/dev.c
commit: b4bd07c20ba0c1fa7ad09ba257e0a5cfc2bf6bb3
32. dmaengine: update kerneldoc
Johannes Weiner Wed, 11 Feb 2009
Some of the kerneldoc comments in the dmaengine header describe
already removed structure members. Remove them.
include/linux/dmaengine.h
commit: 1d93e52eb48df986a3c4d5ad8a520bf1f6837367
33. atmel-mci: fix initialization of dma slave data
Dan Williams Wed, 18 Feb 2009
The conversion of atmel-mci to dma_request_channel missed the
initialization of the channel dma_slave information.
drivers/dma_v3/dmaengine_v3.c
include/linux/dmaengine.h
commit: 287d859222e0adbc67666a6154aaf42d7d5bbb54
34. I/OAT: do not set DCACTRL_CMPL_WRITE_ENABLE for I/OAT ver.3
Maciej Sosnowski Thu, 26 Feb 2009
Flag DCACTRL_CMPL_WRITE_ENABLE is valid only for I/OAT ver.2
so it should not be set for I/OAT ver.3.
drivers/dma_v3/ioat_dma.c
commit: ea9c717d0148d4194f9bd04ecfa6b59b20fc0a08
35. I/OAT: fail initialization on zero channels detection
Maciej Sosnowski Thu, 26 Feb 2009
On some systems with I/OAT ver.2 when DCA is disabled in BIOS
situations have been observed that zero DMA channels are detected instead
of four.
drivers/dma_v3/ioat_dma.c
commit: 8b794b141c633083408d0bfb2229b3406d0ebf99
36. I/OAT: cancel watchdog before dma remove
Maciej Sosnowski Thu, 26 Feb 2009
Channel watchdog should be canceled before the rest of dma remove stuff.
drivers/dma_v3/ioat_dma.c
commit: 2b8a6bf896ef47cc7d84c503079cc7b99789f9fa
37. I/OAT: list usage cleanup
Eric Sesterhenn Thu, 26 Feb 2009
Trivial cleanup, list_del(); list_add_tail() is equivalent to list_move_tail().
drivers/dma_v3/ioat_dma.c
commit: aa2d0b8b97efa1033609ca89b9faa5d3a1232959
38. I/OAT: update driver version and copyright dates
Maciej Sosnowski Thu, 26 Feb 2009
drivers/dca/dca-core.c
drivers/dma_v3/ioat.c
drivers/dma_v3/ioat_dca.c
drivers/dma_v3/ioat_dma.c
drivers/dma_v3/ioatdma.h
drivers/dma_v3/ioatdma_hw.h
drivers/dma_v3/ioatdma_registers.h
commit: 211a22ce08dbb27eb1a66df8a4bdae5e96092bc8
39. I/OAT: add verification for proper APICID_TAG_MAP setting by BIOS
Maciej Sosnowski Thu, 26 Feb 2009
BIOS versions for systems with I/OAT ver.2 have been found
which fail to program APICID_TAG_MAP for DCA.
drivers/dma_v3/ioat_dca.c
commit: 49bc46360d68156ce82b2b1a12badb80078453a0
40. I/OAT: fail self-test if callback test reaches timeout
Dan Williams Mon, 2 Mar 2009
If we miss interrupts in the self test then fail registration of this
channel as it is unsuitable for use as a public channel.
drivers/dma_v3/ioat_dma.c
commit: 0c33e1ca3d80647f2e72e44524fd21e79214da20
41. dmaengine: Add privatecnt to revert DMA_PRIVATE property
Atsushi Nemoto Fri, 6 Mar 2009
Currently dma_request_channel() set DMA_PRIVATE capability but never
clear it.
drivers/dma_v3/dmaengine_v3.c
include/linux/dmaengine.h
commit: 0f571515c332e00b3515dbe0859ceaa30ab66e00
42. dca: add missing copyright/license headers
Maciej Sosnowski Sat, 21 Mar 2009
In two dca files copyright and license headers are missing.
drivers/dca/dca-sysfs.c
include/linux/dca.h
commit: e2fc4d19292ef2eb208f76976ddc3320cc5839b6
43. dmaengine: kill some unused headers
Dan Williams Wed, 25 Mar 2009
The dmaengine redux left some unneeded headers in include/linux/dmaengine.h
include/linux/dmaengine.h
commit: 54aee6a5f560d0e1bf3f39987c6ebe06daeb0ce1
44. dmaengine: initialize tx_list in dma_async_tx_descriptor_init
Dan Williams Wed, 25 Mar 2009
Centralize this common initialization
drivers/dma_v3/dmaengine_v3.c
commit: ccccce229c633a92c42cd1a40c0738d7b0d12644
45. dmaengine: fail device registration if channel registration fails
Dan Williams Wed, 25 Mar 2009
"If alloc_percpu or kzalloc failed, chan_id does not match with its
position in device->channels list."
drivers/dma_v3/dmaengine_v3.c
commit: 257b17ca030387cb17314cd1851507bdd1b4ddd5
46. dma-mapping: replace all DMA_64BIT_MASK macro with DMA_BIT_MASK(64)
Yang Hongyang Tue, 7 Apr 2009
Replace all DMA_64BIT_MASK macro with DMA_BIT_MASK(64)
drivers/dma_v3/ioat.c
commit: 6a35528a8346f6e6fd32ed7e51f04d1fa4ca2c01
47. dma-mapping: replace all DMA_32BIT_MASK macro with DMA_BIT_MASK(32)
Yang Hongyang Tue, 7 Apr 2009
Replace all DMA_32BIT_MASK macro with DMA_BIT_MASK(32)
drivers/dma_v3/ioat.c
commit: 284901a90a9e0b812ca3f5f852cbbfb60d10249d
48. ioatdma: fix "ioatdma frees DMA memory with wrong function"
Maciej Sosnowski Thu, 23 Apr 2009
The ioatdma driver was unmapping all regions using unmap_page.
drivers/dma_v3/dmaengine_v3.c
drivers/dma_v3/ioat_dma.c
include/linux/dmaengine.h
commit: 4f005dbe5584fe54c9f6d6d4f0acd3fb29be84da
Acks would be appreciated. Thanks.
drivers/dca/dca-core.c | 55 +-
drivers/dca/dca-sysfs.c | 21
drivers/dma_v3/dmaengine_v3.c | 831 +++++++++++++++++++++++++++----------
drivers/dma_v3/ioat.c | 102 +---
drivers/dma_v3/ioat_dca.c | 26 +
drivers/dma_v3/ioat_dma.c | 103 ++--
drivers/dma_v3/ioatdma.h | 4
drivers/dma_v3/ioatdma_hw.h | 2
drivers/dma_v3/ioatdma_registers.h | 2
drivers/dma_v3/iovlock_v3.c | 17
include/linux/dca.h | 20
include/linux/dmaengine.h | 218 ++++-----
include/net/netdma.h | 11
net/core/dev.c | 150 ------
net/ipv4/tcp.c | 5
net/ipv4/tcp_input.c | 2
net/ipv4/tcp_ipv4.c | 2
net/ipv6/tcp_ipv6.c | 3
18 files changed, 959 insertions(+), 615 deletions(-)
Signed-off-by: Jarod Wilson <jarod@redhat.com>
diff --git a/drivers/dca/dca-core.c b/drivers/dca/dca-core.c
index ec249d2..25b743a 100644
--- a/drivers/dca/dca-core.c
+++ b/drivers/dca/dca-core.c
@@ -1,5 +1,5 @@
/*
- * Copyright(c) 2007 Intel Corporation. All rights reserved.
+ * Copyright(c) 2007 - 2009 Intel Corporation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the Free
@@ -28,7 +28,7 @@
#include <linux/device.h>
#include <linux/dca.h>
-#define DCA_VERSION "1.4"
+#define DCA_VERSION "1.8"
MODULE_VERSION(DCA_VERSION);
MODULE_LICENSE("GPL");
@@ -60,16 +60,17 @@ int dca_add_requester(struct device *dev)
{
struct dca_provider *dca;
int err, slot = -ENODEV;
+ unsigned long flags;
if (!dev)
return -EFAULT;
- spin_lock(&dca_lock);
+ spin_lock_irqsave(&dca_lock, flags);
/* check if the requester has not been added already */
dca = dca_find_provider_by_dev(dev);
if (dca) {
- spin_unlock(&dca_lock);
+ spin_unlock_irqrestore(&dca_lock, flags);
return -EEXIST;
}
@@ -78,19 +79,21 @@ int dca_add_requester(struct device *dev)
if (slot >= 0)
break;
}
- if (slot < 0) {
- spin_unlock(&dca_lock);
+
+ spin_unlock_irqrestore(&dca_lock, flags);
+
+ if (slot < 0)
return slot;
- }
err = dca_sysfs_add_req(dca, dev, slot);
if (err) {
- dca->ops->remove_requester(dca, dev);
- spin_unlock(&dca_lock);
+ spin_lock_irqsave(&dca_lock, flags);
+ if (dca == dca_find_provider_by_dev(dev))
+ dca->ops->remove_requester(dca, dev);
+ spin_unlock_irqrestore(&dca_lock, flags);
return err;
}
- spin_unlock(&dca_lock);
return 0;
}
EXPORT_SYMBOL_GPL(dca_add_requester);
@@ -103,25 +106,25 @@ int dca_remove_requester(struct device *dev)
{
struct dca_provider *dca;
int slot;
+ unsigned long flags;
if (!dev)
return -EFAULT;
- spin_lock(&dca_lock);
+ spin_lock_irqsave(&dca_lock, flags);
dca = dca_find_provider_by_dev(dev);
if (!dca) {
- spin_unlock(&dca_lock);
+ spin_unlock_irqrestore(&dca_lock, flags);
return -ENODEV;
}
slot = dca->ops->remove_requester(dca, dev);
- if (slot < 0) {
- spin_unlock(&dca_lock);
+ spin_unlock_irqrestore(&dca_lock, flags);
+
+ if (slot < 0)
return slot;
- }
dca_sysfs_remove_req(dca, slot);
- spin_unlock(&dca_lock);
return 0;
}
EXPORT_SYMBOL_GPL(dca_remove_requester);
@@ -135,17 +138,18 @@ u8 dca_common_get_tag(struct device *dev, int cpu)
{
struct dca_provider *dca;
u8 tag;
+ unsigned long flags;
- spin_lock(&dca_lock);
+ spin_lock_irqsave(&dca_lock, flags);
dca = dca_find_provider_by_dev(dev);
if (!dca) {
- spin_unlock(&dca_lock);
+ spin_unlock_irqrestore(&dca_lock, flags);
return -ENODEV;
}
tag = dca->ops->get_tag(dca, dev, cpu);
- spin_unlock(&dca_lock);
+ spin_unlock_irqrestore(&dca_lock, flags);
return tag;
}
@@ -217,11 +221,16 @@ static BLOCKING_NOTIFIER_HEAD(dca_provider_chain);
int register_dca_provider(struct dca_provider *dca, struct device *dev)
{
int err;
+ unsigned long flags;
err = dca_sysfs_add_provider(dca, dev);
if (err)
return err;
+
+ spin_lock_irqsave(&dca_lock, flags);
list_add(&dca->node, &dca_providers);
+ spin_unlock_irqrestore(&dca_lock, flags);
+
blocking_notifier_call_chain(&dca_provider_chain,
DCA_PROVIDER_ADD, NULL);
return 0;
@@ -234,9 +243,15 @@ EXPORT_SYMBOL_GPL(register_dca_provider);
*/
void unregister_dca_provider(struct dca_provider *dca)
{
+ unsigned long flags;
+
blocking_notifier_call_chain(&dca_provider_chain,
DCA_PROVIDER_REMOVE, NULL);
+
+ spin_lock_irqsave(&dca_lock, flags);
list_del(&dca->node);
+ spin_unlock_irqrestore(&dca_lock, flags);
+
dca_sysfs_remove_provider(dca);
}
EXPORT_SYMBOL_GPL(unregister_dca_provider);
@@ -270,6 +285,6 @@ static void __exit dca_exit(void)
dca_sysfs_exit();
}
-module_init(dca_init);
+arch_initcall(dca_init);
module_exit(dca_exit);
diff --git a/drivers/dca/dca-sysfs.c b/drivers/dca/dca-sysfs.c
index 3d47e9d..d436143 100644
--- a/drivers/dca/dca-sysfs.c
+++ b/drivers/dca/dca-sysfs.c
@@ -1,3 +1,24 @@
+/*
+ * Copyright(c) 2007 - 2009 Intel Corporation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the Free
+ * Software Foundation; either version 2 of the License, or (at your option)
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; if not, write to the Free Software Foundation, Inc., 59
+ * Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ *
+ * The full GNU General Public License is included in this distribution in the
+ * file called COPYING.
+ */
+
#include <linux/kernel.h>
#include <linux/spinlock.h>
#include <linux/device.h>
diff --git a/drivers/dma_v3/dmaengine_v3.c b/drivers/dma_v3/dmaengine_v3.c
index 04db9a2..1f91974 100644
--- a/drivers/dma_v3/dmaengine_v3.c
+++ b/drivers/dma_v3/dmaengine_v3.c
@@ -31,32 +31,18 @@
*
* LOCKING:
*
- * The subsystem keeps two global lists, dma_device_list and dma_client_list.
- * Both of these are protected by a mutex, dma_list_mutex.
+ * The subsystem keeps a global list of dma_device structs it is protected by a
+ * mutex, dma_list_mutex.
+ *
+ * A subsystem can get access to a channel by calling dmaengine_get() followed
+ * by dma_find_channel(), or if it has need for an exclusive channel it can call
+ * dma_request_channel(). Once a channel is allocated a reference is taken
+ * against its corresponding driver to disable removal.
*
* Each device has a channels list, which runs unlocked but is never modified
* once the device is registered, it's just setup by the driver.
*
- * Each client is responsible for keeping track of the channels it uses. See
- * the definition of dma_event_callback in dmaengine.h.
- *
- * Each device has a kref, which is initialized to 1 when the device is
- * registered. A kref_get is done for each class_device registered. When the
- * class_device is released, the corresponding kref_put is done in the release
- * method. Every time one of the device's channels is allocated to a client,
- * a kref_get occurs. When the channel is freed, the corresponding kref_put
- * happens. The device's release function does a completion, so
- * unregister_device does a remove event, class_device_unregister, a kref_put
- * for the first reference, then waits on the completion for all other
- * references to finish.
- *
- * Each channel has an open-coded implementation of Rusty Russell's "bigref,"
- * with a kref and a per_cpu local_t. A dma_chan_get is called when a client
- * signals that it wants to use a channel, and dma_chan_put is called when
- * a channel is removed or a client using it is unregistered. A client can
- * take extra references per outstanding transaction, as is the case with
- * the NET DMA client. The release function does a kref_put on the device.
- * -ChrisL, DanW
+ * See Documentation/dmaengine.txt for more details
*/
#include <linux/init.h>
@@ -73,54 +59,89 @@
#include <linux/mutex.h>
#ifndef __GENKSYMS__
#include <linux/jiffies.h>
+#include <linux/idr.h>
#endif
+static inline const char *dma_chan_name(struct dma_chan *chan)
+{
+ return dev_name((struct device *)&chan->device_v3->dev);
+}
+
static DEFINE_MUTEX(dma_list_mutex);
static LIST_HEAD(dma_device_list);
-static LIST_HEAD(dma_client_list);
+static long dmaengine_ref_count;
+static struct idr dma_idr;
/* --- sysfs implementation --- */
+/**
+ * dev_to_dma_chan - convert a device pointer to the its sysfs container object
+ * @dev - device node
+ *
+ * Must be called under dma_list_mutex
+ */
+static struct dma_chan *dev_to_dma_chan(struct class_device *dev)
+{
+ struct dma_chan_dev *chan_dev;
+
+ chan_dev = container_of(dev, typeof(*chan_dev), class_dev);
+ return chan_dev->chan;
+}
+
static ssize_t show_memcpy_count(struct class_device *cd, char *buf)
{
- struct dma_chan *chan = container_of(cd, struct dma_chan, class_dev);
+ struct dma_chan *chan;
unsigned long count = 0;
int i;
+ int err;
- for_each_possible_cpu(i)
- count += per_cpu_ptr(chan->local, i)->memcpy_count;
+ mutex_lock(&dma_list_mutex);
+ chan = dev_to_dma_chan(cd);
+ if (chan) {
+ for_each_possible_cpu(i)
+ count += per_cpu_ptr(chan->local, i)->memcpy_count;
+ err = sprintf(buf, "%lu\n", count);
+ } else
+ err = -ENODEV;
+ mutex_unlock(&dma_list_mutex);
- return sprintf(buf, "%lu\n", count);
+ return err;
}
static ssize_t show_bytes_transferred(struct class_device *cd, char *buf)
{
- struct dma_chan *chan = container_of(cd, struct dma_chan, class_dev);
+ struct dma_chan *chan;
unsigned long count = 0;
int i;
+ int err;
- for_each_possible_cpu(i)
- count += per_cpu_ptr(chan->local, i)->bytes_transferred;
+ mutex_lock(&dma_list_mutex);
+ chan = dev_to_dma_chan(cd);
+ if (chan) {
+ for_each_possible_cpu(i)
+ count += per_cpu_ptr(chan->local, i)->bytes_transferred;
+ err = sprintf(buf, "%lu\n", count);
+ } else
+ err = -ENODEV;
+ mutex_unlock(&dma_list_mutex);
- return sprintf(buf, "%lu\n", count);
+ return err;
}
static ssize_t show_in_use(struct class_device *cd, char *buf)
{
- struct dma_chan *chan = container_of(cd, struct dma_chan, class_dev);
- int in_use = 0;
-
- if (unlikely(chan->slow_ref) &&
- atomic_read(&chan->refcount.refcount) > 1)
- in_use = 1;
- else {
- if (local_read(&(per_cpu_ptr(chan->local,
- get_cpu())->refcount)) > 0)
- in_use = 1;
- put_cpu();
- }
+ struct dma_chan *chan;
+ int err;
- return sprintf(buf, "%d\n", in_use);
+ mutex_lock(&dma_list_mutex);
+ chan = dev_to_dma_chan(cd);
+ if (chan)
+ err = sprintf(buf, "%d\n", chan->client_count);
+ else
+ err = -ENODEV;
+ mutex_unlock(&dma_list_mutex);
+
+ return err;
}
static struct class_device_attribute dma_class_attrs[] = {
@@ -130,76 +151,110 @@ static struct class_device_attribute dma_class_attrs[] = {
__ATTR_NULL
};
-static void dma_async_device_cleanup(struct kref *kref);
-
-static void dma_class_dev_release(struct class_device *cd)
+static void chan_dev_release(struct class_device *dev)
{
- struct dma_chan *chan = container_of(cd, struct dma_chan, class_dev);
- kref_put(&chan->device_v3->refcount, dma_async_device_cleanup);
+ struct dma_chan_dev *chan_dev;
+
+ chan_dev = container_of(dev, typeof(*chan_dev), class_dev);
+ if (atomic_dec_and_test(chan_dev->idr_ref)) {
+ mutex_lock(&dma_list_mutex);
+ idr_remove(&dma_idr, chan_dev->dev_id);
+ mutex_unlock(&dma_list_mutex);
+ kfree(chan_dev->idr_ref);
+ }
+ kfree(chan_dev);
}
static struct class dma_devclass = {
.name = "dma_v3",
.class_dev_attrs = dma_class_attrs,
- .release = dma_class_dev_release,
+ .release = chan_dev_release,
};
/* --- client and device registration --- */
-#define dma_chan_satisfies_mask(chan, mask) \
- __dma_chan_satisfies_mask((chan), &(mask))
+#define dma_device_satisfies_mask(device, mask) \
+ __dma_device_satisfies_mask((device), &(mask))
static int
-__dma_chan_satisfies_mask(struct dma_chan *chan, dma_cap_mask_t *want)
+__dma_device_satisfies_mask(struct dma_device_v3 *device, dma_cap_mask_t *want)
{
dma_cap_mask_t has;
- bitmap_and(has.bits, want->bits, chan->device_v3->cap_mask.bits,
+ bitmap_and(has.bits, want->bits, device->cap_mask.bits,
DMA_TX_TYPE_END);
return bitmap_equal(want->bits, has.bits, DMA_TX_TYPE_END);
}
+static struct module *dma_chan_to_owner(struct dma_chan *chan)
+{
+ return chan->device_v3->dev->driver->owner;
+}
+
/**
- * dma_client_chan_alloc - try to allocate channels to a client
- * @client: &dma_client
+ * balance_ref_count - catch up the channel reference count
+ * @chan - channel to balance ->client_count versus dmaengine_ref_count
*
- * Called with dma_list_mutex held.
+ * balance_ref_count must be called under dma_list_mutex
*/
-static void dma_client_chan_alloc(struct dma_client *client)
+static void balance_ref_count(struct dma_chan *chan)
{
- struct dma_device_v3 *device;
- struct dma_chan *chan;
- int desc; /* allocated descriptor count */
- enum dma_state_client ack;
+ struct module *owner = dma_chan_to_owner(chan);
- /* Find a channel */
- list_for_each_entry(device, &dma_device_list, global_node) {
- /* Does the client require a specific DMA controller? */
- if (client->slave && client->slave->dma_dev
- && client->slave->dma_dev != device->dev)
- continue;
+ while (chan->client_count < dmaengine_ref_count) {
+ __module_get(owner);
+ chan->client_count++;
+ }
+}
- list_for_each_entry(chan, &device->channels, device_node) {
- if (!dma_chan_satisfies_mask(chan, client->cap_mask))
- continue;
+/**
+ * dma_chan_get - try to grab a dma channel's parent driver module
+ * @chan - channel to grab
+ *
+ * Must be called under dma_list_mutex
+ */
+static int dma_chan_get(struct dma_chan *chan)
+{
+ int err = -ENODEV;
+ struct module *owner = dma_chan_to_owner(chan);
+
+ if (chan->client_count) {
+ __module_get(owner);
+ err = 0;
+ } else if (try_module_get(owner))
+ err = 0;
+
+ if (err == 0)
+ chan->client_count++;
+
+ /* allocate upon first client reference */
+ if (chan->client_count == 1 && err == 0) {
+ int desc_cnt = chan->device_v3->device_alloc_chan_resources(chan);
+
+ if (desc_cnt < 0) {
+ err = desc_cnt;
+ chan->client_count = 0;
+ module_put(owner);
+ } else if (!dma_has_cap(DMA_PRIVATE, chan->device_v3->cap_mask))
+ balance_ref_count(chan);
+ }
- desc = chan->device_v3->device_alloc_chan_resources(
- chan, client);
- if (desc >= 0) {
- ack = client->event_callback_v3(client,
- chan,
- DMA_STATE_RESOURCE_AVAILABLE);
+ return err;
+}
- /* we are done once this client rejects
- * an available resource
- */
- if (ack == DMA_ACK) {
- dma_chan_get(chan);
- chan->client_count++;
- } else if (ack == DMA_NAK)
- return;
- }
- }
- }
+/**
+ * dma_chan_put - drop a reference to a dma channel's parent driver module
+ * @chan - channel to release
+ *
+ * Must be called under dma_list_mutex
+ */
+static void dma_chan_put(struct dma_chan *chan)
+{
+ if (!chan->client_count)
+ return; /* this channel failed alloc_chan_resources */
+ chan->client_count--;
+ module_put(dma_chan_to_owner(chan));
+ if (chan->client_count == 0)
+ chan->device->device_free_chan_resources(chan);
}
enum dma_status dma_sync_wait_v3(struct dma_chan *chan, dma_cookie_t cookie)
@@ -221,138 +276,364 @@ enum dma_status dma_sync_wait_v3(struct dma_chan *chan, dma_cookie_t cookie)
EXPORT_SYMBOL(dma_sync_wait_v3);
/**
- * dma_chan_cleanup_v3 - release a DMA channel's resources
- * @kref: kernel reference structure that contains the DMA channel device
+ * dma_cap_mask_all - enable iteration over all operation types
*/
-void dma_chan_cleanup_v3(struct kref *kref)
+static dma_cap_mask_t dma_cap_mask_all;
+
+/**
+ * dma_chan_tbl_ent - tracks channel allocations per core/operation
+ * @chan - associated channel for this entry
+ */
+struct dma_chan_tbl_ent {
+ struct dma_chan *chan;
+};
+
+/**
+ * channel_table - percpu lookup table for memory-to-memory offload providers
+ */
+static struct dma_chan_tbl_ent *channel_table[DMA_TX_TYPE_END];
+
+static int __init dma_channel_table_init(void)
{
- struct dma_chan *chan = container_of(kref, struct dma_chan, refcount);
- chan->device_v3->device_free_chan_resources(chan);
- kref_put(&chan->device_v3->refcount, dma_async_device_cleanup);
+ enum dma_transaction_type cap;
+ int err = 0;
+
+ bitmap_fill(dma_cap_mask_all.bits, DMA_TX_TYPE_END);
+
+ /* 'interrupt', 'private', and 'slave' are channel capabilities,
+ * but are not associated with an operation so they do not need
+ * an entry in the channel_table
+ */
+ clear_bit(DMA_INTERRUPT, dma_cap_mask_all.bits);
+ clear_bit(DMA_PRIVATE, dma_cap_mask_all.bits);
+ clear_bit(DMA_SLAVE, dma_cap_mask_all.bits);
+
+ for_each_dma_cap_mask(cap, dma_cap_mask_all) {
+ channel_table[cap] = alloc_percpu(struct dma_chan_tbl_ent);
+ if (!channel_table[cap]) {
+ err = -ENOMEM;
+ break;
+ }
+ }
+
+ if (err) {
+ pr_err("dmaengine: initialization failure\n");
+ for_each_dma_cap_mask(cap, dma_cap_mask_all)
+ if (channel_table[cap])
+ free_percpu(channel_table[cap]);
+ }
+
+ return err;
}
-EXPORT_SYMBOL(dma_chan_cleanup_v3);
+arch_initcall(dma_channel_table_init);
-static void dma_chan_free_rcu(struct rcu_head *rcu)
+/**
+ * dma_find_channel - find a channel to carry out the operation
+ * @tx_type: transaction type
+ */
+struct dma_chan *dma_find_channel(enum dma_transaction_type tx_type)
{
- struct dma_chan *chan = container_of(rcu, struct dma_chan, rcu);
- int bias = 0x7FFFFFFF;
- int i;
- for_each_possible_cpu(i)
- bias -= local_read(&per_cpu_ptr(chan->local, i)->refcount);
- atomic_sub(bias, &chan->refcount.refcount);
- kref_put(&chan->refcount, dma_chan_cleanup_v3);
+ struct dma_chan *chan;
+ int cpu;
+
+ cpu = get_cpu();
+ chan = per_cpu_ptr(channel_table[tx_type], cpu)->chan;
+ put_cpu();
+
+ return chan;
}
+EXPORT_SYMBOL_GPL(dma_find_channel);
-static void dma_chan_release(struct dma_chan *chan)
+/**
+ * dma_issue_pending_all - flush all pending operations across all channels
+ */
+void dma_issue_pending_all(void)
{
- atomic_add(0x7FFFFFFF, &chan->refcount.refcount);
- chan->slow_ref = 1;
- call_rcu(&chan->rcu, dma_chan_free_rcu);
+ struct dma_device_v3 *device;
+ struct dma_chan *chan;
+
+ rcu_read_lock();
+ list_for_each_entry_rcu(device, &dma_device_list, global_node) {
+ if (dma_has_cap(DMA_PRIVATE, device->cap_mask))
+ continue;
+ list_for_each_entry(chan, &device->channels, device_node)
+ if (chan->client_count)
+ device->device_issue_pending(chan);
+ }
+ rcu_read_unlock();
}
+EXPORT_SYMBOL_GPL(dma_issue_pending_all);
/**
- * dma_chans_notify_available - broadcast available channels to the clients
+ * nth_chan - returns the nth channel of the given capability
+ * @cap: capability to match
+ * @n: nth channel desired
+ *
+ * Defaults to returning the channel with the desired capability and the
+ * lowest reference count when 'n' cannot be satisfied. Must be called
+ * under dma_list_mutex.
*/
-static void dma_clients_notify_available(void)
+static struct dma_chan *nth_chan(enum dma_transaction_type cap, int n)
{
- struct dma_client *client;
+ struct dma_device_v3 *device;
+ struct dma_chan *chan;
+ struct dma_chan *ret = NULL;
+ struct dma_chan *min = NULL;
- mutex_lock(&dma_list_mutex);
+ list_for_each_entry(device, &dma_device_list, global_node) {
+ if (!dma_has_cap(cap, device->cap_mask) ||
+ dma_has_cap(DMA_PRIVATE, device->cap_mask))
+ continue;
+ list_for_each_entry(chan, &device->channels, device_node) {
+ if (!chan->client_count)
+ continue;
+ if (!min)
+ min = chan;
+ else if (chan->table_count < min->table_count)
+ min = chan;
+
+ if (n-- == 0) {
+ ret = chan;
+ break; /* done */
+ }
+ }
+ if (ret)
+ break; /* done */
+ }
- list_for_each_entry(client, &dma_client_list, global_node)
- dma_client_chan_alloc(client);
+ if (!ret)
+ ret = min;
- mutex_unlock(&dma_list_mutex);
+ if (ret)
+ ret->table_count++;
+
+ return ret;
}
/**
- * dma_chans_notify_available - tell the clients that a channel is going away
- * @chan: channel on its way out
+ * dma_channel_rebalance - redistribute the available channels
+ *
+ * Optimize for cpu isolation (each cpu gets a dedicated channel for an
+ * operation type) in the SMP case, and operation isolation (avoid
+ * multi-tasking channels) in the non-SMP case. Must be called under
+ * dma_list_mutex.
*/
-static void dma_clients_notify_removed(struct dma_chan *chan)
+static void dma_channel_rebalance(void)
{
- struct dma_client *client;
- enum dma_state_client ack;
+ struct dma_chan *chan;
+ struct dma_device_v3 *device;
+ int cpu;
+ int cap;
+ int n;
- mutex_lock(&dma_list_mutex);
+ /* undo the last distribution */
+ for_each_dma_cap_mask(cap, dma_cap_mask_all)
+ for_each_possible_cpu(cpu)
+ per_cpu_ptr(channel_table[cap], cpu)->chan = NULL;
- list_for_each_entry(client, &dma_client_list, global_node) {
- ack = client->event_callback_v3(client, chan,
- DMA_STATE_RESOURCE_REMOVED);
+ list_for_each_entry(device, &dma_device_list, global_node) {
+ if (dma_has_cap(DMA_PRIVATE, device->cap_mask))
+ continue;
+ list_for_each_entry(chan, &device->channels, device_node)
+ chan->table_count = 0;
+ }
- /* client was holding resources for this channel so
- * free it
- */
- if (ack == DMA_ACK) {
- dma_chan_put(chan);
- chan->client_count--;
+ /* don't populate the channel_table if no clients are available */
+ if (!dmaengine_ref_count)
+ return;
+
+ /* redistribute available channels */
+ n = 0;
+ for_each_dma_cap_mask(cap, dma_cap_mask_all)
+ for_each_online_cpu(cpu) {
+ if (num_possible_cpus() > 1)
+ chan = nth_chan(cap, n++);
+ else
+ chan = nth_chan(cap, -1);
+
+ per_cpu_ptr(channel_table[cap], cpu)->chan = chan;
}
+}
+
+static struct dma_chan *private_candidate(dma_cap_mask_t *mask, struct dma_device_v3 *dev,
+ dma_filter_fn fn, void *fn_param)
+{
+ struct dma_chan *chan;
+
+ if (!__dma_device_satisfies_mask(dev, mask)) {
+ pr_debug("%s: wrong capabilities\n", __func__);
+ return NULL;
}
+ /* devices with multiple channels need special handling as we need to
+ * ensure that all channels are either private or public.
+ */
+ if (dev->chancnt > 1 && !dma_has_cap(DMA_PRIVATE, dev->cap_mask))
+ list_for_each_entry(chan, &dev->channels, device_node) {
+ /* some channels are already publicly allocated */
+ if (chan->client_count)
+ return NULL;
+ }
- mutex_unlock(&dma_list_mutex);
+ list_for_each_entry(chan, &dev->channels, device_node) {
+ if (chan->client_count) {
+ pr_debug("%s: %s busy\n",
+ __func__, dma_chan_name(chan));
+ continue;
+ }
+ if (fn && !fn(chan, fn_param)) {
+ pr_debug("%s: %s filter said false\n",
+ __func__, dma_chan_name(chan));
+ continue;
+ }
+ return chan;
+ }
+
+ return NULL;
}
/**
- * dma_async_client_register_v3 - register a &dma_client
- * @client: ptr to a client structure with valid 'event_callback' and 'cap_mask'
+ * dma_request_channel - try to allocate an exclusive channel
+ * @mask: capabilities that the channel must satisfy
+ * @fn: optional callback to disposition available channels
+ * @fn_param: opaque parameter to pass to dma_filter_fn
*/
-void dma_async_client_register_v3(struct dma_client *client)
+struct dma_chan *__dma_request_channel(dma_cap_mask_t *mask, dma_filter_fn fn, void *fn_param)
{
- /* validate client data */
- BUG_ON(dma_has_cap(DMA_SLAVE, client->cap_mask) &&
- !client->slave);
+ struct dma_device_v3 *device, *_d;
+ struct dma_chan *chan = NULL;
+ int err;
+ /* Find a channel */
mutex_lock(&dma_list_mutex);
- list_add_tail(&client->global_node, &dma_client_list);
+ list_for_each_entry_safe(device, _d, &dma_device_list, global_node) {
+ chan = private_candidate(mask, device, fn, fn_param);
+ if (chan) {
+ /* Found a suitable channel, try to grab, prep, and
+ * return it. We first set DMA_PRIVATE to disable
+ * balance_ref_count as this channel will not be
+ * published in the general-purpose allocator
+ */
+ dma_cap_set(DMA_PRIVATE, device->cap_mask);
+ device->privatecnt++;
+ err = dma_chan_get(chan);
+
+ if (err == -ENODEV) {
+ pr_debug("%s: %s module removed\n", __func__,
+ dma_chan_name(chan));
+ list_del_rcu(&device->global_node);
+ } else if (err)
+ pr_err("dmaengine: failed to get %s: (%d)\n",
+ dma_chan_name(chan), err);
+ else
+ break;
+ if (--device->privatecnt == 0)
+ dma_cap_clear(DMA_PRIVATE, device->cap_mask);
+ chan->private = NULL;
+ chan = NULL;
+ }
+ }
mutex_unlock(&dma_list_mutex);
+
+ pr_debug("%s: %s (%s)\n", __func__, chan ? "success" : "fail",
+ chan ? dma_chan_name(chan) : NULL);
+
+ return chan;
}
-EXPORT_SYMBOL(dma_async_client_register_v3);
+EXPORT_SYMBOL_GPL(__dma_request_channel);
+
+void dma_release_channel(struct dma_chan *chan)
+{
+ mutex_lock(&dma_list_mutex);
+ WARN_ONCE(chan->client_count != 1,
+ "chan reference count %d != 1\n", chan->client_count);
+ dma_chan_put(chan);
+ /* drop PRIVATE cap enabled by __dma_request_channel() */
+ if (--chan->device_v3->privatecnt == 0)
+ dma_cap_clear(DMA_PRIVATE, chan->device_v3->cap_mask);
+ chan->private = NULL;
+ mutex_unlock(&dma_list_mutex);
+}
+EXPORT_SYMBOL_GPL(dma_release_channel);
/**
- * dma_async_client_unregister_v3 - unregister a client and free the &dma_client
- * @client: &dma_client to free
- *
- * Force frees any allocated DMA channels, frees the &dma_client memory
+ * dmaengine_get - register interest in dma_channels
*/
-void dma_async_client_unregister_v3(struct dma_client *client)
+void dmaengine_get(void)
{
- struct dma_device_v3 *device;
+ struct dma_device_v3 *device, *_d;
struct dma_chan *chan;
- enum dma_state_client ack;
-
- if (!client)
- return;
+ int err;
mutex_lock(&dma_list_mutex);
- /* free all channels the client is holding */
- list_for_each_entry(device, &dma_device_list, global_node)
- list_for_each_entry(chan, &device->channels, device_node) {
- ack = client->event_callback_v3(client, chan,
- DMA_STATE_RESOURCE_REMOVED);
+ dmaengine_ref_count++;
- if (ack == DMA_ACK) {
- dma_chan_put(chan);
- chan->client_count--;
- }
+ /* try to grab channels */
+ list_for_each_entry_safe(device, _d, &dma_device_list, global_node) {
+ if (dma_has_cap(DMA_PRIVATE, device->cap_mask))
+ continue;
+ list_for_each_entry(chan, &device->channels, device_node) {
+ err = dma_chan_get(chan);
+ if (err == -ENODEV) {
+ /* module removed before we could use it */
+ list_del_rcu(&device->global_node);
+ break;
+ } else if (err)
+ pr_err("dmaengine: failed to get %s: (%d)\n",
+ dma_chan_name(chan), err);
}
+ }
+
+ /* if this is the first reference and there were channels
+ * waiting we need to rebalance to get those channels
+ * incorporated into the channel table
+ */
+ if (dmaengine_ref_count == 1)
+ dma_channel_rebalance();
- list_del(&client->global_node);
mutex_unlock(&dma_list_mutex);
}
-EXPORT_SYMBOL(dma_async_client_unregister_v3);
+EXPORT_SYMBOL_GPL(dmaengine_get);
/**
- * dma_async_client_chan_request_v3 - send all available channels to the
- * client that satisfy the capability mask
- * @client - requester
+ * dmaengine_put - let dma drivers be removed when ref_count == 0
*/
-void dma_async_client_chan_request_v3(struct dma_client *client)
+void dmaengine_put(void)
+{
+ struct dma_device_v3 *device;
+ struct dma_chan *chan;
+
+ mutex_lock(&dma_list_mutex);
+ dmaengine_ref_count--;
+ BUG_ON(dmaengine_ref_count < 0);
+ /* drop channel references */
+ list_for_each_entry(device, &dma_device_list, global_node) {
+ if (dma_has_cap(DMA_PRIVATE, device->cap_mask))
+ continue;
+ list_for_each_entry(chan, &device->channels, device_node)
+ dma_chan_put(chan);
+ }
+
+ mutex_unlock(&dma_list_mutex);
+}
+EXPORT_SYMBOL_GPL(dmaengine_put);
+
+static int get_dma_id(struct dma_device_v3 *device)
{
+ int rc;
+
+ idr_retry:
+ if (!idr_pre_get(&dma_idr, GFP_KERNEL))
+ return -ENOMEM;
mutex_lock(&dma_list_mutex);
- dma_client_chan_alloc(client);
+ rc = idr_get_new(&dma_idr, NULL, &device->dev_id);
mutex_unlock(&dma_list_mutex);
+ if (rc == -EAGAIN)
+ goto idr_retry;
+ else if (rc != 0)
+ return rc;
+
+ return 0;
}
-EXPORT_SYMBOL(dma_async_client_chan_request_v3);
/**
* dma_async_device_register_v3 - registers DMA devices found
@@ -360,9 +641,9 @@ EXPORT_SYMBOL(dma_async_client_chan_request_v3);
*/
int dma_async_device_register_v3(struct dma_device_v3 *device)
{
- static int id;
int chancnt = 0, rc;
struct dma_chan* chan;
+ atomic_t *idr_ref;
if (!device)
return -ENODEV;
@@ -389,54 +670,93 @@ int dma_async_device_register_v3(struct dma_device_v3 *device)
BUG_ON(!device->device_issue_pending);
BUG_ON(!device->dev);
- init_completion(&device->done);
- kref_init(&device->refcount);
- device->dev_id = id++;
+ idr_ref = kmalloc(sizeof(*idr_ref), GFP_KERNEL);
+ if (!idr_ref)
+ return -ENOMEM;
+ rc = get_dma_id(device);
+ if (rc != 0) {
+ kfree(idr_ref);
+ return rc;
+ }
+
+ atomic_set(idr_ref, 0);
/* represent channels in sysfs. Probably want devs too */
list_for_each_entry(chan, &device->channels, device_node) {
+ rc = -ENOMEM;
chan->local = alloc_percpu(typeof(*chan->local));
if (chan->local == NULL)
- continue;
+ goto err_out;
+ chan->dev = kzalloc(sizeof(*chan->dev), GFP_KERNEL);
+ if (chan->dev == NULL) {
+ free_percpu(chan->local);
+ chan->local = NULL;
+ goto err_out;
+ }
chan->chan_id = chancnt++;
- chan->class_dev.class = &dma_devclass;
- chan->class_dev.dev = NULL;
+ chan->dev->class_dev.class = &dma_devclass;
+ chan->dev->chan = chan;
+ chan->dev->idr_ref = idr_ref;
+ chan->dev->dev_id = device->dev_id;
+ atomic_inc(idr_ref);
snprintf(chan->class_dev.class_id, BUS_ID_SIZE, "dma%dchan%d",
- device->dev_id, chan->chan_id);
+ device->dev_id, chan->chan_id);
rc = class_device_register(&chan->class_dev);
if (rc) {
- chancnt--;
free_percpu(chan->local);
chan->local = NULL;
+ kfree(chan->dev);
+ atomic_dec(idr_ref);
goto err_out;
}
-
- /* One for the channel, one of the class device */
- kref_get(&device->refcount);
- kref_get(&device->refcount);
- kref_init(&chan->refcount);
chan->client_count = 0;
- chan->slow_ref = 0;
- INIT_RCU_HEAD(&chan->rcu);
}
+ device->chancnt = chancnt;
mutex_lock(&dma_list_mutex);
- list_add_tail(&device->global_node, &dma_device_list);
+ /* take references on public channels */
+ if (dmaengine_ref_count && !dma_has_cap(DMA_PRIVATE, device->cap_mask))
+ list_for_each_entry(chan, &device->channels, device_node) {
+ /* if clients are already waiting for channels we need
+ * to take references on their behalf
+ */
+ if (dma_chan_get(chan) == -ENODEV) {
+ /* note we can only get here for the first
+ * channel as the remaining channels are
+ * guaranteed to get a reference
+ */
+ rc = -ENODEV;
+ mutex_unlock(&dma_list_mutex);
+ goto err_out;
+ }
+ }
+ list_add_tail_rcu(&device->global_node, &dma_device_list);
+ if (dma_has_cap(DMA_PRIVATE, device->cap_mask))
+ device->privatecnt++; /* Always private */
+ dma_channel_rebalance();
mutex_unlock(&dma_list_mutex);
- dma_clients_notify_available();
-
return 0;
err_out:
+ /* if we never registered a channel just release the idr */
+ if (atomic_read(idr_ref) == 0) {
+ mutex_lock(&dma_list_mutex);
+ idr_remove(&dma_idr, device->dev_id);
+ mutex_unlock(&dma_list_mutex);
+ kfree(idr_ref);
+ return rc;
+ }
+
list_for_each_entry(chan, &device->channels, device_node) {
if (chan->local == NULL)
continue;
- kref_put(&device->refcount, dma_async_device_cleanup);
+ mutex_lock(&dma_list_mutex);
+ chan->dev->chan = NULL;
+ mutex_unlock(&dma_list_mutex);
class_device_unregister(&chan->class_dev);
- chancnt--;
free_percpu(chan->local);
}
return rc;
@@ -444,37 +764,30 @@ err_out:
EXPORT_SYMBOL(dma_async_device_register_v3);
/**
- * dma_async_device_cleanup - function called when all references are released
- * @kref: kernel reference object
- */
-static void dma_async_device_cleanup(struct kref *kref)
-{
- struct dma_device_v3 *device;
-
- device = container_of(kref, struct dma_device_v3, refcount);
- complete(&device->done);
-}
-
-/**
- * dma_async_device_unregister_v3 - unregisters DMA devices
+ * dma_async_device_unregister_v3 - unregister a DMA device
* @device: &dma_device
+ *
+ * This routine is called by dma driver exit routines, dmaengine holds module
+ * references to prevent it being called while channels are in use.
*/
void dma_async_device_unregister_v3(struct dma_device_v3 *device)
{
struct dma_chan *chan;
mutex_lock(&dma_list_mutex);
- list_del(&device->global_node);
+ list_del_rcu(&device->global_node);
+ dma_channel_rebalance();
mutex_unlock(&dma_list_mutex);
list_for_each_entry(chan, &device->channels, device_node) {
- dma_clients_notify_removed(chan);
+ WARN_ONCE(chan->client_count,
+ "%s called while %d clients hold a reference\n",
+ __func__, chan->client_count);
+ mutex_lock(&dma_list_mutex);
+ chan->dev->chan = NULL;
+ mutex_unlock(&dma_list_mutex);
class_device_unregister(&chan->class_dev);
- dma_chan_release(chan);
}
-
- kref_put(&device->refcount, dma_async_device_cleanup);
- wait_for_completion(&device->done);
}
EXPORT_SYMBOL(dma_async_device_unregister_v3);
@@ -499,11 +812,14 @@ dma_async_memcpy_buf_to_buf_v3(struct dma_chan *chan, void *dest,
dma_addr_t dma_dest, dma_src;
dma_cookie_t cookie;
int cpu;
+ unsigned long flags;
dma_src = dma_map_single(dev->dev, src, len, DMA_TO_DEVICE);
dma_dest = dma_map_single(dev->dev, dest, len, DMA_FROM_DEVICE);
- tx = dev->device_prep_dma_memcpy(chan, dma_dest, dma_src, len,
- DMA_CTRL_ACK);
+ flags = DMA_CTRL_ACK |
+ DMA_COMPL_SRC_UNMAP_SINGLE |
+ DMA_COMPL_DEST_UNMAP_SINGLE;
+ tx = dev->device_prep_dma_memcpy(chan, dma_dest, dma_src, len, flags);
if (!tx) {
dma_unmap_single(dev->dev, dma_src, len, DMA_TO_DEVICE);
@@ -545,11 +861,12 @@ dma_async_memcpy_buf_to_pg_v3(struct dma_chan *chan, struct page *page,
dma_addr_t dma_dest, dma_src;
dma_cookie_t cookie;
int cpu;
+ unsigned long flags;
dma_src = dma_map_single(dev->dev, kdata, len, DMA_TO_DEVICE);
dma_dest = dma_map_page(dev->dev, page, offset, len, DMA_FROM_DEVICE);
- tx = dev->device_prep_dma_memcpy(chan, dma_dest, dma_src, len,
- DMA_CTRL_ACK);
+ flags = DMA_CTRL_ACK | DMA_COMPL_SRC_UNMAP_SINGLE;
+ tx = dev->device_prep_dma_memcpy(chan, dma_dest, dma_src, len, flags);
if (!tx) {
dma_unmap_single(dev->dev, dma_src, len, DMA_TO_DEVICE);
@@ -593,12 +910,13 @@ dma_async_memcpy_pg_to_pg_v3(struct dma_chan *chan, struct page *dest_pg,
dma_addr_t dma_dest, dma_src;
dma_cookie_t cookie;
int cpu;
+ unsigned long flags;
dma_src = dma_map_page(dev->dev, src_pg, src_off, len, DMA_TO_DEVICE);
dma_dest = dma_map_page(dev->dev, dest_pg, dest_off, len,
DMA_FROM_DEVICE);
- tx = dev->device_prep_dma_memcpy(chan, dma_dest, dma_src, len,
- DMA_CTRL_ACK);
+ flags = DMA_CTRL_ACK;
+ tx = dev->device_prep_dma_memcpy(chan, dma_dest, dma_src, len, flags);
if (!tx) {
dma_unmap_page(dev->dev, dma_src, len, DMA_TO_DEVICE);
@@ -623,12 +941,102 @@ void dma_async_tx_descriptor_init_v3(struct dma_async_tx_descriptor *tx,
{
tx->chan = chan;
spin_lock_init(&tx->lock);
+ INIT_LIST_HEAD(&tx->tx_list);
}
EXPORT_SYMBOL(dma_async_tx_descriptor_init_v3);
+/* dma_wait_for_async_tx - spin wait for a transaction to complete
+ * @tx: in-flight transaction to wait on
+ *
+ * This routine assumes that tx was obtained from a call to async_memcpy,
+ * async_xor, async_memset, etc which ensures that tx is "in-flight" (prepped
+ * and submitted). Walking the parent chain is only meant to cover for DMA
+ * drivers that do not implement the DMA_INTERRUPT capability and may race with
+ * the driver's descriptor cleanup routine.
+ */
+enum dma_status
+dma_wait_for_async_tx(struct dma_async_tx_descriptor *tx)
+{
+ enum dma_status status;
+ struct dma_async_tx_descriptor *iter;
+ struct dma_async_tx_descriptor *parent;
+
+ if (!tx)
+ return DMA_SUCCESS;
+
+ WARN_ONCE(tx->parent, "%s: speculatively walking dependency chain for"
+ " %s\n", __func__, dma_chan_name(tx->chan));
+
+ /* poll through the dependency chain, return when tx is complete */
+ do {
+ iter = tx;
+
+ /* find the root of the unsubmitted dependency chain */
+ do {
+ parent = iter->parent;
+ if (!parent)
+ break;
+ else
+ iter = parent;
+ } while (parent);
+
+ /* there is a small window for ->parent == NULL and
+ * ->cookie == -EBUSY
+ */
+ while (iter->cookie == -EBUSY)
+ cpu_relax();
+
+ status = dma_sync_wait_v3(iter->chan, iter->cookie);
+ } while (status == DMA_IN_PROGRESS || (iter != tx));
+
+ return status;
+}
+EXPORT_SYMBOL_GPL(dma_wait_for_async_tx);
+
+/* dma_run_dependencies - helper routine for dma drivers to process
+ * (start) dependent operations on their target channel
+ * @tx: transaction with dependencies
+ */
+void dma_run_dependencies(struct dma_async_tx_descriptor *tx)
+{
+ struct dma_async_tx_descriptor *dep = tx->next;
+ struct dma_async_tx_descriptor *dep_next;
+ struct dma_chan *chan;
+
+ if (!dep)
+ return;
+
+ /* we'll submit tx->next now, so clear the link */
+ tx->next = NULL;
+ chan = dep->chan;
+
+ /* keep submitting up until a channel switch is detected
+ * in that case we will be called again as a result of
+ * processing the interrupt from async_tx_channel_switch
+ */
+ for (; dep; dep = dep_next) {
+ spin_lock_bh(&dep->lock);
+ dep->parent = NULL;
+ dep_next = dep->next;
+ if (dep_next && dep_next->chan == chan)
+ dep->next = NULL; /* ->next will be submitted */
+ else
+ dep_next = NULL; /* submit current dep and terminate */
+ spin_unlock_bh(&dep->lock);
+
+ dep->tx_submit(dep);
+ }
+
+ chan->device_v3->device_issue_pending(chan);
+}
+EXPORT_SYMBOL_GPL(dma_run_dependencies);
+
static int __init dma_bus_init(void)
{
+ idr_init(&dma_idr);
mutex_init(&dma_list_mutex);
return class_register(&dma_devclass);
}
-subsys_initcall(dma_bus_init);
+arch_initcall(dma_bus_init);
+
+
diff --git a/drivers/dma_v3/ioat.c b/drivers/dma_v3/ioat.c
index 8947880..04e5b01 100644
--- a/drivers/dma_v3/ioat.c
+++ b/drivers/dma_v3/ioat.c
@@ -1,6 +1,6 @@
/*
* Intel I/OAT DMA Linux driver
- * Copyright(c) 2007 Intel Corporation.
+ * Copyright(c) 2007 - 2009 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms and conditions of the GNU General Public License,
@@ -75,60 +75,10 @@ static int ioat_dca_enabled = 1;
module_param(ioat_dca_enabled, int, 0644);
MODULE_PARM_DESC(ioat_dca_enabled, "control support of dca service (default: 1)");
-static int ioat_setup_functionality(struct pci_dev *pdev, void __iomem *iobase)
-{
- struct ioat_device *device = pci_get_drvdata(pdev);
- u8 version;
- int err = 0;
-
- version = readb(iobase + IOAT_VER_OFFSET);
- switch (version) {
- case IOAT_VER_1_2:
- device->dma = ioat_dma_probe(pdev, iobase);
- if (device->dma && ioat_dca_enabled)
- device->dca = ioat_dca_init(pdev, iobase);
- break;
- case IOAT_VER_2_0:
- device->dma = ioat_dma_probe(pdev, iobase);
- if (device->dma && ioat_dca_enabled)
- device->dca = ioat2_dca_init(pdev, iobase);
- break;
- case IOAT_VER_3_0:
- device->dma = ioat_dma_probe(pdev, iobase);
- if (device->dma && ioat_dca_enabled)
- device->dca = ioat3_dca_init(pdev, iobase);
- break;
- default:
- err = -ENODEV;
- break;
- }
- if (!device->dma)
- err = -ENODEV;
- return err;
-}
-
-static void ioat_shutdown_functionality(struct pci_dev *pdev)
-{
- struct ioat_device *device = pci_get_drvdata(pdev);
-
- dev_err(&pdev->dev, "Removing dma and dca services\n");
- if (device->dca) {
- unregister_dca_provider(device->dca);
- free_dca_provider(device->dca);
- device->dca = NULL;
- }
-
- if (device->dma) {
- ioat_dma_remove(device->dma);
- device->dma = NULL;
- }
-}
-
static struct pci_driver ioat_pci_driver = {
.name = "ioatdma_v3",
.id_table = ioat_pci_tbl,
.probe = ioat_probe,
- .shutdown = ioat_shutdown_functionality,
.remove = __devexit_p(ioat_remove),
};
@@ -148,15 +98,15 @@ static int __devinit ioat_probe(struct pci_dev *pdev,
if (err)
goto err_request_regions;
- err = pci_set_dma_mask(pdev, DMA_64BIT_MASK);
+ err = pci_set_dma_mask(pdev, DMA_BIT_MASK(64));
if (err)
- err = pci_set_dma_mask(pdev, DMA_32BIT_MASK);
+ err = pci_set_dma_mask(pdev, DMA_BIT_MASK(32));
if (err)
goto err_set_dma_mask;
- err = pci_set_consistent_dma_mask(pdev, DMA_64BIT_MASK);
+ err = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64));
if (err)
- err = pci_set_consistent_dma_mask(pdev, DMA_32BIT_MASK);
+ err = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(32));
if (err)
goto err_set_dma_mask;
@@ -179,7 +129,29 @@ static int __devinit ioat_probe(struct pci_dev *pdev,
pci_set_master(pdev);
- err = ioat_setup_functionality(pdev, iobase);
+ switch (readb(iobase + IOAT_VER_OFFSET)) {
+ case IOAT_VER_1_2:
+ device->dma = ioat_dma_probe(pdev, iobase);
+ if (device->dma && ioat_dca_enabled)
+ device->dca = ioat_dca_init(pdev, iobase);
+ break;
+ case IOAT_VER_2_0:
+ device->dma = ioat_dma_probe(pdev, iobase);
+ if (device->dma && ioat_dca_enabled)
+ device->dca = ioat2_dca_init(pdev, iobase);
+ break;
+ case IOAT_VER_3_0:
+ device->dma = ioat_dma_probe(pdev, iobase);
+ if (device->dma && ioat_dca_enabled)
+ device->dca = ioat3_dca_init(pdev, iobase);
+ break;
+ default:
+ err = -ENODEV;
+ break;
+ }
+ if (!device->dma)
+ err = -ENODEV;
+
if (err)
goto err_version;
@@ -198,17 +170,21 @@ err_enable_device:
return err;
}
-/*
- * It is unsafe to remove this module: if removed while a requested
- * dma is outstanding, esp. from tcp, it is possible to hang while
- * waiting for something that will never finish. However, if you're
- * feeling lucky, this usually works just fine.
- */
static void __devexit ioat_remove(struct pci_dev *pdev)
{
struct ioat_device *device = pci_get_drvdata(pdev);
- ioat_shutdown_functionality(pdev);
+ dev_err(&pdev->dev, "Removing dma and dca services\n");
+ if (device->dca) {
+ unregister_dca_provider(device->dca);
+ free_dca_provider(device->dca);
+ device->dca = NULL;
+ }
+
+ if (device->dma) {
+ ioat_dma_remove(device->dma);
+ device->dma = NULL;
+ }
kfree(device);
}
diff --git a/drivers/dma_v3/ioat_dca.c b/drivers/dma_v3/ioat_dca.c
index 85bc467..dae8695 100644
--- a/drivers/dma_v3/ioat_dca.c
+++ b/drivers/dma_v3/ioat_dca.c
@@ -1,6 +1,6 @@
/*
* Intel I/OAT DMA Linux driver
- * Copyright(c) 2007 Intel Corporation.
+ * Copyright(c) 2007 - 2009 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms and conditions of the GNU General Public License,
@@ -49,6 +49,23 @@
#define DCA_TAG_MAP_MASK 0xDF
+/* expected tag map bytes for I/OAT ver.2 */
+#define DCA2_TAG_MAP_BYTE0 0x80
+#define DCA2_TAG_MAP_BYTE1 0x0
+#define DCA2_TAG_MAP_BYTE2 0x81
+#define DCA2_TAG_MAP_BYTE3 0x82
+#define DCA2_TAG_MAP_BYTE4 0x82
+
+/* verify if tag map matches expected values */
+static inline int dca2_tag_map_valid(u8 *tag_map)
+{
+ return ((tag_map[0] == DCA2_TAG_MAP_BYTE0) &&
+ (tag_map[1] == DCA2_TAG_MAP_BYTE1) &&
+ (tag_map[2] == DCA2_TAG_MAP_BYTE2) &&
+ (tag_map[3] == DCA2_TAG_MAP_BYTE3) &&
+ (tag_map[4] == DCA2_TAG_MAP_BYTE4));
+}
+
/*
* "Legacy" DCA systems do not implement the DCA register set in the
* I/OAT device. Software needs direct support for their tag mappings.
@@ -284,6 +301,13 @@ struct dca_provider *ioat_dca_init(struct pci_dev *pdev, void __iomem *iobase)
for (i = 0; i < IOAT_TAG_MAP_LEN; i++)
ioatdca->tag_map[i] = tag_map[i];
+ if (!dca2_tag_map_valid(ioatdca->tag_map)) {
+ dev_err(&pdev->dev, "APICID_TAG_MAP set incorrectly by BIOS, "
+ "disabling DCA\n");
+ free_dca_provider(dca);
+ return NULL;
+ }
+
err = register_dca_provider(dca, &pdev->dev);
if (err) {
free_dca_provider(dca);
diff --git a/drivers/dma_v3/ioat_dma.c b/drivers/dma_v3/ioat_dma.c
index f62c95f..b08356b 100644
--- a/drivers/dma_v3/ioat_dma.c
+++ b/drivers/dma_v3/ioat_dma.c
@@ -1,6 +1,6 @@
/*
* Intel I/OAT DMA Linux driver
- * Copyright(c) 2004 - 2007 Intel Corporation.
+ * Copyright(c) 2004 - 2009 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms and conditions of the GNU General Public License,
@@ -188,11 +188,13 @@ static int ioat_dma_enumerate_channels(struct ioatdma_device *device)
ioat_chan->desccount = 0;
INIT_WORK(&ioat_chan->work, (void*)ioat_dma_chan_reset_part2,
&ioat_chan->work);
- if (ioat_chan->device->version != IOAT_VER_1_2) {
- writel(IOAT_DCACTRL_CMPL_WRITE_ENABLE
- | IOAT_DMA_DCA_ANY_CPU,
- ioat_chan->reg_base + IOAT_DCACTRL_OFFSET);
- }
+ if (ioat_chan->device->version == IOAT_VER_2_0)
+ writel(IOAT_DCACTRL_CMPL_WRITE_ENABLE |
+ IOAT_DMA_DCA_ANY_CPU,
+ ioat_chan->reg_base + IOAT_DCACTRL_OFFSET);
+ else if (ioat_chan->device->version == IOAT_VER_3_0)
+ writel(IOAT_DMA_DCA_ANY_CPU,
+ ioat_chan->reg_base + IOAT_DCACTRL_OFFSET);
spin_lock_init(&ioat_chan->cleanup_lock);
spin_lock_init(&ioat_chan->desc_lock);
INIT_LIST_HEAD(&ioat_chan->free_desc);
@@ -524,7 +526,7 @@ static dma_cookie_t ioat1_tx_submit(struct dma_async_tx_descriptor *tx)
}
hw->ctl = IOAT_DMA_DESCRIPTOR_CTL_CP_STS;
- if (new->async_tx.callback) {
+ if (first->async_tx.callback) {
hw->ctl |= IOAT_DMA_DESCRIPTOR_CTL_INT_GN;
if (first != new) {
/* move callback into to last desc */
@@ -616,7 +618,7 @@ static dma_cookie_t ioat2_tx_submit(struct dma_async_tx_descriptor *tx)
}
hw->ctl |= IOAT_DMA_DESCRIPTOR_CTL_CP_STS;
- if (new->async_tx.callback) {
+ if (first->async_tx.callback) {
hw->ctl |= IOAT_DMA_DESCRIPTOR_CTL_INT_GN;
if (first != new) {
/* move callback into to last desc */
@@ -690,7 +692,6 @@ static struct ioat_desc_sw *ioat_dma_alloc_descriptor(
desc_sw->async_tx.tx_submit = ioat2_tx_submit;
break;
}
- INIT_LIST_HEAD(&desc_sw->async_tx.tx_list);
desc_sw->hw = desc;
desc_sw->async_tx.phys = phys;
@@ -733,8 +734,7 @@ static void ioat2_dma_massage_chan_desc(struct ioat_dma_chan *ioat_chan)
* ioat_dma_alloc_chan_resources - returns the number of allocated descriptors
* @chan: the channel to be filled out
*/
-static int ioat_dma_alloc_chan_resources(struct dma_chan *chan,
- struct dma_client *client)
+static int ioat_dma_alloc_chan_resources(struct dma_chan *chan)
{
struct ioat_dma_chan *ioat_chan = to_ioat_chan(chan);
struct ioat_desc_sw *desc;
@@ -807,6 +807,12 @@ static void ioat_dma_free_chan_resources(struct dma_chan *chan)
struct ioat_desc_sw *desc, *_desc;
int in_use_descs = 0;
+ /* Before freeing channel resources first check
+ * if they have been previously allocated for this channel.
+ */
+ if (ioat_chan->desccount == 0)
+ return;
+
tasklet_disable(&ioat_chan->cleanup_task);
ioat_dma_memcpy_cleanup(ioat_chan);
@@ -870,6 +876,7 @@ static void ioat_dma_free_chan_resources(struct dma_chan *chan)
ioat_chan->pending = 0;
ioat_chan->dmacount = 0;
ioat_chan->watchdog_completion = 0;
+ ioat_chan->desccount = 0;
ioat_chan->last_compl_desc_addr_hw = 0;
ioat_chan->watchdog_tcp_cookie =
ioat_chan->watchdog_last_tcp_cookie = 0;
@@ -977,11 +984,9 @@ static struct ioat_desc_sw *ioat_dma_get_next_descriptor(
switch (ioat_chan->device->version) {
case IOAT_VER_1_2:
return ioat1_dma_get_next_descriptor(ioat_chan);
- break;
case IOAT_VER_2_0:
case IOAT_VER_3_0:
return ioat2_dma_get_next_descriptor(ioat_chan);
- break;
}
return NULL;
}
@@ -1058,22 +1063,31 @@ static void ioat_dma_cleanup_tasklet(unsigned long data)
static void
ioat_dma_unmap(struct ioat_dma_chan *ioat_chan, struct ioat_desc_sw *desc)
{
- /*
- * yes we are unmapping both _page and _single
- * alloc'd regions with unmap_page. Is this
- * *really* that bad?
- */
- if (!(desc->async_tx.flags & DMA_COMPL_SKIP_DEST_UNMAP))
- pci_unmap_page(ioat_chan->device->pdev,
- pci_unmap_addr(desc, dst),
- pci_unmap_len(desc, len),
- PCI_DMA_FROMDEVICE);
-
- if (!(desc->async_tx.flags & DMA_COMPL_SKIP_SRC_UNMAP))
- pci_unmap_page(ioat_chan->device->pdev,
- pci_unmap_addr(desc, src),
- pci_unmap_len(desc, len),
- PCI_DMA_TODEVICE);
+ if (!(desc->async_tx.flags & DMA_COMPL_SKIP_DEST_UNMAP)) {
+ if (desc->async_tx.flags & DMA_COMPL_DEST_UNMAP_SINGLE)
+ pci_unmap_single(ioat_chan->device->pdev,
+ pci_unmap_addr(desc, dst),
+ pci_unmap_len(desc, len),
+ PCI_DMA_FROMDEVICE);
+ else
+ pci_unmap_page(ioat_chan->device->pdev,
+ pci_unmap_addr(desc, dst),
+ pci_unmap_len(desc, len),
+ PCI_DMA_FROMDEVICE);
+ }
+
+ if (!(desc->async_tx.flags & DMA_COMPL_SKIP_SRC_UNMAP)) {
+ if (desc->async_tx.flags & DMA_COMPL_SRC_UNMAP_SINGLE)
+ pci_unmap_single(ioat_chan->device->pdev,
+ pci_unmap_addr(desc, src),
+ pci_unmap_len(desc, len),
+ PCI_DMA_TODEVICE);
+ else
+ pci_unmap_page(ioat_chan->device->pdev,
+ pci_unmap_addr(desc, src),
+ pci_unmap_len(desc, len),
+ PCI_DMA_TODEVICE);
+ }
}
/**
@@ -1165,9 +1179,8 @@ static void ioat_dma_memcpy_cleanup(struct ioat_dma_chan *ioat_chan)
if the client is done with the descriptor
*/
if (async_tx_test_ack(&desc->async_tx)) {
- list_del(&desc->node);
- list_add_tail(&desc->node,
- &ioat_chan->free_desc);
+ list_move_tail(&desc->node,
+ &ioat_chan->free_desc);
} else
desc->async_tx.cookie = 0;
} else {
@@ -1357,6 +1370,8 @@ static int ioat_dma_self_test(struct ioatdma_device *device)
dma_cookie_t cookie;
int err = 0;
struct completion cmp;
+ unsigned long tmo;
+ unsigned long flags;
src = kzalloc(sizeof(u8) * IOAT_TEST_SIZE, GFP_KERNEL);
if (!src)
@@ -1375,7 +1390,7 @@ static int ioat_dma_self_test(struct ioatdma_device *device)
dma_chan = container_of(device->common.channels.next,
struct dma_chan,
device_node);
- if (device->common.device_alloc_chan_resources(dma_chan, NULL) < 1) {
+ if (device->common.device_alloc_chan_resources(dma_chan) < 1) {
dev_err(&device->pdev->dev,
"selftest cannot allocate chan resource\n");
err = -ENODEV;
@@ -1386,8 +1401,9 @@ static int ioat_dma_self_test(struct ioatdma_device *device)
DMA_TO_DEVICE);
dma_dest = dma_map_single(dma_chan->device_v3->dev,dest, IOAT_TEST_SIZE,
DMA_FROM_DEVICE);
+ flags = DMA_COMPL_SRC_UNMAP_SINGLE | DMA_COMPL_DEST_UNMAP_SINGLE;
tx = device->common.device_prep_dma_memcpy(dma_chan, dma_dest, dma_src,
- IOAT_TEST_SIZE, 0);
+ IOAT_TEST_SIZE, flags);
if (!tx) {
dev_err(&device->pdev->dev,
"Self-test prep failed, disabling\n");
@@ -1408,9 +1424,10 @@ static int ioat_dma_self_test(struct ioatdma_device *device)
}
device->common.device_issue_pending(dma_chan);
- wait_for_completion_timeout(&cmp, msecs_to_jiffies(3000));
+ tmo = wait_for_completion_timeout(&cmp, msecs_to_jiffies(3000));
- if (device->common.device_is_tx_complete(dma_chan, cookie, NULL, NULL)
+ if (tmo == 0 ||
+ device->common.device_is_tx_complete(dma_chan, cookie, NULL, NULL)
!= DMA_SUCCESS) {
dev_err(&device->pdev->dev,
"Self-test copy timed out, disabling\n");
@@ -1652,6 +1669,13 @@ struct ioatdma_device *ioat_dma_probe(struct pci_dev *pdev,
" %d channels, device version 0x%02x, driver version %s\n",
device->common.chancnt, device->version, IOAT_DMA_VERSION);
+ if (!device->common.chancnt) {
+ dev_err(&device->pdev->dev,
+ "Intel(R) I/OAT DMA Engine problem found: "
+ "zero channels detected\n");
+ goto err_setup_interrupts;
+ }
+
err = ioat_dma_setup_interrupts(device);
if (err)
goto err_setup_interrupts;
@@ -1692,6 +1716,9 @@ void ioat_dma_remove(struct ioatdma_device *device)
struct dma_chan *chan, *_chan;
struct ioat_dma_chan *ioat_chan;
+ if (device->version != IOAT_VER_3_0)
+ cancel_delayed_work(&device->work);
+
ioat_dma_remove_interrupts(device);
dma_async_device_unregister_v3(&device->common);
@@ -1703,10 +1730,6 @@ void ioat_dma_remove(struct ioatdma_device *device)
pci_release_regions(device->pdev);
pci_disable_device(device->pdev);
- if (device->version != IOAT_VER_3_0) {
- cancel_delayed_work(&device->work);
- }
-
list_for_each_entry_safe(chan, _chan,
&device->common.channels, device_node) {
ioat_chan = to_ioat_chan(chan);
diff --git a/drivers/dma_v3/ioatdma.h b/drivers/dma_v3/ioatdma.h
index f59123f..3705f3e 100644
--- a/drivers/dma_v3/ioatdma.h
+++ b/drivers/dma_v3/ioatdma.h
@@ -1,5 +1,5 @@
/*
- * Copyright(c) 2004 - 2007 Intel Corporation. All rights reserved.
+ * Copyright(c) 2004 - 2009 Intel Corporation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the Free
@@ -30,7 +30,7 @@
#include <net/tcp.h>
#include <linux/workqueue.h>
-#define IOAT_DMA_VERSION "3.30-k1"
+#define IOAT_DMA_VERSION "3.64-k1"
enum ioat_interrupt {
none = 0,
diff --git a/drivers/dma_v3/ioatdma_hw.h b/drivers/dma_v3/ioatdma_hw.h
index 006b013..79c289f 100644
--- a/drivers/dma_v3/ioatdma_hw.h
+++ b/drivers/dma_v3/ioatdma_hw.h
@@ -1,5 +1,5 @@
/*
- * Copyright(c) 2004 - 2007 Intel Corporation. All rights reserved.
+ * Copyright(c) 2004 - 2009 Intel Corporation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the Free
diff --git a/drivers/dma_v3/ioatdma_registers.h b/drivers/dma_v3/ioatdma_registers.h
index 0f38e16..0f51176 100644
--- a/drivers/dma_v3/ioatdma_registers.h
+++ b/drivers/dma_v3/ioatdma_registers.h
@@ -1,5 +1,5 @@
/*
- * Copyright(c) 2004 - 2007 Intel Corporation. All rights reserved.
+ * Copyright(c) 2004 - 2009 Intel Corporation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the Free
diff --git a/drivers/dma_v3/iovlock_v3.c b/drivers/dma_v3/iovlock_v3.c
index da29205..7836ad2 100644
--- a/drivers/dma_v3/iovlock_v3.c
+++ b/drivers/dma_v3/iovlock_v3.c
@@ -55,7 +55,6 @@ struct dma_pinned_list *dma_pin_iovec_pages_v3(struct iovec *iov, size_t len)
int nr_iovecs = 0;
int iovec_len_used = 0;
int iovec_pages_used = 0;
- long err;
/* don't pin down non-user-based iovecs */
if (segment_eq(get_fs(), KERNEL_DS))
@@ -72,23 +71,21 @@ struct dma_pinned_list *dma_pin_iovec_pages_v3(struct iovec *iov, size_t len)
local_list = kmalloc(sizeof(*local_list)
+ (nr_iovecs * sizeof (struct dma_page_list))
+ (iovec_pages_used * sizeof (struct page*)), GFP_KERNEL);
- if (!local_list) {
- err = -ENOMEM;
+ if (!local_list)
goto out;
- }
/* list of pages starts right after the page list array */
pages = (struct page **) &local_list->page_list[nr_iovecs];
+ local_list->nr_iovecs = 0;
+
for (i = 0; i < nr_iovecs; i++) {
struct dma_page_list *page_list = &local_list->page_list[i];
len -= iov[i].iov_len;
- if (!access_ok(VERIFY_WRITE, iov[i].iov_base, iov[i].iov_len)) {
- err = -EFAULT;
+ if (!access_ok(VERIFY_WRITE, iov[i].iov_base, iov[i].iov_len))
goto unpin;
- }
page_list->nr_pages = num_pages_spanned(&iov[i]);
page_list->base_address_v3 = iov[i].iov_base;
@@ -109,10 +106,8 @@ struct dma_pinned_list *dma_pin_iovec_pages_v3(struct iovec *iov, size_t len)
NULL);
up_read(¤t->mm->mmap_sem);
- if (ret != page_list->nr_pages) {
- err = -ENOMEM;
+ if (ret != page_list->nr_pages)
goto unpin;
- }
local_list->nr_iovecs = i + 1;
}
@@ -122,7 +117,7 @@ struct dma_pinned_list *dma_pin_iovec_pages_v3(struct iovec *iov, size_t len)
unpin:
dma_unpin_iovec_pages_v3(local_list);
out:
- return ERR_PTR(err);
+ return NULL;
}
void dma_unpin_iovec_pages_v3(struct dma_pinned_list *pinned_list)
diff --git a/include/linux/dca.h b/include/linux/dca.h
index 1a1df85..3223c00 100644
--- a/include/linux/dca.h
+++ b/include/linux/dca.h
@@ -1,3 +1,23 @@
+/*
+ * Copyright(c) 2007 - 2009 Intel Corporation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the Free
+ * Software Foundation; either version 2 of the License, or (at your option)
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; if not, write to the Free Software Foundation, Inc., 59
+ * Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ *
+ * The full GNU General Public License is included in this distribution in the
+ * file called COPYING.
+ */
#ifndef DCA_H
#define DCA_H
/* DCA Provider API */
diff --git a/include/linux/dmaengine.h b/include/linux/dmaengine.h
index 21a17e6..718442e 100644
--- a/include/linux/dmaengine.h
+++ b/include/linux/dmaengine.h
@@ -25,13 +25,11 @@
#include <linux/device.h>
#include <linux/uio.h>
-#include <linux/kref.h>
-#include <linux/completion.h>
-#include <linux/rcupdate.h>
#ifndef __GENKSYMS__
#include <linux/dma-mapping.h>
#endif
+
/**
* enum dma_event - resource PNP/power managment events
* @DMA_RESOURCE_SUSPEND: DMA device going into low power state
@@ -96,6 +94,9 @@ struct dma_chan_percpu {
* @local: per-cpu pointer to a struct dma_chan_percpu
* @device_v3: pointer to dma_device version 3 structure
* @client_count: how many clients are using this channel
+ * @table_count: number of appearances in the mem-to-mem allocation table
+ * @dev: class device for sysfs
+ * @private: private data for certain client-channel associations
*/
struct dma_chan {
struct dma_client *client;
@@ -117,32 +118,29 @@ struct dma_chan {
#ifndef __GENKSYMS__
struct dma_device_v3 *device_v3;
int client_count;
+ int table_count;
+ struct dma_chan_dev *dev;
+ void *private;
#endif
};
+/**
+ * struct dma_chan_dev - relate sysfs device node to backing channel device
+ * @chan - driver channel device
+ * @device - sysfs device
+ * @dev_id - parent dma_device dev_id
+ * @idr_ref - reference count to gate release of dma_device dev_id
+ */
+struct dma_chan_dev {
+ struct dma_chan *chan;
+ struct class_device class_dev;
+ int dev_id;
+ atomic_t *idr_ref;
+};
+
void dma_chan_cleanup_v3(struct kref *kref);
void dma_chan_cleanup(struct kref *kref);
-static inline void dma_chan_get(struct dma_chan *chan)
-{
- if (unlikely(chan->slow_ref))
- kref_get(&chan->refcount);
- else {
- local_inc(&(per_cpu_ptr(chan->local, get_cpu())->refcount));
- put_cpu();
- }
-}
-
-static inline void dma_chan_put(struct dma_chan *chan)
-{
- if (unlikely(chan->slow_ref))
- kref_put(&chan->refcount, dma_chan_cleanup_v3);
- else {
- local_dec(&(per_cpu_ptr(chan->local, get_cpu())->refcount));
- put_cpu();
- }
-}
-
/*
* typedef dma_event_callback - function pointer to a DMA event callback
*/
@@ -197,6 +195,18 @@ struct dma_device {
struct dma_client *dma_async_client_register(dma_event_callback event_callback);
void dma_async_client_unregister(struct dma_client *client);
+#ifdef CONFIG_DMA_ENGINE_V3
+void dmaengine_get(void);
+void dmaengine_put(void);
+#else
+static inline void dmaengine_get(void)
+{
+}
+static inline void dmaengine_put(void)
+{
+}
+#endif
+
void dma_async_client_chan_request(struct dma_client *client,
unsigned int number);
@@ -365,32 +375,6 @@ dma_cookie_t dma_memcpy_pg_to_iovec_v3(struct dma_chan *chan, struct iovec *iov,
struct dma_pinned_list *pinned_list, struct page *page,
unsigned int offset, size_t len);
-/**
- * enum dma_state - resource PNP/power management state
- * @DMA_STATE_RESOURCE_SUSPEND: DMA device going into low power state
- * @DMA_STATE_RESOURCE_RESUME: DMA device returning to full power
- * @DMA_STATE_RESOURCE_AVAILABLE: DMA device available to the system
- * @DMA_STATE_RESOURCE_REMOVED: DMA device removed from the system
- */
-enum dma_state {
- DMA_STATE_RESOURCE_SUSPEND,
- DMA_STATE_RESOURCE_RESUME,
- DMA_STATE_RESOURCE_AVAILABLE,
- DMA_STATE_RESOURCE_REMOVED,
-};
-
-/**
- * enum dma_state_client - state of the channel in the client
- * @DMA_ACK: client would like to use, or was using this channel
- * @DMA_DUP: client has already seen this channel, or is not using this channel
- * @DMA_NAK: client does not want to see any more channels
- */
-enum dma_state_client {
- DMA_ACK,
- DMA_DUP,
- DMA_NAK,
-};
-
#define dma_submit_error(cookie) ((cookie) < 0 ? 1 : 0)
/**
@@ -407,6 +391,7 @@ enum dma_transaction_type {
DMA_MEMSET,
DMA_MEMCPY_CRC32C,
DMA_INTERRUPT,
+ DMA_PRIVATE,
DMA_SLAVE,
};
@@ -414,18 +399,6 @@ enum dma_transaction_type {
#define DMA_TX_TYPE_END (DMA_SLAVE + 1)
/**
- * enum dma_slave_width - DMA slave register access width.
- * @DMA_SLAVE_WIDTH_8BIT: Do 8-bit slave register accesses
- * @DMA_SLAVE_WIDTH_16BIT: Do 16-bit slave register accesses
- * @DMA_SLAVE_WIDTH_32BIT: Do 32-bit slave register accesses
- */
-enum dma_slave_width {
- DMA_SLAVE_WIDTH_8BIT,
- DMA_SLAVE_WIDTH_16BIT,
- DMA_SLAVE_WIDTH_32BIT,
-};
-
-/**
* enum dma_ctrl_flags - DMA flags to augment operation preparation,
* control completion, and communicate status.
* @DMA_PREP_INTERRUPT - trigger an interrupt (callback) upon completion of
@@ -435,12 +408,18 @@ enum dma_slave_width {
* dependency chains
* @DMA_COMPL_SKIP_SRC_UNMAP - set to disable dma-unmapping the source buffer(s)
* @DMA_COMPL_SKIP_DEST_UNMAP - set to disable dma-unmapping the destination(s)
+ * @DMA_COMPL_SRC_UNMAP_SINGLE - set to do the source dma-unmapping as single
+ * (if not set, do the source dma-unmapping as page)
+ * @DMA_COMPL_DEST_UNMAP_SINGLE - set to do the destination dma-unmapping as single
+ * (if not set, do the destination dma-unmapping as page)
*/
enum dma_ctrl_flags {
DMA_PREP_INTERRUPT = (1 << 0),
DMA_CTRL_ACK = (1 << 1),
DMA_COMPL_SKIP_SRC_UNMAP = (1 << 2),
DMA_COMPL_SKIP_DEST_UNMAP = (1 << 3),
+ DMA_COMPL_SRC_UNMAP_SINGLE = (1 << 4),
+ DMA_COMPL_DEST_UNMAP_SINGLE = (1 << 5),
};
/**
@@ -449,34 +428,8 @@ enum dma_ctrl_flags {
*/
typedef struct { DECLARE_BITMAP(bits, DMA_TX_TYPE_END); } dma_cap_mask_t;
-/**
- * struct dma_slave - Information about a DMA slave
- * @dev: device acting as DMA slave
- * @dma_dev: required DMA master device. If non-NULL, the client can not be
- * bound to other masters than this.
- * @tx_reg: physical address of data register used for
- * memory-to-peripheral transfers
- * @rx_reg: physical address of data register used for
- * peripheral-to-memory transfers
- * @reg_width: peripheral register width
- *
- * If dma_dev is non-NULL, the client can not be bound to other DMA
- * masters than the one corresponding to this device. The DMA master
- * driver may use this to determine if there is controller-specific
- * data wrapped around this struct. Drivers of platform code that sets
- * the dma_dev field must therefore make sure to use an appropriate
- * controller-specific dma slave structure wrapping this struct.
- */
-struct dma_slave {
- struct device *dev;
- struct device *dma_dev;
- dma_addr_t tx_reg;
- dma_addr_t rx_reg;
- enum dma_slave_width reg_width;
-};
-
/*
- * typedef dma_event_callback_v3 - function pointer to a DMA event callback
+ * typedef dma_event_callback - function pointer to a DMA event callback
* For each channel added to the system this routine is called for each client.
* If the client would like to use the channel it returns '1' to signal (ack)
* the dmaengine core to take out a reference on the channel and its
@@ -488,10 +441,6 @@ struct dma_slave {
* @chan - channel to be acted upon
* @state - available or removed
*/
-typedef enum dma_state_client (*dma_event_callback_v3)
- (struct dma_client *client,
- struct dma_chan *chan, enum dma_state state);
-
typedef void (*dma_async_tx_callback)(void *dma_async_param);
/**
@@ -527,9 +476,24 @@ struct dma_async_tx_descriptor {
spinlock_t lock;
};
+#ifdef CONFIG_DMA_ENGINE_V3
+enum dma_status dma_wait_for_async_tx(struct dma_async_tx_descriptor *tx);
+void dma_issue_pending_all(void);
+#else
+static inline enum dma_status dma_wait_for_async_tx(struct dma_async_tx_descriptor *tx)
+{
+ return DMA_SUCCESS;
+}
+static inline void dma_issue_pending_all(void)
+{
+ do { } while (0);
+}
+#endif
+
/**
* struct dma_device_v3 - info on the entity supplying DMA services
* @chancnt: how many DMA channels are supported
+ * @privatecnt: how many DMA channels are requested by dma_request_channel
* @channels: the list of struct dma_chan
* @global_node: list_head for global dma_device_list
* @cap_mask: one or more dma_capability flags
@@ -548,23 +512,21 @@ struct dma_async_tx_descriptor {
* @device_prep_dma_interrupt: prepares an end of chain interrupt operation
* @device_prep_slave_sg: prepares a slave dma operation
* @device_terminate_all: terminate all pending operations
+ * @device_is_tx_complete: poll for transaction completion
* @device_issue_pending: push pending transactions to hardware
*/
struct dma_device_v3 {
unsigned int chancnt;
+ unsigned int privatecnt;
struct list_head channels;
struct list_head global_node;
dma_cap_mask_t cap_mask;
int max_xor;
- struct kref refcount;
- struct completion done;
-
int dev_id;
struct device *dev;
- int (*device_alloc_chan_resources)(struct dma_chan *chan,
- struct dma_client *client);
+ int (*device_alloc_chan_resources)(struct dma_chan *chan);
void (*device_free_chan_resources)(struct dma_chan *chan);
struct dma_async_tx_descriptor *(*device_prep_dma_memcpy)(
@@ -596,9 +558,6 @@ struct dma_device_v3 {
/* --- public DMA engine API --- */
-void dma_async_client_register_v3(struct dma_client *client);
-void dma_async_client_unregister_v3(struct dma_client *client);
-void dma_async_client_chan_request_v3(struct dma_client *client);
dma_cookie_t dma_async_memcpy_buf_to_buf_v3(struct dma_chan *chan,
void *dest, void *src, size_t len);
dma_cookie_t dma_async_memcpy_buf_to_pg_v3(struct dma_chan *chan,
@@ -614,6 +573,11 @@ static inline void async_tx_ack(struct dma_async_tx_descriptor *tx)
tx->flags |= DMA_CTRL_ACK;
}
+static inline void async_tx_clear_ack(struct dma_async_tx_descriptor *tx)
+{
+ tx->flags &= ~DMA_CTRL_ACK;
+}
+
static inline bool async_tx_test_ack(struct dma_async_tx_descriptor *tx)
{
return (tx->flags & DMA_CTRL_ACK) == DMA_CTRL_ACK;
@@ -640,6 +604,19 @@ __dma_cap_set(enum dma_transaction_type tx_type, dma_cap_mask_t *dstp)
set_bit(tx_type, dstp->bits);
}
+#define dma_cap_clear(tx, mask) __dma_cap_clear((tx), &(mask))
+static inline void
+__dma_cap_clear(enum dma_transaction_type tx_type, dma_cap_mask_t *dstp)
+{
+ clear_bit(tx_type, dstp->bits);
+}
+
+#define dma_cap_zero(mask) __dma_cap_zero(&(mask))
+static inline void __dma_cap_zero(dma_cap_mask_t *dstp)
+{
+ bitmap_zero(dstp->bits, DMA_TX_TYPE_END);
+}
+
#define dma_has_cap(tx, mask) __dma_has_cap((tx), &(mask))
static inline int
__dma_has_cap(enum dma_transaction_type tx_type, dma_cap_mask_t *srcp)
@@ -690,6 +667,25 @@ static inline enum dma_status dma_async_is_tx_complete(struct dma_chan *chan,
int dma_async_device_register_v3(struct dma_device_v3 *device);
void dma_async_device_unregister_v3(struct dma_device_v3 *device);
+void dma_run_dependencies(struct dma_async_tx_descriptor *tx);
+struct dma_chan *dma_find_channel(enum dma_transaction_type tx_type);
+#define dma_request_channel(mask, x, y) __dma_request_channel(&(mask), x, y)
+void dma_release_channel(struct dma_chan *chan);
+
+/**
+ * typedef dma_filter_fn - callback filter for dma_request_channel
+ * @chan: channel to be reviewed
+ * @filter_param: opaque parameter passed through dma_request_channel
+ *
+ * When this optional parameter is specified in a call to dma_request_channel a
+ * suitable channel is passed to this routine for further dispositioning before
+ * being returned. Where 'suitable' indicates a non-busy channel that
+ * satisfies the given capability mask. It returns 'true' to indicate that the
+ * channel is suitable.
+ */
+typedef bool (*dma_filter_fn)(struct dma_chan *chan, void *filter_param);
+
+struct dma_chan *__dma_request_channel(dma_cap_mask_t *mask, dma_filter_fn fn, void *fn_param);
/**
* struct dma_client - info on the entity making use of DMA services
@@ -699,11 +695,6 @@ void dma_async_device_unregister_v3(struct dma_device_v3 *device);
* @lock: protects access to the channels list
* @channels: the list of DMA channels allocated
* @global_node: list_head for global dma_client_list
- * @event_callback_v3: func ptr to call when something happens
- * @cap_mask: only return channels that satisfy the requested capabilities
- * a value of zero corresponds to any capability
- * @slave: data for preparing slave transfer. Must be non-NULL iff the
- * DMA_SLAVE capability is requested.
*/
struct dma_client {
dma_event_callback event_callback;
@@ -713,12 +704,19 @@ struct dma_client {
spinlock_t lock;
struct list_head channels;
struct list_head global_node;
-#ifndef __GENKSYMS__
- dma_event_callback_v3 event_callback_v3;
- dma_cap_mask_t cap_mask;
- struct dma_slave *slave;
-#endif
};
#endif /* CONFIG_DMA_ENGINE */
+
+#ifdef CONFIG_NET_DMA
+#define net_dmaengine_get() dmaengine_get()
+#define net_dmaengine_put() dmaengine_put()
+#else
+static inline void net_dmaengine_get(void)
+{
+}
+static inline void net_dmaengine_put(void)
+{
+}
+#endif
#endif /* DMAENGINE_H */
diff --git a/include/net/netdma.h b/include/net/netdma.h
index f28c6e0..8ba8ce2 100644
--- a/include/net/netdma.h
+++ b/include/net/netdma.h
@@ -24,17 +24,6 @@
#include <linux/dmaengine.h>
#include <linux/skbuff.h>
-static inline struct dma_chan *get_softnet_dma(void)
-{
- struct dma_chan *chan;
- rcu_read_lock();
- chan = rcu_dereference(__get_cpu_var(softnet_data).net_dma);
- if (chan)
- dma_chan_get(chan);
- rcu_read_unlock();
- return chan;
-}
-
int dma_skb_copy_datagram_iovec(struct dma_chan* chan,
struct sk_buff *skb, int offset, struct iovec *to,
size_t len, struct dma_pinned_list *pinned_list);
diff --git a/net/core/dev.c b/net/core/dev.c
index 04bc9f8..9d4fd04 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -168,27 +168,6 @@ static struct list_head ptype_base[16]; /* 16 way hashed list */
static struct list_head ptype_all; /* Taps */
static struct list_head gro_ptype_base[16];
-#ifdef CONFIG_NET_DMA
-struct net_dma {
- struct dma_client client;
- spinlock_t lock;
- cpumask_t channel_mask;
- struct dma_chan *channels[NR_CPUS];
-};
-
-static enum dma_state_client
- netdev_dma_event(struct dma_client *client, struct dma_chan *chan,
- enum dma_state state);
-
-static struct net_dma net_dma = {
- .client = {
- .event_callback_v3 = netdev_dma_event,
- },
-};
-
-
-#endif
-
/*
* The @dev_base list is protected by @dev_base_lock and the rtnl
* semaphore.
@@ -952,6 +931,11 @@ int dev_open(struct net_device *dev)
dev->flags |= IFF_UP;
/*
+ * Enable NET_DMA
+ */
+ net_dmaengine_get();
+
+ /*
* Initialize multicasting status
*/
dev_mc_upload(dev);
@@ -1026,6 +1010,11 @@ int dev_close(struct net_device *dev)
*/
raw_notifier_call_chain(&netdev_chain, NETDEV_DOWN, dev);
+ /*
+ * Shutdown NET_DMA
+ */
+ net_dmaengine_put();
+
return 0;
}
@@ -2427,14 +2416,7 @@ out:
* There may not be any more sk_buffs coming right now, so push
* any pending DMA copies to hardware
*/
- if (!cpus_empty(net_dma.channel_mask)) {
- int chan_idx;
- for_each_cpu_mask(chan_idx, net_dma.channel_mask) {
- struct dma_chan *chan = net_dma.channels[chan_idx];
- if (chan)
- dma_async_memcpy_issue_pending_v3(chan);
- }
- }
+ dma_issue_pending_all();
#endif
local_irq_enable();
return;
@@ -4024,114 +4006,6 @@ static int dev_cpu_callback(struct notifier_block *nfb,
}
#endif /* CONFIG_HOTPLUG_CPU */
-#ifdef CONFIG_NET_DMA
-/**
- * net_dma_rebalance -
- * This is called when the number of channels allocated to the net_dma_client
- * changes. The net_dma_client tries to have one DMA channel per CPU.
- */
-static void net_dma_rebalance(struct net_dma *net_dma)
-{
- unsigned int cpu, i, n, chan_idx;
- struct dma_chan *chan;
-
- if (cpus_empty(net_dma->channel_mask)) {
- for_each_online_cpu(cpu)
- rcu_assign_pointer(per_cpu(softnet_data, cpu).net_dma, NULL);
- return;
- }
-
- i = 0;
- cpu = first_cpu(cpu_online_map);
-
- for_each_cpu_mask(chan_idx, net_dma->channel_mask) {
- chan = net_dma->channels[chan_idx];
-
- n = ((num_online_cpus() / cpus_weight(net_dma->channel_mask))
- + (i < (num_online_cpus() %
- cpus_weight(net_dma->channel_mask)) ? 1 : 0));
-
- while(n) {
- per_cpu(softnet_data, cpu).net_dma = chan;
- cpu = next_cpu(cpu, cpu_online_map);
- n--;
- }
- i++;
- }
-}
-
-/**
- * netdev_dma_event - event callback for the net_dma_client
- * @client: should always be net_dma_client
- * @chan: DMA channel for the event
- * @event: event type
- */
-static enum dma_state_client
- netdev_dma_event(struct dma_client *client, struct dma_chan *chan,
- enum dma_state state)
-{
- int i, found = 0, pos = -1;
- struct net_dma *net_dma =
- container_of(client, struct net_dma, client);
- enum dma_state_client ack = DMA_DUP; /* default: take no action */
-
- spin_lock(&net_dma->lock);
- switch (state) {
- case DMA_STATE_RESOURCE_AVAILABLE:
- for (i = 0; i < NR_CPUS; i++)
- if (net_dma->channels[i] == chan) {
- found = 1;
- break;
- } else if (net_dma->channels[i] == NULL && pos < 0)
- pos = i;
-
- if (!found && pos >= 0) {
- ack = DMA_ACK;
- net_dma->channels[pos] = chan;
- cpu_set(pos, net_dma->channel_mask);
- net_dma_rebalance(net_dma);
- }
-
- break;
- case DMA_STATE_RESOURCE_REMOVED:
- for (i = 0; i < NR_CPUS; i++)
- if (net_dma->channels[i] == chan) {
- found = 1;
- pos = i;
- break;
- }
-
- if (found) {
- ack = DMA_ACK;
- cpu_clear(pos, net_dma->channel_mask);
- net_dma->channels[i] = NULL;
- net_dma_rebalance(net_dma);
- }
- break;
- default:
- break;
- }
- spin_unlock(&net_dma->lock);
-
- return(ack);
-}
-
-/**
- * netdev_dma_regiser - register the networking subsystem as a DMA client
- */
-static int __init netdev_dma_register(void)
-{
- spin_lock_init(&net_dma.lock);
- dma_cap_set(DMA_MEMCPY, net_dma.client.cap_mask);
- dma_async_client_register_v3(&net_dma.client);
- dma_async_client_chan_request_v3(&net_dma.client);
- return 0;
-}
-
-#else
-static int __init netdev_dma_register(void) { return -ENODEV; }
-#endif /* CONFIG_NET_DMA */
-
/*
* Initialize the DEV module. At boot time this walks the device list and
* unhooks any devices that fail to initialise (normally hardware not
@@ -4225,8 +4099,6 @@ static int __init net_dev_init(void)
atomic_set(&queue->backlog_dev.refcnt, 1);
}
- netdev_dma_register();
-
dev_boot_phase = 0;
open_softirq(NET_TX_SOFTIRQ, net_tx_action, NULL);
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index 8cfa50c..18ab4bc 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -1144,7 +1144,7 @@ int tcp_recvmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg,
if ((available < target) &&
(len > sysctl_tcp_dma_copybreak) && !(flags & MSG_PEEK) &&
!sysctl_tcp_low_latency &&
- __get_cpu_var(softnet_data).net_dma) {
+ dma_find_channel(DMA_MEMCPY)) {
preempt_enable_no_resched();
tp->ucopy.pinned_list =
dma_pin_iovec_pages_v3(msg->msg_iov,
@@ -1356,7 +1356,7 @@ do_prequeue:
if (!(flags & MSG_TRUNC)) {
#ifdef CONFIG_NET_DMA
if (!tp->ucopy.dma_chan && tp->ucopy.pinned_list)
- tp->ucopy.dma_chan = get_softnet_dma();
+ tp->ucopy.dma_chan = dma_find_channel(DMA_MEMCPY);
if (tp->ucopy.dma_chan) {
tp->ucopy.dma_cookie = dma_skb_copy_datagram_iovec(
@@ -1461,7 +1461,6 @@ skip_copy:
/* Safe to free early-copied skbs now */
__skb_queue_purge(&sk->sk_async_wait_queue);
- dma_chan_put(tp->ucopy.dma_chan);
tp->ucopy.dma_chan = NULL;
}
if (tp->ucopy.pinned_list) {
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index 44c7718..4cc00e6 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -3824,7 +3824,7 @@ static int tcp_dma_try_early_copy(struct sock *sk, struct sk_buff *skb, int hlen
return 0;
if (!tp->ucopy.dma_chan && tp->ucopy.pinned_list)
- tp->ucopy.dma_chan = get_softnet_dma();
+ tp->ucopy.dma_chan = dma_find_channel(DMA_MEMCPY);
if (tp->ucopy.dma_chan && skb->ip_summed == CHECKSUM_UNNECESSARY) {
diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
index ec8e0a7..506c781 100644
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c
@@ -1117,7 +1117,7 @@ process:
#ifdef CONFIG_NET_DMA
struct tcp_sock *tp = tcp_sk(sk);
if (!tp->ucopy.dma_chan && tp->ucopy.pinned_list)
- tp->ucopy.dma_chan = get_softnet_dma();
+ tp->ucopy.dma_chan = dma_find_channel(DMA_MEMCPY);
if (tp->ucopy.dma_chan)
ret = tcp_v4_do_rcv(sk, skb);
else
diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c
index 6cfb9bd..e73e1ad 100644
--- a/net/ipv6/tcp_ipv6.c
+++ b/net/ipv6/tcp_ipv6.c
@@ -1312,6 +1312,9 @@ process:
if (!sock_owned_by_user(sk)) {
#ifdef CONFIG_NET_DMA
struct tcp_sock *tp = tcp_sk(sk);
+ if (!tp->ucopy.dma_chan && tp->ucopy.pinned_list)
+ tp->ucopy.dma_chan = dma_find_channel(DMA_MEMCPY);
+
if (tp->ucopy.dma_chan)
ret = tcp_v6_do_rcv(sk, skb);
else
distrib
>
Scientific%20Linux
>
5x
>
x86_64
>
by-pkgid
>
27922b4260f65d317aabda37e42bbbff
>
files
>
708
