From: Mikulas Patocka <mpatocka@redhat.com>
Date: Tue, 19 Aug 2008 14:13:45 -0400
Subject: [md] LVM raid-1 performance fixes
Message-id: Pine.LNX.4.64.0808191410001.12962@hs20-bc2-1.build.redhat.com
O-Subject: [PATCH 2/2 RHEL 5.3] bz438153 LVM raid-1 performance fixes
Bugzilla: 438153
RH-Acked-by: Alasdair G Kergon <agk@redhat.com>
RH-Acked-by: Jonathan Brassow <jbrassow@redhat.com>

This is the second patch for the same bug, to improve performance even
more.

There is missing queue unplug in dm-raid and it caused severe slowdown
(dm-raid1 was 10 times slower than dm-linear) at two customers.

The patch is upstream and is considered to be included in RHEL 4.7.z

Testing: compiled in on my workstation, run test program from the
bugzilla, saw twice performance improvement when doing synchronous writes
with the size 8192 bytes.

Upstream commit: 7ff14a36159d947872870e7a3e9dcaebc46b23eb (2.6.26-rc1)

Avoid 3ms delay in dm-raid and kcopyd.

It is specified that any submitted bio without BIO_RW_SYNC flag may plug the
queue (i.e. block the requests from being dispatched to the physical device).

The queue is unplugged when the caller calls blk_unplug() function. Usually, thesequence is that someone calls submit_bh to submit IO on a buffer. The IO plugs the queue and waits (to be possibly joined with other adjacent bios). Then, whenthe caller calls wait_on_buffer(), it unplugs the queue and submits the IOs to
the disk.

This was happenning:

When doing O_SYNC writes, function fsync_buffers_list() submits a list of
bios to dm_raid1, the bios are added to dm_raid1 write queue and kmirrord is
woken up.

fsync_buffers_list() calls wait_on_buffer() --- that unplugs the queue, but
there are no bios on the device queue (they are still in dm_raid1 queue)

wait_on_buffer() starts waiting until the IO is finished.

kmirrord is scheduled, kmirrord takes bios and submits them to the devices.

the submitted bio plugs the harddisk queue, there is no one to unplug it
(the process that called wait_on_buffer() is already sleeping)

So there is 3ms timeout, after which the queues on the harddisks are
unplugged, and requests are processed.

This 3ms timeout caused that in certain workloads (O_SYNC, 8kb writes), dm-raid1is 10 times slower than md-raid1.

Every time we submit something asynchronously via dm_io, we must unplug the
queue to actually put the request to the device.

This patch adds unplug call to kmirrord --- while processing requests, it keeps the queue plugged (so that adjacent bios can be merged), when it finishes
processing all bios, it unplugs the queue to submit the bios.

It also fixes kcopyd, that has the same potential problem. All kcopyd requests
are submitted with BIO_RW_SYNC.

Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>

diff --git a/drivers/md/dm-io.c b/drivers/md/dm-io.c
index 894c33b..2306ec9 100644
--- a/drivers/md/dm-io.c
+++ b/drivers/md/dm-io.c
@@ -422,7 +422,7 @@ static int sync_io(struct dm_io_client *client, unsigned int num_regions,
 {
 	struct io io;
 
-	if (num_regions > 1 && rw != WRITE) {
+	if (num_regions > 1 && (rw & RW_MASK) != WRITE) {
 		WARN_ON(1);
 		return -EIO;
 	}
@@ -459,7 +459,7 @@ static int async_io(struct dm_io_client *client, unsigned int num_regions,
 {
 	struct io *io;
 
-	if (num_regions > 1 && rw != WRITE) {
+	if (num_regions > 1 && (rw & RW_MASK) != WRITE) {
 		WARN_ON(1);
 		fn(1, context);
 		return -EIO;
@@ -556,6 +556,11 @@ static int dp_init(struct dm_io_request *io_req, struct dpages *dp)
 
 /*
  * New collapsed (a)synchronous interface
+ *
+ * If the IO is asynchronous (i.e. it has notify.fn), you must either unplug the
+ * queue with blk_unplug() some time later or you must set the BIO_RW_SYNC bit in
+ * io_req->bi_rw. If you fail to do one of these, the IO will be submitted to
+ * the disk after q->unplug_delay, which defaults to 3ms in blk-settings.c.
  */
 int dm_io(struct dm_io_request *io_req, unsigned num_regions,
 	  struct io_region *where, unsigned long *sync_error_bits)
diff --git a/drivers/md/dm-raid1.c b/drivers/md/dm-raid1.c
index 92625f7..759f4ef 100644
--- a/drivers/md/dm-raid1.c
+++ b/drivers/md/dm-raid1.c
@@ -1314,6 +1314,8 @@ static void do_mirror(void *data)
 	do_reads(ms, &reads);
 	do_writes(ms, &writes);
 	do_failures(ms, &failures);
+
+	dm_table_unplug_all(ms->ti->table);
 }
 
 /*-----------------------------------------------------------------
diff --git a/drivers/md/kcopyd.c b/drivers/md/kcopyd.c
index edcc9b1..cdcc34b 100644
--- a/drivers/md/kcopyd.c
+++ b/drivers/md/kcopyd.c
@@ -338,7 +338,7 @@ static int run_io_job(struct kcopyd_job *job)
 {
 	int r;
 	struct dm_io_request io_req = {
-		.bi_rw = job->rw,
+		.bi_rw = job->rw | (1 << BIO_RW_SYNC),
 		.mem.type = DM_IO_PAGE_LIST,
 		.mem.ptr.pl = job->pages,
 		.mem.offset = job->offset,