From: Brad Peters <bpeters@redhat.com>
Date: Tue, 19 Aug 2008 17:44:39 -0400
Subject: [openib] lost interrupt after LPAR to LPAR communication
Message-id: 20080819214439.11639.51799.sendpatchset@squad5-lp1.lab.bos.redhat.com
O-Subject: [PATCH RHEL5.3 bz457838] Lost interrupts after LPAR to LPAR communication
Bugzilla: 457838
RH-Acked-by: Pete Zaitcev <zaitcev@redhat.com>
RH-Acked-by: David Howells <dhowells@redhat.com>

RHBZ#:
======
https://bugzilla.redhat.com/show_bug.cgi?id=457838

Description:
===========
Bug fix / Affects all systems running Infiniband with ehca (I believe
this is only IBM at the moment, but not certain)

Running huge traffic over InfiniBand/ehca between two LPARs will
cause missing IRQ events so that completions are handled by the
deadman timer with a lower frequency. The result is a significantly
degraded performance of involved applications.

This patch fixes the problem by periodically triggering EOI
(end-of-interrupt) through H_EOI, if event queues are pending.

RHEL Version Found:
================
RHEL 5.1

kABI Status:
============
No symbols were harmed.

Brew:
=====
Built on all platforms.
http://brewweb.devel.redhat.com/brew/taskinfo?taskID=1423704

Upstream Status:
================
Posted and applied:
http://lkml.org/lkml/2008/6/13/134

Test Status:
============
Tested by Stefan Roscher <IBM> 8/14/08
To recreate the problem I used a p575 with galaxy 1 adapters.

Two LPARS are sharing this adapter.

I ran flood ping from one lpar to the other via ipoib connected mode.
After a while you see a performance decrease. In this situation only the
driver's "deadman timer" is looking every second on the event queue. After
checking the hca hw register we saw that a interrupt is reported to the lpar
and so no new interrupts will be singnaled, but this interrupt never reached
the LPAR.

After applying the patch the ehca driver can sense this situation and switch
off the bit in the register so interrupts will be signaled again.
This is also noticeable with floodping, where you will see that the ping is
comming back to old performance after losing the interrupt.

===============================================================

Brad Peters 1-978-392-1000 x 23183
IBM on-site partner.

Proposed Patch:
===============
This patch is based on 2.6.18-101.el5

diff --git a/drivers/infiniband/hw/ehca/ehca_irq.c b/drivers/infiniband/hw/ehca/ehca_irq.c
index ab8f42c..8c821d4 100644
--- a/drivers/infiniband/hw/ehca/ehca_irq.c
+++ b/drivers/infiniband/hw/ehca/ehca_irq.c
@@ -531,7 +531,7 @@ void ehca_process_eq(struct ehca_shca *shca, int is_irq)
 {
 	struct ehca_eq *eq = &shca->eq;
 	struct ehca_eqe_cache_entry *eqe_cache = eq->eqe_cache;
-	u64 eqe_value;
+	u64 eqe_value, ret;
 	unsigned long flags;
 	int eqe_cnt, i;
 	int eq_empty = 0;
@@ -583,8 +583,13 @@ void ehca_process_eq(struct ehca_shca *shca, int is_irq)
 			ehca_dbg(&shca->ib_device,
 				 "No eqe found for irq event");
 		goto unlock_irq_spinlock;
-	} else if (!is_irq)
+	} else if (!is_irq) {
+		ret = hipz_h_eoi(eq->ist);
+		if (ret != H_SUCCESS)
+			ehca_err(&shca->ib_device,
+				 "bad return code EOI -rc = %ld\n", ret);
 		ehca_dbg(&shca->ib_device, "deadman found %x eqe", eqe_cnt);
+	}
 	if (unlikely(eqe_cnt == EHCA_EQE_CACHE_SIZE))
 		ehca_dbg(&shca->ib_device, "too many eqes for one irq event");
 	/* enable irq for new packets */
diff --git a/drivers/infiniband/hw/ehca/hcp_if.c b/drivers/infiniband/hw/ehca/hcp_if.c
index 7029aa6..7a5496a 100644
--- a/drivers/infiniband/hw/ehca/hcp_if.c
+++ b/drivers/infiniband/hw/ehca/hcp_if.c
@@ -932,3 +932,13 @@ u64 hipz_h_error_data(const struct ipz_adapter_handle adapter_handle,
 				       r_cb,
 				       0, 0, 0, 0);
 }
+
+u64 hipz_h_eoi(int irq)
+{
+	unsigned long xirr;
+
+	iosync();
+	xirr = (0xffULL << 24) | irq;
+
+	return plpar_hcall_norets(H_EOI, xirr);
+}
diff --git a/drivers/infiniband/hw/ehca/hcp_if.h b/drivers/infiniband/hw/ehca/hcp_if.h
index 60ce02b..2c3c6e0 100644
--- a/drivers/infiniband/hw/ehca/hcp_if.h
+++ b/drivers/infiniband/hw/ehca/hcp_if.h
@@ -260,5 +260,6 @@ u64 hipz_h_error_data(const struct ipz_adapter_handle adapter_handle,
 		      const u64 ressource_handle,
 		      void *rblock,
 		      unsigned long *byte_count);
+u64 hipz_h_eoi(int irq);
 
 #endif /* __HCP_IF_H__ */