From: Brad Peters <bpeters@redhat.com>
Date: Tue, 12 Aug 2008 15:14:20 -0400
Subject: [ppc64] EEH: facilitate vendor driver recovery
Message-id: 20080812191420.15375.7843.sendpatchset@squad5-lp1.lab.bos.redhat.com
O-Subject: [PATCH RHEL5.3 457253] Improvements in EEH driver to facilitate vendor driver EEH recovery
Bugzilla: 457253
RH-Acked-by: David Howells <dhowells@redhat.com>
RHBZ#:
======
https://bugzilla.redhat.com/show_bug.cgi?id=457253
Description:
===========
Bug Fix / Affects PPC arch only
lpfc.h change is multi-arch, but as RHEL5.3 does not support PCIe AER
(Advanced Error Recovery), this code is not exposed, and the whole patch
only affects PPC kernel.
While working with Emulex and Qlogic on EEH support in their drivers we
uncovered the following issues:
1.) pdev->error_state was not properly being initialized, this prevented use of
any test of error_state != pci_channel_io_normal in drivers during driver
initializaition.
2.) Drivers contain code paths which loop on hardware register accesses during
both driver shutdown and driver initializaition. These loops were resulting in
EEH driver exceeding EEH_MAX_FAILS and then causing system panic.
3.) Drivers could rely upon error_state value to determine slot was frozen and
use this state to conditionally not execute code paths with looping hardware
register accesses to avoid exceeding EEH_MAX_FAILS in eeh_dn_check_failure,
however, this value was not reset to pci_channel_io_normal until
eeh_report_resume, which is too late for most drivers.
Issue 1) has now been resolved in both distros, by backporting upstream patches
which properly initialize error_state in drivers/pci/probe.c and
arch/powerpc/kernel/pci_64.c
Attempts to correct issue 2) were blocked by issue 3)
To work around issue 3 we implemnted the following hacks in Emulex and QLogic
drivers.
1) add upstream error_state wrapper to avoid execution of driver hardware
access polling loops:
+/* Not present in RHEL5. */
+static inline int pci_channel_offline(struct pci_dev *pdev);
+static inline int
+pci_channel_offline(struct pci_dev *pdev)
+{
+ return (pdev->error_state != pci_channel_io_normal);
+}
2) Reset error_state in drivers slot_reset callback.
+
+
+ /* Workaround on core EEH code does not set pdev->error_state */
+ pdev->error_state = pci_channel_io_normal;
+
As a first step, we should investigate the earliest point when the EEH driver
can reset error_state to pci_channel_io_normal to allow drivers to use the test
above to avoid drivers looping on hardware register accesses when the slot
might still be offline (frozen).
RHEL Version Found:
================
RHEL 5.2
kABI Status:
============
No symbols were harmed.
Brew:
=====
Built on all platforms.
http://brewweb.devel.redhat.com/brew/taskinfo?taskID=1426768
Upstream Status:
================
This patch consolidates multiple changes that bring RHEL EEH support to parity
> with the mainline kernel. Several other bug fixes are dependent on it. I'll
> add references to the mainline commits shortly.
The above mentioned patch backports and consolidates the following mainline
kernel commits:
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
powerpc/eeh: Don't panic when EEH_MAX_FAILS is exceeded
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=f36c5227cd88b6340c40d62b05859e8213740a97
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[POWERPC] Tweak EEH copyright info
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=f36c5227cd88b6340c40d62b05859e8213740a97
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[POWERPC] EEH: Tolerate high mmio
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=2fd30be8dae25386fc5167c34c6d73201334a8d4
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[POWERPC] EEH: wait for slot status
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=9c547768e7d9f456f1b145102e75f79e30f7b709
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[POWERPC] Add CONFIG_PPC_PSERIES_DEBUG to enable debugging for platforms/pseries
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=36f8a2c4c61e3559a95190e457b431c6900859b4
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[POWERPC] EEH: support ibm,get-config-addr-info2 RTAS call
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=147d6a37500348b6bda5738453d84c46678209cf
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[POWERPC] EEH: log all PCI-X and PCI-E AER registers
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=fcf9892b553ba638e6c8f9b6a140c99927c69693
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[POWERPC] EEH: capture and log pci state on error
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=d99bb1db798f49eacb27e087da7e62433361174d
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[POWERPC] Remove dead EEH code
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=42253a68a8e794a38ede33566083af8a80948f60
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[POWERPC] eeh.c: Use for_each_child_of_node
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=acaa617620b8b4071c6a3525b0160f4be07412d7
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[POWERPC] EEH: Work with device endpoint, always
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=307d46e83291ef58b2c3b6d33b7ec6dd163d58b9
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[POWERPC] Rename get_property to of_get_property: arch/powerpc
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=e2eb63927bfcb54232163bfec32440246fd44457
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[POWERPC] EEH: modify order of EEH state checking
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=39d16e295966a1f0025a65eaab4cb59fe5ba8c17
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[POWERPC] EEH: rm un-needed data
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=d0ab95ca9854174029cef2f08acf1859441cb547
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[POWERPC] EEH: handle reset state high
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=90375f53960f2b1e8d2a6af3324d440e3a977bf3
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
PCI: define inline for test of channel error state
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=a7369f1f6533b9efc3209d1df103537bbbf24b8c
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
WorkStruct: make allyesconfig
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=c4028958b6ecad064b1a6303a6a5906d4fe48d73#patch4
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[POWERPC] EEH: Fix PCI bridge handling bug
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=093eda3ce5dc3758c9a5e806ea6573bfffed3ff7
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[PATCH] getting rid of all casts of k[cmz]alloc() calls
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=5cbded585d129d0226cb48ac4202b253c781be26#patch22
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[POWERPC] EEH: Drivers that need reset trump others
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=2a50f144fc6081269468abf79e1907c6669bef22
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[POWERPC] EEH: Avoid a possible NULL pointer dereference
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=b76e5e93982f28666aeef1055264f5eac76311b0
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[POWERPC] EEH: Clean up comments
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=638799b33586339ac007b0fef497a15d81a271aa
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[POWERPC] EEH: verify state change
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=fa1be476a2baa0961f63161caee6733cdc353adb
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[POWERPC] EEH: multifunction recovery bugfix
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=5794dbcbab862e416c4ea4f10fda5e67f5565fd7
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[POWERPC] EEH: hotplug recovery bugfix
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-
2.6.git;a=commitdiff;h=90fdd6130f5c0053c48e8c8e247091739b6e4092
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[POWERPC] EEH: Split up long error msg
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=b455b24cf2512e9578a4ddd66ec91c3100c78127
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[POWERPC] EEH: log error only after driver notification.
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=ede8ca269f9f3e3fa4fb8561671f0699eefc32c5
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[POWERPC] Clarify EEH error message
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=a885902de3394ef18ca415f9175da5d8a8406cca
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[POWERPC] EEH recovery tweaks
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=d0e70341c05f6c31375530e0ae29b319153004a7
dd++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[POWERPC] EEH: restructure multi-function support
http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=4980d5eb750288ffc0bb9daff3feb947e1bac61e
Test Status:
============
have confirmed that the eeh-consolidated-patch-for-rhel2.6.18-98 has been
tested on IBM JS21 blade servers and IBM P5 Power servers with Emulex and
QLogic FC drivers.
Testing has also been confirmed by Emulex with lpfc driver candiate targeted
for RHEL5.3. Emulex patches will be dependent upon this patch.
With this patch applied along with Emulex lfpc RHEL5.3 driver candidate,
systems will sucessfully recover from EEH errors on Power platforms.
Testing with QLogic qla2xxx RHEL5.3 driver is on-going. With this patch applied
systems will sucessufully recover from EEH errors on Power platforms.
===============================================================
Brad Peters 1-978-392-1000 x 23183
IBM on-site partner.
Proposed Patch:
===============
This patch is based on 2.6.18-103.el5
diff --git a/arch/powerpc/platforms/pseries/eeh.c b/arch/powerpc/platforms/pseries/eeh.c
index e055867..34b87a6 100644
--- a/arch/powerpc/platforms/pseries/eeh.c
+++ b/arch/powerpc/platforms/pseries/eeh.c
@@ -1,6 +1,8 @@
/*
* eeh.c
- * Copyright (C) 2001 Dave Engebretsen & Todd Inglett IBM Corporation
+ * Copyright IBM Corporation 2001, 2005, 2006
+ * Copyright Dave Engebretsen & Todd Inglett 2001
+ * Copyright Linas Vepstas 2005, 2006
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -15,6 +17,8 @@
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+ *
+ * Please address comments and feedback to Linas Vepstas <linas@austin.ibm.com>
*/
#include <linux/delay.h>
@@ -25,6 +29,7 @@
#include <linux/rbtree.h>
#include <linux/seq_file.h>
#include <linux/spinlock.h>
+
#include <asm/atomic.h>
#include <asm/eeh.h>
#include <asm/eeh_event.h>
@@ -33,7 +38,6 @@
#include <asm/ppc-pci.h>
#include <asm/rtas.h>
-#undef DEBUG
/** Overview:
* EEH, or "Extended Error Handling" is a PCI bridge technology for
@@ -70,11 +74,14 @@
*/
/* If a device driver keeps reading an MMIO register in an interrupt
- * handler after a slot isolation event has occurred, we assume it
- * is broken and panic. This sets the threshold for how many read
- * attempts we allow before panicking.
+ * handler after a slot isolation event, it might be broken.
+ * This sets the threshold for how many read attempts we allow
+ * before printing an error message.
*/
-#define EEH_MAX_FAILS 100000
+#define EEH_MAX_FAILS 2100000
+
+/* Time to wait for a PCI slot to report status, in milliseconds */
+#define PCI_BUS_RESET_WAIT_MSEC (60*1000)
/* RTAS tokens */
static int ibm_set_eeh_option;
@@ -83,6 +90,7 @@ static int ibm_read_slot_reset_state;
static int ibm_read_slot_reset_state2;
static int ibm_slot_error_detail;
static int ibm_get_config_addr_info;
+static int ibm_get_config_addr_info2;
static int ibm_configure_bridge;
int eeh_subsystem_enabled;
@@ -91,11 +99,21 @@ EXPORT_SYMBOL(eeh_subsystem_enabled);
/* Lock to avoid races due to multiple reports of an error */
static DEFINE_SPINLOCK(confirm_error_lock);
-/* Buffer for reporting slot-error-detail rtas calls */
+/* Buffer for reporting slot-error-detail rtas calls. Its here
+ * in BSS, and not dynamically alloced, so that it ends up in
+ * RMO where RTAS can access it.
+ */
static unsigned char slot_errbuf[RTAS_ERROR_LOG_MAX];
static DEFINE_SPINLOCK(slot_errbuf_lock);
static int eeh_error_buf_size;
+/* Buffer for reporting pci register dumps. Its here in BSS, and
+ * not dynamically alloced, so that it ends up in RMO where RTAS
+ * can access it.
+ */
+#define EEH_PCI_REGS_LOG_LEN 4096
+static unsigned char pci_regs_buf[EEH_PCI_REGS_LOG_LEN];
+
/* System monitoring statistics */
static unsigned long no_device;
static unsigned long no_dn;
@@ -103,7 +121,6 @@ static unsigned long no_cfg_addr;
static unsigned long ignored_check;
static unsigned long total_mmio_ffs;
static unsigned long false_positives;
-static unsigned long ignored_failures;
static unsigned long slot_resets;
#define IS_BRIDGE(class_code) (((class_code)<<16) == PCI_BASE_CLASS_BRIDGE)
@@ -111,7 +128,8 @@ static unsigned long slot_resets;
/* --------------------------------------------------------------- */
/* Below lies the EEH event infrastructure */
-void eeh_slot_error_detail (struct pci_dn *pdn, int severity)
+static void rtas_slot_error_detail(struct pci_dn *pdn, int severity,
+ char *driver_log, size_t loglen)
{
int config_addr;
unsigned long flags;
@@ -129,7 +147,8 @@ void eeh_slot_error_detail (struct pci_dn *pdn, int severity)
rc = rtas_call(ibm_slot_error_detail,
8, 1, NULL, config_addr,
BUID_HI(pdn->phb->buid),
- BUID_LO(pdn->phb->buid), NULL, 0,
+ BUID_LO(pdn->phb->buid),
+ virt_to_phys(driver_log), loglen,
virt_to_phys(slot_errbuf),
eeh_error_buf_size,
severity);
@@ -140,6 +159,113 @@ void eeh_slot_error_detail (struct pci_dn *pdn, int severity)
}
/**
+ * gather_pci_data - copy assorted PCI config space registers to buff
+ * @pdn: device to report data for
+ * @buf: point to buffer in which to log
+ * @len: amount of room in buffer
+ *
+ * This routine captures assorted PCI configuration space data,
+ * and puts them into a buffer for RTAS error logging.
+ */
+static size_t gather_pci_data(struct pci_dn *pdn, char * buf, size_t len)
+{
+ struct pci_dev *dev = pdn->pcidev;
+ u32 cfg;
+ int cap, i;
+ int n = 0;
+
+ n += scnprintf(buf+n, len-n, "%s\n", pdn->node->full_name);
+ printk(KERN_WARNING "EEH: of node=%s\n", pdn->node->full_name);
+
+ rtas_read_config(pdn, PCI_VENDOR_ID, 4, &cfg);
+ n += scnprintf(buf+n, len-n, "dev/vend:%08x\n", cfg);
+ printk(KERN_WARNING "EEH: PCI device/vendor: %08x\n", cfg);
+
+ rtas_read_config(pdn, PCI_COMMAND, 4, &cfg);
+ n += scnprintf(buf+n, len-n, "cmd/stat:%x\n", cfg);
+ printk(KERN_WARNING "EEH: PCI cmd/status register: %08x\n", cfg);
+
+ if (!dev) {
+ printk(KERN_WARNING "EEH: no PCI device for this of node\n");
+ return n;
+ }
+
+ /* Gather bridge-specific registers */
+ if (dev->class >> 16 == PCI_BASE_CLASS_BRIDGE) {
+ rtas_read_config(pdn, PCI_SEC_STATUS, 2, &cfg);
+ n += scnprintf(buf+n, len-n, "sec stat:%x\n", cfg);
+ printk(KERN_WARNING "EEH: Bridge secondary status: %04x\n", cfg);
+
+ rtas_read_config(pdn, PCI_BRIDGE_CONTROL, 2, &cfg);
+ n += scnprintf(buf+n, len-n, "brdg ctl:%x\n", cfg);
+ printk(KERN_WARNING "EEH: Bridge control: %04x\n", cfg);
+ }
+
+ /* Dump out the PCI-X command and status regs */
+ cap = pci_find_capability(dev, PCI_CAP_ID_PCIX);
+ if (cap) {
+ rtas_read_config(pdn, cap, 4, &cfg);
+ n += scnprintf(buf+n, len-n, "pcix-cmd:%x\n", cfg);
+ printk(KERN_WARNING "EEH: PCI-X cmd: %08x\n", cfg);
+
+ rtas_read_config(pdn, cap+4, 4, &cfg);
+ n += scnprintf(buf+n, len-n, "pcix-stat:%x\n", cfg);
+ printk(KERN_WARNING "EEH: PCI-X status: %08x\n", cfg);
+ }
+
+ /* If PCI-E capable, dump PCI-E cap 10, and the AER */
+ cap = pci_find_capability(dev, PCI_CAP_ID_EXP);
+ if (cap) {
+ n += scnprintf(buf+n, len-n, "pci-e cap10:\n");
+ printk(KERN_WARNING
+ "EEH: PCI-E capabilities and status follow:\n");
+
+ for (i=0; i<=8; i++) {
+ rtas_read_config(pdn, cap+4*i, 4, &cfg);
+ n += scnprintf(buf+n, len-n, "%02x:%x\n", 4*i, cfg);
+ printk(KERN_WARNING "EEH: PCI-E %02x: %08x\n", i, cfg);
+ }
+
+ cap = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ERR);
+ if (cap) {
+ n += scnprintf(buf+n, len-n, "pci-e AER:\n");
+ printk(KERN_WARNING
+ "EEH: PCI-E AER capability register set follows:\n");
+
+ for (i=0; i<14; i++) {
+ rtas_read_config(pdn, cap+4*i, 4, &cfg);
+ n += scnprintf(buf+n, len-n, "%02x:%x\n", 4*i, cfg);
+ printk(KERN_WARNING "EEH: PCI-E AER %02x: %08x\n", i, cfg);
+ }
+ }
+ }
+
+ /* Gather status on devices under the bridge */
+ if (dev->class >> 16 == PCI_BASE_CLASS_BRIDGE) {
+ struct device_node *dn;
+
+ for_each_child_of_node(pdn->node, dn) {
+ pdn = PCI_DN(dn);
+ if (pdn)
+ n += gather_pci_data(pdn, buf+n, len-n);
+ }
+ }
+
+ return n;
+}
+
+void eeh_slot_error_detail(struct pci_dn *pdn, int severity)
+{
+ size_t loglen = 0;
+ pci_regs_buf[0] = 0;
+
+ rtas_pci_enable(pdn, EEH_THAW_MMIO);
+ loglen = gather_pci_data(pdn, pci_regs_buf, EEH_PCI_REGS_LOG_LEN);
+
+ rtas_slot_error_detail(pdn, severity, pci_regs_buf, loglen);
+}
+
+/**
* read_slot_reset_state - Read the reset state of a device node's slot
* @dn: device node to read
* @rets: array to return results in
@@ -168,6 +294,55 @@ static int read_slot_reset_state(struct pci_dn *pdn, int rets[])
}
/**
+ * eeh_wait_for_slot_status - returns error status of slot
+ * @pdn pci device node
+ * @max_wait_msecs maximum number to millisecs to wait
+ *
+ * Return negative value if a permanent error, else return
+ * Partition Endpoint (PE) status value.
+ *
+ * If @max_wait_msecs is positive, then this routine will
+ * sleep until a valid status can be obtained, or until
+ * the max allowed wait time is exceeded, in which case
+ * a -2 is returned.
+ */
+int
+eeh_wait_for_slot_status(struct pci_dn *pdn, int max_wait_msecs)
+{
+ int rc;
+ int rets[3];
+ int mwait;
+
+ while (1) {
+ rc = read_slot_reset_state(pdn, rets);
+ if (rc) return rc;
+ if (rets[1] == 0) return -1; /* EEH is not supported */
+
+ if (rets[0] != 5) return rets[0]; /* return actual status */
+
+ if (rets[2] == 0) return -1; /* permanently unavailable */
+
+ if (max_wait_msecs <= 0) break;
+
+ mwait = rets[2];
+ if (mwait <= 0) {
+ printk (KERN_WARNING
+ "EEH: Firmware returned bad wait value=%d\n", mwait);
+ mwait = 1000;
+ } else if (mwait > 300*1000) {
+ printk (KERN_WARNING
+ "EEH: Firmware is taking too long, time=%d\n", mwait);
+ mwait = 300*1000;
+ }
+ max_wait_msecs -= mwait;
+ msleep (mwait);
+ }
+
+ printk(KERN_WARNING "EEH: Timed out waiting for slot status\n");
+ return -2;
+}
+
+/**
* eeh_token_to_phys - convert EEH address token to phys address
* @token i/o token, should be address in the form 0xA....
*/
@@ -196,7 +371,7 @@ struct device_node * find_device_pe(struct device_node *dn)
return dn;
}
-/** Mark all devices that are peers of this device as failed.
+/** Mark all devices that are children of this device as failed.
* Mark the device driver too, so that it can see the failure
* immediately; this is critical, since some drivers poll
* status registers in interrupts ... If a driver is polling,
@@ -204,9 +379,11 @@ struct device_node * find_device_pe(struct device_node *dn)
* an interrupt context, which is bad.
*/
-static void __eeh_mark_slot (struct device_node *dn, int mode_flag)
+static void __eeh_mark_slot(struct device_node *parent, int mode_flag)
{
- while (dn) {
+ struct device_node *dn;
+
+ for_each_child_of_node(parent, dn) {
if (PCI_DN(dn)) {
/* Mark the pci device driver too */
struct pci_dev *dev = PCI_DN(dn)->pcidev;
@@ -216,10 +393,8 @@ static void __eeh_mark_slot (struct device_node *dn, int mode_flag)
if (dev && dev->driver)
dev->error_state = pci_channel_io_frozen;
- if (dn->child)
- __eeh_mark_slot (dn->child, mode_flag);
+ __eeh_mark_slot(dn, mode_flag);
}
- dn = dn->sibling;
}
}
@@ -229,7 +404,7 @@ void eeh_mark_slot (struct device_node *dn, int mode_flag)
dn = find_device_pe (dn);
/* Back up one, since config addrs might be shared */
- if (PCI_DN(dn) && PCI_DN(dn)->eeh_pe_config_addr)
+ if (!pcibios_find_pci_bus(dn) && PCI_DN(dn->parent))
dn = dn->parent;
PCI_DN(dn)->eeh_mode |= mode_flag;
@@ -239,19 +414,19 @@ void eeh_mark_slot (struct device_node *dn, int mode_flag)
if (dev)
dev->error_state = pci_channel_io_frozen;
- __eeh_mark_slot (dn->child, mode_flag);
+ __eeh_mark_slot(dn, mode_flag);
}
-static void __eeh_clear_slot (struct device_node *dn, int mode_flag)
+static void __eeh_clear_slot(struct device_node *parent, int mode_flag)
{
- while (dn) {
+ struct device_node *dn;
+
+ for_each_child_of_node(parent, dn) {
if (PCI_DN(dn)) {
PCI_DN(dn)->eeh_mode &= ~mode_flag;
PCI_DN(dn)->eeh_check_count = 0;
- if (dn->child)
- __eeh_clear_slot (dn->child, mode_flag);
+ __eeh_clear_slot(dn, mode_flag);
}
- dn = dn->sibling;
}
}
@@ -263,12 +438,12 @@ void eeh_clear_slot (struct device_node *dn, int mode_flag)
dn = find_device_pe (dn);
/* Back up one, since config addrs might be shared */
- if (PCI_DN(dn) && PCI_DN(dn)->eeh_pe_config_addr)
+ if (!pcibios_find_pci_bus(dn) && PCI_DN(dn->parent))
dn = dn->parent;
PCI_DN(dn)->eeh_mode &= ~mode_flag;
PCI_DN(dn)->eeh_check_count = 0;
- __eeh_clear_slot (dn->child, mode_flag);
+ __eeh_clear_slot(dn, mode_flag);
spin_unlock_irqrestore(&confirm_error_lock, flags);
}
@@ -293,8 +468,8 @@ int eeh_dn_check_failure(struct device_node *dn, struct pci_dev *dev)
int rets[3];
unsigned long flags;
struct pci_dn *pdn;
- enum pci_channel_state state;
int rc = 0;
+ const char *location;
total_mmio_ffs++;
@@ -305,6 +480,7 @@ int eeh_dn_check_failure(struct device_node *dn, struct pci_dev *dev)
no_dn++;
return 0;
}
+ dn = find_device_pe(dn);
pdn = PCI_DN(dn);
/* Access to IO BARs might get this far and still not want checking. */
@@ -333,17 +509,15 @@ int eeh_dn_check_failure(struct device_node *dn, struct pci_dev *dev)
rc = 1;
if (pdn->eeh_mode & EEH_MODE_ISOLATED) {
pdn->eeh_check_count ++;
- if (pdn->eeh_check_count >= EEH_MAX_FAILS) {
- printk (KERN_ERR "EEH: Device driver ignored %d bad reads, panicing\n",
- pdn->eeh_check_count);
+ if (pdn->eeh_check_count % EEH_MAX_FAILS == 0) {
+ location = (char *) of_get_property(dn, "ibm,loc-code", NULL);
+ printk (KERN_ERR "EEH: %d reads ignored for recovering device at "
+ "location=%s driver=%s pci addr=%s\n",
+ pdn->eeh_check_count, location,
+ dev->driver->name, pci_name(dev));
+ printk (KERN_ERR "EEH: Might be infinite loop in %s driver\n",
+ dev->driver->name);
dump_stack();
-
- /* re-read the slot reset state */
- if (read_slot_reset_state(pdn, rets) != 0)
- rets[0] = -1; /* reset state unknown */
-
- /* If we are here, then we hit an infinite loop. Stop. */
- panic("EEH: MMIO halt (%d) on device:%s\n", rets[0], pci_name(dev));
}
goto dn_unlock;
}
@@ -366,25 +540,25 @@ int eeh_dn_check_failure(struct device_node *dn, struct pci_dev *dev)
goto dn_unlock;
}
- /* If EEH is not supported on this device, punt. */
- if (rets[1] != 1) {
- printk(KERN_WARNING "EEH: event on unsupported device, rc=%d dn=%s\n",
- ret, dn->full_name);
+ /* Note that config-io to empty slots may fail;
+ * they are empty when they don't have children. */
+ if ((rets[0] == 5) && (rets[2] == 0) && (dn->child == NULL)) {
false_positives++;
rc = 0;
goto dn_unlock;
}
- /* If not the kind of error we know about, punt. */
- if (rets[0] != 2 && rets[0] != 4 && rets[0] != 5) {
+ /* If EEH is not supported on this device, punt. */
+ if (rets[1] != 1) {
+ printk(KERN_WARNING "EEH: event on unsupported device, rc=%d dn=%s\n",
+ ret, dn->full_name);
false_positives++;
rc = 0;
goto dn_unlock;
}
- /* Note that config-io to empty slots may fail;
- * we recognize empty because they don't have children. */
- if ((rets[0] == 5) && (dn->child == NULL)) {
+ /* If not the kind of error we know about, punt. */
+ if (rets[0] != 1 && rets[0] != 2 && rets[0] != 4 && rets[0] != 5) {
false_positives++;
rc = 0;
goto dn_unlock;
@@ -398,17 +572,12 @@ int eeh_dn_check_failure(struct device_node *dn, struct pci_dev *dev)
eeh_mark_slot (dn, EEH_MODE_ISOLATED);
spin_unlock_irqrestore(&confirm_error_lock, flags);
- state = pci_channel_io_normal;
- if ((rets[0] == 2) || (rets[0] == 4))
- state = pci_channel_io_frozen;
- if (rets[0] == 5)
- state = pci_channel_io_perm_failure;
- eeh_send_failure_event (dn, dev, state, rets[2]);
+ eeh_send_failure_event (dn, dev);
/* Most EEH events are due to device driver bugs. Having
* a stack trace will help the device-driver authors figure
* out what happened. So print that out. */
- if (rets[0] != 5) dump_stack();
+ dump_stack();
return 1;
dn_unlock:
@@ -457,38 +626,6 @@ EXPORT_SYMBOL(eeh_check_failure);
/* The code below deals with error recovery */
/**
- * eeh_slot_availability - returns error status of slot
- * @pdn pci device node
- *
- * Return negative value if a permanent error, else return
- * a number of milliseconds to wait until the PCI slot is
- * ready to be used.
- */
-static int
-eeh_slot_availability(struct pci_dn *pdn)
-{
- int rc;
- int rets[3];
-
- rc = read_slot_reset_state(pdn, rets);
-
- if (rc) return rc;
-
- if (rets[1] == 0) return -1; /* EEH is not supported */
- if (rets[0] == 0) return 0; /* Oll Korrect */
- if (rets[0] == 5) {
- if (rets[2] == 0) return -1; /* permanently unavailable */
- return rets[2]; /* number of millisecs to wait */
- }
- if (rets[0] == 1)
- return 250;
-
- printk (KERN_ERR "EEH: Slot unavailable: rc=%d, rets=%d %d %d\n",
- rc, rets[0], rets[1], rets[2]);
- return -2;
-}
-
-/**
* rtas_pci_enable - enable MMIO or DMA transfers for this slot
* @pdn pci device node
*/
@@ -511,9 +648,13 @@ rtas_pci_enable(struct pci_dn *pdn, int function)
function);
if (rc)
- printk(KERN_WARNING "EEH: Cannot enable function %d, err=%d dn=%s\n",
+ printk(KERN_WARNING "EEH: Unexpected state change %d, err=%d dn=%s\n",
function, rc, pdn->node->full_name);
+ rc = eeh_wait_for_slot_status (pdn, PCI_BUS_RESET_WAIT_MSEC);
+ if ((rc == 4) && (function == EEH_THAW_MMIO))
+ return 0;
+
return rc;
}
@@ -624,36 +765,24 @@ int rtas_set_slot_reset(struct pci_dn *pdn)
{
int i, rc;
- __rtas_set_slot_reset(pdn);
+ /* Take three shots at resetting the bus */
+ for (i=0; i<3; i++) {
+ __rtas_set_slot_reset(pdn);
- /* Now double check with the firmware to make sure the device is
- * ready to be used; if not, wait for recovery. */
- for (i=0; i<10; i++) {
- rc = eeh_slot_availability (pdn);
+ rc = eeh_wait_for_slot_status(pdn, PCI_BUS_RESET_WAIT_MSEC);
if (rc == 0)
return 0;
- if (rc == -2) {
- printk (KERN_ERR "EEH: failed (%d) to reset slot %s\n",
- i, pdn->node->full_name);
- __rtas_set_slot_reset(pdn);
- continue;
- }
-
if (rc < 0) {
- printk (KERN_ERR "EEH: unrecoverable slot failure %s\n",
- pdn->node->full_name);
+ printk(KERN_ERR "EEH: unrecoverable slot failure %s\n",
+ pdn->node->full_name);
return -1;
}
-
- msleep (rc+100);
+ printk(KERN_ERR "EEH: bus reset %d failed on slot %s, rc=%d\n",
+ i+1, pdn->node->full_name, rc);
}
- rc = eeh_slot_availability (pdn);
- if (rc)
- printk (KERN_ERR "EEH: timeout resetting slot %s\n", pdn->node->full_name);
-
- return rc;
+ return -1;
}
/* ------------------------------------------------------- */
@@ -727,11 +856,8 @@ void eeh_restore_bars(struct pci_dn *pdn)
if ((pdn->eeh_mode & EEH_MODE_SUPPORTED) && !IS_BRIDGE(pdn->class_code))
__restore_bars (pdn);
- dn = pdn->node->child;
- while (dn) {
+ for_each_child_of_node(pdn->node, dn)
eeh_restore_bars (PCI_DN(dn));
- dn = dn->sibling;
- }
}
/**
@@ -787,17 +913,49 @@ struct eeh_early_enable_info {
unsigned int buid_lo;
};
+static int get_pe_addr (int config_addr,
+ struct eeh_early_enable_info *info)
+{
+ unsigned int rets[3];
+ int ret;
+
+ /* Use latest config-addr token on power6 */
+ if (ibm_get_config_addr_info2 != RTAS_UNKNOWN_SERVICE) {
+ /* Make sure we have a PE in hand */
+ ret = rtas_call (ibm_get_config_addr_info2, 4, 2, rets,
+ config_addr, info->buid_hi, info->buid_lo, 1);
+ if (ret || (rets[0]==0))
+ return 0;
+
+ ret = rtas_call (ibm_get_config_addr_info2, 4, 2, rets,
+ config_addr, info->buid_hi, info->buid_lo, 0);
+ if (ret)
+ return 0;
+ return rets[0];
+ }
+
+ /* Use older config-addr token on power5 */
+ if (ibm_get_config_addr_info != RTAS_UNKNOWN_SERVICE) {
+ ret = rtas_call (ibm_get_config_addr_info, 4, 2, rets,
+ config_addr, info->buid_hi, info->buid_lo, 0);
+ if (ret)
+ return 0;
+ return rets[0];
+ }
+ return 0;
+}
+
/* Enable eeh for the given device node. */
static void *early_enable_eeh(struct device_node *dn, void *data)
{
unsigned int rets[3];
struct eeh_early_enable_info *info = data;
int ret;
- char *status = get_property(dn, "status", NULL);
- u32 *class_code = (u32 *)get_property(dn, "class-code", NULL);
- u32 *vendor_id = (u32 *)get_property(dn, "vendor-id", NULL);
- u32 *device_id = (u32 *)get_property(dn, "device-id", NULL);
- u32 *regs;
+ const char *status = of_get_property(dn, "status", NULL);
+ const u32 *class_code = of_get_property(dn, "class-code", NULL);
+ const u32 *vendor_id = of_get_property(dn, "vendor-id", NULL);
+ const u32 *device_id = of_get_property(dn, "device-id", NULL);
+ const u32 *regs;
int enable;
struct pci_dn *pdn = PCI_DN(dn);
@@ -807,7 +965,7 @@ static void *early_enable_eeh(struct device_node *dn, void *data)
pdn->eeh_freeze_count = 0;
if (status && strncmp(status, "ok",2) != 0)
- return NULL; /* ignore devices with bad status */
+ return NULL; /* ignore devices with bad status */
/* Ignore bad nodes. */
if (!class_code || !vendor_id || !device_id)
@@ -822,7 +980,7 @@ static void *early_enable_eeh(struct device_node *dn, void *data)
/* Ok... see if this device supports EEH. Some do, some don't,
* and the only way to find out is to check each and every one. */
- regs = (u32 *)get_property(dn, "reg", NULL);
+ regs = of_get_property(dn, "reg", NULL);
if (regs) {
/* First register entry is addr (00BBSS00) */
/* Try to enable eeh */
@@ -836,15 +994,7 @@ static void *early_enable_eeh(struct device_node *dn, void *data)
/* If the newer, better, ibm,get-config-addr-info is supported,
* then use that instead. */
- pdn->eeh_pe_config_addr = 0;
- if (ibm_get_config_addr_info != RTAS_UNKNOWN_SERVICE) {
- ret = rtas_call (ibm_get_config_addr_info, 4, 2, rets,
- pdn->eeh_config_addr,
- info->buid_hi, info->buid_lo,
- 0);
- if (ret == 0)
- pdn->eeh_pe_config_addr = rets[0];
- }
+ pdn->eeh_pe_config_addr = get_pe_addr(pdn->eeh_config_addr, info);
/* Some older systems (Power4) allow the
* ibm,set-eeh-option call to succeed even on nodes
@@ -915,6 +1065,7 @@ void __init eeh_init(void)
ibm_read_slot_reset_state = rtas_token("ibm,read-slot-reset-state");
ibm_slot_error_detail = rtas_token("ibm,slot-error-detail");
ibm_get_config_addr_info = rtas_token("ibm,get-config-addr-info");
+ ibm_get_config_addr_info2 = rtas_token("ibm,get-config-addr-info2");
ibm_configure_bridge = rtas_token ("ibm,configure-bridge");
if (ibm_set_eeh_option == RTAS_UNKNOWN_SERVICE)
@@ -983,7 +1134,8 @@ static void eeh_add_device_early(struct device_node *dn)
void eeh_add_device_tree_early(struct device_node *dn)
{
struct device_node *sib;
- for (sib = dn->child; sib; sib = sib->sibling)
+
+ for_each_child_of_node(dn, sib)
eeh_add_device_tree_early(sib);
eeh_add_device_early(dn);
}
@@ -1013,7 +1165,7 @@ static void eeh_add_device_late(struct pci_dev *dev)
pdn = PCI_DN(dn);
pdn->pcidev = dev;
- pci_addr_cache_insert_device (dev);
+ pci_addr_cache_insert_device(dev);
}
void eeh_add_device_tree_late(struct pci_bus *bus)
@@ -1087,12 +1239,9 @@ static int proc_eeh_show(struct seq_file *m, void *v)
"no config address=%ld\n"
"check not wanted=%ld\n"
"eeh_total_mmio_ffs=%ld\n"
- "eeh_false_positives=%ld\n"
- "eeh_ignored_failures=%ld\n"
"eeh_slot_resets=%ld\n",
no_device, no_dn, no_cfg_addr,
ignored_check, total_mmio_ffs,
- false_positives, ignored_failures,
slot_resets);
}
@@ -1104,7 +1253,7 @@ static int proc_eeh_open(struct inode *inode, struct file *file)
return single_open(file, proc_eeh_show, NULL);
}
-static struct file_operations proc_eeh_operations = {
+static const struct file_operations proc_eeh_operations = {
.open = proc_eeh_open,
.read = seq_read,
.llseek = seq_lseek,
@@ -1118,9 +1267,8 @@ static int __init eeh_init_proc(void)
if (machine_is(pseries)) {
e = create_proc_entry("ppc64/eeh", 0, NULL);
if (e)
- e->proc_fops = &proc_eeh_operations;
+ e->proc_fops = &proc_eeh_operations;
}
-
return 0;
}
__initcall(eeh_init_proc);
diff --git a/arch/powerpc/platforms/pseries/eeh_cache.c b/arch/powerpc/platforms/pseries/eeh_cache.c
index b6b462d..8dead87 100644
--- a/arch/powerpc/platforms/pseries/eeh_cache.c
+++ b/arch/powerpc/platforms/pseries/eeh_cache.c
@@ -2,7 +2,8 @@
* eeh_cache.c
* PCI address cache; allows the lookup of PCI devices based on I/O address
*
- * Copyright (C) 2004 Linas Vepstas <linas@austin.ibm.com> IBM Corporation
+ * Copyright IBM Corporation 2004
+ * Copyright Linas Vepstas <linas@austin.ibm.com> 2004
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -27,7 +28,6 @@
#include <asm/pci-bridge.h>
#include <asm/ppc-pci.h>
-#undef DEBUG
/**
* The pci address cache subsystem. This subsystem places
@@ -153,7 +153,7 @@ pci_addr_cache_insert(struct pci_dev *dev, unsigned long alo,
return piar;
}
}
- piar = (struct pci_io_addr_range *)kmalloc(sizeof(struct pci_io_addr_range), GFP_ATOMIC);
+ piar = kmalloc(sizeof(struct pci_io_addr_range), GFP_ATOMIC);
if (!piar)
return NULL;
@@ -224,6 +224,10 @@ void pci_addr_cache_insert_device(struct pci_dev *dev)
{
unsigned long flags;
+ /* Ignore PCI bridges */
+ if ((dev->class >> 16) == PCI_BASE_CLASS_BRIDGE)
+ return;
+
spin_lock_irqsave(&pci_io_addr_cache_root.piar_lock, flags);
__pci_addr_cache_insert_device(dev);
spin_unlock_irqrestore(&pci_io_addr_cache_root.piar_lock, flags);
@@ -284,16 +288,13 @@ void __init pci_addr_cache_build(void)
spin_lock_init(&pci_io_addr_cache_root.piar_lock);
while ((dev = pci_get_device(PCI_ANY_ID, PCI_ANY_ID, dev)) != NULL) {
- /* Ignore PCI bridges */
- if ((dev->class >> 16) == PCI_BASE_CLASS_BRIDGE)
- continue;
pci_addr_cache_insert_device(dev);
dn = pci_device_to_OF_node(dev);
if (!dn)
continue;
- pci_dev_get (dev); /* matching put is in eeh_remove_device() */
+ pci_dev_get(dev); /* matching put is in eeh_remove_device() */
PCI_DN(dn)->pcidev = dev;
}
diff --git a/arch/powerpc/platforms/pseries/eeh_driver.c b/arch/powerpc/platforms/pseries/eeh_driver.c
index b524b78..36cc395 100644
--- a/arch/powerpc/platforms/pseries/eeh_driver.c
+++ b/arch/powerpc/platforms/pseries/eeh_driver.c
@@ -1,6 +1,7 @@
/*
* PCI Error Recovery Driver for RPA-compliant PPC64 platform.
- * Copyright (C) 2004, 2005 Linas Vepstas <linas@linas.org>
+ * Copyright IBM Corp. 2004 2005
+ * Copyright Linas Vepstas <linas@linas.org> 2004, 2005
*
* All rights reserved.
*
@@ -19,8 +20,7 @@
* along with this program; if not, write to the Free Software
* Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
*
- * Send feedback to <linas@us.ibm.com>
- *
+ * Send comments and feedback to Linas Vepstas <linas@austin.ibm.com>
*/
#include <linux/delay.h>
#include <linux/interrupt.h>
@@ -105,17 +105,18 @@ static void eeh_report_error(struct pci_dev *dev, void *userdata)
return;
rc = driver->err_handler->error_detected (dev, pci_channel_io_frozen);
+
+ /* A driver that needs a reset trumps all others */
+ if (rc == PCI_ERS_RESULT_NEED_RESET) *res = rc;
if (*res == PCI_ERS_RESULT_NONE) *res = rc;
- if (*res == PCI_ERS_RESULT_DISCONNECT &&
- rc == PCI_ERS_RESULT_NEED_RESET) *res = rc;
}
/**
* eeh_report_mmio_enabled - tell drivers that MMIO has been enabled
*
- * Report an EEH error to each device driver, collect up and
- * merge the device driver responses. Cumulative response
- * passed back in "userdata".
+ * Tells each device driver that IO ports, MMIO and config space I/O
+ * are now enabled. Collects up and merges the device driver responses.
+ * Cumulative response passed back in "userdata".
*/
static void eeh_report_mmio_enabled(struct pci_dev *dev, void *userdata)
@@ -123,17 +124,16 @@ static void eeh_report_mmio_enabled(struct pci_dev *dev, void *userdata)
enum pci_ers_result rc, *res = userdata;
struct pci_driver *driver = dev->driver;
- // dev->error_state = pci_channel_mmio_enabled;
-
if (!driver ||
!driver->err_handler ||
!driver->err_handler->mmio_enabled)
return;
rc = driver->err_handler->mmio_enabled (dev);
+
+ /* A driver that needs a reset trumps all others */
+ if (rc == PCI_ERS_RESULT_NEED_RESET) *res = rc;
if (*res == PCI_ERS_RESULT_NONE) *res = rc;
- if (*res == PCI_ERS_RESULT_DISCONNECT &&
- rc == PCI_ERS_RESULT_NEED_RESET) *res = rc;
}
/**
@@ -158,7 +158,8 @@ static void eeh_report_reset(struct pci_dev *dev, void *userdata)
return;
rc = driver->err_handler->slot_reset(dev);
- if (*res == PCI_ERS_RESULT_NONE) *res = rc;
+ if ((*res == PCI_ERS_RESULT_NONE) ||
+ (*res == PCI_ERS_RESULT_RECOVERED)) *res = rc;
if (*res == PCI_ERS_RESULT_DISCONNECT &&
rc == PCI_ERS_RESULT_NEED_RESET) *res = rc;
}
@@ -170,14 +171,19 @@ static void eeh_report_reset(struct pci_dev *dev, void *userdata)
static void eeh_report_resume(struct pci_dev *dev, void *userdata)
{
struct pci_driver *driver = dev->driver;
+ struct device_node *dn = pci_device_to_OF_node(dev);
dev->error_state = pci_channel_io_normal;
if (!driver)
return;
- if (!driver->err_handler)
- return;
- if (!driver->err_handler->resume)
+
+ if ((PCI_DN(dn)->eeh_mode) & EEH_MODE_IRQ_DISABLED) {
+ PCI_DN(dn)->eeh_mode &= ~EEH_MODE_IRQ_DISABLED;
+ enable_irq(dev->irq);
+ }
+ if (!driver->err_handler ||
+ !driver->err_handler->resume)
return;
driver->err_handler->resume(dev);
@@ -243,6 +249,7 @@ static void eeh_report_failure(struct pci_dev *dev, void *userdata)
static int eeh_reset_device (struct pci_dn *pe_dn, struct pci_bus *bus)
{
+ struct device_node *dn;
int cnt, rc;
/* pcibios will clear the counter; save the value */
@@ -258,23 +265,20 @@ static int eeh_reset_device (struct pci_dn *pe_dn, struct pci_bus *bus)
if (rc)
return rc;
- /* New-style config addrs might be shared across multiple devices,
- * Walk over all functions on this device */
- if (pe_dn->eeh_pe_config_addr) {
- struct device_node *pe = pe_dn->node;
- pe = pe->parent->child;
- while (pe) {
- struct pci_dn *ppe = PCI_DN(pe);
- if (pe_dn->eeh_pe_config_addr == ppe->eeh_pe_config_addr) {
- rtas_configure_bridge(ppe);
- eeh_restore_bars(ppe);
- }
- pe = pe->sibling;
+ /* Walk over all functions on this device. */
+ dn = pe_dn->node;
+ if (!pcibios_find_pci_bus(dn) && PCI_DN(dn->parent))
+ dn = dn->parent->child;
+
+ while (dn) {
+ struct pci_dn *ppe = PCI_DN(dn);
+ /* On Power4, always true because eeh_pe_config_addr=0 */
+ if (pe_dn->eeh_pe_config_addr == ppe->eeh_pe_config_addr) {
+ rtas_configure_bridge(ppe);
+ eeh_restore_bars(ppe);
}
- } else {
- rtas_configure_bridge(pe_dn);
- eeh_restore_bars(pe_dn);
- }
+ dn = dn->sibling;
+ }
/* Give the system 5 seconds to finish running the user-space
* hotplug shutdown scripts, e.g. ifdown for ethernet. Yes,
@@ -294,7 +298,7 @@ static int eeh_reset_device (struct pci_dn *pe_dn, struct pci_bus *bus)
/* The longest amount of time to wait for a pci device
* to come back on line, in seconds.
*/
-#define MAX_WAIT_FOR_RECOVERY 15
+#define MAX_WAIT_FOR_RECOVERY 150
struct pci_dn * handle_eeh_events (struct eeh_event *event)
{
@@ -306,18 +310,18 @@ struct pci_dn * handle_eeh_events (struct eeh_event *event)
const char *location, *pci_str, *drv_str;
frozen_dn = find_device_pe(event->dn);
- frozen_bus = pcibios_find_pci_bus(frozen_dn);
-
if (!frozen_dn) {
- location = (char *) get_property(event->dn, "ibm,loc-code", NULL);
+ location = of_get_property(event->dn, "ibm,loc-code", NULL);
location = location ? location : "unknown";
printk(KERN_ERR "EEH: Error: Cannot find partition endpoint "
"for location=%s pci addr=%s\n",
location, pci_name(event->dev));
return NULL;
}
- location = (char *) get_property(frozen_dn, "ibm,loc-code", NULL);
+
+ frozen_bus = pcibios_find_pci_bus(frozen_dn);
+ location = of_get_property(frozen_dn, "ibm,loc-code", NULL);
location = location ? location : "unknown";
/* There are two different styles for coming up with the PE.
@@ -336,13 +340,6 @@ struct pci_dn * handle_eeh_events (struct eeh_event *event)
return NULL;
}
-#if 0
- /* We may get "permanent failure" messages on empty slots.
- * These are false alarms. Empty slots have no child dn. */
- if ((event->state == pci_channel_io_perm_failure) && (frozen_device == NULL))
- return;
-#endif
-
frozen_pdn = PCI_DN(frozen_dn);
frozen_pdn->eeh_freeze_count++;
@@ -357,19 +354,12 @@ struct pci_dn * handle_eeh_events (struct eeh_event *event)
if (frozen_pdn->eeh_freeze_count > EEH_MAX_ALLOWED_FREEZES)
goto excess_failures;
- /* If the reset state is a '5' and the time to reset is 0 (infinity)
- * or is more then 15 seconds, then mark this as a permanent failure.
- */
- if ((event->state == pci_channel_io_perm_failure) &&
- ((event->time_unavail <= 0) ||
- (event->time_unavail > MAX_WAIT_FOR_RECOVERY*1000)))
- goto hard_fail;
-
- eeh_slot_error_detail(frozen_pdn, 1 /* Temporary Error */);
printk(KERN_WARNING
- "EEH: This PCI device has failed %d times since last reboot: "
- "location=%s driver=%s pci addr=%s\n",
- frozen_pdn->eeh_freeze_count, location, drv_str, pci_str);
+ "EEH: This PCI device has failed %d times in the last hour:\n",
+ frozen_pdn->eeh_freeze_count);
+ printk(KERN_WARNING
+ "EEH: location=%s driver=%s pci addr=%s\n",
+ location, drv_str, pci_str);
/* Walk the various device drivers attached to this slot through
* a reset sequence, giving each an opportunity to do what it needs
@@ -379,20 +369,38 @@ struct pci_dn * handle_eeh_events (struct eeh_event *event)
*/
pci_walk_bus(frozen_bus, eeh_report_error, &result);
+ /* Get the current PCI slot state. This can take a long time,
+ * sometimes over 3 seconds for certain systems. */
+ rc = eeh_wait_for_slot_status (frozen_pdn, MAX_WAIT_FOR_RECOVERY*1000);
+ if (rc < 0) {
+ printk(KERN_WARNING "EEH: Permanent failure\n");
+ goto hard_fail;
+ }
+
+ /* Since rtas may enable MMIO when posting the error log,
+ * don't post the error log until after all dev drivers
+ * have been informed.
+ */
+ eeh_slot_error_detail(frozen_pdn, EEH_LOG_TEMP_FAILURE);
+
/* If all device drivers were EEH-unaware, then shut
* down all of the device drivers, and hope they
* go down willingly, without panicing the system.
*/
if (result == PCI_ERS_RESULT_NONE) {
rc = eeh_reset_device(frozen_pdn, frozen_bus);
- if (rc)
+ if (rc) {
+ printk(KERN_WARNING "EEH: Unable to reset, rc=%d\n", rc);
goto hard_fail;
+ }
}
/* If all devices reported they can proceed, then re-enable MMIO */
if (result == PCI_ERS_RESULT_CAN_RECOVER) {
rc = rtas_pci_enable(frozen_pdn, EEH_THAW_MMIO);
+ if (rc < 0)
+ goto hard_fail;
if (rc) {
result = PCI_ERS_RESULT_NEED_RESET;
} else {
@@ -405,26 +413,37 @@ struct pci_dn * handle_eeh_events (struct eeh_event *event)
if (result == PCI_ERS_RESULT_CAN_RECOVER) {
rc = rtas_pci_enable(frozen_pdn, EEH_THAW_DMA);
+ if (rc < 0)
+ goto hard_fail;
if (rc)
result = PCI_ERS_RESULT_NEED_RESET;
+ else
+ result = PCI_ERS_RESULT_RECOVERED;
}
/* If any device has a hard failure, then shut off everything. */
- if (result == PCI_ERS_RESULT_DISCONNECT)
+ if (result == PCI_ERS_RESULT_DISCONNECT) {
+ printk(KERN_WARNING "EEH: Device driver gave up\n");
goto hard_fail;
+ }
/* If any device called out for a reset, then reset the slot */
if (result == PCI_ERS_RESULT_NEED_RESET) {
rc = eeh_reset_device(frozen_pdn, NULL);
- if (rc)
+ if (rc) {
+ printk(KERN_WARNING "EEH: Cannot reset, rc=%d\n", rc);
goto hard_fail;
+ }
result = PCI_ERS_RESULT_NONE;
pci_walk_bus(frozen_bus, eeh_report_reset, &result);
}
/* All devices should claim they have recovered by now. */
- if (result != PCI_ERS_RESULT_RECOVERED)
+ if ((result != PCI_ERS_RESULT_RECOVERED) &&
+ (result != PCI_ERS_RESULT_NONE)) {
+ printk(KERN_WARNING "EEH: Not recovered\n");
goto hard_fail;
+ }
/* Tell all device drivers that they can resume operations */
pci_walk_bus(frozen_bus, eeh_report_resume, NULL);
@@ -439,7 +458,8 @@ excess_failures:
*/
printk(KERN_ERR
"EEH: PCI device at location=%s driver=%s pci addr=%s \n"
- "has failed %d times and has been permanently disabled. \n"
+ "has failed %d times in the last hour "
+ "and has been permanently disabled. \n"
"Please try reseating this device or replacing it.\n",
location, drv_str, pci_str, frozen_pdn->eeh_freeze_count);
goto perm_error;
@@ -452,7 +472,7 @@ hard_fail:
location, drv_str, pci_str);
perm_error:
- eeh_slot_error_detail(frozen_pdn, 2 /* Permanent Error */);
+ eeh_slot_error_detail(frozen_pdn, EEH_LOG_PERM_FAILURE);
/* Notify all devices that they're about to go down. */
pci_walk_bus(frozen_bus, eeh_report_failure, NULL);
diff --git a/arch/powerpc/platforms/pseries/eeh_event.c b/arch/powerpc/platforms/pseries/eeh_event.c
index 45ccc68..8b5cdeb 100644
--- a/arch/powerpc/platforms/pseries/eeh_event.c
+++ b/arch/powerpc/platforms/pseries/eeh_event.c
@@ -37,7 +37,7 @@
/* EEH event workqueue setup. */
static DEFINE_SPINLOCK(eeh_eventlist_lock);
LIST_HEAD(eeh_eventlist);
-static void eeh_thread_launcher(void *);
+static void eeh_thread_launcher(struct work_struct *);
DECLARE_WORK(eeh_event_wq, eeh_thread_launcher, NULL);
/* Serialize reset sequences for a given pci device */
@@ -103,7 +103,7 @@ static int eeh_event_handler(void * dummy)
* eeh_thread_launcher
* @dummy - unused
*/
-static void eeh_thread_launcher(void *dummy)
+static void eeh_thread_launcher(struct work_struct *dummy)
{
if (kernel_thread(eeh_event_handler, NULL, CLONE_KERNEL) < 0)
printk(KERN_ERR "Failed to start EEH daemon\n");
@@ -118,17 +118,15 @@ static void eeh_thread_launcher(void *dummy)
* (from a workqueue).
*/
int eeh_send_failure_event (struct device_node *dn,
- struct pci_dev *dev,
- enum pci_channel_state state,
- int time_unavail)
+ struct pci_dev *dev)
{
unsigned long flags;
struct eeh_event *event;
- char *location;
+ const char *location;
if (!mem_init_done) {
printk(KERN_ERR "EEH: event during early boot not handled\n");
- location = (char *) get_property(dn, "ibm,loc-code", NULL);
+ location = of_get_property(dn, "ibm,loc-code", NULL);
printk(KERN_ERR "EEH: device node = %s\n", dn->full_name);
printk(KERN_ERR "EEH: PCI location = %s\n", location);
return 1;
@@ -144,8 +142,6 @@ int eeh_send_failure_event (struct device_node *dn,
event->dn = dn;
event->dev = dev;
- event->state = state;
- event->time_unavail = time_unavail;
/* We may or may not be called in an interrupt context */
spin_lock_irqsave(&eeh_eventlist_lock, flags);
diff --git a/drivers/scsi/lpfc/lpfc_sli.h b/drivers/scsi/lpfc/lpfc_sli.h
index 09fafb7..026a38a 100644
--- a/drivers/scsi/lpfc/lpfc_sli.h
+++ b/drivers/scsi/lpfc/lpfc_sli.h
@@ -19,14 +19,6 @@
*******************************************************************/
-/* This inline function is present in upstream but not in SLES 10
- */
-static inline int
-pci_channel_offline(struct pci_dev *pdev)
-{
- return (pdev->error_state != pci_channel_io_normal);
-}
-
/* forward declaration for LPFC_IOCB_t's use */
struct lpfc_hba;
struct lpfc_vport;
diff --git a/include/asm-powerpc/eeh.h b/include/asm-powerpc/eeh.h
index 9c5099b..dbfdf7a 100644
--- a/include/asm-powerpc/eeh.h
+++ b/include/asm-powerpc/eeh.h
@@ -44,6 +44,13 @@ static inline void mmio_read_fixup(const volatile void __iomem *addr)
extern int eeh_subsystem_enabled;
+/* Add new of_find_property macro from mainline 2.6.26 include/linux/of.h
+ * Remove when of.h is available
+ */
+#define for_each_child_of_node(parent, child) \
+ for (child = of_get_next_child(parent, NULL); child != NULL; \
+ child = of_get_next_child(parent, child))
+
/* Values for eeh_mode bits in device_node */
#define EEH_MODE_SUPPORTED (1<<0)
#define EEH_MODE_NOCHECK (1<<1)
diff --git a/include/asm-powerpc/eeh_event.h b/include/asm-powerpc/eeh_event.h
index dc6bf0f..cc3cb04 100644
--- a/include/asm-powerpc/eeh_event.h
+++ b/include/asm-powerpc/eeh_event.h
@@ -30,8 +30,6 @@ struct eeh_event {
struct list_head list;
struct device_node *dn; /* struct device node */
struct pci_dev *dev; /* affected device */
- enum pci_channel_state state; /* PCI bus state for the affected device */
- int time_unavail; /* milliseconds until device might be available */
};
/**
@@ -46,9 +44,7 @@ struct eeh_event {
* (from a workqueue).
*/
int eeh_send_failure_event (struct device_node *dn,
- struct pci_dev *dev,
- enum pci_channel_state state,
- int time_unavail);
+ struct pci_dev *dev);
/* Main recovery function */
struct pci_dn * handle_eeh_events (struct eeh_event *);
diff --git a/include/asm-powerpc/ppc-pci.h b/include/asm-powerpc/ppc-pci.h
index f85e393..dadf5b6 100644
--- a/include/asm-powerpc/ppc-pci.h
+++ b/include/asm-powerpc/ppc-pci.h
@@ -61,15 +61,18 @@ struct pci_dev *pci_get_device_by_addr(unsigned long addr);
/**
* eeh_slot_error_detail -- record and EEH error condition to the log
- * @severity: 1 if temporary, 2 if permanent failure.
+ * @pdn: pci device node
+ * @severity: EEH_LOG_TEMP_FAILURE or EEH_LOG_PERM_FAILURE
*
- * Obtains the the EEH error details from the RTAS subsystem,
+ * Obtains the EEH error details from the RTAS subsystem,
* and then logs these details with the RTAS error log system.
*/
+#define EEH_LOG_TEMP_FAILURE 1
+#define EEH_LOG_PERM_FAILURE 2
void eeh_slot_error_detail (struct pci_dn *pdn, int severity);
/**
- * rtas_pci_enableo - enable IO transfers for this slot
+ * rtas_pci_enable - enable IO transfers for this slot
* @pdn: pci device node
* @function: either EEH_THAW_MMIO or EEH_THAW_DMA
*
@@ -81,6 +84,7 @@ int rtas_pci_enable(struct pci_dn *pdn, int function);
/**
* rtas_set_slot_reset -- unfreeze a frozen slot
+ * @pdn: pci device node
*
* Clear the EEH-frozen condition on a slot. This routine
* does this by asserting the PCI #RST line for 1/8th of
@@ -90,9 +94,11 @@ int rtas_pci_enable(struct pci_dn *pdn, int function);
* Returns a non-zero value if the reset failed.
*/
int rtas_set_slot_reset (struct pci_dn *);
+int eeh_wait_for_slot_status(struct pci_dn *pdn, int max_wait_msecs);
/**
* eeh_restore_bars - Restore device configuration info.
+ * @pdn: pci device node
*
* A reset of a PCI device will clear out its config space.
* This routines will restore the config space for this
@@ -103,6 +109,7 @@ void eeh_restore_bars(struct pci_dn *);
/**
* rtas_configure_bridge -- firmware initialization of pci bridge
+ * @pdn: pci device node
*
* Ask the firmware to configure all PCI bridges devices
* located behind the indicated node. Required after a
@@ -116,16 +123,22 @@ int rtas_write_config(struct pci_dn *, int where, int size, u32 val);
int rtas_read_config(struct pci_dn *, int where, int size, u32 *val);
/**
+ * eeh_mark_slot -- set mode flags for pertition endpoint
+ * @pdn: pci device node
+ *
* mark and clear slots: find "partition endpoint" PE and set or
* clear the flags for each subnode of the PE.
*/
void eeh_mark_slot (struct device_node *dn, int mode_flag);
void eeh_clear_slot (struct device_node *dn, int mode_flag);
-/* Find the associated "Partiationable Endpoint" PE */
+/**
+ * find_device_pe -- Find the associated "Partitionable Endpoint" PE
+ * @pdn: pci device node
+ */
struct device_node * find_device_pe(struct device_node *dn);
-#endif
+#endif /* CONFIG_EEH */
#endif /* __KERNEL__ */
#endif /* _ASM_POWERPC_PPC_PCI_H */
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 3b2bcb1..9794a99 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -196,6 +196,11 @@ struct pci_dev {
#define to_pci_dev(n) container_of(n, struct pci_dev, dev)
#define for_each_pci_dev(d) while ((d = pci_get_device(PCI_ANY_ID, PCI_ANY_ID, d)) != NULL)
+static inline int pci_channel_offline(struct pci_dev *pdev)
+{
+ return (pdev->error_state != pci_channel_io_normal);
+}
+
static inline struct pci_cap_saved_state *pci_find_saved_cap(
struct pci_dev *pci_dev,char cap)
{
distrib
>
Scientific%20Linux
>
5x
>
x86_64
>
by-pkgid
>
fc11cd6e1c513a17304da94a5390f3cd
>
files
>
2938
