From: AMEET M. PARANJAPE <aparanja@redhat.com> Date: Thu, 30 Oct 2008 09:52:31 -0500 Subject: [openib] ppc64: fix using SDP on 64K page systems Message-id: 4909CA2F.4030308@REDHAT.COM O-Subject: Re: [PATCH RHEL5.3 BZ468872] Running applications over SDP causing system crash when configured for 64K pages Bugzilla: 468872 RH-Acked-by: David Howells <dhowells@redhat.com> RH-Acked-by: David Howells <dhowells@redhat.com> RH-Acked-by: Doug Ledford <dledford@redhat.com> RHBZ#: ====== https://bugzilla.redhat.com/show_bug.cgi?id=468872 Description: =========== When 64K pages are in use, the skb_frag size can become larger than the skb_frag can address. An skb_frag's max size is 64K-1. This patch defines SDP_MAX_PAYLOAD as 64K - SDP_HEADER_SIZE. The patch changes sdp_post_recv() and sdp_sendmsg() to use the smaller of PAGE_SIZE or SDP_MAX_PAYLOAD as it segment size. The OFED packages are available here: http://www.openfabrics.org/ RHEL Version Found: ================ RHEL 5.2 and 5.3 alpha kABI Status: ============ No symbols were harmed. Brew: ===== Built on all platforms. http://brewweb.devel.redhat.com/brew/taskinfo?taskID=1544100 Upstream Status: ================ The patch has been checked into the ofed Sockets Direct Protocol (SDP) matainer's git and will be in 1.4 release: https://bugs.openfabrics.org/show_bug.cgi?id=1300 Test Status: ============ Without this patch running netperf suite over Sockets Direct Protocol (SDP) induces a system crash when 64k pages are in use. This crash is not seen with this patch. =============================================================== Ameet Paranjape 978-392-3903 ext 23903 IBM on-site partner Proposed Patch: =============== diff --git a/drivers/infiniband/ulp/sdp/sdp.h b/drivers/infiniband/ulp/sdp/sdp.h index 24f5caf..57a95ce 100644 --- a/drivers/infiniband/ulp/sdp/sdp.h +++ b/drivers/infiniband/ulp/sdp/sdp.h @@ -50,6 +50,7 @@ extern int sdp_data_debug_level; #define SDP_MAX_SEND_SKB_FRAGS (PAGE_SIZE > 0x8000 ? 1 : 0x8000 / PAGE_SIZE) #define SDP_HEAD_SIZE (PAGE_SIZE / 2 + sizeof(struct sdp_bsdh)) #define SDP_NUM_WC 4 +#define SDP_MAX_PAYLOAD ((1 << 16) - SDP_HEAD_SIZE) #define SDP_MIN_ZCOPY_THRESH 1024 #define SDP_MAX_ZCOPY_THRESH 1048576 diff --git a/drivers/infiniband/ulp/sdp/sdp_bcopy.c b/drivers/infiniband/ulp/sdp/sdp_bcopy.c index 1aeb264..845935b 100644 --- a/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ b/drivers/infiniband/ulp/sdp/sdp_bcopy.c @@ -288,11 +288,11 @@ static void sdp_post_recv(struct sdp_sock *ssk) frag = &skb_shinfo(skb)->frags[i]; frag->page = page; frag->page_offset = 0; - frag->size = PAGE_SIZE; + frag->size = min(PAGE_SIZE, SDP_MAX_PAYLOAD); ++skb_shinfo(skb)->nr_frags; - skb->len += PAGE_SIZE; - skb->data_len += PAGE_SIZE; - skb->truesize += PAGE_SIZE; + skb->len += frag->size; + skb->data_len += frag->size; + skb->truesize += frag->size; } rx_req = ssk->rx_ring + (id & (SDP_RX_SIZE - 1)); diff --git a/drivers/infiniband/ulp/sdp/sdp_main.c b/drivers/infiniband/ulp/sdp/sdp_main.c index c46fd60..0ac9954 100644 --- a/drivers/infiniband/ulp/sdp/sdp_main.c +++ b/drivers/infiniband/ulp/sdp/sdp_main.c @@ -1620,6 +1620,10 @@ int sdp_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg, iov++; + /* Limiting the size_goal is required when using 64K pages*/ + if (size_goal > SDP_MAX_PAYLOAD) + size_goal = SDP_MAX_PAYLOAD; + bz = sdp_bz_setup(ssk, from, seglen, size_goal); while (seglen > 0) {