From: George Beshers <gbeshers@redhat.com>
Date: Thu, 11 Sep 2008 17:53:42 -0400
Subject: [misc] holdoffs in hrtimer_run_queues
Message-id: 20080911214233.23760.97225.sendpatchset@dhcp-100-2-194.bos.redhat.com
O-Subject: [RHEL5.3 PATCH 1/2] [v3] BZ442148: holdoffs in hrtimer_run_queues
Bugzilla: 442148
RH-Acked-by: Prarit Bhargava <prarit@redhat.com>
RH-Acked-by: Jason Baron <jbaron@redhat.com>

BZ#442148: holdoffs in hrtimer_run_queues on >64p systems

Upstream:
http://git2.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=833883d9ac4cfb31c1c4419335e68e6895a05b6b
http://git2.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=259aae864ceeb2b34e7bafa1ce18d096a357fab2

Brew with both patches:
http://brewweb.devel.redhat.com/brew/taskinfo?taskID=1467867

Patch 1/2:

hrtimer: reduce calls to hrtimer_get_softirq_time()

It seems that hrtimer_run_queues() is calling hrtimer_get_softirq_time() more
often than it needs to.  This can cause frequent contention on systems with
large numbers of processors/cores.

With this patch, hrtimer_run_queues only calls hrtimer_get_softirq_time() if
there is a pending timer in one of the hrtimer bases, and only once.

This also combines hrtimer_run_queues() and the inline run_hrtimer_queue()
into one function.

Signed-off-by: Dimitri Sivanich <sivanich@sgi.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>

hrtimer: optimize the softirq time optimization

The previous optimization did not take the case into account where a
clock provides its own softirq_get_time() function.

Check for the availablitiy of the clock get time function first and
then check if we need to retrieve the time for both clocks via
hrtimer_softirq_gettime() to avoid a double evaluation of time in that
case as well.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>

diff --git a/kernel/hrtimer.c b/kernel/hrtimer.c
index 46ee71f..f543284 100644
--- a/kernel/hrtimer.c
+++ b/kernel/hrtimer.c
@@ -605,59 +605,61 @@ int hrtimer_get_res(const clockid_t which_clock, struct timespec *tp)
 EXPORT_SYMBOL_GPL(hrtimer_get_res);
 
 /*
- * Expire the per base hrtimer-queue:
+ * Called from timer softirq every jiffy, expire hrtimers.
+ *
+ * Expire the per base hrtimer-queues:
  */
-static inline void run_hrtimer_queue(struct hrtimer_base *base)
+void hrtimer_run_queues(void)
 {
 	struct rb_node *node;
+	struct hrtimer_base *cpu_base = __get_cpu_var(hrtimer_bases);
+	struct hrtimer_base *base;
+	int index;
+	int gst = 1;
 
-	if (!base->first)
-		return;
 
-	if (base->get_softirq_time)
-		base->softirq_time = base->get_softirq_time();
+	for (index = 0; index < MAX_HRTIMER_BASES; index++) {
+		base = &cpu_base[index];
 
-	spin_lock_irq(&base->lock);
+		if (!base->first)
+			continue;
 
-	while ((node = base->first)) {
-		struct hrtimer *timer;
-		int (*fn)(struct hrtimer *);
-		int restart;
+		if (gst) {
+			hrtimer_get_softirq_time(cpu_base);
+			gst = 0;
+		}
 
-		timer = rb_entry(node, struct hrtimer, node);
-		if (base->softirq_time.tv64 <= timer->expires.tv64)
-			break;
+		if (base->get_softirq_time)
+			base->softirq_time = base->get_softirq_time();
 
-		fn = timer->function;
-		set_curr_timer(base, timer);
-		__remove_hrtimer(timer, base);
-		spin_unlock_irq(&base->lock);
+		spin_lock_irq(&base->lock);
 
-		restart = fn(timer);
+		while ((node = base->first)) {
+			struct hrtimer *timer;
+			int (*fn)(struct hrtimer *);
+			int restart;
 
-		spin_lock_irq(&base->lock);
+			timer = rb_entry(node, struct hrtimer, node);
+			if (base->softirq_time.tv64 <= timer->expires.tv64)
+				break;
 
-		if (restart != HRTIMER_NORESTART) {
-			BUG_ON(hrtimer_active(timer));
-			enqueue_hrtimer(timer, base);
-		}
-	}
-	set_curr_timer(base, NULL);
-	spin_unlock_irq(&base->lock);
-}
+			fn = timer->function;
+			set_curr_timer(base, timer);
+			__remove_hrtimer(timer, base);
+			spin_unlock_irq(&base->lock);
 
-/*
- * Called from timer softirq every jiffy, expire hrtimers:
- */
-void hrtimer_run_queues(void)
-{
-	struct hrtimer_base *base = __get_cpu_var(hrtimer_bases);
-	int i;
+			restart = fn(timer);
 
-	hrtimer_get_softirq_time(base);
+			spin_lock_irq(&base->lock);
 
-	for (i = 0; i < MAX_HRTIMER_BASES; i++)
-		run_hrtimer_queue(&base[i]);
+			if (restart != HRTIMER_NORESTART) {
+				BUG_ON(hrtimer_active(timer));
+				enqueue_hrtimer(timer, base);
+			}
+		}
+		set_curr_timer(base, NULL);
+		spin_unlock_irq(&base->lock);
+	}
 }
 
 /*