kernel.git/arch/i386/kernel/process.c, branch linux-2.6.19.y

[PATCH] sched: fix bad missed wakeups in the i386, x86_64, ia64, ACPI and APM idle code

2007-01-10T19:05:19Z

Fernando Lopez-Lezcano reported frequent scheduling latencies and audio xruns starting at the 2.6.18-rt kernel, and those problems persisted all until current -rt kernels. The latencies were serious and unjustified by system load, often in the milliseconds range. After a patient and heroic multi-month effort of Fernando, where he tested dozens of kernels, tried various configs, boot options, test-patches of mine and provided latency traces of those incidents, the following 'smoking gun' trace was captured by him: _------=> CPU# / _-----=> irqs-off | / _----=> need-resched || / _---=> hardirq/softirq ||| / _--=> preempt-depth |||| / ||||| delay cmd pid ||||| time | caller \ / ||||| \ | / IRQ_19-1479 1D..1 0us : __trace_start_sched_wakeup (try_to_wake_up) IRQ_19-1479 1D..1 0us : __trace_start_sched_wakeup <<...>-5856> (37 0) IRQ_19-1479 1D..1 0us : __trace_start_sched_wakeup (c01262ba 0 0) IRQ_19-1479 1D..1 0us : resched_task (try_to_wake_up) IRQ_19-1479 1D..1 0us : __spin_unlock_irqrestore (try_to_wake_up) ... -0 1...1 11us!: default_idle (cpu_idle) ... -0 0Dn.1 602us : smp_apic_timer_interrupt (c0103baf 1 0) ... <...>-5856 0D..2 618us : __switch_to (__schedule) <...>-5856 0D..2 618us : __schedule <-0> (20 162) <...>-5856 0D..2 619us : __spin_unlock_irq (__schedule) <...>-5856 0...1 619us : trace_stop_sched_switched (__schedule) <...>-5856 0D..1 619us : trace_stop_sched_switched <<...>-5856> (37 0) what is visible in this trace is that CPU#1 ran try_to_wake_up() for PID:5856, it placed PID:5856 on CPU#0's runqueue and ran resched_task() for CPU#0. But it decided to not send an IPI that no CPU - due to TS_POLLING. But CPU#0 never woke up after its NEED_RESCHED bit was set, and only rescheduled to PID:5856 upon the next lapic timer IRQ. The result was a 600+ usecs latency and a missed wakeup! the bug turned out to be an idle-wakeup bug introduced into the mainline kernel this summer via an optimization in the x86_64 tree: commit 495ab9c045e1b0e5c82951b762257fe1c9d81564 Author: Andi Kleen Date: Mon Jun 26 13:59:11 2006 +0200 [PATCH] i386/x86-64/ia64: Move polling flag into thread_info_status During some profiling I noticed that default_idle causes a lot of memory traffic. I think that is caused by the atomic operations to clear/set the polling flag in thread_info. There is actually no reason to make this atomic - only the idle thread does it to itself, other CPUs only read it. So I moved it into ti->status. the problem is this type of change: if (!hlt_counter && boot_cpu_data.hlt_works_ok) { - clear_thread_flag(TIF_POLLING_NRFLAG); + current_thread_info()->status &= ~TS_POLLING; smp_mb__after_clear_bit(); while (!need_resched()) { local_irq_disable(); this changes clear_thread_flag() to an explicit clearing of TS_POLLING. clear_thread_flag() is defined as: clear_bit(flag, &ti->flags); and clear_bit() is a LOCK-ed atomic instruction on all x86 platforms: static inline void clear_bit(int nr, volatile unsigned long * addr) { __asm__ __volatile__( LOCK_PREFIX "btrl %1,%0" hence smp_mb__after_clear_bit() is defined as a simple compile barrier: #define smp_mb__after_clear_bit() barrier() but the explicit TS_POLLING clearing introduced by the patch: + current_thread_info()->status &= ~TS_POLLING; is not an atomic op! So the clearing of the TS_POLLING bit is freely reorderable with the reading of the NEED_RESCHED bit - and both now reside in different memory addresses. CPU idle wakeup very much depends on ordered memory ops, the clearing of the TS_POLLING flag must always be done before we test need_resched() and hit the idle instruction(s). [Symmetrically, the wakeup code needs to set NEED_RESCHED before it tests the TS_POLLING flag, so memory ordering is paramount.] Fernando's dual-core Athlon64 system has a sufficiently advanced memory ordering model so that it triggered this scenario very often. ( And it also turned out that the reason why these latencies never triggered on my testsystems is that i routinely use idle=poll, which was the only idle variant not affected by this bug. ) The fix is to change the smp_mb__after_clear_bit() to an smp_mb(), to act as an absolute barrier between the TS_POLLING write and the NEED_RESCHED read. This affects almost all idling methods (default, ACPI, APM), on all 3 x86 architectures: i386, x86_64, ia64. Signed-off-by: Ingo Molnar Tested-by: Fernando Lopez-Lezcano [chrisw: backport to 2.6.19.1] Signed-off-by: Chris Wright

[PATCH] i386/x86_64: ACPI cpu_idle_wait() fix

2006-11-17T16:20:09Z

The scheduler on Andreas Friedrich's hyperthreading system stopped working properly: the scheduler would never move tasks to another CPU! The lask known working kernel was 2.6.8. After a couple of attempts to corner the bug, the following smoking gun was found: BIOS reported wrong ACPI idfor the processor CPU#1: set_cpus_allowed(), swapper:1, 3 -> 2 [] show_trace_log_lvl+0x34/0x4a [] show_trace+0x2c/0x2e [] dump_stack+0x2b/0x2d [] set_cpus_allowed+0x52/0xec [] cpu_idle_wait+0x2e/0x100 [] acpi_processor_power_exit+0x45/0x58 [] acpi_processor_remove+0x46/0xea [] acpi_start_single_object+0x47/0x54 [] acpi_bus_register_driver+0xa4/0xd3 [] acpi_processor_init+0x57/0x77 [] init+0x146/0x2fd [] kernel_thread_helper+0x7/0x10 a quick look at cpu_idle_wait() shows how broken that code is on i386: it changes the init task's affinity map but never restores it ... and because all userspace tasks get forked by init, they all inherited that single-CPU affinity mask. x86_64 cloned this bug too. Signed-off-by: Ingo Molnar Cc: Andreas Friedrich Cc: Wolfgang Erig Cc: Andrew Morton Cc: Adrian Bunk Signed-off-by: Linus Torvalds

[PATCH] x86: Revert new unwind kernel stack termination

2006-10-21T16:37:02Z

Jan convinced me that it was unnecessary because the assembly stubs do this already on the stack. Cc: jbeulich@novell.com Signed-off-by: Andi Kleen

ACPI: Processor native C-states using MWAIT

2006-10-14T04:35:39Z

Intel processors starting with the Core Duo support support processor native C-state using the MWAIT instruction. Refer: Intel Architecture Software Developer's Manual http://www.intel.com/design/Pentium4/manuals/253668.htm Platform firmware exports the support for Native C-state to OS using ACPI _PDC and _CST methods. Refer: Intel Processor Vendor-Specific ACPI: Interface Specification http://www.intel.com/technology/iapc/acpi/downloads/302223.htm With Processor Native C-state, we use 'MWAIT' instruction on the processor to enter different C-states (C1, C2, C3). We won't use the special IO ports to enter C-state and no SMM mode etc required to enter C-state. Overall this will mean better C-state support. One major advantage of using MWAIT for all C-states is, with this and "treat interrupt as break event" feature of MWAIT, we can now get accurate timing for the time spent in C1, C2, .. states. Signed-off-by: Venkatesh Pallipadi Signed-off-by: Andrew Morton Signed-off-by: Len Brown

[PATCH] x86: Terminate the kernel stacks for the unwinder

2006-10-05T16:47:22Z

Always make sure RIP/EIP is 0 in the registers stored on the top of the stack of a kernel thread. This makes sure the unwinder code won't try a fallback but knows the stack has ended. AK: this patch is a bit mysterious. in theory they should be terminated anyways, but it seems to fix at least one crash. Anyways double termination probably doesn't hurt. Signed-off-by: Andi Kleen

[PATCH] namespaces: utsname: use init_utsname when appropriate

2006-10-02T14:57:21Z

In some places, particularly drivers and __init code, the init utsns is the appropriate one to use. This patch replaces those with a the init_utsname helper. Changes: Removed several uses of init_utsname(). Hope I picked all the right ones in net/ipv4/ipconfig.c. These are now changed to utsname() (the per-process namespace utsname) in the previous patch (2/7) [akpm@osdl.org: CIFS fix] Signed-off-by: Serge E. Hallyn Cc: Kirill Korotaev Cc: "Eric W. Biederman" Cc: Herbert Poetzl Cc: Andrey Savochkin Cc: Serge Hallyn Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds

[PATCH] kmemdup: some users

2006-10-01T07:39:19Z

Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds

[PATCH] i386: Allow a kernel not to be in ring 0

2006-09-26T08:52:39Z

We allow for the fact that the guest kernel may not run in ring 0. This requires some abstraction in a few places when setting %cs or checking privilege level (user vs kernel). This is Chris' [RFC PATCH 15/33] move segment checks to subarch, except rather than using #define USER_MODE_MASK which depends on a config option, we use Zach's more flexible approach of assuming ring 3 == userspace. I also used "get_kernel_rpl()" over "get_kernel_cs()" because I think it reads better in the code... 1) Remove the hardcoded 3 and introduce #define SEGMENT_RPL_MASK 3 2) Add a get_kernel_rpl() macro, and don't assume it's zero. And: Clean up of patch for letting kernel run other than ring 0: a. Add some comments about the SEGMENT_IS_*_CODE() macros. b. Add a USER_RPL macro. (Code was comparing a value to a mask in some places and to the magic number 3 in other places.) c. Add macros for table indicator field and use them. d. Change the entry.S tests for LDT stack segment to use the macros Signed-off-by: Rusty Russell Signed-off-by: Zachary Amsden Signed-off-by: Jeremy Fitzhardinge Signed-off-by: Andrew Morton Signed-off-by: Andi Kleen

[PATCH] i386: move kernel_thread_helper into entry.S

2006-09-26T08:52:35Z

And add proper CFI annotation to it which was previously impossible. This prevents "stuck" messages by the dwarf2 unwinder when reaching the top of a kernel stack. Includes feedback from Jan Beulich Cc: jbeulich@novell.com Signed-off-by: Andi Kleen

[PATCH] i386/x86-64: Don't randomize stack top when no randomization personality is set

2006-09-26T08:52:28Z

Based on patch from Frank van Maarseveen , but extended. Signed-off-by: Andi Kleen