kernel.git/mm/percpu.c, branch linux-2.6.32.y

percpu: fix chunk range calculation

2011-12-21T21:04:52Z

commit a855b84c3d8c73220d4d3cd392a7bee7c83de70e upstream. Percpu allocator recorded the cpus which map to the first and last units in pcpu_first/last_unit_cpu respectively and used them to determine the address range of a chunk - e.g. it assumed that the first unit has the lowest address in a chunk while the last unit has the highest address. This simply isn't true. Groups in a chunk can have arbitrary positive or negative offsets from the previous one and there is no guarantee that the first unit occupies the lowest offset while the last one the highest. Fix it by actually comparing unit offsets to determine cpus occupying the lowest and highest offsets. Also, rename pcu_first/last_unit_cpu to pcpu_low/high_unit_cpu to avoid confusion. The chunk address range is used to flush cache on vmalloc area map/unmap and decide whether a given address is in the first chunk by per_cpu_ptr_to_phys() and the bug was discovered by invalid per_cpu_ptr_to_phys() translation for crash_note. Kudos to Dave Young for tracking down the problem. Signed-off-by: Tejun Heo Reported-by: WANG Cong Reported-by: Dave Young Tested-by: Dave Young LKML-Reference: <4EC21F67.10905@redhat.com> Signed-off-by: Thomas Renninger Signed-off-by: Greg Kroah-Hartman

percpu: fix pcpu_last_unit_cpu

2010-09-27T00:21:27Z

commit 46b30ea9bc3698bc1d1e6fd726c9601d46fa0a91 upstream. pcpu_first/last_unit_cpu are used to track which cpu has the first and last units assigned. This in turn is used to determine the span of a chunk for man/unmap cache flushes and whether an address belongs to the first chunk or not in per_cpu_ptr_to_phys(). When the number of possible CPUs isn't power of two, a chunk may contain unassigned units towards the end of a chunk. The logic to determine pcpu_last_unit_cpu was incorrect when there was an unused unit at the end of a chunk. It failed to ignore the unused unit and assigned the unused marker NR_CPUS to pcpu_last_unit_cpu. This was discovered through kdump failure which was caused by malfunctioning per_cpu_ptr_to_phys() on a kvm setup with 50 possible CPUs by CAI Qian. Signed-off-by: Tejun Heo Reported-by: CAI Qian Signed-off-by: Greg Kroah-Hartman

percpu: restructure pcpu_extend_area_map() to fix bugs and improve readability

2009-11-12T15:55:35Z

pcpu_extend_area_map() had the following two bugs. * It should return 1 if pcpu_lock was dropped and reacquired but it returned 0. This could lead to oops if free_percpu() races with area map extension. * pcpu_mem_free() was called under pcpu_lock. pcpu_mem_free() might end up calling vfree() which isn't IRQ safe. This could lead to deadlock through lock order inversion via IRQ. In addition, Linus pointed out that the temporary lock dropping and subtle three-way return value of pcpu_extend_area_map() was very ugly and suggested to split the function into two - pcpu_need_to_extend() and pcpu_extend_area_map(). This patch restructures pcpu_extend_area_map() as suggested and fixes the two bugs. Signed-off-by: Tejun Heo Acked-by: Linus Torvalds Cc: Ingo Molnar

percpu: allow pcpu_alloc() to be called with IRQs off

2009-10-28T15:25:59Z

pcpu_alloc() and pcpu_extend_area_map() perform a series of spin_lock_irq()/spin_unlock_irq() calls, which make them unsafe with respect to being called from contexts which have IRQs off. This patch converts the code to perform save/restore of flags instead, making pcpu_alloc() (or __alloc_percpu() respectively) to be called from early kernel startup stage, where IRQs are off. This is needed for proper initialization of per-cpu rq_weight data from sched_init(). tj: added comment explaining why irqsave/restore is used in alloc path. Signed-off-by: Jiri Kosina Acked-by: Ingo Molnar Signed-off-by: Tejun Heo

percpu: fix compile warnings

2009-10-12T08:04:42Z

Fix the following two compile warnings which show up on i386. mm/percpu.c:1873: warning: comparison of distinct pointer types lacks a cast mm/percpu.c:1879: warning: format '%lx' expects type 'long unsigned int', but argument 2 has type 'size_t' Signed-off-by: Tejun Heo Reported-by: Randy Dunlap

percpu: make allocation failures more verbose

2009-09-29T00:17:58Z

Warn and dump stack when percpu allocation fails. percpu allocator is still young and unchecked NULL percpu pointer usage can result in random memory corruption when combined with the pointer shifting in access macros. Allocation failures should be rare and the warning message will be disabled after certain times. Signed-off-by: Tejun Heo

percpu: make pcpu_setup_first_chunk() failures more verbose

2009-09-29T00:17:57Z

The parameters to pcpu_setup_first_chunk() come from different sources depending on architecture and can be quite complex. The function runs various sanity checks on the parameters and triggers BUG() if something isn't right. However, this is very early during the boot and not reporting exactly what the problem is makes debugging even harder. Add PCPU_SETUP_BUG() macro which prints out enough information about the parameters. As the macro still puts separate BUG() for each check, it won't lose any information even on the situations where only the program counter can be retrieved. While at it, also bump pcpu_dump_alloc_info() message to KERN_INFO so that it's visible on the console if boot fails to complete. Signed-off-by: Tejun Heo

percpu: make embedding first chunk allocator check vmalloc space size

2009-09-29T00:17:57Z

Embedding first chunk allocator maintains the distances between units in the vmalloc area and thus needs vmalloc space to be larger than the maximum distances between units; otherwise, it wouldn't be able to create any dynamic chunks. This patch makes the embedding first chunk allocator check vmalloc space size and if the maximum distance between units is larger than 75% of it, print warning and, if page mapping allocator is available, fail initialization so that the system falls back onto it. This should work around percpu allocation failure problems on certain sparc64 configurations where distances between NUMA nodes are larger than the vmalloc area and makes percpu allocator more robust for future configurations. Signed-off-by: Tejun Heo

percpu: make pcpu_build_alloc_info() clear static buffers

2009-09-29T00:17:57Z

pcpu_build_alloc_info() may be called multiple times when percpu is falling back to different first chunk allocator. Make it clear static buffers so that they don't contain values from previous runs. Signed-off-by: Tejun Heo

percpu: fix unit_map[] verification in pcpu_setup_first_chunk()

2009-09-29T00:17:56Z

pcpu_setup_first_chunk() incorrectly used NR_CPUS as the impossible unit number while unit number can equal and go over NR_CPUS with sparse unit map. This triggers BUG_ON() spuriously on machines which have non-power-of-two number of cpus. Use UINT_MAX instead. Signed-off-by: Tejun Heo Reported-and-tested-by: Tony Vroon