kernel.git/arch/x86/include/asm/percpu.h, branch linux-4.13.y

x86: Remove x86_test_and_clear_bit_percpu()

2016-11-18T11:07:57Z

Upon removal of the "is_idle" flag, x86_test_and_clear_bit_percpu() is no longer used. Signed-off-by: Len Brown Acked-by: Ingo Molnar Acked-by: Peter Zijlstra (Intel) Link: http://lkml.kernel.org/r/b334ae6819507e3dfc0a4b33ed974714d067eb4a.1479449716.git.len.brown@intel.com Signed-off-by: Thomas Gleixner

percpu: eliminate two sparse warnings

2016-09-22T16:14:29Z

Fix two cases where a __percpu pointer cast drops __percpu. Signed-off-by: Lance Richardson Signed-off-by: Tejun Heo

x86, asm: Use CC_SET()/CC_OUT() in

2016-06-08T19:41:20Z

Remove open-coded uses of set instructions to use CC_SET()/CC_OUT() in . Signed-off-by: H. Peter Anvin Link: http://lkml.kernel.org/r/1465414726-197858-8-git-send-email-hpa@linux.intel.com Reviewed-by: Andy Lutomirski Reviewed-by: Borislav Petkov Acked-by: Peter Zijlstra (Intel)

x86, asm: use bool for bitops and other assembly outputs

2016-06-08T19:41:20Z

The gcc people have confirmed that using "bool" when combined with inline assembly always is treated as a byte-sized operand that can be assumed to be 0 or 1, which is exactly what the SET instruction emits. Change the output types and intermediate variables of as many operations as practical to "bool". Signed-off-by: H. Peter Anvin Link: http://lkml.kernel.org/r/1465414726-197858-3-git-send-email-hpa@linux.intel.com Reviewed-by: Andy Lutomirski Reviewed-by: Borislav Petkov Acked-by: Peter Zijlstra (Intel)

x86, bitops: remove use of "sbb" to return CF

2016-06-08T19:41:20Z

Use SETC instead of SBB to return the value of CF from assembly. Using SETcc enables uniformity with other flags-returning pieces of assembly code. Signed-off-by: H. Peter Anvin Link: http://lkml.kernel.org/r/1465414726-197858-2-git-send-email-hpa@linux.intel.com Reviewed-by: Andy Lutomirski Reviewed-by: Borislav Petkov Acked-by: Peter Zijlstra (Intel)

x86-64: Use RIP-relative addressing for most per-CPU accesses

2014-11-04T19:43:14Z

Observing that per-CPU data (in the SMP case) is reachable by exploiting 64-bit address wraparound (building on the default kernel load address being at 16Mb), the one byte shorter RIP-relative addressing form can be used for most per-CPU accesses. The one exception are the "stable" reads, where the use of the "P" operand modifier prevents the compiler from using RIP-relative addressing, but is unavoidable due to the use of the "p" constraint (side note: with gcc 4.9.x the intended effect of this isn't being achieved anymore, see gcc bug 63637). With the dependency on the minimum kernel load address, arbitrarily low values for CONFIG_PHYSICAL_START are now no longer possible. A link time assertion is being added, directing to the need to increase that value when it triggers. Signed-off-by: Jan Beulich Link: http://lkml.kernel.org/r/5458A1780200007800044A9D@mail.emea.novell.com Signed-off-by: Thomas Gleixner

x86: Convert a few more per-CPU items to read-mostly ones

2014-11-04T19:13:28Z

Both this_cpu_off and cpu_info aren't getting modified post boot, yet are being accessed on enough code paths that grouping them with other frequently read items seems desirable. For cpu_info this at the same time implies removing the cache line alignment (which afaict became pointless when it got converted to per-CPU data years ago). Signed-off-by: Jan Beulich Link: http://lkml.kernel.org/r/54589BD20200007800044A84@mail.emea.novell.com Signed-off-by: Thomas Gleixner

percpu: invoke __verify_pcpu_ptr() from the generic part of accessors and operations

2014-06-17T23:12:40Z

__verify_pcpu_ptr() is used to verify that a specified parameter is actually an percpu pointer by percpu accessor and operation implementations. Currently, where it's called isn't clearly defined and we just ensure that it's invoked at least once for all accessors and operations. The lack of clarity on when it should be called isn't nice and given that this is a completely generic issue, there's no reason to make archs worry about it. This patch updates __verify_pcpu_ptr() invocations such that it's always invoked from the final generic wrapper once per access or operation. As this is already the case for {raw|this}_cpu_*() definitions through __pcpu_size_*(), only the {raw|per|this}_cpu_ptr() accessors need to be updated. This change makes it unnecessary for archs to worry about __verify_pcpu_ptr(). x86's arch_raw_cpu_ptr() is updated accordingly. Signed-off-by: Tejun Heo Cc: Christoph Lameter Cc: Thomas Gleixner Cc: Ingo Molnar Cc: "H. Peter Anvin"

percpu: introduce arch_raw_cpu_ptr()

2014-06-17T23:12:34Z

Currently, archs can override raw_cpu_ptr() directly; however, we wanna build a layer of indirection in the generic part of percpu so that we can implement generic features there without affecting archs. Introduce arch_raw_cpu_ptr() which is used to define raw_cpu_ptr() by generic percpu code. The two are identical for now. x86 is currently the only arch which overrides raw_cpu_ptr() and is converted to define arch_raw_cpu_ptr() instead. This doesn't introduce any functional difference. Signed-off-by: Tejun Heo Cc: Christoph Lameter Cc: Thomas Gleixner Cc: Ingo Molnar Cc: "H. Peter Anvin"

percpu: add raw_cpu_ops

2014-04-07T23:36:13Z

The kernel has never been audited to ensure that this_cpu operations are consistently used throughout the kernel. The code generated in many places can be improved through the use of this_cpu operations (which uses a segment register for relocation of per cpu offsets instead of performing address calculations). The patch set also addresses various consistency issues in general with the per cpu macros. A. The semantics of __this_cpu_ptr() differs from this_cpu_ptr only because checks are skipped. This is typically shown through a raw_ prefix. So this patch set changes the places where __this_cpu_ptr() is used to raw_cpu_ptr(). B. There has been the long term wish by some that __this_cpu operations would check for preemption. However, there are cases where preemption checks need to be skipped. This patch set adds raw_cpu operations that do not check for preemption and then adds preemption checks to the __this_cpu operations. C. The use of __get_cpu_var is always a reference to a percpu variable that can also be handled via a this_cpu operation. This patch set replaces all uses of __get_cpu_var with this_cpu operations. D. We can then use this_cpu RMW operations in various places replacing sequences of instructions by a single one. E. The use of this_cpu operations throughout will allow other arches than x86 to implement optimized references and RMV operations to work with per cpu local data. F. The use of this_cpu operations opens up the possibility to further optimize code that relies on synchronization through per cpu data. The patch set works in a couple of stages: I. Patch 1 adds the additional raw_cpu operations and raw_cpu_ptr(). Also converts the existing __this_cpu_xx_# primitive in the x86 code to raw_cpu_xx_#. II. Patch 2-4 use the raw_cpu operations in places that would give us false positives once they are enabled. III. Patch 5 adds preemption checks to __this_cpu operations to allow checking if preemption is properly disabled when these functions are used. IV. Patches 6-20 are patches that simply replace uses of __get_cpu_var with this_cpu_ptr. They do not depend on any changes to the percpu code. No preemption tests are skipped if they are applied. V. Patches 21-46 are conversion patches that use this_cpu operations in various kernel subsystems/drivers or arch code. VI. Patches 47/48 (not included in this series) remove no longer used functions (__this_cpu_ptr and __get_cpu_var). These should only be applied after all the conversion patches have made it and after we have done additional passes through the kernel to ensure that none of the uses of these functions remain. This patch (of 46): The patches following this one will add preemption checks to __this_cpu ops so we need to have an alternative way to use this_cpu operations without preemption checks. raw_cpu_ops will be the basis for all other ops since these will be the operations that do not implement any checks. Primitive operations are renamed by this patch from __this_cpu_xxx to raw_cpu_xxxx. Also change the uses of the x86 percpu primitives in preempt.h. These depend directly on asm/percpu.h (header #include nesting issue). Signed-off-by: Peter Zijlstra Signed-off-by: Christoph Lameter Acked-by: Ingo Molnar Cc: Tejun Heo Cc: "James E.J. Bottomley" Cc: "Paul E. McKenney" Cc: Alex Shi Cc: Arnd Bergmann Cc: Benjamin Herrenschmidt Cc: Bryan Wu Cc: Catalin Marinas Cc: Chris Metcalf Cc: Daniel Lezcano Cc: David Daney Cc: David Miller Cc: David S. Miller Cc: Dimitri Sivanich Cc: Dipankar Sarma Cc: Eric Dumazet Cc: Fenghua Yu Cc: Frederic Weisbecker Cc: Greg Kroah-Hartman Cc: H. Peter Anvin Cc: Haavard Skinnemoen Cc: Hans-Christian Egtvedt Cc: Hedi Berriche Cc: Heiko Carstens Cc: Helge Deller Cc: Ivan Kokshaysky Cc: James Hogan Cc: Jens Axboe Cc: John Stultz Cc: Martin Schwidefsky Cc: Masami Hiramatsu Cc: Matt Turner Cc: Mike Frysinger Cc: Mike Travis Cc: Neil Brown Cc: Nicolas Pitre Cc: Paul Mackerras Cc: Paul Mundt Cc: Rafael J. Wysocki Cc: Ralf Baechle Cc: Richard Henderson Cc: Robert Richter Cc: Russell King Cc: Russell King Cc: Rusty Russell Cc: Steven Rostedt Cc: Thomas Gleixner Cc: Tony Luck Cc: Will Deacon Cc: Wim Van Sebroeck Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds