Commit 4b783176 authored by Linus Torvalds's avatar Linus Torvalds

Merge branch 'x86-pti-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

Pull STIBP fallout fixes from Thomas Gleixner:
 "The performance destruction department finally got it's act together
  and came up with a cure for the STIPB regression:

   - Provide a command line option to control the spectre v2 user space
     mitigations. Default is either seccomp or prctl (if seccomp is
     disabled in Kconfig). prctl allows mitigation opt-in, seccomp
     enables the migitation for sandboxed processes.

   - Rework the code to handle the conditional STIBP/IBPB control and
     remove the now unused ptrace_may_access_sched() optimization
     attempt

   - Disable STIBP automatically when SMT is disabled

   - Optimize the switch_to() logic to avoid MSR writes and invocations
     of __switch_to_xtra().

   - Make the asynchronous speculation TIF updates synchronous to
     prevent stale mitigation state.

  As a general cleanup this also makes retpoline directly depend on
  compiler support and removes the 'minimal retpoline' option which just
  pretended to provide some form of security while providing none"

* 'x86-pti-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (31 commits)
  x86/speculation: Provide IBPB always command line options
  x86/speculation: Add seccomp Spectre v2 user space protection mode
  x86/speculation: Enable prctl mode for spectre_v2_user
  x86/speculation: Add prctl() control for indirect branch speculation
  x86/speculation: Prepare arch_smt_update() for PRCTL mode
  x86/speculation: Prevent stale SPEC_CTRL msr content
  x86/speculation: Split out TIF update
  ptrace: Remove unused ptrace_may_access_sched() and MODE_IBRS
  x86/speculation: Prepare for conditional IBPB in switch_mm()
  x86/speculation: Avoid __switch_to_xtra() calls
  x86/process: Consolidate and simplify switch_to_xtra() code
  x86/speculation: Prepare for per task indirect branch speculation control
  x86/speculation: Add command line control for indirect branch speculation
  x86/speculation: Unify conditional spectre v2 print functions
  x86/speculataion: Mark command line parser data __initdata
  x86/speculation: Mark string arrays const correctly
  x86/speculation: Reorder the spec_v2 code
  x86/l1tf: Show actual SMT state
  x86/speculation: Rework SMT state change
  sched/smt: Expose sched_smt_present static key
  ...
parents 88058417 55a97402
...@@ -4199,9 +4199,13 @@ ...@@ -4199,9 +4199,13 @@
spectre_v2= [X86] Control mitigation of Spectre variant 2 spectre_v2= [X86] Control mitigation of Spectre variant 2
(indirect branch speculation) vulnerability. (indirect branch speculation) vulnerability.
The default operation protects the kernel from
user space attacks.
on - unconditionally enable on - unconditionally enable, implies
off - unconditionally disable spectre_v2_user=on
off - unconditionally disable, implies
spectre_v2_user=off
auto - kernel detects whether your CPU model is auto - kernel detects whether your CPU model is
vulnerable vulnerable
...@@ -4211,6 +4215,12 @@ ...@@ -4211,6 +4215,12 @@
CONFIG_RETPOLINE configuration option, and the CONFIG_RETPOLINE configuration option, and the
compiler with which the kernel was built. compiler with which the kernel was built.
Selecting 'on' will also enable the mitigation
against user space to user space task attacks.
Selecting 'off' will disable both the kernel and
the user space protections.
Specific mitigations can also be selected manually: Specific mitigations can also be selected manually:
retpoline - replace indirect branches retpoline - replace indirect branches
...@@ -4220,6 +4230,48 @@ ...@@ -4220,6 +4230,48 @@
Not specifying this option is equivalent to Not specifying this option is equivalent to
spectre_v2=auto. spectre_v2=auto.
spectre_v2_user=
[X86] Control mitigation of Spectre variant 2
(indirect branch speculation) vulnerability between
user space tasks
on - Unconditionally enable mitigations. Is
enforced by spectre_v2=on
off - Unconditionally disable mitigations. Is
enforced by spectre_v2=off
prctl - Indirect branch speculation is enabled,
but mitigation can be enabled via prctl
per thread. The mitigation control state
is inherited on fork.
prctl,ibpb
- Like "prctl" above, but only STIBP is
controlled per thread. IBPB is issued
always when switching between different user
space processes.
seccomp
- Same as "prctl" above, but all seccomp
threads will enable the mitigation unless
they explicitly opt out.
seccomp,ibpb
- Like "seccomp" above, but only STIBP is
controlled per thread. IBPB is issued
always when switching between different
user space processes.
auto - Kernel selects the mitigation depending on
the available CPU features and vulnerability.
Default mitigation:
If CONFIG_SECCOMP=y then "seccomp", otherwise "prctl"
Not specifying this option is equivalent to
spectre_v2_user=auto.
spec_store_bypass_disable= spec_store_bypass_disable=
[HW] Control Speculative Store Bypass (SSB) Disable mitigation [HW] Control Speculative Store Bypass (SSB) Disable mitigation
(Speculative Store Bypass vulnerability) (Speculative Store Bypass vulnerability)
......
...@@ -92,3 +92,12 @@ Speculation misfeature controls ...@@ -92,3 +92,12 @@ Speculation misfeature controls
* prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_STORE_BYPASS, PR_SPEC_ENABLE, 0, 0); * prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_STORE_BYPASS, PR_SPEC_ENABLE, 0, 0);
* prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_STORE_BYPASS, PR_SPEC_DISABLE, 0, 0); * prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_STORE_BYPASS, PR_SPEC_DISABLE, 0, 0);
* prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_STORE_BYPASS, PR_SPEC_FORCE_DISABLE, 0, 0); * prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_STORE_BYPASS, PR_SPEC_FORCE_DISABLE, 0, 0);
- PR_SPEC_INDIR_BRANCH: Indirect Branch Speculation in User Processes
(Mitigate Spectre V2 style attacks against user processes)
Invocations:
* prctl(PR_GET_SPECULATION_CTRL, PR_SPEC_INDIRECT_BRANCH, 0, 0, 0);
* prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_INDIRECT_BRANCH, PR_SPEC_ENABLE, 0, 0);
* prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_INDIRECT_BRANCH, PR_SPEC_DISABLE, 0, 0);
* prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_INDIRECT_BRANCH, PR_SPEC_FORCE_DISABLE, 0, 0);
...@@ -444,10 +444,6 @@ config RETPOLINE ...@@ -444,10 +444,6 @@ config RETPOLINE
branches. Requires a compiler with -mindirect-branch=thunk-extern branches. Requires a compiler with -mindirect-branch=thunk-extern
support for full protection. The kernel may run slower. support for full protection. The kernel may run slower.
Without compiler support, at least indirect branches in assembler
code are eliminated. Since this includes the syscall entry path,
it is not entirely pointless.
config INTEL_RDT config INTEL_RDT
bool "Intel Resource Director Technology support" bool "Intel Resource Director Technology support"
depends on X86 && CPU_SUP_INTEL depends on X86 && CPU_SUP_INTEL
...@@ -1004,13 +1000,7 @@ config NR_CPUS ...@@ -1004,13 +1000,7 @@ config NR_CPUS
to the kernel image. to the kernel image.
config SCHED_SMT config SCHED_SMT
bool "SMT (Hyperthreading) scheduler support" def_bool y if SMP
depends on SMP
---help---
SMT scheduler support improves the CPU scheduler's decision making
when dealing with Intel Pentium 4 chips with HyperThreading at a
cost of slightly increased overhead in some places. If unsure say
N here.
config SCHED_MC config SCHED_MC
def_bool y def_bool y
......
...@@ -220,9 +220,10 @@ KBUILD_CFLAGS += -fno-asynchronous-unwind-tables ...@@ -220,9 +220,10 @@ KBUILD_CFLAGS += -fno-asynchronous-unwind-tables
# Avoid indirect branches in kernel to deal with Spectre # Avoid indirect branches in kernel to deal with Spectre
ifdef CONFIG_RETPOLINE ifdef CONFIG_RETPOLINE
ifneq ($(RETPOLINE_CFLAGS),) ifeq ($(RETPOLINE_CFLAGS),)
KBUILD_CFLAGS += $(RETPOLINE_CFLAGS) -DRETPOLINE $(error You are building kernel with non-retpoline compiler, please update your compiler.)
endif endif
KBUILD_CFLAGS += $(RETPOLINE_CFLAGS)
endif endif
archscripts: scripts_basic archscripts: scripts_basic
......
...@@ -41,9 +41,10 @@ ...@@ -41,9 +41,10 @@
#define MSR_IA32_SPEC_CTRL 0x00000048 /* Speculation Control */ #define MSR_IA32_SPEC_CTRL 0x00000048 /* Speculation Control */
#define SPEC_CTRL_IBRS (1 << 0) /* Indirect Branch Restricted Speculation */ #define SPEC_CTRL_IBRS (1 << 0) /* Indirect Branch Restricted Speculation */
#define SPEC_CTRL_STIBP (1 << 1) /* Single Thread Indirect Branch Predictors */ #define SPEC_CTRL_STIBP_SHIFT 1 /* Single Thread Indirect Branch Predictor (STIBP) bit */
#define SPEC_CTRL_STIBP (1 << SPEC_CTRL_STIBP_SHIFT) /* STIBP mask */
#define SPEC_CTRL_SSBD_SHIFT 2 /* Speculative Store Bypass Disable bit */ #define SPEC_CTRL_SSBD_SHIFT 2 /* Speculative Store Bypass Disable bit */
#define SPEC_CTRL_SSBD (1 << SPEC_CTRL_SSBD_SHIFT) /* Speculative Store Bypass Disable */ #define SPEC_CTRL_SSBD (1 << SPEC_CTRL_SSBD_SHIFT) /* Speculative Store Bypass Disable */
#define MSR_IA32_PRED_CMD 0x00000049 /* Prediction Command */ #define MSR_IA32_PRED_CMD 0x00000049 /* Prediction Command */
#define PRED_CMD_IBPB (1 << 0) /* Indirect Branch Prediction Barrier */ #define PRED_CMD_IBPB (1 << 0) /* Indirect Branch Prediction Barrier */
......
...@@ -3,6 +3,8 @@ ...@@ -3,6 +3,8 @@
#ifndef _ASM_X86_NOSPEC_BRANCH_H_ #ifndef _ASM_X86_NOSPEC_BRANCH_H_
#define _ASM_X86_NOSPEC_BRANCH_H_ #define _ASM_X86_NOSPEC_BRANCH_H_
#include <linux/static_key.h>
#include <asm/alternative.h> #include <asm/alternative.h>
#include <asm/alternative-asm.h> #include <asm/alternative-asm.h>
#include <asm/cpufeatures.h> #include <asm/cpufeatures.h>
...@@ -162,11 +164,12 @@ ...@@ -162,11 +164,12 @@
_ASM_PTR " 999b\n\t" \ _ASM_PTR " 999b\n\t" \
".popsection\n\t" ".popsection\n\t"
#if defined(CONFIG_X86_64) && defined(RETPOLINE) #ifdef CONFIG_RETPOLINE
#ifdef CONFIG_X86_64
/* /*
* Since the inline asm uses the %V modifier which is only in newer GCC, * Inline asm uses the %V modifier which is only in newer GCC
* the 64-bit one is dependent on RETPOLINE not CONFIG_RETPOLINE. * which is ensured when CONFIG_RETPOLINE is defined.
*/ */
# define CALL_NOSPEC \ # define CALL_NOSPEC \
ANNOTATE_NOSPEC_ALTERNATIVE \ ANNOTATE_NOSPEC_ALTERNATIVE \
...@@ -181,7 +184,7 @@ ...@@ -181,7 +184,7 @@
X86_FEATURE_RETPOLINE_AMD) X86_FEATURE_RETPOLINE_AMD)
# define THUNK_TARGET(addr) [thunk_target] "r" (addr) # define THUNK_TARGET(addr) [thunk_target] "r" (addr)
#elif defined(CONFIG_X86_32) && defined(CONFIG_RETPOLINE) #else /* CONFIG_X86_32 */
/* /*
* For i386 we use the original ret-equivalent retpoline, because * For i386 we use the original ret-equivalent retpoline, because
* otherwise we'll run out of registers. We don't care about CET * otherwise we'll run out of registers. We don't care about CET
...@@ -211,6 +214,7 @@ ...@@ -211,6 +214,7 @@
X86_FEATURE_RETPOLINE_AMD) X86_FEATURE_RETPOLINE_AMD)
# define THUNK_TARGET(addr) [thunk_target] "rm" (addr) # define THUNK_TARGET(addr) [thunk_target] "rm" (addr)
#endif
#else /* No retpoline for C / inline asm */ #else /* No retpoline for C / inline asm */
# define CALL_NOSPEC "call *%[thunk_target]\n" # define CALL_NOSPEC "call *%[thunk_target]\n"
# define THUNK_TARGET(addr) [thunk_target] "rm" (addr) # define THUNK_TARGET(addr) [thunk_target] "rm" (addr)
...@@ -219,13 +223,19 @@ ...@@ -219,13 +223,19 @@
/* The Spectre V2 mitigation variants */ /* The Spectre V2 mitigation variants */
enum spectre_v2_mitigation { enum spectre_v2_mitigation {
SPECTRE_V2_NONE, SPECTRE_V2_NONE,
SPECTRE_V2_RETPOLINE_MINIMAL,
SPECTRE_V2_RETPOLINE_MINIMAL_AMD,
SPECTRE_V2_RETPOLINE_GENERIC, SPECTRE_V2_RETPOLINE_GENERIC,
SPECTRE_V2_RETPOLINE_AMD, SPECTRE_V2_RETPOLINE_AMD,
SPECTRE_V2_IBRS_ENHANCED, SPECTRE_V2_IBRS_ENHANCED,
}; };
/* The indirect branch speculation control variants */
enum spectre_v2_user_mitigation {
SPECTRE_V2_USER_NONE,
SPECTRE_V2_USER_STRICT,
SPECTRE_V2_USER_PRCTL,
SPECTRE_V2_USER_SECCOMP,
};
/* The Speculative Store Bypass disable variants */ /* The Speculative Store Bypass disable variants */
enum ssb_mitigation { enum ssb_mitigation {
SPEC_STORE_BYPASS_NONE, SPEC_STORE_BYPASS_NONE,
...@@ -303,6 +313,10 @@ do { \ ...@@ -303,6 +313,10 @@ do { \
preempt_enable(); \ preempt_enable(); \
} while (0) } while (0)
DECLARE_STATIC_KEY_FALSE(switch_to_cond_stibp);
DECLARE_STATIC_KEY_FALSE(switch_mm_cond_ibpb);
DECLARE_STATIC_KEY_FALSE(switch_mm_always_ibpb);
#endif /* __ASSEMBLY__ */ #endif /* __ASSEMBLY__ */
/* /*
......
...@@ -53,12 +53,24 @@ static inline u64 ssbd_tif_to_spec_ctrl(u64 tifn) ...@@ -53,12 +53,24 @@ static inline u64 ssbd_tif_to_spec_ctrl(u64 tifn)
return (tifn & _TIF_SSBD) >> (TIF_SSBD - SPEC_CTRL_SSBD_SHIFT); return (tifn & _TIF_SSBD) >> (TIF_SSBD - SPEC_CTRL_SSBD_SHIFT);
} }
static inline u64 stibp_tif_to_spec_ctrl(u64 tifn)
{
BUILD_BUG_ON(TIF_SPEC_IB < SPEC_CTRL_STIBP_SHIFT);
return (tifn & _TIF_SPEC_IB) >> (TIF_SPEC_IB - SPEC_CTRL_STIBP_SHIFT);
}
static inline unsigned long ssbd_spec_ctrl_to_tif(u64 spec_ctrl) static inline unsigned long ssbd_spec_ctrl_to_tif(u64 spec_ctrl)
{ {
BUILD_BUG_ON(TIF_SSBD < SPEC_CTRL_SSBD_SHIFT); BUILD_BUG_ON(TIF_SSBD < SPEC_CTRL_SSBD_SHIFT);
return (spec_ctrl & SPEC_CTRL_SSBD) << (TIF_SSBD - SPEC_CTRL_SSBD_SHIFT); return (spec_ctrl & SPEC_CTRL_SSBD) << (TIF_SSBD - SPEC_CTRL_SSBD_SHIFT);
} }
static inline unsigned long stibp_spec_ctrl_to_tif(u64 spec_ctrl)
{
BUILD_BUG_ON(TIF_SPEC_IB < SPEC_CTRL_STIBP_SHIFT);
return (spec_ctrl & SPEC_CTRL_STIBP) << (TIF_SPEC_IB - SPEC_CTRL_STIBP_SHIFT);
}
static inline u64 ssbd_tif_to_amd_ls_cfg(u64 tifn) static inline u64 ssbd_tif_to_amd_ls_cfg(u64 tifn)
{ {
return (tifn & _TIF_SSBD) ? x86_amd_ls_cfg_ssbd_mask : 0ULL; return (tifn & _TIF_SSBD) ? x86_amd_ls_cfg_ssbd_mask : 0ULL;
...@@ -70,11 +82,7 @@ extern void speculative_store_bypass_ht_init(void); ...@@ -70,11 +82,7 @@ extern void speculative_store_bypass_ht_init(void);
static inline void speculative_store_bypass_ht_init(void) { } static inline void speculative_store_bypass_ht_init(void) { }
#endif #endif
extern void speculative_store_bypass_update(unsigned long tif); extern void speculation_ctrl_update(unsigned long tif);
extern void speculation_ctrl_update_current(void);
static inline void speculative_store_bypass_update_current(void)
{
speculative_store_bypass_update(current_thread_info()->flags);
}
#endif #endif
...@@ -11,9 +11,6 @@ struct task_struct *__switch_to_asm(struct task_struct *prev, ...@@ -11,9 +11,6 @@ struct task_struct *__switch_to_asm(struct task_struct *prev,
__visible struct task_struct *__switch_to(struct task_struct *prev, __visible struct task_struct *__switch_to(struct task_struct *prev,
struct task_struct *next); struct task_struct *next);
struct tss_struct;
void __switch_to_xtra(struct task_struct *prev_p, struct task_struct *next_p,
struct tss_struct *tss);
/* This runs runs on the previous thread's stack. */ /* This runs runs on the previous thread's stack. */
static inline void prepare_switch_to(struct task_struct *next) static inline void prepare_switch_to(struct task_struct *next)
......
...@@ -79,10 +79,12 @@ struct thread_info { ...@@ -79,10 +79,12 @@ struct thread_info {
#define TIF_SIGPENDING 2 /* signal pending */ #define TIF_SIGPENDING 2 /* signal pending */
#define TIF_NEED_RESCHED 3 /* rescheduling necessary */ #define TIF_NEED_RESCHED 3 /* rescheduling necessary */
#define TIF_SINGLESTEP 4 /* reenable singlestep on user return*/ #define TIF_SINGLESTEP 4 /* reenable singlestep on user return*/
#define TIF_SSBD 5 /* Reduced data speculation */ #define TIF_SSBD 5 /* Speculative store bypass disable */
#define TIF_SYSCALL_EMU 6 /* syscall emulation active */ #define TIF_SYSCALL_EMU 6 /* syscall emulation active */
#define TIF_SYSCALL_AUDIT 7 /* syscall auditing active */ #define TIF_SYSCALL_AUDIT 7 /* syscall auditing active */
#define TIF_SECCOMP 8 /* secure computing */ #define TIF_SECCOMP 8 /* secure computing */
#define TIF_SPEC_IB 9 /* Indirect branch speculation mitigation */
#define TIF_SPEC_FORCE_UPDATE 10 /* Force speculation MSR update in context switch */
#define TIF_USER_RETURN_NOTIFY 11 /* notify kernel of userspace return */ #define TIF_USER_RETURN_NOTIFY 11 /* notify kernel of userspace return */
#define TIF_UPROBE 12 /* breakpointed or singlestepping */ #define TIF_UPROBE 12 /* breakpointed or singlestepping */
#define TIF_PATCH_PENDING 13 /* pending live patching update */ #define TIF_PATCH_PENDING 13 /* pending live patching update */
...@@ -110,6 +112,8 @@ struct thread_info { ...@@ -110,6 +112,8 @@ struct thread_info {
#define _TIF_SYSCALL_EMU (1 << TIF_SYSCALL_EMU) #define _TIF_SYSCALL_EMU (1 << TIF_SYSCALL_EMU)
#define _TIF_SYSCALL_AUDIT (1 << TIF_SYSCALL_AUDIT) #define _TIF_SYSCALL_AUDIT (1 << TIF_SYSCALL_AUDIT)
#define _TIF_SECCOMP (1 << TIF_SECCOMP) #define _TIF_SECCOMP (1 << TIF_SECCOMP)
#define _TIF_SPEC_IB (1 << TIF_SPEC_IB)
#define _TIF_SPEC_FORCE_UPDATE (1 << TIF_SPEC_FORCE_UPDATE)
#define _TIF_USER_RETURN_NOTIFY (1 << TIF_USER_RETURN_NOTIFY) #define _TIF_USER_RETURN_NOTIFY (1 << TIF_USER_RETURN_NOTIFY)
#define _TIF_UPROBE (1 << TIF_UPROBE) #define _TIF_UPROBE (1 << TIF_UPROBE)
#define _TIF_PATCH_PENDING (1 << TIF_PATCH_PENDING) #define _TIF_PATCH_PENDING (1 << TIF_PATCH_PENDING)
...@@ -145,8 +149,18 @@ struct thread_info { ...@@ -145,8 +149,18 @@ struct thread_info {
_TIF_FSCHECK) _TIF_FSCHECK)
/* flags to check in __switch_to() */ /* flags to check in __switch_to() */
#define _TIF_WORK_CTXSW \ #define _TIF_WORK_CTXSW_BASE \
(_TIF_IO_BITMAP|_TIF_NOCPUID|_TIF_NOTSC|_TIF_BLOCKSTEP|_TIF_SSBD) (_TIF_IO_BITMAP|_TIF_NOCPUID|_TIF_NOTSC|_TIF_BLOCKSTEP| \
_TIF_SSBD | _TIF_SPEC_FORCE_UPDATE)
/*
* Avoid calls to __switch_to_xtra() on UP as STIBP is not evaluated.
*/
#ifdef CONFIG_SMP
# define _TIF_WORK_CTXSW (_TIF_WORK_CTXSW_BASE | _TIF_SPEC_IB)
#else
# define _TIF_WORK_CTXSW (_TIF_WORK_CTXSW_BASE)
#endif
#define _TIF_WORK_CTXSW_PREV (_TIF_WORK_CTXSW|_TIF_USER_RETURN_NOTIFY) #define _TIF_WORK_CTXSW_PREV (_TIF_WORK_CTXSW|_TIF_USER_RETURN_NOTIFY)
#define _TIF_WORK_CTXSW_NEXT (_TIF_WORK_CTXSW) #define _TIF_WORK_CTXSW_NEXT (_TIF_WORK_CTXSW)
......
...@@ -169,10 +169,14 @@ struct tlb_state { ...@@ -169,10 +169,14 @@ struct tlb_state {
#define LOADED_MM_SWITCHING ((struct mm_struct *)1) #define LOADED_MM_SWITCHING ((struct mm_struct *)1)
/* Last user mm for optimizing IBPB */
union {
struct mm_struct *last_user_mm;
unsigned long last_user_mm_ibpb;
};
u16 loaded_mm_asid; u16 loaded_mm_asid;
u16 next_asid; u16 next_asid;
/* last user mm's ctx id */
u64 last_ctx_id;
/* /*
* We can be in one of several states: * We can be in one of several states:
......
This diff is collapsed.
...@@ -40,6 +40,8 @@ ...@@ -40,6 +40,8 @@
#include <asm/prctl.h> #include <asm/prctl.h>
#include <asm/spec-ctrl.h> #include <asm/spec-ctrl.h>
#include "process.h"
/* /*
* per-CPU TSS segments. Threads are completely 'soft' on Linux, * per-CPU TSS segments. Threads are completely 'soft' on Linux,
* no more per-task TSS's. The TSS size is kept cacheline-aligned * no more per-task TSS's. The TSS size is kept cacheline-aligned
...@@ -252,11 +254,12 @@ void arch_setup_new_exec(void) ...@@ -252,11 +254,12 @@ void arch_setup_new_exec(void)
enable_cpuid(); enable_cpuid();
} }
static inline void switch_to_bitmap(struct tss_struct *tss, static inline void switch_to_bitmap(struct thread_struct *prev,
struct thread_struct *prev,
struct thread_struct *next, struct thread_struct *next,
unsigned long tifp, unsigned long tifn) unsigned long tifp, unsigned long tifn)
{ {
struct tss_struct *tss = this_cpu_ptr(&cpu_tss_rw);
if (tifn & _TIF_IO_BITMAP) { if (tifn & _TIF_IO_BITMAP) {
/* /*
* Copy the relevant range of the IO bitmap. * Copy the relevant range of the IO bitmap.
...@@ -395,32 +398,85 @@ static __always_inline void amd_set_ssb_virt_state(unsigned long tifn) ...@@ -395,32 +398,85 @@ static __always_inline void amd_set_ssb_virt_state(unsigned long tifn)
wrmsrl(MSR_AMD64_VIRT_SPEC_CTRL, ssbd_tif_to_spec_ctrl(tifn)); wrmsrl(MSR_AMD64_VIRT_SPEC_CTRL, ssbd_tif_to_spec_ctrl(tifn));
} }
static __always_inline void intel_set_ssb_state(unsigned long tifn) /*
* Update the MSRs managing speculation control, during context switch.
*
* tifp: Previous task's thread flags
* tifn: Next task's thread flags
*/
static __always_inline void __speculation_ctrl_update(unsigned long tifp,
unsigned long tifn)
{ {
u64 msr = x86_spec_ctrl_base | ssbd_tif_to_spec_ctrl(tifn); unsigned long tif_diff = tifp ^ tifn;
u64 msr = x86_spec_ctrl_base;
bool updmsr = false;
/*
* If TIF_SSBD is different, select the proper mitigation
* method. Note that if SSBD mitigation is disabled or permanentely
* enabled this branch can't be taken because nothing can set
* TIF_SSBD.
*/
if (tif_diff & _TIF_SSBD) {
if (static_cpu_has(X86_FEATURE_VIRT_SSBD)) {
amd_set_ssb_virt_state(tifn);
} else if (static_cpu_has(X86_FEATURE_LS_CFG_SSBD)) {
amd_set_core_ssb_state(tifn);
} else if (static_cpu_has(X86_FEATURE_SPEC_CTRL_SSBD) ||
static_cpu_has(X86_FEATURE_AMD_SSBD)) {
msr |= ssbd_tif_to_spec_ctrl(tifn);
updmsr = true;
}
}
/*
* Only evaluate TIF_SPEC_IB if conditional STIBP is enabled,
* otherwise avoid the MSR write.
*/
if (IS_ENABLED(CONFIG_SMP) &&
static_branch_unlikely(&switch_to_cond_stibp)) {
updmsr |= !!(tif_diff & _TIF_SPEC_IB);
msr |= stibp_tif_to_spec_ctrl(tifn);
}
wrmsrl(MSR_IA32_SPEC_CTRL, msr); if (updmsr)
wrmsrl(MSR_IA32_SPEC_CTRL, msr);
} }
static __always_inline void __speculative_store_bypass_update(unsigned long tifn) static unsigned long speculation_ctrl_update_tif(struct task_struct *tsk)
{ {
if (static_cpu_has(X86_FEATURE_VIRT_SSBD)) if (test_and_clear_tsk_thread_flag(tsk, TIF_SPEC_FORCE_UPDATE)) {
amd_set_ssb_virt_state(tifn); if (task_spec_ssb_disable(tsk))
else if (static_cpu_has(X86_FEATURE_LS_CFG_SSBD)) set_tsk_thread_flag(tsk, TIF_SSBD);
amd_set_core_ssb_state(tifn); else
else clear_tsk_thread_flag(tsk, TIF_SSBD);
intel_set_ssb_state(tifn);
if (task_spec_ib_disable(tsk))
set_tsk_thread_flag(tsk, TIF_SPEC_IB);
else
clear_tsk_thread_flag(tsk, TIF_SPEC_IB);
}
/* Return the updated threadinfo flags*/
return task_thread_info(tsk)->flags;
} }
void speculative_store_bypass_update(unsigned long tif) void speculation_ctrl_update(unsigned long tif)
{ {
/* Forced update. Make sure all relevant TIF flags are different */
preempt_disable(); preempt_disable();
__speculative_store_bypass_update(tif); __speculation_ctrl_update(~tif, tif);
preempt_enable(); preempt_enable();
} }
void __switch_to_xtra(struct task_struct *prev_p, struct task_struct *next_p, /* Called from seccomp/prctl update */
struct tss_struct *tss) void speculation_ctrl_update_current(void)
{
preempt_disable();
speculation_ctrl_update(speculation_ctrl_update_tif(current));
preempt_enable();
}
void __switch_to_xtra(struct task_struct *prev_p, struct task_struct *next_p)
{ {
struct thread_struct *prev, *next; struct thread_struct *prev, *next;
unsigned long tifp, tifn; unsigned long tifp, tifn;
...@@ -430,7 +486,7 @@ void __switch_to_xtra(struct task_struct *prev_p, struct task_struct *next_p, ...@@ -430,7 +486,7 @@ void __switch_to_xtra(struct task_struct *prev_p, struct task_struct *next_p,
tifn = READ_ONCE(task_thread_info(next_p)->flags); tifn = READ_ONCE(task_thread_info(next_p)->flags);
tifp = READ_ONCE(task_thread_info(prev_p)->flags); tifp = READ_ONCE(task_thread_info(prev_p)->flags);
switch_to_bitmap(tss, prev, next, tifp, tifn); switch_to_bitmap(prev, next, tifp, tifn);
propagate_user_return_notify(prev_p, next_p); propagate_user_return_notify(prev_p, next_p);
...@@ -451,8 +507,15 @@ void __switch_to_xtra(struct task_struct *prev_p, struct task_struct *next_p, ...@@ -451,8 +507,15 @@ void __switch_to_xtra(struct task_struct *prev_p, struct task_struct *next_p,
if ((tifp ^ tifn) & _TIF_NOCPUID) if ((tifp ^ tifn) & _TIF_NOCPUID)
set_cpuid_faulting(!!(tifn & _TIF_NOCPUID)); set_cpuid_faulting(!!(tifn & _TIF_NOCPUID));
if ((tifp ^ tifn) & _TIF_SSBD) if (likely(!((tifp | tifn) & _TIF_SPEC_FORCE_UPDATE))) {
__speculative_store_bypass_update(tifn); __speculation_ctrl_update(tifp, tifn);
} else {
speculation_ctrl_update_tif(prev_p);
tifn = speculation_ctrl_update_tif(next_p);
/* Enforce MSR update to ensure consistent state */
__speculation_ctrl_update(~tifn, tifn);
}
} }
/* /*
......
// SPDX-License-Identifier: GPL-2.0
//
// Code shared between 32 and 64 bit
#include <asm/spec-ctrl.h>
void __switch_to_xtra(struct task_struct *prev_p, struct task_struct *next_p);
/*
* This needs to be inline to optimize for the common case where no extra
* work needs to be done.
*/
static inline void switch_to_extra(struct task_struct *prev,
struct task_struct *next)
{
unsigned long next_tif = task_thread_info(next)->flags;
unsigned long prev_tif = task_thread_info(prev)->flags;
if (IS_ENABLED(CONFIG_SMP)) {
/*
* Avoid __switch_to_xtra() invocation when conditional
* STIPB is disabled and the only different bit is