Skip to content
Snippets Groups Projects
  1. Oct 28, 2022
  2. Oct 03, 2022
    • Jiebin Sun's avatar
      ipc/msg: mitigate the lock contention with percpu counter · 72d1e611
      Jiebin Sun authored
      The msg_bytes and msg_hdrs atomic counters are frequently updated when IPC
      msg queue is in heavy use, causing heavy cache bounce and overhead. 
      Change them to percpu_counter greatly improve the performance.  Since
      there is one percpu struct per namespace, additional memory cost is
      minimal.  Reading of the count done in msgctl call, which is infrequent. 
      So the need to sum up the counts in each CPU is infrequent.
      
      Apply the patch and test the pts/stress-ng-1.4.0
      -- system v message passing (160 threads).
      
      Score gain: 3.99x
      
      CPU: ICX 8380 x 2 sockets
      Core number: 40 x 2 physical cores
      Benchmark: pts/stress-ng-1.4.0
      -- system v message passing (160 threads)
      
      [akpm@linux-foundation.org: coding-style cleanups]
      [jiebin.sun@intel.com: avoid negative value by overflow in msginfo]
        Link: https://lkml.kernel.org/r/20220920150809.4014944-1-jiebin.sun@intel.com
      [akpm@linux-foundation.org: fix min() warnings]
      Link: https://lkml.kernel.org/r/20220913192538.3023708-3-jiebin.sun@intel.com
      
      
      Signed-off-by: default avatarJiebin Sun <jiebin.sun@intel.com>
      Reviewed-by: default avatarTim Chen <tim.c.chen@linux.intel.com>
      Cc: Alexander Mikhalitsyn <alexander.mikhalitsyn@virtuozzo.com>
      Cc: Alexey Gladkov <legion@kernel.org>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: Davidlohr Bueso <dave@stgolabs.net>
      Cc: Dennis Zhou <dennis@kernel.org>
      Cc: "Eric W . Biederman" <ebiederm@xmission.com>
      Cc: Manfred Spraul <manfred@colorfullife.com>
      Cc: Shakeel Butt <shakeelb@google.com>
      Cc: Tejun Heo <tj@kernel.org>
      Cc: Vasily Averin <vasily.averin@linux.dev>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      72d1e611
  3. Sep 03, 2021
    • Vasily Averin's avatar
      memcg: enable accounting of ipc resources · 18319498
      Vasily Averin authored
      When user creates IPC objects it forces kernel to allocate memory for
      these long-living objects.
      
      It makes sense to account them to restrict the host's memory consumption
      from inside the memcg-limited container.
      
      This patch enables accounting for IPC shared memory segments, messages
      semaphores and semaphore's undo lists.
      
      Link: https://lkml.kernel.org/r/d6507b06-4df6-78f8-6c54-3ae86e3b5339@virtuozzo.com
      
      
      Signed-off-by: default avatarVasily Averin <vvs@virtuozzo.com>
      Reviewed-by: default avatarShakeel Butt <shakeelb@google.com>
      Cc: Alexander Viro <viro@zeniv.linux.org.uk>
      Cc: Alexey Dobriyan <adobriyan@gmail.com>
      Cc: Andrei Vagin <avagin@gmail.com>
      Cc: Borislav Petkov <bp@alien8.de>
      Cc: Borislav Petkov <bp@suse.de>
      Cc: Christian Brauner <christian.brauner@ubuntu.com>
      Cc: Dmitry Safonov <0x7f454c46@gmail.com>
      Cc: "Eric W. Biederman" <ebiederm@xmission.com>
      Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
      Cc: "H. Peter Anvin" <hpa@zytor.com>
      Cc: Ingo Molnar <mingo@redhat.com>
      Cc: "J. Bruce Fields" <bfields@fieldses.org>
      Cc: Jeff Layton <jlayton@kernel.org>
      Cc: Jens Axboe <axboe@kernel.dk>
      Cc: Jiri Slaby <jirislaby@kernel.org>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Cc: Kirill Tkhai <ktkhai@virtuozzo.com>
      Cc: Michal Hocko <mhocko@kernel.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Roman Gushchin <guro@fb.com>
      Cc: Serge Hallyn <serge@hallyn.com>
      Cc: Tejun Heo <tj@kernel.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Vladimir Davydov <vdavydov.dev@gmail.com>
      Cc: Yutian Yang <nglaive@gmail.com>
      Cc: Zefan Li <lizefan.x@bytedance.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      18319498
  4. Jul 01, 2021
  5. May 23, 2021
    • Varad Gautam's avatar
      ipc/mqueue, msg, sem: avoid relying on a stack reference past its expiry · a11ddb37
      Varad Gautam authored
      do_mq_timedreceive calls wq_sleep with a stack local address.  The
      sender (do_mq_timedsend) uses this address to later call pipelined_send.
      
      This leads to a very hard to trigger race where a do_mq_timedreceive
      call might return and leave do_mq_timedsend to rely on an invalid
      address, causing the following crash:
      
        RIP: 0010:wake_q_add_safe+0x13/0x60
        Call Trace:
         __x64_sys_mq_timedsend+0x2a9/0x490
         do_syscall_64+0x80/0x680
         entry_SYSCALL_64_after_hwframe+0x44/0xa9
        RIP: 0033:0x7f5928e40343
      
      The race occurs as:
      
      1. do_mq_timedreceive calls wq_sleep with the address of `struct
         ext_wait_queue` on function stack (aliased as `ewq_addr` here) - it
         holds a valid `struct ext_wait_queue *` as long as the stack has not
         been overwritten.
      
      2. `ewq_addr` gets added to info->e_wait_q[RECV].list in wq_add, and
         do_mq_timedsend receives it via wq_get_first_waiter(info, RECV) to call
         __pipelined_op.
      
      3. Sender calls __pipelined_op::smp_store_release(&this->state,
         STATE_READY).  Here is where the race window begins.  (`this` is
         `ewq_addr`.)
      
      4. If the receiver wakes up now in do_mq_timedreceive::wq_sleep, it
         will see `state == STATE_READY` and break.
      
      5. do_mq_timedreceive returns, and `ewq_addr` is no longer guaranteed
         to be a `struct ext_wait_queue *` since it was on do_mq_timedreceive's
         stack.  (Although the address may not get overwritten until another
         function happens to touch it, which means it can persist around for an
         indefinite time.)
      
      6. do_mq_timedsend::__pipelined_op() still believes `ewq_addr` is a
         `struct ext_wait_queue *`, and uses it to find a task_struct to pass to
         the wake_q_add_safe call.  In the lucky case where nothing has
         overwritten `ewq_addr` yet, `ewq_addr->task` is the right task_struct.
         In the unlucky case, __pipelined_op::wake_q_add_safe gets handed a
         bogus address as the receiver's task_struct causing the crash.
      
      do_mq_timedsend::__pipelined_op() should not dereference `this` after
      setting STATE_READY, as the receiver counterpart is now free to return.
      Change __pipelined_op to call wake_q_add_safe on the receiver's
      task_struct returned by get_task_struct, instead of dereferencing `this`
      which sits on the receiver's stack.
      
      As Manfred pointed out, the race potentially also exists in
      ipc/msg.c::expunge_all and ipc/sem.c::wake_up_sem_queue_prepare.  Fix
      those in the same way.
      
      Link: https://lkml.kernel.org/r/20210510102950.12551-1-varad.gautam@suse.com
      
      
      Fixes: c5b2cbdb ("ipc/mqueue.c: update/document memory barriers")
      Fixes: 8116b54e ("ipc/sem.c: document and update memory barriers")
      Fixes: 0d97a82b ("ipc/msg.c: update and document memory barriers")
      Signed-off-by: default avatarVarad Gautam <varad.gautam@suse.com>
      Reported-by: default avatarMatthias von Faber <matthias.vonfaber@aox-tech.de>
      Acked-by: default avatarDavidlohr Bueso <dbueso@suse.de>
      Acked-by: default avatarManfred Spraul <manfred@colorfullife.com>
      Cc: Christian Brauner <christian.brauner@ubuntu.com>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: "Eric W. Biederman" <ebiederm@xmission.com>
      Cc: <stable@vger.kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      a11ddb37
  6. Jun 08, 2020
  7. Feb 04, 2020
    • Lu Shuaibing's avatar
      ipc/msg.c: consolidate all xxxctl_down() functions · 889b3317
      Lu Shuaibing authored
      A use of uninitialized memory in msgctl_down() because msqid64 in
      ksys_msgctl hasn't been initialized.  The local | msqid64 | is created in
      ksys_msgctl() and then passed into msgctl_down().  Along the way msqid64
      is never initialized before msgctl_down() checks msqid64->msg_qbytes.
      
      KUMSAN(KernelUninitializedMemorySantizer, a new error detection tool)
      reports:
      
      ==================================================================
      BUG: KUMSAN: use of uninitialized memory in msgctl_down+0x94/0x300
      Read of size 8 at addr ffff88806bb97eb8 by task syz-executor707/2022
      
      CPU: 0 PID: 2022 Comm: syz-executor707 Not tainted 5.2.0-rc4+ #63
      Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
      Call Trace:
       dump_stack+0x75/0xae
       __kumsan_report+0x17c/0x3e6
       kumsan_report+0xe/0x20
       msgctl_down+0x94/0x300
       ksys_msgctl.constprop.14+0xef/0x260
       do_syscall_64+0x7e/0x1f0
       entry_SYSCALL_64_after_hwframe+0x44/0xa9
      RIP: 0033:0x4400e9
      Code: 18 89 d0 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 fb 13 fc ff c3 66 2e 0f 1f 84 00 00 00 00
      RSP: 002b:00007ffd869e0598 EFLAGS: 00000246 ORIG_RAX: 0000000000000047
      RAX: ffffffffffffffda RBX: 00000000004002c8 RCX: 00000000004400e9
      RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
      RBP: 00000000006ca018 R08: 0000000000000000 R09: 0000000000000000
      R10: 00000000ffffffff R11: 0000000000000246 R12: 0000000000401970
      R13: 0000000000401a00 R14: 0000000000000000 R15: 0000000000000000
      
      The buggy address belongs to the page:
      page:ffffea0001aee5c0 refcount:0 mapcount:0 mapping:0000000000000000 index:0x0
      flags: 0x100000000000000()
      raw: 0100000000000000 0000000000000000 ffffffff01ae0101 0000000000000000
      raw: 0000000000000000 0000000000000000 00000000ffffffff 0000000000000000
      page dumped because: kumsan: bad access detected
      ==================================================================
      
      Syzkaller reproducer:
      msgctl$IPC_RMID(0x0, 0x0)
      
      C reproducer:
      // autogenerated by syzkaller (https://github.com/google/syzkaller)
      
      int main(void)
      {
        syscall(__NR_mmap, 0x20000000, 0x1000000, 3, 0x32, -1, 0);
        syscall(__NR_msgctl, 0, 0, 0);
        return 0;
      }
      
      [natechancellor@gmail.com: adjust indentation in ksys_msgctl]
        Link: https://github.com/ClangBuiltLinux/linux/issues/829
        Link: http://lkml.kernel.org/r/20191218032932.37479-1-natechancellor@gmail.com
      Link: http://lkml.kernel.org/r/20190613014044.24234-1-shuaibinglu@126.com
      
      
      Signed-off-by: default avatarLu Shuaibing <shuaibinglu@126.com>
      Signed-off-by: default avatarNathan Chancellor <natechancellor@gmail.com>
      Suggested-by: default avatarArnd Bergmann <arnd@arndb.de>
      Cc: Davidlohr Bueso <dave@stgolabs.net>
      Cc: Manfred Spraul <manfred@colorfullife.com>
      Cc: NeilBrown <neilb@suse.com>
      From: Andrew Morton <akpm@linux-foundation.org>
      Subject: drivers/block/null_blk_main.c: fix layout
      
      Each line here overflows 80 cols by exactly one character.  Delete one tab
      per line to fix.
      
      Cc: Shaohua Li <shli@fb.com>
      Cc: Jens Axboe <axboe@kernel.dk>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      889b3317
    • Manfred Spraul's avatar
      ipc/msg.c: update and document memory barriers · 0d97a82b
      Manfred Spraul authored
      Transfer findings from ipc/mqueue.c:
      
      - A control barrier was missing for the lockless receive case So in
        theory, not yet initialized data may have been copied to user space -
        obviously only for architectures where control barriers are not NOP.
      
      - use smp_store_release().  In theory, the refount may have been
        decreased to 0 already when wake_q_add() tries to get a reference.
      
      Link: http://lkml.kernel.org/r/20191020123305.14715-5-manfred@colorfullife.com
      
      
      Signed-off-by: default avatarManfred Spraul <manfred@colorfullife.com>
      Cc: Waiman Long <longman@redhat.com>
      Cc: Davidlohr Bueso <dave@stgolabs.net>
      Cc: <1vier1@web.de>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Will Deacon <will.deacon@arm.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      0d97a82b
  8. Jan 25, 2019
    • Arnd Bergmann's avatar
      ipc: rename old-style shmctl/semctl/msgctl syscalls · 275f2214
      Arnd Bergmann authored
      
      The behavior of these system calls is slightly different between
      architectures, as determined by the CONFIG_ARCH_WANT_IPC_PARSE_VERSION
      symbol. Most architectures that implement the split IPC syscalls don't set
      that symbol and only get the modern version, but alpha, arm, microblaze,
      mips-n32, mips-n64 and xtensa expect the caller to pass the IPC_64 flag.
      
      For the architectures that so far only implement sys_ipc(), i.e. m68k,
      mips-o32, powerpc, s390, sh, sparc, and x86-32, we want the new behavior
      when adding the split syscalls, so we need to distinguish between the
      two groups of architectures.
      
      The method I picked for this distinction is to have a separate system call
      entry point: sys_old_*ctl() now uses ipc_parse_version, while sys_*ctl()
      does not. The system call tables of the five architectures are changed
      accordingly.
      
      As an additional benefit, we no longer need the configuration specific
      definition for ipc_parse_version(), it always does the same thing now,
      but simply won't get called on architectures with the modern interface.
      
      A small downside is that on architectures that do set
      ARCH_WANT_IPC_PARSE_VERSION, we now have an extra set of entry points
      that are never called. They only add a few bytes of bloat, so it seems
      better to keep them compared to adding yet another Kconfig symbol.
      I considered adding new syscall numbers for the IPC_64 variants for
      consistency, but decided against that for now.
      
      Signed-off-by: default avatarArnd Bergmann <arnd@arndb.de>
      275f2214
  9. Aug 27, 2018
    • Arnd Bergmann's avatar
      y2038: globally rename compat_time to old_time32 · 9afc5eee
      Arnd Bergmann authored
      
      Christoph Hellwig suggested a slightly different path for handling
      backwards compatibility with the 32-bit time_t based system calls:
      
      Rather than simply reusing the compat_sys_* entry points on 32-bit
      architectures unchanged, we get rid of those entry points and the
      compat_time types by renaming them to something that makes more sense
      on 32-bit architectures (which don't have a compat mode otherwise),
      and then share the entry points under the new name with the 64-bit
      architectures that use them for implementing the compatibility.
      
      The following types and interfaces are renamed here, and moved
      from linux/compat_time.h to linux/time32.h:
      
      old				new
      ---				---
      compat_time_t			old_time32_t
      struct compat_timeval		struct old_timeval32
      struct compat_timespec		struct old_timespec32
      struct compat_itimerspec	struct old_itimerspec32
      ns_to_compat_timeval()		ns_to_old_timeval32()
      get_compat_itimerspec64()	get_old_itimerspec32()
      put_compat_itimerspec64()	put_old_itimerspec32()
      compat_get_timespec64()		get_old_timespec32()
      compat_put_timespec64()		put_old_timespec32()
      
      As we already have aliases in place, this patch addresses only the
      instances that are relevant to the system call interface in particular,
      not those that occur in device drivers and other modules. Those
      will get handled separately, while providing the 64-bit version
      of the respective interfaces.
      
      I'm not renaming the timex, rusage and itimerval structures, as we are
      still debating what the new interface will look like, and whether we
      will need a replacement at all.
      
      This also doesn't change the names of the syscall entry points, which can
      be done more easily when we actually switch over the 32-bit architectures
      to use them, at that point we need to change COMPAT_SYSCALL_DEFINEx to
      SYSCALL_DEFINEx with a new name, e.g. with a _time32 suffix.
      
      Suggested-by: default avatarChristoph Hellwig <hch@infradead.org>
      Link: https://lore.kernel.org/lkml/20180705222110.GA5698@infradead.org/
      
      
      Signed-off-by: default avatarArnd Bergmann <arnd@arndb.de>
      9afc5eee
  10. Aug 22, 2018
  11. Jun 22, 2018
    • NeilBrown's avatar
      rhashtable: split rhashtable.h · 0eb71a9d
      NeilBrown authored
      
      Due to the use of rhashtables in net namespaces,
      rhashtable.h is included in lots of the kernel,
      so a small changes can required a large recompilation.
      This makes development painful.
      
      This patch splits out rhashtable-types.h which just includes
      the major type declarations, and does not include (non-trivial)
      inline code.  rhashtable.h is no longer included by anything
      in the include/ directory.
      Common include files only include rhashtable-types.h so a large
      recompilation is only triggered when that changes.
      
      Acked-by: default avatarHerbert Xu <herbert@gondor.apana.org.au>
      Signed-off-by: default avatarNeilBrown <neilb@suse.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      0eb71a9d
  12. Apr 20, 2018
    • Arnd Bergmann's avatar
      y2038: ipc: Report long times to user space · c2ab975c
      Arnd Bergmann authored
      
      The shmid64_ds/semid64_ds/msqid64_ds data structures have been extended
      to contain extra fields for storing the upper bits of the time stamps,
      this patch does the other half of the job and and fills the new fields on
      32-bit architectures as well as 32-bit tasks running on a 64-bit kernel
      in compat mode.
      
      There should be no change for native 64-bit tasks.
      
      Signed-off-by: default avatarArnd Bergmann <arnd@arndb.de>
      c2ab975c
    • Arnd Bergmann's avatar
      y2038: ipc: Use ktime_get_real_seconds consistently · 2a70b787
      Arnd Bergmann authored
      
      In some places, we still used get_seconds() instead of
      ktime_get_real_seconds(), and I'm changing the remaining ones now to
      all use ktime_get_real_seconds() so we use the full available range for
      timestamps instead of overflowing the 'unsigned long' return value in
      year 2106 on 32-bit kernels.
      
      Signed-off-by: default avatarArnd Bergmann <arnd@arndb.de>
      2a70b787
  13. Apr 11, 2018
    • Davidlohr Bueso's avatar
      ipc/msg: introduce msgctl(MSG_STAT_ANY) · 23c8cec8
      Davidlohr Bueso authored
      There is a permission discrepancy when consulting msq ipc object
      metadata between /proc/sysvipc/msg (0444) and the MSG_STAT shmctl
      command.  The later does permission checks for the object vs S_IRUGO.
      As such there can be cases where EACCESS is returned via syscall but the
      info is displayed anyways in the procfs files.
      
      While this might have security implications via info leaking (albeit no
      writing to the msq metadata), this behavior goes way back and showing
      all the objects regardless of the permissions was most likely an
      overlook - so we are stuck with it.  Furthermore, modifying either the
      syscall or the procfs file can cause userspace programs to break (ie
      ipcs).  Some applications require getting the procfs info (without root
      privileges) and can be rather slow in comparison with a syscall -- up to
      500x in some reported cases for shm.
      
      This patch introduces a new MSG_STAT_ANY command such that the msq ipc
      object permissions are ignored, and only audited instead.  In addition,
      I've left the lsm security hook checks in place, as if some policy can
      block the call, then the user has no other choice than just parsing the
      procfs file.
      
      Link: http://lkml.kernel.org/r/20180215162458.10059-4-dave@stgolabs.net
      
      
      Signed-off-by: default avatarDavidlohr Bueso <dbueso@suse.de>
      Reported-by: default avatarRobert Kettler <robert.kettler@outlook.com>
      Cc: Eric W. Biederman <ebiederm@xmission.com>
      Cc: Kees Cook <keescook@chromium.org>
      Cc: Manfred Spraul <manfred@colorfullife.com>
      Cc: Michael Kerrisk <mtk.manpages@gmail.com>
      Cc: Michal Hocko <mhocko@kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      23c8cec8
  14. Apr 02, 2018
  15. Mar 27, 2018
    • Eric W. Biederman's avatar
      ipc: Directly call the security hook in ipc_ops.associate · 50ab44b1
      Eric W. Biederman authored
      
      After the last round of cleanups the shm, sem, and msg associate
      operations just became trivial wrappers around the appropriate security
      method.  Simplify things further by just calling the security method
      directly.
      
      Signed-off-by: default avatar"Eric W. Biederman" <ebiederm@xmission.com>
      50ab44b1
    • Eric W. Biederman's avatar
      ipc/msg: Fix msgctl(..., IPC_STAT, ...) between pid namespaces · 39a4940e
      Eric W. Biederman authored
      
      Today msg_lspid and msg_lrpid are remembered in the pid namespace of
      the creator and the processes that last send or received a sysvipc
      message.  If you have processes in multiple pid namespaces that is
      just wrong.  The process ids reported will not make the least bit of
      sense.
      
      This fix is slightly more susceptible to a performance problem than
      the related fix for System V shared memory.  By definition the pids
      are updated by msgsnd and msgrcv, the fast path of System V message
      queues.  The only concern over the previous implementation is the
      incrementing and decrementing of the pid reference count.  As that is
      the only difference and multiple updates by of the task_tgid by
      threads in the same process have been shown in af_unix sockets to
      create a cache line ping-pong between cpus of the same processor.
      
      In this case I don't expect cache lines holding pid reference counts
      to ping pong between cpus.  As senders and receivers update different
      pids there is a natural separation there.  Further if multiple threads
      of the same process either send or receive messages the pid will be
      updated to the same value and ipc_update_pid will avoid the reference
      count update.
      
      Which means in the common case I expect msg_lspid and msg_lrpid to
      remain constant, and reference counts not to be updated when messages
      are sent.
      
      In rare cases it may be possible to trigger the issue which was
      observed for af_unix sockets, but it will require multiple processes
      with multiple threads to be either sending or receiving messages.  It
      just does not feel likely that anyone would do that in practice.
      
      This change updates msgctl(..., IPC_STAT, ...) to return msg_lspid and
      msg_lrpid in the pid namespace of the process calling stat.
      
      This change also updates cat /proc/sysvipc/msg to return print msg_lspid
      and msg_lrpid in the pid namespace of the process that opened the proc
      file.
      
      Fixes: b488893a ("pid namespaces: changes to show virtual ids to user")
      Reviewed-by: default avatarNagarathnam Muthusamy <nagarathnam.muthusamy@oracle.com>
      Signed-off-by: default avatar"Eric W. Biederman" <ebiederm@xmission.com>
      39a4940e
  16. Mar 24, 2018
  17. Mar 23, 2018
  18. Feb 07, 2018
  19. Nov 02, 2017
    • Greg Kroah-Hartman's avatar
      License cleanup: add SPDX GPL-2.0 license identifier to files with no license · b2441318
      Greg Kroah-Hartman authored
      
      Many source files in the tree are missing licensing information, which
      makes it harder for compliance tools to determine the correct license.
      
      By default all files without license information are under the default
      license of the kernel, which is GPL version 2.
      
      Update the files which contain no license information with the 'GPL-2.0'
      SPDX license identifier.  The SPDX identifier is a legally binding
      shorthand, which can be used instead of the full boiler plate text.
      
      This patch is based on work done by Thomas Gleixner and Kate Stewart and
      Philippe Ombredanne.
      
      How this work was done:
      
      Patches were generated and checked against linux-4.14-rc6 for a subset of
      the use cases:
       - file had no licensing information it it.
       - file was a */uapi/* one with no licensing information in it,
       - file was a */uapi/* one with existing licensing information,
      
      Further patches will be generated in subsequent months to fix up cases
      where non-standard license headers were used, and references to license
      had to be inferred by heuristics based on keywords.
      
      The analysis to determine which SPDX License Identifier to be applied to
      a file was done in a spreadsheet of side by side results from of the
      output of two independent scanners (ScanCode & Windriver) producing SPDX
      tag:value files created by Philippe Ombredanne.  Philippe prepared the
      base worksheet, and did an initial spot review of a few 1000 files.
      
      The 4.13 kernel was the starting point of the analysis with 60,537 files
      assessed.  Kate Stewart did a file by file comparison of the scanner
      results in the spreadsheet to determine which SPDX license identifier(s)
      to be applied to the file. She confirmed any determination that was not
      immediately clear with lawyers working with the Linux Foundation.
      
      Criteria used to select files for SPDX license identifier tagging was:
       - Files considered eligible had to be source code files.
       - Make and config files were included as candidates if they contained >5
         lines of source
       - File already had some variant of a license header in it (even if <5
         lines).
      
      All documentation files were explicitly excluded.
      
      The following heuristics were used to determine which SPDX license
      identifiers to apply.
      
       - when both scanners couldn't find any license traces, file was
         considered to have no license information in it, and the top level
         COPYING file license applied.
      
         For non */uapi/* files that summary was:
      
         SPDX license identifier                            # files
         ---------------------------------------------------|-------
         GPL-2.0                                              11139
      
         and resulted in the first patch in this series.
      
         If that file was a */uapi/* path one, it was "GPL-2.0 WITH
         Linux-syscall-note" otherwise it was "GPL-2.0".  Results of that was:
      
         SPDX license identifier                            # files
         ---------------------------------------------------|-------
         GPL-2.0 WITH Linux-syscall-note                        930
      
         and resulted in the second patch in this series.
      
       - if a file had some form of licensing information in it, and was one
         of the */uapi/* ones, it was denoted with the Linux-syscall-note if
         any GPL family license was found in the file or had no licensing in
         it (per prior point).  Results summary:
      
         SPDX license identifier                            # files
         ---------------------------------------------------|------
         GPL-2.0 WITH Linux-syscall-note                       270
         GPL-2.0+ WITH Linux-syscall-note                      169
         ((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause)    21
         ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause)    17
         LGPL-2.1+ WITH Linux-syscall-note                      15
         GPL-1.0+ WITH Linux-syscall-note                       14
         ((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause)    5
         LGPL-2.0+ WITH Linux-syscall-note                       4
         LGPL-2.1 WITH Linux-syscall-note                        3
         ((GPL-2.0 WITH Linux-syscall-note) OR MIT)              3
         ((GPL-2.0 WITH Linux-syscall-note) AND MIT)             1
      
         and that resulted in the third patch in this series.
      
       - when the two scanners agreed on the detected license(s), that became
         the concluded license(s).
      
       - when there was disagreement between the two scanners (one detected a
         license but the other didn't, or they both detected different
         licenses) a manual inspection of the file occurred.
      
       - In most cases a manual inspection of the information in the file
         resulted in a clear resolution of the license that should apply (and
         which scanner probably needed to revisit its heuristics).
      
       - When it was not immediately clear, the license identifier was
         confirmed with lawyers working with the Linux Foundation.
      
       - If there was any question as to the appropriate license identifier,
         the file was flagged for further research and to be revisited later
         in time.
      
      In total, over 70 hours of logged manual review was done on the
      spreadsheet to determine the SPDX license identifiers to apply to the
      source files by Kate, Philippe, Thomas and, in some cases, confirmation
      by lawyers working with the Linux Foundation.
      
      Kate also obtained a third independent scan of the 4.13 code base from
      FOSSology, and compared selected files where the other two scanners
      disagreed against that SPDX file, to see if there was new insights.  The
      Windriver scanner is based on an older version of FOSSology in part, so
      they are related.
      
      Thomas did random spot checks in about 500 files from the spreadsheets
      for the uapi headers and agreed with SPDX license identifier in the
      files he inspected. For the non-uapi files Thomas did random spot checks
      in about 15000 files.
      
      In initial set of patches against 4.14-rc6, 3 files were found to have
      copy/paste license identifier errors, and have been fixed to reflect the
      correct identifier.
      
      Additionally Philippe spent 10 hours this week doing a detailed manual
      inspection and review of the 12,461 patched files from the initial patch
      version early this week with:
       - a full scancode scan run, collecting the matched texts, detected
         license ids and scores
       - reviewing anything where there was a license detected (about 500+
         files) to ensure that the applied SPDX license was correct
       - reviewing anything where there was no detection but the patch license
         was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied
         SPDX license was correct
      
      This produced a worksheet with 20 files needing minor correction.  This
      worksheet was then exported into 3 different .csv files for the
      different types of files to be modified.
      
      These .csv files were then reviewed by Greg.  Thomas wrote a script to
      parse the csv files and add the proper SPDX tag to the file, in the
      format that the file expected.  This script was further refined by Greg
      based on the output to detect more types of files automatically and to
      distinguish between header and source .c files (which need different
      comment types.)  Finally Greg ran the script using the .csv files to
      generate the patches.
      
      Reviewed-by: default avatarKate Stewart <kstewart@linuxfoundation.org>
      Reviewed-by: default avatarPhilippe Ombredanne <pombredanne@nexb.com>
      Reviewed-by: default avatarThomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarGreg Kroah-Hartman <gregkh@linuxfoundation.org>
      b2441318
  20. Oct 11, 2017
  21. Sep 09, 2017
    • Guillaume Knispel's avatar
      ipc: optimize semget/shmget/msgget for lots of keys · 0cfb6aee
      Guillaume Knispel authored
      ipc_findkey() used to scan all objects to look for the wanted key.  This
      is slow when using a high number of keys.  This change adds an rhashtable
      of kern_ipc_perm objects in ipc_ids, so that one lookup cease to be O(n).
      
      This change gives a 865% improvement of benchmark reaim.jobs_per_min on a
      56 threads Intel(R) Xeon(R) CPU E5-2695 v3 @ 2.30GHz with 256G memory [1]
      
      Other (more micro) benchmark results, by the author: On an i5 laptop, the
      following loop executed right after a reboot took, without and with this
      change:
      
          for (int i = 0, k=0x424242; i < KEYS; ++i)
              semget(k++, 1, IPC_CREAT | 0600);
      
                       total       total          max single  max single
         KEYS        without        with        call without   call with
      
            1            3.5         4.9   µs            3.5         4.9
           10            7.6         8.6   µs            3.7         4.7
           32           16.2        15.9   µs            4.3         5.3
          100           72.9        41.8   µs            3.7         4.7
         1000        5,630.0       502.0   µs             *           *
        10000    1,340,000.0     7,240.0   µs             *           *
        31900   17,600,000.0    22,200.0   µs             *           *
      
       *: unreliable measure: high variance
      
      The duration for a lookup-only usage was obtained by the same loop once
      the keys are present:
      
                       total       total          max single  max single
         KEYS        without        with        call without   call with
      
            1            2.1         2.5   µs            2.1         2.5
           10            4.5         4.8   µs            2.2         2.3
           32           13.0        10.8   µs            2.3         2.8
          100           82.9        25.1   µs             *          2.3
         1000        5,780.0       217.0   µs             *           *
        10000    1,470,000.0     2,520.0   µs             *           *
        31900   17,400,000.0     7,810.0   µs             *           *
      
      Finally, executing each semget() in a new process gave, when still
      summing only the durations of these syscalls:
      
      creation:
                       total       total
         KEYS        without        with
      
            1            3.7         5.0   µs
           10           32.9        36.7   µs
           32          125.0       109.0   µs
          100          523.0       353.0   µs
         1000       20,300.0     3,280.0   µs
        10000    2,470,000.0    46,700.0   µs
        31900   27,800,000.0   219,000.0   µs
      
      lookup-only:
                       total       total
         KEYS        without        with
      
            1            2.5         2.7   µs
           10           25.4        24.4   µs
           32          106.0        72.6   µs
          100          591.0       352.0   µs
         1000       22,400.0     2,250.0   µs
        10000    2,510,000.0    25,700.0   µs
        31900   28,200,000.0   115,000.0   µs
      
      [1] http://lkml.kernel.org/r/20170814060507.GE23258@yexl-desktop
      
      Link: http://lkml.kernel.org/r/20170815194954.ck32ta2z35yuzpwp@debix
      
      
      Signed-off-by: default avatarGuillaume Knispel <guillaume.knispel@supersonicimagine.com>
      Reviewed-by: default avatarMarc Pardo <marc.pardo@supersonicimagine.com>
      Cc: Davidlohr Bueso <dave@stgolabs.net>
      Cc: Kees Cook <keescook@chromium.org>
      Cc: Manfred Spraul <manfred@colorfullife.com>
      Cc: Alexey Dobriyan <adobriyan@gmail.com>
      Cc: "Eric W. Biederman" <ebiederm@xmission.com>
      Cc: "Peter Zijlstra (Intel)" <peterz@infradead.org>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
      Cc: Serge Hallyn <serge@hallyn.com>
      Cc: Andrey Vagin <avagin@openvz.org>
      Cc: Guillaume Knispel <guillaume.knispel@supersonicimagine.com>
      Cc: Marc Pardo <marc.pardo@supersonicimagine.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      0cfb6aee
  22. Sep 04, 2017
  23. Aug 03, 2017
  24. Jul 16, 2017
  25. Jul 12, 2017
Loading