diff options
| author | Jann Horn <jannh@google.com> | 2020-03-17 01:28:45 +0100 |
|---|---|---|
| committer | Greg Kroah-Hartman <gregkh@linuxfoundation.org> | 2020-03-20 11:55:59 +0100 |
| commit | 30f6cae722654caef2ab4bacb2e910bfd766866b (patch) | |
| tree | f711ab604638687f368ba3d63bc01e473e1096a7 /mm/slub.c | |
| parent | c1a9559a24b0ae78ea1f7a6ac453cc5aedda32b3 (diff) | |
| download | linux-stable-30f6cae722654caef2ab4bacb2e910bfd766866b.tar.gz linux-stable-30f6cae722654caef2ab4bacb2e910bfd766866b.tar.bz2 linux-stable-30f6cae722654caef2ab4bacb2e910bfd766866b.zip | |
mm: slub: add missing TID bump in kmem_cache_alloc_bulk()
commit fd4d9c7d0c71866ec0c2825189ebd2ce35bd95b8 upstream.
When kmem_cache_alloc_bulk() attempts to allocate N objects from a percpu
freelist of length M, and N > M > 0, it will first remove the M elements
from the percpu freelist, then call ___slab_alloc() to allocate the next
element and repopulate the percpu freelist. ___slab_alloc() can re-enable
IRQs via allocate_slab(), so the TID must be bumped before ___slab_alloc()
to properly commit the freelist head change.
Fix it by unconditionally bumping c->tid when entering the slowpath.
Cc: stable@vger.kernel.org
Fixes: ebe909e0fdb3 ("slub: improve bulk alloc strategy")
Signed-off-by: Jann Horn <jannh@google.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Diffstat (limited to 'mm/slub.c')
| -rw-r--r-- | mm/slub.c | 9 |
1 files changed, 9 insertions, 0 deletions
diff --git a/mm/slub.c b/mm/slub.c index 9c3937c5ce38..764a1023ed87 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3104,6 +3104,15 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, if (unlikely(!object)) { /* + * We may have removed an object from c->freelist using + * the fastpath in the previous iteration; in that case, + * c->tid has not been bumped yet. + * Since ___slab_alloc() may reenable interrupts while + * allocating memory, we should bump c->tid now. + */ + c->tid = next_tid(c->tid); + + /* * Invoking slow path likely have side-effect * of re-populating per CPU c->freelist */ |