kernel.git/fs/eventpoll.c, branch linux-4.4.y

ep_create_wakeup_source(): dentry name can change under you...

2020-10-14T07:46:21Z

commit 3701cb59d892b88d569427586f01491552f377b1 upstream. or get freed, for that matter, if it's a long (separately stored) name. Signed-off-by: Al Viro Signed-off-by: Greg Kroah-Hartman

epoll: EPOLL_CTL_ADD: close the race in decision to take fast path

2020-10-14T07:46:21Z

commit fe0a916c1eae8e17e86c3753d13919177d63ed7e upstream. Checking for the lack of epitems refering to the epoll we want to insert into is not enough; we might have an insertion of that epoll into another one that has already collected the set of files to recheck for excessive reverse paths, but hasn't gotten to creating/inserting the epitem for it. However, any such insertion in progress can be detected - it will update the generation count in our epoll when it's done looking through it for files to check. That gets done under ->mtx of our epoll and that allows us to detect that safely. We are *not* holding epmutex here, so the generation count is not stable. However, since both the update of ep->gen by loop check and (later) insertion into ->f_ep_link are done with ep->mtx held, we are fine - the sequence is grab epmutex bump loop_check_gen ... grab tep->mtx // 1 tep->gen = loop_check_gen ... drop tep->mtx // 2 ... grab tep->mtx // 3 ... insert into ->f_ep_link ... drop tep->mtx // 4 bump loop_check_gen drop epmutex and if the fastpath check in another thread happens for that eventpoll, it can come * before (1) - in that case fastpath is just fine * after (4) - we'll see non-empty ->f_ep_link, slow path taken * between (2) and (3) - loop_check_gen is stable, with ->mtx providing barriers and we end up taking slow path. Note that ->f_ep_link emptiness check is slightly racy - we are protected against insertions into that list, but removals can happen right under us. Not a problem - in the worst case we'll end up taking a slow path for no good reason. Signed-off-by: Al Viro Signed-off-by: Greg Kroah-Hartman

epoll: replace ->visited/visited_list with generation count

2020-10-14T07:46:21Z

commit 18306c404abe18a0972587a6266830583c60c928 upstream. removes the need to clear it, along with the races. Signed-off-by: Al Viro Signed-off-by: Greg Kroah-Hartman

epoll: do not insert into poll queues until all sanity checks are done

2020-10-14T07:46:20Z

commit f8d4f44df056c5b504b0d49683fb7279218fd207 upstream. Signed-off-by: Al Viro Signed-off-by: Greg Kroah-Hartman

fix regression in "epoll: Keep a reference on files added to the check list"

2020-09-12T09:45:28Z

[ Upstream commit 77f4689de17c0887775bb77896f4cc11a39bf848 ] epoll_loop_check_proc() can run into a file already committed to destruction; we can't grab a reference on those and don't need to add them to the set for reverse path check anyway. Tested-by: Marc Zyngier Fixes: a9ed4a6560b8 ("epoll: Keep a reference on files added to the check list") Signed-off-by: Al Viro Signed-off-by: Sasha Levin

do_epoll_ctl(): clean the failure exits up a bit

2020-08-26T08:27:10Z

commit 52c479697c9b73f628140dcdfcd39ea302d05482 upstream. Signed-off-by: Al Viro Signed-off-by: Marc Zyngier Signed-off-by: Greg Kroah-Hartman

epoll: Keep a reference on files added to the check list

2020-08-26T08:27:10Z

commit a9ed4a6560b8562b7e2e2bed9527e88001f7b682 upstream. When adding a new fd to an epoll, and that this new fd is an epoll fd itself, we recursively scan the fds attached to it to detect cycles, and add non-epool files to a "check list" that gets subsequently parsed. However, this check list isn't completely safe when deletions can happen concurrently. To sidestep the issue, make sure that a struct file placed on the check list sees its f_count increased, ensuring that a concurrent deletion won't result in the file disapearing from under our feet. Cc: stable@vger.kernel.org Signed-off-by: Marc Zyngier Signed-off-by: Al Viro Signed-off-by: Marc Zyngier Signed-off-by: Greg Kroah-Hartman

fs/epoll: drop ovflist branch prediction

2019-02-20T09:13:14Z

[ Upstream commit 76699a67f3041ff4c7af6d6ee9be2bfbf1ffb671 ] The ep->ovflist is a secondary ready-list to temporarily store events that might occur when doing sproc without holding the ep->wq.lock. This accounts for every time we check for ready events and also send events back to userspace; both callbacks, particularly the latter because of copy_to_user, can account for a non-trivial time. As such, the unlikely() check to see if the pointer is being used, seems both misleading and sub-optimal. In fact, we go to an awful lot of trouble to sync both lists, and populating the ovflist is far from an uncommon scenario. For example, profiling a concurrent epoll_wait(2) benchmark, with CONFIG_PROFILE_ANNOTATED_BRANCHES shows that for a two threads a 33% incorrect rate was seen; and when incrementally increasing the number of epoll instances (which is used, for example for multiple queuing load balancing models), up to a 90% incorrect rate was seen. Similarly, by deleting the prediction, 3% throughput boost was seen across incremental threads. Link: http://lkml.kernel.org/r/20181108051006.18751-4-dave@stgolabs.net Signed-off-by: Davidlohr Bueso Reviewed-by: Andrew Morton Cc: Al Viro Cc: Jason Baron Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds Signed-off-by: Sasha Levin

epoll: fix race between ep_poll_callback(POLLFREE) and ep_free()/ep_remove()

2017-09-07T06:34:10Z

commit 138e4ad67afd5c6c318b056b4d17c17f2c0ca5c0 upstream. The race was introduced by me in commit 971316f0503a ("epoll: ep_unregister_pollwait() can use the freed pwq->whead"). I did not realize that nothing can protect eventpoll after ep_poll_callback() sets ->whead = NULL, only whead->lock can save us from the race with ep_free() or ep_remove(). Move ->whead = NULL to the end of ep_poll_callback() and add the necessary barriers. TODO: cleanup the ewake/EPOLLEXCLUSIVE logic, it was confusing even before this patch. Hopefully this explains use-after-free reported by syzcaller: BUG: KASAN: use-after-free in debug_spin_lock_before ... _raw_spin_lock_irqsave+0x4a/0x60 kernel/locking/spinlock.c:159 ep_poll_callback+0x29f/0xff0 fs/eventpoll.c:1148 this is spin_lock(eventpoll->lock), ... Freed by task 17774: ... kfree+0xe8/0x2c0 mm/slub.c:3883 ep_free+0x22c/0x2a0 fs/eventpoll.c:865 Fixes: 971316f0503a ("epoll: ep_unregister_pollwait() can use the freed pwq->whead") Reported-by: 范龙飞 Signed-off-by: Oleg Nesterov Signed-off-by: Linus Torvalds Signed-off-by: Greg Kroah-Hartman

epoll: optimize setting task running after blocking

2015-02-14T05:21:40Z

After waking up a task waiting for an event, we explicitly mark it as TASK_RUNNING (which is necessary as we do the checks for wakeups as TASK_INTERRUPTIBLE). Once running and dealing with actually delivering the events, we're obviously not planning on calling schedule, thus we can relax the implied barrier and simply update the state with __set_current_state(). Signed-off-by: Davidlohr Bueso Cc: Alexander Viro Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds