kernel.git/fs/gfs2/util.c, branch linux-6.7.y

gfs2: Fix withdraw race

2023-09-05T13:58:17Z

Function gfs2_withdraw() tries to synchronize concurrent callers by atomically setting the SDF_WITHDRAWN flag in the first caller, setting the SDF_WITHDRAW_IN_PROG flag to indicate that a withdraw is in progress, performing the actual withdraw, and clearing the SDF_WITHDRAW_IN_PROG flag when done. All other callers wait for the SDF_WITHDRAW_IN_PROG flag to be cleared before returning. This leaves a small window in which callers can find the SDF_WITHDRAWN flag set before the SDF_WITHDRAW_IN_PROG flag has been set, causing them to return prematurely, before the withdraw has been completed. Fix that by setting the SDF_WITHDRAWN and SDF_WITHDRAW_IN_PROG flags atomically. Signed-off-by: Andreas Gruenbacher

gfs2: Fix asynchronous thread destruction

2023-09-05T13:58:17Z

The kernel threads are currently stopped and destroyed synchronously by gfs2_make_fs_ro() and gfs2_put_super(), and asynchronously by signal_our_withdraw(), with no synchronization, so the synchronous and asynchronous contexts can race with each other. First, when creating the kernel threads, take an extra task struct reference so that the task struct won't go away immediately when they terminate. This allows those kthreads to terminate immediately when they're done rather than hanging around as zombies until they are reaped by kthread_stop(). When kthread_stop() is called on a terminated kthread, it will return immediately. Second, in signal_our_withdraw(), once the SDF_JOURNAL_LIVE flag has been cleared, wake up the logd and quotad wait queues instead of stopping the logd and quotad kthreads. The kthreads are then expected to terminate automatically within short time, but if they cannot, they will not block the withdraw. For example, if a user process and one of the kthread decide to withdraw at the same time, only one of them will perform the actual withdraw and the other will wait for it to be done. If the kthread ends up being the one to wait, the withdrawing user process won't be able to stop it. Signed-off-by: Andreas Gruenbacher

gfs2: Stop using gfs2_make_fs_ro for withdraw

2023-09-05T13:58:17Z

[ 81.372851][ T5532] CPU: 1 PID: 5532 Comm: syz-executor.0 Not tainted 6.2.0-rc1-syzkaller-dirty #0 [ 81.382080][ T5532] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/12/2023 [ 81.392343][ T5532] Call Trace: [ 81.395654][ T5532] [ 81.398603][ T5532] dump_stack_lvl+0x1b1/0x290 [ 81.418421][ T5532] gfs2_assert_warn_i+0x19a/0x2e0 [ 81.423480][ T5532] gfs2_quota_cleanup+0x4c6/0x6b0 [ 81.428611][ T5532] gfs2_make_fs_ro+0x517/0x610 [ 81.457802][ T5532] gfs2_withdraw+0x609/0x1540 [ 81.481452][ T5532] gfs2_inode_refresh+0xb2d/0xf60 [ 81.506658][ T5532] gfs2_instantiate+0x15e/0x220 [ 81.511504][ T5532] gfs2_glock_wait+0x1d9/0x2a0 [ 81.516352][ T5532] do_sync+0x485/0xc80 [ 81.554943][ T5532] gfs2_quota_sync+0x3da/0x8b0 [ 81.559738][ T5532] gfs2_sync_fs+0x49/0xb0 [ 81.564063][ T5532] sync_filesystem+0xe8/0x220 [ 81.568740][ T5532] generic_shutdown_super+0x6b/0x310 [ 81.574112][ T5532] kill_block_super+0x79/0xd0 [ 81.578779][ T5532] deactivate_locked_super+0xa7/0xf0 [ 81.584064][ T5532] cleanup_mnt+0x494/0x520 [ 81.593753][ T5532] task_work_run+0x243/0x300 [ 81.608837][ T5532] exit_to_user_mode_loop+0x124/0x150 [ 81.614232][ T5532] exit_to_user_mode_prepare+0xb2/0x140 [ 81.619820][ T5532] syscall_exit_to_user_mode+0x26/0x60 [ 81.625287][ T5532] do_syscall_64+0x49/0xb0 [ 81.629710][ T5532] entry_SYSCALL_64_after_hwframe+0x63/0xcd In this backtrace, gfs2_quota_sync() takes quota data references and then calls do_sync(). Function do_sync() encounters filesystem corruption and withdraws the filesystem, which (among other things) calls gfs2_quota_cleanup(). Function gfs2_quota_cleanup() wrongly assumes that nobody is holding any quota data references anymore, and destroys all quota data objects. When gfs2_quota_sync() then resumes and dereferences the quota data objects it is holding, those objects are no longer there. Function gfs2_quota_cleanup() deals with resource deallocation and can easily be delayed until gfs2_put_super() in the case of a filesystem withdraw. In fact, most of the other work gfs2_make_fs_ro() does is unnecessary during a withdraw as well, so change signal_our_withdraw() to skip gfs2_make_fs_ro() and perform the necessary steps directly instead. Thanks to Edward Adam Davis for the initial patches. Link: https://lore.kernel.org/all/0000000000002b5e2405f14e860f@google.com Reported-by: syzbot+3f6a670108ce43356017@syzkaller.appspotmail.com Signed-off-by: Andreas Gruenbacher

gfs2: Add quota_change type

2023-07-03T20:30:48Z

Function do_qc has two main uses: (1) to re-sync the local quota changes (qd) to the master quotas, and (2) normal quota changes. In the case of normal quota changes, the change can be positive or negative, as the quota usage goes up and down. Before this patch function do_qc was distinguishing one from another by whether the resulting value is or isn't zero: In the case of a re-sync (called do_sync) the quota value is moved from the temporary value to a master value, so the amount is added to one and subtracted from the other. The problem is that since the values can be positive or negative we can occasionally run into situations where we are not doing a re-sync but the quota change just happens to cancel out the previous value. In the case of a re-sync extra references and locks are taken, and so do_qc needs to release them. In the case of a normal quota change, no extra references and locks are taken, so it must not try to release them. The problem is: if the quota change is not a re-sync but the value just happens to cancel out the original quota change, the resulting zero value fools do_qc into thinking this is a re-sync and therefore it must release the extra references. This results in problems, mainly having to do with slot reference numbers going smaller than zero. This patch introduces new constants, QC_SYNC and QC_CHANGE so do_qc can really tell the difference. For QC_SYNC calls it must release the extra references acquired by gfs2_quota_unlock's call to qd_check_sync. For QC_CHANGE calls it does not have extra references to put. Note that this allows quota changes back to a value of zero, and so I removed an assert warning related to that. Signed-off-by: Bob Peterson Signed-off-by: Andreas Gruenbacher

gfs2: gfs2_freeze_lock_shared cleanup

2023-07-03T20:30:26Z

All the remaining users of gfs2_freeze_lock_shared() set freeze_gh to &sdp->sd_freeze_gh and flags to 0, so remove those two parameters. Signed-off-by: Andreas Gruenbacher

gfs2: Rework freeze / thaw logic

2023-07-03T20:25:02Z

So far, at mount time, gfs2 would take the freeze glock in shared mode and then immediately drop it again, turning it into a cached glock that can be reclaimed at any time. To freeze the filesystem cluster-wide, the node initiating the freeze would take the freeze glock in exclusive mode, which would cause the freeze glock's freeze_go_sync() callback to run on each node. There, gfs2 would freeze the filesystem and schedule gfs2_freeze_func() to run. gfs2_freeze_func() would re-acquire the freeze glock in shared mode, thaw the filesystem, and drop the freeze glock again. The initiating node would keep the freeze glock held in exclusive mode. To thaw the filesystem, the initiating node would drop the freeze glock again, which would allow gfs2_freeze_func() to resume on all nodes, leaving the filesystem in the thawed state. It turns out that in freeze_go_sync(), we cannot reliably and safely freeze the filesystem. This is primarily because the final unmount of a filesystem takes a write lock on the s_umount rw semaphore before calling into gfs2_put_super(), and freeze_go_sync() needs to call freeze_super() which also takes a write lock on the same semaphore, causing a deadlock. We could work around this by trying to take an active reference on the super block first, which would prevent unmount from running at the same time. But that can fail, and freeze_go_sync() isn't actually allowed to fail. To get around this, this patch changes the freeze glock locking scheme as follows: At mount time, each node takes the freeze glock in shared mode. To freeze a filesystem, the initiating node first freezes the filesystem locally and then drops and re-acquires the freeze glock in exclusive mode. All other nodes notice that there is contention on the freeze glock in their go_callback callbacks, and they schedule gfs2_freeze_func() to run. There, they freeze the filesystem locally and drop and re-acquire the freeze glock before re-thawing the filesystem. This is happening outside of the glock state engine, so there, we are allowed to fail. From a cluster point of view, taking and immediately dropping a glock is indistinguishable from taking the glock and only dropping it upon contention, so this new scheme is compatible with the old one. Thanks to Li Dong for reporting a locking bug in gfs2_freeze_func() in a previous version of this commit. Signed-off-by: Andreas Gruenbacher

gfs2: Rename SDF_{FS_FROZEN => FREEZE_INITIATOR}

2023-06-15T07:57:38Z

Rename the SDF_FS_FROZEN flag to SDF_FREEZE_INITIATOR to indicate more clearly that the node that has this flag set is the initiator of the freeze. Signed-off-by: Andreas Gruenbacher

gfs2: Rename gfs2_freeze_lock{ => _shared }

2023-06-15T07:57:38Z

Rename gfs2_freeze_lock to gfs2_freeze_lock_shared to make it a bit more obvious that this function establishes the "thawed" state of the freeze glock. Signed-off-by: Andreas Gruenbacher

gfs2: Rename the {freeze,thaw}_super callbacks

2023-06-15T07:57:38Z

Rename gfs2_freeze to gfs2_freeze_super and gfs2_unfreeze to gfs2_thaw_super to match the names of the corresponding super operations. Signed-off-by: Andreas Gruenbacher

gfs2: Rename remaining "transaction" glock references

2023-06-15T07:57:38Z

The transaction glock was repurposed to serve as the new freeze glock years ago. Don't refer to it as the transaction glock anymore. Also, to be more precise, call it the "freeze glock" instead of the "freeze lock". Ditto for the journal glock. Signed-off-by: Andreas Gruenbacher