Writes to queue/scheduler failing with EBUSY

If you’ve ever tuned a disk’s I/O scheduler on Linux, you’ve probably written to its queue/scheduler sysfs file, or let a udev rule do it for you at boot. On Linux 7.2, a regression report showed up where those writes started failing with -EBUSY, even when nothing else was touching that disk’s scheduler.

Tracing the -EBUSY led straight to a trylock on a lock that was much wider than a single disk. Commit 4ff58d6bc9dd (“block: serialize elevator changes for the same queue using a writer lock”) had changed the sysfs write path to take set->update_nr_hwq_lock for write. Because of an earlier commit, ce8ee8583ed8 (“block: use trylock to avoid lockdep circular dependency in sysfs”), that lock has to be acquired with a trylock, returning -EBUSY if anyone else is holding it.

Here’s the catch: set->update_nr_hwq_lock belongs to the entire tag set, not an individual disk’s queue. So if you have multiple disks on the same SCSI host or NVMe controller, switching the scheduler on one disk blocks scheduler writes on all its siblings. Adding or removing a sibling disk triggers the same failure. When udev brings up several disks at once, you run into it right away.

Looking closer, the write lock wasn’t actually needed across the whole tag set. It was only there to keep two concurrent scheduler switches on the same queue from stepping on each other inside elevator_change_done(), which updates sysfs and debugfs entries outside q->elevator_lock. Separate queues in the same tag set didn’t need to block each other at all.

My first attempt added a per-queue mutex to serialize those switches. During review on the list, a cleaner idea came up: reuse the existing QUEUE_FLAG_NO_ELV_SWITCH flag as a busy bit instead of introducing another lock. I also tested taking q->sysfs_lock around the switch. That fixed the cross-disk failures too, but it made concurrent writers to the same queue wait on the lock, and commit 1bf70d08cc3b had already moved elevator switching off q->sysfs_lock.

So v2 went with the flag. The sysfs write goes back to taking set->update_nr_hwq_lock for read, sets QUEUE_FLAG_NO_ELV_SWITCH before elevator_change(), clears it when done, and returns -EBUSY if the flag is already set. Since disk removal already sets that same flag under the write lock, no new switch can sneak in while a disk is going away.

Before: write lock on tag set update_nr_hwq_lock (WRITE trylock) nvme0n1 switching scheduler holds lock nvme0n2 udev writes queue/scheduler -EBUSY (blocked) After: read lock + per-queue flag update_nr_hwq_lock (READ — shared) nvme0n1 NO_ELV_SWITCH bit set on q1 switching nvme0n2 NO_ELV_SWITCH bit set on q2 runs in parallel

Concurrent writes to the same queue, or writes racing with an nr_hw_queues update, an nr_requests write, or disk removal, still return -EBUSY just like before. The only small change is that writing to a disk while it’s being removed now returns -EBUSY instead of -ENOENT. Running concurrent writers across NVMe namespaces and SCSI LUNs dropped the cross-disk -EBUSY failures from hundreds per run down to zero, while keeping same-queue behavior and blktests unchanged.

The fix landed in mainline as 603523fd5466 (lore thread).

Comments