udev, Explained: From Kernel Events to an Automated Backup

Every plugged-in device on a Linux machine passes through the same layer before it becomes usable. This article covers what that layer, udev, actually is and how it connects to the kernel, then walks through a real, working example: a udev rule and systemd service that back up a project folder automatically whenever a specific USB drive is connected.

What udev actually is

udev is the device manager used by modern Linux systems. It is the layer responsible for everything that happens between a piece of hardware getting plugged in and a usable file for it showing up in /dev.

It is not part of the kernel. udev is run entirely in user space, as a background daemon called udevd. This distinction is what actually matters here: the kernel's job stops at detecting hardware and describing it. Naming the device, setting its permissions, and reacting to it are all handled outside the kernel, by udev, once the kernel has already finished its part.

It is kept this way on purpose. Device-naming policy is left out of the kernel so it can be changed, customized, or broken without touching kernel code itself. It also means the logic can be inspected and edited as plain text files, rather than something compiled into the operating system.

The kernel connection

What happens the moment a device is plugged in is best understood as a fixed sequence, illustrated in Figure 1. Each step hands off to the next, and no step is skipped.

The kernel's role is found to end at step two. It is never the kernel that decides what a device should be called or who is allowed to use it, that decision is left entirely to udev, driven by whatever rules a distro or a person has written.

This is also why device naming on Linux can appear inconsistent to anyone who has never touched it. Without a matching rule in place, a USB drive might be shown as /dev/sdb today and /dev/sdc tomorrow, depending on what else happens to be connected at the time. udev rules are what make that predictable.

Where rules live

Rules are plain text files ending in .rules. They are read from three locations, and the difference between them is worth knowing before writing anything of your own:

• /usr/lib/udev/rules.d/: shipped by the distro and by installed packages. Not meant to be edited directly.
• /etc/udev/rules.d/: where custom rules belong.
• /run/udev/rules.d/: temporary, runtime-only rules, cleared on reboot.

Anatomy of a rule

A rule is best read as a set of conditions followed by an action, all written on a single line. Figure 2 breaks the same rule apart piece by piece.

# match a USB storage device by its filesystem label, then run a script
ACTION=="add", SUBSYSTEM=="block", ENV{ID_FS_LABEL}=="MyDrive", RUN+="/usr/local/bin/handle-drive.sh"

It is read left to right: when a block device is added, and its filesystem label is found to be "MyDrive," this script is run. Every comma-separated piece either has to match, for the rule to apply at all, or is assigned, as an action to be carried out once it does.

• ACTION, the type of event: add, remove, or change.
• SUBSYSTEM, the category of device: block, usb, net, and so on.
• ENV{ } / ATTR{ }, a specific property to match, like a filesystem label, vendor ID, or serial number.
• RUN+=, the action to take once every condition matches, most often running a program.

What the number prefix is actually doing

Files are processed in alphabetical order. It is common convention to prefix a custom file with a number, such as 99-my-rule.rules, so that it is obvious at a glance when it runs relative to everything else already in place.

Sorting is the only thing being controlled here. Rule files are read and applied in the order their filenames sort alphabetically, and since the prefix is the first thing in the filename, it is the number that decides that order, not anything inside the file itself. A rule in 10-usb.rules is applied before a rule in 50-network.rules, which is applied before 99-usb-backup.rules.

Low numbers, typically in the 10 to 20 range, are used by base system rules that set up fundamentals, disk identification and basic permissions among them. Numbers in the middle, around 50 to 70, are where most distro-shipped hardware rules are found. It is convention, not a rule enforced by udev, to leave 99 for anything custom, since it guarantees the rule is applied last, after everything the system has already set up. This matters in practice: if a custom rule depends on a property that an earlier rule sets, such as a device already having been assigned a persistent name, using a high number is what ensures that property already exists by the time the custom rule is checked.

It should not be assumed the number needs to be exactly 99. Any number higher than the rules it depends on is enough, 99 is simply used because it reads unambiguously as "last" to the next person who opens the file.

Reading device properties and reloading rules

Before a rule can be written, it is usually necessary to know what properties a connected device actually has. udev provides a tool for exactly this:

udevadm info --query=all --name=/dev/sdb

Every property udev knows about that device is printed here, labels, serial numbers, vendor and product IDs, so the correct thing to match on is known ahead of time rather than guessed at.

Once a rule file has been written or edited, it is not required to reboot for it to take effect:

sudo udevadm control --reload-rules

A practical case: backing up automatically when a drive is plugged in

One of the more useful things udev can be put to work on is a backup that starts the instant a specific USB drive is connected, with no cron job involved and nothing left to be remembered by hand.

The rule is matched on something unique to that one drive, its filesystem label in this case, so that it is not fired for every USB device that happens to be plugged in:

# /etc/udev/rules.d/99-usb-backup.rules
ACTION=="add", SUBSYSTEM=="block", ENV{ID_FS_LABEL}=="BackupDrive", TAG+="systemd", ENV{SYSTEMD_WANTS}="usb-backup.service"

This rule does not call the backup script directly with RUN+=. The work is handed off to systemd instead, through SYSTEMD_WANTS. This is not a stylistic choice. It reflects a real constraint in how udev is designed to operate, and it is worth understanding before it becomes the cause of a backup that silently never runs.

It is stated directly in udev's own documentation that anything placed under RUN+= is expected to be a short, foreground task. Network access is not allowed, mounting or unmounting a filesystem is not allowed, and nothing long-running is allowed. Anything still active once the event finishes is killed outright. A real backup job breaks each of these rules in turn, which is exactly why udev hands the work off to systemd, where none of these restrictions apply.

The service that systemd is told to start looks like this:

# /etc/systemd/system/usb-backup.service
[Unit]
Description=Backup on USB plug-in

[Service]
Type=oneshot
ExecStartPre=/bin/sleep 15
ExecStart=/usr/local/bin/backup.sh
User=youruser

The fifteen-second delay ahead of the script matters more than it appears to. udev is fired the moment the kernel sees the device, not once the filesystem has actually been mounted. When the delay is skipped, the script is often run against a drive that isn't accessible yet, and it exits having done nothing, usually without any error to point at why.

Figure 3 lays out where the restriction actually sits, and why the handoff happens exactly where it does.

Setting the whole thing up, step by step

The pieces covered so far come together in a fixed build order. Seven steps are involved, and none of them can be skipped.

1. The drive's label is found first. Before a rule can be matched, it needs to be known what the drive is actually called. This is checked with:

   bash

   copy

   lsblk -o NAME,LABEL
   

   The label shown here is what gets used in the rule, copied exactly, not retyped from memory.

2. The backup script is written and placed at /usr/local/bin/backup.sh, a standard location for local executables. It is made runnable with:

   bash

   copy

   chmod +x /usr/local/bin/backup.sh
   

3. The udev rule is written to /etc/udev/rules.d/99-usb-backup.rules, matched on the label found in step one:

   text

   copy

   # /etc/udev/rules.d/99-usb-backup.rules
   ACTION=="add", SUBSYSTEM=="block", ENV{ID_FS_LABEL}=="BackupDrive", TAG+="systemd", ENV{SYSTEMD_WANTS}="usb-backup.service"
   

   The 99 prefix is used here for the reason already covered, it guarantees this rule is read after every base system rule that identifies the drive in the first place.

4. The systemd service is written to /etc/systemd/system/usb-backup.service:

   text

   copy

   # /etc/systemd/system/usb-backup.service
   [Unit]
   Description=Backup on USB plug-in
   
   [Service]
   Type=oneshot
   ExecStartPre=/bin/sleep 15
   ExecStart=/usr/local/bin/backup.sh
   User=youruser
   

5. Both subsystems are reloaded, since neither one picks up a new or edited file on its own:

   bash

   copy

   sudo udevadm control --reload-rules
   sudo systemctl daemon-reload
   

6. The drive is plugged in. Nothing needs to be run by hand at this point, the rule from step three is what triggers the service from step four.

7. The result is checked, rather than assumed:

   bash

   copy

   sudo systemctl status usb-backup.service
   sudo journalctl -u usb-backup.service --since today
   

   A clean exit is shown as Type=oneshot reporting success. If it isn't, the journal output is where the actual reason is found, most often one of the two causes covered next.

One honest caveat

It should not be assumed that handing work off to systemd removes every failure mode. A mismatched filesystem label is the most common cause of a rule that silently never fires, and it is found this way more often than any documentation bug or udev version issue. The label used in the rule has to match the drive's actual label exactly, checked with lsblk -o NAME,LABEL, not guessed at or copied from memory.

It is also worth being clear that the fifteen-second delay is a workaround, not a real fix. It is not being detected here whether the drive has actually finished mounting, only assumed that fifteen seconds is enough time for it to have done so. On a slower drive or a heavily loaded system, it may not be.

None of this should be taken to mean the approach is fragile. It is seen to work reliably once the label is confirmed and the delay is set generously. The failure modes are narrow, and they are the same two every time.

The takeaway

udev is the reason plugging in hardware on Linux is mostly found to "just work," consistent names, sane permissions, and no manual setup required. Because all of that logic lives in readable rule files rather than in kernel code, it is also the reason what "just works" can be redefined for a given machine, persistent device names, custom permissions, or a script reacting the instant specific hardware appears.

Once a rule can be read and traced back to the kernel event that triggered it, udev stops being treated as a mysterious background service and starts being used as a tool that is actually under control.