System Rollback on ext4 Using rsync and Hardlinks: A Practical Implementation
Standard Linux installations on ext4 lack built-in snapshot support, making quick rollbacks to previous states challenging. Backup tools like borg or restic focus on long-term data storage rather than local rollback. Restoring from a full backup takes hours and involves processing large volumes, which is inefficient for fixing local configuration errors or post-update issues.
The solution minimizes dependencies: no need to change the file system, use LVM, or adopt specialized FS like Btrfs/ZFS. The approach relies on rsync with hardlink support for deduplication.
How It Works: rsync + Hardlinks
rsync synchronizes directories by copying only changes. The --link-dest option allows referencing a previous snapshot: unchanged files are reused via hardlinks, minimizing storage usage. A hardlink is an additional name for the same inode, ensuring zero overhead for unchanged data.
Basic command to create a snapshot:
rsync -aHAX --numeric-ids --delete --link-dest="$PREV" "$SRC" "$DEST_TMP/"
-a: Archive mode (recursive, preserves permissions, timestamps).-H: Preserves hardlinks.-A: Preserves ACLs.-X: Preserves extended attributes.--numeric-ids: Uses numeric UID/GID.--delete: Removes extra files in the target directory.
This brings the current state to a snapshot, copying only the delta.
Snapshot Storage Structure
Snapshots are stored in .infra_snapshots/ as regular directories with timestamps:
.infra_snapshots/
├── system/
│ ├── 2026-03-01_00-30-00/
│ ├── 2026-03-02_00-30-00/
│ └── LATEST -> 2026-03-02_00-30-00/
├── docker/
│ ├── 2026-03-01_23-30-00/
│ └── LATEST -> 2026-03-01_23-30-00/
The LATEST symlink points to the most recent snapshot for automation. Separating system and docker isolates system files from container data: Docker volumes are excluded to avoid issues with mutable data.
Creation process:
- Create a temporary directory
.tmp-YYYY-MM-DD_HH-MM-SS. - Run rsync into it.
- Rename atomically:
mv -T "$TMP_DEST" "$FINAL_DEST". - Update
LATEST:ln -sfn "$FINAL_DEST" "$LATEST_LINK".
This ensures atomicity: the snapshot is either complete or absent.
Restoring State
Rollback involves reverse synchronization with a dry-run for preview:
rsync -aHAX --numeric-ids --delete --dry-run "$SNAP/" "/"
After verification, remove --dry-run to apply. rsync will bring the root system to the snapshot state, deleting new files and restoring old ones via hardlinks.
For Docker: similarly, but only on container directories, excluding volumes.
Limitations of the Approach
- Lack of System Atomicity: rsync reads files sequentially; the system may change during creation. Snapshots are file-consistent but not instantaneous.
- Mutable Data: Logs, databases, and temp files can be in an intermediate state. Exclude them from rsync using
--exclude. - Disk Space: New file versions occupy space; periodically clean up old snapshots.
- Not for High-Load Production: Suitable for servers/desktops with low activity during snapshot creation.
| Scenario | Creation Time (approx.) | Disk Usage |
|----------|---------------------------|---------------|
| Daily system | 2-5 min | 100-500 MB |
| Docker after updates | 1-3 min | 50-200 MB |
Automation
Integrate into cron for daily snapshots:
- 00:30: system.
- 23:30: docker.
The script should log rsync output, check exit codes, and notify on errors.
Key Points:
- rsync with
--link-destemulates snapshots on ext4 without changing the FS. - Hardlinks provide deduplication: unchanged files are not duplicated.
- Atomicity via tmp-directories prevents partial snapshots.
- Separating system/docker improves rollback accuracy.
- Dry-run is essential before restore for safety.
The approach is transparent, uses only standard Linux utilities, and scales to any ext4 system.
— Editorial Team
No comments yet.