maxpack: deduplication of versioned data

maxpack: Efficient Deduplication for Versioned File Sets

Inter-file deduplication in maxpack analyzes the entire file set as a single data space, eliminating duplicates across versions and objects. On 100 MB of Node.js, compression reaches 3.9x; on 10 GB, it hits 50.1x by reducing unique content to 0.6% per megabyte. The tool is designed for scenarios with high data overlap.

Real-World Applications

maxpack excels on data sets with repetitive content:

• Git project versions: Shared code is stored only once.
• Logs and database dumps: Adjacent exports differ minimally.
• Blockchain node snapshots: Overlaps in database state.
• Versioned datasets: Model checkpoints, images.
• VM and container snapshots: Shared state layers.
• Frontend builds: Byte-level matches despite hashes in filenames.

The effect intensifies with increased volume and duplicate ratio.

Benchmarks on Node.js v20

Testing on Apple M4 with 4 threads. Scaling from 100.1 MB to 10 GB via git tags:

• Initially, all methods ~3.9x.
• maxpack L3 on 10 GB: 50.1x.
• tar+zstd: ~4.8x.
• 7z: ~8.6x.

The share of unique content drops, allowing maxpack to boost compression.

Results on CPython 3.12 (817.3 MB, 8 tags)

| Method | Archive | Compression | Pack | Unpack |

|----------------|-----------|-------------|--------|--------|

| maxpack L3 | 31.0 MB | 26.4x | 0.9 s | 3.1 s |

| tar+zstd -3 | 215.3 MB | 3.8x | 16.5 s | 11.9 s |

| tar+zstd -19 | 170.3 MB | 4.8x | 113.7 s| 17.0 s |

| tar+xz | 168.3 MB | 4.9x | 41.6 s | 13.7 s |

maxpack reduces archive size by 7x compared to tar+zstd -3, with faster times.

Results on Go 1.23 (974.8 MB, 8 tags)

| Method | Archive | Compression | Pack | Unpack |

|----------------|-----------|-------------|---------|--------|

| maxpack L3 | 31.0 MB | 31.4x | 14.4 s | 14.7 s |

| tar+zstd -3 | 210.1 MB | 4.6x | 40.5 s | 35.1 s |

| tar+xz | 149.5 MB | 6.5x | 294.0 s | 40.5 s |

| 7z -mx=9 | 70.7 MB | 13.8x | 312.4 s | 11.4 s |

Archive is 2.3x smaller than 7z, with lower CPU cost for packing.

On projects fd, lsd, bat, size reduction is 41.6–70.3%, with deduplication up to 81.3%.

Controlled Datasets

High-dedup (100 files, 90% duplicates, 12 MB):

| Dataset | Method | Archive | Compression | vs tar+zstd-3 |

|-------------|-------------|-------------|-------------|---------------|

| High-dedup | tar+zstd-3 | 4,102,948 | 3.1x | – |

| High-dedup | maxpack L3 | 1,290,154 | 10.0x | -68.6% |

Archive is 3.2x smaller. On sets without duplicates—parity or slight overhead.

Speed Comparison

Unpacking is 1.5–2.2x faster:

• Diverse Unpack: maxpack 21 ms vs 32 ms.
• High-dedup Unpack: 29 ms vs 65 ms.
• Big Unpack: 71 ms vs 120 ms.

Packing depends on structure: parity on duplicate data, overhead on unique.

Usage Limitations

• Unique data: Minimal deduplication.
• Small sets: Analysis not cost-effective.
• Random access: Inferior to specialized storage.

Tool for versioned sets with strong overlap.

Positioning vs. Alternatives

maxpack combines features of packing and snapshot systems:

| Feature | tar+zstd/7z | borg/restic | maxpack |

|-------------------|-------------|-------------|-----------|

| Deduplication | No | Per-chunk | Yes |

| Solid compression | Yes | No | Yes |

| Result | Single file | Repository | Single file |

| Incrementality | No | Yes | Append |

| Encryption | Depends | Yes | AES-256-GCM |

Suitable for portable version archives.

Testing on Your Data

maxpack pack your_data/ -o test.maxpack
tar cf - your_data/ | zstd -3 > test.tar.zst
ls -lh test.maxpack test.tar.zst

Key Points

• Compression scales with versioned data volume up to 50x+.
• Deduplication up to 90% on highly repetitive sets.
• Unpacking 1.5–2.2x faster than tar+zstd.
• Archives 2–7x smaller than competitors on real projects.
• Support for incrementality and AES-256-GCM.

— Editorial Team