Parallel Chunk Merging Manticore Search

Parallel Chunk Merging for RT Tables in Manticore Search 24.4.0

In Manticore Search 24.4.0, RT table compaction now runs in parallel. Disk chunk merges launch simultaneously, and each job can combine more than two chunks. Key parameters: parallel_chunk_merges controls the number of parallel merge jobs, while merge_chunks_per_job sets chunks per job. The OPTIMIZE command leverages a pool of background workers with N-way merging.

This fixes compaction delays under constant data loads with low optimize_cutoff. Tables are searchable right after inserts, but full optimization to the target chunk count used to take minutes.

Benchmark with 10 Million Documents

We tested an RT table with fields id bigint, name text (10–100 words), type int, and optimize_cutoff='16'. Loading via Manticore Load Emulator:

manticore-load \
  --cache-gen-workers=5 \
  --drop \
  --batch-size=1000 \
  --threads=5 \
  --total=10000000 \
  --init="CREATE TABLE test(id bigint, name text, type int) optimize_cutoff='16'" \
  --load="INSERT INTO test(id,name,type) VALUES(<increment>,'<text/10/100>',<int/1/100>)" \
  --wait

Old mode (parallel_chunk_merges=1, merge_chunks_per_job=2):

• Loading 10M docs: 1:18.
• Full compaction: 3:23.
• At 1:18: 53 chunks.

New mode (parallel_chunk_merges=3, merge_chunks_per_job=5):

• Loading: 1:18.
• Full compaction: 1:31.
• At 1:18: 23 chunks.

Win: total time down 55%, merge tail latency cut by 90%.

Storage Type Comparison

| Storage Type | Best Settings | Time | Worst Settings | Time | Improvement |

|--------------|-----------------------------------|-------|-------------------------------|--------|-------------|

| Row-wise | parallel_chunk_merges=4, merge_chunks_per_job=5 | 14:35 | parallel_chunk_merges=1, merge_chunks_per_job=2 | 67:15 | 4.61x |

| Columnar | parallel_chunk_merges=4, merge_chunks_per_job=5 | 15:10 | parallel_chunk_merges=1, merge_chunks_per_job=2 | 99:14 | 6.80x |

Optimal values: parallel_chunk_merges=4–5, merge_chunks_per_job=4–5. New defaults clock in at 1:57 for the same 10M docs.

Key tuning factors:

• Low merge_chunks_per_job enables more parallel jobs.
• High value speeds up individual merges but caps parallelism.
• Balance hinges on I/O and workload.

Parameter Balance

parallel_chunk_merges fires off multiple jobs at once. merge_chunks_per_job dictates workload per job. With many chunks, lighter jobs (low merge_chunks_per_job) free up resources for parallelism. Heavy jobs cut steps but hog the worker pool.

Benchmarks show 3–5 on both params minimizes time. For RT workloads with bulk inserts, this slashes optimization wait times without slowing inserts.

Key takeaways:

• Data is searchable immediately; compaction runs in the background.
• Parallel merging cuts tail latency by 90% (from 2:05 to 0:13).
• Up to 6.8x speedup on columnar storage.
• New defaults are 40% faster than old mode.
• Fewer chunks during loading (23 vs 53).

Tuning Recommendations

• Start with Manticore Search 24.4.0 defaults.
• Monitor chunk count and I/O during loads.
• Bump parallel_chunk_merges to 4–5 with spare CPU.
• Test merge_chunks_per_job=4–5 for large RT tables.
• Run OPTIMIZE to force compaction after batches.

For continuous insert streams with low optimize_cutoff, these changes keep tables stable without long-running background tasks.

— Editorial Team