Matrix downsampling filter 50M points

Matrix Filter for Pixel-Perfect Downsampling of Scatter Plots with 50M Points

The matrix filter delivers 100% coverage and visual fidelity when rendering 50 million pressure points from oil wells. Traditional algorithms like LTTB lose up to 59% of data density. This approach works in a single pass without sorting, capping output to screen pixel count.

Scatter plots are essential for irregular data with gaps: lines distort breaks and hide clusters. Standard methods are tuned for line charts, overlooking 2D structure.

Limitations of Traditional Algorithms

MinMax, M4, LTTB, and hybrids bin the X-axis and pick 1–4 Y points per bin. In dense columns, they drop 96–99% of data: clouds collapse to extremes.

• MinMax: min and max — captures peaks but ignores distribution.
• M4: first, last, min/max — pixel-perfect for lines.
• LTTB: largest triangle area — curve silhouette.
• MinMaxLTTB: locks extremes, then smooths.

For scatter plots, this is a core flaw: algorithms ignore Y-coordinates within bins.

Matrix Filter Principle

Like the Voxel Grid Filter in graphics: projects points onto a W×H pixel grid. One point remains per cell in traversal order.

public static void downsample(
        double[] times, double[] values,
        int width, int height,
        List<double[]> result) {

    int n = times.length;
    if (n == 0) return;

    // Step 1: bounds
    double tMin = times[0], tMax = times[0];
    double vMin = values[0], vMax = values[0];
    for (int i = 1; i < n; i++) {
        if (times[i]  < tMin) tMin = times[i];
        if (times[i]  > tMax) tMax = times[i];
        if (values[i] < vMin) vMin = values[i];
        if (values[i] > vMax) vMax = values[i];
    }
    double tRange = Math.max(tMax - tMin, 1e-10);
    double vRange = Math.max(vMax - vMin, 1e-10);

    // Step 2: grid
    boolean[] grid = new boolean[width * height];

    // Step 3: filter
    for (int i = 0; i < n; i++) {
        int px = (int) Math.min(width  - 1, (times[i]  - tMin) / tRange * (width  - 1));
        int py = (int) Math.min(height - 1, (values[i] - vMin) / vRange * (height - 1));
        int idx = py * width + px;
        if (!grid[idx]) {
            grid[idx] = true;
            result.add(new double[]{times[i], values[i]});
        }
    }
}

The method mimics rendering: lights up pixels that would glow with the full set. It's strictly for visualization, not statistics.

Testing Methodology

~3000 well telemetry datasets: 19k to 50M+ points. Pixel-by-pixel comparison with originals in scatter (2×2px) and lines (1px) modes.

Metrics:

• Coverage: % of original pixels covered by the algorithm.
• Precision: % of algorithm pixels in the original.
• F1-score: harmonic mean.
• Visual Score: F1 with ±1px tolerance (subpixel accounting).

Classic algorithms budgeted to match MatrixFilter output at 1920×1080. Benchmark: i7-7700K, Java 17, single thread.

Results on Real Data

Regular (~20k points, 2930 files)

Scatter: MatrixFilter 1920×1080 — 100% across all metrics. MatrixFilter 800×600 — Visual 99.6%, half the points. Classics: Coverage 63–84%, Visual 76–95%.

| Algorithm | Avg Out | Total ms | Cover | Prec | F1 | Visual |

|------------------|---------|----------|-------|------|------|--------|

| Matrix 1920×1080 | 4939 | 695 | 100% | 100% | 100% | 100% |

| Matrix 800×600 | 2867 | 497 | 84.6% | 100% | 91.5%| 99.6% |

| MinMaxLTTB x4 | 4932 | 494 | 84.1% | 100% | 90.9%| 95.2% |

Medium and Large

On 500k–50M points, Matrix holds 100% Visual at ~ms/10k points. LTTB drops to 16–40% Coverage on dense data.

Key Takeaways

• Matrix filter delivers pixel-perfect scatter without sorting, O(n) time.
• Traditional algorithms lose dense cloud structure, even with generous budgets.
• Visual Score with ±1px matches perception: Matrix 800×600 is indistinguishable from full render.
• Strictly for visuals: use other methods for lines/stats.
• Tested on industrial oilfield datasets with irregular gaps.

— Editorial Team