NEES/NIS Diagnosis of Kalman Filters under Nonlinearity

# Diagnosing Kalman Filters under Nonlinearity and Non-Gaussian Noises: NEES/NIS Analysis of KF, UKF, and Particle Filter

The Kalman Filter remains a fundamental tool for state estimation in navigation, robotics, and signal processing tasks. However, its classical form assumes linear dynamics and Gaussian noises—conditions rarely met in real-world systems. This article examines how three popular approaches—the linear Kalman Filter (KF), Unscented Kalman Filter (UKF), and Particle Filter (PF)—perform when these assumptions are violated, and which metrics allow for an objective assessment of their performance.

Why Gaussianity Matters—and When It's Not Essential

The Kalman Filter is optimal in terms of minimizing mean squared error only in the linear-Gaussian setting. This follows from two key properties of the normal distribution: closure under affine transformations and conjugacy of the likelihood in Bayesian updates. When Gaussianity is violated, these properties are lost, but the recursion over mean and covariance remains computationally tractable. As a result, the KF continues to function, but its estimates are no longer optimal, and the covariance matrix may inaccurately reflect true uncertainty.

In practice, this manifests as overconfidence or underconfidence in the filter's estimates, which is especially critical in decision-making systems. For example, with outliers in measurements, the KF can get "clogged" and start diverging from the true trajectory, while adaptive methods maintain robustness.

Experimental Setup: 2×2 Factorial Design

A Python simulator using only NumPy and SciPy was implemented for systematic analysis. The architecture ensures experimental purity: filters receive only noisy measurements without access to the true state.

Simulation scenarios are organized as a full factorial experiment:

• Linear dynamics + Gaussian noises—baseline case.
• Linear dynamics + non-Gaussian noises—robustness check against anomalies.
• Nonlinear dynamics + Gaussian noises—test of nonlinearity handling.
• Nonlinear dynamics + non-Gaussian noises—stress test for all algorithms.

The nonlinear model is specifically designed to amplify effects:

x₁[k+1] = x₁[k] + x₂[k] + 0.5·sin(x₁[k])
x₂[k+1] = 0.8·x₂[k] + 0.2·cos(x₁[k])

Measurements include a quadratic dependency: y₂ = x₂² + η₂, which loses information about velocity sign and creates local bimodality.

Process noises are modeled as Gaussian mixtures (background + outliers), and measurement noises as Laplace distribution with heavy tails.

Evaluation Metrics: RMSE, NEES, and NIS

Three metrics were used for comprehensive evaluation:

• RMSE (Root Mean Square Error)—overall accuracy measure, but uninformative about uncertainty consistency.
• NEES (Normalized Estimated Error Squared)—shows how well empirical error matches predicted covariance. For a correct filter, NEES should follow a χ² distribution with n degrees of freedom (n = state dimension).
• NIS (Normalized Innovation Squared)—NEES analog for operational mode without ground truth. Based on innovations (differences between measurements and predictions).

Results showed that:

• In the linear-Gaussian case, all filters exhibit similar RMSE values, with NEES within the χ² confidence interval.
• With non-Gaussian noises, KF maintains functionality, but NEES systematically exceeds bounds, indicating underestimation of uncertainty.
• UKF handles nonlinearity better but is sensitive to non-Gaussian measurement noise due to retained Gaussian assumption.
• PF shows the best NEES consistency across all scenarios, especially with multimodal and heavy-tailed noises.

Practical Recommendations for Filter Selection

Based on the experiments, the following recommendations can be made:

• Use KF if the system is close to linear and noises only mildly deviate from Gaussianity (e.g., weak outliers). This ensures minimal computational cost.
• Switch to UKF if nonlinearity is significant but noises remain Gaussian or nearly so. UKF effectively approximates nonlinear transformations without dimensionality increase.
• Choose PF when noises are clearly non-Gaussian (outliers, multimodality) or observability is impaired (e.g., quadratic measurements). Drawback: high computational complexity and risk of particle depletion.

It's crucial to regularly monitor NIS in real systems: sustained exceedance of the χ² threshold signals the need to revise noise models or switch to a more flexible filter.

Key Takeaways

• The Kalman Filter doesn't "break" under non-Gaussian noises but loses optimality and consistency.
• NEES is the key metric for filter validation in simulation; NIS for real-time diagnostics.
• UKF compensates for nonlinearity but doesn't solve non-Gaussianity.
• The Particle Filter is the most versatile but requires careful tuning and substantial resources.
• The combination of nonlinearity and non-Gaussian noises poses the greatest challenges for all approaches.

— Editorial Team