Anomaly Detection in File Storage Logs: Isolation Forest, CatBoost, and Autoencoder
Machine learning effectively identifies threats in file storage logs, where millions of read, write, and delete operations are recorded daily. Three algorithms—Isolation Forest, CatBoost, and autoencoder—analyze multidimensional time series of user activity, distinguishing normal behavior from suspicious patterns without labeled data.
Data is aggregated into time intervals: for each slot, the intensity of operations by type is counted. This results in a feature vector where deviations signal insider copying or mass encryption.
Isolation Forest: Isolating Anomalies with Randomness
Isolation Forest builds an ensemble of trees with random splits of features. Anomalies are isolated faster because their features differ significantly from the majority. The algorithm is unsupervised: it requires no labeling and focuses on the path from root to leaf.
from sklearn.ensemble import IsolationForest
if_model = IsolationForest(n_estimators=100, contamination=0.05, bootstrap=False, random_state=42)For time series, lag features are added:
data['prev_10min'] = (data.rolling(window='10min', min_periods=1)['events'].sum()).astype(int)
data['prev_5min'] = (data.rolling(window='5min', min_periods=1)['events'].sum()).astype(int)
data['prev_3min'] = (data.rolling(window='3min', min_periods=1)['events'].sum()).astype(int)Key Hyperparameters:
- n_estimators: balance of accuracy and speed
- contamination: expected proportion of anomalies (0.05)
- max_samples: subsample size (256 by default)
- bootstrap: False for diversity
The anomaly_score via decision_function is normalized [-0.5, 0.5]. Values < -0.25 are confident anomalies:
data['anomaly'] = np.where(if_model.decision_function(data[['events', 'prev_3min', 'prev_5min', 'prev_10min']]) > -0.25, 0, 1)SHAP analysis shows dominance of 10-minute lags. Advantages:
- O(n log n) complexity for real-time processing
- Runs on CPU without storing data
- Easy deployment
CatBoost for Predicting Normal Activity
CatBoostRegressor is trained on historical data to predict expected intensity. Anomalies occur where |actual - predicted| exceeds a threshold.
from catboost import CatBoostRegressor
model = CatBoostRegressor()
expected_value = model.predict()
anomaly_score = abs(actual_value - expected_value)Data Splitting Scheme:
- Training (3 weeks): normal activity
- Validation (1 week): tuning
- Test (2 days): anomalies
Features expand context:
- Temporal: day of week, hour
- Lag: events over 1h, 30m, 10m, 5m, 3m
- Rolling: mean/std/quantile(0.8) over 3/7 days
Rolling statistics are computed without leakage:
# Rolling statistics
data['week_mean'] = round(data.rolling(window=7, min_periods=1)['events'].mean(), 2)Rolling learning updates the model daily on the full historical pool.
Autoencoder: Compression and Reconstruction
A neural autoencoder compresses normal sequences into latent space, while anomalies are poorly reconstructed (high reconstruction error).
Architecture: LSTM/GRU encoder + decoder. Training on normal data minimizes MSE. An error threshold determines anomalies.
Advantages for series: captures long-term dependencies. Disadvantages: requires GPU, sensitive to hyperparameters.
Comparison of Approaches:
| Model | Speed | Interpretability | Resources | Adaptation to Trends |
|--------|----------|--------------------|---------|---------------------|
| Isolation Forest | High | SHAP | CPU | Lags |
| CatBoost | Medium | Feature importance | CPU/GPU | Rolling learning |
| Autoencoder | Low | Attention | GPU | Sequences |
Key Takeaways
- Isolation Forest leads in speed and cost-effectiveness for real-time use
- CatBoost is accurate in predicting trends with expanded features
- Autoencoder excels on complex sequences but is resource-intensive
- All models are unsupervised: they work without labeling
- Combining lags and rolling statistics improves accuracy by 15-20%
SIEM integration: models update online, alerts trigger on anomaly_score > threshold. Testing on synthetic anomalies confirms F1-score > 0.85 for all.
— Editorial Team
No comments yet.