A Systematic Approach to Load Testing: From Token Failures to Stable Metrics
The backend, acting as a proxy between the frontend and an external service, cached an access token via opcache. Under load, several parallel requests simultaneously detected an expired token and attempted to refresh it. The old token overwrote the fresh one in the cache, leading to 403 errors.
Solution: We added a lock for token refresh and moved the cache to clustered memcached for a unified state across processes. Testing in Postman with parallel requests confirmed the race condition was eliminated.
Metrics before: Response time grew exponentially, success rate dropped below 90% at 300 RPS. After: Success rate 100%, 5xx errors <0.05%, average response time 99-104 ms at 226 RPS.
Backend Preparation: Optimization Checklist
Before load testing (LT), a backend developer follows a systematic checklist. This minimizes bottlenecks at the code and infrastructure levels.
Database Optimization
- Index all fields in WHERE and JOIN clauses.
- Analyze EXPLAIN for slow queries.
- Master-slave replication, connection pooling (e.g., pgbouncer or equivalent).
Caching
- Memcached for repetitive data.
- HTTP response caching in Varnish or nginx.
- Twig template caching.
Asynchronous Processing
- RabbitMQ or Redis queues for background tasks.
- Offload email sending, PDF generation to background jobs.
PHP Code
- Eliminate N+1 queries.
- Optimize loops, avoid deep recursion.
- Free memory with unset() for large objects.
Infrastructure
- php-fpm: pm.max_children > 100, pm.max_requests = 500.
- MinIO for file storage.
- HPA in Kubernetes based on CPU/RAM.
- Rate limiter (nginx or Traefik) for throttling.
Frontend: Reducing Incoming Load
The frontend generates the main traffic. Optimizations here can reduce backend RPS by 30-50%.
- Static assets: nginx cache for JS/CSS/images.
- Client-side API cache: localStorage or memcached for rarely changing data (e.g., catalogs, configs).
- Pagination: Use limit/offset or infinite scroll instead of full data loads.
- SSG/ISR: Next.js for static page rendering.
- SSE/Webhooks: Replace polling with push notifications.
Load Testing Workflow
LT is mandatory for features that change server logic, caching, or databases. Conduct it on staging/production at night during minimal user load.
- Planning (one week ahead): Lead + PM + DevOps estimate target RPS (N requests/sec).
- DevOps role: Configure pods (CPU/RAM), HPA, real-time monitoring.
- Execution: Use k6 or similar, with scenarios for API endpoints.
- Monitoring: Grafana (RPS, latency, error rate), Graylog for logs.
- Analysis: Identify degradation points, fix bottlenecks, repeat.
Success criteria: Success rate >99%, latency <150 ms at target RPS with a 20% buffer.
Key Takeaways
- Process races (e.g., tokens, sessions) are only revealed under parallel load.
- Clustered cache (memcached) instead of local opcache prevents race conditions.
- LT on prod-like environments: database replication, external services.
- DevOps in the loop: Dynamically adjust resources during testing.
- Frontend optimizations reduce load more effectively than backend fixes.
Success Metrics and Benchmarks
Example from Grafana after fixes:
| Metric | Average | Peak | Target |
|--------|---------|------|--------|
| Success Rate | 100% | 100% | >99% |
| 5xx errors | 0.017% | 0.042% | <0.1% |
| RPS | 226 | 321 | 300+ |
| Latency (ms) | 99-104 | 123 | <150 |
Stabilization at 226 RPS without degradation. Failure point: ~450 RPS (100% buffer).
Recommendations for Middle/Senior Developers
- Integrate LT into CI/CD: automated tests on merge.
- Monitor p99 latency, not just average.
- Test worst-case scenarios: 80% traffic on a single endpoint.
- Log request IDs for tracing (e.g., Jaeger/OpenTelemetry).
This approach identifies issues during development, reducing incidents in production.
— Editorial Team
No comments yet.