High-Performance Analytics for High-Volume Retail
The Context
ShopScale S.r.l. manages over 50 e-commerce brands tailored for the European market. They needed a unified "Mission Control" dashboard to visualize revenue, inventory, and user behavior in real-time during high-traffic events like Black Friday. Existing tools suffered from data latency of up to 15 minutes—consequences that meant lost revenue during flash sales.
The Engineering Challenge
The system needed to ingest, aggregate, and visualize 50,000+ events per second with sub-50ms latency for dashboard updates, while maintaining strict data consistency for financial reporting.
Architectural Solution: CQRS & Event Sourcing
We moved away from a traditional CRUD architecture to a Command Query Responsibility Segregation (CQRS) pattern.
1. Write Side (High Throughput)
- Events are ingested via a high-performance Golang service.
- Buffered into Apache Kafka for durability.
- Processed by workers that update the source-of-truth PostgreSQL database.
2. Read Side (Low Latency)
- Materialized views in Redis are updated asynchronously.
- The dashboard frontend connects via WebSockets to receive delta updates only.
- Complex aggregations are pre-computed using Tinybird for millisecond-speed queries over billions of rows.
Tech Stack
- Frontend: React, D3.js for custom visualizations, Redux Toolkit
- Backend: Node.js (WebSocket server), Go (Ingestor)
- Data: PostgreSQL (TimescaleDB), Redis, Tinybird
- DevOps: Kubernetes (EKS), Prometheus, Grafana
Outcome
- Zero latency: Dashboard updates effectively in real-time (< 200ms lag).
- 100% uptime during Black Friday 2024 peak load.
- Granular insights: Enabled marketing teams to adjust ad spend dynamically based on real-time inventory levels.