HALO - Infrastructure Monitoring

The client operated a large-scale SaaS platform serving 2,000+ enterprise customers across multiple AWS regions. Their monitoring stack — a patchwork of Datadog, PagerDuty, and custom CloudWatch alarms — was generating approximately 12,000 alerts per day. The operations team had determined through careful analysis that 97% of these alerts were false positives or non-actionable noise: threshold breaches that self-resolved, correlated symptoms of a single root cause firing as independent alerts, and alerts on metrics that had drifted from their original baselines without threshold updates.

The human cost was severe. The SRE team of eight engineers was spending 60% of their time triaging alerts rather than improving infrastructure. Mean time to resolution (MTTR) sat at 47 minutes because engineers had to wade through noise to find the signal. Alert fatigue had set in — the team had unconsciously developed a habit of dismissing alerts that "looked routine," which meant real incidents were occasionally missed. Two P1 incidents in the previous quarter had been caught by customers before the on-call team, resulting in SLA breaches that cost the company $180K in credits.

The client had attempted to solve this with increasingly complex static thresholds and alert routing rules. They had over 3,400 alerting rules in their system, many of which contradicted each other or referenced deprecated infrastructure. The maintenance burden of the rules themselves had become a full-time job. They needed a fundamentally different approach — one that could learn what "normal" looks like for each metric, adapt to seasonal patterns, and correlate related alerts into actionable incidents automatically.

The platform is architected as a streaming data pipeline built for real-time processing at scale. The ingestion layer collects metrics via a Kafka-based message bus that handles 50,000+ metrics per second. Prometheus exporters, CloudWatch metric streams, and custom application instrumentations all publish to topic-partitioned Kafka queues. A Flink-based stream processor performs real-time aggregation, downsampling, and feature extraction before routing processed metrics to the analysis layer.

The analysis layer runs the ML ensemble as a set of containerized microservices on Kubernetes. Each model type (STL, isolation forest, LSTM autoencoder) runs as an independent service with its own scaling characteristics — the LSTM models are GPU-accelerated for inference speed, while the statistical models run on CPU-optimized instances. Model outputs are aggregated by an ensemble coordinator service that applies voting logic and anomaly confidence scoring. The correlation engine runs as a separate stateful service that maintains a sliding window of recent anomalies and applies graph-based correlation algorithms using the infrastructure topology as a constraint graph.

The storage layer uses TimescaleDB for time-series metric storage with automated retention policies, PostgreSQL for incident records and correlation metadata, and Redis for real-time state caching (current baselines, active incidents, model feature buffers). The presentation layer is a React dashboard served via FastAPI, with WebSocket connections for real-time alert streaming. The dashboard provides a topology-aware incident view, metric drill-downs with anomaly overlays, and a correlation explanation panel that shows why specific alerts were grouped. The entire platform runs on AWS EKS with Terraform-managed infrastructure, and model retraining is orchestrated via Airflow on a weekly cadence.