Company

CP-TO Watchdog Leading Indicator

A leading indicator and alert pipeline that catches operational issues before the lagging backlog does

Period | 2026.01 ~ 2026.02

  • #Python
  • #Grafana
  • #Google Sheets
  • #SQLite

Operational issues that only showed up after the backlog grew

Problem

  • The existing trigger relied on backlog, a lagging metric, so issues were detected late
  • Per-region metric lookups triggered repeated N+1 queries

My Role

  • Designed the Target On Duty formula to detect anomalies earlier than backlog
  • Built a pipeline of Grafana metrics, Google Sheets thresholds, a 20-minute cooldown and logging
  • Restructured per-region reads with SQLite composite indexes and batch queries

Result

  • Backtest on Gangnam data: 13 minutes earlier than the existing trigger on average, AD10 diff +28.98%
  • Per-region N+1 queries cut from 100 to 1
  • Measured a 3.5% false alarm rate and defined tuning work toward under 1%

Technical Decisions

  • Target On Duty as a leading indicator

    Backlog only rises after problems pile up, so an earlier signal was defined as the metric

  • SQLite composite indexes and batched reads

    Replaced per-region N+1 queries with a single read

Running in Production

  • 20-minute cooldown so the same alert does not repeat
  • Measured a 3.5% false alarm rate and set tuning work toward under 1%
AS-IS / TO-BE: from a backlog trigger to the Target On Duty leading indicator
AS-IS / TO-BE: from a backlog trigger to the Target On Duty leading indicator