Alerts & Monitoring is a key best practice for any data pipeline. But sometimes data pipelines can seem like they’re working, not trigger any alerts, and actually have something massively wrong. This is where we recommend Impossible Reports.
Quite simply, these are reports you set up in any of your existing reporting tools (Amplitude, MixPanel, Tableau, Looker, Pickaxe, etc) that are designed to capture something that couldn’t ever happen in real life. For example, a negative number of gross new viewers! Or more Checkouts than Add to Carts! Or fewer Brand New Donations than Brand New Donors!
These reports should be blank or empty every single day, so you won’t have to monitor them. But you should set up notifications so that if they ever do populate, they’ll act as a canary in the coal mine, alerting you that something in your data is getting tracked incorrectly.
How Do You Monitor The Impossible?
- In your reporting tool, create a folder called “Impossible Reports” and save a bunch of these, particularly for any metrics that depend on multiple systems, or where you’ve had upstream data quality issues in the past.
- You can also create “Very Unlikely Reports” that are conceptually similar but register thresholds that seem unlikely (like if your Funnel normally converts at 25%, this might fire if it converts at 12%).
