This year, an estimated 123.7 million people watched the Super Bowl. 120 million of those viewers watched the broadcast on CBS, making it the biggest audience for a single network live telecast ever. And for only the second time in history, the game went into overtime. All that, plus the flurry of social media engagement that happens during the game, means that advertisers will (hopefully) get a truck-load of engagement – and with that engagement comes a truck-load of data.
Chances are, your data infrastructure is designed to handle the amount of data you’re collecting and moving regularly. This makes sense – why pay for a massive U-Haul when you’re just moving a desk? And on a day-to-day basis, that’s how it should be.
But high-traffic events like the Super Bowl, a concert announcement, or a ticket release can easily break a system. Here are some ways to handle massive influxes of data gracefully, efficiently, and effectively:
1. Know that the data deluge is coming
As both Benjamin Franklin and your high school math teacher have (probably) said, an ounce of prevention is worth a pound of cure. The most important way to keep your system intact is to make sure you’re prepared to handle any predictable data influx, and you know what’s coming. It’s easy to find out when the Super Bowl is coming, or when high-demand concert tickets will be released to the public. Making sure your data team is apprised of marketing campaigns or product drops that will result in a lot of traffic is an important part of keeping things running smoothly.
2. Design a scalable infrastructure (and know how scalable it is)
The next step, though, is to have an understanding of how “stretchy” your system is in practice. When figuring out how stretchy your infrastructure needs to be, you can ask questions like:
- If we get an enormous amount of data very quickly, will we be able to handle it?
- Will we be able to scale our system on the fly?
- How much space can we add, and how quickly can we add it?
Understanding the capacity and capabilities of your system is key to keeping you afloat when a data flood comes.
3. Understand why you’re collecting data in the first place
When big data events happen, a common impulse is to collect as much data as possible, and figure out how to use it later. After all, it’s better to have too much than too little, right?
But if you don’t have a plan for the data you’re collecting, you may end up with a giant pile of information that will be incredibly daunting (and maybe even expensive) to sift through. And it may be so massive that it’s not possible to do much with it at all.
Instead of coming up with a plan retroactively, know what information you want and what you plan to do with it before any big data event. This will allow you to be more thoughtful about your data pipeline, and any data points you want to flag. Being strategic about what you collect in the first place will ensure that you’ll get the most out of your data.
4. Figure out where your new data is going to go
5. Make sure everyone understands your shiny new data
You’ve built a data collection strategy, decided how you’re going to store it, and you’re ready to dig in. The next question is – will your analysts understand it?
Especially if you’ve created a whole new system, or you’ve collected a ton of data from an unexpected event, your analysts need to be able to recognize patterns and spot outliers.
This can be done by carefully organizing what you collect, or by using a program like Pickaxe MIX that’s designed to analyze unique events within the historical context of your other data. However you do it, ensuring that analysts have what they need to process this new information is key to making it useful.
