Let’s give the Modern Data Stack a Chance

Recently, many people who are deep into data science have been discussing the idea that the Modern Data Stack has outlived its usefulness. (Here’s an example.)

The arguments tend to go like this: when the idea of a Modern Data Stack was introduced in 2016, it was revolutionary, but as time has gone on and the term has become more ubiquitous, it’s now too vague to be meaningful. At this point, pretty much every tool is built for the cloud, and so what does MDS even mean anymore? Plus, they argue, Modern Data Stacks aren’t as effective as they could be. 

There’s a lot of focus on what the Modern Data Stack can’t do, and those critiques are valid. The MDS isn’t perfect, and it’s not a solution to every problem. It’s definitely not a panacea. However, it’s important to keep in mind that a lot of organizations – even big companies – haven’t even begun building a Modern Data Stack. 
But wait - haven’t Modern Data Stacks been around since 2016? How is this still new?
To explain what’s going in with the MDS today, let’s go back in time to when the concept was developed. Back in the 2010s, when “big data” was a buzzword de rigueur and cheap cloud storage became readily available, companies started collecting data at unprecedented rates in hopes that at some point, it would be useful. Traditional data stacks, which rely heavily on a company’s physical data center, weren’t equipped to handle that amount of information. This meant companies were collecting tons of data, but had no way to make use of it.

In response to this dilemma, startups began developing 3rd-party cloud-based tools to fill gaps in traditional stacks. Gradually, organizations began smushing these tools together into “Frankenstacks” – arrangements selected specifically for an organization’s needs, to make a unique infrastructure that could help them process the data they already had in cloud storage. This was revolutionary – suddenly, data that was congealing in a big data swamp could be useful.
So everyone uses a Modern Data Stack now, right?

Not even remotely.

While building infrastructure to handle data is of the utmost importance (at least, in our opinion), it can seem like a big and expensive project. Plenty of organizations have other priorities, and because maintaining a data lake is relatively easy and cheap, creating an MDS falls to the bottom of the to-do list. As a result, data scientists and analysts are left to figure out solutions on their own.

At Pickaxe, we’ve met many clients who were still using Excel to manage their data at the beginning of our working relationship. And some of these are large corporations working across multiple verticals. They’d been hoarding data in a big swamp for over a decade and hadn’t begun looking into it at all, mostly for cost reasons. They’re just now beginning to invest in this technology. Even though it seems passé to us, it’s brand new to them. 

In our experience, once we introduce the concept of a Modern Data Stack and begin building warehouse infrastructure, our clients are thrilled. They find that it solves problems they didn’t even know were problems. And it’s a massive improvement over what they were doing before.

Does “Modern Data Stack” still mean anything?

As time has gone on – especially in the last 12-18 months – many of the 3rd-party tools  commonly used in Modern Data Stacks have been acquired or copied by big companies. The term “Modern Data Stack” is cropping up in contexts that may seem irrelevant to those who are experts in the subject.

The ubiquity of the term might make it seem like MDS has become a buzzword rather than a salient concept, which could be an indication that the Modern Data Stack is dying

However, the ubiquity of the term “Modern Data Stack” could also imply that the concept is becoming mainstream. And working with big companies means tools can be souped up with more functionality – for example, you can have things like observability, lineage, and orchestration built into the tool, because there are more resources behind them. 

Is the Modern Data Stack perfect? Definitely not. However, as data scientists who are closely following the development of these tools, it’s natural to assume that they’re more prevalent than they are. 

It’s easy to hyperfocus on the next thing when you’ve got deep industry knowledge, and you know what’s happening on the bleeding edges of data. But outside of our specialized circle, most people haven’t put an MDS into practice, and many don’t even know what it is.

Before discarding the Modern Data Stack to move to the next thing, let’s try to put it into practice fully, and then think about what we can do to improve. 

The ubiquity of the term might make it seem like MDS has become a buzzword rather than a salient concept, which could be an indication that the Modern Data Stack is dying.

Explore more

learn more

Let us show you how we can help you make the most of your data.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Thank You!

We'll be in touch!


Let us give you a customized walkthrough of all of our platform’s capabilities. Reach out to learn more! 

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.