The modern data stack (MDS) refers to a collection of cloud-based tools and technologies that organizations use to manage, store, process, and analyze data. It represents an evolution from traditional on-premises data systems to a more scalable, flexible, and efficient approach enabled by the cloud. While the modern data stack offers robust data capabilities, it also has limitations.

“The future is already here, it's just not evenly distributed.” ― William Gibson

You’ve probably heard this quote before. It’s almost a platitude in silicon valley, but it’s a good reminder that your reality is not necessarily shared by others. In major tech hubs, all we hear and talk about is the newest trends, and are generally uninterested in the old. This is great for building new innovative technologies, as it fosters a large dense market of early adopters. However, it’s important to remember that just because some technology is ubiquitous in San Francisco, it doesn’t mean that it is elsewhere in the country.

Let’s take general cloud adoption as an example. AWS launched in 2006. Almost two decades later a little more than half of enterprise IT environments are in the cloud (vs. on prem), and only 13% of enterprises have a fully deployed multi cloud infrastructure/architecture. Although cloud (and multi cloud) environments are clearly the future, it takes a long time for this trend to “distribute evenly”.

Now, let’s look at the modern data stack, which consists of cloud-native data tools. Companies building products in the data stack have raised more than $220B in the last 4 years alone and the space has seen multiple decacorns reaching $500M ARR. Gartner estimates that data management is the largest infrastructure market and by 2026, it’s expected to grow to $200B and represent 32% of the total infrastructure market.

Clearly a lot of companies are buying a lot of cloud data management products, and if you’re somewhere like San Francisco, cloud data products (or the modern data stack) are considered industry standard by most. At this point, these tools may even be old news for some folks, as they start to look ahead to what’s next (e.g. data mesh). However this is not the reality that most others live in. It will take time to distribute evenly. In the following sections we’ll get into the specific shortcomings of MDS and reasons why MDS doesn’t replace the need for embedded analytics. Howevert, it’s important to remember that most companies cannot solve their data needs with a modern data stack becasue they simply have not fully adopted it yet.

The evolution of the modern data stack (MDS) has generally made data operations easier, faster, and more reliable. For data practitioners and the business as a whole, the modern data stack provides a ton of benefits. However for the average employee not much has changed, they still don’t have access to data infrastructure, if they did they are unlikely to know the data well enough to navigate it, and if they had the needed access and context they’d still need to know sql (or python) – to be clear, the MDS is not just one thing, but nearly every component is designed with data professionals in mind, and would be very difficult for a non-technical business user to navigate.

There’s a new wave of BI tools built for non-technical users now, but 1) adoption is still very early and it’s not clear how good these tools are, and 2) even if someone had clean controlled access to a tool like this, it still would be at a disadvantage to native analytics in the product(s) they’re already using. This leads to the next modern data stack shortcoming – users don’t want to context switch, and generally don’t want to adopt new tools.

Although MDS has made centralizing data & BI much easier for companies, the desire from users to access analytics in the tools they already use is unlikely to go anywhere. This is illustrated by the rise in popularity of reverse ETL tools, like Hightouch, Census, and Rudderstack, which sync data directly from a data warehouse to the operational systems used by your marketing, advertising, and operations teams (aka the SaaS apps they already use).

The adoption of reverse ETL tools suggests that most businesses still need to move data out of a centralized stack back out to the various tools that users actually use every day, since there’s a massive adoption advantage to having analytics within your regular tools. Not only is there no need to switch between apps but the user is also able to see analytics along with operational context.

Another issue your users or employees may face is dependence on internal data teams. The quality and availability of analytics depends heavily on the internal data team's expertise and resources, which can vary significantly. Data teams across industries have increased in headcount and efficiency (often thanks to adopting MDS). However, they’re still usually limited from both a resource and domain expertise perspective, which creates a bottleneck within most enterprises.

From a resource perspective, data teams have to deal with skill gaps and competing priorities. When data teams are the middleman between users and the data stack, all requests go through them. Top-level executives may have higher leverage when it comes to getting analytics requests handled quickly by an internal data team, but most users request’s will go to the bottom of the queue. At worst it may never get turned around, and even if it does there could be a delay of days to weeks, at which point the result may no longer be as useful or a decision has already been made.

From a domain perspective, data teams only have so much context and expertise. Data teams will usually have particularly good insight into their own business, but this doesn’t necessarily mean they understand marketing analytics or sales forecasting particularly well. They’ll never know the intricacies of how a specific organization (e.g. sales, HR, support, or product) works as well as the actual employees in that org or the vendors who serve that specific employee. The data team will also have a more shallow understanding of application specific data – they won’t know the details of how this data was created, stored, and processed, before it was ingested by them.

Embedded analytics does not face these bottlenecks. Analytics are always up-to-date (often real-time), and are maintained by the SaaS vendor who understands the entire lifecycle and lineage of this data. Additionally, the SaaS vendor has a deep understanding of their customers’ use-cases and best practices, plus they possess a wealth of data across their customer base. This allows them to aggregate and analyze data at scale, and to identify trends, benchmarks, and patterns that individual customers might miss. This domain specific knowledge and data enables them to provide targeted insights that address industry-specific challenges or optimize domain-specific workflows.

Ultimately, the popularity of the MDS is well founded and brings a number of benefits to data practitioners and the businesses they serve; however, it doesn’t exactly solve data access or analytics for the average user or employee. This is because it doesn’t provide a comfortable or familiar UX for most users, it doesn’t natively integrate data into a user’s existing workflows and context, and the insights themselves may be limited by the data team’s domain knowledge or expertise (or lack thereof).

For these reasons embedded analytics experiences that are carefully tailored to the end user will always have structural advantages. As a consequence, rather than replacing the need for embedded analytics, the MDS is actually complementary to and enhances embedded analytics.

If you’re a SaaS vendor considering embedded analytics, book some time with Quill. We’re happy to chat, learn more about your business, and share best practices. If you’re already interested in embedded or customer-facing analytics and are evaluating solutions, take a look at our docs or book an intro call with us here.