How should you set up your BI data stack?

If you’re a fast-growing business, chances are that you have data scattered in multiple tools, across different departments. This fragmented set-up makes it difficult for business leaders to get a clear picture of what’s happening in the organization, and how to steer their business in the right direction.

Data is a treasure trove of gold that business teams can do wonders with – if they are able to consume it properly. If it isn’t organized, how can they make sense of it and effectively make use of the data so that it adds value? How can business teams make data-driven decisions? It’s only going to get worse over time as it continues to sit in this mess, so if you haven’t already, it’s time to do something about it!

Fortunately, the answer isn’t rocket science: create a data stack that serves as your Business Intelligence architecture. This architecture is a unified framework that encompasses the various technologies that are implemented in order for data to be visualized and analyzed. It’s a full flow of the IT systems and software tools that are responsible for one or some component(s) of the overall stack – for example, collecting, storing, or analyzing the BI data. When the stack in place, that’s when business users can actually derive value from data.

As always, it’s easier said than done, so let’s take a look at the different components that make up a BI data stack. For clarity, we’ll split this up into 2 sections: 1) the technical components of the stack itself, and the 2) functional components around methodology and organization - essentially, how to make better use of the technical components to analyze data more successfully.

Technical components:

1. Source systems: gather data from the various scattered data sources, such as databases, CRM and ERP systems, files, APIs, marketing stack, and more. This is where data is already inputted and exists - a key starting point since data needs to be collected and centralized before it can be transformed and turned into anything useful.
2. Data integration: stepping stone to centralizing data. The collected data is integrated into a centralized system, most commonly through an ETL process (Extract, Transform, Load). “Extract” is when data is pulled from external sources; “Transform” is when data is processed and adapted to new standards according to the warehouse; “Load” is when data gets loaded into the data warehouse. Solutions like Fivetran, Rudderstack, Stitch, Segment, Portable and tackle this step specifically. There are plenty of bespoke ETL software available, as shown in Portable's comprehensive list of Top 100 ETL Tools. Integrated business intelligence platforms like Whaly also come with pre-existing data connectors to a wide variety of sources to make this a seamless process.
3. Data storage: through the ETL process, the data enters a data warehouse, which is a central repository where data is processed, stored, and cleaned. Data warehouses must scale as usage goes up, and are fundamental to the success of your BI projects. Thankfully, they’re now cheaper and more accessible than ever. Data warehouses like Snowflake, BigQuery, and Redshift allow for 100% transparency over your data, as opposed to clunky BI solutions that may store your data in unknown places. Data storage is an increasingly sensitive topic, so make sure you have full transparency on where you and your customers’ data lives.
4. Transformation: where the action begins. Raw data in the data warehouse gets transformed into actionable business data. Solutions like dbt, Whaly, y42 help with this stage.
5. Data analysis: this step allows users to consume data from the warehouse without technical knowledge. With the assistance of the right tools, the data is visualized, usually in the form of graphs, dashboards, and charts, and digested so business teams can start making sense of it. Many tools exist, such as Google Data Studio and Metabase, but self-service business intelligence platforms like Whaly take this process to the next level in terms of user-friendliness and ease of use - offering features that help people wade through their data more efficiently and showcase it in clear, meaningful ways.
6. Ad-hoc data discovery: data analysts can then experiment and start answering new business questions in a sandbox. Solutions like Hex and PopSQL help with this.
7. Data cataloging: an organized inventory of data assets within the organization, empowering data analysts to quickly find the data they are looking for. This can be done with the help of solutions like Castor, Atlan, and Amundsen.
8. Data distribution: the visualized data can then be distributed in different formats, like dashboards, to inform strategic business decisions. With Whaly, it can be pushed in the format of your choice or your stakeholder’s choice, to be consumed in the best-suited way.

Functional components to set you up for success:

1. Set up your team: identify at least one single point of contact that will own the data. This person or team will be responsible for data topics, defining data metrics, and fielding requests in a centralized way.
2. Define key success metrics with business users: at the end of the day, it's the business users who will use the solution and get the answers they need from the data. Defining success metrics in advance allows you to align business teams on why the project is being put in place, and any expectations around the project. This step is often overlooked, but it’s an important one for business teams to weigh in on the metrics and information they need.
3. Iterate with the business teams: the business teams should participate or even lead a metric definition workshop. After all, they’re the end users who know best about the business needs. The data team should provide inputs to help them understand what’s feasible in the current state of the solution. The more you work together with the business teams throughout the entire process, the higher your chances for a successful project. In parallel to the analysis stage, business teams should experiment with the data and give feedback to the data teams. Platforms like Whaly even allow business teams with no coding experience to play around with data in a safe way and drill down to better understand the figures. You can actually allow business users to go wild, without them bugging you for help or messing up the tool. What’s more, you won’t have to spend hours cleaning up the mess the business users might have made.
4. Measure the impact of the data project on your company: how will you know if it’s successful or worthwhile without measuring? Keep track of the number of insights that have been generated, and the ROI of the project.

The end goal is to land on actionable insights derived from data, and for the business to leverage them to make data-driven, informed decisions on company finances, revenue, and overall growth.

We hope this was a helpful breakdown of all the pieces of the puzzle in your stack, and how to achieve the best outcomes from an organizational standpoint. Different tools exist for each step, but finding an integrated platform that checks off at least a few of these is a good idea for driving efficiencies and maximizing data adoption across business teams.

That said, it’s important to connect your visualization strategy with your ETL strategy as best as possible. These days, nested data structures are becoming increasingly common but can pose a challenge to modern data platforms that aren’t equipped to analyze them. Advanced self-service visualization capabilities can allow you to analyze nested data sets without needing to think twice about the discrepancies and how it might be stored and processed, but some solutions may not offer this, so be aware.

This may be confusing as a ton of data tools are out there (Fivetran, dbt, Looker, Tableau), each tool covering one or some of the stack components - do your research to figure out which step the tool tackles. An integrated, user-friendly platform is the way to go to be covered on as many of these steps as possible, and ensure that data can be integrated and visualized seamlessly without issues, by both data and business teams.

To learn more about integrated and self-service BI platforms, or to dive deeper into the stack, we’d love to hear from you! Get in touch here: https://whaly.io/form/discover