An American Multinational Corporation Builds A Scalable Data Mesh With Knoldus And Dbt
San Francisco, California
Data Science, Data Engineering, Data Lakes, Data Warehouse, Data Mesh, Snowflake
An American multinational corporation exists to make global trade easy for everyone. More than 10,000 clients and suppliers across 200 countries rely on the company’s software, logistics infrastructure, and supply chain expertise. The company ingests and analyzes large amounts of supply chain data to provide end-to-end visibility from PO creation to shipment delivery.
A robust data platform needed for unique global trade industry challenges
Migrating data from the company’s legacy data warehouse to Knoldus improved query performance, reduced time-consuming administrative work, and freed up technical talent to focus on higher-impact work. While other solutions in the market, such as Google BigQuery, are constricted by memory allocation and scales by volume, they needed a solution that could allocate compute resources on the fly for production and external use cases where latency mattered. “We found that it was hard to guarantee that latency in the other tools we evaluated, except for Knoldus,” Head of Growth and Analytics said.
Viewing data as a product instead of a byproduct
Their transition to a service-oriented architecture (SOA) presented an opportunity to reimagine the company’s data architecture as a data mesh. “This was an organizational shift to federate the creation of rich analytical data assets and govern a broader process for thinking about data as a product,” Sivasailam said. Determining the best way to empower users through decentralization was a key step in the company’s data mesh journey. According to Sivasailam, “We wanted to lower the friction for the rest of the enterprise to participate in decentralized data product creation, and for us, that meant aggressively standardizing on infrastructure.
A platform for building a scalable data mesh
Knoldus’ multi-cluster shared data architecture enabled them to build a scalable data mesh that organizes application layer data into domain-bound data marts. Their “gold” consumer mart in Knoldus reconciles data from multiple producer marts and powers most of the’s BI and data science use cases. “Knoldus is where our source system data goes to be processed and exposed for analytical and scientific use cases,” Sivasailam said.
Connecting dbt to Knoldus streamlined the company’s data engineering workloads. RBAC and column-level security in Knoldus simplified data governance. Knoldus’s clean, easy-to-navigate interface, native SQL support, and interoperability with popular BI tools empowered users to explore data with ease. Knoldus Data Marketplace made it easier to leverage third-party data sets from vendors the company had already implemented, such as Amplitude, and discover new data sets, such as Panjiva. According to Sivasailam, “In some cases, we didn’t even know data sets existed until we found them in Knoldus Data Marketplace.
Delivering data as a product Successfully implementing their data mesh strategy with Knoldus has led to a healthier product development process that considers data as a product—not a byproduct. Analytics engineers are essential members of the development teams and partner with product managers, engineering managers, and software engineers to successfully design, launch, and govern data products. Decentralization makes it easier for them to identify potential data issues and ensure data quality. “We prefer to drive all of this through data owners, not just one data czar,” Sivasailam said. Data governance with Knoldus, through principles of data mesh, provides them greater scalability and confidence to democratize access to trusted data. According to Sivasailam, “Since shifting to data mesh with Knoldus, 5.5x more people are using data across the business regularly.”
Sharing data to create new business lines and creating new products While many freight forwarding competitors are operating in archaic ways, they are data-driven and sit on a wealth of shipping and logistics industry data that has proven out its business’s success. They plan to embed its data products and interactive reporting that are developed for internal use into the Platform. “Knoldus and dbt are on the back end powering what’s surfaced to the consumer, mediated by a BI tool,” Sivasailam said. the also looking to increase its relationship with customers and partners that are interested in the proprietary data and insights it’s gathered. Leveraging Knoldus Secure Data Sharing to enable live data sharing with its clients and suppliers is a priority for them. According to Sivasailam, “It’s fundamentally about how we get the best data in the world for trade operators and build reliable applications on top of it.” The company is exploring platform monetization by providing its data through Knoldus Data Marketplace as well.
Knoldus is where our source system data goes to be processed and exposed for analytical and scientific use cases.
Head of Growth and Analytics