Our client is a leading US-based logistics service provider offering logistics management, trucking, warehousing, freight forwarding, brokerage, supply chain management, distribution services, etc.
The client had multiple data sources feeding into their data lake but the data dictionary and understanding of data lineage was missing. They didn’t have a silver layer having clean and standardized data before moving into the data lake. Besides, the performance of Databricks and Power BI was not good. All of these led to an incomplete road map for data lake adoption. The client wanted our help in cleaning the data from the raw/bronze layer and move it to the silver layer from where the data can be pulled by the data science and analytics team.
Fixed multiple challenges related to data architecture and integration including data quality issues, data catalog, automated code deployment/management, etc.
Designed “Data Quality Framework” for automated data quality monitoring, including alerts when thresholds are exceeded