Business Objective
Our client, a leading insurance provider in the US, had over 200 predictive models deployed in production that were monitored and retrained in silos.
The client’s data science team partnered with Tiger to accomplish the following objectives:
- Develop a framework that underlies a consistent and ongoing model management solution
- Build an automated solution to significantly reduce the time taken to monitor and re-train models
Challenges
- Exceptionally fragmented and laborious model management process, which required reaching out to multiple stakeholders to align objectives and solution approach
- Lack of any model inventory that could help the team gain an initial understanding of models
Solution MethodologyÂ
- Model Inventory: Captured business purpose, data source information, model process flows, quality check measures, SOPs (standard operating procedures), and made them available through a web-based tool
- Data and Model Monitoring System: Captured data health reports, model execution status, model run times, model execution failure statistics, and notification/alert system
- Model Retrain System: Included a web-based application in Python Flask that orchestrated the overall automation – data extraction, feature engineering, EDA, model retrain, result validation
- Visualization Suite: Provided dashboards with an ability to drill down on model failure details in Kibana, Elastic Stack – such as failure instances, stage of failure, causes
Business Impact
- Migrated hundreds of models to the new, custom-built platform for ongoing monitoring
- Reduced effort by more than 90% allowing data scientists to focus on core development
- Saved license costs by using open-source technology
