Automated Annual Model Monitoring to Drive Efficiency at Scale

Business Objective

Our client is a leading credit reporting agency that offers credit bureau and decision analytics services to businesses, along with providing credit reports to individual customers. The client conducts periodic model monitoring exercises for various generic and custom scorecard models for multiple institutions.

They want an Automated Annual Model Monitoring tool. The objective was to streamline, automate, and sequence the entire exercise of model monitoring so that it can run independently as a single process with minimal intervention and generate error-free results. The result is a distribution-ready set of summary reports and associated commentary on model performance and validation metrics.

Challenges

The exercise involved validating over one hundred models for consumers and small businesses for different product portfolios, coming from disparate data sources
The existing process had multiple steps that required manual intervention at various steps, leaving room for errors
Several hundred different programs used hardcoded references and paths. Hundreds of intermediary files were generated and had to be checked manually before creating validation metrics

Solution Methodology

Restructured the code and streamlined the data collection process with in-built quality checks at each stage of data in motion
Different analytical platforms such as SAS and Python were streamlined with Microsoft Office integration tools so that there was no manual editing, hardcoding or intervention required
Generated a dashboard so that model performance can be easily shared with stakeholders
Automated creation of validation presentations with commentary and analyses updated at regular cycles

Business Impact

Increase of efficiency by over 92% (from 12 weeks to 1 week) for the model monitoring exercise
Reduction of defect opportunity by minimizing manual steps

Privacy Notice Cookie Policy