Scaling Enterprise Machine Learning Through Governance & MLOps

Background

What is MLOps?

Why is Deploying AIML in Production So Hard?

Figure 1: Sample MLOps Workflow
  • Product & Program Management — person to rank AIML projects to ensure data science team focusing on highest-value AIML projects, not just the coolest (e.g. geekiest) or easiest to complete. Someone to track relevant milestones
  • Calculate ROI — determine metrics for measuring which AIML projects are the best investment for your company, evaluation of efficiency or profitability of production AIML
  • Cost Controls — putting guardrails in place to make sure AIML cost remain within budget, infrastructure components used optimally
  • Identity & Access Management — identify who is accessing what, auditing, prevent unauthorized access
  • Persona Enablement — ensure AIML platform meets current and future needs of personas, enables knowledge and asset sharing
  • Explainability — how to explain the underlying model logic, prove predictions aren’t biased and don’t violate compliance requirements
  • Model & Data Governance model risk management to govern development, validation, approval, modification, implementation, retirement and inventorying of models. Functional framework to manage data across the ten data management functions

Enterprise-Grade AIML Platforms

Figure 2: Example of an Enterprise AIML Platform’s Components by Algorithmia
  • Users: What personas do I need to enable? (e.g. data scientist, citizen data scientist, software engineers, business analyst, ML engineers, IT operations)
  • User Interface (UI) & Experience (UX): Does the platform offer intuitive, easy-to-use UI for technical, non-technical? Is training, support, enablement readily available for end-users?
  • Supported AIML Operations: What operation, should this platform support? (e.g. Collaboration. data ingestion, preparation, exploration and governance. Feature engineering, training, hyperparamater optimization. Model creation, testing, deployment and monitoring. Bias detection, explainability. Business value tracking)
  • Integrations & Interoperability: What applications should my platform integrate with?
  • Infrastructure: Managed vs unmanaged? What type of deployment technologies best meet existing skill sets? (e.g. Containers, VMs, Serverless). Should my platform be singe-cloud, multi-cloud, on-prem?

Determining ROI with AIML

  • Optimization Effort. Improves productivity, reduces cost with a 2x ROI on average. Example is a car manufacturer using sentiment analysis to classify customer support emails in queues making it easier for support staff to pick or prioritize the appropriate tickets for immediate response. Increase customer satisfaction, reduces churn, reduces average time support agents spend on tickets lowering the average OpEx cost to support customers
  • Improved Decision Making. Improves a customer experience, revenue or margin with a 10x ROI on average. Imagine the same car manufacturer using natural language processing to create a chatbot that can respond to customer’s support needs faster with reduced number of needed human support agents, further reducing OpEx
  • Business Model Innovation. Disrupts industries. Creates new markets, businesses or revenue streams with an average ROI of 100x. Picture the car manufacturer adding sensors to their cars and using sensor data with deep learning for object detection and classification to create a new autonomous vehicle driving services. New industry, new revenue streams, C.R.E.A.M. to the ceiling 🤑

Assess Your Organization’s AIML Maturity Level

Figure 3: Categories to Assess ML Maturity

Organizational Alignment

  • Low Maturity no dedicated data scientist, tools chosen ad-hoc by AIML practitioners, IT Operations (IT Ops) are mostly opportunistic
  • Medium Maturity multiple data science teams with tools chosen by needs, IT integrated AIML operations with some IT planning
  • High Maturity data science centralized, standardized platform and operations for management of tooling, IT Ops has established KPIs and understands AIML performance requirements across business groups

Data & Training

  • Low Maturity individual data owners, few or new data scientist using shared or personal file storage for data, limited IT oversight across ad-hoc projects
  • Medium Maturity department-level data administration, data scientist using general corporate shared file storage systems, IT Ops applying emerging data governance principles against concurrent AIML projects
  • High Maturity executive owner of data (e.g. Chief Data Officer), IT Ops creating data-first strategy, IT Ops understands challenges, complexities, and value of effective data management

Deployment & Operations Management

  • Low Maturity data scientist doing everything, limited tooling and workbenches, need to manually handoff, limited or no integration across tools
  • Medium Maturity more personas (e.g. data scientist, developers, DevOps) for deployment & operations, some automation of the AIML deployment process, platform-specific infrastructure management, IT Ops providing non-standardized workload-specific deployment automation
  • High Maturity multiple experienced deployment & operations personas with clearly defined roles and responsibilities, persona and role specific tooling, IT Ops providing one-button deployment, CI/CD pipelines, infrastructure agnostic tooling with performance monitoring

Governance

  • Low Maturity no governance from IT Ops, data scientist using spreadsheets, sticky notes or napkins to track efforts
  • Medium Maturity more personas involved with governance (e.g. DevOps, developers), reporting tools disconnected from other AIML efforts, ad-hoc management of AIML efforts from IT Ops
  • High Maturity executive ownership of governance (e.g. Chief Risk Officer), MLOps governance platform with IT Ops managing access, reporting and policies that integrate with existing IT Ops

Closing the Maturity Gap

Conclusion

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store