AI value: The growing AI operations challenge

AI projects are successful only if they keep working after they are deployed.

It may sound obvious, but one of the persistent obstacles to realizing value from AI implementation is a tendency to treat its deployment as a finish line, not a starting point. Organizations often invest heavily up front in AI systems, and then expect the same teams and processes that built and launched them to keep them running, if they invest in this stage at all. But running AI solutions reliably, at scale, with maintainable systems and predictable outcomes is a specialized discipline which requires its own expertise, processes, operational cadences, and technology.

To consider what this looks like in practice, imagine an insurer deploying a hybrid workforce with AI agents working alongside humans. The AI handles most customer calls through generated voice and an agentic workflow that decides claim validity, payout, and messaging. At launch, everything looks promising. But then a customer complains that the agent was rude. Another cannot reach the helpdesk. Another receives a payout ten times higher than expected. Another is denied despite a clearly valid contract.

This is the point where hard questions arise. Who or what gets alerted? How much business value is at risk? How do you diagnose root causes across the model, tools, data, and workflows? How do you detect issues before they affect customers? Failure to address these questions leads to problematic issues for the organization, such as reputational risk and customers losing trust.

In agentic systems, the blast radius expands because the system acts autonomously, not just predictively. Many of the most common failure modes may not even show up in demos or test environments. Examples include:

• tools breaking when upstream APIs change
• retrievals that miss critical context as data evolves
• performance degrading as models and workloads evolve
• policy violations that surface in edge cases
• spiking costs as usage patterns shift or platforms refresh pricing
• growing latency as workflows become more complex

These are the problems that Live AI Operations (LiveOps) is designed to solve. LiveOps, developed by QuantumBlack, AI by McKinsey, is an end-to-end AI operations capability that comprises a set of practices to ensurethat AI continues to deliver value in production. The “end-to-end” scope of LiveOps means that it considers all aspects of operating AI post-deployment, with the wide range of actions it drives, including:

• AI performance management and value realization
• monitoring and observability for quality, drift, cost, latency, safety, and business outcomes
• governance, risk management, and human oversight where needed
• production hardening and readiness
• incident response and triage
• remediation through hotfixes, prompt and workflow updates, retraining, model upgrades, and rollback
• controlled model upgrades with change and release management
• knowledge continuity through runbooks and decision records

Put simply, LiveOps transforms a one-time AI deployment into a dependable, long-term service, with clear accountability for how it operates and a reliable path to recovery when things go wrong, while freeing up developer capacity to focus on delivering new product releases on time.

AI systems will degrade without AI operations in place

Even the best-performing AI system will degrade after launch. They may do so quietly over time, or loudly and abruptly, in some cases almost immediately after being deployed. Performance often degrades through small changes that accumulate: data shifts, new edge cases, outdated evaluations, rising costs, and declining response quality. When degradation is abrupt, value can fall off a cliff, for example, due to a sudden, significant shift in data due to a major event or change in operating conditions.

Beyond the technical impact, degradation erodes trust. Users see inconsistent results and revert to old workflows, while stakeholders lose confidence as business value fades. Without an operations discipline in place, teams can end up stuck in a pattern of repeatedly patching systems to try to achieve their original value goal instead of continuously improving them and realizing further value.

Relying on AI builders to handle operations only goes so far

Without LiveOps, your best developers can quickly become full-time firefighters, while users lose confidence and scaling hits a ceiling fast.

Agentic AI makes this even harder. Teams can easily get pulled into unstructured data cleanup, prompt and workflow tuning, fixing tool chain breakages, and repeated validation of what agents are allowed to do. That work is not optional if the organization expects stable performance and responsible AI use in production. The fact that agentic capabilities can be implemented so quickly can also put pressure on teams to speed up release times, often at the cost of stability and without sufficient consideration of the true business value added.

Gen AI and agentic workflows raise the stakes

As AI systems shift from providing model predictions to delivering automated decisions and agentic actions, the number of moving parts compounds. Retrieval, tools, workflows, voice, policies, and external APIs are all needed, and all create more failure modes. Personalization and context open up new kinds of risk exposure related to privacy, bias, safety, and reputation.

When organizations struggle to achieve AI impact at scale, it can often be traced to the gap between what was working in a single pilot compared to what is needed for a sustainable system to run at scale. In production, small changes can have outsized, compounding effects, as systems drift from the conditions under which they were validated.

A number of pressure points commonly show up in production across the full AI stack:

• Agentic AI systems can behave unpredictably as prompts, tools, models, and context evolve. Unstructured data introduces inconsistencies and bias that drive downstream mistakes and create operational drag.
• Maintaining quality, managing latency, optimising costs, and achieving reliability becomes an ongoing operational challenge.
• Even when the technology performs well, poor change management can slow user adoption and limit business impact.
• Organizations need the right skills in place to operate and troubleshoot complex, ever-evolving technologies. Human intervention in tandem with tools and agents remains essential to delivering consistently good output.
• When new privacy, fairness, and security risks arise, ethical and regulatory considerations can suddenly become paramount.

The QuantumBlack LiveOps approach

QuantumBlack, AI by McKinsey, has made advances in this space for years, working with clients across industries and organizations of all sizes to sustain AI value. We’ve developed the frameworks and tooling that makes AI value sustainable and scalable.

Our LiveOps framework is designed as a multi-phase operations approach that overlays on the existing well-established AI lifecycle of Build and Launch → Operate → Maintain and Improve.

During the Build and Launch stage, LiveOps requirements such as scalability and reliability are considered from the start, along with a clear set of readiness validation checks prior to models, agents, and/or workflows being deployed to production.

During the Operate stage, the AI workflow is triggered and runs to produce an output. This is validated from a technical and business perspective (e.g., validation rules, guardrails, evaluations, key performance indicators). Ongoing monitoring with logging and observability is essential for tracking AI performance, costs, and overall business value and surfacing issues which may risk value. User feedback is vital at this stage to understand AI performance and impact and identify areas to fix or improve.

During the Maintain and Improve stage, interventions are implemented to address issues detected during the operate stage and enhancements are delivered to increase value.

Four critical operations elements to get right from the start

To effectively deliver on AI operations goals, it’s critical to start early on before the solution is live. Doing so means focusing on four key elements throughout the lifecycle, starting during the Build phase.

For each element, we’ve provided a brief example from a recent project with a large-scale enterprise client, working to maintain their Gen AI-driven solution, followed by key takeaways.

1. Evaluations

Automated, systematic performance assessments across development, deployment, and post-production; golden datasets; user feedback

Example: We found it was vital to set up a clear process for gathering user feedback and to share this as part of onboarding. Feedback from user evaluations was then applied directly to prompts during finetuning.

Takeaway: Setting user feedback loops early ensures feedback can be captured and used from day one.

2. Architecture, scalability, and tooling

Token limits; disaster recovery; client tech-stack compatibility

Example: We had to scale existing functionality as adoption increased, taking into account tool and plugin constraints which could have required architectural changes. An increase in users brought in more perspectives on the best way to show outputs, leading to further prompt refinement.

Takeaway: When selecting an architecture, take future scale into consideration and plan accordingly.

3. Observability

Logging across all layers; tracking both solution performance and business impact (FinOps)

Example: We used a ‘single pane of glass’ monitoring view to track key metrics, including cost-related business KPIs, failed responses in the application, and query response/load times. This monitoring triggered interventions such as refactoring and architectural improvements.

Takeaway: Identify early what will need monitoring and tracking, so that logging and observability requirements can be built for from the outset.

4. Model changes and reliability

Multi-model strategy; planning for model drift and provider updates; primary/secondary model fallback

Example: We faced issues related to LLM outages that required adopting a multi-agent setup, with a secondary LLM to fall back on. Data drift validation enabled us to identify issues through alerts and address them before the pipeline ran, avoiding any impact for business users.

Takeaway: Plan for LLM availability issues, and decide in advance how to catch problems fast enough to prevent them from impacting business users.