Back to Blog
Artificial Intelligence

AI Kill Criteria: Why Only 17% of CIOs Can Retire Low-Value AI

By Marine Pimenta

Two product team members sorting yellow sticky notes on a glass wall while prioritizing a project plan

Most product teams have a process for shipping a feature. Far fewer have one for turning a feature off. With AI, that gap now shows up in the numbers. In its 2027 CIO and Technology Executive Agenda survey, Gartner found that only 13% of CIOs say their enterprise has realized significant or transformational value from AI. It also found that only 17% have criteria for abandoning low-value AI that is already in production. If you own an AI roadmap going into 2027 planning, AI kill criteria may be the cheapest improvement available to you.

What the Gartner survey says about AI exit rules

Gartner surveyed 2,002 CIOs and technology executives between May and July 2026 and released the results on October 19 at its IT Symposium/Xpo in Orlando. It compared the full sample with a group it calls “vanguards”: CIOs who deliver better business outcomes and also report high CEO-perceived value from AI.

Two governance habits stood out:

  • Pilot gates. Organizations with criteria for stopping a pilot from moving to production are 2.3 times more likely to be vanguards. Only 31% of CIOs have them.
  • Production exit rules. Organizations with criteria for abandoning low-value AI already in production are 2.6 times more likely to be vanguards. Only 17% have them.

Gartner VP Analyst Alicia Mullery put it in four words: “Volume is not value.” Two caveats apply. These are CIO self-assessments, and, as eWEEK points out, the data shows an association, not proof that exit rules cause better returns. Still, the 14-point drop from pilot rules to production rules says something. Most organizations will question an AI idea before launch. Once it ships, almost nobody asks again.

Why low-value AI features survive in production

Gartner doesn’t explain why production exit rules are so rare. From a product seat, the usual suspects are easy to name. A pilot has a budget line and a deadline, so someone has to decide. A live feature has users, an internal champion and a dashboard that looks fine, so the default is to keep it running.

AI makes that default more expensive than it is for ordinary software. Mullery’s point is that “AI costs are probabilistic and determined at run time.” A traditional feature costs roughly the same whether ten people use it or ten thousand. An AI feature pays for every call, so a use case that looked cheap in a pilot can lose money once adoption grows. The same survey, per eWEEK, found that 58% of CIOs face more pressure to deliver AI cost improvements and 51% expect AI to raise their solutions’ total cost of ownership.

The bills are already reaching the top. Mavvrik’s 2026 State of AI Cost Governance Report, which covered 396 enterprises, found that nearly half had AI spending surprises escalate to the board, and one-third had imposed emergency spending freezes. A freeze is a kill decision too, just made late, under pressure, and without criteria.

Pruning is how you fund the AI that works

Exit criteria are not a vote against AI. A recent CIO Dive report on an Ocient survey of more than 500 US IT and data leaders, published November 5, shows the trade-off. Three-fourths said their costs are increasing, and 38% said they have seen significant ROI from AI. Nearly one-quarter have reduced or paused underperforming initiatives, while about 45% have increased investment in the ones that work. In the report’s words: “Pruning isn’t pulling back — it’s making room for what’s worth growing.”

Product leaders know this from feature portfolios. Each feature you keep has a maintenance cost, a support cost and an attention cost. With AI, you also pay for inference, evaluation and monitoring. Retiring a weak use case frees budget and engineers for the two or three that actually move a business metric.

A practical AI kill criteria framework for product teams

You don’t need a new governance body to start. Most of this fits into the product reviews you already run. Here is the version I would put in front of a product council:

  1. Record the baseline before launch. Write down the current cost, cycle time or error rate of the process the AI is supposed to improve. Without it, every review turns into a debate about anecdotes.
  2. Write the stop rule on day one, with a number in it. For example: “If the assistant doesn’t cut average handling time by 15% within two quarters of general availability, we sunset it.” Pick the number before anyone is emotionally invested.
  3. Add a unit-cost trigger. Track cost per successful outcome (per resolved ticket, per approved document, per qualified lead), not just total spend. If that unit cost rises past an agreed ceiling as usage grows, the feature goes back to review.
  4. Separate the sponsor from the scorekeeper. The person who championed the use case shouldn’t be the only one measuring it. Mullery called criteria “a team sport”: CIOs need business partners who are willing to walk away from low-value investments.
  5. Put production AI on a review calendar. Pilot gates happen once. Production reviews should repeat, quarterly for most use cases, so the 17% problem doesn’t come back next year.
  6. Plan the sunset as part of the launch. Know how you will turn the feature off, what users fall back to and how you will communicate it. A feature that can’t be removed cleanly tends to stay forever.

None of this is exotic. Back in 2024, Gartner predicted that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, citing poor data quality, inadequate risk controls, escalating costs or unclear business value. Those projects got cut either way. The difference exit criteria make is whether you cut on schedule, with evidence, or in a budget freeze.

Where agents raise the stakes

The next wave makes this more urgent. At the same symposium, Gartner analyst Kristin Moyer said only 21% of CIOs have agentic AI governance today, while 71% will deploy agentic AI within 12 months, according to InformationWeek. She called that “out of alignment.” Agents run multi-step tasks and make more model calls per job, so run-time cost and risk both grow. Among Gartner’s broader recommendations at the event: demonstrate measurable business value before increasing AI investment. For agents, that means setting the exit rule before you scale.

Gartner also flagged a people gap. Organizations with explicit plans to redesign roles for a human-agent workforce are 2.4 times more likely to be vanguards, but only 4% of CIOs have such plans. When you review whether a use case earns its keep, include the workflow around it, not just the model.

Conclusion

The organizations getting value from AI aren’t necessarily the ones running the most use cases. According to Gartner’s data, they are more likely to have written down when to stop. For a product leader, that is a familiar discipline applied to a new kind of feature: entry criteria, exit criteria, a unit-cost metric and an owner who is allowed to say no.

If you’re reviewing your AI portfolio for 2027 and want help deciding what to scale, rebuild or retire, talk to the Luby team.