Should This Be AI At All?

The capability trap is shipping a feature because the model can do it, not because anyone needs it done. It’s the most common failure in AI product design, and it’s recognizable by its origin story: the idea started with “what could we do with the API?” rather than “what do our users struggle with?” The 2023–2024 wave of “summarize this” buttons is the canonical case — bolted onto documents, emails, tickets, and dashboards across the industry, many of them summarizing content the person could read in eight seconds, most of them quietly demoted or removed within a year. The model could summarize. Nobody had the problem.

The trap is seductive because capability demos are genuinely impressive, and impressiveness is easy to mistake for value. But a demo answers “can it?” and a product must answer “should it, here, for these people, at this failure rate?” Those are different questions with different owners. The first belongs to the model provider. The second is yours, and no one upstream will answer it for you.

The tell that you’re in the trap: the feature has no specific user, no specific moment, and no baseline. Compare with the features that survived — Gmail’s Smart Reply attached to a real, frequent, measurable behavior (short replies on mobile); GitHub Copilot attached to the constant, costly act of writing boilerplate. Both started from a task, and the model was the implementation detail. Start with the task or don’t start.

The core trade is simple to state: AI buys you probabilistic value at deterministic cost. The value side is variable — sometimes brilliant, sometimes wrong, always uncertain. The cost side is fixed and guaranteed: latency, compute spend, an interface that must now handle failure, users who must now calibrate trust, and a QA process that can no longer just assert equality. You pay the deterministic costs on every single interaction. You collect the probabilistic value only when the model gets it right and the person notices.

So the first question is never “which model?” It’s “what’s the deterministic baseline?” If a rules engine, a template, a saved filter, or a well-placed button solves 90% of the problem with zero failure modes, the model has to beat that — not a blank page. Date parsing, unit conversion, sorting, address validation: these have correct answers, and a system that is right 99% of the time loses to one that is right always. Stripe doesn’t use a language model to compute sales tax, and no one wishes it did.

The model earns its place where the input space is too open-ended to enumerate: understanding intent in freeform language, drafting where there is no template, synthesizing across sources that were never designed to be joined. That’s the honest test — could you write the rules? If yes, write the rules. If you genuinely can’t, you’ve found a problem worth the probabilistic bet, and everything in the rest of this guide is about designing for it.

Three categories should trigger a hard stop, or at minimum a human gate. First: high stakes with low tolerance for error. When a wrong answer costs money, legal exposure, health, or safety, and the person can’t easily verify it, a system that fails fluently is a liability engine. Air Canada learned this in tribunal in 2024, when its support chatbot invented a bereavement-fare refund policy and the airline was held liable for honoring it. The chatbot didn’t crash — it confabulated, confidently, in the company’s voice. In these domains AI can assist a human decider; it cannot be the decider.

Second: tasks where authorship is the point. A condolence note, a performance review, a wedding toast, an apology — the value of these artifacts is that a person spent themselves writing them. An AI-drafted apology that reads as AI-drafted is worse than no apology; it converts a gesture of care into evidence of its absence. Assist with structure, sure. But when the reader is asking “did you mean this?”, generation hollows out the answer.

Third: tasks that are cheaper by hand. If typing the thing takes less time than prompting, reviewing, and correcting the model — and for a huge class of short, known-answer tasks, it does — the AI is a tax dressed as a gift. This is also where honest capability disclosure pays: a product that says what it’s bad at, up front, spends less trust than one that lets people discover it. Declining a category of work is a design decision, and often the right one.

Define what success means before you design a pixel, because AI features rot the usual metrics. Engagement is the worst offender. A confused user re-prompting five times generates more “engagement” than a satisfied one who got the answer in one shot; a hallucination that sends someone down a verification rabbit hole looks like a highly engaged session. For deterministic software, time-in-product loosely correlates with value. For AI, it just as often measures friction. Any team celebrating message volume on an assistant is celebrating a number that cannot distinguish delight from struggle.

The metrics that survive are about outcomes, not activity. Task completion: did the person accomplish the thing they came for, measured at the task level, not the message level? Correction rate: how often do people edit, redo, or abandon what the AI produced? GitHub has published research on Copilot’s acceptance rate for exactly this reason — the fraction of suggestions developers actually keep is a far more honest signal than the number shown. And retained trust, the longitudinal one: do people come back to the feature for consequential work, or did they try it twice in week one and quietly route around it forever? Retention on high-stakes tasks is the metric that predicts whether the feature has a future.

This is contract-setting turned inward. Before you promise users what the system does, promise your team what “working” means — in numbers, with thresholds, written down. A feature without a success definition can never fail, which means it can never be finished, fixed, or killed.

Your system will be wrong. The design decision — and it is a design decision, not an ML tuning detail — is which direction it should be wrong in. Every classifier and every generative feature trades precision against recall: fail by missing things, or fail by flagging things that don’t matter. You cannot maximize both, so someone chooses. If the designer doesn’t, the model’s defaults will, and the model has no idea what a false positive costs your users.

The tool for making the choice explicit is the confusion matrix, priced in human terms. A spam filter’s false positive — a real message buried — costs a missed job offer; its false negative costs three seconds of deleting. So spam filters correctly bias toward letting borderline mail through. Flip the domain and the answer flips: a fraud alert that occasionally bothers you about legitimate purchases is annoying; a fraud system that misses theft to avoid annoying you is negligent. Copilot-style code suggestions bias toward precision — a wrong suggestion interrupts flow and erodes the habit of accepting, while a missing one costs nothing because the developer just keeps typing. Same math, three different right answers.

Write the sentence down for your feature: “when this system is wrong, we prefer it to be wrong by ___, because the alternative costs ___.” That sentence drives thresholds, drives what the interface emphasizes and hedges, and drives where you spend review ceremony. Effort should concentrate on the expensive failure direction — that is what matching effort to stakes means at the system level, before a single screen exists.

Before launch, write down what would make you remove the feature. Not “iterate on” — remove. Specific, measurable, dated: “if correction rate stays above 40% after two quarters,” “if fewer than 15% of weekly actives touch it by Q3,” “if a hallucination reaches a customer in a regulated flow, we pull it that day.” Kill criteria written before launch are analysis; the same criteria drafted afterward are politics, argued by people whose roadmaps and reputations now depend on the answer coming out one way.

AI features are unusually hard to kill, which is exactly why they need this. They demo well forever — there is always a cherry-picked transcript that looks like the future. Sunk model spend, announced roadmaps, and executive sponsorship all conspire to keep a mediocre feature alive, degrading trust interaction by interaction. The industry-wide retreat of first-wave AI summary buttons happened a year later than the usage data justified, at every company, for exactly these reasons. A pre-registered threshold is the only force that reliably outweighs them.

Kill criteria also sharpen the launch itself. A team that can’t articulate what failure looks like doesn’t understand the bet it’s making — the criteria are the falsifiable form of your success definition from the previous section. And “kill” has gradients worth specifying in advance: retreat to a smaller scope, demote from default to opt-in, gate behind a human review step, or remove outright. Deciding the ladder now, while you’re calm, is the entire point. This is the go/no-go discipline, and it’s the worksheet this chapter pairs with.