Ninety-nine percent satisfaction meant one in a hundred trajectories failed. The team celebrated the ninety-nine. Justin examined the one.
"Show me the failures," he said at the Monday review. Not the successes. Not the dashboards. The failures. The 1 percent. The trajectories that didn't satisfy.
Jay had them ready. He'd been cataloging the failing trajectories for weeks, sorting them by category, by severity, by root cause. He pulled up the list on the conference room monitor.
The first category was timing. Seven percent of the failures were race conditions in multi-service orchestration. The Okta twin provisioned a user, but the Jira twin hadn't finished processing the SCIM event before the Slack twin tried to create the channel. A millisecond gap that manifested as a missing channel. The user story was unsatisfied not because anything was wrong but because things happened in the wrong order.
The second category was state. Twelve percent of the failures involved stale state. A scenario where a user's permissions were updated in Okta, but the Google Drive twin cached the old permissions for thirty seconds, and during that window the user could access files they shouldn't have been able to access. A security-relevant failure. Brief, transient, but real.
The third category was the interesting one. Navan called them "imagination failures." Trajectories where the scenario described a user intent that the code technically satisfied but didn't actually fulfill. A user wanted to create a shared document and invite three collaborators. The code created the document and sent the invitations. But the invitations went to the wrong Slack channel because the channel-naming convention had an edge case where channels with hyphens were parsed differently from channels with underscores.
"The code was correct," Navan said. "The scenario was satisfied. But the user would have been confused."
"Then the scenario wasn't precise enough," Justin said. "If the user would have been confused, the scenario should have caught it. The 1 percent is a map of our blind spots."
They spent three hours on the 1 percent. Three hours on the failures while the 99 percent sat untouched, unexamined, uninteresting. Because the 99 percent worked. The 99 percent was the expected case. The 1 percent was where the lessons lived.
Jay wrote new scenarios for each failure category. Timing scenarios with explicit ordering requirements. State scenarios with cache invalidation windows. Intent scenarios with ambiguous naming conventions. Each failure became a new scenario, and each new scenario made the next round of agent work better.
"The 1 percent teaches us more than the 99 percent," Justin said at the end of the session. "Always examine the failures. The successes tell you what works. The failures tell you what's next."
Navan drew a pie chart in his notebook. A tiny sliver, one percent. He labeled it: here be dragons.
The satisfaction metric climbed to 99.3 percent the following week. The 1 percent had shrunk to 0.7 percent. But the 0.7 percent had new failures in it. Different failures. Harder ones. The frontier moved but it didn't disappear.
It never disappeared. That was the point. There was always a 1 percent. The work was to keep making the 1 percent more interesting.
"The 1 percent is a map of our blind spots." This is the most useful sentence I've read about quality engineering in years. The successes confirm what you know. The failures reveal what you don't.