Welcome, Guest | Browse

Software Factory Archive

← Previous Work All Works Next Work →

The Replication

Rating:
General Audiences
Fandom:
StrongDM Software Factory
Characters:
Justin McCarthy Jay Taylor Navan Chauhan
Tags:
Replication Science Methodology Reproducibility
Words:
472
Published:
2026-01-05

The first team was in Berlin. Lautlos GmbH, four people, compliance tooling. They had been in contact with Justin for months. Their results were expected.

The second team was in Bangalore. A seven-person outfit called Verdant Labs that built environmental monitoring software. They had read the published documentation, cloned the repositories, and built their factory without ever contacting anyone at StrongDM. They didn't announce themselves until they had results. Their satisfaction metric on their primary scenario suite was 0.87. They had spent sixty-three days and roughly ninety thousand dollars in tokens.

The third team was in Toronto. Two people. A former Google SRE and a former Amazon principal engineer. They had met at a conference, disagreed about whether the factory approach was real, and decided to settle the argument empirically. They built a factory in twenty-two days for a fintech API integration product. Their satisfaction metric was 0.81. The former skeptic owed the former believer dinner.

Three independent replications. Three different domains. Three different team sizes. All three used the open-source tools—Attractor, Leash, Agate, CXDB. All three followed the NLSpec methodology. All three observed the compounding-correctness property: after an initial plateau, satisfaction began to climb in a pattern that matched a logarithmic curve approaching an asymptote.

Jay made a chart. Three curves, different colors, same shape. The x-axis was days. The y-axis was satisfaction. The curves didn't overlap—different starting points, different rates, different domains—but the shape was the same. The same underlying dynamical behavior, manifesting in three unrelated contexts.

"That's a result," Jay said, staring at his own chart. "That's an actual scientific result."

"It's three data points," Navan cautioned.

"It's three independent replications of a compounding-correctness curve across unrelated domains. In any other field, that would be a publication."

Justin was quiet. He had the chart up on his own screen. He had been quiet for the better part of an hour, the kind of silence that Jay and Navan had learned to recognize as productive. Justin was not ignoring them. He was integrating.

"The Berlin team had a coach," Justin said finally. "Us. The Bangalore team had documentation only. The Toronto team had documentation and active skepticism. And the curve shape is the same." He looked at Jay. "You're right. The methodology is reproducible independent of team composition, domain, and level of external support. The convergent property is inherent to the architecture, not the people."

"Is that good?" Navan asked.

"It's the best possible outcome. It means it's not about us. It was never about us."

Navan wrote it in his notebook. Jay refined the chart, added error bars, added a trend line. Justin sent a brief email to all three teams, thanking them and asking if they would consent to their anonymized data being included in a published analysis.

All three said yes within the day.

Science, such as it was in the messy world of software engineering, was satisfied.

Kudos: 178

replication_crisis 2026-01-07

"It means it's not about us. It was never about us." The most important sentence in this entire archive. If only more tech leaders felt this way about their work.

data_viz_nerd 2026-01-08

I desperately want to see that chart. Three convergence curves in different colors, same shape, different domains. That's beautiful data.

← Previous Work All Works Next Work →