Building AI Systems That Scale: Office Hours w/ Calvin French-Owen
When generating code becomes cheaper, the harder work shifts toward deciding what to build and verifying that it works. Calvin French-Owen, cofounder of Segment and a former OpenAI engineer who worked on Codex, joins Tomasz Tunguz on Office Hours to explore that shift. Drawing on his experience building coding agents and using them, Calvin discusses specifications, architecture, evaluations, and the human judgment required to turn faster implementation into reliable software.
In our conversation, we discuss:
- How early reasoning-model experiments evolved into Codex
- Why planning and architecture become more valuable as code gets cheaper
- How small components and established patterns help agents work effectively
- Why short feedback loops often beat long unattended runs
- How inspecting failures helps teams build useful evaluations
- Where code review, validation, and missing business context create bottlenecks
- How AI changes hiring and productivity without removing engineering ownership
Three takeaways from this conversation:
1. Give agents a clear task and a fast way to check their work. Calvin emphasizes a loop of specification, execution, and verification. Well-defined components, interfaces, and existing testing patterns give an agent useful constraints. Larger features can be broken into milestones rather than treated as one extended run. Writing a plan also forces the engineer to understand the problem, making it easier to judge the implementation and spot a wrong turn before it spreads.
2. Build evaluations from the failures you can recognize. Before a team can score an agent reliably, it needs to know what good behavior looks like. Calvin describes learning from direct use and inspecting outputs, then turning observed problems into evaluation criteria. Those evaluations support regression checks and model choices, but they need to evolve as capabilities improve. Start with prompts and context when correcting behavior, and consider more expensive approaches when those tools stop being sufficient.
3. More code makes review and judgment more important. Generating an implementation does not resolve architectural tradeoffs, supply missing business context, or prove the change is ready to ship. Calvin identifies review and validation as growing bottlenecks and stresses the responsibility to inspect work before handing it to a colleague. Engineering productivity should ultimately reflect useful outcomes. Teams still need people who can define the problem, choose among tradeoffs, and own the resulting system.
