The past two years of Large Language Model (LLM) development have been governed by “scaling laws” in the simplest sense of the words. Collect more data, build a bigger model, and train it for longer. Sit back and watch new capabilities emerge.

In parallel, we have seen exciting research from groups like Google (1, 2) and Anthropic (1, 2, 3) on mechanistic understanding of how LLMs represent information and how we might interact with them.

OpenAI’s o1 model is not just a better-reasoning LLM; it represents a paradigm shift in how foundation models are developed and deployed. It is the first major instance of training-time reinforcement learning and inference-time search in a foundation model. What does that mean and how will it impact the next generation of LLM applications?

The two sources of LLM improvement

Fundamentally, there are two ways that foundation model systems can improve:

1. Data distribution modeling: How well does the model learn to approximate the true distribution of the data it’s trained on?
2. Search: How can you find the best answer in that model’s distribution?

We have only recently started to formalize, separate, and scale efforts to pursue both of these categories. Let’s build an intuitive model of them:

Data distribution modeling

At their core, the only thing foundation models do is approximate the distribution of their training data. For LLMs, their task during training is simple next-token prediction: given a sequence, predict the most likely next token. At any point, there are many potential next tokens, like branches of a tree. As LLMs see more training data, they learn more about the conditional probabilities of these branches.

Now consider the model training across billions (or trillions) of sequences. Zooming out, individual tree branches blur together into clusters of knowledge. In reality, these exist in high-dimensional space that is impossible to visualize directly, but you can imagine them in 2D space. While the LLM is trained on all its training data together, there are many different data distributions that represent different types of knowledge or skillsets. For example, the data distribution includes different clusters for basketball stats, English grammar, and types of humor.

One important capability of LLMs is that they can combine different knowledge and skills, even in novel ways. LLM’s training data includes plenty of information on physics, and plenty of content on how to speak pig latin. When prompted, LLMs can combine the two to generate descriptions of physics in pig latin, even if there wasn’t much data in their training set that combines the two.

Prior to o1, nearly all foundation model progress came from better data distribution modeling. Adding more (or higher quality) data expands the size and quality of the distribution you can model. Making the model bigger, training it longer, or implementing architecture improvements allows your model to represent the data distribution more accurately.

Search

As described above, the shape and clusters of the model’s data distribution are learned through following billions of tree branches during training. At inference time, LLMs just make their way down a single tree, randomly choosing which branches to go down based on its learned probabilities.

As you could imagine, depending on the branches followed you may or may not end up in the exact place that contains the best answer. Inference-time search is a way to better explore your data distribution. It can’t help your model learn brand new skills, but allows you to find the best answer it can make using its knowledge.

Search is a mature field with many strategies; some being applied to LLM systems are:

• Best of n: Run the model n times and choose the best answer
• Monte Carlo Tree Search (MCTS): An algorithm that explores trees probabilistically, balancing exploration and exploitation.
• Beam Search: An algorithm that explores trees deterministically, keeping the top n at each step.

All of these strategies require a reward estimation model that learns from successful trajectories to predict if an answer (or intermediate reasoning step) is good or bad.

There are also some new LLM-specific approaches to help reach the best output, such as:

• Chain of thought reasoning: Prompt the model towards certain paths that we have seen lead to better outcomes. Before starting a logic puzzle, you could say “think step by step.” Or after a block of reasoning, you could inject “reflect on the previous step.”
• LLMs as critics: After generating a response, use an LLM to critique or improve the initial answer.

The principles of inference-time search have been known and experimented with for some time, even informally with prompting tricks like “think step by step.” But only recently has it been formalized and prioritized at a foundation model level.

There are two major reasons why inference-time search is now being prioritized:

1. There’s a practical limit on how big models can be due to inference compute and latency constraints; we can’t just keep making them bigger.
2. Reasonable-size models have gotten good: there’s a lot of “unsqueezed juice” in their knowledge and skillsets that we were underutilizing.

o1

While they describe few details, it’s clear that the step change improvement in o1’s reasoning capabilities comes from a combination of both data distribution modeling and search strategies.

Data distribution modeling: There has been much debate over LLMs’ poor planning ability. Understanding data distribution modeling explains this. Planning skill is a data distribution, no different from knowledge of European History or Python syntax. LLMs sometimes appear to plan just because there are some examples of planning in its training data. But outside of programming how-tos and math problem sets, humans typically don’t write down their plans. Without relevant data in its training set, asking an LLM to perform more complex or use-case specific planning will fail.

For o1, OpenAI generated a huge number of reasoning tasks, either math / logic-based problems that can be empirically validated, or other ones that could be rated by humans. It then trained a reward model and reinforcement learning system to help it learn to pursue reasoning paths that led to the right answer. This task helped it learn a robust distribution of reasoning strategies and execution.

Search strategy: With this model trained to have strong reasoning skills, OpenAI employs search strategies to help use these skills during inference. Besides chain of thought reasoning, they don’t specify exactly which search method(s) they use, but at a conceptual level it doesn’t really matter. The key point is that instead of following one random inference tree, they’re exploring a bigger area of their data distribution.

O1’s amazing reasoning capabilities come from a combination of these two categories. It’s impossible to say exactly how much each contributed without an ablation study. But it’s clear both helped significantly, and probably benefited from each other: better data distribution modeling gives o1 the skill to solve reasoning problems, and better search gives it ability to use those skills in practice. It seems near-certain that future foundation models will pursue both levers for improvement.

Three implications for companies building LLM applications

How will the shift from purely training-time to training and inference time optimization impact LLM applications? Here are three of our ideas:

LLM applications will have a time budget

Engineering leaders are already used to thinking about latency and cost/compute budget. But with inference-time search strategies, the scope of these decisions will be expanded.

With a continuous lever you can use to spend more time thinking and get a better answer, teams shouldn’t just think about latency in terms of “lower is better.” Instead, features will have a “time budget” balancing response time and quality. Our hypothesis is the distribution of optimal time budget is bimodal:

• For real-time applications like coding copilots, phone agents, or transactional applications, the faster the better.
• For any other application (e.g. answering a support email or categorizing a business transaction), the time budget will often be much longer, as it will be compared to a typical human response time. If a human would take 15 minutes to get to a task and another 15 to complete it, the difference between a model taking 2 seconds, 30 seconds, or even 5 minutes might be trivial if you can get to a better answer.

Good reasoning requires good data; companies can build moats around their domain

O1 is trained to be much stronger on reasoning tasks. While primarily trained on things like math and science, along the way it learned some general reasoning too. For example, it might have learned how to structure a plan, or reflect on a previous step to catch errors.

But many valuable workflows involve domain-specific reasoning that is still not represented in any training distribution. Imagine you are trying to investigate security alerts (like Theory portfolio company Dropzone AI) or prioritize between different supplier tradeoffs as a procurement director. Humans reasoning about these tasks use accumulated experience and tribal knowledge that is often not written down or exists only across scattered internal systems.

Today, it is possible to build domain-specific reasoning around LLMs with rules-based systems using expert knowledge. This makes intuitive sense: if you’re staffing a call center, no matter how smart your new hires are you would still create standard operating procedures to ensure they follow the same guidelines and don’t start from scratch every time they pick up the phone.

Over time, we expect that workflow-specific reasoning data will also be a key source of defensibility for AI application startups. In domains like security operations, there’s no single platform where investigations are conducted; analysts use many separate tools. By creating an end-to-end workflow automation platform (supported by rules-based systems and human input), you also generate unique workflow reasoning records that can help train future models.

Foundation models are becoming more complex systems

Until now, foundation model endpoints made a single query to a single model (perhaps with simple caching or routing). Using o1 might feel the same, but on the back end, it’s triggering a more complex system: orchestrating many separate LLM inference calls, likely in combination with other learned models to assist with search.

The training of o1 was also much more complex, involving synthetic data generation of reasoning problems, and reward models / reinforcement learning to guide the model’s behavior.

To date, open-source foundation models have been able to replicate many capabilities of closed-source models with some lag. But as foundation model systems continue to become more complex both in training and deployment, it’s unclear whether we’ll be able to replicate them with open source efforts. On the other hand, it’s possible that the power of smaller models combined with inference-time search will support a proliferation of open source model systems. Either way, these approaches will change the dynamic of foundation model development.

If you’re building LLM applications using reasoning, please reach out to at@theory.ventures!

‍