Theories
Stay up-to-date on our team's latest theses and research
Stay up-to-date on our team's latest theses and research
Five years ago, a simple workflow automation task seemed not worth the effort. Now, everyone’s asking themselves how aspects of our workflows can be transitioned to automation via AI, or at least augmented by AI.
Unfortunately, this process has headwinds. The most common issues are usually related to the complexity of the workflow, the availability of interconnected tools, or model performance. Of these, model performance is often the easiest to overcome, as its challenges can frequently be mitigated through effective human-AI collaboration.
During my internship at Theory, I focused on model performance within workflows: turning unstructured and nonuniform information from startup board decks into clean, structured data that investors could utilize.
Startup’s board decks often contain crucial information about a company’s performance and priorities; however, they arrive in a variety of formats. Some have clean tables. Some have beautiful but dense slide designs. Many decks hide important metrics inside charts, graphs, and images.
We want easy access to these data, which today sit locked in disparate sources. We also want the ability to slice, join, and analyze them without burdening our investors or portfolio companies with manual data entry.
Our key criteria for this work were:
Hence, the Board Deck Extraction project was born.
To satisfy our first criterion, we needed a frictionless method to get the decks into our processing system. Our first solution was to send the board deck PDFs to a service email account and use an Apps Script to poll that email for board deck-related emails. No deployments necessary, just a lightweight, scheduled polling script.
This ended up working well enough, so we stuck with it. Additionally, Google Apps Script is completely free, and we were already using Google Cloud Services as our cloud provider.
Page layout, page co-occurrence, and multi-page context are all crucial in extracting the meaning of text. This means that pure text extraction was likely not enough.
Our goal was to send the LLM strings of text for processing, so our initial naive solution was to run the PDF through Tesseract OCR in Python. While this worked for text recognition, it flattened all formatting, losing the page structure, table alignment, and any sense of where data lived in context. A number in a table could be sitting beside the wrong label, and Gemini wouldn’t know the difference.
That’s when we turned to Google Document AI’s Layout Parser. Unlike traditional OCR, it:
This meant we could rebuild the document as text with layout preserved, giving Gemini the context it needed to correctly interpret table data, extract numbers embedded in charts and graphs via image OCR, and understand relationships between headers, subheaders, and values. While there are other services specifically designed for applications like this, the Layout Parser was an easy and effective option already pre-existing in our GCS infrastructure.
The result? Accuracy jumped, and Gemini could now pull metrics from places that had been previously out of reach.
Evals are a crucial part of any LLM capability. We used some simple hand labeling of board decks data and a homegrown app to try different prompts, products, and services for these tasks. We didn’t go too deep on measuring fancy statistics or metrics about the quality of extractions; we just stuck to simple accuracy and instruction following for these tasks.
However, we did test out multiple evaluation platforms and each of their usefulness to this project. We performed an evaluation on the evaluators.
It’s easy to frame the objective as a simple yes/no, “did it extract the text correctly?”, but the reality is more nuanced. This task is closer to translation than to straightforward text recognition.
Let’s use a simple example: some metrics (e.g., burn rate) aren’t always present and must be calculated. As is known, LLMs are arithmetically challenged, so how should we get the LLM to calculate it? The answer is via tools. Using Gemini with Vertex AI, you can ask it to extract metrics and write code to calculate new ones. This is extremely useful, especially with metrics that have a triangular relationship. For example, if we are provided a cash balance and a monthly runway, we can extract them with Gemini and use Vertex AI to generate code to calculate burn rate.
.png)
Evals became even more essential in this stage; we are asking the model to perform a series of steps where a mistake at one step ruins the chances for the rest of the steps. By creating a ground truth for every metric extraction, we are able to evaluate each model and prompt’s accuracy.
The prompt was essentially built through iteration: write prompt, analyze errors, incorporate into prompt context, repeat. There are plenty of platforms that offer an application workflow for this process, and we even developed our own internal app to streamline this. However, for much of this work, we used Google Sheets.
This workflow deals with some critical data. For this reason, we decided to create a human review step. While humans should not be relegated to computer verifiers, this is a good use case while we build confidence in the consistency of the capability.
To make this work as pleasant as possible, we created a UI experience optimized for rapid review:
Now that we have the data extracted, we can finally start unearthing insights.
Think of a collection of board decks as a story, and each deck is a chapter. With these metric extractions, we can easily summarize while maintaining and visualizing the key points that drove the story.
Here, we can easily surmise that something isn’t going well. The raw board deck itself cannot display this level of clarity to a company's story.
Five years ago, hiring a dedicated worker for these kinds of tasks was normal. Automation tasks for unstructured data just couldn’t be viable. Today, it is.
With effective human–AI collaboration, we can overcome common pitfalls and achieve workflows that are both streamlined and reliable, delivering high-accuracy and trusted data.
At Theory, board deck extraction is only a fraction of the beginning. We’ve built an entire system of internal services that speed up our workflows, focused on speed and quality. This is just one service among many, and we intend to keep pushing the boundaries of how far AI augmentation can take us.
The two hottest new roles in the bay are the Forward Deployed AI Engineer and Agent Product Manager. We think they shouldn’t exist.
Okay, that’s a bit hyperbolic. But the best approach to implement AI applications is shifting from ad-hoc context engineering to productized context platforms.
These roles were created to address a real need: closing the gap between AI’s capabilities in the lab versus the real world. We recently wrote about how context, not intelligence, is the limiting factor to making AI work in enterprise. Most of the work FDAIEs and APMs do is gathering context from their customers to manually update system prompts and evals.
This approach works, but it’s slow, labor-intensive, and non-scalable. As AI promises to automate every aspect of work, why is its implementation so manual?
It doesn’t have to be this way: context creation and implementation will become more productized as part of a core platform, instead of a post-sales service.
As we wrote previously, emerging context platforms we see all share three core capabilities:
Let’s compare them to today’s context engineering:
Products built around context platforms will be faster and cheaper to deploy. They’ll do tasks more reliably because context is more easily kept up to date.
And critically, they will allow customers to own and manage their operational processes: creating a valuable data asset/intellectual property, as opposed to ceding control and information to their vendors.
Forward-deployed resources will still be critical, but will spend most of their time on things like integrations, edge case handling, and change management; not writing and tweaking prompts.
Meanwhile, as most business jobs shift from doing work to managing AI agents, these context platforms will become key systems of record and operational tools to manage agent behavior.
Context will be a critical part of any enterprise AI application. But there are some areas where it will just be a component of vertical platforms, versus others where the context platform is the dominant feature and might even be a standalone company.
In spaces like cybersecurity, there is a ton of sophisticated domain knowledge required to do the job successfully: writing complex queries, reasoning about attack paths, etc. Much of this knowledge is universal – phishing attacks and cloud misconfigurations look similar anywhere. Customer-specific context is still very useful, but the most important thing is having the latest security expertise. These functions will best be served by sophisticated vertical applications like Maze and Dropzone, which build features for customer context.
In other functions, company-specific context massively outweighs general domain knowledge. Consider procurement or customer support. Individual tasks might be simpler, like looking up data in one system and entering it in another. But knowing what to do entirely depends on how your organization operates and makes decisions. In these domains, getting AI to work requires huge amounts of context gathering, and the context platform will actually be the most important feature driving product value and scalability.
Two areas where we think context platforms will be critical are:
Operations: Ops are extremely complex and variable, requiring coordination across every function in the business. Processes are documented in scattered SOPs, past conversations, and tribal knowledge. Outside of basic project management and brittle RPA tools, ops leaders have no core software platform where they can view or manage how their operations are executed. We wrote about this opportunity extensively in our Business Context Layer article.
EPD: A huge amount of context and history underlie every product and engineering decision. Today’s PRDs are not sufficient to direct coding agents: they might understand the desired features and general best practices, but won’t have the background context on priorities, tradeoffs, and historical decisions that inform the right way to build them. It’s possible that context is a byproduct of coding agents, but we think there is an opportunity to build a context platform across engineering, product, design, and customer teams.
It’s clear that productizing context will be a key unlock for deploying the next generation of AI applications.
If you’re building context platforms for enterprise, or AI applications where context is a core component of your product, I’d love to hear from you: at@theoryvc.com.
