Search is one of the hardest technical problems in computer science. Only a handful of products like Google, Amazon, and Instagram do it well.

Until recently, most products didn’t need great search – it wasn’t core to the user experience.

But with the explosion of LLMs and retrieval systems to support them, every LLM company suddenly needs to have world-class search embedded within their product just to make it work.

With this emerging need, how will this new wave of AI companies solve search?

Retrieval is a critical component of LLM systems, and isn’t going away

Retrieval-augmented generation (RAG) systems deliver relevant information to an LLM to help it respond to a query. This grounds LLM generation in real and relevant information.

Imagine an LLM is answering a question on a history test. Without RAG, the LLM would have to recall the information from things it’s learned in the past. With RAG, it’s like an open-book test: along with the question, the model is provided with the paragraph from the textbook that contains the answer. It’s clear why the latter is much easier.

Finding the right paragraph in the textbook may not be easy. Now imagine trying to find one code snippet in a massive codebase, or the relevant item in a stack of thousands of invoices. Retrieval systems are designed to tackle this challenge.

New LLMs have longer context windows, which allow them to process larger inputs at once. Why take the effort to find a paragraph out of the textbook if you could just load in the entire book?

For most applications, we think that retrieval won’t go away even with >1M token context windows:

• Businesses often have multiple versions of similar documents, and delivering them all to an LLM could present conflicting information.
• Most interesting use cases will require some type of role and context-based access control for security reasons.
• Even if computation becomes more efficient, there’s no need to incur the cost and latency to process thousands or millions of tokens that you don’t need

Semantic similarity search is just one piece of the puzzle

As LLM prototyping exploded, people quickly turned to semantic similarity search.

This approach has been used for decades. First, separate data into chunks (e.g. each paragraph in a word document). Next, run each chunk through a text embedding model that outputs a vector which encodes semantic meaning of the data. During retrieval, embed the query and retrieve the chunks with the nearest vector representations. These chunks contain the data that (in theory) have the most similar meaning to the query.

Semantic similarity is simple to build, but results in pretty mediocre search. Some key limitations of this approach are:

• It can miss useful content that is semantically different from the query. Users don’t always clearly articulate what they mean, or a query might not include context that would be relevant to their request. (e.g., a customer describing a product vaguely, or not mentioning a recent purchase).
• The approach is sensitive to the embedding model. A general text embedding model might not perform well in your domain.
• They’re very sensitive to how you process input data. The system will function differently depending on how you parse, transform, and chunk your input data. Dealing with different types of data (e.g. tables) is complex.
• Even with optimizations, text embeddings are expensive to compute. This hinders the ability to iterate on ingestion and embedding pipelines, and to serve applications that need near real-time data.

To the first point, this approach only searches based on the semantic meaning of the query. If you look at any of the companies that do search well, semantic similarity is only one piece of the puzzle.

The goal of search is to return the best results, not the most similar ones.

YouTube combines the meaning of your search query with vectorized predictions of what videos you’re most likely to watch, based on global popularity and your viewing history. Amazon makes sure to prioritize previous purchases in search results, which it knows you were probably looking to re-order.

The future of retrieval systems

Google was founded on the PageRank algorithm, a simple way to rank web pages. But today’s Google search would be unrecognizable to the initial team: it is an incredibly complex system that uses many approaches to return the best results.

Similarly, teams building RAG systems started with simple semantic similarity search. We believe they will quickly become more complex and end up looking like today’s production search or recommender systems. The problems are not that different: from a large set of candidate items, select a small subset that is most likely to achieve some goal.

Today, most retrieval systems look like:

A future system might look something like:

Retrieval systems will have dramatic impacts on the capabilities of LLM applications: their effective memory, response quality, reliability, and performance/latency. We think that for many applications, these systems will have more of an impact on end capabilities than the LLM itself.

Because of this, we think that most companies will build these systems in-house as a core competency and differentiator. These builders will rely on a new set of infrastructure to build retrieval systems specific to their application.

To date, most investment has gone to databases to store vectors and retrieve nearest neighbors. But thinking through a future stack like the one described above, the database is only a small part of the solution.

Building these new systems will require better tooling to:

• Create, manipulate, and use multiple types of vectors that can encode different types of signals (e.g. combining semantic search with recommendation or behavioral vectors). Superlinked, a Theory portfolio company, is building this tooling.
• Ingest, parse, and process different types of input data.
• Orchestrate retrieval systems and execute them in near real-time; these processes (especially embeddings) are expensive and many applications will not tolerate minutes or hours of latency.
• Provide observability and monitoring to the retrieval stack.

Although most companies will build retrieval systems themselves, it’s possible that the stack consolidates in a couple ways:

• Infrastructure providers could expand their offerings to cover broader parts of the infra stack (e.g. ingestion, manipulation, and storage).
• Companies could build productized “retrieval as a service” that focus on specific applications or modalities – for example, retrieval systems for e-commerce websites, or for chat applications.

We’re excited for the evolution of retrieval and search as product enablers. If you’re building infrastructure for retrieval systems, or retrieval systems to power a new application, we’d love to chat!

There’s no space where reinforcements are as desperately needed as in security operations centers (SOCs). These teams are perpetually understaffed and face a barrage of alerts, each requiring a cumbersome manual investigation even though most will be false positives.

Dropzone AI is a team of AI and security experts building AI analysts that reinforce and expand the capabilities of SOC teams. We are thrilled to partner with them and lead their $16.85 million Series A round with support from Decibel Ventures, Pioneer Square Ventures, In-Q-Tel, and numerous security experts.

The pains of running a security operations center

Growing companies face increasing cybersecurity risks, customer scrutiny, and regulatory compliance burden. To deal with these, they procure detection tools to cover their technical surface area – email, network, endpoints, cloud infrastructure, etc. Each tool generates alerts: some are true attacks but many are false positives. Companies then set up a (or contract with an outsourced) security operations center, or SOC. These are teams of on-call analysts that triage and investigate alerts as they come in.

Any CISO will tell you it’s not easy to staff and manage a SOC.

It’s expensive: running a 24/7 SOC requires at least 5 analysts, which is more than most companies under 5,000 employees can afford. Even with a sufficient budget, there are only ~5.5M security analysts today and an estimated shortage of ~4M. 67% of security organizations reported talent shortages, the top reason being “can’t find enough qualified talent.” Typical security analyst churn is north of 20%, and new employees face a steep learning curve as they are familiarized with an organization and all its tools. Take all these factors and imagine trying to staff a 24/7/365 global operation.

It’s clear why there are staffing challenges – it’s a hard and tedious job. Analysts work long shifts where they look at dozens of near-identical alerts to determine which are real or not. They have to deal with fragmented tools and data, each with their own query languages and product nuances. Nearly every analyst mentions alert fatigue and false positive overload. The result is a slow response time and true attacks that slip through the cracks – either because an alert was ignored or an analyst wasn’t able to connect the dots.

This situation is only getting worse. Generative AI makes it much easier for anyone to launch attacks (e.g. sending personalized spear-phishing emails at scale). More attacks also mean more security tools, each generating a new set of alerts. The volume of tickets a SOC receives already grows each year, and will further accelerate in coming years.

SOC automation, past and future

The first attempts to tackle these problems were Security Orchestration, Automation, and Response (SOAR) platforms founded in the 2010s. Their rules-based automation playbooks are fundamentally limited. They can reliably automate simple workflows, like “if there is an endpoint compromise alert from an executive’s device, isolate their account”. But they can’t actually automate any of the analysts’ investigatory work: each investigation requires slightly different workflows, evaluation, and manipulation of data.

In recent years, “hyperautomation” platforms have promised to make SOAR more useful. They provide more integrations, and low/no-code interfaces to create playbooks. They are indeed much more capable and user-friendly. But at their core, they still rely on rules-based automation, and even with dozens of playbooks, you can’t replicate the work of a human investigation.

But recent innovations in AI – particularly LLM agent systems – provide new capabilities that dramatically change what’s possible in security automation:

1. Data extraction and manipulation:  LLMs can learn each platform’s unique querying language and data structures, operating like a subject matter expert to query and parse results from dozens of tools.
2. Interpretation and planning: LLMs can be trained to interpret results with common-sense security analysis. They can plan each step of an investigation, and then make a final determination if a set of evidence is most likely malicious or benign.

Dropzone AI is the first truly AI-native SOC automation platform. Security teams don’t need to configure more rules-based playbooks. Instead, Dropzone AI analysts are integrated just like another member of the SOC team. They perform Tier 1 investigations on their own – escalating real issues for human review and closing benign ones. These virtual analysts can respond to threats in minutes, 24/7. They never get tired or forget the context of previous alerts. And with feedback from other analysts, they continuously learn more about the organization, systems, and investigation best practices.

Building autonomous AI systems is not easy – especially in security where reliability is critical. It requires deep knowledge of AI, security operations, data, and UX. The Dropzone team brings expertise in all these areas from their time building AI products at ExtraHop. Their agents are deployed today, investigating alerts at multiple enterprises and MSSPs.

The impact of AI analysts

AI analysts will transform SOCs, for the first time allowing these teams to be proactive instead of reactive. They will address alerts faster and more accurately. The same team can cover an increasing number of issues, 24/7, without staffing challenges. Analysts can focus on more advanced analysis or proactive work, instead of dealing with rote triage tasks.

Software systems are becoming increasingly more sophisticated and interconnected. These systems have more capabilities, but also more noise, more data, more surface area, and more issues. Technical teams are already strained and will only get more so.

Fortunately, advances in LLMs mean that AI agents can work the way a human would. Traditional rules-based automation quickly hits its limits. But LLM-based automation can operate flexibly and learn, effectively becoming another member of the team.

Dropzone AI is one of the first companies to tackle this issue head-on. Instead of silencing the wave of new data, they look at all of it – using AI to handle the cognitive workload and freeing up humans to focus on what’s most important. It’s amazing to watch Dropzone’s agents in action, and we can’t wait to support Edward and the team as they transform security operations forever.