AI Council 2026 Recap

Last month was AI Council. Every year at this conference at the intersection of AI and Data, we see familiar faces and seek to learn what will be common in 6-9 months. This year, we were involved in an event, a keynote, and a talk track. Here’s what we learned.

Choppy waters in AI adoption

In conjunction with AI Council, we hosted an event called Ship It, co-organized with portfolio companies LanceDB and Motherduck. How better to spend an evening after a great conference, than a slow boat out to sea!

The gap between a promising demo and a tool people reach for every day remains the real barrier, and it has more to do with behavior than with model capability. A senior engineering leader at a major digital media company put it plainly at Ship It: their single biggest obstacle to rolling out a promising AI tool was getting non-technical stakeholders to try it at all, well ahead of capability and cost. Adoption tends to look lumpy rather than smooth: usage stays flat across a team for weeks, and due to iteration on the workflow or on education, usage spikes. An interesting question worth obsessing over is how you trigger that first behavior change.

The head of data at a large research and advisory firm, a heavy Claude Code user running several sessions at once, argued that most enterprises burn tokens with wild inefficiency, that the same job often runs on a frontier model when a small local one would clear the bar, and that this is why he has been tuning local-model setups for his own workflows, routing the cheap high-volume calls to models that run on his own hardware and reserving the frontier models for the few jobs that truly need them, while he watches where inference economics and model routing eventually settle. His line: a bazooka where a slingshot would do.

A CISO at a major insurance company pushed back on the fear narrative around AI security. His view is that AI is at least as powerful for defenders as for attackers, and that the window of exploitability is shrinking, because AI-written code gets reviewed, pen-tested, and patched faster than any human pipeline can manage. His sharper claim was that security teams are becoming engineering teams. Every security person on his team is an engineer, and they have built an internal AI platform with agents that read threat intel and confirm whether vulnerable methods even run in production. The layer he flagged as most underbuilt today is identity and policy for agents. Existing IAM was built around human users who sign in a few times a day, so it strains against the 200 to 10,000 agents that may live on a single endpoint.

Our next event is on June 30th adjacent to the AI Engineer World’s Fair. Catch us with ping-pong paddles in our hands at https://luma.com/localserve. 

What's making waves in Inference Systems

In the good old days of 2024, AI engineers at the application layer could spend their days on the TypeScript, Python, and prompt-engineering layers, mastering context construction, retrieval pipelines, and model management. In 2026 and 2027, those same engineers will need to pay a lot more attention to inference systems. Whence I spent AI Council on inference: moderating the keynote panel and curating a dedicated speaker track.

I moderated the keynote panel "Modern Inference for Modern Workloads" with Charles Zedlweski, the Chief Product Officer of Together AI; Tuomas Tintamaki, the Lead Research Scientist on NVIDIA's OSS LLMs; and Charles Frye, a Member of Technical Staff at Modal. The goal was to surface what the major inference providers are seeing early and to pull back the curtain a bit. The blunt question I put to the group is the one worth asking anyone at this layer: why is a dedicated inference provider necessary when OpenAI already exists? The honest answer runs deeper than "price matters." AI usage profiles are diverse, and the acceleration of serving profiles and strong open-source models keeps widening the reasons to control where and how a model runs.

Some highlights:

• Fine-tuning is evolving, and it is far from dead. It is seeing a resurgence, often rebranded as reinforcement learning, and more organizations are finding alpha in their private data, using post-training to specialize and optimize models for specific domains.
• The shift to agentic workflows yields different inference demands. The industry has moved from chat-centric interactions to long-horizon tool-using agents, and that shift is the primary driver behind the surge in token usage. (See our recent investment in Sail Research)
• Model routing is the application developer's job. The panelists argued that deciding which model to call for a given task is the responsibility, and the competitive advantage, of the application developer rather than the inference provider. Decomposing complex requests into smaller, optimized sub-models is a real defensibility moat.

You can view the full conversation here.

I also led a talk track on inference systems, aimed at people outside the inference layer who want to understand its key ideas and why they should care. You have probably seen the "intelligence vs price" charts. Behind that simple tradeoff sits a panoply of mechanisms: you can quantize a model to shrink its memory footprint, which moves tok/s and GPU usage; you can optimize the cache it reuses across requests to cut the inference passes needed for the next tokens; you can build specialized tokenizers and internal representations tuned to your domain; you can even question basic assumptions, like whether the generated language should be "natural" at all. Each of these has a startup or two forming around it, with early traction. The track was a guided tour of that frontier, and I'm grateful to the speakers:

1. Omead (PrismML) explained quantization and why making models smaller is both Art and Science. 

2. Neil (Sail Research) talked about why if inference is cheap and available, agent impact goes way up. 

3. Vik (Moondream) showed why you might want to train a tokenizer, and how their VLM is much better. 

4. John (Mozilla) taught us what's going to be essential for open source models to be successful. 

5. Diogo (Typesafe) admitted to his original sin in building ChatGPT: Instruction Tuning. 

6. Yaroslav (Incept Labs) showed us that we may not be shackled to backprop forever. 

‍

Outside my track, LanceDB spoke on trillion-scale retrieval and multimodal data management.

Inference engineering is no longer just for the CUDA kids, so expect to hear a lot more about it over the next 12 months.

‍