HubSpot Engineering

Latest posts

Writing lately

Aviator: HubSpot's Agent Harness

In this post

In June 2025, we started looking for an agent framework that teams across HubSpot could build on. Agents were quickly becoming a core part of how we built software, both internally and in our product, but agent implementations at the time varied wildly in terms of features, support, and reliability. Each team building an agent would make different choices, from model selection to architecture.

As many people have discovered, writing the first version of an agentic loop is easy. One can find a plethora of posts talking about how they wrote an agent in 100 lines of code. Scaling a harness to hundreds of teams is a different problem. As teams began building agents at HubSpot, we found we were repeating the same mistakes over and over, leading to agents that were slower, more expensive, and less reliable than they needed to be. Prompt caching, tool calling, and context management are easy to get subtly wrong, and each team was learning that on its own (even with AI writing most of the code!). We wanted to solve those problems once, so that a fix or an improvement would reach every agent, and every customer using one, at the same time.

While a wealth of tooling already existed in languages like TypeScript and Python, there were few options for building agents in Java, and none we found fit what we needed. Most of HubSpot's backend is Java, and we wanted agents to live inside the services teams already owned, using the same tools, data, and deployment process as the rest of their code. So we decided to build our own. That became Aviator, which is now the core harness that powers agents within HubSpot.

#How we got here

Some teams at HubSpot had begun building agents, while most were still making direct, one-off LLM calls, sometimes with custom tool calling setups. Our DevEx AI team had been experimenting with smolagents for internal tooling. It worked, but integrating Python agents with our Java services was a lot of work. We wanted teams to be able to add an agent to a service they already owned and deploy it with the rest of their code.

At the same time, we had been developing our internal CHIRP1 tools framework as part of our MCP servers, which had drastically lowered the barrier to building AI tools. Teams could add annotations to an existing RPC and expose it as a tool, with the schema generated from its request type.

This led us to ask: what if we could make building an agent just as easy? To this end we set a target: make it possible to write an agent in under 20 lines of Java.

#An agent in 20 lines

The framework we ended up with, called Aviator, lets us write an agent anywhere in our infra in just a few lines of Java code:

PipelineAssistant.java
AviatorAgent agent = agentFactory.newBuilder()    .withSystemPrompt("You are a sales pipeline assistant...")    .withModelClientProvider(() -> glmClient)    .addChirpTool("DealTools#searchDeals")    .addChirpTool("DealTools#getDealActivity")    .build(); try (AviatorAgentConversation convo = agent.startConversation()) {    AviatorMessage answer = convo        .sendAsync(textPrompt("Which deals closing this quarter need follow-up?"))        .getTerminalMessage();}

Critically addChirpTool allowed us to simply re-use RPCs from our existing RPC framework to quickly and easily add capabilities to our agents, without requiring engineers to take on a lot of net new work.

It became quickly apparent just how powerful this could be. In one example, during our infrastructure hackathon, an infra team that had never built an agent before had a debugging agent working in about fifteen minutes using their existing RPCs as tools.

As more teams adopted Aviator, we wanted the same approach to hold: they should be able to add capabilities and try new models without building another agent stack.

#Switzerland of Models

Teams were already building custom agents with different models, so Aviator needed to be model agnostic from the start. Our goal was to abstract away vendor-specific API details so an agent would never be locked into a particular model or provider.

Aviator defines a provider-agnostic transcript format for messages, tool calls, and results. Model clients translate between that format and each provider's API. Switching providers includes adapting the history: remapping tool-call identifiers and removing incompatible reasoning blocks so the next model can consume it.

We can even switch models mid-conversation, picking up with the existing messages and tool results. That lets us fall back to another provider during an outage or move to a more capable model when a task needs it.

[  {    "role": "ASSISTANT",    "content": [      {        "type": "TOOL_USE",        "id": "fc_68a1",        "callId": "call_abc123",        "name": "searchDeals",        "input": { "closeDate": "this_quarter" }      }    ]  },  {    "role": "USER",    "content": [      {        "type": "TOOL_RESULT",        "toolUseId": "call_abc123",        "toolName": "searchDeals",        "content": "{ \"deals\": [ ... ] }"      }    ]  }]
Fig. 1One tool call and its result, made while the conversation ran on OpenAI: as Aviator stores it, and as it is sent to each provider (shortened). When the next turn goes to Anthropic, the call's ID is rewritten with the prefix Anthropic expects, and the result's reference to it changes to match.

This common format has had a number of other benefits: The same format lets all of our agents speak to each other. We can pass a transcript from one agent to another, or resume a conversation with a different agent's prompt, tools, and model. This has allowed us to easily implement some interesting patterns: for example, we can run a fast triaging agent on an error and later "upgrade" that transcript to a more complex coding agent, without losing context.

The transcript is also the complete state of a conversation, which effectively gives our agents a durable log. It serializes to JSON, so an application can checkpoint it to storage and resume from it later, on a different machine, minutes or days afterwards. An agent can pause while it waits for a human to approve a tool call or for a subagent to finish, and when it resumes it picks up exactly where it left off.

We have built rich debugging and tracing tooling based on this format, allowing developers to easily ask questions like “would this conversation have gone better with a system prompt tweak?”

02040The whole turn33 sModel call 13.4 sLoad 2 skills and read a…0 sModel call 25.8 s2 CRM searches and a fi…1.1 sModel call 36.3 sA record summary and i…1.1 sModel call 47.6 sDraw the answer's cards0 sModel call 5: the answer7.1 sSpanTime since the prompt arrived (s)
010203040The whole turn33 sModel call 13.4 sLoad 2 skills and read a file0 sModel call 25.8 s2 CRM searches and a file read1.1 sModel call 36.3 sA record summary and its timeline1.1 sModel call 47.6 sDraw the answer's cards0 sModel call 5: the answer7.1 sSpanTime since the prompt arrived (s)
010203040The whole turn33 sModel call 13.4 sLoad 2 skills and read a file0 sModel call 25.8 s2 CRM searches and a file read1.1 sModel call 36.3 sA record summary and its timeline1.1 sModel call 47.6 sDraw the answer's cards0 sModel call 5: the answer7.1 sSpanTime since the prompt arrived (s)
An Aviator transcript rendered as a trace.
spanstartenddepth
The whole turn032.60
Model call 103.41
Load 2 skills and read a file3.363.392
Model call 23.49.21
2 CRM searches and a file read9.210.32
Model call 310.416.71
A record summary and its timeline16.717.82
Model call 417.825.41
Draw the answer's cards25.4225.462
Model call 5: the answer25.532.61
Fig. 2An Aviator transcript rendered as a trace.

#Scaling tool calling

Nearly immediately, we ran into the problem of having far too many tools. Reusing existing services made it easy to add tools and accelerated adoption. Very quickly, agents like Sidekick, our primary engineering assistant, needed access to over 3,000 tools. Aviator bundles a number of solutions which agent builders can choose from to help scale tool calling in their agents.

Aviator allows engineers to mark tools as "deferred." Their definitions stay out of the agent's initial context and become available through tool search.

Java
builder    .addDeferredChirpTool("DealTools#searchDeals")    .addDeferredChirpTool("DealTools#getDealActivity");

Here again, Aviator helps abstract over various vendor APIs. Anthropic and OpenAI, for example, each have hosted (but subtly different) tool search implementations, which Aviator leverages when available in order to be as in-distribution as possible for that model. For models we self-host, we generally lack these sorts of server-side tools, and instead Aviator provides a dedicated internal search index, generated on the fly based on the specific agent configuration.

#Packaging knowledge into skills

Finding a tool is useful, but often fails to give the agent context on how and when to use a tool. The industry has converged around the Skill spec as a way to provide agents with process-level instructions on how to complete a task, and Aviator brings native SKILL.md support to our engineering team. Beyond the standard SKILL.md, we add load-tools and suggested-tools metadata so that skills can bring their tools with them. A SKILL.md might look like this:

review-pipeline/SKILL.md
---name: review-pipelinenote 1description: Review the sales pipeline and identify deals that need follow-up.load-tools: searchDeals getDealActivitynote 2suggested-tools: getCompanyDetailsnote 3--- Find deals closing this quarter and check their recent activity beforenote 4recommending a follow-up. Use ./references/sales-stages.md to interpreteach stage, then run ./scripts/build_report.py to generate the report.note 5
  1. 1
    The name and description are all the agent sees at first: one line per skill, in every prompt.
  2. 2
    When the agent loads the skill, these tools become callable right away.
  3. 3
    These become findable through tool search, without being loaded.
  4. 4
    The instructions enter the agent's context only when it loads the skill.
  5. 5
    References and scripts stay on the filesystem until the agent reads or runs them.

The agent loads these skills as needed. Structuring capabilities in this way has been a massive productivity boost for our engineering teams working on flagship agents like Breeze Assistant2. This approach allows us to break down capabilities into independent skills that teams can own, evaluate, and roll out. Indeed, Breeze Assistant today uses only a very minimal embedded tool set, instead relying on a set of curated skills to dynamically load its required functionality.

One of the most useful features of skills, however, is their ability to embed code and scripts, and to enable that we need to go one step further and give our agents a code execution environment.

#Giving agents computers

Aviator can give any agent a "computer" by way of an embedded Python environment through Boomslang, as well as a sandboxed Bash environment, both running in WASM3. That lets the sandbox ship inside the Aviator Java library. Agents that execute code need no external dependencies. The sandbox is also secure by default: code can only reach the few egress points we define. And it starts quickly enough that we can spin up tens of thousands of short-lived environments across our agents.

To make this environment useful, we also provide the agent with a virtual filesystem. Through this filesystem, agents are given access to CRM data, HubSpot's context layer4, skills and more.

On Breeze Engine5, an agent's filesystem looks like this by default:

/├── work/                      read-write   scratch space for this conversation├── skills/                    read-only    the skills this agent can load│   └── review-pipeline/│       ├── SKILL.md│       └── scripts/build_report.py├── context/                   read-only    fetched on first read│   ├── index.md                            a manifest of everything below│   ├── foundation/                         HubSpot's context layer│   │   ├── company/icps.md│   │   ├── brand/voice.md│   │   └── user/README.md│   └── agent/<agent-id>/                   context the agent's own team supplies├── memories/                  read-write   what the agent remembers across conversations├── artifacts/                 read-write   documents the agent produces for the user├── uploads/                   read-only    files the user attached└── tmp/                       read-write
Fig. 3An example of a virtualized filesystem provided to an agent in Aviator.

Those files can come from an API, live in memory, or persist across conversations. We’ll dive more into our approach to filesystems in a future post!

#Aviator across HubSpot

Aviator now powers Breeze Assistant and agents across HubSpot's products and internal tools. Teams use it for everything from prospecting research, outreach, CRM insights, to code review, alert triage and more. Some agents interact directly with users; others run in the background, quietly executing on tasks. Sidekick Code, our cloud coding agent running on Aviator, now merges more than 2,500 pull requests a week.

Jan 2026Mar 2026May 2026Jul 2026Sep 2026Active deployments
Jan 2026Mar 2026May 2026Jul 2026Sep 2026Active deployments
Jan 2026Mar 2026May 2026Jul 2026Sep 2026Active deployments
Fig. 4Aviator adoption over time: production deployments using Aviator each week, December 2025 to September 2026. Counts omitted.

The teams building those agents own their prompts, tools, and product behavior. They share the runtime underneath. Adding a model provider, improving tool discovery, or extending the sandbox becomes work we can do once and make available across those applications.

We started with twenty lines of Java because we wanted building an agent to be an ordinary part of building a HubSpot service. The same idea has guided the rest of Aviator: let teams use the code and data they already have, enhanced with the power of agents.

This post has only scratched the surface of what Aviator can do, and there is a lot more we're excited to share. In future posts, we'll go deeper into the WASM runtimes, our virtualized filesystem implementation, how we write and manage agentic evals, and how we use tuning techniques to improve agents, tools, and skills automatically. Stay tuned!

Newsletter

For Agents

View Markdown

Share

More posts

From Data Silos to Feature Serving: How We Unified Fragmented Data for Training, Inference, and Reuse

HubSpot's feature store gives ML teams one place to find and serve features, whether they're ingested into it or left where the data already lives: a CRM property, a team's own service, or a team's Iceberg table.

Building the AI Retrieval Infrastructure Behind 20 Billion+ Vectors at HubSpot

Discover how HubSpot built a scalable AI retrieval infrastructure, managing over 20 billion vectors with Qdrant, to enhance semantic search and support diverse applications.

Automated Code Review: The 6-Month Evolution

Discover the evolution of HubSpot's Sidekick automated code review system and its impact on developer efficiency.