---
title: "Our Journey to Multi-Region: Routing API Traffic"
summary: "This is the second installment of a five-part series about HubSpot’s year-long project to develop an EU data center, launched in July 2021."
date: 2022-04-21
authors: [jonathan-haber]
topics: [Infrastructure]
canonical: https://product.hubspot.com/blog/routing-api-traffic
cover:
  scene: routing-api-traffic
  alt: "A door stands on its own in a field; people queue up holding their tokens, and each time it swings open on its hinge the doorway glows with that person's own town, a European one with a church spire in the afternoon sun or a North American one by moonlight, and they walk straight through into it."
---

**This is the second installment of a five-part series about HubSpot’s year-long project to rework our platform for multi-region support. Check out the introduction to the series&#x20;**[**here**](/blog/developing-an-eu-data-center/)**.**

In this post, we will explore HubSpot’s strategy for adding multi-region support to incoming API calls. For some background, we expose a public API so that external developers can integrate with, or build on top of, the HubSpot platform. This public API is hosted on _api.hubapi.com_ (you can find more details [here](https://developers.hubspot.com/docs/api/overview)). Requests are authenticated using either OAuth or a HubSpot API key (_hapikey&#x20;_&#x66;or short). As we started to plan the multi-region project, it became clear that this sort of global domain would be a challenge. For US customers, API calls should go straight to our US data center. Similarly for EU customers, API calls should go straight to our EU data center. \
\
Theoretically, an easy solution would have been to split _api.hubapi.com_ into _api-na1.hubapi.com_ and _api-eu1.hubapi.com_. This would solve the problem on our end, as each domain could trivially route traffic to the correct Hublet. However, it would push the problem onto our external developers, who would have to update their code to use one of these new domains (based on where each HubSpot customer is hosted). This approach would effectively break all existing code written against HubSpot's public API.

Any solution which required external developers to update their code would take a very long time to roll out - and it would be a big project in its own right, with its own risks and challenges. One of the things we learned throughout this project is the value of _detangling_. It was very tempting to say "if we just change X, then challenge Y goes away or becomes much simpler". But if we went down that path, then completing the project would become dependent on a tangled web of tangentially related projects. Pretty quickly, we’d end up in a constant state of [yak shaving](https://americanexpress.io/yak-shaving/). And any delay or setback in one of these related projects would likely jeopardize the timeline or success of the overall project.&#x20;

![Diagram of a GOAL with curly, looping arrows out to four blockers, one blocker looping on to another and one to a SUB-BLOCKER.](/assets/blog/routing-api-traffic/blockers.svg)

Sometimes tangling is the right way to go, but we wanted to exhaust all other options before going that route. In this case, it was clear that we should detangle these initiatives. We could design a new, _Hublet-ized_ way to interact with our public APIs, but that shouldn't be a prerequisite for launching the EU Hublet. To unblock the launch, we needed to find a way to route API calls that worked transparently with all existing 3rd party code.

## Solution

One approach is to route all API calls to a central routing service, hosted in the US, for example. This service could inspect the request, figure out which Hublet it should get routed to, and forward it there.&#x20;

![World map of a central routing detour from a User in Sweden to a Routing Service in the US and back to the EU Data Center in Germany.](/assets/blog/routing-api-traffic/screen-shot-2022-04-20-at-9-09-48-am.webp)

This would've worked from a functional perspective, but it failed to deliver on the project goals (outlined [here](/blog/developing-an-eu-data-center/)). In particular, this solution would hurt performance, reliability, and data localization. From a performance perspective, imagine a HubSpot user in Sweden, hosted on our EU data center. If they made an API call, this should get routed directly to the EU Hublet in Germany. However, with a central routing service, the API call would travel to the US and then back to the EU, resulting in hundreds of milliseconds of unnecessary latency. Hosting the central routing service in the EU wouldn't solve this problem; it would have created the opposite problem, where API calls from the US would become slow. From a reliability perspective, this central routing service would've been a global point of failure, which is exactly what we were trying to avoid. And from a data localization perspective, we wanted to be able to tell our EU customers that their data is processed and stored in the EU. We can't do that if all of their API calls (many of which contain PII) were routed through the US.

At first glance this looked like a very tricky problem. We needed to take API calls coming in on a global domain and route them to the correct Hublet without sacrificing performance, reliability, or data localization. One option was to still use a routing service, but to run copies of it all around the world, using something like GeoDNS to automatically route calls to the nearest instance. This seemed promising, but it was pretty complicated and sounded a lot like a CDN. At HubSpot, we use Cloudflare as our CDN, and all incoming API calls already flow through it (in order to apply [DDoS protection](https://www.cloudflare.com/ddos/) and [WAF](https://www.cloudflare.com/waf/) rules). Around this time we were starting to use [Cloudflare Workers](https://workers.cloudflare.com/) to power other parts of the HubSpot product. This is a really exciting technology that allows you to run code directly in Cloudflare's edge data centers.&#x20;

If we could implement our routing service as a Cloudflare Worker, then it would be like having a distributed routing service running in all of Cloudflare's 250+ data centers around the world. This seemed like a good approach, so we decided to do a proof of concept. The idea was to turn _api.hubapi.com_ into a thin facade backed by a Cloudflare Worker. This worker would decide whether each request should go to _api-na1.hubapi.com_ or _api-eu1.hubapi.com_ and forward it there. To minimize the performance and reliability risk, we wanted the worker to be able to make these routing decisions without any network calls, solely based on the request data itself. To make this possible, we decided to embed Hublet information directly into our API keys and OAuth tokens. Any existing tokens that predate this change were assumed to belong to the na1 Hublet. The pseudo code for the worker looked like this:

```js
async function forwardRequest(request) {
  let targetHublet;
  if (hasApiKey(request)) {
    targetHublet = extractHubletFromApiKey(request); 
  } else if (hasOAuthToken(request)) {
    targetHublet = extractHubletFromOAuthToken(request);
  } else {
    return new Response('Request is missing auth', { status: 401 });
  }

  return fetch(request, {
    cf: { resolveOverride: `api-${targetHublet}.hubapi.com` },
  });
}

addEventListener('fetch', event => {
  event.respondWith(forwardRequest(event.request));
});
```

We rolled this out to QA for testing and it worked great. Using this approach, based on Cloudflare Workers, we were able to route API calls directly at the edge, without harming performance or reliability. If that same Swedish user made an API call, it would get routed to a Cloudflare data center in Sweden, and then straight to HubSpot's EU data center in Germany. After testing extensively in QA, we rolled this out in Production. For more than a year, all API calls to _api.hubapi.com_ have been flowing through this codepath.

## Learnings and future improvements

Overall we've been very happy with our solution using Cloudflare Workers. There are definitely pain points around local testing and observability, but we've been discussing these issues with Cloudflare and they're investing a lot in the Workers developer experience.&#x20;

One implementation issue we ran into is that, at first, our Worker was changing the URL when firing the subrequest. When testing, we ran into issues with some of our endpoints that use signature validation. It turns out that the client was computing a signature based on _api.hubapi.com_. But when the request reached our API after passing through the Worker, the Host header had changed to _api-na1.hubapi.com_, which invalidated the signature. The fix was to use the special [resolveOverride](https://developers.cloudflare.com/workers/runtime-apis/request/#requestinitcfproperties) directive, which allows you to forward a request without changing the Host header.

Looking forward, we still want to get traffic off of _api.hubapi.com_ and onto more granular domains. This will allow us to use tools such as Cloudflare [Regional Services](https://blog.cloudflare.com/introducing-regional-services/) and Cloudflare [Geo Key Manager](https://blog.cloudflare.com/introducing-cloudflare-geo-key-manager/) in order to provide even stronger data localization guarantees to our customers.

Stay tuned for our next post in this series, explaining the system we built to ensure that our MySQL auto-increment IDs are globally unique.

#### Other multi-region blog posts:

- [Introduction to our multi-region project](/blog/developing-an-eu-data-center/)
- [How we made our MySQL auto-increment IDs globally unique](/blog/our-journey-to-multiregion-vtickets/)
- [How we replicate SQL tables that store global configuration](/blog/mysql-data-replication/)
- [How we handle Kafka messages that need to be consumed cross-region](/blog/kafka-aggregation/)

_These are the types of challenges we solve for on a daily basis at HubSpot. If projects like this sound exciting to you, we’re hiring! Check out our [open positions and apply](https://hubs.life/blog-product-careers)._
