HubSpot Engineering

Latest posts

Writing lately

Embrace and Replace: Migrating ZooKeeper into Kubernetes

In this post

We recently migrated hundreds of ZooKeeper instances from individual server instances to Kubernetes without downtime. Our approach used powerful Kubernetes features like endpoints to ease the process, so we’re sharing the high level outline of the approach for anyone who wants to follow in our footsteps. See the end for important networking prerequisites.

#Traditional ZooKeeper Migrations

ZooKeeper is the foundation of many distributed systems, allowing them a powerful platform to rendezvous to take attendance and form clusters. Behind the scenes, it relies on a comparatively basic approach to forming clusters: each server instance has a config file listing all of the member hostnames and numeric ids, and all servers have the same list of servers, like this:

server.1=host1:2888:3888server.2=host2:2888:3888server.3=host3:2888:3888

Each server has a unique file called myid to tell it which numeric id it corresponds to in that list.

Adding and removing hosts can be done as long as a key rule isn’t violated: each server must be able to reach a quorum, defined as a simple majority, of the servers listed in its config file. The traditional way to migrate a ZooKeeper server to a new instance involves, at a high level:

  1. Configure and start a new host with “server.4=host:4…” in its server list
  2. Update the config files on the existing hosts to add the new server entry and remove the retired host from their server list
  3. Rolling restart the old hosts (no dynamic server configuration in the 3.4x branch)
  4. Update connection strings in clients (perhaps just changing CNAME records if clients re-resolve DNS on errors)

The downside of this approach is many config file changes and rolling restarts, which you might or might not have solid automation for. When we set out to move ZooKeeper to Kubernetes we started thinking about this approach, but figured out an easier way. Safer too, because in our experience each new leader election has a small risk of taking long enough to bring down the systems relying on them.

#New Approach

Our approach involves wrapping existing ZooKeeper servers in Kubernetes services and then does one-for-one server-to-pod replacements using the same ZooKeeper id. This requires just one rolling restart to reconfigure existing ZK instances, then shutting down the servers one by one. We won’t get into the weeds on the ways to configure Kubernetes topologies for ZooKeeper here, nor the low-level readiness checks, because there are many ways to do that with various pros and cons. The concepts discussed below work the same no matter the top-level topology.

We’ll proceed in five steps:

  1. Complete prerequisites to ensure our ZooKeeper cluster is ready to migrate
  2. Create ClusterIP services in Kubernetes that wrap ZooKeeper services
  3. Configure ZooKeeper clients to connect to ClusterIP services
  4. Configure ZooKeeper server instances to perform peer-to-peer transactions over the ClusterIP service addresses
  5. Replace each ZooKeeper instance running on a server with a ZooKeeper instance in a Kubernetes pod

For each of the steps below, we’ll include a diagram of our infrastructure topology. The diagrams will contain only two ZooKeeper instances for ease of understanding, even though one wouldn’t want to ever create a cluster with fewer than three.

Complete Prerequisites

Starting with a working ZooKeeper cluster, we’ll want to ensure that services on the host are capable of communicating with our Kubernetes cluster. We include a few ways to accomplish that at the end of this article.

Diagram of two boxes, ZK1 and ZK2, linked by arrows both ways, with a ZK Clients cloud whose arrows go straight to each of them.
Fig. 1Our starting state. A two-instance ZooKeeper cluster and some clients

Create ClusterIP Services

Create a ClusterIP service with matching Endpoint resources for each ZooKeeper server. They should pass the client port (2181) as well as the cluster-internal ports (2888,3888). With that done you should be able to connect to your ZooKeeper cluster via those service hostnames. Kubernetes ClusterIP services are useful here because they give you static IP addresses that act as load balancers to backend pods. In this case we're using them with a 1:1 mapping of service to pod so that we have a static IP address for each pod.

Diagram adding a Kubernetes service and endpoint for each of ZK1 and ZK2, while ZK Clients still connect straight to them.
Fig. 2Our cluster, with ZooKeeper still on physical hardware, is reachable via ClusterIP Services

Reconfigure Clients

Once you’re able to connect to your ZooKeeper cluster via Kubernetes ClusterIP services, this is a good time to pause to reconfigure all your clients. If you’re using CNAME records in ZooKeeper connection strings, change the DNS records. Restart all the clients if they don’t have recent clients that will re-resolve DNS entries on connection failures. If you’re not using CNAME records then you’ll need to roll out new connection strings and restart all client processes. At this time, old and new connection strings will still work.

Diagram in which ZK Clients now go through each server's service and endpoint, while ZK1 and ZK2 still link directly.
Fig. 3Clients are now communicating with our ZooKeeper cluster using ClusterIP service instances

Reconfigure ZooKeeper Instances

Next up, we’ll make our ZooKeeper servers do their peer-to-peer communication via those ClusterIP services. To do this, we’ll modify the config files to incorporate the addresses of the ClusterIP services. It’s also essential to configure the zk_quorum_listen_all_ips flag here: without it the ZK instance will unsuccessfully attempt to bind to an ip address that doesn’t exist on any interface on the host, because it's a Kube service IP.

server.1=zk1-kube-svc-0:2888:3888server.2=zk2-kube-svc-1:2888:3888server.3=zk3-kube-svc-2:2888:3888zk_quorum_listen_all_ips: true

Rolling restart those hosts and now we’re ready to start replacing hosts with pods.

Diagram in which ZK1 and ZK2 no longer link directly but reach each other through the other's service, as ZK Clients do.
Fig. 4ZooKeeper instances are now communicating with their peers using ClusterIP service instances

Replace ZooKeeper Hosts with Pods

One server at a time we’ll do the following steps:

  1. Select a ZK server and its corresponding ClusterIP service
  2. Shut down ZK process on the server
  3. Start a pod configured with the same server list and myid file as the shut-down ZK server
  4. Wait until ZK in the pod has started and synced data from the other ZK nodes

That’s it, your ZooKeeper cluster is now running in Kubernetes with all of the previous data.

Diagram in which ZK1 has become a Kubernetes pod, still reached through ZK1 Service and ZK1 Endpoint, while ZK2 is unchanged.
Fig. 5Cluster after one round of pod-replacement. ZK1 is now running in a pod without ZK2 knowing anything has changed

#Networking Prerequisites

For these steps to work well, there’s some network setup to handle. You need to take steps to ensure the following:

  1. Kubernetes pod IP addresses need to be routable from all servers that need to connect to ZooKeeper
  2. All servers connecting to ZooKeeper must be able to resolve Kubernetes service hostnames
  3. Kube-proxy must be running on all servers that need to connect to ZooKeeper so that they can reach the ClusterIp services

These can be accomplished in a few ways. We use an in-house network plugin similar to Lyft's https://github.com/lyft/cni-ipvlan-vpc-k8s plugin or AWS's https://github.com/aws/amazon-vpc-cni-k8s which assign AWS VPC IP addresses to pods directly, instead of using a virtual overlay network, so all of our pod IPs are routable from any instance. Overlay networks like flannel (https://github.com/coreos/flannel) would work too, as long as all of your servers are attached to the overlay network.

Newsletter

For Agents

View Markdown

Share

More posts

Hedging Erasure-Coded Reads in HDFS

Erasure coding halves the disk our HBase data needs, but every byte lives on one DataNode. Hedging slow reads by rebuilding the slow cell from its stripe's parity cut slow-read host-minutes on our case-study cluster by 98%.

Building Observable Access Control at Scale: A Rule Engine Approach to JITA

Discover how HubSpot transformed its access control system with a parallel rule engine, enhancing security and observability while managing thousands of daily requests.

Building the AI Retrieval Infrastructure Behind 20 Billion+ Vectors at HubSpot

Discover how HubSpot built a scalable AI retrieval infrastructure, managing over 20 billion vectors with Qdrant, to enhance semantic search and support diverse applications.