HubSpot Engineering

Latest posts

Writing lately

Reducing Downtime from HBase Table Descriptor Modifications

In this post

#Background

#Descriptor Modifications at Scale

At HubSpot, our HBase clusters serve over 25 million requests per second at our daily peak traffic. We automatically run countless table descriptor modifications every day to maintain optimal performance. These modifications are necessary to facilitate maintenance like correcting inefficient table properties, tuning target region sizes as cluster sizes change, and coordinating HBase’s normalizer. HBase table descriptor modifications are one of the most fundamental admin operations, and, by default, they cause a totally unacceptable degree of downtime for our end users.

In this post I will explain how you can trivially configure two new settings in HBase to reduce — perhaps totally eliminate — downtime, latency, and user-facing errors induced by table descriptor modifications. First, I’ll proactively explain a few key terms.

#Glossary

HBase

HBase is a distributed, scalable, big data store, modeled after Google's Bigtable. It's part of the Apache Hadoop ecosystem and designed to provide random, low-latency read/write access to huge datasets.

HBase Tables

Data in HBase is organized and accessed through tables, which consist of rows and columns. Each table is horizontally partitioned into regions, which are the basic building block for scalability and distribution. Tables have “Table Descriptors”, which dictate their configuration, specifying details such as compression type.

HBase Regions

A region is a subset of a table containing all the rows between a start key and an end key. As data grows, regions can be split and merged automatically to maintain system performance. Regions are distributed and managed across the servers in your cluster to distribute the read/write workload for your table.

99th Percentile Client-Side Latency

We measure the duration that each request to HBase takes from the client’s perspective. By taking the 99th percentile values of this histogram we are able to quickly understand the worst-case user experience with respect to product “slowness.”

Client Error Rates

We measure the volume of exceptions thrown by HBase clients at HubSpot. We monitor the number of exceptions divided by the total number of requests to get a quick sense for our system reliability from the HubSpot product perspective.

Table Descriptor Modifications

HBase allows the modification of existing table descriptors. Table descriptor modifications allow for the altering of table properties. These properties may include the table’s column families, compression type, data block encoding, target region size/count, normalizer configuration, and bloom filter configuration — just to name a few.

Descriptor Modification Downtime

HBase is a strongly consistent datastore by default. This means that there can be only one live copy of any given region at a time to eliminate the possibility for inconsistencies.

A table descriptor modification tells HBase that all regions for the given table must be “reopened” — moved from an OPEN state, to a CLOSED state, and back to an OPEN state — in order to apply the desired configuration changes. Because there is only one live copy of any given region, a region being in a non-OPEN state is effectively downtime for the region from the user’s perspective.

Until recently, HBase’s only behavior was to accept a table modification request and then attempt to reopen every region as quickly as possible. This expeditious pace sounds like a righteous goal, but we found that, in practice, it was disruptive for our user experience.

#The Problem

#Modifications Cause Measurable User Downtime

Historically, our automation has executed all table modifications in the middle of the night. It does this to mitigate customer facing pain due to mass region re-opening. We found that, even in our quietest hours, descriptor modifications were still too disruptive. Two of our most important metrics for gauging user experience are 99th percentile client-side latency and client-side error rates. See the chart below where a table descriptor modification at approximately 1 am EST caused latency to spike from a few milliseconds to 30 seconds:

Panel of 99th percentile read latencies: around 01:07 several lines jump from 0 to as high as 30 s, then fall back by 01:16.

This elevated latency not only caused slowness for users, but also breached timeouts which caused client facing error volume to spike:

Client Error Rate panel: flat at 0% until about 01:06, then climbing to nearly 0.08% at 01:13 and back to 0% by 01:15.

#The Solution

#Eliminating the downtime

I opened up HBASE-28215 which suggested that we could solve table descriptor modification induced downtime by throttling HBase’s ReopenTableRegionsProcedure — the ReopenTableRegionsProcedure being the operation kicked off as a result of any table modification. Again, the problem with the reopen procedure is that it kicks off sub-procedures to reopen each individual region, and it does so as quickly as possible which is too disruptive.

In the corresponding pull request we accomplished exactly what we sought out to do. We introduced two new configuration options:

  • hbase.reopen.table.regions.progressive.batch.size.max
    • This is an integer which represents the number of individual region reopen procedures to be created by the parent ReopenTableRegionsProcedure in a single batch. This defaults to Java’s Integer.MAX_VALUE (2147483647) so it should be a no-op for any table with fewer than billions of regions.
  • hbase.reopen.table.regions.progressive.batch.backoff.ms
    • This is a long which represents the maximum milliseconds to be waited between reopen batches. Hyperbolically, you could configure your batch size to 1 and your backoff milliseconds to 60,000 — this would result in reopening no more than 1 region per minute due to a single ReopenTableRegionsProcedure. This value defaults to 0 which implies that we will do no additional throttling without explicit configuration.

#Our configuration

We introduced the following customization to our HBase configurations at HubSpot:

  1. hbase.reopen.table.regions.progressive.batch.size.max = 50
  2. hbase.reopen.table.regions.progressive.batch.backoff.ms = 1000

With these configurations in place we ensure that we will cool down for at least one second between reopen batches, and we ensure that no batch will contain more than 50 regions.

We chose these initial values for a few reasons:

  1. Our automated region size tuning aims for, on the large end, 10GB regions
  2. So our tables with upwards of 1PB on disk would have about 100,000 regions
  3. With a batch size of 50 and a backoff of one second, we would take between 30 and 60 minutes to reopen 100,000 regions

Accomplishing any given descriptor modification in no more than an hour felt like a reasonable starting point. With these initial configurations, we saw such compelling improvements that we have felt little need to iterate on these values.

#Evidence of improvement

After applying the aforementioned changes we ran another table descriptor modification of the same table and at approximately the same time (1 am EST). It had no clear impact on 99th percentile latency:

Panel of 99th percentile read latencies: the main lines wander between 10 and 50 ms all night, with no spike around 01:00.

Similarly, the modification caused approximately zero additional runtime exceptions for HubSpot users:

Client Error Rate panel: flat at 0% the whole time, on the same 0 to 0.08% scale as before.

The sustained, lower volume, region reopenings are preferable to users because low server-side error volume can be hidden “under the hood.” This is possible due to the HBase client’s implicit retrying, and the retries' increased propensity for success due to the mild nature of the disruption.

#A Bonus Suggestion

There are some very specific cases where reopening regions may not be necessary for your table modification. For example, if your change has no immediate effect on the composition of regions, or if you just don’t care whether your change is applied expeditiously. If you find yourself in this situation, then you can make use of a feature introduced in HBASE-25549: what we refer to as a “lazy table modification” at HubSpot. One can use this via the Admin interface like so:

Java
import org.apache.hadoop.hbase.client.Admin;import org.apache.hadoop.hbase.client.TableDescriptor; try (Admin admin = getAdmin()) {  TableDescriptor modifiedTableDescriptor = getModifiedTableDescriptor();  boolean reopenRegions = false; // don't reopen regions after modification  admin.modifyTable(modifiedTableDescriptor, reopenRegions);}

As the release notes suggest, one should use this API with care because regions may be in an inconsistent state from when they are lazily modified to when they have all reopened.

#Conclusion

By simply configuring both hbase.reopen.table.regions.progressive.batch.size.max and hbase.reopen.table.regions.progressive.batch.backoff.ms, you can turn a downtime inducing admin operation into a trivial one.

These features, plus “lazy table modifications”, and many more are available in HBase’s recent 2.6 release. Consider this your sign to begin an upgrade!

Are you an engineer ready to make an impact? Check out our careers page for your next opportunity! And to learn more about our culture, follow us on Instagram @HubSpotLife.

Newsletter

For Agents

View Markdown

Share

More posts

Incident Report: August 11, 2026

A change to a service's team-ownership metadata cut its new instances off from a cache, and a rollback restarted every instance, so CRM record pages failed for 18 minutes on August 11. What happened, and what we're changing.

From Data Silos to Feature Serving: How We Unified Fragmented Data for Training, Inference, and Reuse

HubSpot's feature store gives ML teams one place to find and serve features, whether they're ingested into it or left where the data already lives: a CRM property, a team's own service, or a team's Iceberg table.

Hedging Erasure-Coded Reads in HDFS

Erasure coding halves the disk our HBase data needs, but every byte lives on one DataNode. Hedging slow reads by rebuilding the slow cell from its stripe's parity cut slow-read host-minutes on our case-study cluster by 98%.