---
title: "From Data Silos to Feature Serving: How We Unified Fragmented Data for Training, Inference, and Reuse"
summary: "HubSpot's feature store gives ML teams one place to find and serve features, whether they're ingested into it or left where the data already lives: a CRM property, a team's own service, or a team's Iceberg table."
date: 2026-11-10
authors: [leena-bhandari, andrea-cicalese, durai-ramasubramanian]
topics: [Data, AI]
canonical: https://product.hubspot.com/blog/from-data-silos-to-feature-serving-how-we-unified-fragmented-data-for-training-inference-and-reuse
cover:
  scene: from-data-silos-to-feature-serving-how-we-unified-fragmented-data-for-training-inference-and-reuse
  alt: "Teams carry their own tables onto a lawn and set them end to end, a farmhouse table with a pot, a tea trolley with a teapot and a little folding table propped up on a book, while the model sitting at the middle table gets a helping from every one and tucks in."
---

## Why is fragmented data a bottleneck for engineering teams?

Every machine learning model runs on features which are signals it learns from and reasons over. In a large organization they are scattered across operational databases, warehouse tables, event streams, and the pipelines that individual teams have built. An ML engineer starting a new model first has to find the signals they need, make sense of the data, then write one-off code to extract and transform it. The next team that needs an overlapping signal builds it all over again. This makes shipping a new functionality slow and hard. The need for a unified store has sharpened as teams build LLM and agent powered features needing a shared, trustworthy place to find and serve context at scale.

A _Feature store_ is a central layer where features are defined once, then discovered and reused across teams and across the training-to-serving boundary. But in a large organization, most of that data already lives somewhere, and copying all of it into one more store would add another silo. We built HubSpot’s feature store as a single interface for all these signals. Teams can ingest features directly into the store, or leave the data where it already lives, whether that is a SQL table a team already owns, a property in the CRM object, or a value served by a team’s own service.

## Feature Store: What is It

Feature stores like Uber’s [Michelangelo Palette](https://www.uber.com/pa/en/blog/from-predictive-to-generative-ai/), the [Databricks Feature Store](https://www.databricks.com/blog/what-is-a-feature-store), and open-source [Feast](https://feast.dev/) are built as part of the ML infrastructure that defines, stores, and serves features for ML models. It lets teams discover and reuse features instead of rebuilding them, and keeps the feature values consistent between training and inference. HubSpot’s Feature store is shaped by how data is organized at HubSpot. Each feature is keyed by an identifier, we call this an **entity group**. For example, say there is a feature “deal_amount_changes_30d” representing the number of times a deal's amount changed in the last 30 days. Here,

- **Feature value** is the computed value like `5`
- The **entity group** is what the value describes, here this is `DEAL`
- Entity group is made of one or more **entities** like `accountId` & `dealId`
- Every value also carries a **feature timestamp** i.e. the moment it was valid

Feature store is built from three components -

1. **Registry** - This is the feature catalog that stores the feature definitions used for discovering features.
2. **Online store** - This serves the latest value of each feature with low latency, for models making model inferences in real time. HubSpot uses [Apache HBase](https://hbase.apache.org/) as its online store.
3. **Offline store** - This keeps a full history of the feature values for analysis, model training and batch processing in [Apache Iceberg](https://iceberg.apache.org). HubSpot standardized on Apache Iceberg as its open data lake format, where a single copy of data sits in low-cost cloud storage and many engines such as Spark and Snowflake query it directly.

Online and the offline stores are kept in sync with the online store serving the latest features for low-latency retrieval and the offline store storing the entire feature history designed for analysis and batch processing.

### Keeping the Offline Store in Sync with the Online Store

The online store only holds the latest value of each feature for an entity group. Hence, it needs to be synced to the offline store regularly to make the feature history available to the users. If we stored the updates as they arrive, each feature would have its own history, and answering ‘what did this `deal` look like at time `t`?’ would mean joining many feature tables, and running such complex queries again for every training example. Instead, we build point-in-time rows as we write the history, so a single lookup returns all feature values for an entity group at that moment.

Every write to the online store also publishes an update event. A Spark streaming job collects all updates and stages them as partitioned files on S3. A loader job runs daily and loads each entity group into Apache Iceberg. The loader builds point-in-time correctness into the data as it writes so that the training does not need joins of many smaller tables within an entity group. This gives the offline store the following characteristics:

- **Wide, point-in-time rows**: each row holds the value of every feature for an entity, at a specific moment. i.e., a record retrieved for time `t` sees only values that existed at `t`.
- **Correct timeline despite late events**: a late event fixes its own row and every later row that carried the old value forward.
- **Optimized wide rows:** the loader merges events into existing rows where it can, and keeps fast- and slow-changing features in separate tables, which limits the cost of copying values forward.
- **No manual schema update**: new features get their columns automatically, and backfilled historical values land on the same point-in-time timeline.

The offline store format has proved useful for training models for use cases like similar deals, next best action recommendation and some internal models.

## How do we bring the data together and make it discoverable?

We built the feature store to abstract away a feature's source so that all features need the same contracts for retrieval and are easy to find.

![The feature store's architecture: a feature registry on top resolves each feature's source, for online retrieval from HBase and offline retrieval from Iceberg.](feature-store-architecture.svg)

We built the feature store from three components, each meeting a core need of its users:

- **Discovering features**: how the registry and the data catalog help teams find features and locate each feature’s source
- **Serving features at inference time**: how features are served from the online store, read from the CRM, or fetched from a team’s own service
- **Training and analysis on offline data**: how Apache Iceberg tables plug in as offline sources with no ingestion, (‘Bring Your Own Table’) alongside the feature store’s own Iceberg catalog

### Feature Discoverability

Features hosted in different sources are only useful if they can be found and located. The feature store's services need to look up each feature’s source, and the users need to browse available features.

When a model asks for a feature, the retrieval service checks the registry to decide where the value should come from. The registry, backed by MySQL/Vitess, stores the online and offline source for each feature, along with its description, tags, metadata, and owners, and is replicated in all regions for the services to query it.

To make features discoverable to feature store users, the registry is connected to [OpenMetadata](https://open-metadata.org/), which HubSpot uses as its data catalog. Features show up there alongside other data assets, with their definitions, ownership, and lineage, so that teams can find them and any other data at HubSpot. To make features easier to find, we also capture details for each feature such as a summary of what a feature represents, a sample value, its schema, coverage, and where it is already used. Some of this is generated by analyzing the pipelines that compute the feature in the organization’s source code. We built a discovery skill on this catalog that uses the context so that an ML engineer can describe their use case in plain language and get the relevant features surfaced. This capability is live and still maturing, as we continue to improve how much context is captured automatically and how well the skill matches a use case.

### Serving Online Features from Multiple Sources

A feature that has been ingested into the feature store is stored in HBase and served from there. But when a signal already lives in another system, copying it into HBase would create a second copy to be kept in sync. To avoid data duplication & data out-of-sync issues, the feature store serves such values directly from their source at read time. This is useful for data that already exist in operational systems and need only lightweight transformations such as type casts or date conversions. Feature store supports consumption from source in the following ways -

#### 1. Mirroring data from the CRM

Most of the useful features for new AI capabilities are found to be within the CRM platform itself. These are CRM object properties written and updated by users and automated systems. These are registered as features in the registry by linking the object & property names to the feature’s entity group in the online source and the value is read on demand from the CRM. Today, 10% of the total features registered are directly served from CRM and _~11M feature values are retrieved daily_ from CRM.

#### 2. Served from other RPC/API

Some teams keep feature values in their own data stores and services. For these, the feature store defines a standard interface on top of our internal RPC framework, with a fixed request and response format for feature retrieval. Once the interface is implemented by a service, it can be registered as the online source for corresponding features in the registry & the service becomes a dependency for that feature’s retrieval. This lets the feature store serve features that a team’s service either stores or computes on demand. 5% of the total features fall into this category and accounts for _3% of the total feature retrievals_.

### Offline Feature Retrieval from external sources

Any team that already keeps its data in Apache Iceberg can plug it into the offline store as a source, with no ingestion needed. This is called “Bring Your Own Table” (BYOT). It is set up in the following way:

1. We need tables to follow a minimal specification like one column for feature value, feature timestamp, entity value and optionally one column for feature metadata.
2. The data in the tables should be written in append mode to preserve feature history.
3. The table owner grants read-only access on that table to the feature store’s shared access role, which represents all feature store users. The grant covers only that table, so the rest of the team’s data stays untouched.
4. Register the features with their offline source mapped to the Iceberg table specifying columns for feature value, entities & timestamp.

Upon successful registration, these features are served through the offline store. Today, 23% of the total features fall into this category.

We built automation for surfacing Offline store tables in Snowflake for feature exploration and analysis. We provide Spark libraries for offline batch processing which is useful for offline inference & backfills. If the features are selected for online retrieval, their latest values get synced to the online store. All of these functionalities work out of the box for any features without additional tooling.

## The Benefits and Trade-offs of Serving from the Source

There are benefits and trade-offs of serving features from multiple sources instead of ingesting all the feature values into a centralized feature store. Let’s explore them in this section.

#### Benefits of serving data from external source

- **Storage saved**: The data isn’t copied into the feature store, so there is no extra storage to pay for.
  - Existing data is used as is without the need to ingest into another system for feature exploration & analysis. On average, we see that this saves _~1M records per use case_.
- **No sync pipelines**: There are no pipelines to build or maintain to keep a second copy in sync with the source.
  - Eliminating the training-serving skew - Because training-time feature values are synced to the online store, backend teams read the same values at inference time. This removes an error-prone re-implementation, along with the tight coordination needed between ML and backend engineers. In one use case, we saved _~6 weeks of development time_.

#### Trade-offs of serving data from external source

- The feature store cannot guarantee latency and availability for these features, because retrieval depends on an external system.
- Data quality, coverage, and lineage are harder to guarantee, because the feature store has limited visibility into the source pipelines.
- The Data has to be translatable to the feature store format to be useful.
  - For Offline serving, the data has to already be in Iceberg in the required format. Not every table is laid out for the point-in-time reads the feature store optimizes for. This affects the query and batch-processing performance.

## Closing Insights

Teams started adopting the feature store quickly. We observe a daily volume of _~1.4B feature ingestions_ into the online feature store. They get a low-effort way to register features from different sources, one interface to retrieve them, and tooling for training and batch processing that works out of the box.

Not having to keep multiple sources’ data in sync is an underrated benefit. At HubSpot’s scale of billions of records, it means no complex pipelines to build and run, no data freshness issues between the source and the feature store, alongside preventing duplicate storage.

ML engineers get from development to production faster with the feature store, which speeds up the release of new functionality. By bringing together features from multiple sources, feature store enabled many functionalities such as -

1. [Deal Scoring](https://knowledge.hubspot.com/records/use-deal-scores)
2. [Buyer Intent Signals](https://knowledge.hubspot.com/reports/use-intent-signals)
3. [Prospecting agent](https://knowledge.hubspot.com/prospecting/use-the-prospecting-agent)
4. Many internal HubSpot models like signup anomaly detection, fraud/abuse detection, lead scoring, etc.

These comprise ~20% of the total use cases built in the past year powered by the feature store.
