HubSpot Engineering

Latest posts

Writing lately

Kafka Night: Confluent and HubSpot on Real-Time Data Processing (Video)

Confluent diagram with Kafka at the centre, fed by databases, logs and sensors and feeding monitoring, analytics and Hadoop.

Neha Narkhede is the co-founder and head of engineering at stream data platform company Confluent, and one of the original authors of Apache Kafka. Earlier this week, Neha joined us at HubSpot for Boston Kafka Night where she talked to us about Kafka's architecture, discussed how companies are leveraging it for advanced real-time data processing, and discovered Samza on the Rocks.

Neha walked through the evolution of LinkedIn's data systems from an increasingly coupled architecture to separate systems linked with Kafka as a message bus. Similarly, Kafka is now an instrumental part of HubSpot's infrastructure. After Neha's presentation, two of our engineers, Ze'ev Klapow and Mike Axiak, spoke on how we've operationally integrated Kafka. Mike spoke about how HubSpot has dealt with availability and failover, and Ze'ev covered the consumer metrics we've developed to gauge our system's performance and health.

watch the video

Save the date: The Apache Kafka Summit is in San Francisco on April 26th, 2016.

Newsletter

For Agents

View Markdown

Share

More posts

From Data Silos to Feature Serving: How We Unified Fragmented Data for Training, Inference, and Reuse

HubSpot's feature store gives ML teams one place to find and serve features, whether they're ingested into it or left where the data already lives: a CRM property, a team's own service, or a team's Iceberg table.

Hedging Erasure-Coded Reads in HDFS

Erasure coding halves the disk our HBase data needs, but every byte lives on one DataNode. Hedging slow reads by rebuilding the slow cell from its stripe's parity cut slow-read host-minutes on our case-study cluster by 98%.

Building the AI Retrieval Infrastructure Behind 20 Billion+ Vectors at HubSpot

Discover how HubSpot built a scalable AI retrieval infrastructure, managing over 20 billion vectors with Qdrant, to enhance semantic search and support diverse applications.