On August 11, 2026, between 3:09 PM and 3:27 PM EDT (UTC-4), HubSpot customers experienced an outage of the CRM Record Page. During this time, a large portion of CRM record page loads failed. No data was lost during this incident, and services outside of the CRM record page were unaffected. We have completed a thorough analysis of this incident, and below is a detailed account of what happened, why it occurred, and the steps we are taking to prevent similar issues in the future.
#What Happened
On Tuesday, August 11th, our engineering team modified metadata on a core CRM backend service, updating team ownership to reflect organizational changes. This update caused any newly launched service instance to lack access to a distributed cache. When two new instances launched in response to scaling signals, they failed cache record lookups and queried the underlying database instead. This created pressure on the database’s resources and caused minor elevated latency and timeouts for the requests they served.
Our reliability automation alerted the engineering team of traffic degradation due to the two instances that had been launched since the code change. In an effort to minimize customer impact, the team followed our incident best practices and rolled back to a previous version of the service from before the degradation started. However, this permissioning issue was not tied to the version of the service, and rather was managed by a separate permissioning system. Rolling back globally relaunched all instances, causing all service instances to lose access to the cache, exhausting database connection pools and triggering this outage.
#Customer Impact
Between approximately 3:09 PM and 3:27 PM EDT, HubSpot Customers across all regions attempting to view CRM record pages (contact, companies, deal, tickets) experienced consistent failures. Non-CRM record pages were not affected. Preceding this outage, some customers in Europe and the western North America regions may have experienced minor degradation in the form of slower load time and timeouts.
#Technical Impact Details
When the metadata update took effect on new service instances, it created a chain of resource constraints:
- Cache Unavailability: The internal permissioning system denied new instances access to the distributed cache layer due to the updated team ownership metadata.
- Database Resource Exhaustion: Without the cache, the backend instances routed all data retrieval requests directly to the underlying database. The database connection pools were under pressure due to the volume of direct queries, leading to elevated latency and timeouts.
- Rollback Amplification: The standard rollback procedure, which normally mitigates code-level defects by reverting to a known good state, inadvertently amplified this configuration issue. By restarting all instances globally, it applied the problematic permissions universally, turning a minor, localized degradation into a systemic outage.
#Timeline of Events (August 11, 2026)
- 1:21 PM EDT: Minor degradation starts when team ownership metadata is changed.
- 2:10 PM EDT: Issue is detected via reliability automation alerts. The engineering team begins investigating.
- 2:41 PM EDT: Engineering team identifies permissions as the root cause issue.
- 3:07 PM EDT: Engineering team tries to mitigate the issue by initiating a rollback of the service. This action restarts all service instances globally, causing widespread failures of CRM record page loads. Minor degradation increases to widespread outage.
- 3:16 PM EDT: An internal incident is automatically declared and escalated.
- 3:24 PM EDT: Engineering team reverts team ownership metadata to permanently fix the issue. Service instances are restarted to safely restore cache access.
- 3:27 PM EDT: Services return to normal. The internal incident is closed.
#What We're Doing to Improve
- Decoupling Permissions: Our permissioning system currently allows both service and team-based permission. When our team changed the team ownership metadata for the service, this inadvertently removed permissions that were associated with the service via team-based permission. We will audit and ensure our critical services, including those directly involved in this incident, move towards service-based permissions.
- Configuration Deployment Consistency: We will update our systems to ensure that authorization configuration changes follow the same staged rollout process as code deployments, preventing them from being applied globally all at once.
- Validation Safeguards: We are implementing automated checks that verify permissions remain intact before any organizational metadata changes can be made.
- Enhanced Alerting: We will add targeted alerts to identify spikes in authorization check failures or caching layer fall-throughs to accelerate detection.
#Our Commitment to Trust
We know that quick, reliable access to CRM records is foundational to how your teams use HubSpot. We sincerely apologize for the disruption this incident caused to your day. This event highlighted opportunities to improve how we manage internal permissions and validate configuration changes. We are committed to being transparent with our findings and applying these learnings to improve the resilience and reliability of the HubSpot platform.

