About Intuit
Intuit is a financial software company that serves more than 100 million consumers, small businesses, and self-employed customers through products like TurboTax (tax filing), QuickBooks (accounting and payroll), and Credit Karma (personal finance and credit management).
End-to-end data freshness of 4 seconds or less was a hard requirement. Beyond that threshold, model accuracy degrades and recommendations stop being relevant.
Key takeaway · Freshness is a model-accuracy requirement
To support these products, the Intuit Persistence Service (IPS) platform provides a unified data foundation for client developers. The IPS team abstracts away system-specific complexities, so developers can focus on building real-time ML models, identifying trends, and running what-if scenarios. Rather than each product team managing its own persistence layer, IPS delivers scalable, reliable data services that power mission-critical features.
During peak tax season, IPS processes more than 140 billion transactions, with bursts exceeding 100,000 transactions per second. As demand grew for OLAP and real-time clickstream use cases, the IPS team moved away from Apache Druid to an OLAP database that could deliver query performance with SLAs as low as four seconds.
The challenge
Real-time analytical processing at massive scale
As Intuit's products evolved to deliver AI-driven experiences, the IPS team faced real-time analytical processing at massive scale.
Intuit's customer support relies on ML models that analyze the customer's last 30 in-product clicks to power two experiences:
- Reactive models: Enable Intuit Assist (Intuit's generative AI chatbot) to provide contextual help when customers reach out.
- Proactive models: Detect when customers are struggling and surface information before they ask.
The requirement was non-negotiable: end-to-end data freshness of 4 seconds or less. Beyond that threshold, model accuracy degrades exponentially, leading to irrelevant recommendations and eroded customer trust.
Marketing teams also needed real-time aggregates across billions of records, with two types of complexity:
- Composition use cases: Determining total employee count across QuickBooks Online and QB Payroll required joining events from multiple real-time data streams.
- Stateful processing: Tracking metrics like active customer counts meant comparing current and previous state for each incoming event.
Where Apache Druid fell short
Intuit's existing Apache Druid deployment couldn't meet these demands:
- 40% more expensive for standard traffic, and 150% more during peak periods
- Poor multi-table join performance forced complex denormalization workarounds
- An operationally complex architecture drained engineering resources
Alternatives evaluated
The IPS team evaluated alternatives, but each had disqualifying limitations:
- ClickHouse: Multi-table joins weren't performant enough, upserts required workarounds, and horizontal scaling demanded custom operational tooling.
- DuckDB: Its single-node architecture was an operational non-starter.
- Pinot: It couldn't match the required query performance.
The IPS team needed a system that delivered sub-4-second end-to-end latency, handled real-time upserts across billions of records, sustained 100,000+ events per second during peaks, and performed multi-table joins efficiently, while keeping the "paved road" philosophy that made IPS successful for transactional workloads.
The solution
Why Intuit chose StarRocks
After evaluating ClickHouse, DuckDB, Pinot, and the existing Druid deployment, Intuit selected StarRocks based on these capabilities:
- Multi-table join performance at scale: Essential for real-time composition across product domains.
- Native real-time upsert support: Enabled stateful processing with primary key tables, without custom workarounds.
- Horizontal and vertical scaling: An operational requirement for handling unpredictable peak loads.
- Standard SQL interface: Reduced adoption friction for teams already proficient in SQL, with features like asynchronous materialized views and table partitioning and bucketing.
The IPS OLAP platform
The new IPS OLAP platform consists of three layers:
- Automated ingestion layer: Apache Flink pipelines, automatically provisioned from developer configurations, support real-time streaming from Kafka, historical backfills, and zero-downtime data replays straight into StarRocks, with ingestion optimization and query acceleration configurations built in.
- StarRocks shared-data architecture: Compute and storage separation backed by AWS S3, with multiple data warehouses for resource isolation and resource groups to prevent noisy-neighbor problems on multi-tenant clusters.
- Query REST layer with guard rails: A lightweight abstraction (10–20 ms overhead) in front of StarRocks that validates queries, enforces proper use of partitioning and indexes, provides attribute-based access control, and enables automatic query profiling.
IPS developers define entities in a simple schema language. The platform handles DDL generation, deployment, monitoring, and operational concerns, and secures the datastore with StarRocks features like query timeouts, a SQL blacklist for risky patterns, and resource groups for multi-tenant isolation.
The results
Beyond raw performance, the move from Druid to StarRocks changed how Intuit approaches real-time analytics:
- Eliminated complex workarounds: Replacing Druid with StarRocks removed the denormalization pipelines and workarounds that multi-table joins had required.
- Simplified operations: Consolidating OLAP infrastructure onto one platform reduced the operational burden. Instead of managing complex Druid clusters, teams work with a system that abstracts complexity while maintaining performance.
- Meeting mission-critical SLAs: The 4-second end-to-end requirement, previously unattainable, became consistently achievable, supporting both reactive customer support and proactive assistance without performance degradation.
Performance results:
- Real-time ML predictions: 2-second total latency (1-second data freshness + sub-500ms query latency at TP99) while handling 100K+ events per second.
- Marketing data aggregations:
| Workload | Before | With StarRocks |
|---|---|---|
| Small datasets (up to 2B records) | 35 minutes | 1 minute |
| Medium datasets (up to 5B records) | 2 hours | 2.5 minutes |
| Large datasets (>10B records) | 5 hours | 5 minutes |
| Composition queries (multi-domain joins) | NA | 30 seconds |
What's next
After this initial success with StarRocks as its OLAP engine, Intuit plans to expand its use:
- Read replicas to support development, exploration, and reporting without affecting production.
- Unified analytics by connecting directly to the data lake for real-time and historical insights.
- Stronger data consistency with multi-table transactions for reliable, atomic updates.
To hear the story from the Intuit team, watch Gaurav Doon and Daniel Russotto's keynote from StarRocks Summit 2025, covering the architecture behind Intuit's real-time OLAP platform and lessons in performance, scalability, and reliability.