About Demandbase
Demandbase is the leading account-based go-to-market platform for B2B enterprises. The company helps businesses identify and target the right customers at the right time with the right message through unified intent data, AI-powered insights, and prescriptive actions.
Proof-of-concept tests showed significantly better query latency from StarRocks native tables than from querying Iceberg directly, so frequently accessed data is loaded into StarRocks and rarely used tables stay as external Iceberg references.
Key takeaway · Native tables for hot data, Iceberg for the rest
Thousands of companies rely on Demandbase to maximize revenue and consolidate their data and tech stacks into one platform. The platform combines and processes marketing data from numerous sources, handling both bulk imports and streaming events. Core to the product is processing data at scale and delivering flexible, fast reporting and insights. To support this, Demandbase needed a high-performance data infrastructure that could handle growing and complex workloads while maintaining near real-time responsiveness.
The challenge
Demandbase faced a critical architectural decision as its data volumes scaled into the petabyte range. The company needed a solution that could handle massive data growth and support both batch and streaming workloads without compromising query performance.
The platform required several capabilities that traditional data warehouses struggled to deliver together:
- Change data capture from PostgreSQL needed to flow into both a data lake and a query engine for immediate customer access. Enrichment processes running nightly had to transform raw lake data into queryable warehouse tables without creating operational bottlenecks.
- Cost became a major concern as data volumes grew. Storage expenses threatened to spiral as the platform stored more customer and third-party data. The team needed cost-effective, scalable storage at petabyte scale.
- Disaster recovery and upgrade strategies also presented challenges. Losing a production cluster couldn't mean hours of downtime while teams scrambled to restore data. Upgrading to new database versions needed a strategy that allowed testing and validation before cutting over production traffic.
The company evaluated multiple architectures but struggled to find one that met these requirements. It needed to support concurrent writes from multiple systems, separate storage from compute for cost efficiency and flexibility, and maintain strong separation between batch processing and real-time analytics. Traditional data warehouses either couldn't scale cost-effectively or required complex workarounds to support streaming and batch workflows together.
The solution
After evaluating its architecture needs, Demandbase built a data lakehouse combining Apache Iceberg as the storage layer with StarRocks as the analytics engine.
Two layers
Apache Iceberg provides the storage foundation:
- Open, cost-effective, scalable storage backed by cloud providers
- Strong separation of storage and compute
- Time travel and rollbacks
- Comprehensive schema evolution, including partitioning
StarRocks serves as the low-latency query engine:
- Fast, real-time analytics for a responsive user experience
- High-performance join queries to meet SLAs and reduce denormalization
- Stream loading with upsert handling
- Query engine integration with Iceberg via external catalogs
The architecture connects the two through batch and streaming patterns.
Batch loads from Iceberg
For batch operations, StarRocks' INSERT OVERWRITE supports SELECT statements that let the Demandbase team query Iceberg directly. This allows partition-level updates from Iceberg on demand. StarRocks' built-in task queue manages these operations with configurable parallelization.
Streaming loads from Kafka
On the streaming side, Spark jobs read from and write to Kafka for StarRocks consumption. With StarRocks routine loads, users can pull from Kafka topics and update designated tables on the fly, while maintaining state for tracking and supporting full-table, partial, and conditional update patterns. The data can then be inserted into primary key tables to enable create, update, and delete operations.
Net-new records
For Demandbase, a critical use case was net-new records in the data pipeline. Using partial routine loads, StarRocks streams create and delete changes for columns filled on creation, providing immediate visibility. After nightly Iceberg enrichment completes, StarRocks loads the full enriched data via INSERT INTO, acting as upserts for existing records.
Hot data in StarRocks, the rest in Iceberg
Performance comparisons during proof-of-concept testing showed significantly better query latency from StarRocks native tables than from querying Iceberg directly via external catalogs. This justified loading frequently accessed data into StarRocks while keeping rarely used tables as external Iceberg references.
Disaster recovery and upgrades
Iceberg now acts as the source of truth, which enables enterprise disaster recovery strategies. Teams can stand up new StarRocks clusters with region failover and load directly from Iceberg. For upgrades, they use blue-green deployment: stand up a second cluster with the newer version, load it from Iceberg, mirror production traffic, then start routine loads with different consumer groups. Traffic then migrates via Route 53.
A load service on top of StarRocks
Demandbase also built a lightweight load service on top of StarRocks to streamline user requests and resource delegation. Internal teams submit requests specifying tables and partitions, and the service handles task submission, completion tracking, and retries using StarRocks' task management features. A similar service manages routine loads, automatically handling schema changes by stopping loads, validating upstream data, and creating updated routine loads.
The results
The StarRocks and Iceberg architecture delivered operational and cost benefits:
- Petabyte-scale data storage with cost-effective lakehouse backing
- Near real-time data freshness through partial routine loads
- Disaster recovery with Iceberg as the source of truth
- Upgrades via blue-green deployment loaded from Iceberg
- Simplified operations through the load service abstraction
- Performance on normalized schemas, avoiding expensive denormalization
- Fast analytics backing customer-facing reporting
The architecture proved particularly valuable for concurrent access patterns. Multiple systems write to Iceberg simultaneously while StarRocks queries the same data for analytics. Partitioned workloads during enrichment avoid conflicts, and teams report minimal issues with concurrent writes to the same Iceberg tables.
What's next
Demandbase continues to expand its StarRocks and Iceberg integration:
- Conditional stream loading support for predicate-based updates in routine loads
- Priority retry handling for failed load tasks, so retries don't queue behind new submissions
- Wider use of federated queries across environments
Connor Clark, Staff Software Engineer at Demandbase, presented this architecture at StarRocks Summit 2025, covering nightly INSERT OVERWRITE batch loads with StarRocks Submit Task, Spark-driven CDC streams, and the dual-loading strategy across Iceberg and StarRocks.