Originally presented at StarRocks Summit 2025: Real-World Case Studies at Alibaba Group: How StarRocks+Apache Paimon Power the Streaming Lakehouse
About Alibaba Group
Alibaba Group operates the e-commerce platform Taobao. Taobao Flash Sales is a promotion program that Taobao runs with the food delivery platform Ele.me as instant retail grows. Brands pool resources to give shoppers deep discounts or exclusive offers in short, limited-quantity sales, and Ele.me delivers the orders.
Keep the DWD-to-ADS processing in Paimon so data stays in the lake, and use StarRocks materialized views only as the last acceleration step at the ADS layer.
Key takeaway · Where materialized views fit
The talk was presented by Zhen Fan, Staff Engineer at Alibaba Cloud, and Kai Kang, StarRocks Tech Lead at Alibaba Group.
During a sale, operators watch payments and orders and adjust the campaign as it runs, which needs data that is minutes old.
The challenge
Ele.me's data infrastructure used T+1 offline processing, so analysis of a sale arrived the next day and business owners could not act on it within minutes. The team needed to ingest, clean, analyze and visualize very large data flows within minutes.
The common setup of the past decade, a Lambda architecture with a Hadoop offline warehouse for T+1 data and a Flink real-time warehouse for second-level data, means two code bases, no transactions (small updates rewrite whole partitions with INSERT OVERWRITE), and a central Hive Metastore that limits scale and holds no file-level statistics for engines like StarRocks to optimize with. Lakehouse formats solve the transaction and metadata issues, but in the speakers' view Apache Iceberg, Delta Lake and Apache Hudi are built for batch work and cannot keep up with minute-interval updates at scale.
Why Apache Paimon
Apache Paimon puts an LSM tree on cloud storage, which gives low-latency ingestion, efficient upserts and changelogs that Flink can read as a stream. It targets minutes-level freshness, integrates natively with Flink, works with Spark, StarRocks and Presto, and keeps ACID transactions, time travel and schema evolution.
The solution
In the reference pipeline, Flink handles streaming and Spark handles batch ETL through the ODS, DWD and DWS layers, all stored as Paimon tables. Final results go into StarRocks at the ADS layer, and StarRocks can also query the Paimon tables at any layer directly.
For Taobao Flash Sales, Flink writes into Paimon, whose LSM-tree storage takes real-time writes and handles deduplication and aggregation automatically. StarRocks queries the Paimon tables in place through an external catalog, which removes an ETL hop and its delay. The team also tuned StarRocks for the lakehouse, for example with native support for Paimon deletion vectors, which improved query response times.
Materialized views on Paimon
StarRocks on Paimon covered the freshness needs at first. As the business grew very fast, peak-hour complex queries went from seconds to minutes and dashboard refreshes fell far behind.
Optimization 1: materialized views for dashboards. As more users came on, stability suffered under concurrent access and queries slowed. The team put StarRocks materialized views, refreshed every ten minutes, on the Paimon tables behind the BI dashboards, which takes the load of large queries off the cluster.
Optimization 2: long-range comparisons. After launch, operators wanted daily and weekly comparisons, which need more history than the real-time tables keep. An offline INSERT task loads the previous day's data into a StarRocks internal table. A view combines it with the real-time materialized view through UNION, so both are aggregated together at the finest grain.
Optimization 3: partitioned materialized views. Using partitioned views with partition alignment across multiple Paimon base tables, the team replaced the first view with one that keeps long history through PARTITION BY on the base table's partition field, which is much easier to run. One limit: if a lake table's lifecycle is shortened, the view drops those partitions at its next refresh, so data that must outlive the base table still needs another approach.
The team's conclusion: materialized views work best as the final accelerator at the ADS layer, with DWD-to-ADS processing kept in Paimon.
RoaringBitmap for exact real-time deduplication
Exact distinct counts are costly when rows must be shuffled between nodes several times. With bitmaps, each dimension value holds one bitmap of user IDs, deduplication is a bitwise OR, and UV is the count of set bits, using far less storage than raw IDs. Plain bitmaps waste space on sparse dimensions, so the team stores RoaringBitmap data in Paimon and queries it with StarRocks bitmap functions.
Scenario 1: real-time UV over many traffic-log dimensions
Later, the business wanted flexible real-time metrics across channels, including non-Flash Sale data. Views directly on the DWD lake table could not keep up, and large views competing for resources refreshed late. The new pipeline:
- An auto-incrementing sequence maps each UID to an integer in real time.
- A Flink UDAF performs bitmap aggregation and union, storing results serialized.
- City, user segmentation and other warehouse tables are brought in.
- Message-queue data lands in the lake; Flink SQL stream-reads the Paimon detail data and runs lookup joins against the UID mapping and dimension tables.
- Mapped UIDs become bitmaps in downstream lake tables, per minute and then per hour.
- StarRocks computes real-time deduplicated counts with its bitmap union count function.
Scenario 2: bitmaps across domains
Segmentation criteria then changed often, and refreshing dimension tables each time caused large fluctuations, so the team processed the crowd dimension separately. They also wanted one real-time bitmap asset for all traffic dimensions and visit-to-purchase rates that use order data. The revised model:
- A middle DWS layer keeps the finest-grained aggregate state and emits bitmaps for common and single dimensions.
- Only the UID mapping table stays in the stream; other joins move to the application layer.
- Batch jobs build crowd-dimension bitmap lake tables, and their intersections serve crowd drill-downs.
- The UDAF handles retraction streams to support updates, and order-domain lake tables are converted to bitmaps too.
The results
- Freshness: analytics moved from next-day offline results to a lakehouse built for minutes-level freshness, which StarRocks queries in place.
- Dashboards at peak: materialized views refreshed every ten minutes serve BI and keep large queries off the cluster.
- Longer history: daily and weekly comparisons run on a union of real-time and historical data, later simplified with partitioned views.
- Deduplication: querying RoaringBitmap data in Paimon with StarRocks bitmap functions made UV and cross-domain metrics much fresher and faster, including crowd drill-downs, visit-to-purchase rates and long-period traffic metrics.