Originally published at: StarRocks blog
About DiDi
DiDi (Didi Chuxing Technology Co., Ltd.) is a ride-sharing company based in Beijing and one of the largest ride-hailing platforms in the world. It serves 550 million users in more than 400 cities, and riders completed 7.43 billion trips on the platform in 2017.
With Apache Druid, Presto and ClickHouse running side by side, users had to work out which engine fit each use case and why the SQL and delete behavior differed between them. Moving to StarRocks as the only OLAP engine removed that choice.
Key takeaway · One engine instead of three
The original post was published on the StarRocks blog; it does not name an author.
DiDi generates petabytes of data every day across a large number of systems and applications. Its IT team runs that infrastructure and builds new services on top of it, and analytics on this data supports decisions across the business.
The challenge
DiDi's OLAP stack grew one engine at a time. Raw data lived in Apache Hive. Apache Druid became the first OLAP layer in 2016, Apache Kylin was added in 2017 to speed up Druid queries, Presto took Kylin's place in 2018 for ad-hoc queries, and ClickHouse joined in 2020. Each step fixed one problem and introduced another.
By the time DiDi ran Presto, ClickHouse and Druid together, the problems were clear:
- Operational overhead. Keeping several OLAP systems running required specialized skills from the data engineering team, made troubleshooting expensive, and raised the risk of the same data diverging between systems.
- Confusion for users. People did not know which system to pick for a given business case. Queries had to be written differently on each one, and data that could be deleted in one system could not be deleted in another.
- Scale and real-time gaps. Data volume grew exponentially with the business, and the three engines did not scale with it, so query latency rose. DiDi also needed low-latency analytics on fresh data, including real-time streaming inserts, updates and deletes, which the existing setup could not provide.
- No path to open table formats. DiDi planned to move to the newer open table formats, which none of Presto, ClickHouse or Druid supported at the time.
The timeline above also records the trade-offs DiDi saw in each engine: Druid's queries were fast but complex and its data pipelines heavy; Kylin had limited ad-hoc support, was resource intensive and made data model management complex; Presto had stability issues; ClickHouse was the fastest of the group but weak at JOINs, without real-time updates, and costly to operate.
The solution
After evaluating its options, DiDi moved its OLAP workloads to StarRocks and, by 2022, made it the only engine in the OLAP environment, replacing Presto, Druid and ClickHouse. The reasons map to the problems above:
- One platform for real-time and historical data, so separate OLAP systems were no longer needed.
- Horizontal scaling, so capacity could grow with data volume and query load.
- Fast queries on both normalized and denormalized data, with real-time upserts, deletes and inserts and a simpler deployment model.
- Lakehouse support: StarRocks can run warehouse-style queries on Delta Lake, Apache Hive, Apache Hudi and Apache Iceberg.
Financial Data Portal
DiDi's Financial Data Portal is its internal platform for financial product data. It gives the finance business team the analysis it uses to keep improving operations. The OLAP engine behind it has to handle custom time dimensions and JOINs across datasets, answer within seconds, and take in about 1 billion records a day. StarRocks meets the seconds-level target using:
- Data partitioning and bucketing
- Prefix, ZoneMap, Bloom Filter and Bitmap indexes
- Materialized views
The results
Monitoring and alarm system
DiDi's monitoring and alarm system business unit ingests 450,000 entries per second and about 12 TB of data per day. After the move to StarRocks:
- Queries got faster. Query performance improved 4x, and P90 query time dropped from 500 ms to 150 ms.
- Data management got simpler. With one engine, there were fewer moving parts and more consistent data. Because StarRocks handles JOINs at scale, much of the pipeline work that moved and reshaped data was no longer needed.
- Infrastructure cost fell. The cluster went from more than 60 nodes to fewer than 10, stored data shrank by about 40%, and overall cost dropped by more than 80%.
Across DiDi
As of 2023, DiDi runs more than 30 StarRocks clusters holding over 300 TB of data. StarRocks serves several business lines, including ride-hailing, carpooling, two-wheelers, finance and energy.
What's next
DiDi keeps StarRocks as its core OLAP platform. Its plans are to bring real-time analytics to more parts of the business, to explore machine learning on top of its OLAP data, and to keep contributing to the StarRocks open-source project.