About Zepp Health
Zepp Health is a cloud-based health service provider with smart wearable technologies. Its vision is to create smart products that promote a healthy lifestyle.
Pre-computing metrics and serving them from Apache HBase allowed only point queries. With Bitmap data in StarRocks, users analyze metrics across combinations of dimensions and time periods without pre-computing redundant metrics.
Key takeaway · Pre-computation limits flexibility
This article was written by the Zepp Health engineering team.
At Zepp Health, a huge amount of event tracking data is generated every day. When analyzing it, users often query metrics in a single dimension or a combination of dimensions. A dimension can be time, event name, city or device attribute. A metric can be the number of users or the number of times an event is tracked.
The challenge
Our previous Apache HBase computation and analytics system was inefficient and inflexible for queries against data at this scale. To analyze event tracking data, engineers collected metric statistics, used Apache Spark or Apache Hive to pre-calculate metrics such as page views (PVs) and unique views (UVs), and then wrote the pre-calculated data to Apache HBase for point queries.
This solution had three drawbacks:
- Data in Apache HBase is stored as key-value pairs, which can only be used for point queries. Complex analytics is impossible.
- Bitmap-related technologies cannot be used. Engineers have to compute the required metrics in advance, which is not flexible and not suitable for set operations.
- The data processing chain is long and hard to maintain, and model abstraction is not supported. All of this complicates business upgrades.
Building an event tracking analytics platform on an efficient OLAP engine had become an important part of business development.
Evaluating OLAP engines
We were looking for a high-performance, simple and easy-to-maintain database to replace the existing Apache Spark + Apache HBase architecture. We also wanted to go beyond the point queries of Apache HBase and use real-time multi-table joins to support more business use cases.
We compared Apache Impala, Apache Druid, ClickHouse and StarRocks. After a thorough comparison, StarRocks stood out for its read/write performance, and its active community offered a good development and user experience. We chose StarRocks as the OLAP engine to replace Apache HBase.
| Product | Advantages | Disadvantages | O&M cost | Performance |
|---|---|---|---|---|
| ClickHouse | High single-table query performance; support for various database models; complete ecosystem | No support for high concurrency; low stability, prone to out of memory (OOM); complex scaling, high O&M cost | High | High |
| StarRocks | Extraordinary single-table query performance; powerful real-time multi-table joins; support for high concurrency | Complex data import ecosystem; limited batch processing capability in current version | Low | High |
| Apache Impala | Friendly to updates; sound Apache Hadoop ecosystem | Complex environment; low stability; poor import performance | Medium | Low |
| Apache Druid | Real-time data read/write; support for high concurrency | No support for standard SQL; no support for distributed joins | High | Medium |
The solution
Architecture
- Event tracking data flows from API Gateway to Apache Kafka, and is cleansed, filtered and transformed using Apache Hudi on Apache Flink. The OLAP preprocessing layer is built on this streaming data lake.
- The data warehouse detail (DWD) layer is built on Apache Hudi's Upsert or Append write mode. Which mode is used depends on data characteristics, desired write performance and cost.
- Data is processed offline on a schedule and then moved to the data warehouse summary (DWS) layer.
- DWS data is imported to StarRocks offline on a schedule, for cost optimization.
- Data in StarRocks is queried through a unified query and analytics platform.
Data processing
- Transform the original data and insert it into Apache Hudi in Upsert or Append mode based on data characteristics. Business data that is not sensitive to duplication is written directly to Apache Hudi in Append mode.
- Use Broker Load to write data into the aggregate key model with Bitmap fields to generate Bitmap data.
- Perform customized set operations on the Bitmap data based on business requirements. Metrics such as PV and UV are generated in this step.
- Users run self-service queries for metrics on the query and analytics platform.
While building the platform, we contributed three improvements that were merged into the main branch of StarRocks: fixes for bugs that occurred when creating materialized views, support for more object storage types during data import, and more parameter configurations for some data import scenarios.
The results
Performance
StarRocks' monitoring platform shows that the average query latency is about 100 ms and the P99 latency is about 250 ms, which fits our business needs well.
Outcomes
- Efficient: The system responds quickly to users' queries. The latency of large queries is reduced from minutes to seconds.
- Flexible: Users can analyze metrics across combinations of dimensions and time periods. There is no need to pre-compute redundant metrics.
- Low storage cost: StarRocks' storage structure reduces the cost of storing the same volume of data by 20%.
- Simple: StarRocks is easy to operate and maintain, which reduces maintenance labor cost compared to ClickHouse.
- Easy to use: Users can run self-service queries, which improves the data retrieval experience. The latency of point queries for some metrics has been reduced from minutes to seconds, and for some metrics to milliseconds.
What's next
- Today, to generate Bitmap data, engineers must incrementally import detailed data into the aggregate key model. Parquet files with Bitmap fields cannot be imported into StarRocks directly, which is inefficient in heterogeneous storage scenarios. Zepp Health's big data team is working with the StarRocks community on a solution (see https://github.com/StarRocks/starrocks/issues/3279).
- Connect more workloads to the OLAP platform and use StarRocks in more smart wearable health scenarios.