Originally published at: StarRocks Blog
About Haezoom
Haezoom runs virtual power plant (VPP) operations in Korea, with a focus on solar energy. Its platform serves 2.3 million users and more than 6,000 power stations, and the data engineering team manages the solar data behind generation forecasts and energy market optimization.
Some of Haezoom's sources send data only once an hour, yet Druid's ingestion tasks held compute all day. StarRocks Routine Load reads data as it arrives instead.
Key takeaway · Match ingestion to how data arrives
The original post was written by Kim Byung Ju of CloudShift.
CloudShift is a Korean consulting firm founded in 2020. For five years it has delivered data platform projects across many industries, both in the cloud and on premises. Haezoom worked with CloudShift on the migration described here.
The challenge
Haezoom used Apache Druid as its main time-series database to process large volumes of IoT data and serve real-time analytics. Druid gave the team fast queries and supported real-time bidding. As the platform grew, though, its limits became harder to work around.
The original pipeline ran on AWS across two regions. Weather data from NOAA and KMA (the Korea Meteorological Administration) arrived over FTP and was pulled in by AWS Lambda functions, with raw files kept in Amazon S3. Readings from RTUs and meters, plus data from the KMA API Hub, were collected by FastAPI scrapers on Amazon EKS. The data flowed into Confluent Cloud and was stream-ingested into a Druid cluster on EKS, with Amazon RDS holding metadata. A FastAPI platform API served the supply and demand VPP apps, and data scientists worked with the data through Amazon SageMaker.
Five problems stood out:
- Wasted compute on ingestion. Not every source suited a streaming pipeline. Some sources updated only hourly, but Druid's peon processes kept using compute around the clock, whatever the actual ingestion rate.
- High infrastructure cost. Running Druid on EKS with high availability meant standby master pods, a ZooKeeper ensemble and deep storage. Keeping all of these reliable added a large and growing cost.
- Scaling for the worst query. A few analytical queries needed far more resources than the rest. The team had to scale the whole cluster for those peaks, even though most queries used very little.
- Hard to integrate other sources. Haezoom wanted one data platform over S3, PostgreSQL and Kafka. Druid's architecture made that difficult.
- Limited SQL for the AI team. Druid queries work best with few JOIN and WITH clauses. To avoid them, data had to be fully preprocessed before ingestion, which cost compute and development time as the AI team's needs grew.
The solution
Evaluating StarRocks against Druid
Haezoom compared StarRocks with Druid feature by feature, and rated how worthwhile it would be to move each workload:
| Druid workload | Migration fit | With StarRocks |
|---|---|---|
| Kafka real-time ingestion | Low, if real-time ingestion is the only need | Cheaper, periodic ingestion with Routine Load |
| Kinesis real-time ingestion | Low: StarRocks has no native Kinesis ingestion | Switch to Flink or StarRocks Pipe |
| Batch loading | High: many batch ingestion methods | Several loading strategies, including INSERT |
| Real-time queries on denormalized data | Low where real-time requirements are strict, although StarRocks has every function needed | Broader ANSI SQL support for complex queries through the SQL API |
| Join queries and long-term report queries | High: Druid's newer MSQ engine handles them, but inefficiently because it runs on MiddleManagers | Join queries run reliably |
| Data lake export | High: Druid exports only CSV, through MSQ | Multiple export formats; External Catalog |
| Data lake federation | High: Druid's early support has architectural inefficiencies | Multiple federation environments through External Catalog |
| BI connection | High: Druid uses the Avatica JDBC driver | Works with the MySQL connector driver |
| Real-time BI (Imply Pivot) | Needs deeper analysis; consider alternatives if it is business-critical | Metabase works, but some features need a paid license |
The comparison showed that StarRocks addressed the cost, scaling and query flexibility problems directly.
Migration plan
Most of Haezoom's data already lived in Confluent (Kafka) and S3, so the team loaded StarRocks from those sources rather than from Druid. If data ever has to come out of Druid, the plan is to export it to S3 with MSQ and load it from there. CloudShift and Haezoom worked through five steps:
- Deploy StarRocks on EKS. Size the cluster and set parameters from current workload patterns and expected growth; choose node types, storage classes and resources for FE and BE nodes, based on Druid's cluster specs, data volumes and query patterns.
- Ingest from Kafka. Use Routine Load jobs and Stream Load, tuning parallelism, batch size and commit intervals, with a full mapping of topics to tables, partitioning and retention.
- Check SQL compatibility. Run the existing Druid SQL against StarRocks, benchmark execution time, resource use and concurrency, and document any syntax changes in migration scripts.
- Connect BI and Python. Move Metabase dashboards over the MySQL protocol, connect data science workflows through SQLAlchemy and PyMySQL, and verify every visualization.
- Run in parallel, then retire Druid. Run both systems side by side for up to two weeks with validation checks, then cut over in phases and shut down Druid.
The results
Compared with Druid, the StarRocks deployment improved Haezoom's platform in five ways:
- Resource efficiency: Routine Load consumes data on demand, as it actually arrives, so ingestion no longer holds compute continuously the way Druid's peon processes did.
- 1.74x higher throughput: The new architecture handles 1.74 times as many requests as the Druid setup, which supports Haezoom's growing base of 2.3 million users.
- 44.3% lower average response time: Average query response time fell by 44.3%, giving the solar energy forecasting models faster answers.
- Up to 4x faster complex queries: Specific complex analytical queries that used to bottleneck the system ran up to 4x faster, so the AI team can run its models without the preprocessing overhead.
- 30% lower infrastructure cost: Better resource use and the removal of unneeded standby components cut overall data infrastructure costs by 30%.
What's next
- Performance tuning: Refine query plans and resource allocation for Haezoom's time-series solar data to raise throughput and lower latency.
- Data lake integration: Build a data lake on Apache Iceberg, using External Catalogs to bring together S3, PostgreSQL and streaming sources.
- Self-service analytics: Give data analysts and scientists an environment where they can explore and query data without waiting on engineering.