Originally published at: Grab Tech Blog

About Grab

Grab is a Southeast Asian superapp for deliveries, mobility and digital financial services, serving more than 800 cities in eight countries.

InfluxDB handled time-series numbers but struggled with string metadata, joins and pre-built summaries. Moving to StarRocks put metrics, metadata and job history into one relational store.

Key takeaway · Observability data is more than time series

The original post was written by Huong Vuong and Hai Nam Cao at Grab.

To understand its Apache Spark workloads, Grab built Iris, an in-house tool that collects metrics and metadata for each Spark job and reports resource usage, performance and query patterns. Where most monitoring stops at the EC2 instance, Iris works at the Spark application level and exposes results as tables, so users do not have to parse Spark event log JSON.

The challenge

The first version of Iris ran on a Telegraf, InfluxDB and Grafana (TIG) stack, with Superset for historical views. As usage grew, three problems became clear:

Two places to look. Real-time data lived in Grafana and historical data in Superset. The Grafana dashboards were hard for non-technical users, and the lack of fine-grained permissions held back adoption.
A heavy offline pipeline. Getting data into the lake for offline analysis took several hops and complex transformations.
Split data management. Real-time data sat in InfluxDB while offline data sat in the data lake, and string-type metadata was awkward to handle.

The team also found specific limits in InfluxDB:

Queries are written in Flux rather than full SQL.
Data from Kafka needs an external agent (Telegraf) before it reaches the database.
Pre-aggregation works only on time windows and indexed columns, never on strings.
It is built for numeric time series, is slow on strings and joins, and has no materialized views for pre-built job-run summaries.
There is no official way to back up or stream data to the data lake.
Indexing high-cardinality identifiers such as app_id creates very large indexes and slow queries.

Reviewing a job's last 30 days of runs meant querying the data lake with Spark and Presto, which the team later found slower than querying StarRocks directly. Grab wanted one web application for Spark monitoring, backed by a database that could serve real-time and historical queries. It chose StarRocks.

The solution

New architecture

Iris architecture: Spark jobs send metrics and metadata through a Metrics Collector to a Kafka queue; StarRocks ingests them into Iris Tables and aggregates them into Iris Materialized Views, which the Iris WebUI and Dashboard query; data is backed up to Iris Lake Tables for offline use — New Iris architecture with StarRocks

A metrics collector in each Spark cluster publishes job metrics and metadata to Kafka. From there:

StarRocks replaces InfluxDB for real-time and historical data and reads directly from Kafka, so Telegraf is gone.
A custom web application, the Iris UI, takes over from the Grafana dashboards.
Superset stays, now connected straight to StarRocks with the same real-time data as the Iris UI.
Scheduled backups from StarRocks to Amazon S3 replace the earlier data lake pipelines.

Data model and ingestion

Iris covers both shared ad-hoc clusters and clusters created per job, using three tables: cluster_worker_metadata_raw, cluster_worker_metrics_raw and cluster_spark_metrics_raw. worker_uuid links metadata to worker metrics, which arrive every 5 seconds, and app_id links metadata to Spark event metrics, which arrive as events occur. Queries can target a worker, a job or a cluster, and history shows auto-scaling behavior, peak worker counts and memory use over time.

All three are Duplicate Key tables, range-partitioned on report_date, hash-distributed on report_date and platform, with three replicas for fault tolerance and parallel reads. Routine Load jobs pull JSON messages from Kafka into them. Several load tasks read different topic partitions in parallel, so new metrics show up in Iris tables within seconds.

Materialized views and retention

Asynchronous materialized views join metadata, worker metrics and Spark metrics into one summary row per job run, so the job-run summary screen and SQL users skip the joins at query time. Views refresh when base tables change or on an interval, such as every minute. Two settings keep them cheap:

PROPERTIES (
  "partition_ttl" = "33 DAY",
  "auto_refresh_partitions_limit" = "3"
)

The TTL caps each view at 33 days of history, and the refresh limit recomputes only the latest partitions, where new data lands.

Base tables use dynamic partitioning over the same 33-day window, so partitions are created and dropped automatically and date pruning keeps scans small. Once a day, a cron job copies data past 30 days to Amazon S3 as Parquet with INSERT INTO FILES, and Spark maps it into Delta Lake tables, keeping older history queryable without loading the observability cluster.

Iris web application

A Go backend queries the StarRocks tables and materialized views and handles authentication and authorization. The frontend shows job runs, status, metadata, driver logs, the Spark UI, and resource usage and cost.

Iris job overview screen for one Spark job with total runs, platform, average duration, status, and 7-day charts of runs, duration and costs — Job overview screen in the Iris UI

StarRocksApache KafkaApache SparkAmazon S3Delta LakeSupersetGo

The results

A second materialized view, refreshed daily, summarizes each job's last 30 days of runs with p95 values for instances, job and CPU hours, utilization, data read and written, and spill. A recommendation API builds on these trends: when a job's resource utilization stays below 25%, Iris suggests halving the worker size. Users can also get recommendations in Slack through SpellVault, Grab's GenAI platform.

Iris job page with a recommendation banner suggesting a downgrade of the worker node size from 4 CPUs to 2 CPUs — Recommendation for a job with low resource utilization

The team summarized what it learned:

One store for real-time and historical data improved query performance and simplified the architecture.
Materialized views for pre-aggregation shortened response times for common UI actions.
Dynamic partitioning keeps performance steady as data grows and handles retention automatically.
Direct Kafka ingestion cut latency and complexity from the pipeline.
A relational model allows more complex queries and simpler metadata handling than InfluxDB's time-series model.

With job metadata and metrics together, Grab can run showback and chargeback per job. Iris now lives inside One DE, Grab's data engineering app. Real-time CPU and memory charts are moving from Grafana to the Iris UI, after which the Grafana dashboard will be retired; Superset stays for platform metrics and BI.

What's next

Grab plans to extend recommendations to bottlenecks and Spark configuration changes, add deeper analytics, connect Iris to more internal tools, and explore machine learning for predicting Spark performance.