Based on Xiaobing Xia's session at NextQuery 2026, PhoenixAI's Data + AI Infrastructure Summit. Xia is a Member of Technical Staff at Microsoft AI. Watch the session
About Microsoft AI
Microsoft AI (MAI) runs a reinforcement learning (RL) training platform where researchers launch more than 10,000 experiments every week. In each experiment the model gets a problem and responds, a grader scores the attempt, and a single job repeats that loop millions of times.
Microsoft AI researchers do not wait for a run to finish. They watch trajectories, grades and performance while the job is still going, and use what they see to decide what to try next. The sooner the data shows up, the sooner they can change course.
The RL training platform takes in 40 TB of compressed Parquet a day, and the raw data is many times larger. Volume has more than doubled month over month for the past few months, so the team plans capacity three to six months ahead.

The challenge
Up to seven hours before new data was queryable
In Microsoft AI's original setup for RL training analytics, RL jobs sent metric events to Kafka, and streaming ingestion loaded them into a proprietary data warehouse behind an analytics UI. The raw tables landed with duplicates and without aggregation, so they were not ready to query.
Periodic materialized view refreshes in that warehouse handled primary-key deduplication, pre-aggregation and pre-joins over more than 1 PB of data. New data was not visible until the next refresh finished. As volume grew, that lag reached as much as 7 hours, and researchers waited that long to see how their RL training runs were going.
Fresher data and faster queries cost more
On the proprietary warehouse, which held more than 1 PB of RL training data, fresher data meant more frequent materialized view refreshes, and each refresh was expensive. Complex queries took more than 20 minutes, and making them faster meant a bigger cluster. Security rules kept outside compute away from Microsoft AI's internal storage, so the data was shipped across the continent to be processed and queried.
With volume more than doubling every month on top of 40 TB of compressed data a day, keeping the experience good for researchers would have required a much larger budget. Microsoft AI decided to build a new stack for RL training analytics.
The solution
An open lakehouse with StarRocks on top
Microsoft AI's data platform team collected requirements from researchers and from its own experience running the platform. Data would live in an open lake format in Microsoft AI's own Azure Blob Storage, and a query engine would read it there.
That engine needed MPP distributed execution, a vectorized engine, and native support for external lake tables, materialized views and a data cache. StarRocks met those requirements. Apache Paimon became the table format on Azure Blob Storage, and Apache Flink the ingestion layer, with continuous writes, primary-key upserts and exactly-once delivery that can be replayed from Kafka.
Three goals drove the choice: low maintenance, fast queries on the lake, and low infrastructure cost as data grows. With four engineers, the team could not take on a stack that needed constant care.

For StarRocks to query the lake as is, the data had to land clean. Microsoft AI moved deduplication, pre-aggregation, pre-joins, compaction and index building out of batch jobs and into the Flink ingestion job, where they run continuously.
Paimon tables use a primary key with the deduplicate merge engine, so duplicates are dropped as Flink writes and compacts. Pre-aggregations declared in Flink SQL become materialized tables, and bloom filter and bitmap indexes are built during ingestion. Compaction runs inline in the same job; only a few very large tables get a dedicated offline compaction. Data lands in Paimon deduplicated, aggregated, compacted and indexed, and is queryable in StarRocks 2 to 8 minutes after it is produced.

Three layers of acceleration in StarRocks
A query on a lake has two slow steps: planning, which works out which files hold the data, and reading those files from object storage. StarRocks has a built-in feature for each, and materialized views for the slowest query patterns. Microsoft AI counts these three features as a main reason most researcher queries are fast.
- Metadata cache. Frontend nodes cache Paimon metadata and refresh it periodically, so planning skips the step of collecting file information.
- Data cache. When a query reads a range of a Parquet file, backend nodes cache those blocks. The team warms the cache on a schedule through StarRocks data cache management, for example reloading the last few days of selected tables every 12 minutes. Researchers tend to query similar ranges, so most of their queries read data that is already cached.
- Materialized views. When a query pattern stays slow, the team builds a materialized view over the last 7 days, the range most researchers look at, with its own partition and sort keys. The last 7 days are kept hot in StarRocks internal tables, and older data stays cold in Paimon.
Microsoft AI also built a query insights page that lists every StarRocks query, whether it came from a person or an agent. Opening a slow one shows its plan and profile, and the root cause points to the fix: warm the data cache for that table, adjust the metadata cache, or add a materialized view for that pattern.
One StarRocks cluster, run by four engineers
A StarRocks cluster at Microsoft AI serves researchers' ad hoc queries, a growing number of queries from AI agents, and the team's own cache warm-ups and materialized view refreshes. One expensive query should not be able to bring down the cluster for everyone else.
StarRocks resource groups prevent that. Each group of users has a quota, with a maximum number of CPUs per node and a share of memory, enforced on every query from that group's accounts.
Running Flink, Paimon and StarRocks creates routine work that grows with every new dataset and table. Microsoft AI's four engineers built an internal control plane that turns each task into one click. On the StarRocks side, that covers provisioning and resizing clusters, version upgrades, cache warm-ups, materialized views, resource groups and server settings, with each cluster's frontend and backend nodes and utilization on one page.
The same control plane restarts a Flink job from a checkpoint or replays Kafka from a point in time, such as nine hours earlier, with new CPU and memory settings. With these options, the team expects to recover from most incidents in about five minutes.
The results
About two months after Microsoft AI began rebuilding its RL training data warehouse:
- Data freshness: end-to-end lag fell from as much as 7 hours to 2 to 8 minutes, and stays under 8 minutes most of the time.
- Query speed: p95 latency on StarRocks is under 4 seconds. Complex queries on the previous warehouse took more than 20 minutes.
- Maintenance: four engineers spend less than 10% of their time keeping the platform running end to end.
- Cost: infrastructure cost is roughly 50× lower than the previous solution. Running in Microsoft AI's own data center is the main reason for the drop.
Microsoft AI researchers now see results from a run within minutes instead of hours. Alibaba took a similar route for flash sale analytics, querying Paimon tables with StarRocks; read how Alibaba built it.
What's next
More and more of the queries on Microsoft AI's RL training platform come from AI agents, and Microsoft AI's next steps follow that shift:
- Agent-facing queries. A semantic layer will describe the storage layout and the caches and materialized tables available in Paimon and StarRocks, and expand automatically, so agents can write efficient queries.
- AI-native maintenance. Agents will read key metrics from the control plane and decide when to resize a StarRocks cluster, reschedule a Flink job or trigger a full compaction.
- More processing on ingest. No-code and low-code Flink processing, such as GPU inference, will make data more useful by the time it lands.
- Semantic and image search. Agents will combine image search, semantic search and analytics in multi-step queries on one platform. Tencent runs vector search for RAG and image management on StarRocks.