Microsoft AIRL training analytics

Building a Real-Time Data Warehouse for Microsoft AI

Microsoft AI's researchers run more than 10,000 reinforcement learning experiments a week and need to see results while runs are still going. The team replaced a proprietary data warehouse with an open lakehouse queried by StarRocks. New data is now queryable in under 8 minutes instead of up to 7 hours, and p95 queries return in under 4 seconds.

<8min

end-to-end data lag

Down from up to 7 hours

<4s

p95 query latency

Complex queries took 20+ minutes before

~50×

lower infrastructure cost

From significant performance improvements and running in Microsoft AI's own data center

CompanyMicrosoft AI
Use caseRL training analytics
Previous solutionProprietary cloud data warehouse
CloudAzure

Based on Xiaobing Xia's session at NextQuery 2026, PhoenixAI's Data + AI Infrastructure Summit. Xia is a Member of Technical Staff at Microsoft AI. Watch the session

About Microsoft AI

Microsoft AI (MAI) runs a reinforcement learning (RL) training platform where researchers launch more than 10,000 experiments every week. In each experiment the model gets a problem and responds, a grader scores the attempt, and a single job repeats that loop millions of times.

Microsoft AI researchers do not wait for a run to finish. They watch trajectories, grades and performance while the job is still going, and use what they see to decide what to try next. The sooner the data shows up, the sooner they can change course.

The RL training platform takes in 40 TB of compressed Parquet a day, and the raw data is many times larger. Volume has more than doubled month over month for the past few months, so the team plans capacity three to six months ahead.

Diagram of the RL loop: pose a problem, the model responds, a grader scores each attempt, repeated millions of times per job, while researchers monitor trajectories, grades and performance. Below it, three figures: 10K+ experiments per week, 40 TB ingested per day compressed, and about 2x growth per month
The RL training loop and the data it produces. Source: Microsoft AI, NextQuery 2026

The challenge

Up to seven hours before new data was queryable

In Microsoft AI's original setup for RL training analytics, RL jobs sent metric events to Kafka, and streaming ingestion loaded them into a proprietary data warehouse behind an analytics UI. The raw tables landed with duplicates and without aggregation, so they were not ready to query.

Periodic materialized view refreshes in that warehouse handled primary-key deduplication, pre-aggregation and pre-joins over more than 1 PB of data. New data was not visible until the next refresh finished. As volume grew, that lag reached as much as 7 hours, and researchers waited that long to see how their RL training runs were going.

Fresher data and faster queries cost more

On the proprietary warehouse, which held more than 1 PB of RL training data, fresher data meant more frequent materialized view refreshes, and each refresh was expensive. Complex queries took more than 20 minutes, and making them faster meant a bigger cluster. Security rules kept outside compute away from Microsoft AI's internal storage, so the data was shipped across the continent to be processed and queried.

With volume more than doubling every month on top of 40 TB of compressed data a day, keeping the experience good for researchers would have required a much larger budget. Microsoft AI decided to build a new stack for RL training analytics.

The solution

An open lakehouse with StarRocks on top

Microsoft AI's data platform team collected requirements from researchers and from its own experience running the platform. Data would live in an open lake format in Microsoft AI's own Azure Blob Storage, and a query engine would read it there.

That engine needed MPP distributed execution, a vectorized engine, and native support for external lake tables, materialized views and a data cache. StarRocks met those requirements. Apache Paimon became the table format on Azure Blob Storage, and Apache Flink the ingestion layer, with continuous writes, primary-key upserts and exactly-once delivery that can be replayed from Kafka.

Three goals drove the choice: low maintenance, fast queries on the lake, and low infrastructure cost as data grows. With four engineers, the team could not take on a stack that needed constant care.

Reference architecture: RL jobs send metric events to Kafka; Apache Flink deduplicates by primary key, maintains materialized tables, builds bloom filter and bitmap indexes and compacts inline; Apache Paimon on Azure Blob Storage holds deduplicated, aggregated, compacted and indexed tables; StarRocks queries those tables with a metadata cache, a data cache, 7-day materialized views and per-workload resource groups, serving researchers and AI agents. An internal control plane manages all three layers
Microsoft AI's real-time data warehouse for RL training

For StarRocks to query the lake as is, the data had to land clean. Microsoft AI moved deduplication, pre-aggregation, pre-joins, compaction and index building out of batch jobs and into the Flink ingestion job, where they run continuously.

Paimon tables use a primary key with the deduplicate merge engine, so duplicates are dropped as Flink writes and compacts. Pre-aggregations declared in Flink SQL become materialized tables, and bloom filter and bitmap indexes are built during ingestion. Compaction runs inline in the same job; only a few very large tables get a dedicated offline compaction. Data lands in Paimon deduplicated, aggregated, compacted and indexed, and is queryable in StarRocks 2 to 8 minutes after it is produced.

Two pipelines compared. Top, the proprietary warehouse: Kafka, streaming ingest that copies raw rows, raw tables with duplicates, then periodic batch jobs for pre-aggregation, pre-join and primary-key dedup, about 7 hours to query. Bottom, Flink and Paimon: a continuous Flink ingestion job does pre-aggregation, primary-key dedup, transformation, compaction and index building before data lands in Paimon on Blob Storage, 2 to 8 minutes to query
Shaping data after landing versus during ingestion. Source: Microsoft AI, NextQuery 2026

Three layers of acceleration in StarRocks

A query on a lake has two slow steps: planning, which works out which files hold the data, and reading those files from object storage. StarRocks has a built-in feature for each, and materialized views for the slowest query patterns. Microsoft AI counts these three features as a main reason most researcher queries are fast.

  • Metadata cache. Frontend nodes cache Paimon metadata and refresh it periodically, so planning skips the step of collecting file information.
  • Data cache. When a query reads a range of a Parquet file, backend nodes cache those blocks. The team warms the cache on a schedule through StarRocks data cache management, for example reloading the last few days of selected tables every 12 minutes. Researchers tend to query similar ranges, so most of their queries read data that is already cached.
  • Materialized views. When a query pattern stays slow, the team builds a materialized view over the last 7 days, the range most researchers look at, with its own partition and sort keys. The last 7 days are kept hot in StarRocks internal tables, and older data stays cold in Paimon.

Microsoft AI also built a query insights page that lists every StarRocks query, whether it came from a person or an agent. Opening a slow one shows its plan and profile, and the root cause points to the fix: warm the data cache for that table, adjust the metadata cache, or add a materialized view for that pattern.

One StarRocks cluster, run by four engineers

A StarRocks cluster at Microsoft AI serves researchers' ad hoc queries, a growing number of queries from AI agents, and the team's own cache warm-ups and materialized view refreshes. One expensive query should not be able to bring down the cluster for everyone else.

StarRocks resource groups prevent that. Each group of users has a quota, with a maximum number of CPUs per node and a share of memory, enforced on every query from that group's accounts.

Running Flink, Paimon and StarRocks creates routine work that grows with every new dataset and table. Microsoft AI's four engineers built an internal control plane that turns each task into one click. On the StarRocks side, that covers provisioning and resizing clusters, version upgrades, cache warm-ups, materialized views, resource groups and server settings, with each cluster's frontend and backend nodes and utilization on one page.

The same control plane restarts a Flink job from a checkpoint or replays Kafka from a point in time, such as nine hours earlier, with new CPU and memory settings. With these options, the team expects to recover from most incidents in about five minutes.

StarRocksApache PaimonApache FlinkApache KafkaAzure Blob Storage

The results

About two months after Microsoft AI began rebuilding its RL training data warehouse:

  • Data freshness: end-to-end lag fell from as much as 7 hours to 2 to 8 minutes, and stays under 8 minutes most of the time.
  • Query speed: p95 latency on StarRocks is under 4 seconds. Complex queries on the previous warehouse took more than 20 minutes.
  • Maintenance: four engineers spend less than 10% of their time keeping the platform running end to end.
  • Cost: infrastructure cost is roughly 50× lower than the previous solution. Running in Microsoft AI's own data center is the main reason for the drop.

Microsoft AI researchers now see results from a run within minutes instead of hours. Alibaba took a similar route for flash sale analytics, querying Paimon tables with StarRocks; read how Alibaba built it.

What's next

More and more of the queries on Microsoft AI's RL training platform come from AI agents, and Microsoft AI's next steps follow that shift:

  • Agent-facing queries. A semantic layer will describe the storage layout and the caches and materialized tables available in Paimon and StarRocks, and expand automatically, so agents can write efficient queries.
  • AI-native maintenance. Agents will read key metrics from the control plane and decide when to resize a StarRocks cluster, reschedule a Flink job or trigger a full compaction.
  • More processing on ingest. No-code and low-code Flink processing, such as GPU inference, will make data more useful by the time it lands.
  • Semantic and image search. Agents will combine image search, semantic search and analytics in multi-step queries on one platform. Tencent runs vector search for RAG and image management on StarRocks.