Originally published at: LinkedIn
About Roblox
Roblox runs a large internal observability platform built on open-source components. As LLM-based tools spread across the company, its observability team extended that platform to cover AI agents, bots and coding tools, and added StarRocks as the store for LLM logs and traces.
Roblox did not build separate infrastructure for AI workloads. It extended its existing OpenTelemetry pipeline at a few points: an LLM gateway, StarRocks for LLM logs and traces, and Grafana plugins.
Key takeaway · Extend the existing platform
The original post, "GenAI Observability at Roblox: Where We Started," was written by Jay Moorthi of Roblox.
The challenge
Within a short time, Roblox had dozens of LLM-based systems: AI agents, classifiers, support bots and coding harnesses. Some were custom builds, some used LangChain, and some wrapped Claude or OpenCode. Roblox's other services had reliable observability data; these new systems mostly did not, even though LLM behavior varies widely from one request to the next.
There were no internal design patterns or industry standards to follow yet. Each system produced telemetry in its own format, sent it to its own destination and used no shared schema. Bug reports from engineers described symptoms such as a tool that seemed slow, a session that hung or a bot that gave confused answers, and the team could neither act on them nor reproduce them.
Capacity was not the main problem. Roblox's existing pipeline, which combines OpenTelemetry clients and the OpenTelemetry Collector with Apache Kafka, Logstash, Elastic, VictoriaMetrics, Ceph Object Storage and Grafana, already carries more than 10 million messages per second, or trillions each week. The harder questions were about the data itself. LLM telemetry has a different shape from other telemetry, raises new security concerns, and its outliers are often where the important problems show up.
Design principles
Before choosing components, the team agreed on six principles:
- Meet users where they are: no new UI, query language or workflow.
- Make LLMs part of the service graph: the same diagnostic support whether or not a system calls an LLM.
- Sample only as a last resort: outlier agent behavior often matters most, so keep 100% of LLM traffic, with no downsampling or throttling.
- Build in security from day one: LLM traffic gets the same guardrails and access controls as PII.
- Reuse existing scale: run on internal systems that already meet strict SLAs.
- Stay on open standards and open source: remain vendor-agnostic and contribute fixes upstream.
These principles pointed to four additions: one place to instrument LLM traffic, a standard telemetry stream, storage that could keep full-fidelity LLM data, and a means of showing that data in tools engineers already used.
The solution
A gateway as the instrumentation point
While this work was in progress, Roblox rolled out a LiteLLM-based LLM gateway, which became the single instrumentation point. Services behind it emit a standard telemetry stream without custom onboarding code or extra work for their engineers.
Routing LLM data to StarRocks
Telemetry leaves the gateway and enters the OpenTelemetry Collector, which normalizes, enriches and redacts it, then routes it into the existing pipeline. Kafka acts as a durable buffer. For the new kind of data, LLM logs and traces, Roblox chose StarRocks: open-source OLAP that runs on object storage and supports RBAC and query isolation.
Because routing is handled in the OpenTelemetry Collector, the team could add StarRocks quickly and with little risk to the rest of the pipeline. Running StarRocks on object storage keeps storage costs at HDD level, while compute scales independently for both interactive queries and large analytical jobs.
In the diagram, StarRocks, labeled as disaggregated OLAP with 100% sampling, serves three consumers: Grafana, data access through MCP servers, a CLI and a troubleshooting skill, and benchmarking with agent-bench and a regression suite. Its storage scales out to object storage that needs no SSDs, and object storage feeds batch analytics in the data lake in Parquet format. Kafka also feeds near-real-time streaming analytics for intent and sentiment, and agent-bench runs flow back into the pipeline as a new source.
Grafana for engineers
Grafana was already in wide use at Roblox, so the team built Grafana plugins that let engineers browse LLM sessions and turns there. The plugins follow the OpenTelemetry semantic conventions for LLM data.
The results
All LLM telemetry now flows through one OpenTelemetry-based platform. Every service behind the internal model gateway is covered by default, sampled at 100%, with no onboarding code. The platform reports tokens and cost for human and service usage alike, and code fingerprinting traces every AI-authored line of code back to where it came from.
Results reported so far:
- Scale: more than 50 production LLM services, 12 coding harnesses and 50 models, at millions of tokens per second.
- Troubleshooting: engineers find the root cause of problems with their coding agents on their own.
- Cost visibility: one set of cost dashboards covers all of Roblox's AI coding tools.
- Performance: within one month, the number of users who hit slow first-token times fell by 20x.
- Code attribution: all new code at Roblox is fingerprinted. Most of it is attributed to either a coding harness and model pair or an automated code generator, backed by an agent trace.
What's next
Roblox is adding batch and streaming analytics, continuous classifiers for intent, sentiment, safety and other signals, and benchmarking and evaluation tools.