InstrumentalManufacturing failure analysis

Intelligent Agents in Manufacturing Processes at Instrumental

Instrumental's analysis engine, Solve, uses AI agents to find why electronics fail on the production line by querying factory tables with billions of rows. After moving Solve from Amazon Redshift to StarRocks, a full investigation takes 5 minutes or less instead of more than 30.

≤5min

per Solve investigation

Down from 30+ minutes on Amazon Redshift

<30%

of session time spent on database queries

Down from over 80% on Amazon Redshift

$2M

in parts recovered from failed units

Recovered by customers using Solve's findings

CompanyInstrumental
Use caseManufacturing failure analysis
Previous solutionAmazon Redshift

This story is based on a NextQuery 2026 session by Xuyen On, Tech Lead of the Data Platform at Instrumental. NextQuery is PhoenixAI's Data + AI Infrastructure Summit. Watch the session

About Instrumental

Instrumental started as an automated optical inspection (AOI) company. Its stations sit on electronics production lines: as each unit is built, a worker photographs it at the station, and Instrumental's AI returns a green, yellow or red light and points to the component that needs fixing before the unit is run again.

Instrumental's customers make phones, VR headsets and printed circuit board assemblies. Most of its business today is AI server racks, which are expensive and go through many assembly steps. Catching a defect at the station saves those customers time and money and gets products to market sooner.

Each station's inspection images and test results are combined with process data from the factory into a single record for every unit, so engineers can trace a unit across builds and factories. On top of these records, Instrumental built Solve, an analysis engine that uses AI agents to find out why units fail.

The challenge

How Solve answers a question

With Instrumental's Solve, an engineer asks a question in plain language, such as which components failed most at a site in the last week. Solve's AI agents interpret the question with Instrumental's proprietary context knowledge, write SQL against the factory data, and return a chart with a short written answer. They correlate test results with inspection images and process data to rank the likely causes of a failure.

Pulling that analysis together by hand, across different data sources, can take an engineer weeks or months. Solve does it in seconds or minutes. That depends on the database answering the agents' queries quickly.

Solve chat session: an engineer asks for the highest-failure components at a factory in the last week; the Solve orchestrator loads 503 context cards and calls its query_starrocks tool 13 times, each returning a few rows. Beside it, the three steps: ask in natural language, AI agents plus context, query StarRocks
One Solve question: 503 context cards and 13 agent-written queries. Source: Instrumental, NextQuery 2026

Why Solve was slow on Amazon Redshift

Instrumental's Solve draws on many sources. Visual inspection data comes from Instrumental's own PostgreSQL database, and customers' process data arrives through integrations with Databricks, BigQuery and PostgreSQL, or as Parquet, CSV and Excel files.

Most tables hold tens to hundreds of millions of rows, and one of the biggest holds more than 48 billion. A typical Solve session sends anywhere from 13 to more than 100 queries, and the agents write wildcard searches and complex joins that are not typical data warehouse operations.

Solve first ran on Amazon Redshift. It worked, but a typical session took more than 30 minutes before the user reached an answer. Users described typing a question, going to get a coffee, and coming back hoping the answer would be useful.

More than 80% of that time went to database queries. Instrumental hired the tech lead of its data platform in part to find a way to scale Solve.

The solution

Testing the queries nothing else could finish

When Instrumental evaluated StarRocks as a faster database for Solve, the data for the analytical use case Instrumental had in mind was not ready yet. Instrumental tested StarRocks with what it had instead: the hardest OLTP queries from its own application.

These queries joined tables of more than 8 billion rows with other tables in the billions, on a data model with very large JSON columns. They returned small result sets, but their complicated filters made them time out on both PostgreSQL and Redshift.

On StarRocks they completed in 5 minutes. Instrumental's app dev lead was surprised, because Instrumental had never been able to complete these queries, and that result is what brought StarRocks into Instrumental.

The worst query in Instrumental's application traverses a table of more than 8 billion rows and joins another table in the billions, with large JSON columns and a small result set. It crashed on PostgreSQL and on Amazon Redshift and finished in about 5 minutes on StarRocks
The application query that settled the choice. Source: Instrumental, NextQuery 2026

Instrumental took this away as a best practice: start with the worst queries, then find out where the bottlenecks are.

One warehouse for agents and dashboards

StarRocks is now the data warehouse behind Solve and behind Instrumental's near real-time dashboards. Factory data from customers' platforms, databases and files lands in StarRocks, and Solve's agents query it directly.

For live data, Flink CDC streams row-level changes from Instrumental's PostgreSQL database into StarRocks, so dashboards and agent-generated charts query fresh factory data in near real time.

Reference architecture: customer factory data from Databricks, BigQuery, PostgreSQL and Parquet, CSV and Excel files, plus Instrumental's inspection data streamed from PostgreSQL through Flink CDC, lands in a StarRocks data warehouse with Primary Key tables, partitions sized to memory and tables of billions of rows. Solve's AI agents send SQL to StarRocks and turn the results into a chart and short answer, and near real-time dashboards read from the same warehouse
Instrumental's reference architecture for Solve

For real-time updates and dashboards, Instrumental uses StarRocks Primary Key tables. They take quick updates and keep every row, where the partition overwrites common in data warehouses are slower. The lesson Instrumental took away: get correctness right at ingestion.

Ingestion also ran into memory pressure. Each node has about 50 GB of RAM, so Instrumental uses monthly partitions that keep each chunk around 50 GB and let the index fit in memory. Without that, the ingest rate slows to a crawl.

Engineers can also ask Solve open-ended questions, such as the failure rate across a product's whole history. Those can run long, and too many table scans can slow the warehouse, so Solve steers users toward one- to two-week windows where possible.

StarRocksApache Flink CDCPostgreSQLDatabricksGoogle BigQuery

The results

Faster Solve investigations and answers engineers can act on

Instrumental's move to StarRocks changed both how long a Solve investigation takes and where that time goes.

  • Investigation time: a full Solve session now takes 5 minutes or less, down from more than 30 minutes on Redshift, more than six times faster. Engineers can now work with Solve interactively.
  • Query speed: analytic queries that took minutes on Redshift finish in tens of seconds, and the simpler, standard queries the agents write come back quickly.
  • Time in the database: queries now take less than 30% of a Solve session, down from more than 80%. Most of the time is now spent in the LLM processing the results.

In one example, Solve found that a single connector, failing at two positions on one board, was the top failing component by a wide margin. It accounted for more than 500 distinct units, 1.5% of the more than 33,000 boards tested. Boards without issues were filtered out, and about 47% of the genuinely defective units traced to that one part.

Solve answer for one site and one week: a Pareto chart of distinct failing units by component, led by part-0001, with a short answer naming connector part-0001 at positions J288 and J39 on board-0001. Callouts show about 47% of genuine defect units trace to that connector and about 69% of failures come from the top three components, all interconnects. Part and board names are anonymized
A Solve answer from billions of rows of factory data. Source: Instrumental, NextQuery 2026

Some of Instrumental's customers recovered $2 million worth of compute parts from the boneyard based on insights from Solve. For Instrumental, that result alone paid for StarRocks.

Eightfold also moved from Redshift to StarRocks, for customer-facing analytics and agents; read how Eightfold made the move. For another team running agent-generated SQL in production, see how Conductor built agentic analytics.

What's next

With the query bottleneck gone, the next one for Instrumental is managing context for Solve's agents. Hundreds of context cards were manageable at first, but Solve now uses more than 500. Each card is a text field with an on/off switch, and creating and filtering cards is manual.

Instrumental is working on three fronts:

  • Context: Instrumental is evaluating Contextbase, PhoenixAI's context management system for AI agents. It has not been fully tested yet, and early results from simple use cases look very promising.
  • Semantics: Instrumental's and its customers' proprietary process knowledge is managed as context today, and the agents' own logic handles query generation and guardrails. A proof of concept of Semantic SQL would move query generation and guardrails out of the agents and into a semantic layer.
  • Evals: Instrumental has just started running evals on Solve to track how the agents perform as their logic improves. Building test harnesses and managing test data is a large project, and Instrumental is running a proof of concept with PhoenixAI's tooling for agent evaluations.