Advertising optimization, payment risk, commerce operations, and incident response all depend on seeing recent changes quickly. A result that arrives a minute late can mean a missed adjustment or a delayed response.
Ingestion, storage, and query engines have all become faster. But what users read is query results, not ingested rows. If data lands within seconds and the query over it still scans large amounts of detail and takes tens of seconds or minutes, the result is not real time.
Real-Time Analytics Must Be Both Fast and Fresh
Query optimization helps, but for stable, frequently executed workloads such as advertising metrics, log rollups, and transaction statistics, recomputing from raw data on every request rebuilds the same result again and again. Precomputation calculates those results ahead of time so that queries can read them directly.
Precomputed results also have to stay fresh. A dashboard that responds in 100 milliseconds but shows data from an hour ago is not real time, and recomputing the full history every few seconds to keep it fresh costs too much.
PhoenixAI asynchronous materialized views already use partition change tracking (PCT) to limit recomputation to changed partitions. The problem is that real-time writes usually concentrate in one hot partition. If today's partition holds 30 million rows and the latest batch adds 10,000, the change can still trigger recomputation of all 30 million rows. As the partition grows, refresh time stretches from seconds to minutes and can approach or exceed the refresh interval. Refresh cost needs to depend on what changed in this cycle, not on how much data has accumulated.
The Hard Part Is Maintaining Changes Correctly
“Process only what changed” sounds simple, but real-time data is not an append-only stream. Orders change state, refunds retract revenue, and advertising conversions can be reattributed. An UPDATE must add the new value and retract the old one. When a row moves between groups, both groups must change. When the current maximum is deleted, the correct MAX cannot be recovered with simple subtraction.
Real-time precomputation must therefore capture changes accurately, understand how they propagate through JOINs and aggregations, preserve state across refreshes, and commit the result together with the consumed position. A failure cannot skip data, and a retry cannot duplicate it. General-purpose stream processors such as Flink can solve these problems, but maintaining one stable analytical query should not always require a separate CDC pipeline, streaming job, checkpointing system, state backend, and target-table workflow.
PhoenixAI Incremental Materialized Views (IMVs) for native tables provide a database-native path. Data continues to land in PhoenixAI cloud-native tables, while the database understands each change, maintains the necessary state, and updates the materialized result. IMV is designed for real-time analytical workloads where query logic is stable, each refresh changes far less data than a conventional recomputation would touch, and both low query latency and high data freshness matter.
PhoenixAI IMV: Turning Row-Level Changes into Correct Materialized Results
To users, PhoenixAI IMV is a familiar materialized view: define the query in SQL, build the complete result on the first refresh, and let the database maintain it from then on. Underneath, the system has to answer three questions: What changed in this refresh window? Which query results do those changes affect? And when is it safe to advance the consumed position?
PhoenixAI first answers “what changed” through Bookmarks and CDC. The system reads the continuous window (last successful position, current position], captures every INSERT and DELETE, and represents an UPDATE as “retract the old value, then insert the new value.” If the same row changes multiple times within the window, PhoenixAI first collapses those events into the net change that must be applied to the result. This gives the incremental computation an accurate, continuous input with no gaps.
Next, the optimizer answers “what is affected” with a Delta plan. It rewrites the original query into an incremental maintenance plan that propagates changes correctly through filters, projections, JOINs, UNION ALL, GROUP BY, and aggregations. The hard part is not adding one more value to SUM; it is retraction-aware incremental computation. After an update or delete, JOIN and aggregate results must be corrected. If a row moves between groups, both the old and new group must change. If a group disappears entirely, its result must be deleted. For non-invertible aggregates such as MIN and MAX, PhoenixAI recomputes only the affected groups—not the entire hot partition. The result is a precise set of UPSERTs and DELETEs against the materialized view.
Finally, PhoenixAI answers “when can the position advance” with exactly-once transactions. The materialized view's UPSERTs and DELETEs are committed in the same transaction as the Bookmark position. On success, the result and the position advance together. On failure, neither advances, and the same window is retried. This prevents data loss from advancing the position too early and prevents duplicate results when a window is consumed again.
With all three in place, queries read a materialized result; refresh work depends mainly on the current change set rather than total history; results match a full recomputation even after UPDATEs and DELETEs; and there is no separate CDC, state, and recovery pipeline to run outside the database.
Reading less data is the easier part. The harder part is keeping the result identical to a full recomputation after updates, deletes, and failures. That is why IMV combines row-level change capture, retraction-aware incremental computation, and an exactly-once commit, rather than only narrowing the scan.
At 9 Billion Rows, Refresh Time Remains Between 1 and 2 Seconds
For a precomputation mechanism, the question is whether refresh time stays stable as data accumulates. In a real-time market-data CostBench test, we compared PCT with native-table IMV using the same PhoenixAI table, ingestion pattern, and aggregation definition. Three 8-core compute nodes handled the workload. In each cycle, 10 million rows were loaded in parallel into a DUPLICATE KEY native table before the materialized view was refreshed.
Figure 1. PCT and PhoenixAI IMV refresh time as the native table grows from 300 million to 9 billion rows.
At 9 billion rows, IMV refreshed about 70× faster than PCT. The shape of the two curves matters more than that single point. As the native table grew 30×, from 300 million to 9 billion rows, IMV refresh time moved from 1.1 to 1.9 seconds, so historical data volume no longer drove refresh cost. PCT refresh time grew with the hot partition, from about 15 seconds to more than two minutes.
Steady-state ingestion throughput stayed in the same range for both approaches, so IMV did not move refresh cost onto the write path. IMV did use more storage to keep Row IDs and intermediate aggregate state: 318 MB in this test, against 117 MB for the equivalent PCT materialized view. The tradeoff is extra state storage for stable refresh latency, fresher results, and less repeated computation.
Where IMV Fits
Native-table IMV fits continuously ingested workloads such as dashboards for advertising impressions, clicks, conversions, and revenue; minute-level log and observability rollups; order, payment, refund, and operational-state analysis; and IoT device events. These workloads have three things in common: stable query logic that runs repeatedly, a change set much smaller than what a conventional refresh would recompute, and a need for both low query latency and fresh data.
General-purpose stream processors are still the right tool for complex event orchestration, arbitrary streaming transformations, and highly customized multi-stream processing. Conventional materialized views are still the more general choice when the SQL changes often, refresh is not the bottleneck, or the workload needs a broader query surface.
For workloads that fit, IMV is a shorter path than an external streaming pipeline and more stable than recomputing hot partitions. Data, state, computation, checkpoints, results, and observability stay inside the database, and users work with SQL and materialized views instead of running an additional stack for each real-time aggregate.
Roadmap: From Strict Incremental to Automatic Mode Selection
PhoenixAI 26.2 has shipped the first release with strict incremental maintenance for shared-data cloud-native tables. It supports append workloads and, with CDC enabled on PRIMARY KEY tables, handles UPSERTs, UPDATEs, and DELETEs. Supported query shapes include filters, projections, INNER and CROSS JOINs, GROUP BY, and common aggregates such as SUM, COUNT, AVG, MIN, and MAX. Refresh can be triggered by base-table changes, a schedule, or manually, with observable source-version ranges and refresh history. In strict mode, a query or data change that cannot be maintained safely is rejected explicitly rather than silently degrading into an unpredictable full recomputation.
PhoenixAI 26.3 is planned to introduce AUTO mode. At creation and refresh time, the system will decide whether the current query and data changes can be maintained incrementally. When they can, it keeps using IMV. When a recomputation is needed, it can use PCT to rebuild the baseline and then resume incremental refreshes. With AUTO, users get incremental refresh where it applies without knowing every maintenance boundary in advance, and refresh keeps running as data and workloads change.
Summary
For real-time analytics, the useful measure is the time and resources between a data change and the moment users can act on the updated result, and whether that stays stable as data grows.
Native-table IMV handles row-level change capture, incremental plan rewriting, result retractions, aggregate-state maintenance, checkpoint management, and failure recovery inside the database.
Refresh works on the rows that changed, instead of the full history at query time or the full hot partition at refresh time.