Originally presented at a StarRocks webinar: When StarRocks Meets Vector Index: An Overview of StarRocks' Exciting New Feature
About Tencent
Tencent runs StarRocks across many of its best-known products, including Tencent Games, WeChat, Tencent Video and Tencent Hunyuan, for scenarios such as ad-hoc analysis and BI dashboards. Its StarRocks footprint adds up to more than 20,000 cores in over 100 clusters, the largest of which has thousands of machines, and it handles PB-scale analytical queries.
Tencent's image library needed BI dashboards and similarity search over the same data. Building vector indexes into StarRocks let one system serve both, instead of keeping a separate vector database in sync.
Key takeaway · One dataset, two workloads
The original talk was given by Petri (Pengcheng Zhang), Senior Engineer at Tencent Big Data, China.
Tencent's big data team has added vector search to StarRocks and uses it for two AI workloads: retrieval-augmented generation (RAG) for internal chatbots, and management of the image library behind Tencent's text-to-image models.
The challenge
To train and fine-tune its text-to-image models, Tencent built a content library holding billions of images. Model trainers pull images from it for training and fine-tuning. The raw files live in an object store. Each image also carries structured metadata (text description, source, URL, size, quality) and two embeddings: a feature vector for the image itself and one for its text description.
Two kinds of queries run against this data:
- BI dashboards that track image counts over time, image size statistics and quality, to help the team manage the library.
- Similarity search, where a user uploads an image or a text description and its feature vector is used to find the closest images in the library.
The team's first idea was a dedicated vector database. It ran into three problems.
Scale and compatibility
The library has billions of rows and more than 100 TB of data. The team doubted that the existing vector databases would stay stable and available at that size. A new system would also have to fit Tencent's platform: MySQL-compatible interfaces, data sync tools, profiling tools and cluster monitoring. Closing those gaps could mean new technology stacks and languages.
Complex search
The use case needs more than plain nearest-neighbor lookups:
- Top-k search: the k images closest to a query.
- Range search: all images within a given similarity range.
- Hybrid search: top-k or range search combined with filters on structured attributes, for example the k most similar images where
image_quality = "high"andimage_size > 1024.
When the team looked in June 2023, no existing vector database supported all three.
Heterogeneous workloads
The dashboards need aggregations and joins, which vector databases handle poorly. Pairing a vector database with StarRocks would cover both query types, but it would mean copying data between systems, keeping them consistent, and paying to operate both.
The solution
Instead of adding a new system, the team chose to build vector database features into StarRocks. StarRocks had been running in Tencent production for years across many scenarios, it came with mature management and maintenance tooling, and engineers already knew it well. A new vector database would have lacked that tooling and been costly to bring in. The team addressed each challenge with a matching piece of work.
Vector indexes inside StarRocks
Tables can now define a vector index on an embedding column. Two index types are supported, HNSW and IVF-PQ, with L2 distance or cosine similarity as the metric. Top-k, range and hybrid queries are written in SQL, and the optimizer plans a scan of the vector index that returns approximate results.
CREATE TABLE `vector_table` (
`id` varchar(100) NOT NULL COMMENT "",
...
`question_embedding` array<float> NOT NULL COMMENT "",
INDEX my_index (`question_embedding`) USING VECTOR_INDEX PROPERTIES (
"dim" = "768",
"index_type" = "hnsw",
"metric_type" = "l2_distance",
"efSearch" = "40",
"M" = "32",
"efConstruction" = "40",
"is_vector_normed" = "false"
) COMMENT 'embedding vector index'
) ENGINE=OLAP
PRIMARY KEY(`id`)
DISTRIBUTED BY HASH(`id`) BUCKETS 1;
-- Filtered top-k search
select * from vector_table
where source = "wiki"
ORDER BY approx_l2_distance('0.1, 0.2, 0.3, ...., 0.1', question_embedding)
limit 10;In storage, each tablet keeps its rows in segment files and its vector indexes in separate index files. Because StarRocks uses MVCC, updates produce new versions of both. Every flush of data files also builds and flushes the matching vector index, the files stay immutable, and older versions are compacted on a schedule. All existing StarRocks ingestion methods, including Stream Load and Routine Load, work unchanged. The team recommends batching updates and loading them periodically rather than doing frequent small upserts.
TenANN, a dedicated index library
StarRocks routes every vector index operation through TenANN, a library the team wrote to integrate and extend existing index libraries. Today it works as an extended version of Faiss. It handles building, saving, loading and caching, and searching indexes, with index metadata kept in RocksDB.
| TenANN | Faiss | Comments | |
|---|---|---|---|
| Index File Cache | Yes | No | TenANN offers out-of-the-box support for LRU index file caching. |
| Range Search over HNSW | Yes | No | |
| Range Search over IVF-PQ | Yes | No | |
| IVF-PQ BlockCache | Yes | No | Faiss reads indexes file-by-file; TenANN reads and caches in data blocks, using half the memory for equivalent in-memory performance and recall. |
One system for both workloads
With vector indexes, the same StarRocks cluster serves dashboard queries and similarity search. The team is also building a serving engine based on scatter-gather execution, and workload isolation, to support the two query types better.
The results
AIGC image management
The image library used to run on two separate systems, one for OLAP queries and one for vector search. After both moved to StarRocks, similarity search runs twice as fast and total cost has fallen by half.
RAG for chatbots
Some Tencent chatbots use StarRocks as the vector store for RAG. Documents are split into chunks, embedded and stored in StarRocks. For each question, the relevant chunks are retrieved and added to the prompt sent to the LLM.
Performance
In production, Tencent reports:
- Top-10 search over 1M 1024-dimension vectors with an HNSW index on one 8-core, 16 GB machine: recall above 95% and an average latency of 10 ms.
- Top-10 search over 1B 1024-dimension vectors (more than 100 TB of data) with an IVF-PQ index on 26 machines (84 cores and 300 GB each): recall above 80% and an average latency of 10 s.
The team also ran VectorDBBench in its own environment. On serial P99 latency, StarRocks landed in the middle of the dedicated vector databases tested, such as Milvus, Zilliz Cloud and Pinecone. The team attributes the gap to fixed overhead from StarRocks' query scheduling and optimizer and is working to reduce it.
What's next
- Reduce the fixed per-query overhead to speed up vector search.
- Finish the scatter-gather serving engine and workload isolation.
- The team is contributing this work back to StarRocks; at the time of the talk, the host said it was planned for the StarRocks 3.4 release.