Stop Training Codebooks: How TurboVec Shrinks 31 GB Vector Stores into 4 GB with Zero-Rebuild SIMD Search

A 10 million document corpus stored as standard 32-bit floating-point vectors consumes roughly 31 GB of RAM. When memory budgets tighten or local inference pipelines require low footprint, vector stores usually force engineers into standard Product Quantization (PQ). That means spending massive computational overhead running k-means clustering across large batches just to build offline codebooks, only to watch search recall drift the moment your data distribution shifts.

TurboVec throws out that offline training pipeline entirely. Built on Google Research's TurboQuant algorithm and implemented in Rust with native Python bindings, TurboVec compresses that same 10 million document corpus into 4 GB of memory. It eliminates the separate training step, achieves near-optimal distortion through mathematical guarantees, and executes filtered searches faster than FAISS IndexPQFastScan across modern ARM and x86 hardware.

Here is an architectural look at how TurboVec works under the hood, how it handles dynamic online updates, and how its hand-tuned SIMD kernels outpace traditional quantized index implementations.

The Mathematics of Data-Oblivious Quantization

Traditional vector quantization schemes like FAISS IndexPQ depend on data-dependent codebook training (typically k-means++). If your incoming data drifts or you need continuous online ingest, those static cluster centroids degrade, requiring expensive index rebuilds.

TurboQuant circumvents this by relying on the geometry of high-dimensional hyperspheres. The quantization pipeline follows six distinct phases:

[Raw Vector (FP32)]
       │
       ▼
 1. Extract & Store Norm (L2) ──► Unit Direction on Hypersphere
       │
       ▼
 2. Random Orthogonal Rotation Matrix ──► Coordinates follow Beta / Gaussian N(0, 1/d)
       │
       ▼
 3. TQ+ Coordinate Calibration ──► Fit scale/shift to codebook outer bounds
       │
       ▼
 4. Lloyd-Max Scalar Quantization ──► 2-bit (4 buckets) or 4-bit (16 buckets)
       │
       ▼
 5. Bit-Packing ──► Pack coordinates tightly into bytes (e.g., 6144 B -> 384 B)
       │
       ▼
 6. Length-Renormalized Scoring ──► Store ||v|| / ⟨u, x̂⟩ to eliminate inner-product bias

1. Normalization and Random Rotation

Every vector is stripped of its L2 norm, saving the magnitude as a single float and treating the rest as a unit direction on a hypersphere. TurboVec then multiplies all vectors by the same random orthogonal matrix.

Because of the properties of high-dimensional hyperspheres, rotating an arbitrary vector with a random orthogonal matrix forces each coordinate to independently follow a Beta distribution. In high dimensions, this distribution converges to a canonical Gaussian $N(0, 1/d)$. The transformation makes the coordinate distribution predictable regardless of the underlying semantic content.

2. TQ+ Per-Coordinate Calibration

At finite dimensions (such as GloVe $d=200$ or low-bit configurations), coordinates can slightly drift from the asymptotic distribution. TurboVec addresses this via TQ+ calibration. Calling index.calibrate(sample) with roughly 1,024 representative rows fits two scalars per coordinate (a shift and a scale) to map empirical quantiles onto the outermost Lloyd-Max centroids. The probability levels track the bit width (~0.933 at 2-bit, ~0.996 at 4-bit). Once fitted, this calibration is locked and applied to every subsequent vector addition with zero index rebuilds.

3. Lloyd-Max Scalar Quantization and Bit-Packing

Because the distribution is mathematically known, TurboVec uses precomputed Lloyd-Max codebooks:

2-bit quantization: 4 discrete buckets
4-bit quantization: 16 discrete buckets

These bucket boundaries minimize Mean Squared Error (MSE) and achieve distortion within a factor of 2.7x of Shannon's information-theoretic distortion-rate limit.

Coordinates are converted into small integers and packed into byte streams. A 1536-dimensional vector drops from 6,144 bytes in float32 down to 384 bytes in 2-bit mode, yielding a 16x compression ratio.

4. Length-Renormalized Scoring

Scalar quantization tends to underestimate inner products because the reconstructed unit vector $\hat{x}$ is slightly shorter than the original unit vector $u$. TurboVec adapts a correction technique from RaBitQ: during encoding, it computes one extra $d$-dimensional dot product $\langle u, \hat{x} \rangle$ and stores the scalar:

$$\text{Correction Factor} = \frac{|v|}{\langle u, \hat{x} \rangle}$$

At query time, the SIMD search kernel multiplies candidate scores by this factor prior to heap insertion. This eliminates downward estimation bias at zero search-time computational cost.

Hardware-Tuned SIMD Execution

Quantization reduces memory bandwidth bottlenecks, but scoring throughput depends entirely on hardware-level execution. Instead of decompressing compressed vectors back to float32, TurboVec rotates the query vector once into the transformed domain and computes dot products directly against quantized codes using lookup tables (LUTs).

The underlying engine includes custom kernels for diverse instruction sets:

ARM NEON: Uses SDOT and SMMLA instructions to score vector-major layouts directly.
x86 AVX-512: Leverages AVX-512 VNNI dot-product kernels and vpermb LUT permutations for short 2-bit accumulation loops.
Fallbacks: Provides runtime-detected AVX2 implementations alongside scalar fallbacks for older x86-64-v2 hardware.

Search Performance Benchmarks

Configuration: 100K vectors, 1K queries, $k=64$, median of 5 runs vs FAISS IndexPQFastScan.

Platform	Architecture	Bit Width	TurboVec vs FAISS Speedup
GCP c4a-standard-8	Google Axion (ARM, 8 vCPUs)	4-bit	3.5× faster (3.4x to 3.7x across cells)
GCP c4a-standard-8	Google Axion (ARM, 8 vCPUs)	2-bit	26% faster (22% to 29% across cells)
Intel Xeon Platinum 8481C	Sapphire Rapids (x86, 8 vCPUs)	4-bit	3.4× faster (3.2x to 3.5x across cells)
Intel Xeon Platinum 8481C	Sapphire Rapids (x86, 8 vCPUs)	2-bit	20% faster (5% to 32% across cells)

Low-Latency Mutations: Ingestion and Deletion

Most vector indexes optimized for fast scanning suffer when records need to be added or deleted in real time. FAISS IndexPQFastScan requires costly codebook training before ingestion and completely repacks stored codes when dropping records using IndexIDMap.

TurboVec avoids codebook training entirely, enabling immediate online writes and $O(1)$ removals.

Mutation Latency (100K Vectors Baseline)
┌─────────────────────────────────────────────────────────────┐
│ Single Vector Ingestion (add)                               │
│ TurboVec: 6.3 µs - 19.7 µs   [7.6x - 13.9x FAISS Speedup]   │
│                                                             │
│ Batch Ingestion (100 vectors)                              │
│ TurboVec: 4.6 µs - 16.3 µs/vector [4.6x - 15.1x FAISS]      │
│                                                             │
│ Single Deletion by ID (remove)                              │
│ TurboVec (IdMapIndex): 0.44 µs - 1.22 µs                    │
│ FAISS (IndexIDMap over FastScan): 0.19 s - 1.02 s           │
└─────────────────────────────────────────────────────────────┘

Because IdMapIndex.remove(id) relies on an internal swap-and-pop strategy with ID-map bookkeeping, deleting a vector executes in roughly one microsecond. In contrast, FAISS repacks the underlying fast-scan buffer, taking hundreds of milliseconds to over a full second for a single deletion.

Search-Time Filtering: Block-Granular Short-Circuiting

Standard filtered search often requires either post-filtering (which ruins top-k recall when filters are restrictive) or pre-filtering that completely bypasses vectorized SIMD scoring paths.

TurboVec implements allowlist filtering directly inside its SIMD execution loops at 32-vector block granularity:

[Incoming Search Request + Allowlist IDs]
                     │
                     ▼
       ┌───────────────────────────┐
       │ 32-Vector Block Evaluation│
       └─────────────┬─────────────┘
                     │
         Any Allowed Slots in Block?
        ┌────────────┴────────────┐
       YES                        NO
        │                         │
        ▼                         ▼
┌───────────────────────┐  ┌───────────────────────────┐
│ Run SIMD LUT Scoring  │  │ Short-Circuit Entire Block│
│ & Insert Allowed to   │  │ Skip LUT & Arithmetic     │
│ Top-K Min-Heap        │  └───────────────────────────┘
└───────────────────────┘

When an allowlist covers only a small fraction of the database, the engine skips LUT transformations and arithmetic across entire 32-vector chunks. The result set always returns $\min(k, n_{\text{allowed}})$ distinct vectors without requiring over-fetching hacks.

Python and Rust Implementation

TurboVec exposes both native Rust crates and a streamlined Python interface.

Python API with Stable IDs and Filtering

python
import numpy as np
from turbovec import IdMapIndex

# Initialize a 4-bit quantized index for 1536-dimensional embeddings
dim = 1536
index = IdMapIndex(dim=dim, bit_width=4)

# Vectors must be strict float32 2D arrays
vectors = np.random.randn(1000, dim).astype(np.float32)
doc_ids = np.arange(1000, 2000, dtype=np.uint64)

# Add vectors with external 64-bit IDs
index.add_with_ids(vectors, doc_ids)

# Filtered search using an ID allowlist
allowed_subset = np.array([1001, 1005, 1050, 1100], dtype=np.uint64)
query = np.random.randn(1, dim).astype(np.float32)

scores, matched_ids = index.search(query, k=10, allowlist=allowed_subset)

# O(1) removal
index.remove(1005)

# Durable incremental persistence (single fsync)
index.sync("production_index.tvim")

Native Rust Implementation

rust
use turbovec::IdMapIndex;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let dim = 1536;
    let bit_width = 4;
    let mut index = IdMapIndex::new(dim, bit_width)?;

    let vectors: Vec<f32> = vec![0.0; 1536 * 3]; // Flattened vectors
    let ids: Vec<u64> = vec![1001, 1002, 1003];

    index.add_with_ids(&vectors, &ids)?;

    let query: Vec<f32> = vec![0.0; 1536];
    let (scores, matched_ids) = index.search(&query, 10);

    index.remove(1002);
    index.sync("production_index.tvim")?;

    Ok(())
}

Framework Ecosystem Integration

TurboVec provides drop-in replacements for in-memory and reference stores across the RAG ecosystem:

LangChain: pip install turbovec[langchain] replaces InMemoryVectorStore
LlamaIndex: pip install turbovec[llama-index] replaces SimpleVectorStore
Haystack: pip install turbovec[haystack] replaces InMemoryDocumentStore
Agno: pip install turbovec[agno] replaces LanceDb

Recall Quality: TurboQuant vs FAISS

Compression without recall is useless in production. Below is the recall comparison between calibrated TurboQuant (TQ+) and FAISS IndexPQ (configured with LUT256, nbits=8, float32 LUT, sub-quantizer counts matched to TurboQuant bit rates).

Dataset	Dimensions	Bit Width	Metric	TQ+ vs FAISS IndexPQ
OpenAI Embeddings	$d=1536$	2-bit & 4-bit	Recall@1	TQ+ leads by 0.9 to 2.9 points on 3 of 4 cells
OpenAI Embeddings	$d=3072$	2-bit & 4-bit	Recall@8	Both systems achieve $\ge 0.997$ at $k \le 4$ and $1.0$ at $k=8$
GloVe Word Vectors	$d=200$	4-bit	Recall@1	TQ+ leads by +1.9 points
GloVe Word Vectors	$d=200$	2-bit	Recall@1	TQ+ reaches 0.572 vs FAISS 0.564 (+0.8 points)

Even in low-dimensional regimes ($d=200$) where asymptotic Gaussian assumptions loosen, TQ+ coordinate calibration compensates for the drift, retaining high recall without needing iterative k-means codebook generation.

Persistence and Safety

For systems requiring durability under frequent mutations, standard write and load methods handle full-file snapshotting. In addition, TurboVec exposes index.sync(path).

sync(path) writes out only the modifications introduced since the previous sync call using a single fsync followed by an atomic rename. Appending a small vector batch or removing a single ID costs only a few milliseconds to make durable, regardless of how large the primary index file is on disk.

When designing embedded or air-gapped retrieval pipelines where memory capacity, predictable CPU utilization, and zero-downtime writes are critical, TurboVec presents a performant alternative to traditional training-heavy vector stores.

References

https://github.com/RyanCodrai/turbovec
TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate (ICLR 2026)
RaBitQ: Quantizing High-Dimensional Vectors with a Theoretical Error Bound for Approximate Nearest Neighbor Search (SIGMOD 2024)
FAISS Fast accumulation of PQ and AQ codes

Tags : PerformanceEngineering MachineLearning SIMD Rust VectorDatabase