A 10-million-document vector corpus stored as standard float32 embeddings demands 31 GB of memory. In high-throughput production environments, that memory footprint translates directly to bloated cloud bills, high cache churn, and prohibitive infrastructure requirements.

TurboVec alters this equation completely. By implementing Google Research's TurboQuant algorithm in a standalone Rust engine with Python bindings, TurboVec shrinks that 31 GB corpus down to 4 GB of RAM. More importantly, it achieves this while outpacing FAISS IndexPQFastScan in search latency across both ARM and x86 architectures, eliminating the traditional trade-offs between memory footprint, recall, and search throughput.

+-------------------------------------------------------------------------+
|                  Float32 Corpus: 31 GB Memory Footprint                 |
+-------------------------------------------------------------------------+
                                     |
                          TurboVec Compression (4-bit)
                                     v
+------------------------------------+
|   TurboVec Corpus: 4 GB Footprint  |  ---> 3.4x Faster SIMD Search
+------------------------------------+

---

The Core Problem with Production Vector Quantization

Most approximate nearest neighbor (ANN) pipelines rely on Product Quantization (PQ). While PQ offers solid compression, it introduces substantial friction into real-world production systems:

1. Mandatory Training Steps: Standard PQ requires running k-means clustering across a large, representative sample of vectors before indexing a single document.
2. Rebuild Headaches: As your data distribution shifts, quantization codebooks degrade, requiring index rebuilds and downtime.
3. Expensive Mutability: Removing or updating vectors in heavily packed structures like FAISS IndexPQFastScan requires repacking the underlying codes, turning a single delete into a multi-second operation.

TurboVec bypasses offline codebook training altogether. It relies on a data-oblivious quantization scheme that delivers near-optimal distortion rates from the very first vector you insert.

---

How TurboQuant Works: From Direction to Bit-Packing

TurboVec compresses high-dimensional vectors by treating them as points on a hypersphere. The mathematical pipeline converts unconstrained vectors into packed, bias-corrected coordinates through six core stages.

[ Input Vector (FP32) ]
          |
          v
   1. Normalization         --> Strip vector norm: ||v|| stored as float
          |
          v
   2. Random Rotation       --> Multiply by orthogonal matrix (Beta distribution)
          |
          v
   3. TQ+ Calibration       --> Fit coordinate shift and scale (1024 sample rows)
          |
          v
   4. Lloyd-Max Bucketing   --> Map to precomputed centroids (4 or 16 buckets)
          |
          v
   5. Bit-Packing           --> Compress coordinates (2-bit or 4-bit)
          |
          v
   6. Length Renormalization--> Store ||v|| / <u, x̂> for unbiased inner products

1. Normalization

The norm $||v||$ is extracted and preserved as a single float. The remaining vector is treated purely as a unit direction on the unit hypersphere.

2. Random Orthogonal Rotation

The unit vector is multiplied by a shared, random orthogonal matrix. This operation enforces a critical property: regardless of the original data distribution, the rotated coordinates independently follow a Beta distribution that asymptotically converges to a Gaussian distribution $\mathcal{N}(0, 1/d)$ in high dimensions. The rotation makes the coordinate distribution mathematically predictable without analyzing the dataset.

3. Per-Coordinate Calibration (TQ+)

At finite dimensions (such as GloVe $d=200$ embeddings), coordinates can drift from the asymptotic ideal shape. TurboVec addresses this with TQ+ calibration. By passing a tiny sample (~1024 rows) to index.calibrate(sample), the engine fits two scalars per coordinate: a shift and a scale. This maps empirical quantiles directly to the codebook's outermost centroids (~0.933 probability level for 2-bit, ~0.996 for 4-bit). It is a one-time step that never requires retraining or re-indexing.

4. Lloyd-Max Scalar Quantization

Because the coordinate distribution is known mathematically, optimal bucket boundaries and centroids are computed analytically via the Lloyd-Max algorithm rather than derived from data via k-means. For 2-bit quantization, coordinates are mapped to 4 buckets; for 4-bit quantization, they are mapped to 16 buckets. This achieves distortion within a factor of 2.7x of Shannon's theoretical distortion-rate limit.

5. Bit-Packing

Quantized coordinate indices are packed into byte streams:

• A 1536-dimensional vector at 32-bit float precision consumes 6,144 bytes.
• The same vector at 2-bit precision consumes just 384 bytes (a 16x reduction).

6. Length-Renormalized Scoring

Scalar quantization naturally underestimates inner products because the reconstructed unit vector is slightly shorter than the original. TurboVec adapts a correction technique from RaBitQ: during encoding, it calculates the dot product between the rotated unit vector and its quantized centroid reconstruction $\langle u, \hat{x} \rangle$. It stores the scalar factor:

$$\text{Scalar} = \frac{||v||}{\langle u, \hat{x} \rangle}$$

During search, the scoring kernel multiplies candidate scores by this stored scalar before heap insertion. This eliminates downward estimation bias at zero search-time compute overhead.

---

Hardware-Level SIMD Execution

TurboVec avoids full vector decompression at query time. The query vector is rotated once into the transformed domain and evaluated against lookup tables (LUTs) using hand-crafted SIMD kernels.

Architecture	Primary SIMD Instructions	Fallback Paths
ARM (Axion / Neoverse)	NEON SDOT, SMMLA dot-product kernels	Scalar fallback
x86_64 (Sapphire Rapids)	AVX-512 VNNI, vpermb LUT scans	AVX2, Scalar fallback

All x86_64 builds target an x86-64-v2 baseline (SSE4.2 compatibility). High-performance AVX-512 and AVX2 paths are gated using runtime feature detection via is_x86_feature_detected!, ensuring binary portability across cloud instances.

Query Vector ---> [ Rotate Domain ] ---> [ Construct Nibble LUTs ]
                                                |
                                                v
                              +-----------------------------------+
                              |   SIMD Kernel Execution Loop      |
                              |   (32-Vector Block Granularity)   |
                              +-----------------------------------+
                                                |
                        +-----------------------+-----------------------+
                        |                                               |
              [ ARM: NEON SDOT/SMMLA ]                        [ x86: AVX-512 VNNI / vpermb ]

---

Empirical Benchmarks: TurboVec vs. FAISS

Benchmarks were performed on 100,000 vectors with 1,000 queries at $k=64$, evaluating median performance across 5 runs.

1. Search Throughput Across Architectures

TurboVec consistently outperforms FAISS IndexPQFastScan across various vector dimensions and bit widths.

Machine / Processor	Bit Width	vs. FAISS IndexPQFastScan Speedup	Kernel Mechanics
GCP c4a-standard-8 (Google Axion, 8 vCPUs)	4-bit	3.5x average (3.4x to 3.7x range)	Direct vector-major scoring via SDOT/SMMLA
GCP c4a-standard-8 (Google Axion, 8 vCPUs)	2-bit	26% faster (22% to 29% range)	Compact bit-scan execution
Intel Xeon 8481C (Sapphire Rapids, 8 vCPUs)	4-bit	3.4x average (3.2x to 3.5x range)	AVX-512 VNNI on vector-major layout
Intel Xeon 8481C (Sapphire Rapids, 8 vCPUs)	2-bit	20% faster (5% to 32% range)	vpermb LUT scan across short accumulate loop

2. Recall Performance ($k=64$)

Recall was evaluated against standard production FAISS IndexPQ (LUT256, nbits=8, float32 LUT).

   Recall @ 1 Comparison (OpenAI d=1536 / d=3072)
  -------------------------------------------------------------
   TurboVec (TQ+) : [===========>                      ] Beats FAISS by 0.9 to 2.9 pp
   FAISS IndexPQ  : [==========>                       ] Baseline
  -------------------------------------------------------------
   By k=4: Both methods reach >= 0.997
   By k=8: Both methods reach 1.000

• OpenAI $d=1536$ & $d=3072$: Calibrated TurboQuant (TQ+) outperforms FAISS at Recall@1 on three of four test configurations by 0.9 to 2.9 percentage points. Both indices reach $\ge 0.997$ recall by $k \le 4$ and hit $1.0$ by $k=8$.
• GloVe $d=200$: In low-dimensional spaces where the asymptotic Beta distribution assumption is loose, TQ+ calibration captures +1.9 points at 4-bit and +0.8 points at 2-bit for Recall@1 (0.572 vs FAISS's 0.564).

3. Mutation Latency: Inserts and Removals

Production indices require fast writes and clean deletions. FAISS IndexPQFastScan repacks code arrays on deletion, incurring massive latency spikes. TurboVec leverages an internal swap-and-pop id-map structure.

Operation	Metric / Configuration	TurboVec Latency	FAISS Latency	Speedup Factor
Single Insert ($n=1$)	Per-vector add()	6.3 to 19.7 µs	47.9 to 273.8 µs	7.6x to 13.9x faster
Batch Insert ($n=100$)	Amortized per-vector add()	4.6 to 16.3 µs	21.2 to 246.1 µs	4.6x to 15.1x faster
Delete by ID	remove(id) on 100K index	0.44 to 1.37 µs	0.19 to 1.02 seconds	Up to 1,000,000x faster

---

Filtered Search Without Query Degradation

Traditional post-filtering over-fetches candidates and discards non-matching records, which destroys recall on selective queries. Pre-filtering outside the index wastes CPU time by running unvectorized scalar loops.

TurboVec evaluates allowlists directly inside the SIMD kernel at 32-vector block granularity:

[ Allowlist Bitmask / IDs ] 
             |
             v
+-----------------------------+
| 32-Vector SIMD Block Check  |
+-----------------------------+
      /                 \
[ No Matches ]    [ Matches Found ]
      |                  |
      v                  v
(Skip Block LUT     (Execute SIMD Dot Product
 & Scoring Entirely) & Filter Slots at Heap Insert)

1. If zero vectors within a 32-vector block match the allowlist, the entire block is skipped before running LUT lookups or SIMD scoring loops.
2. If vectors match, SIMD scoring executes, and non-allowed slots are filtered out during heap insertion.
3. Search results return exactly $\min(k, n_{\text{allowed}})$ distinct results without padding or null values.

---

Developer Workflows: Python and Rust Implementation

Python API

python
import numpy as np
from turbovec import IdMapIndex

# Initialize index for 1536-dimensional vectors at 4-bit quantization
dim = 1536
index = IdMapIndex(dim=dim, bit_width=4)

# Vectors must be strict 2D float32 numpy arrays
vectors = np.random.randn(1000, dim).astype(np.float32)
doc_ids = np.arange(1000, 2000, dtype=np.uint64)

# Optional one-time calibration on a sample
index.calibrate(vectors[:500])

# Add vectors with stable 64-bit unsigned IDs
index.add_with_ids(vectors, doc_ids)

# Filtered search (Stage 1 candidate set from SQL/ACL/metadata)
allowed_ids = np.array([1001, 1005, 1050, 1100], dtype=np.uint64)
query = np.random.randn(1, dim).astype(np.float32)

scores, matched_ids = index.search(query, k=10, allowlist=allowed_ids)

# O(1) removals by ID
index.remove(1005)

# Fast incremental persistence (single fsync, crash-safe)
index.sync("production_index.tvim")

Rust API

rust
use turbovec::IdMapIndex;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let dim = 1536;
    let bit_width = 4;
    let mut index = IdMapIndex::new(dim, bit_width)?;

    let vectors: Vec<f32> = vec![0.0; 1536 * 3]; // Flat 2D array representation
    let ids: Vec<u64> = vec![1001, 1002, 1003];

    index.add_with_ids(&vectors, &ids)?;

    let query: Vec<f32> = vec![0.0; 1536];
    let (scores, result_ids) = index.search(&query, 10);

    // O(1) deletion and atomic snapshot
    index.remove(1002);
    index.write("index_snapshot.tvim")?;

    let loaded_index = IdMapIndex::load("index_snapshot.tvim")?;
    Ok(())
}

---

Ecosystem Integration and Persistence Architecture

TurboVec integrates directly into modern LLM orchestration frameworks as an in-memory vector store replacement:

• LangChain: pip install turbovec[langchain] (Replaces langchain_core.vectorstores.InMemoryVectorStore)
• LlamaIndex: pip install turbovec[llama-index] (Replaces llama_index.core.vector_stores.SimpleVectorStore)
• Haystack: pip install turbovec[haystack] (Replaces haystack.document_stores.in_memory.InMemoryDocumentStore)
• Agno: pip install turbovec[agno] (Replaces agno.vectordb.lancedb.LanceDb)

Storage Mechanics: Snapshots vs. Incremental Sync

• write(path) / load(path): Writes out a full snapshot of the index layout and metadata.
• sync(path): Incremental engine that persists only dirty memory blocks modified since the last call. Executing an append or deletion on a multi-gigabyte index costs only a few milliseconds, backed by a single fsync call that remains crash-safe at every byte boundary.

TurboVec operates entirely locally in-process. For teams deploying air-gapped retrieval-augmented generation (RAG) applications on isolated VPCs, it delivers bare-metal search speed without cloud egress or third-party database dependencies.

---

References

• https://github.com/RyanCodrai/turbovec