A 10 million document vector corpus stored as standard float32 embeddings demands roughly 31 GB of raw RAM. For engineers running memory-constrained environments, edge hardware, or localized RAG architectures, that footprint is a non-starter. You either pay exorbitant cloud bills for high-memory instances or accept the heavy maintenance tax of traditional Product Quantization (PQ) indices like FAISS IndexPQ, which force you through upfront codebook training, slow vector-repacking removes, and complex parameter tuning.
TurboVec upends this trade-off. Built in Rust with Python bindings, TurboVec implements Google Research’s TurboQuant algorithm to squash that same 31 GB dataset down to 4 GB. It does this without requiring a offline training phase, without rebuilds as your corpus grows, and while outperforming FAISS IndexPQFastScan search latency across both ARM and x86 architectures.
+-----------------------------------------------------------------------+
| 10M Document FP32 Corpus |
| 31 GB |
+-----------------------------------------------------------------------+
|
v
+-----------------------------------------------------------------------+
| TurboVec Quantized Index |
| 4 GB (16x) |
+-----------------------------------------------------------------------+
The Quantization Trap: Why Traditional PQ Hinders Real-Time Ingest
Standard Product Quantization algorithms like FAISS IndexPQ achieve high compression by clustering vector sub-spaces using algorithms such as k-means++. However, this approach introduces three major systemic bottlenecks:
- Training Dependencies: You cannot index vectors incrementally without first training codebooks on a representative sample. If your domain shifts, your quantization quality degrades until you rebuild the entire index.
- Dynamic Ingest Cost: Adding or removing vectors on a live FAISS index requires expensive layout modifications. Removing an ID via
IndexIDMapoverIndexPQFastScanforces a complete repacking of stored codes, turning a simple deletion into an operational block that can take anywhere from 0.19 to 1.02 seconds for a 100K vector index. - Hardware Inefficiencies: Standard PQ search relies heavily on multi-byte Look-Up Table (LUT) sweeps that struggle to fully saturate SIMD vector registers across heterogeneous hardware.
TurboVec bypasses data-dependent codebook training entirely by leveraging data-oblivious quantization.
Inside the TurboQuant Pipeline: Zero Training, Optimal Distortion
Instead of inspecting your dataset to find optimal cluster centers, TurboQuant exploits a key mathematical property of high-dimensional space: after a random rotation, any coordinate distribution converges to a predictable shape regardless of the input data.
Raw Vector (float32)
│
▼
1. Normalize Lengths ──────────────► Store ||v|| as float32
│
▼
2. Random Rotation ───────────────► Multiply by Random Orthogonal Matrix
│ (Coordinates convert to Beta / Gaussian)
▼
3. Per-Coordinate Calibration ────► TQ+ fits shift/scale scalars via index.calibrate()
│
▼
4. Lloyd-Max Quantization ────────► Map coordinates to fixed theoretical buckets
│ (2-bit: 4 buckets | 4-bit: 16 buckets)
▼
5. Nibble Bit-Packing ────────────► Compress 1536 dims into 384 bytes
│
▼
6. Length-Renormalized Scoring ───► Correct inner product bias using ||v|| / ⟨u, x̂⟩
Step 1: Length Normalization
The vector's original magnitude (norm) is extracted and saved as an isolated float32 scalar. The remaining directional component is normalized onto a unit hypersphere.
Step 2: Data-Oblivious Random Rotation
The unit vector is multiplied by a fixed random orthogonal matrix. This rotation redistributes energy across all dimensions. Regardless of whether your raw input consists of sparse text embeddings or dense audio vectors, the rotated coordinates independently follow a Beta distribution that converges to a Gaussian $N(0, 1/d)$ in high dimensions.
Step 3: Per-Coordinate Calibration (TQ+)
While the Beta distribution holds asymptotically, finite-dimensional embeddings (especially low-bit regimes or word-vector styles) experience subtle coordinate drift. The TQ+ variant solves this by fitting two scalars per coordinate (a shift and a scale) to align empirical quantiles with the codebook's outermost centroids. Calling index.calibrate(sample) on a small draw (~1024 vectors) permanently locks in this state without triggering full-index retraining or rebuilds. TQ+ yields recall gains of up to +2.2 percentage points at R@1 on drifting datasets like GloVe d=200.
Step 4: Mathematical Lloyd-Max Quantization
Because the post-rotation coordinate distribution is known mathematically, optimal bucket boundaries and centroids are calculated directly from probability theory using the Lloyd-Max algorithm. For 2-bit index configurations, coordinates fall into 4 buckets; for 4-bit, 16 buckets. No k-means, no iterations over data rows. The codebook achieves a distortion rate within a factor of 2.7x of Shannon's theoretical distortion-rate limit.
Step 5: High-Density Bit-Packing
Quantized coordinate indices (0-3 for 2-bit, 0-15 for 4-bit) are packed directly into byte streams. A 1536-dimensional vector drops from 6,144 bytes in FP32 down to 384 bytes in 2-bit mode, yielding a 16x compression ratio.
Step 6: Length-Renormalized Scoring
Scalar quantization inherently contracts vector magnitudes, causing inner products to be systematically underestimated. Adapted from the RaBitQ framework, TurboVec computes a scalar correction factor during encoding:
$$\frac{|v|}{\langle u, \hat{x} \rangle}$$
Where $u$ is the rotated unit direction and $\hat{x}$ is the centroid reconstruction. Multiplying candidate scores by this factor during search removes downward estimator bias at zero query-time cost and zero extra storage overhead.
Low-Level SIMD Execution on ARM and x86
TurboVec writes kernel execution paths natively for target hardware using explicit SIMD intrinsics rather than relying on autovectorization.
Query Input Vector
│
▼
Query Random Rotation
│
▼
+---------------------------+
| SIMD Kernel Dispatch |
+---------------------------+
│
┌──────────────────┴──────────────────┐
▼ ▼
ARM Architecture x86 Architecture
(NEON SDOT / SMMLA) (AVX-512 VNNI / vpermb)
│ │
└──────────────────┬──────────────────┘
▼
32-Vector Block Filtering Check
│
┌─────────────┴─────────────┐
▼ ▼
[Block Allowed?] [Block Excluded?]
│ │
▼ ▼
Nibble-Split LUT Scan Short-Circuit Block
│ (0 SIMD ops)
▼
Unbiased Score Accumulation
│
▼
Min-Heap Top-K Insert
ARM Architectures (Google Axion / GCP c4a)
On ARM64, search queries utilize hand-written NEON kernels leveraging SDOT (Signed Dot Product) and SMMLA (Matrix Multiply Accumulate) instructions. The vector-major memory layout enables direct dot-product accumulation across packed byte structures.
x86 Architectures (Intel Sapphire Rapids / Xeon)
Modern x86 chips utilize AVX-512 VNNI (Vector Neural Network Instructions) combined with vpermb for fast byte-level lookup table (LUT) evaluations. The kernel falls back dynamically to AVX2 implementations or scalar routines on older CPUs based on runtime feature detection via is_x86_feature_detected!.
Short-Circuiting Block-Level Filtering
Unlike post-filtering architectures that score candidate items only to discard forbidden items later, TurboVec handles filtering directly inside its SIMD execution loop. Operations run over contiguous 32-vector blocks:
- If an allowlist bitmask indicates that zero slots inside a 32-vector block are permitted, the entire SIMD block is short-circuited before performing any LUT lookups or scoring logic.
- If a block is partially allowed, invalid slots are filtered out during final min-heap insertion.
This ensures selective filters eliminate computational work proportionally to their selectivity.
Hardware Benchmarks: TurboVec vs FAISS
The following benchmark metrics reflect 100K vector indices queried with 1K input vectors at $k=64$ (median of 5 runs).
Search Latency Metrics
In throughput tests against FAISS IndexPQFastScan (with sub-quantizer counts matched identically to TurboVec bit rates), TurboVec dominates across both ARM and x86 hardware.
| Architecture | CPU Target | Bit Depth | FAISS Latency | TurboVec Latency | Speedup Factor |
|---|---|---|---|---|---|
| ARM | Google Axion (8 vCPU) | 4-bit | Baseline | 3.5x Faster | 3.5x |
| ARM | Google Axion (8 vCPU) | 2-bit | Baseline | 1.26x Faster | 1.26x |
| x86 | Sapphire Rapids (8 vCPU) | 4-bit | Baseline | 3.4x Faster | 3.4x |
| x86 | Sapphire Rapids (8 vCPU) | 2-bit | Baseline | 1.20x Faster | 1.20x |
Dynamic Mutability: Ingest & Removal Performance
Where TurboVec sets an engineering milestone is dynamic index modification. FAISS incurs heavy penalties when modifying indexed structures post-creation.
Single Removal Latency (100K Vector Index)
FAISS IndexIDMap: [========================================] ~1,020,000 µs (Repack)
TurboVec IdMap: [=] 1.22 µs (O(1) Swap-and-Pop)
| Operation | Target Config | FAISS Performance | TurboVec Performance | Efficiency Gain |
|---|---|---|---|---|
| Single Add ($n=1$) | Warm Index | High Overhead | 6.3 - 19.7 µs | 7.6x - 13.9x Faster |
| Batch Add ($n=100$) | Warm Index | High Overhead | 4.6 - 16.3 µs / vec | 4.6x - 15.1x Faster |
| Delete by ID ($n=1$) | IdMapIndex | 0.19 - 1.02 seconds | 0.44 - 1.22 µs | ~100,000x Faster |
FAISS relies on memory repacking during vector removals, causing second-long spikes. TurboVec's IdMapIndex executes deletions using an $O(1)$ swap-and-pop technique paired with index metadata tracking, resulting in sub-microsecond latency.
Crash-Safe Incremental Persistence
Standard in-memory engines require full file dumps to persist updates to disk. TurboVec introduces incremental state syncing:
pythonfrom turbovec import IdMapIndex import numpy as np index = IdMapIndex(dim=1536, bit_width=4) index.add_with_ids(vectors, ids) # Full file snapshot index.write("index.tvim") # Make small updates index.remove(1002) # Incremental update: Flushes only modified bytes via single fsync index.sync("index.tvim")
The sync(path) call isolates modified byte ranges, issues a single atomic fsync, and guarantees crash safety across arbitrary byte boundaries. Appending or deleting vectors on multi-gigabyte indices completes in milliseconds.
Pragmatic Integration Guide
TurboVec provides native bindings for both Python and Rust environments, along with direct drop-in integration classes for common orchestration frameworks.
Python Native API
Python arrays must be standard float32 2D contiguous arrays.
pythonimport numpy as np from turbovec import TurboQuantIndex, IdMapIndex # Initialize uncalibrated or calibrate on sample vectors = np.random.randn(10000, 1536).astype(np.float32) index = TurboQuantIndex(dim=1536, bit_width=4) index.add(vectors) # Perform SIMD search queries = np.random.randn(5, 1536).astype(np.float32) scores, indices = index.search(queries, k=10) # Stable ID tracking with dynamic filters id_index = IdMapIndex(dim=1536, bit_width=4) id_index.add_with_ids(vectors, np.arange(10000, dtype=np.uint64)) # Filtered Search: Executed directly inside the 32-vector SIMD block allowed_ids = np.array([10, 25, 99, 402], dtype=np.uint64) scores, ids = id_index.search(queries, k=10, allowlist=allowed_ids)
Rust Native API
Add turbovec to your Cargo.toml:
rustuse turbovec::{TurboQuantIndex, IdMapIndex}; fn main() -> Result<(), Box<dyn std.error::Error>> { let mut index = TurboQuantIndex::new(1536, 4)?; // Pass vector slices directly index.add(&vectors); let results = index.search(&queries, 10); // Persistent disk storage index.write("index.tv")?; let loaded = TurboQuantIndex::load("index.tv")?; Ok(()) }
Framework Drop-In Integration
Swap out standard heavy vector stores in your existing application pipeline without rewriting pipeline abstractions:
bashpip install turbovec[langchain] pip install turbovec[llama-index] pip install turbovec[haystack] pip install turbovec[agno]
- LangChain: Replaces
langchain_core.vectorstores.InMemoryVectorStore - LlamaIndex: Replaces
llama_index.core.vector_stores.SimpleVectorStore - Haystack: Replaces
haystack.document_stores.in_memory.InMemoryDocumentStore - Agno: Replaces
agno.vectordb.lancedb.LanceDb
Architecting Air-Gapped Local Vector Pipelines
By combining pure local execution, low memory overhead, and SIMD hardware acceleration, TurboVec provides a lean foundation for high-performance vector retrieval. Eliminating external API calls, background training routines, and managed database dependencies allows you to run millions of vector embeddings locally on a single machine.