Home / Documentation / Vector Engine

Vector Engine

High-performance similarity search powered by FAISS

Table of Contents

Overview

The Vector Engine in ShibuDb provides high-performance similarity search capabilities powered by FAISS (Facebook AI Similarity Search). It enables efficient storage and retrieval of high-dimensional vectors for applications like recommendation systems, image search, natural language processing, and machine learning.

Key Features

  • Multiple Index Types: Support for Flat, HNSW, IVF, and PQ indexes
  • Various Distance Metrics: L2, Inner Product, L1, and more
  • High Performance: Optimized for large-scale vector operations
  • Automatic Training: Index training happens automatically
  • Real-time Search: Fast similarity search with configurable parameters

Architecture

Vector Engine Architecture 
┌─────────────────────────────────────┐
│           Vector Space              │
├─────────────────────────────────────┤
│  FAISS Index (similarity search)    │
├─────────────────────────────────────┤
│  In-Memory Buffer (batch ops)       │
├─────────────────────────────────────┤
│  Write-Ahead Log (durability)       │
├─────────────────────────────────────┤
│  Data Files (persistent storage)    │
└─────────────────────────────────────┘

Vector Space Management

Vector data is organized in spaces with specific index types and distance metrics.

Creating a Vector Space

Create Vector Space 
# Create a basic vector space (128 dimensions, Flat index, L2 metric)
CREATE-SPACE embeddings --engine vector --dimension 128

# Create with specific index type
CREATE-SPACE image_vectors --engine vector --dimension 512 --index-type HNSW32 --metric L2

# Create with custom parameters
CREATE-SPACE text_embeddings --engine vector --dimension 768 --index-type IVF32 --metric InnerProduct

# Create with different HNSW configurations
CREATE-SPACE fast_search --engine vector --dimension 128 --index-type HNSW64 --metric L2
CREATE-SPACE ultra_fast --engine vector --dimension 128 --index-type HNSW256 --metric L2

# Create with different IVF configurations
CREATE-SPACE large_dataset --engine vector --dimension 128 --index-type IVF64 --metric L2
CREATE-SPACE huge_dataset --engine vector --dimension 128 --index-type IVF256 --metric L2

# Create with different PQ configurations
CREATE-SPACE memory_efficient --engine vector --dimension 128 --index-type PQ8 --metric L2
CREATE-SPACE ultra_efficient --engine vector --dimension 128 --index-type PQ32 --metric L2

# Create with composite indices
CREATE-SPACE accurate_large --engine vector --dimension 128 --index-type IVF32,Flat --metric L2
CREATE-SPACE fast_accurate --engine vector --dimension 128 --index-type HNSW64,Flat --metric L2
CREATE-SPACE efficient_accurate --engine vector --dimension 128 --index-type PQ8,Flat --metric L2
CREATE-SPACE balanced_large --engine vector --dimension 128 --index-type IVF64,PQ16 --metric L2
CREATE-SPACE fast_efficient --engine vector --dimension 128 --index-type HNSW128,PQ32 --metric L2

# Create with WAL enabled (for enhanced durability)
CREATE-SPACE durable_embeddings --engine vector --dimension 128 --enable-wal

# Create with WAL disabled (default, for maximum performance)
CREATE-SPACE fast_embeddings --engine vector --dimension 128 --disable-wal

Parameters:

  • --engine vector: Specifies vector engine type
  • --dimension N: Vector dimension (required for vector spaces)
  • --index-type TYPE: FAISS index type (default: Flat)
  • --metric METRIC: Distance metric (default: L2)
  • --enable-wal: Enable Write-Ahead Logging for enhanced durability (default: disabled for vector spaces)
  • --disable-wal: Disable Write-Ahead Logging for maximum performance (default for vector spaces)

Minimum Vector Requirements

Different index types have different minimum vector requirements before search operations become available:

  • Flat: No minimum required (search available immediately)
  • HNSW{n}: No minimum required (search available immediately)
  • IVF{n}: Minimum n vectors required (n = number of clusters)
  • PQ{n}: Minimum 256 vectors required (for training)
  • Composite indices: Follow the higher requirement of their components
Examples 
# HNSW32 - search available immediately
CREATE-SPACE hnsw_space --engine vector --dimension 128 --index-type HNSW32
USE hnsw_space
INSERT-VECTOR 1 1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0
SEARCH-TOPK 1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0 5  # Works immediately

# IVF32 - search available after 32 vectors
CREATE-SPACE ivf_space --engine vector --dimension 128 --index-type IVF32
USE ivf_space
# Need to insert at least 32 vectors before search works
INSERT-VECTOR 1 1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0
# ... insert 31 more vectors ...
INSERT-VECTOR 32 1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0
SEARCH-TOPK 1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0 5  # Now works

# PQ8 - search available after 256 vectors
CREATE-SPACE pq_space --engine vector --dimension 128 --index-type PQ8
USE pq_space
# Need to insert at least 256 vectors before search works
# ... insert 256 vectors ...
SEARCH-TOPK 1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0 5  # Now works

Using a Vector Space

Switch Space 
# Switch to vector space
USE embeddings

# Verify current space (prompt will show current space)
[embeddings]>

WAL Configuration for Vector Spaces

Vector spaces support configurable Write-Ahead Logging (WAL) to balance performance and durability:

Default Behavior (WAL Disabled)

  • Performance: Maximum write performance
  • Durability: Basic durability through data file persistence
  • Recovery: Limited recovery capabilities
  • Use Case: High-performance vector operations where some data loss is acceptable

WAL Enabled

  • Performance: Slower due to additional logging overhead
  • Durability: Enhanced durability with full crash recovery
  • Recovery: Complete recovery from crashes and unexpected shutdowns
  • Use Case: Production systems requiring strong durability guarantees

WAL Configuration Examples

WAL Configuration 
# Create vector space with WAL enabled (enhanced durability)
CREATE-SPACE production_vectors --engine vector --dimension 128 --index-type HNSW32 --enable-wal

# Create vector space with WAL disabled (maximum performance, default)
CREATE-SPACE fast_vectors --engine vector --dimension 128 --index-type HNSW32 --disable-wal

# Create vector space with WAL disabled (explicit, same as default)
CREATE-SPACE performance_vectors --engine vector --dimension 128 --index-type HNSW32

FAISS Index Types

ShibuDb supports various FAISS index types with hardcoded configurations and composite indices for different use cases.

Supported Index Types

Index Types Comparison 
| Index Type | Description | Use Case | Memory | Speed | Min Vectors Required |
|------------|-------------|----------|--------|-------|---------------------|
| `Flat` | Exact search | Small datasets, high accuracy | High | Slow | 0 |
| `HNSW{n}` | Hierarchical Navigable Small World | Fast similarity search | Medium | Fast | 0 |
| `IVF{n}` | Inverted file index | Large datasets | Low | Medium | n |
| `PQ{n}` | Product quantization | Very large datasets | Very Low | Fast | 256 |

Hardcoded Index Variants

HNSW Indices

HNSW{n} where n is a power of 2 from 2 to 256

  • Examples: HNSW2, HNSW4, HNSW8, HNSW16, HNSW32, HNSW64, HNSW128, HNSW256
  • Minimum vectors required: 0 (search enabled immediately)
  • Use case: Fast approximate similarity search with configurable neighbor count

IVF Indices

IVF{n} where n is a power of 2 from 2 to 256

  • Examples: IVF2, IVF4, IVF8, IVF16, IVF32, IVF64, IVF128, IVF256
  • Minimum vectors required: n (number of clusters)
  • Use case: Large dataset indexing with configurable cluster count

PQ Indices

PQ{n} where n is a power of 2 from 2 to 256

  • Examples: PQ2, PQ4, PQ8, PQ16, PQ32, PQ64, PQ128, PQ256
  • Minimum vectors required: 256 (always required for PQ training)
  • Use case: Memory-efficient indexing for very large datasets

Composite Index Types

Composite indices combine multiple index types for enhanced performance and functionality:

Composite Indices 
| Composite Index | Description | Min Vectors Required | Use Case |
|-----------------|-------------|---------------------|----------|
| `IVF{n},Flat` | IVF clustering with exact search refinement | max(n, 1) | Large datasets with high accuracy |
| `HNSW{n},Flat` | HNSW search with exact search refinement | 0 | Fast search with high accuracy |
| `PQ{n},Flat` | PQ quantization with exact search refinement | 256 | Memory-efficient with high accuracy |
| `IVF{n},PQ{m}` | IVF clustering with PQ quantization | max(n, 256) | Very large datasets with balanced performance |
| `HNSW{n},PQ{m}` | HNSW search with PQ quantization | 256 | Fast search with memory efficiency |

Composite Index Examples:

  • IVF32,Flat: 32 clusters with exact search refinement (min 32 vectors)
  • HNSW64,Flat: 64 neighbors with exact search refinement (min 0 vectors)
  • PQ8,Flat: 8-bit quantization with exact search refinement (min 256 vectors)
  • IVF64,PQ16: 64 clusters with 16-bit quantization (min 256 vectors)
  • HNSW128,PQ32: 128 neighbors with 32-bit quantization (min 256 vectors)

1. Flat Index (Exact Search)

Best for: Small datasets (< 1M vectors), high accuracy requirements

Flat Index 
# Create flat index
CREATE-SPACE exact_search --engine vector --dimension 128 --index-type Flat --metric L2
USE exact_search

# Insert vectors
INSERT-VECTOR 1 1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0
INSERT-VECTOR 2 1.1,2.1,3.1,4.1,5.1,6.1,7.1,8.1
INSERT-VECTOR 3 9.0,8.0,7.0,6.0,5.0,4.0,3.0,2.0

# Search for similar vectors
SEARCH-TOPK 1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0 3

Note: Only numerical IDs are supported for vector spaces.

Characteristics:

  • 100% accuracy (exact search)
  • High memory usage
  • Slower for large datasets
  • No training required

2. HNSW Index (Hierarchical Navigable Small World)

Best for: Fast similarity search, medium datasets

HNSW Index 
# Create HNSW index
CREATE-SPACE fast_search --engine vector --dimension 128 --index-type HNSW32 --metric L2
USE fast_search

# Insert vectors
INSERT-VECTOR 1 1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0
INSERT-VECTOR 2 1.1,2.1,3.1,4.1,5.1,6.1,7.1,8.1

# Search with HNSW
SEARCH-TOPK 1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0 5

3. IVF Index (Inverted File)

Best for: Large datasets with balanced performance

IVF Index 
# Create IVF index
CREATE-SPACE large_dataset --engine vector --dimension 128 --index-type IVF32 --metric L2
USE large_dataset

# Insert many vectors
INSERT-VECTOR 1 1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0
INSERT-VECTOR 2 1.1,2.1,3.1,4.1,5.1,6.1,7.1,8.1
# ... insert more vectors

# Search with IVF
SEARCH-TOPK 1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0 10

Basic Vector Operations

Core operations for managing vector data.

INSERT-VECTOR - Store Vectors

Insert Vectors 
# Insert a single vector
INSERT-VECTOR 1 1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0

# Insert with numeric ID
INSERT-VECTOR 1001 1.5,2.5,3.5,4.5,5.5,6.5,7.5,8.5

Format: INSERT-VECTOR <id> <comma-separated-floats>

GET-VECTOR - Retrieve Vectors

Get Vectors 
# Get vector by ID
GET-VECTOR 1

SEARCH-TOPK - Similarity Search

Similarity Search 
# Find top 5 most similar vectors
SEARCH-TOPK 1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0 5

# Find top 1 most similar vector
SEARCH-TOPK 0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8 1

Format: SEARCH-TOPK <query-vector> <k>

RANGE-SEARCH - Radius Search

Radius Search 
# Find all vectors within radius 0.5
RANGE-SEARCH 1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0 0.5

# Find all vectors within radius 1.0
RANGE-SEARCH 0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8 1.0

Format: RANGE-SEARCH <query-vector> <radius>

Advanced Search Operations

Advanced features for efficient data management and querying.

Batch Vector Operations

Batch Operations 
# Insert multiple vectors efficiently
INSERT-VECTOR 1 1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0
INSERT-VECTOR 2 1.1,2.1,3.1,4.1,5.1,6.1,7.1,8.1
INSERT-VECTOR 3 1.2,2.2,3.2,4.2,5.2,6.2,7.2,8.2
INSERT-VECTOR 4 1.3,2.3,3.3,4.3,5.3,6.3,7.3,8.3
INSERT-VECTOR 5 1.4,2.4,3.4,4.4,5.4,6.4,7.4,8.4

# Search for similar vectors
SEARCH-TOPK 1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0 10

Multi-Query Search

Multi-Query Search 
# Search with different query vectors
SEARCH-TOPK 1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0 5
SEARCH-TOPK 9.0,8.0,7.0,6.0,5.0,4.0,3.0,2.0 5
SEARCH-TOPK 0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8 5

Hybrid Search Strategies

Hybrid Search 
# Use range search to find candidates
RANGE-SEARCH 1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0 1.0

# Then use top-k search for ranking
SEARCH-TOPK 1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0 10

Distance Metrics

Different distance metrics for measuring vector similarity.

Supported Metrics

Distance Metrics 
| Metric | Description | Use Case | Formula |
|--------|-------------|----------|---------|
| `L2` | Euclidean distance | General purpose | √(Σ(x₁-y₁)²) |
| `InnerProduct` | Inner product similarity | Cosine similarity (normalized vectors) | Σ(x₁y₁) |
| `L1` | Manhattan distance | Robust to outliers | Σ|x₁-y₁| |
| `Lp` | Lp norm distance | Configurable norm | (Σ|x₁-y₁|ᵖ)^(1/p) |
| `Canberra` | Canberra distance | Weighted differences | Σ|x₁-y₁|/(|x₁|+|y₁|) |
| `BrayCurtis` | Bray-Curtis distance | Ecological data | Σ|x₁-y₁|/Σ(x₁+y₁) |
| `JensenShannon` | Jensen-Shannon divergence | Probability distributions | JS(P||Q) |
| `Linf` | L-infinity distance | Maximum difference | max|x₁-y₁| |

Choosing the Right Metric

L2 (Euclidean Distance) - Default

L2 Metric 
# Good for general-purpose similarity
CREATE-SPACE general --engine vector --dimension 128 --metric L2

InnerProduct (Cosine Similarity)

InnerProduct Metric 
# Good for normalized vectors (embeddings)
CREATE-SPACE embeddings --engine vector --dimension 768 --metric InnerProduct

L1 (Manhattan Distance)

L1 Metric 
# Good for robust similarity (outlier-resistant)
CREATE-SPACE robust --engine vector --dimension 128 --metric L1

Performance Optimization

Tips for optimizing vector search performance.

Index Selection Guidelines

Small Datasets (< 100K vectors)

Small Datasets 
# Use Flat index for exact search
CREATE-SPACE small_dataset --engine vector --dimension 128 --index-type Flat

# Or use HNSW for faster approximate search
CREATE-SPACE small_fast --engine vector --dimension 128 --index-type HNSW16

Medium Datasets (100K - 1M vectors)

Medium Datasets 
# Use HNSW for fast approximate search
CREATE-SPACE medium_dataset --engine vector --dimension 128 --index-type HNSW32

# For higher accuracy, use HNSW64 or HNSW128
CREATE-SPACE medium_accurate --engine vector --dimension 128 --index-type HNSW64

# For very high accuracy with exact refinement
CREATE-SPACE medium_exact --engine vector --dimension 128 --index-type HNSW32,Flat

Large Datasets (1M - 10M vectors)

Large Datasets 
# Use IVF for balanced performance
CREATE-SPACE large_dataset --engine vector --dimension 128 --index-type IVF32

# For larger datasets, use more clusters
CREATE-SPACE large_many_clusters --engine vector --dimension 128 --index-type IVF64

# For high accuracy with exact refinement
CREATE-SPACE large_accurate --engine vector --dimension 128 --index-type IVF32,Flat

Very Large Datasets (> 10M vectors)

Very Large Datasets 
# Use PQ for memory efficiency
CREATE-SPACE huge_dataset --engine vector --dimension 128 --index-type PQ8

# For better accuracy, use higher PQ bits
CREATE-SPACE huge_accurate --engine vector --dimension 128 --index-type PQ16

# For balanced performance with clustering
CREATE-SPACE huge_balanced --engine vector --dimension 128 --index-type IVF64,PQ16

# For fast search with memory efficiency
CREATE-SPACE huge_fast --engine vector --dimension 128 --index-type HNSW128,PQ32

Index Variant Selection Guidelines

HNSW Variants

  • HNSW2-HNSW16: Very fast, lower accuracy, good for real-time applications
  • HNSW32-HNSW64: Balanced speed and accuracy, good for most applications
  • HNSW128-HNSW256: Higher accuracy, slower, good for precision-critical applications

IVF Variants

  • IVF2-IVF16: Good for smaller large datasets (1M-5M vectors)
  • IVF32-IVF64: Good for medium large datasets (5M-20M vectors)
  • IVF128-IVF256: Good for very large datasets (20M+ vectors)

PQ Variants

  • PQ2-PQ8: Very memory efficient, lower accuracy
  • PQ16-PQ32: Balanced memory and accuracy
  • PQ64-PQ256: Higher accuracy, more memory usage

Memory Usage Optimization

Vector Dimension Impact

Dimension Impact 
# Lower dimensions = less memory
CREATE-SPACE low_dim --engine vector --dimension 64 --index-type HNSW32

# Higher dimensions = more memory
CREATE-SPACE high_dim --engine vector --dimension 1024 --index-type HNSW32

Index Type Memory Usage

Memory Usage by Index Type 
# Flat: Highest memory usage
CREATE-SPACE flat_index --engine vector --dimension 128 --index-type Flat

# HNSW: Medium memory usage (varies by neighbor count)
CREATE-SPACE hnsw_small --engine vector --dimension 128 --index-type HNSW16
CREATE-SPACE hnsw_medium --engine vector --dimension 128 --index-type HNSW32
CREATE-SPACE hnsw_large --engine vector --dimension 128 --index-type HNSW64

# IVF: Lower memory usage (varies by cluster count)
CREATE-SPACE ivf_small --engine vector --dimension 128 --index-type IVF16
CREATE-SPACE ivf_medium --engine vector --dimension 128 --index-type IVF32
CREATE-SPACE ivf_large --engine vector --dimension 128 --index-type IVF64

# PQ: Lowest memory usage (varies by quantization bits)
CREATE-SPACE pq_small --engine vector --dimension 128 --index-type PQ4
CREATE-SPACE pq_medium --engine vector --dimension 128 --index-type PQ8
CREATE-SPACE pq_large --engine vector --dimension 128 --index-type PQ16

# Composite indices: Memory usage depends on components
CREATE-SPACE composite_accurate --engine vector --dimension 128 --index-type IVF32,Flat
CREATE-SPACE composite_efficient --engine vector --dimension 128 --index-type HNSW64,PQ16

Search Performance Tuning

K Value Impact

K Value Impact 
# Smaller k = faster search
SEARCH-TOPK 1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0 1

# Larger k = slower search
SEARCH-TOPK 1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0 100

Radius Impact

Radius Impact 
# Smaller radius = fewer results, faster
RANGE-SEARCH 1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0 0.1

# Larger radius = more results, slower
RANGE-SEARCH 1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0 10.0

Best Practices

Recommended practices for using the vector engine effectively.

Index Selection

  • Use Flat for datasets < 1M vectors and high accuracy requirements
  • Use HNSW for fast similarity search with good accuracy
  • Use IVF for large datasets with balanced performance
  • Use PQ for very large datasets with memory constraints

Vector Preparation

  • Normalize vectors for consistent distance calculations
  • Use appropriate dimensions for your use case
  • Preprocess vectors to remove noise and outliers
  • Consider dimensionality reduction for high-dimensional vectors

Search Optimization

  • Start with default parameters and tune based on results
  • Use batch operations for multiple queries
  • Monitor search latency and adjust parameters accordingly
  • Consider using approximate search for real-time applications

Examples and Use Cases

Common use cases and practical examples.

Image Similarity Search

Image Search 
# Create image vectors space
CREATE-SPACE image_search --engine vector --dimension 512 --index-type HNSW32 --metric L2
USE image_search

# Insert image embeddings
INSERT-VECTOR img_001 0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0
INSERT-VECTOR img_002 0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0,0.1
INSERT-VECTOR img_003 0.9,0.8,0.7,0.6,0.5,0.4,0.3,0.2,0.1,0.0

# Search for similar images
SEARCH-TOPK 0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0 5

Text Embedding Search

Text Search 
# Create text embeddings space
CREATE-SPACE text_search --engine vector --dimension 768 --index-type IVF32 --metric InnerProduct
USE text_search

# Insert text embeddings
INSERT-VECTOR doc_001 0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8
INSERT-VECTOR doc_002 0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9
INSERT-VECTOR doc_003 0.8,0.7,0.6,0.5,0.4,0.3,0.2,0.1

# Search for similar documents
SEARCH-TOPK 0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8 10

Recommendation System

Recommendations 
# Create user preferences space
CREATE-SPACE recommendations --engine vector --dimension 128 --index-type HNSW32 --metric L2
USE recommendations

# Insert user preference vectors
INSERT-VECTOR user_001 0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8
INSERT-VECTOR user_002 0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9
INSERT-VECTOR user_003 0.8,0.7,0.6,0.5,0.4,0.3,0.2,0.1

# Find similar users for recommendations
SEARCH-TOPK 0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8 5