Home / Documentation / Architecture

Architecture

A modular design with clear separation of concerns

Table of Contents

Overview

ShibuDB is a lightweight database system designed for high-performance storage and retrieval with vector search capabilities. The architecture follows a modular design pattern with clear separation of concerns.

System Architecture

text 
┌─────────────────────────────────────────────────────────────┐
│                    Client Applications                     │
└─────────────────────┬───────────────────────────────────────┘
                      │ TCP/JSON
┌─────────────────────▼───────────────────────────────────────┐
│                    Network Layer                           │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────┐ │
│  │   TCP Server    │  │  Connection     │  │   Protocol  │ │
│  │                 │  │   Manager       │  │   Handler    │ │
│  └─────────────────┘  └─────────────────┘  └─────────────┘ │
└─────────────────────┬───────────────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────────────┐
│                    Query Engine                            │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────┐ │
│  │   Query Parser  │  │  Query Router   │  │  Response   │ │
│  │                 │  │                 │  │  Builder    │ │
│  └─────────────────┘  └─────────────────┘  └─────────────┘ │
└─────────────────────┬───────────────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────────────┐
│                    Authentication                          │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────┐ │
│  │   Auth Manager  │  │  Role Manager   │  │  Permission │ │
│  │                 │  │                 │  │  Checker    │ │
│  └─────────────────┘  └─────────────────┘  └─────────────┘ │
└─────────────────────┬───────────────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────────────┐
│                    Storage Layer                           │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────┐ │
│  │  Space Manager  │  │  Key-Value      │  │  Vector     │ │
│  │                 │  │  Storage        │  │  Storage    │ │
│  └─────────────────┘  └─────────────────┘  └─────────────┘ │
└─────────────────────┬───────────────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────────────┐
│                    Index Layer                             │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────┐ │
│  │   B-Tree Index  │  │  Vector Index   │  │  Metadata   │ │
│  │                 │  │  (FAISS)        │  │  Index      │ │
│  └─────────────────┘  └─────────────────┘  └─────────────┘ │
└─────────────────────┬───────────────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────────────┐
│                    Persistence Layer                       │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────┐ │
│  │   WAL (Write-   │  │  Data Files     │  │  Checkpoint │ │
│  │   Ahead Log)    │  │                 │  │  Manager    │ │
│  └─────────────────┘  └─────────────────┘  └─────────────┘ │
└─────────────────────┬───────────────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────────────┐
│                    File System                             │
└─────────────────────────────────────────────────────────────┘

Core Components

1. Network Layer

The network layer handles all client communication via TCP connections.

  • TCP Server: listens for incoming connections
  • Connection Manager: manages connection lifecycle
  • Protocol Handler: parses JSON messages and routes to the query engine

2. Query Engine

The query engine is the central coordinator for all database operations.

  • Query Parser: validates and parses incoming queries
  • Query Router: routes queries to appropriate handlers
  • Response Builder: constructs standardized responses

3. Authentication System

The authentication system provides role-based access control.

  • Auth Manager: handles user authentication
  • Role Manager: manages roles and permissions
  • Permission Checker: validates operation permissions

4. Storage Layer

The storage layer manages data persistence and retrieval.

  • Space Manager: manages spaces (namespaces)
  • Key-Value Storage: key-value operations
  • Vector Storage: vector data and similarity search

5. Index Layer

The index layer provides fast data access patterns.

  • B-Tree Index: fast key-value lookups
  • Vector Index (FAISS): similarity search
  • Metadata Index: tracks system metadata

6. Persistence Layer

The persistence layer ensures data durability and recovery.

  • WAL (Write-Ahead Log): optional crash-recovery log (enabled per-space)
  • Data Files: store actual data on disk
  • Checkpointing / fsync: periodic durability mechanisms (engine-dependent)

Data Flow

Write Operation Flow

text 
Client Request → Network Layer → Query Engine → Auth Check → Storage Layer → Index Update → WAL Write → Response

Read Operation Flow

text 
Client Request → Network Layer → Query Engine → Auth Check → Index Lookup → Storage Layer → Response

Vector Search Flow

text 
Client Request → Network Layer → Query Engine → Auth Check → Vector Index → Similarity Search → Response

Data Organization

Spaces

ShibuDB organizes data into logical spaces (similar to databases in traditional systems).

text 
Space: "users"
├── Key: "user:123:profile" → Value: {"name": "John", "email": "john@example.com"}
├── Key: "user:123:settings" → Value: {"theme": "dark", "notifications": true}
└── Key: "user:456:profile" → Value: {"name": "Jane", "email": "jane@example.com"}

Space: "products"
├── Key: "product:789:info" → Value: {"name": "Laptop", "price": 999.99}
└── Vector: "product:789:embedding" → Vector: [0.1, 0.2, 0.3, ...]

Indexes

B-Tree Index: provides fast key-value lookups.

text 
Key: "user:123:profile" → File Offset: 1024
Key: "user:123:settings" → File Offset: 2048
Key: "user:456:profile" → File Offset: 3072

Vector Index (FAISS): enables similarity search.

text 
Space: "products"
├── Index Type: IVF (Inverted File)
├── Vector Dimension: 128
├── Number of Clusters: 100
└── Distance Metric: L2

Storage Format

Data Files

text 
shibudb_data.db
├── Header (Magic Number, Version, Metadata)
├── Data Blocks
│   ├── Block 1: Key-Value Pairs
│   ├── Block 2: Key-Value Pairs
│   └── ...
└── Footer (Checksum, Size)

WAL (Write-Ahead Log)

text 
shibudb_wal.db
├── Log Entry 1: {"op": "PUT", "space": "users", "key": "user:123", "value": "..."}
├── Log Entry 2: {"op": "DELETE", "space": "users", "key": "user:456"}
└── ...