hyperspace-db

• Fractal Knowledge Graphs: Euclidean vectors fail at hierarchies. Our native Hyperbolic engine compresses massive trees (like codebases or taxonomies) into 64-dimensional spaces, reducing RAM usage by 50x without losing semantic context.
• Continuous Reconsolidation: AI agents need to "sleep" and organize memories. With our Fast Upsert Path, CDC Event Streams, and built-in Riemannian Math SDK (Fréchet mean, parallel transport), your agents can continuously shift and prune vectors dynamically.
• Edge-to-Cloud & Offline-First: Drones and humanoid robots can't wait for cloud latency. HyperspaceDB runs directly on Edge hardware, using a Merkle Tree Delta Sync protocol (SyncHandshake, SyncPull, SyncPush) to asynchronously handshake and sync episodic memory chunks with the Cloud when the network is available.
• Serverless at Billion-Scale: HyperspaceDB dynamically unloads idle logic to disk/S3, enabling you to host millions of vectors across thousands of tenants on a single commodity server, acting as the "Neon of Vector Search."

🚀 Core Pillars (v4.2.3)

🕸️ Native Vector-Graph (KAG): Why HyperspaceDB vs The Market?

Traditional vector databases treat documents as isolated points in Euclidean space. In complex regulatory, cybersecurity, aerospace, and medical domains, pure vector search fails to retrieve 77.9% of structural prerequisites and causal containment steps.

HyperspaceDB solves this by unifying HNSW vector indexing and a Property Compressed Sparse Row (CSR) Graph within the same shared-memory address space and L1/L2 cache line.

[!IMPORTANT]

🏆 Empirical Proof Across 12 Production Playbook Collections (16,001 Vectors, 18,402 Edges):

• Causal Context Completeness: 100.0% Recall with KAG vs 22.1% in Pure Vector Search.
• Zero Lost Steps: Pure vector search misses up to 93.8% of containment steps (e.g. in SOC MITRE TTPs and SEC capital rules). KAG deterministically captures every prerequisite.
• Sub-Millisecond Graph Traversal: ~5 µs native CSR bitset traversal at the CPU core, 2.36 ms (p50) end-to-end over HTTP.
• Total Decision Latency: 3.37 ms (p50) combining vector entry match + multi-hop graph reachability.

⚔️ Comprehensive Competitive Matrix

⚡ L0 Hot Tier Caching & WriteBuffer (Real-Time Ingestion)

To maximize read throughput and achieve true real-time ingestion, HyperspaceDB implements a transparent, dual-component memory tier:

graph TD
    Request[Incoming Request] --> Action{Write or Search?}
    
    Action -->|Insert| WB[WriteBuffer in RAM]
    WB --> WAL[WAL Log]
    WB --> IndexQueue[Active Indexing Queue]
    
    Action -->|Search| CacheCheck{L0 Cache Hit?}
    CacheCheck -->|Yes L1/L2 Hit| FastOut[Instant Result ~1 µs - 100 µs]
    CacheCheck -->|No Cache Miss| SearchMerge[Merged Scatter-Gather Search]
    SearchMerge --> HNSWS[HNSW Index search]
    SearchMerge --> WBS[WriteBuffer Rayon Parallel Scan]
    HNSWS --> Deduplicate[Deduplicate by external ID & Re-rank]
    WBS --> Deduplicate

1. Transparent L0 Hot Tier Cache (hyperspace-cache)

The L0 Cache resides directly on the read path, enabling blistering microsecond-level vector retrievals:

• L1 exact-match Cache: Uses a 64-shard thread-safe DashMap for key-value point lookups. Latency is an imperceptible ~1 µs.
• L2 approximate-match Cache: Uses an in-memory HNSW graph (instant-distance) with ArcSwap for atomic lock-free reads during background updates. Latency is ~100 µs.
• Time-To-Live (TTL): Automatic scanning of the __ttl metadata field every 100 ms. Expired keys are invalidated, and tombstones are added to L2.
• Auto-rebuild and Cooldown: When L2 tombstones exceed 30%, a self-healing background task rebuilds the L2 graph, protected by a 5-second cooldown to prevent CPU-rebuild thrashing.
• Metadata Warmup: Rebuilding or starting up automatically warmloads key-value metadata from metadata.forward, ensuring that payloads are instantly available upon cache hits.

2. Real-Time WriteBuffer

A lock-free in-memory buffer ensuring newly inserted vectors are searchable from the very first millisecond:

• Instant Searchability: Newly inserted vectors are placed in the WriteBuffer while HNSW graph links are built in the background.
• Rayon-Parallel Linear Scan: Queries perform multi-threaded linear scans over the WriteBuffer utilizing AVX2/NEON vector instructions, computing Euclidean/Hyperbolic distances and running complex box/cone region filters in < 2.0 ms for up to 50,000 active entries.
• Deduplicated Merging: Results from the main HNSW graph and the WriteBuffer are merged, sorted, and deduplicated by external ID to return highly accurate, real-time lists.
• Snapshot Promotion: Upon HNSW graph integration, background workers evict indexed entries from the WriteBuffer, keeping the RAM footprint minimal.

📊 L0 Cache & Ingestion Performance Results

A rigorous 22-test validation suite confirms the stability and performance of the memory acceleration layer:

• Cache-Hit Ratio: Achieves >98% Recall accuracy on recurrent L1 lookups, with a zero-copy fast path for de-quantized floats.
• rebuild_cooldown Verification: Successfully throttles back-to-back index updates, queuing pending vectors correctly.
• tombstone_trigger Auto-rebuild: Automatically resolves up to 5,000 concurrent deletions, reclaiming memory map segments with zero leaks.
• WriteBuffer linear scan: Demonstrates Rayon multi-threaded lookup times under 1.8 ms for bulk vector batches.

🤖 Target Use Cases

1. Robotics & Autonomous Drones: On-device semantic memory, Hierarchical SLAM, and offline-first edge synchronization.
2. Continuous Learning Systems (AGI): Frameworks doing Riemannian optimization, memory reconsolidation, and Hausdorff-based graph pruning.
3. Enterprise Graph AI: Merging relational logic with semantic proximity for massive multi-scale data analysis (Code ASTs, Medical Taxonomies).
4. High-Load RAG & SaaS: Traditional search, but significantly cheaper to operate due to Serverless Idle Eviction and multi-tenant isolation.

⚡ 1 Million Vectors Benchmark (v3.0)

We pushed HyperspaceDB v3.0 to the limit with a 1 Million Vector Dataset. The results define a new standard for performance and efficiency.

🏆 Hyperbolic Efficiency (Poincaré 64d)

When using the native Hyperbolic (Poincaré) metric, HyperspaceDB achieves unparalleled throughput by reducing dimensionality (64d) while preserving semantic structure achievable only with 1024d in Euclidean space.

⚔️ Euclidean Performance (1024d)

Even in standard Euclidean mode, HyperspaceDB outperforms competitors on standard hardware.

📉 Zero Degradation Architecture

While other databases slow down as data grows, HyperspaceDB maintains consistent throughput.

• Weaviate degraded from 738 QPS -> 491 QPS (-33%).
• Milvus fluctuated between 6k and 11k QPS.
• HyperspaceDB held steady at ~156k QPS (Hyperbolic) and ~17.8k QPS (Euclidean).

💾 50% Less Disk Usage

Store more, pay less. HyperspaceDB's 1-bit quantization and efficient storage engine require half the disk space of Milvus for the exact same dataset.

• HyperspaceDB: 9.0 GB (Euclidean) / 0.7 GB (Hyperbolic)
• Milvus: 18.5 GB

Benchmark Config: 1M Vectors, 1024 Dimensions (Euclidean) vs 64 Dimensions (Hyperbolic), Batch Size 1000.

⚡ v4.0.0 Grand Benchmark: Thread-per-Core & Direct I/O

HyperspaceDB v4.0.0 introduces a high-performance Thread-per-Core (Shared-Nothing) architecture with Hardware Direct I/O (4096-byte DMA alignment with O_DIRECT/F_NOCACHE), hierarchical Spatial AABB segment pruning, and autonomous startup migration.

🏆 Empirical VectorDBBench Verification (v3.x vs v4.0.0)

📌 Detailed comparative data available in BENCHMARK_V3_VS_V4_REPORT.md and BENCHMARK_GRAND_REPORT.md.

🗜️ Multi-Tier Vector Quantization Suite (v4.0.0)

HyperspaceDB v4.0.0 supports 7 quantization precision levels verified empirically across 20,000 unit-normalized 1024D embeddings (see BENCHMARK_QUANTIZATION_SWEEP.md):

🔒 Security

• API Keys: Secure endpoints with HYPERSPACE_API_KEY environment variable.
• Header: Clients must send x-api-key: <key>.
• Zero-Knowledge: Server stores only SHA-256 hash of the key in memory.

🤝 Federated Clustering & P2P Swarm (v3.0)

HyperspaceDB implements two distinct clustering architectures designed for both high availability in the Cloud and dynamic Edge-to-Edge discovery for robotics swarms.

1. Leader-Follower Replication (Cloud / High Availability)

• Node Identity: Each node generates a unique UUID (node_id) and maintains a Lamport logical clock.
• Leader: Handles Writes (Coordinator). Streams WAL events. Manages Cluster Topology.
• Follower: Read-Only replica. Can be promoted to Leader.

2. Edge-to-Edge Gossip Swarm (Robotics / Local-First)

Designed for robotic swarms without a central Leader. Uses raw UDP multicasting to form a decentralized, self-healing network.

• Zero-Dependency: Built on raw tokio::net::UdpSocket (no heavy libp2p dependencies).
• Heartbeats: Nodes broadcast state via UDP. Disconnected nodes are automatically evicted after a TTL interval.
• Auto-Discovery: Discover peers and instantly initiate a Delta Sync handshake to resolve diverging graphs.
• Enable: Set HS_GOSSIP_PEERS (e.g. 192.168.1.10:7946) or HS_GOSSIP_PORT to join the swarm.

Data Synchronization (Edge-to-Cloud Delta Sync)

HyperspaceDB uses a 256-bucket Merkle Tree for efficient data drift detection, ideal for WASM/Edge targets updating offline:

• Granular Hashing: Each collection is partitioned into 256 buckets (by vector ID % 256)
• XOR Rolling Hash: Each bucket maintains an incremental hash of its vectors
• Fast Diffing: Compare bucket hashes to identify which partition is out of sync
• Bandwidth Optimization: Sync only affected buckets instead of full collection

WASM Sync Example

When your robot or web client comes back online, initiating a Sync is mathematically minimal:

// 1. Handshake: Send local 256 bucket hashes
const { diffBuckets } = await client.syncHandshake(collection, localBuckets);

if (diffBuckets.length > 0) {
    // 2. Pull only the modified/missing buckets from Cloud
    const stream = client.syncPull(collection, diffBuckets);
    stream.on('data', (vectorData) => applyLocal(vectorData));
    
    // 3. Push local offline edits back to Cloud
    client.syncPush(localEditsQueue);
}

Digest API

# HTTP
GET /api/collections/{name}/digest

# gRPC
rpc GetDigest(DigestRequest) returns (DigestResponse)

Response includes:

• logical_clock: Lamport timestamp
• state_hash: Root hash (XOR of all buckets)
• buckets: Array of 256 bucket hashes
• count: Total vector count

Cluster Topology API

View the logic state of the cluster via HTTP:

# Get Replication State
curl http://localhost:50050/api/cluster/status

# Get Decentralized Swarm Peers (Gossip)
curl http://localhost:50050/api/swarm/peers

{
  "gossip_enabled": true,
  "peer_count": 2,
  "peers": [
    {
       "node_id": "e8...0e",
       "role": "Leader",
       "addr": "192.168.1.20:50050",
       "logical_clock": 42,
       "healthy": true
    }
  ]
}

Starting a Cluster

# Start Leader
./hyperspace-server --port 50051 --role leader

# Start Follower
./hyperspace-server --port 50052 --role follower --leader http://127.0.0.1:50051

🕸️ WebAssembly (WASM) Support

HyperspaceDB can run directly in the browser via WebAssembly, enabling Local-First AI applications with zero network latency.

• Zero Latency: Search runs in-memory on the client.
• Privacy: Data never leaves the device.
• Optimized: Uses RAMVectorStore backend for browser environments.

👉 Read the WASM Documentation

⚖️ Heterogeneous Tribunal Framework (Tribunal Router)

HyperspaceDB natively supports the confrontational model of LLM routing (Architect vs. Tribunal) directly on the vector graph.

Using the Cognitive Math SDK and the Graph Traversal API, the SDK calculates a Geometric Trust Score for any LLM claim by verifying the logical path length between concepts in the latent hyperbolic space.

If the geodesic distance (hops) between "Claim A" and "Claim B" on the graph is too large (or disconnected), the Trust Score drops to 0.0 (Hallucination).

from hyperspace.agents import TribunalContext

tribunal = TribunalContext(client, collection_name="knowledge_graph")

# Evaluates structural graph distance between concepts. 
# 1.0 = Truth (Identical), 0.0 = Hallucination (Disconnected)
score = tribunal.evaluate_claim(concept_a_id=12, concept_b_id=45)

🧠 Hybrid Search (BM25 + RRF)

Combine the power of Hyperbolic Embeddings with advanced BM25 lexical ranking.

# Search for semantic similarity AND keyword match (e.g. "iPhone 15")
results = client.search(
    vector=[0.1]*8, 
    top_k=5, 
    hybrid_query="iPhone 15", 
    hybrid_alpha=0.7,  # 70% vector, 30% lexical
    bm25={
        "method": "bm25plus",
        "language": "english"
    }
)

📊 Quantization Modes

All quantization is configured per-collection at creation time via HS_QUANTIZATION_LEVEL:

⚠️ Metric Compatibility Notice:

• Cosine & L2: Compatible with ALL quantization modes (none, medium, medium_plus, turbo, opq, pq, extreme).
• Poincaré & Lorentz: Only compatible with none and medium (Scalar SQ8). Product Quantization (pq/opq) and 1-bit (extreme) assume Euclidean metrics and orthonormal rotations that violate conformal Poincaré boundary conditions ($|x| < 1$) and Lorentzian pseudo-Riemannian signature $(-1, +1, \dots, +1)$.
• Hybrid (801D = 33D Lorentz + 768D Euclidean): Uses AsymmetricHybrid, where the 33D Lorentz head is kept unquantized/FP64 while the 768D Euclidean tail can be compressed with turbo, opq, or extreme.

The default scalar mode uses a ScaNN-inspired anisotropic loss $L = |e_\parallel|^2 + t_w \cdot |e_\perp|^2$ ($t_w=10$), which penalizes directional error more than magnitude error, improving Recall@10 by +5.3% (Cosine) and +3.8% (L2) versus isotropic SQ8.

Each geometry has its own independent backend, now featuring native support for Qwen3-Embedding (0.6B) and YAR v5 models:

Example — Config for Qwen3 (1024d) and YAR v5 (128d):

HYPERSPACE_EMBED=true

# Cosine via Qwen3 (HuggingFace)
HS_EMBED_COSINE_PROVIDER=huggingface
HS_EMBED_COSINE_HF_MODEL_ID=onnx-community/Qwen3-Embedding-0.6B-ONNX
HS_EMBED_COSINE_DIM=1024
HS_EMBED_COSINE_CHUNK_SIZE=4096   # 32K context window

# Poincaré via YAR Labs v5
HS_EMBED_POINCARE_PROVIDER=huggingface
HS_EMBED_POINCARE_HF_MODEL_ID=YARlabs/v5_Embedding_0.5B
HS_EMBED_POINCARE_DIM=128

Direct Search from SDK:

# Server-side text-to-vector search (v3.0.1)
results = client.search_text("Find similar robotics docs", top_k=5)

For details see embeddings.md.

📉 Binary Quantization (1-bit)

Use Binary quantization mode to compress vectors by 32x-64x (vs f32/f64). Ideal for large-scale datasets where memory is the bottleneck.

🛠 Architecture

HyperspaceDB strictly follows a Command-Query Separation (CQS) pattern:

graph TD
    Client["Client (gRPC)"] -->|Insert| S["Server Service"]
    Client -->|Search| S
    
    subgraph Persistence_Layer ["Persistence Layer"]
        S -->|"1. Append"| WAL["Write-Ahead Log"]
        S -->|"2. Append"| VS["Vector Store (mmap)"]
    end
    
    subgraph Indexing_Layer ["Indexing Layer"]
        S -->|"3. Send ID"| Q["Async Queue"]
        Q -->|Pop| W["Indexer Worker"]
        W -->|Update| HNSW["HNSW Graph (RAM)"]
    end

1. Transport: gRPC/Tonic server accepts requests (Insert/Search).
2. Persistence: Data is immediately persisted to WAL and segmented Mmap storage.
3. Indexing: A background worker updates the HNSW graph asynchronously.
4. Recovery: Graph snapshots (via rkyv zero-copy) ensure near-instant restarts.

👉 *For deep dive, read ARCHITECTURE.md*

🛠 Operations & Maintenance

Queue Monitoring

Check ingestion backlog via API or collections stats:

{
  "count": 150000,
  "indexing_queue": 45  // Items pending index insertion
}

Rebuild Index (Defragmentation)

Trigger a graph rebuild to optimize layout and remove deleted nodes:

curl -X POST http://localhost:50050/api/collections/my_col/rebuild

Memory Management (Jemalloc)

HyperspaceDB uses Jemalloc for efficient memory allocation. You can tune its behavior via the MALLOC_CONF environment variable:

• Low RAM (Aggressive Release): MALLOC_CONF=background_thread:true,dirty_decay_ms:0,muzzy_decay_ms:0 - Releases unused memory immediately to OS. Increases CPU usage slightly.
• Balanced (Default): MALLOC_CONF=background_thread:true,dirty_decay_ms:5000,muzzy_decay_ms:5000 - Keeps some memory for reuse, balanced performance.

To create a manual memory vacuum request (e.g., after large deletions):

curl -X POST http://localhost:50050/api/admin/vacuum

💻 System Requirements

HyperspaceDB is designed to run efficiently on commodity hardware, but specific instruction sets are required for hardware acceleration.

CPU (Critical)

• Architecture: x86-64 or ARM64.
• Instructions:
• x86-64: Must support AVX2 (Intel Haswell 2013+ or AMD Zen 2017+).
• ARM64: Must support NEON (Standard on Apple Silicon M1/M2/M3 and AWS Graviton).
• Note: The database will crash or fail to compile on CPUs without SIMD support.

Storage (I/O)

• Disk Type: SSD / NVMe is highly recommended.
• HyperspaceDB uses mmap for random access. Spinning HDDs (mechanical drives) will severely degrade search latency due to seek times.

Memory (RAM)

• Minimum: 512 MB.
• Recommended: Enough RAM to cache the "hot" part of your dataset.
• Thanks to ScalarI8 quantization, 1 Million vectors (8-dim) take only ~12 MB of disk space. Even large datasets fit easily into RAM.
• If the dataset exceeds RAM, the OS will swap pages to disk (performance will depend on SSD speed).

Operating System

• Linux: Kernel 5.10+ recommended (for efficient memory mapping).
• macOS: 12.0+ (fully supported).
• Windows: Supported via WSL2 (native Windows build is experimental).

🏃 Quick Start

1. Build and Start Server

Make sure you have just and nightly rust installed.

# Build release binary
cargo build --release

# Run server (Default HTTP port: 50050)
./target/release/hyperspace-server

# Or with custom ports
./target/release/hyperspace-server --port 50051 --http-port 50050

2. Access Web Dashboard

The built-in React Dashboard provides real-time monitoring and management:

http://localhost:50050

Dashboard Features:

• 📊 System Overview: Real-time metrics (RAM, CPU, vector count)
• 🗂️ Collections Manager: Create, delete, and inspect collections
• �️ Cluster Nodes: Visualize node topology and replication status
• �🔍 Data Explorer: View recent vectors and test search queries
• ⚙️ Settings: Integration snippets (Python, cURL) and live logs
• 📈 Graph Explorer: Visualize HNSW graph structure, Poincaré manifolds, and agent momentum trajectories in real-time.

Authentication: If HYPERSPACE_API_KEY is set, you'll be prompted to enter it on first visit. The key is stored in localStorage for subsequent sessions.

Build Dashboard from Source:

cd dashboard
npm install
npm run build
# Assets are embedded in Rust binary via rust-embed

3. Launch TUI Monitor

Open a new terminal to monitor the database:

./target/release/hyperspace-cli

3. Use Python SDK

pip install hyperspacedb==3.1.2

from hyperspace import HyperspaceClient

# Connect to local instance
client = HyperspaceClient()

# Create a collection with Schema-Driven Cascade (128D Poincaré + MRL)
client.create_collection(
    name="world_model", 
    schema={
        "components": [
            {"name": "primary", "metric": "poincare", "full_dimension": 128, "weight": 1.0}
        ],
        "cascade_pipeline": [
            {"component_name": "primary", "cutoff_dimension": 32, "store_in_ram": True, "rerank_top_k": 100}
        ]
    }
)

# Insert text document (you can provide your own embeddings)
client.insert(id=1, collection="world_model", document="Hyperspace is autonomous.")

# Search 
results = client.search(query_text="autonomous engine", top_k=5)
print(results)

🏘️ Collections Management

HyperspaceDB v1.1+ supports Multi-Tenancy via Collections. Each collection is an independent vector index with its own dimension and metric.

Via Web Dashboard

Access the dashboard at http://localhost:50050:

1. Create Collection: Enter name, select dimension (8D, 768D, 1024D, 1536D), click Create
2. View Collections: See all active collections with their stats
3. Delete Collection: Remove collections you no longer need

Via gRPC/SDK

from hyperspace import HyperspaceClient

client = HyperspaceClient()

# Create a new collection with Schema
client.create_collection(
    name="my_vectors", 
    schema={
        "components": [
            {"name": "main", "metric": "cosine", "full_dimension": 1536}
        ]
    }
)

# Insert into specific collection
client.insert(id=1, document="...", collection="my_vectors")

# Search in specific collection
results = client.search(query_text="...", collection="my_vectors", top_k=5)

# List all collections
collections = client.list_collections()

# Delete a collection
client.delete_collection("my_vectors")

Note: If no collection is specified, operations default to the "default" collection.

🏙️ SaaS & Multi-Tenancy (v2.0)

HyperspaceDB is built for SaaS. Isolate thousands of users on a single node.

1. User Isolation

Data is logically separated by user_id. Each user sees only their own collections.

How to use: Pass the x-hyperspace-user-id header in your requests.

curl -H "x-hyperspace-user-id: tenant_123" http://localhost:50050/api/collections

2. Billing API

Admins can query usage statistics for all tenants:

curl -H "x-hyperspace-user-id: admin" http://localhost:50050/api/admin/usage

⚙️ Configuration & Presets

HyperspaceDB v1.2 introduces flexible configuration presets to support both Scientific (Hyperbolic) and Classic (Euclidean) use cases.

Configure these via .env file or environment variables:

[!IMPORTANT] Since v3.1.0, HS_DIMENSION and HS_METRIC are deprecated. Use CollectionSchema via SDK or CLI for granular control.

🔄 Инструкция по автоматической миграции (v3.x → v4.0.0 Migration Guide)

В версии v4.0.0 реализована раздельная архитектура хранения данных:

• data/: Папка legacy-данных v3.x.
• data_v4/: Папка нового формата v4.0.0 (Direct I/O, Thread-per-core, Spatial AABB).

Как работает миграция (Zero-Config Upgrade):

1. Автоматический запуск: При первом старте HyperspaceDB v4.0.0 сервер автоматически обнаруживает папку data/.
2. Бесшовная конвертация: Все найденные v3-коллекции конвертируются в формат v4.0.0 и записываются в data_v4/{collection_name}/:
   • Векторы выравниваются по границам страниц 4096 байт для прямого DMA-доступа (Direct I/O).
   • Автоматически рассчитываются и сохраняются пространственные сегменты Spatial AABB.
   • Таблица безопасности и прав доступа (data/security.db) автоматически переносится в data_v4/security.db.
3. Безопасное архивирование: Исходные v3 файлы атомарно перемещаются в data/legacy_v3_migrated/{collection_name}/ с созданием манифеста целостности .migration_v4_complete.json (с таймштампом и количеством векторов).
4. Повторные / ручные переносы: Если в будущем пользователь скопирует старые v3 файлы или целую коллекцию в data/ (или data/legacy_v3/), система при следующем перезапуске автоматически обнаружит их и смигрирует в data_v4/.

Настройка путей через переменные окружения:

• HS_DATA_DIR: Путь к активному v4 хранилищу (по умолчанию: data_v4).
• HS_LEGACY_V3_DIR: Путь к legacy v3 хранилищу для поиска миграций (по умолчанию: data).

Проверка успешности миграции:

• В логах сервера отображаются записи:

  🔄 [MigrationEngine] Found 1 legacy v3.x collection(s) to migrate to v4.0.0 (from data to data_v4)
  🚀 [MigrationEngine] Migrating 'my_collection' from data/my_collection to v4 format in data_v4...
  ✅ [MigrationEngine] Successfully migrated 'my_collection' (50000 vectors in 142 ms)
  📦 [MigrationEngine] Archived legacy files to data/legacy_v3_migrated/my_collection
  

• В папке каждой коллекции data_v4/{collection}/ создается файл .migration_v4_complete.json.

🎯 Supported Presets

1. Classic RAG (Default) Optimized for standard embeddings from OpenAI, Cohere, Voyage, etc.

• Metric: cosine (Cosine Similarity) - recommended for OpenAI/BGE embeddings
• Dimensions: 1024, 1536, 2048
• Note: cosine mode automatically normalizes vectors on insert/search (with zero-copy fast path for already normalized vectors) and uses HNSW-friendly squared L2 ranking internally. For magnitude-sensitive workloads, use l2 with HS_QUANTIZATION_LEVEL=none.

2. Scientific / Hyperbolic Optimized for hierarchical data, graph embeddings, and low-dimensional efficiency.

• Metric: poincare
• Dimensions: 16, 32, 64, 128 (Common: 64)
• Requirement: Input vectors must strictly satisfy ||x|| < 1.0 (Poincaré ball constraint). Server will reject invalid vectors.

📊 Best Practices

HyperspaceDB follows the microservices philosophy: One Index per Instance. To manage multiple datasets, we recommend deploying separate Docker containers or using Metadata Filtering for logical separation within a single index.

1. Vector Dimensionality

• Recommendation: Choose dimensions matching your embedding model.
• Support: Native support for 1024 (BGE-M3), 1536 (OpenAI), 768 (BERT), and 8 (Hyperbolic).
• Reason: HyperspaceDB now uses Const Generics to optimize for specific dimensions at compile time.

2. Quantization Strategy

• Mode: Use Binary quantization for maximum memory savings.
• Trade-off: Binary mode reduces precision but compresses vectors by 32x-64x compared to floating-point.
• When to use: Large-scale datasets where memory is the bottleneck.

3. Indexing Parameters

• ef_construction: Controls index build time vs. search quality. Higher values = better recall but slower indexing.
• ef_search: Controls search time vs. recall. Higher values = better recall but slower search.
• Tuning: Adjust via gRPC without restarting the server.

4. Hybrid Search

• Enable: Use hybrid_query parameter in search requests.
• Tuning: Adjust hybrid_alpha (0.0 to 1.0) to balance semantic similarity and keyword matching.

⚡ RAG Performance Benchmarks (v3.1)

We conducted a rigorous evaluation of HyperspaceDB v3.1 against popular vector databases (Qdrant, ChromaDB, Milvus, Weaviate) across 15 distinct RAGBench datasets (ranging from 1,000 to 100,000 documents).

The results highlight HyperspaceDB's Lock-Free ArcSwap Architecture and optimized cache layers.

📋 Benchmark: HYBRID Search (Lexical BM25 + Semantic)

Evaluating retrieval accuracy (Recall@10, Context Precision) and throughput (QPS, Latency) averaged across 15 domains.

📊 Index Size: 1,000 Documents (HYBRID 1000x100)

📊 Index Size: 100,000 Documents (HYBRID 100000x1000)

🏆 Concurrency & Stress Testing

Measured on the CovidQA dataset under concurrent search queries.

👥 100 Parallel Clients

👥 1,000 Parallel Clients

📉 Why is HyperspaceDB so fast?

1. Lock-Free Architecture: Readers never block readers using ArcSwap and lock-free atomic references.
2. L0 Two-Level Caching: Sharded L1 DashMap (~1 µs lookups) coupled with a background-rebuilt L2 HNSW fallback graph (~100 µs lookups).
3. Anisotropic Quantization: Custom ScaNN-inspired scalar I8 loss quantization maintains high Recall while saving 8x RAM.
4. Zero-Copy Serialization: Fast memory mapping (mmap) of segment files ensures immediate startup recovery without parsing overhead.

RAG Benchmarks Hardware Config: Apple M4 Pro (64GB RAM), NVMe SSD storage.

🐳 Deployment

Official Docker Images

HyperspaceDB publishes three specialized Docker distribution tags to match different deployment architectures:

Quick Start

1. Run Standard Engine with PDK Plugins

docker run -d \
  --name hyperspace \
  -p 50051:50051 \
  -p 50050:50050 \
  -v $(pwd)/hs_data:/app/data \
  -v $(pwd)/plugins:/app/plugins \
  glukhota/hyperspace-db:latest

2. Run All-in-One Engine (With Built-In ONNX Embedding Model)

docker run -d \
  --name hyperspace-embed \
  -p 50051:50051 \
  -p 50050:50050 \
  -v $(pwd)/hs_data:/app/data \
  -v $(pwd)/plugins:/app/plugins \
  glukhota/hyperspace-db:latest-embed

Access the Web Dashboard and Plugins Workbench at http://localhost:50050 and gRPC at localhost:50051.

Docker Compose

Run the full stack with mounted data and plugins directories:

docker-compose up -d

🔒 Security & SOC 2 Compliance

HyperspaceDB comes with advanced security controls out-of-the-box, ensuring compatibility with standard enterprise compliance requirements (e.g. SOC 2, ISO 27001):

• Mutual TLS (mTLS): Built-in TLS and client verification support for both gRPC (port 50051) and HTTP Dashboard/API (port 50050). Configure via .env:
  • HS_TLS_CERT=/path/to/server.crt
  • HS_TLS_KEY=/path/to/server.key
  • HS_TLS_CA=/path/to/ca.crt (enables mutual TLS verification)
• Role-Based Access Control (RBAC): Fine-grained user role permissions (Admin, ReadWrite, ReadOnly) managed via transaction-safe redb database.
• Cross-Tenant Collection Sharing: Securely share collections between tenants with read-only/read-write privileges using Namespaced URI references (owner/collection_name).
• Structured Audit Logs: Structured JSON audit logging (HS_AUDIT_LOG_LEVEL=full) written to stdout or a file, with tenant-isolated logs streamed securely to authenticated users.

[!TIP] For SOC 2 compliance, we recommend deploying HyperspaceDB on LUKS-encrypted drives or encrypted cloud block volumes within isolated Virtual Private Clouds (VPC).

🔌 AI Integrations

HyperspaceDB is ready for the modern AI stack with official high-level integrations:

🛠️ Model Context Protocol (MCP)

Connect HyperspaceDB directly to Claude Desktop, Cursor, Windsurf, Antigravity, or any MCP-compliant agent to give them persistent cognitive spatial memory.

🧠 1. Zero-Code Cognitive Agent Memory (mcp-hyperspace-memory)

Drop-in cognitive agent memory server with 8 specialized memory tools (memory_remember, memory_recall, memory_forget, memory_update, memory_list_sessions, memory_explore_hierarchy, memory_consolidate, memory_verify_claim):

# Start Memory MCP server via npx
npx mcp-hyperspace-memory@latest

Option A: Managed Cloud Memory (Zero Infrastructure)

{
  "mcpServers": {
    "hyperspace-memory": {
      "command": "npx",
      "args": ["-y", "mcp-hyperspace-memory@latest"],
      "env": {
        "HYPERSPACE_API_KEY": "sk_YOUR_YAR_API_KEY",
        "MEMORY_COLLECTION": "agent_memory"
      }
    }
  }
}

Option B: Local DB Instance + Cloud v5 Embedding API Key (Default Auth)

Run glukhota/hyperspace-db:latest locally on your machine. Your vector graph and memory payloads stay 100% private on localhost, while text is vectorized via the state-of-the-art continuous v5_Embedding_801 model in the cloud:

{
  "mcpServers": {
    "hyperspace-memory": {
      "command": "npx",
      "args": ["-y", "mcp-hyperspace-memory@latest"],
      "env": {
        "HYPERSPACE_HOST": "localhost:50051",
        "HYPERSPACE_API_KEY": "sk_YOUR_YAR_API_KEY",
        "MEMORY_COLLECTION": "agent_memory"
      }
    }
  }
}

Option C: Local DB with Custom Password + Cloud v5 Embedding API Key (Two Keys)

If your local .env has a custom security password (HYPERSPACE_API_KEY=my_local_secret), provide two separate keys:

{
  "mcpServers": {
    "hyperspace-memory": {
      "command": "npx",
      "args": ["-y", "mcp-hyperspace-memory@latest"],
      "env": {
        "HYPERSPACE_HOST": "localhost:50051",
        "HYPERSPACE_API_KEY": "my_local_secret",
        "YAR_API_KEY": "sk_YOUR_YAR_API_KEY",
        "MEMORY_COLLECTION": "agent_memory"
      }
    }
  }
}

• HYPERSPACE_API_KEY: Authenticates to local gRPC database (localhost:50051).
• YAR_API_KEY (or legacy CDE_API_KEY): Authenticates to cloud https://the.yar.ink/v1/embeddings (v5_Embedding_801). (Get your sk_ API key at yar.ink/dashboard).

⚡ 2. Full Vector Database & Graph MCP Server (mcp-hyperspacedb)

Complete database administration, graph traversal, and cognitive analytics:

# Start Full Database MCP server via npx
npx mcp-hyperspacedb@latest

Available Tools (30+):

• hyperspace_search_text / hyperspace_search_wasserstein: Semantic & Optimal Transport search.
• hyperspace_insert_text / hyperspace_delete_points / hyperspace_get_points: Data CRUD.
• hyperspace_create_collection / hyperspace_delete_collection: Collection management.
• hyperspace_graph_traverse / hyperspace_explore_graph / hyperspace_get_neighbors: Graph navigation.
• hyperspace_get_subsumption_tree / hyperspace_get_concept_parents: Hierarchy traversal.
• hyperspace_analyze_thought_stability / hyperspace_predict_momentum / hyperspace_get_trust_score: Cognitive AI.
• hyperspace_verify_generation: Tribunal geometric hallucination verification.
• hyperspace_query_planner: Natural-language sub-query decomposition and planned retrieval.
• hyperspace_analyze_geometry: Gromov Delta → metric recommendation (cosine/l2/lorentz/poincare/hybrid).
• hyperspace_rebuild_index / hyperspace_vacuum / hyperspace_cache_*: System maintenance.

📦 SDKs (v4.x Milestone)

Official 1st-party drivers with full Functional Parity, Tribunal Verification (verify_generation), Delta Sync, and Cognitive Math:

🧠 AI Agent Skills (hyperspacedb-skills) are structured SKILL.md files that teach Cursor, Claude Code, Windsurf, Antigravity, and custom agents how to correctly use HyperspaceDB — covering core CRUD, graph traversal, cognitive AI (Lyapunov/Koopman), Tribunal verification, query planning, DePIN node economics, and the full 30+ MCP tool reference. Install once, all your agents understand HyperspaceDB automatically.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

Copyright © 2026 YARlabs