SPTVector Architecture Plan

Purpose-Built Vector Store for SmartPoints Technology

Date: 2026-02-15 Status: DRAFT — Pending Philippe's Review Source Repo: spt (v2.0.3, 98 Rust crates) Target Repo: sptvector (greenfield) Alignment: SmartPoints White Paper, Provisional Patent v7.0, US Patent 8,170,957 B2

1. Strategic Objectives

Objective	Rationale
Rename all packages from `Spt`/`spt`/`spt` to `spt`/`sptvector`/`spt`	IP separation, brand identity for SmartPoints Technology
Change `.spt` file extension to `.spt`	Patent claims reference `.spt` actor containers
Add native Go support	NATS ecosystem is Go-native; Go SDK enables first-class NATS JetStream integration
Quantum-resistant crypto as default	White paper mandates ML-DSA, ML-KEM/Kyber, SLH-DSA for all operations
Actor model for `.spt` files	Each container is an actor in the AIL identity graph with directed messaging
NATS protocol integration	SVS messaging backbone per white paper Section 3.2

2. Scope: What to Fork vs. What to Skip

Core Crates to Port (Priority 1 — ~20 crates)

These are essential for the SPT vector store and actor runtime:

spt Crate	sptvector Name	Purpose
`spt-core`	`sptvector-core`	Vector database engine (HNSW, storage, search)
`spt-node`	`sptvector-node`	Node.js native bindings (NAPI-RS)
`spt-wasm`	`sptvector-wasm`	WebAssembly for edge/browser
`spt-types`	`spt-types`	Segment types, format constants, magic bytes
`spt-wire`	`spt-wire`	Serialization/deserialization (async tokio)
`spt-crypto`	`spt-crypto`	PQC signing, encryption, witness chains
`spt-runtime`	`spt-runtime`	Container loading, MicroVM spawning
`spt-store`	`spt-store`	Persistent storage layer
`spt-manifest`	`spt-manifest`	Manifest/metadata management
`spt-cow`	`spt-cow`	Copy-on-Write branching
`spt-witness`	`spt-witness`	Tamper-evident audit trails
`spt-cli`	`spt-cli`	Command-line interface
`spt-membership`	`spt-membership`	Vector membership filters
`spt-delta`	`spt-delta`	Sparse delta patches
`hnsw-rs`	`spt-hnsw`	HNSW index (or keep as dependency)
`spt-graph`	`sptvector-graph`	Cypher/graph queries for identity graph
`spt-distributed`	`sptvector-distributed`	Raft consensus, replication

New SPT-Specific Crates (to create)

Crate	Purpose
`spt-actor`	Actor runtime — message handling, capability checks, lifecycle
`spt-relationships`	Pre-negotiated relationship management, directional terms
`spt-discovery`	Discovery service for new relationship requests
`spt-noise`	Noise protocol extensions for asymmetric SLA negotiation
`spt-nats`	NATS client integration (Rust side, via `async-nats`)
`spt-identity`	DID/VC management, AIL graph integration

Go Package (new)

Package	Purpose
`go/sptvector`	Go module — NATS-native client for sptvector operations
`go/sptvector/sptfile`	`.spt` file read/write in Go
`go/sptvector/actor`	Actor primitives in Go (receive/send via NATS)
`go/sptvector/crypto`	PQC wrappers (via `circl` or `liboqs-go`)
`go/sptvector/nats`	NATS JetStream integration for actor messaging
`go/sptvector/wire`	Wire protocol (gRPC/protobuf for Rust<->Go interop)

Crates to Defer/Skip (Phase 1)

spt Crate	Reason to Defer
`sptllm`	LLM inference — not core to actor model
`ruqu-*`	Quantum coherence research — premature
`rvdna`	Genomics — domain-specific, not needed
`spt-attention-*` (40+)	Attention mechanisms — research layer
`spt-sona`	Self-optimizing neural — defer to Phase 3
`spt-postgres`	PostgreSQL extension — defer
`tiny-dancer`	FastGRNN — research

3. Renaming Strategy

File Extension

.spt -> .spt throughout all code, tests, docs, CLI flags
Magic bytes: "SPT\x1f\x8b" -> "SPTS\x1f\x8b" (4-byte magic preserved)

Rust Crates

All spt-* -> sptvector-*
All spt-* -> spt-*
All Spt* structs -> Sptvector* or Spt*
All spt_* module names -> spt_*
All SptHnsw*, SptLtra*, Spt* prefixes -> Spt*

npm Packages

@smartpointstech/* -> @sptvector/*
spt -> sptvector

Documentation

All references to "SPT" -> "SPTVector"
All references to "SPF" (legacy name) -> "SPT Format"
All spt.io domain references -> smartpoints.tech

4. `.spt` File Format Specification

Header (64 bytes)

Offset  Size    Field
0x00    4       Magic: "SPTS"
0x04    2       Version (u16)
0x06    4       Segment count (u32)
0x0A    8       Total size (u64)
0x12    32      Root hash (Blake3)
0x32    16      Actor DID hash (truncated)
0x42    2       Flags (discovery_enabled, ephemeral, etc.)

Segment Types (inherited from SPT + new)

rust

pub enum SegmentType {
    // Inherited from SPT (renumbered as needed)
    Vec           = 0x01,  // Vector embeddings
    Index         = 0x02,  // HNSW adjacency
    Manifest      = 0x05,  // Segment directory
    Quant         = 0x06,  // Quantization codebooks
    Meta          = 0x07,  // Key-value metadata
    Crypto        = 0x0C,  // PQC keys, signatures
    Kernel        = 0x0E,  // Embedded runtime (WASM/QEMU)
    Ebpf          = 0x0F,  // eBPF filters
    CowMap        = 0x20,  // COW cluster mapping
    Witness       = 0x0A,  // Audit trails

    // NEW: SPT Actor Segments
    ActorManifest = 0x30,  // DID, VCs, blueprint, discovery flag
    Relationships = 0x31,  // Pre-negotiated directional capabilities
    MessageQueue  = 0x32,  // Inbound directed vector buffer
    WitnessActor  = 0x33,  // Actor-specific message receipt logs
    NoiseState    = 0x34,  // Noise handshake state, PSKs
    IdentityGraph = 0x35,  // Local shard of AIL graph context
}

Crypto (default PQC)

Signatures: ML-DSA-65 (all segments, messages)
Key Exchange: ML-KEM-768 / Kyber (Noise handshakes)
Hash: SHAKE-256 + Blake3 (witness chains)
Stateless sigs: SLH-DSA-128s (long-term identity)
Hybrid fallback: Ed25519 + ML-DSA during transition

5. Actor Model Architecture

Core Trait

rust

#[async_trait]
pub trait SptActor: Send + Sync {
    /// Load from .spt file, verify PQC root, resolve AIL graph
    async fn load(path: &Path, ail: &AilClient) -> Result<Self>
    where Self: Sized;

    /// Handle inbound directed vector message
    async fn handle_message(&mut self, msg: DirectedVector) -> Result<Response>;

    /// Actor DID
    fn did(&self) -> &Did;

    /// Current relationships
    fn relationships(&self) -> &[Relationship];

    /// Request new relationship via discovery
    async fn discover(&self, proposal: RelationshipProposal) -> Result<Relationship>;
}

Message Flow

1. Sender creates DirectedVector (embedding + metadata + PQC signature)
2. NATS publishes to subject: spt.actor.{recipient_did}.in.{relationship_id}
3. Recipient actor receives, validates:
   a. Is sender_did in my RELATIONSHIPS segment?
   b. Does PQC signature verify against sender's pubkey in AIL?
   c. Is message type in negotiated allowed_types?
   d. If all pass -> process. Else -> silent drop + witness log.
4. Response via: spt.actor.{sender_did}.in.{reverse_relationship_id}

NATS Subject Scheme

spt.actor.{did}.in.{rel_id}           — Inbound directed messages
spt.actor.{did}.out.{rel_id}          — Outbound directed messages
spt.discovery.request                  — Relationship proposals
spt.discovery.response.{request_id}    — Discovery responses
spt.governance.{zone_id}.handshake     — Double-Handshake Protocol
spt.witness.{did}.log                  — Audit trail events

6. Go Package Design

Why Go

NATS server and reference client are Go
Go's concurrency model (goroutines) maps naturally to actor patterns
Go is the lingua franca for cloud-native infrastructure (K8s, Docker)
Enables SPT to ship a single binary that embeds NATS + sptvector

Architecture Options

Option	Approach	Pros	Cons
A: CGo/FFI	Go calls Rust via C ABI	Full Rust engine reuse	CGo complexity, cross-compilation pain
B: Wire Protocol	Go speaks gRPC/protobuf to Rust server	Clean separation, independent scaling	Extra network hop, deployment complexity
C: Pure Go	Reimplement core in Go	No FFI, NATS-native, single binary	Duplicate effort, feature drift
D: Hybrid (recommended)	Wire protocol for vector ops + pure Go for actor/NATS/crypto	Best of both worlds	Moderate complexity

Recommendation: Option D (Hybrid)

Go package handles: NATS messaging, actor lifecycle, .spt file I/O, Noise handshakes, PQC via circl (Cloudflare's Go PQC library)
Rust engine handles: HNSW indexing, vector search, quantization, heavy compute
Protocol: gRPC with protobuf over Unix socket or TCP
This gives us a Go binary that can embed NATS and talk to sptvector-core

Go Module Structure

go/
├── go.mod                    # module github.com/SmartPointsTech/sptvector/go
├── cmd/
│   └── sptd/                 # SPT daemon (NATS + sptvector server)
│       └── main.go
├── pkg/
│   ├── sptfile/              # .spt file format (read/write/validate)
│   │   ├── header.go
│   │   ├── segment.go
│   │   └── crypto.go
│   ├── actor/                # Actor runtime
│   │   ├── actor.go          # SptActor interface
│   │   ├── handler.go        # Message handler with capability checks
│   │   └── lifecycle.go      # Load, spawn, shutdown
│   ├── nats/                 # NATS integration
│   │   ├── client.go         # Subject-based pub/sub for actors
│   │   ├── discovery.go      # Discovery service
│   │   └── jetstream.go      # Durable streams for SLA logs
│   ├── noise/                # Noise protocol extensions
│   │   ├── handshake.go      # NK/XK patterns with directional terms
│   │   └── psk.go            # PSK management for relationships
│   ├── crypto/               # PQC wrappers
│   │   ├── mldsa.go          # ML-DSA signatures (via circl)
│   │   ├── mlkem.go          # ML-KEM key exchange
│   │   └── witness.go        # Witness chain (Blake3 + PQC)
│   ├── identity/             # DID/VC management
│   │   ├── did.go
│   │   └── vc.go
│   └── wire/                 # gRPC client to Rust engine
│       ├── proto/
│       │   └── sptvector.proto
│       └── client.go
└── internal/
    └── ail/                  # AIL graph client
        └── graph.go

7. Phased Roadmap

Phase 1: Fork, Rename, Build (Weeks 1-2)

Goal: Working sptvector repo with .spt files, all tests passing.

Task	Details	Estimate
1.1 Fork spt to SmartPointsTech/sptvector	`gh repo fork` or clean copy of selected crates	Day 1
1.2 Select ~20 core crates, remove research crates	Keep core vector engine + SPT format crates	Day 1
1.3 Rename all Rust crates: `spt-` -> `sptvector-`, `spt-` -> `spt-`	Cargo.toml, module names, imports	Days 2-3
1.4 Change magic bytes, file extension `.spt` -> `.spt`	`spt-types`, CLI, all tests	Day 3
1.5 Rename structs/types: `Spt` -> `Spt`, `Spt` -> `Spt`	Global search-replace with verification	Days 3-4
1.6 Update npm packages: `@smartpointstech/` -> `@sptvector/`	package.json, TypeScript bindings	Day 4
1.7 `cargo test --workspace` — fix all breakages	Iterative fix cycle	Days 4-5
1.8 Update README, docs, license headers	Brand as SmartPoints Technology	Day 5
1.9 CI/CD: GitHub Actions for build/test	Adapt from spt workflows	Day 5
1.10 Verify `.spt` file create/read/write works end-to-end	Integration test	Day 5

Exit Criteria: cargo test passes, spt-cli creates and reads .spt files.

Phase 2: PQC Hardening + New Segments (Weeks 3-4)

Goal: .spt files have mandatory PQC and actor-specific segments.

Task	Details	Estimate
2.1 Upgrade `spt-crypto`: ML-DSA-65 as default signature	Replace optional PQC with mandatory	Days 1-2
2.2 Add ML-KEM-768 for key exchange	Integrate `pqcrypto-mlkem` or `liboqs-rust`	Day 2
2.3 Implement new segment types: `ActorManifest`, `Relationships`, `MessageQueue`, `WitnessActor`, `NoiseState`, `IdentityGraph`	Extend `spt-types`	Days 3-5
2.4 Wire serialization for new segments	Extend `spt-wire`	Days 5-6
2.5 Witness chain upgrade: SHAKE-256 + ML-DSA	`spt-witness`	Day 7
2.6 Tests: PQC sign/verify round-trip, new segment I/O	Unit + integration	Days 7-8

Exit Criteria: .spt files contain actor manifests and relationships with PQC.

Phase 3: Actor Runtime + NATS (Weeks 5-7)

Goal: .spt files load as actors, communicate via NATS with capability enforcement.

Task	Details	Estimate
3.1 Implement `SptActor` trait in Rust	`spt-actor` crate	Days 1-3
3.2 Message validation pipeline: graph lookup -> PQC verify -> type check	Core security enforcement	Days 3-5
3.3 NATS integration (`async-nats`): subject-based pub/sub	`spt-nats` crate	Days 5-7
3.4 Discovery service: proposal -> Noise handshake -> relationship creation	`spt-discovery` crate	Days 7-9
3.5 Noise protocol extensions: NK/XK + PSK for directional terms	`spt-noise` crate	Days 9-11
3.6 Integration test: two actors negotiate, exchange directed vectors	End-to-end scenario	Days 11-12

Exit Criteria: Two .spt actors communicate via NATS with PQC-verified directed messaging.

Phase 4: Go Package (Weeks 8-10)

Goal: Go SDK for sptvector with NATS-native actor support.

Task	Details	Estimate
4.1 Go module scaffolding: `go.mod`, package structure	Per design in Section 6	Day 1
4.2 `.spt` file reader/writer in Go	`pkg/sptfile` — header, segments, validation	Days 2-4
4.3 PQC in Go via `circl` (Cloudflare)	ML-DSA, ML-KEM wrappers	Days 4-5
4.4 NATS actor runtime in Go	`pkg/actor` + `pkg/nats`	Days 5-8
4.5 Noise handshake in Go	`pkg/noise` using `flynn/noise` + PQC	Days 8-9
4.6 gRPC wire protocol to Rust engine	protobuf schema + client	Days 9-11
4.7 `sptd` daemon: embeds NATS + sptvector server	`cmd/sptd`	Days 11-13
4.8 Go tests: actor lifecycle, NATS messaging, PQC verification	Full test suite	Days 13-14

Exit Criteria: Go binary creates .spt actors, communicates via embedded NATS.

Phase 5: AIL Integration + Production Hardening (Weeks 11-14)

Goal: Full identity graph, governance, and production readiness.

Task	Details	Estimate
5.1 AIL graph overlay on sptvector's Raft consensus	DAG-based identity storage	Week 11
5.2 Cypher queries for relationship resolution	`sptvector-graph`	Week 11
5.3 Control Zones + Double-Handshake Protocol	Governance sidecar integration	Week 12
5.4 Ephemeral memory: TTL shards + cryptographic zeroization	Cognitive Layer	Week 12
5.5 Benchmarking: <100ms handshakes, 61us queries, 10K+ TPS	Performance validation	Week 13
5.6 Security audit: PQC verification, lateral movement tests	Red team scenarios	Week 13
5.7 Documentation, examples, deployment guides	Production readiness	Week 14

Exit Criteria: Patent prototype complete with benchmark data.

8. Key Technical Decisions

Decision	Choice	Rationale
PQC Library (Rust)	`pqcrypto` + `liboqs-rust`	Most complete NIST PQC coverage
PQC Library (Go)	`cloudflare/circl`	Production-proven, ML-DSA + ML-KEM
Actor Framework (Rust)	Custom on `tokio`	Avoid external dependency for core security primitive
Actor Framework (Go)	Custom on goroutines	NATS-native, no framework overhead
NATS Client (Rust)	`async-nats`	Official async Rust client
NATS Client (Go)	`nats-io/nats.go`	Reference implementation
Noise (Rust)	`snow` crate	Mature Noise implementation
Noise (Go)	`flynn/noise`	Well-maintained Go Noise
Wire Protocol	gRPC + protobuf	Language-agnostic, efficient, well-tooled
Graph Queries	Cypher (retained from spt)	Existing implementation, expressive for identity graphs
File Format	Binary, append-only (evolved from the original format)	Crash-safe, progressive loading

9. Risk Register

Risk	Impact	Likelihood	Mitigation
Renaming breaks subtle internal references	Medium	High	Comprehensive grep + CI
PQC adds unacceptable latency	High	Low	Hardware accel (ANE/CUDA), hybrid fallbacks
Go<->Rust interop complexity	Medium	Medium	Clean gRPC boundary, integration tests
Scope creep from spt's 98 crates	High	High	Strict Phase 1 crate selection, defer research crates
NATS subject explosion with many actors	Medium	Medium	Hierarchical subjects, JetStream partitioning
Noise PQC hybrid patterns not standardized	Low	Medium	Follow NIST + IETF drafts, modular design

10. Success Metrics

Metric	Target	Source
Vector query latency	<61us p50 (match spt)	Benchmark suite
Handshake latency (PQC + Noise)	<100ms end-to-end	Integration test
Throughput	10,000+ TPS	Load test
PQC overhead	<10ms per operation	Crypto benchmarks
`.spt` boot time	<125ms (self-booting actor)	Runtime test
Go test coverage	>80%	CI
Rust test coverage	>80% (core crates)	CI
Lateral movement	Zero in Control Zone tests	Security audit

SPTVector Architecture Plan ​

Purpose-Built Vector Store for SmartPoints Technology ​

1. Strategic Objectives ​

2. Scope: What to Fork vs. What to Skip ​

Core Crates to Port (Priority 1 — ~20 crates) ​

New SPT-Specific Crates (to create) ​

Go Package (new) ​

Crates to Defer/Skip (Phase 1) ​

3. Renaming Strategy ​

File Extension ​

Rust Crates ​

npm Packages ​

Documentation ​

4. .spt File Format Specification ​

Header (64 bytes) ​

Segment Types (inherited from SPT + new) ​

Crypto (default PQC) ​

5. Actor Model Architecture ​

Core Trait ​

Message Flow ​

NATS Subject Scheme ​

6. Go Package Design ​

Why Go ​

Architecture Options ​

Go Module Structure ​

7. Phased Roadmap ​

Phase 1: Fork, Rename, Build (Weeks 1-2) ​

Phase 2: PQC Hardening + New Segments (Weeks 3-4) ​

Phase 3: Actor Runtime + NATS (Weeks 5-7) ​

Phase 4: Go Package (Weeks 8-10) ​

Phase 5: AIL Integration + Production Hardening (Weeks 11-14) ​

8. Key Technical Decisions ​

9. Risk Register ​

10. Success Metrics ​