spt-indexer

Universal indexing pipeline for processing files, emails, and other sources into chunks for vector storage

Overview

spt-indexer provides a document indexing pipeline that parses files and other content sources into text chunks, then stores them in spt-core's VectorDB. Embeddings are not generated by this crate --- the platform bridge layer is responsible for injecting real embeddings via the cognitive engine.

Key Features

• Document Parsing --- pluggable DocumentParser trait with built-in plain text support
• Text Chunking --- configurable chunking strategy for splitting documents into vector-sized pieces
• Pipeline Orchestration --- IndexingPipeline chains parsing, chunking, and storage
• Directory Indexing --- recursively index an entire directory in one call
• Content Type Detection --- identifies content types for parser routing

Dependencies

Key internal dependencies:

• spt-core (storage, hnsw) --- VectorDB for storing indexed chunks

Usage

use spt_indexer::{IndexingPipeline, PlainTextParser, DocumentParser};
use spt_core::{VectorDB, types::DbOptions};
use std::path::Path;

let db = VectorDB::new(DbOptions::default()).unwrap();
let parsers: Vec<Box<dyn DocumentParser>> = vec![Box::new(PlainTextParser)];
let pipeline = IndexingPipeline::new(db, parsers);

let result = pipeline.index_directory(Path::new("./docs")).unwrap();

API Reference

Core Types

Type	Description
IndexingPipeline	Main pipeline orchestrating parse, chunk, store
PipelineConfig	Configuration for chunking and indexing behavior
Chunk	A text chunk with metadata ready for embedding
IndexableItem	Input item to be indexed
IndexedFile	Record of a file that has been indexed
IndexResult	Summary of an indexing run (items processed, chunks created)
ContentType	Enum of recognized content types
IndexerError	Crate error type

Key Traits

Trait	Description
DocumentParser	Parse a file or byte stream into text
PlainTextParser	Built-in parser for plain text files

Key Functions

Function	Description
chunk_text()	Split text into chunks with configurable overlap