spt-import

Import tools for migrating data from JSON, CSV, and NumPy formats into SPT stores

Overview

spt-import provides parsers and batch ingestion for migrating vector data from common formats (JSON, CSV/TSV, NumPy .npy) into SPT stores. It includes both a library API and a CLI binary (spt-import) for command-line usage. Records are validated against the target dimension and ingested in configurable batches with progress reporting.

Key Features

• JSON importer -- parse vector arrays with optional metadata from JSON files
• CSV/TSV importer -- columnar vector import with header detection
• NumPy importer -- read .npy format embeddings directly
• Batch ingestion -- configurable batch sizes with progress callbacks
• CLI binary -- spt-import command-line tool with clap-based argument parsing

Dependencies

Key internal dependencies:

• spt-runtime -- SptStore for writing vectors to disk
• spt-types -- shared error and type definitions
• csv -- CSV/TSV parsing
• clap -- CLI argument parsing

Usage

use spt_import::{VectorRecord, import_to_new_store};
use std::path::Path;

let records = vec![VectorRecord {
    id: 1,
    vector: vec![0.1, 0.2, 0.3],
    metadata: vec![],
}];
let result = import_to_new_store(Path::new("out.spt"), 3, &records, 1000, None).unwrap();
println!("Imported {} vectors", result.total_imported);

API Reference

Core Types

Type	Description
VectorRecord	A single vector with id, embedding, and optional metadata
ImportResult	Summary of an import (imported count, rejected count, batches)
ingest_records	Batch-ingest records into an existing SptStore
import_to_new_store	Create a new store and ingest all records
ProgressReporter	Trait for reporting ingestion progress