spt-mincut-gated-transformer

Ultra low latency transformer inference with mincut-gated coherence control

Overview

spt-mincut-gated-transformer is a deterministic, allocation-free transformer inference engine governed by dynamic minimum-cut coherence signals. It integrates Mixture-of-Depths, early exit, sparse attention, spike-driven inference, and energy-based gating from recent research to achieve bounded p99 latency with explainable gate decisions. Every inference step produces a witness record documenting what state changes were allowed.

Key Features

• Mincut-gated coherence -- graph-based gate controller decides which state writes are permitted
• Early exit -- skip layers when coherence is already sufficient, reducing latency 30-50%
• Flash attention -- optimized attention with int8 and standard forward paths
• Spike-driven scheduling -- event-driven compute that skips idle time steps (87x energy reduction)
• Allocation-free hot path -- arena-based weight management with zero heap allocations after init

Dependencies

Key internal dependencies:

• thiserror -- error types
• serde -- configuration serialization
• getrandom -- optional WASM random support

Usage

use spt_mincut_gated_transformer::{
    MincutGatedTransformer, TransformerConfig, GatePolicy,
    GatePacket, InferInput, InferOutput,
};

let config = TransformerConfig::micro();
let policy = GatePolicy::default();
// let mut transformer = MincutGatedTransformer::new(config, policy, weights)?;

API Reference

Core Types

Type	Description
MincutGatedTransformer	Main inference engine with gate and spike scheduler
TransformerConfig	Model dimensions, layer count, and feature toggles
GatePolicy	Thresholds for coherence gating decisions
GatePacket	Min-cut signals passed into each inference step
InferInput	Input tokens/embeddings plus gate and spike data
InferOutput	Output logits plus witness record
GateController	Decides tier of compute based on coherence signals
CoherenceEarlyExit	Layer-skipping logic based on coherence thresholds
WeightArena	Pre-allocated arena for model weights
AdaptiveKVCache	Three-tier KV cache (hot/warm/archive)