spt-synthlang

Prompt compression engine for LLM token reduction using SynthLang patterns

Overview

spt-synthlang provides a modular compression pipeline that reduces token usage when communicating with LLMs by 40-70%. It applies abbreviation, vowel removal, and symbolic encoding using SynthLang patterns. Compression is configurable across multiple levels and includes cost estimation utilities for common model pricing.

Key Features

• Multi-Level Compression --- None, Low, Medium, High levels with increasing aggressiveness
• Pipeline Architecture --- composable compressor stages (abbreviation, vowel removal, symbols)
• Bidirectional --- compress and decompress (note: some stages are lossy)
• Token Estimation --- estimate token counts and cost savings per model pricing
• Benchmarkable --- includes Criterion benchmarks for compression throughput

Dependencies

Key external dependencies:

• regex / lazy_static --- pattern matching for compression rules
• serde / serde_json --- serialization of configs and metrics

No internal SPT crate dependencies.

Usage

use spt_synthlang::{compress, CompressionLevel};

let text = "Please analyze the database configuration and generate a report";
let result = compress(text, CompressionLevel::Medium);
println!("Savings: {:.1}%", result.metrics.savings_pct * 100.0);

API Reference

Core Types

Type	Description
CompressionResult	Compressed text and associated metrics
CompressionPipeline	Configurable pipeline of compressor stages
CompressionConfig	Pipeline configuration
CompressionLevel	Enum: None, Low, Medium, High
CompressionMetrics	Original/compressed char counts and savings
StageMetrics	Per-stage compression statistics
CostEstimate	Estimated token cost savings
ModelPricing	Pricing info for a specific LLM model

Key Functions

Function	Description
compress()	Compress text at a given level using default pipeline
decompress()	Attempt to reverse compression (lossy stages may not fully restore)
estimate_tokens()	Estimate token count for a string
estimate_savings()	Estimate cost savings for a compression result