Vetch SDK
Inference waste control for production LLM systems.
Vetch observes LLM call metadata (model, tokens, latency, region), detects waste patterns without storing prompt or completion text, and can stop confirmed stalled loops before they compound.
It tracks model, tokens, latency, region, tags, session context, finish reason, visible output character count, estimated cost, energy, and carbon.
Install
pip install vetch
import vetch
vetch.instrument(
region="us-east-1",
tags={"service": "chat-api"}
)Metadata only
No prompt or completion text stored. Output text may be counted, then discarded.
Fail-open
If Vetch fails, your LLM call continues.
Detection
Patterns Vetch can surface from inference metadata.
Twelve detection advisories run in real time, grouped into three areas: agent and session, cost and caching, and output and model fit. All deterministic and auditable, with no LLM judge required. A thirteenth finding, PREMIUM-001, is surfaced only during an audit.
Agent & session
Patterns in session-level behaviour – loops, drift, and runaway context that compound across calls.
STALL-001 · Stalled agent loop
Agent is looping – producing little output while repeating the same inputs. Vetch can warn, stop, or reroute before the next wasted call.
ZOMBIE-001 · Post-completion drift
Calls keep coming after the task looks done. Spending money on calls made after work already completed.
CTX-001 · Context snowball
Context balloons on every call while the agent makes no progress. Usually repeated tool failures piling up in an already-stuck prompt.
ERROR-001 · Error storm
High error rate – three or more consecutive failures, or above 40% of recent calls erroring. Covers outages, quota limits, and malformed requests.
Cost & caching
Patterns that inflate token spend – caching gaps, bloated retrieval context, and unconstrained generation.
CACHE-001 · Prompt caching opportunity
Repeated input-token patterns suggest a cache opportunity. Uncached calls pay standard input pricing instead of a cache-read discount.
CACHE-002 · Cache not active
Same inputs, no cache hits. Prompt caching is supported by this provider but isn’t turned on yet.
RAG-001 · RAG bloat
Sending large retrieved contexts but getting short answers back. Worth checking whether all that retrieved content is actually needed.
BABBLE-001 · Excessive generation
Outputs running much longer than needed. Usually a missing output limit – extra tokens with no added value.
Output & model fit
Patterns in what the model sends back – truncation, empty output, and models not working as configured.
TRUNC-001 · Repeated truncation
Responses keep hitting the output limit and getting cut off – broken JSON, incomplete tool calls, truncated answers.
EMPTY-001 · Invisible output burn
Tokens used, almost nothing returned. Often structured output gone wrong, or a silent error producing empty responses.
STREAM-001 · Cancelled streams
Streaming responses keep getting cancelled before they finish. You pay for every token generated, then throw the partial answer away.
REASONING-001 · Reasoning not engaged
A reasoning model is returning no reasoning tokens, suggesting the reasoning path may not be engaged.
Audit-only finding
Not a real-time advisory. Surfaced during a review, once there is enough evidence to suggest a change worth testing.
PREMIUM-001 · Large model overuse
A consistent workflow using a premium model when a cheaper one might do the same job. Queued for offline review, never an automatic downgrade.
Intervention
When a pattern is confirmed, Vetch acts.
Automatic intervention is currently wired to stall detection. A broader policy engine is on the roadmap.
Warn
Log the advisory and continue. Useful during setup and customer audits.
Kill
Raise StallDetected before the next wasted call. Recover deliberately.
Reroute
Substitute a cheaper fallback model for the stalled call when configured.
vetch.instrument()
vetch.set_stall_action("kill") # or "warn" / "reroute"
with vetch.Session(tags={"feature": "agent-loop"}) as session:
try:
while True:
response = client.chat.completions.create(...) # intercepted automatically
agent.step(response)
except vetch.StallDetected:
fix_agent_state()
session.clear_stall() # re-arm, then resumeAudit workflow
Run the scan for the decision you need.
A few hours can validate setup and catch obvious loops. Around a week is a useful paid-review baseline. Longer windows are better for trends, budgets, and before/after comparisons.
1 · Instrument
Add one import and tag calls by service, feature, customer, workflow, or environment.
2 · Observe
Run warn-only while real traffic builds enough evidence for the decision.
3 · Review
Inspect advisories, attribution, token patterns, and data-quality warnings.
4 · Act
Promote only high-confidence patterns to controls, then validate with your own evals.
$ vetch audit --window 7d --tags team=ml --format text
[CRITICAL] STALL-001 · Stalled agent loop
Session: rag-search · enterprise · prod
Evidence: 16 of last 20 calls <5 output tokens, 80% input similarity
Action: review threshold -> consider set_stall_action("kill")
[WARNING] CACHE-001 · Prompt caching opportunity
Session: document-qa · prod
Evidence: 72% of calls share identical input token count
Action: inspect provider cache settings
Summary: 1,842 requests · $34.20 · 180 Wh · 72 g CO2e
Data quality: all events attributed. Window: 7 days.Illustrative output. Vetch estimates avoidable waste from your own metadata; it does not guarantee savings.
Integrations
Built for production LLM systems, not dashboards alone.
Hook into the providers and frameworks you already run. Export into the observability stack you already trust.
Wire once. Cover the endpoints you ship on.
OpenAI-compatible paths included, so custom and self-hosted stacks land the same way.
LangChain and LlamaIndex
Callback handlers ship today. Chain-level session wrapping is planned.
Into the stack you already run
OTLP export, MCP server, and background-job support for production observability.
Caveats
What the signals are for, and what they are not.
Directional estimates
Cost, energy, carbon, and water are for finding waste and guiding FinOps, engineering, and sustainability decisions. Not a substitute for formal carbon certification or water accounting.
Evidence, not a scanner
Repeated tool failures, truncation, empty visible output, and runaway context can support AppSec reviews of agent behaviour. Vetch is not a security scanner; it gives teams metadata evidence to ask sharper questions.