llm-batcher

an openai-compatible proxy that sits in front of anthropic’s claude. clients keep their openai sdk and POST /v1/chat/completions; the proxy translates both ways, coalesces concurrent requests into short admission windows, and measures itself. distributed-systems instincts pointed at llm serving.

what it does

1. translation — accepts openai-shaped chat completions, calls the anthropic messages api, translates the response back. drop-in for existing openai clients.
2. microbatching — under load, groups concurrent requests into an admission window that flushes on a size or time trigger, dispatched behind a concurrency cap so the upstream never sees a thundering herd.
3. observability — an opt-in GET /metrics snapshot with per-request p50/p95/p99 latency (nearest-rank over a bounded window), token accounting, and an estimated dollar cost with an unpriced bucket for unknown models.

visualizations

• microbatching — watch the admission window fill and flush
• observability — latency percentiles and cost breakdown

write-ups

• I Put an OpenAI-Shaped Proxy in Front of Claude
• I Taught My LLM Proxy to Wait on Purpose
• I Taught My LLM Proxy to Watch Itself