LLM Inference Toolkit

A middleware layer for LLM inference exposing an OpenAI-compatible REST API. Features a semantic response cache that returns cached results for semantically similar prompts — cutting API costs without exact-match requirements — and a context compression engine that auto-summarises old conversation turns when approaching a model's context limit. Supports 100+ LLM providers via litellm.

Technologies Used

Python

Redis

Docker