resk-caching
Semantic LLM Response Caching with Vector Database Integration
Cut LLM API Costs Without Cutting Quality
resk-caching is a Bun-based backend library and server that caches LLM responses using vector databases. When a new query arrives, it finds the most semantically similar cached response and returns it instead of calling your provider — dramatically reducing API costs while keeping response quality and relevance.
Key Features
- Up to 90% API cost reduction through intelligent semantic caching, with real-time cost tracking and per-provider pricing (OpenAI, Anthropic, Google…)
- Sub-5ms cached responses vs 500ms+ API calls, with intelligent cache warming strategies
- OpenAI-compatible API for zero-cost offline development — mock LLM provider included
- Scalability & availability: cache-first rate-limit relief, circuit breaker patterns, health monitoring, graceful degradation
- Built for AI chatbot backends with Redis integration
How It Works
- You maintain a database of high-quality LLM responses to common queries, stored as vector embeddings.
- When a new query arrives, the system finds the most semantically similar cached response.
- The cached response is returned instead of making a new API call — advanced selection algorithms support business logic, user preferences, and A/B testing.
Quick Start
npm install resk-caching
Cache, Securely
A cache is a second copy of your model's answers — treat it like one. resk-caching was built inside a security company, and it shows: cache keys are isolated per user and per policy, so one tenant's prompts and completions never serve another's requests. Sensitive-data screening keeps PII and secrets out of the store (what never lands in the cache cannot leak from it), TTLs and invalidation hooks bound how long an answer stays authoritative, and full request logging means every cached hit is auditable like a live call.
The failure mode it prevents is real: an unscoped semantic cache happily serves user A's medical question to user B because the embeddings were close. Similarity thresholds, namespace scoping and hit auditing are what make caching safe for multi-tenant LLM apps.
When Semantic Caching Pays Off
Highest returns, in order: support and FAQ assistants (thousands of near-identical questions), internal copilots (repetitive queries over stable docs), test and CI environments (the mock provider replays recorded responses deterministically at zero cost), and rate-limited workloads (cache-first absorption smooths provider throttling). RAG pipelines and highly personalized generation see smaller gains — measure your hit rate before assuming the 90% headline applies to you.
FAQ
- Is semantic caching safe for private data? With per-user scoping and sensitive-data screening enabled, yes — that is the default posture. Disable them and you are back to a shared cache: do that only for public, non-personal workloads.
- What happens on a cache miss? The request proxies to your provider as usual, and the response is embedded and stored for next time. Latency on a miss is the provider call plus milliseconds.
- Which vector databases are supported? Pluggable backends with Redis integration built in — the embeddings store is a configuration choice, not a rewrite.
- How does the mock provider help development? The OpenAI-compatible API replays cached responses offline, so CI and local dev run free of API keys, rate limits and cost — deterministically.
Get Started with resk-caching
Reduce costs and latency for your LLM applications today.
View on GitHub View on npm Documentation API ReferenceRelated Guides
Going deeper on protecting what your LLM stores and serves: data protection for AI systems covers PII handling and access logging, the LLM security hub maps the full defense stack, and securing AI applications shows where caching fits in a production architecture.