A cache that uses Vercel KV as the backing store.
Lookup LLM generations in cache by prompt and associated LLM key.
Sets a custom key encoder function for the cache. This function should take a prompt and an LLM key and return a string that will be used as the cache key.
Update the cache with the given generations.
Note this overwrites any existing generations for the given prompt and LLM key.
const cache = new VercelKVCache({
ttl: 3600, // Optional: Cache entries will expire after 1 hour
});
// Initialize the OpenAI model with Vercel KV cache for caching responses
const model = new ChatOpenAI({
model: "gpt-4o-mini",
cache,
});
await model.invoke("How are you today?");
const cachedValues = await cache.lookup("How are you today?", "llmKey");