SummarizationMiddleware(
self,
model: str | BaseChatModel,
*,
trigger: ContextSizeStream transformer factories registered by the middleware.
Logic to run before the agent execution starts.
Async logic to run before the agent execution starts.
Logic to run after the model is called.
Async logic to run after the model is called.
Intercept and control model execution via handler callback.
Intercept and control async model execution via handler callback.
Logic to run after the agent execution completes.
Async logic to run after the agent execution completes.
Intercept tool execution for retries, monitoring, or modification.
Intercept and control async tool execution via handler callback.
| Name | Type | Description |
|---|---|---|
model* | str | BaseChatModel | The language model to use for generating summaries. |
trigger | ContextSize | TriggerClause | list[ContextSize | TriggerClause] | None | Default: None |
keep | ContextSize | Default: ('messages', _DEFAULT_MESSAGES_TO_KEEP) |
token_counter | TokenCounter | Default: count_tokens_approximately |
summary_prompt | str | Default: DEFAULT_SUMMARY_PROMPT |
trim_tokens_to_summarize | int | None | Default: _DEFAULT_TRIM_TOKEN_LIMIT |
summarizer | Summarizer | None | Default: None |
| Name | Type |
|---|---|
| model | str | BaseChatModel |
| trigger | ContextSize | TriggerClause | list[ContextSize | TriggerClause] | None |
| keep | ContextSize |
| token_counter | TokenCounter |
| summary_prompt | str |
| trim_tokens_to_summarize | int | None |
| summarizer | Summarizer | None |
| deprecated_kwargs | Any |
Summarizes conversation history when token limits are approached.
This middleware monitors message token counts and automatically summarizes older messages when a threshold is reached, preserving recent messages and maintaining context continuity by ensuring AI/Tool message pairs remain together.
Summary-generation errors are retried in-process, up to 3 attempts total, via
Runnable.with_retry, unless the error is a ModelError marked non-retryable
(e.g. authentication or invalid request errors). If the summary prompt exceeds
the model's context window (ContextOverflowError), the call is retried once with
the middle of the history cut out. If a summary call still fails, the
underlying error propagates rather than fabricating a summary.
To generate summaries with something other than a chat model, pass a
summarizer. The middleware still decides when to summarize and which
messages to summarize; only the summary text comes from the summarizer.
Function to count tokens in messages.
Prompt template for generating summaries.
Maximum tokens to keep when preparing messages for the summarization call.
Pass None to skip trimming entirely.
One or more thresholds that trigger summarization.
Provide a single
ContextSize
tuple, or a single
TriggerClause
dict, or a list mixing either form.
A ContextSize tuple expresses one threshold. A TriggerClause dict
expresses multiple thresholds that must all be met (AND). When a list is
provided, summarization runs if any item is met (OR).
# Trigger summarization when 50 messages is reached
("messages", 50)
# Trigger summarization when 3000 tokens is reached
("tokens", 3000)
# Trigger summarization either when 80% of model's max input tokens
# is reached or when 100 messages is reached (whichever comes first)
[("fraction", 0.8), ("messages", 100)]
# Trigger when tokens >= 4000 AND messages >= 10
{"tokens": 4000, "messages": 10}
# Trigger when (tokens >= 5000 AND messages >= 3) OR
# (tokens >= 3000 AND messages >= 6)
[{"tokens": 5000, "messages": 3}, {"tokens": 3000, "messages": 6}]
See ContextSize
for more details.
Context retention policy applied after summarization.
Provide a ContextSize
tuple to specify how much history to preserve.
Defaults to keeping the most recent 20 messages.
Does not support multiple values like trigger.
# Keep the most recent 20 messages
("messages", 20)
# Keep the most recent 3000 tokens
("tokens", 3000)
# Keep the most recent 30% of the model's max input tokens
("fraction", 0.3)Optional replacement for the summarization model call.
model is still used for its profile (fractional limits) and
token counting, but is not called to write summaries.
init_chat_model, writes summaries in place of model. It is
prompted with summary_prompt and retried like model would
be. Use this to summarize with a different model than the one
whose context window you're managing. To summarize with a chat
model, pass its name or a BaseChatModel instance; wrapped
models such as model.with_config(...) are treated as a
Runnable (below).Runnable or function receives the messages to summarize
(after trimming) and returns the summary text. summary_prompt
is not used. Functions may be sync or async; an async function
requires async invocation (e.g. ainvoke), and a sync function
runs in a thread pool when invoked asynchronously.# Manage gpt-6-sol's context window, summarize with gpt-6-luna
SummarizationMiddleware(
model="openai:gpt-6-sol",
trigger=("fraction", 0.8),
summarizer="openai:gpt-6-luna",
)
# Summarize with a non-LLM service
async def summarize(messages: Sequence[BaseMessage]) -> str:
return await my_summarization_service(get_buffer_string(messages))
SummarizationMiddleware(
model="openai:gpt-6-luna",
trigger=("fraction", 0.8),
summarizer=summarize,
)