Tracer that runs a run evaluator whenever a run is persisted.
EvaluatorCallbackHandler(
self,
evaluators: Sequence[langsmith.RunEvaluator],
client: langsmith.Client | None = None,
example_id: UUID | str | None = None,
skip_unfinished: bool = True,
project_name: str | None = 'evaluators',
max_concurrency: int | None = None,
**kwargs: Any = {}
)| Name | Type | Description |
|---|---|---|
evaluators* | Sequence[langsmith.RunEvaluator] | The run evaluators to apply to all top level runs. |
client | langsmith.Client | None | Default: NoneThe LangSmith client instance to use for evaluating the runs. If not specified, a new instance will be created. |
example_id | UUID | str | None | Default: NoneThe example ID to be associated with the runs. |
skip_unfinished | bool | Default: TrueWhether to skip unfinished runs. |
project_name | str | None | Default: 'evaluators'The LangSmith project name to be organize eval chain runs under. |
max_concurrency | int | None | Default: NoneThe maximum number of concurrent evaluators to run. |
The example ID associated with the runs.
The LangSmith client instance used for evaluating the runs.
The sequence of run evaluators to be executed.
The thread pool executor used for running the evaluators.
The set of futures representing the running evaluators.
Whether to skip runs that are not finished or raised an error.
The LangSmith project name to be organize eval chain runs under.
Start a trace for a chat model run.
Start a trace for an LLM run.
Run on new LLM token.
Run on retry.
End a trace for an LLM or chat model run.
Handle an error for an LLM run.
Start a trace for a chain run.
End a trace for a chain run.
Handle an error for a chain run.
Start a trace for a tool run.
End a trace for a tool run.
Handle an error for a tool run.
Run when the Retriever starts running.
Run when Retriever errors.
Run when the Retriever ends running.
Whether to raise an error if an exception occurs.
Whether to run the callback inline.
Whether to ignore LLM callbacks.
Whether to ignore retry callbacks.
Whether to ignore chain callbacks.
Whether to ignore agent callbacks.
Whether to ignore retriever callbacks.
Whether to ignore chat model callbacks.
Ignore custom event.