Tracer that runs a run evaluator whenever a run is persisted.
EvaluatorCallbackHandler(
self,
evaluators: Sequence[langsmith.RunEvaluator],
client: langsmith.Client | None = None,
example_id: UUID | str | None = None,
skip_unfinished: bool = True,
project_name: str | None = 'evaluators',
max_concurrency: int | None = None,
**kwargs: Any = {}
)| Name | Type | Description |
|---|---|---|
evaluators* | Sequence[langsmith.RunEvaluator] | The run evaluators to apply to all top level runs. |
client | langsmith.Client | None | Default: NoneThe LangSmith client instance to use for evaluating the runs. If not specified, a new instance will be created. |
example_id | UUID | str | None | Default: NoneThe example ID to be associated with the runs. |
skip_unfinished | bool | Default: TrueWhether to skip unfinished runs. |
project_name | str | None | Default: 'evaluators'The LangSmith project name to be organize eval chain runs under. |
max_concurrency | int | None | Default: NoneThe maximum number of concurrent evaluators to run. |
The example ID associated with the runs.
The LangSmith client instance used for evaluating the runs.
The sequence of run evaluators to be executed.
The thread pool executor used for running the evaluators.
The set of futures representing the running evaluators.
Whether to skip runs that are not finished or raised an error.
The LangSmith project name to be organize eval chain runs under.
Start a trace for a chat model run.
Start a trace for a (non-chat model) LLM run.
Run on new output token.
Run on retry.
End a trace for a model run.
Handle an error for an LLM run.
Start a trace for a chain run.
End a trace for a chain run.
Handle an error for a chain run.
Start a trace for a tool run.
End a trace for a tool run.
Run when tool errors.
Run when Retriever starts running.
Run when Retriever errors.
Run when Retriever ends running.
Whether to raise an error if an exception occurs.
Whether to ignore LLM callbacks.
Whether to ignore retry callbacks.
Whether to ignore chain callbacks.
Whether to ignore agent callbacks.
Whether to ignore retriever callbacks.
Whether to ignore chat model callbacks.
Ignore custom event.