A class for building LLM-as-a-judge evaluators.
.. deprecated:: 0.5.0
LLMEvaluator is deprecated. Use openevals instead: https://github.com/langchain-ai/openevals
LLMEvaluator(
self,
*,
prompt_template: Union[str, list[tuple[str, str]]],
score_config: Union[CategoricalScoreConfig, ContinuousScoreConfig],
map_variables: Optional[Callable[[Run, Optional[Example]], dict]] = None,
model_name: str = 'gpt-4o',
model_provider: str = 'openai',
**kwargs = {}
)| Name | Type | Description |
|---|---|---|
prompt_template* | Union[str, List[Tuple[str, str]] | The prompt template to use for the evaluation. If a string is provided, it is assumed to be a human / user message. |
score_config* | Union[CategoricalScoreConfig, ContinuousScoreConfig] | The configuration for the score, either categorical or continuous. |
map_variables | Optional[Callable[[Run, Example], dict]] | Default: NoneA function that maps the run and example to the variables in the prompt. If |
model_name | Optional[str] | Default: 'gpt-4o'The model to use for the evaluation. |
model_provider | Optional[str] | Default: 'openai'The model provider to use for the evaluation. |