uptrain_callback

UpTrain Callback Handler

UpTrain is an open-source platform to evaluate and improve LLM applications. It provides grades for 20+ preconfigured checks (covering language, code, embedding use cases), performs root cause analyses on instances of failure cases and provides guidance for resolving them.

This module contains a callback handler for integrating UpTrain seamlessly into your pipeline and facilitating diverse evaluations. The callback handler automates various evaluations to assess the performance and effectiveness of the components within the pipeline.

The evaluations conducted include:

1. RAG:

   • Context Relevance: Determines the relevance of the context extracted from the query to the response.
   • Factual Accuracy: Assesses if the Language Model (LLM) is providing accurate information or hallucinating.
   • Response Completeness: Checks if the response contains all the information requested by the query.

2. Multi Query Generation: MultiQueryRetriever generates multiple variants of a question with similar meanings to the original question. This evaluation includes previous assessments and adds:

   • Multi Query Accuracy: Ensures that the multi-queries generated convey the same meaning as the original query.

3. Context Compression and Reranking: Re-ranking involves reordering nodes based on relevance to the query and selecting top n nodes. Due to the potential reduction in the number of nodes after re-ranking, the following evaluations are performed in addition to the RAG evaluations:

   • Context Reranking: Determines if the order of re-ranked nodes is more relevant to the query than the original order.
   • Context Conciseness: Examines whether the reduced number of nodes still provides all the required information.

These evaluations collectively ensure the robustness and effectiveness of the RAG query engine, MultiQueryRetriever, and the re-ranking process within the pipeline.

Useful links: Github: https://github.com/uptrain-ai/uptrain Website: https://uptrain.ai/ Docs: https://docs.uptrain.ai/getting-started/introduction