LangChain NVIDIA AI Foundation Model Playground Integration
You can import langchain_nvidia instead.
This comprehensive module integrates NVIDIA's state-of-the-art AI Foundation Models, featuring advanced models for conversational AI and semantic embeddings, into the LangChain framework. It provides robust classes for seamless interaction with NVIDIA's AI models, particularly tailored for enriching conversational experiences and enhancing semantic understanding in various applications.
Features
Chat Models (ChatNVIDIA): This class serves as the primary interface for interacting with NVIDIA's Foundation chat models. Users can effortlessly utilize NVIDIA's advanced models like 'Mistral' to engage in rich, context-aware conversations, applicable across diverse domains from customer support to interactive storytelling.
Semantic Embeddings (NVIDIAEmbeddings): The module offers capabilities to generate sophisticated embeddings using NVIDIA's AI models. These embeddings are instrumental for tasks like semantic analysis, text similarity assessments, and contextual understanding, significantly enhancing the depth of NLP applications.
Installation
Install this module easily using pip:
pip install langchain-nvidia-ai-endpoints
Utilizing Chat Models
After setting up the environment, interact with NVIDIA AI Foundation models::
from langchain_nvidia_ai_endpoints import ChatNVIDIA
ai_chat_model = ChatNVIDIA(model="meta/llama2-70b")
response = ai_chat_model.invoke("Tell me about the LangChain integration.")
Generating Semantic Embeddings
Use NVIDIA's models for creating embeddings, useful in various NLP tasks::
from langchain_nvidia_ai_endpoints import NVIDIAEmbeddings
embed_model = NVIDIAEmbeddings(model="nvolveqa_40k")
embedding_output = embed_model.embed_query("Exploring AI capabilities.")Register a model as a known model.
Must be done at the beginning of a program, at least before the model is used or available models are listed.
For instance:
from langchain_nvidia_ai_endpoints import register_model, Model
register_model(
Model(
id="my-custom-model-name",
model_type="chat",
client="ChatNVIDIA",
endpoint="http://host:port/path-to-my-model"
)
)
llm = ChatNVIDIA(model="my-custom-model-name")
Be sure that the id matches the model parameter the endpoint expects.
Supported model clients are ChatNVIDIA, NVIDIAEmbeddings, NVIDIARerank,
and NVIDIA. The endpoint must accept the payload shape for the selected client.
Endpoint is required.
Use this instead of passing base_url to a client constructor when the model's
endpoint supports inference and not /v1/models listing.
Use base_url when the model's endpoint supports /v1/models listing and inference
on a known path, e.g. /v1/chat/completions.
Return the active inference priority, or None if unset.
Model information.
NVIDIA chat model.
ChatNVIDIA subclass that injects nvext.agent_hints into requests
for Dynamo KV cache routing optimization.
A unique prefix_id is auto-generated for every request.
Set inference priority for all LLM calls within scope.
Lower number = higher priority (priority=1 is most urgent).
Works as both a decorator and a context manager::
# decorator — deprioritize background work
@inference_priority(priority=10)
def background_research(query):
return llm.invoke(query)
# context manager
with inference_priority(priority=1):
result = llm.invoke(query)
# async decorator
@inference_priority(priority=10)
async def background_async(query):
return await llm.ainvoke(query)
Precedence (wins first → last):
inference_priority contextChatNVIDIADynamo(priority=1)Nesting: inner scopes fully replace outer scopes.
Client to NVIDIA embeddings models.
LangChain LLM that uses the Completions API with NVIDIA NIMs.
LangChain Document Compressor that uses the NVIDIA NeMo Retriever Reranking API.
LangChain retriever that queries the NVIDIA RAG Blueprint /v1/search endpoint.
Targets containerized RAG deployments where the rag-server exposes the search API. Supports all DocumentSearch parameters from the RAG server schema.
ChatNVIDIA subclass with Dynamo KV cache optimization support.
Chat Model Components Derived from ChatModel/NVIDIA
Inference-priority decorator for LangChain chat models.
Provides a decorator / context manager that sets priority for every
:class:~langchain_core.language_models.BaseChatModel call in scope.
The mechanism is universal: any BaseChatModel subclass whose Pydantic
model_fields include priority will automatically receive the value as
a keyword argument — no per-model integration required.
Lower number = higher priority (priority=1 is most urgent).
Example — decorator (deprioritize background work)::
from langchain_nvidia_ai_endpoints import ChatNVIDIADynamo, inference_priority
llm = ChatNVIDIADynamo(model="my-model", base_url="http://localhost:8099/v1")
@inference_priority(priority=10)
def background_research(query: str) -> str:
return llm.invoke(query).content
Example — context manager::
with inference_priority(priority=10):
result = llm.invoke("background task")
Callback Handler that prints to std out.
NVIDIARAGRetriever for NVIDIA RAG Blueprint /search endpoint.
Connects to a containerized NVIDIA RAG server that exposes the /v1/search endpoint. Supports all DocumentSearch API parameters.
Model profile data. All edits should be made in profile_augmentations.toml.