Chat Models are a variation on language models.
While Chat Models use language models under the hood, the interface they expose is a bit different. Rather than expose a "text in, text out" API, they expose an interface where "chat messages" are the inputs and outputs.
Anyscale Chat large language models.
See https://www.anyscale.com/ for information about Anyscale.
To use, you should have the openai python package installed, and the
environment variable ANYSCALE_API_KEY set with your API key.
Alternatively, you can use the anyscale_api_key keyword argument.
Any parameters that are valid to be passed to the openai.create call can be passed
in, even if not explicitly saved on this class.
Baichuan chat model integration.
Baidu Qianfan chat model integration.
ChatCoze chat models API by coze.com
For more information, see https://www.coze.com/open/docs/chat
A chat model that uses the DeepInfra API.
EdenAI chat large language models.
EdenAI is a versatile platform that allows you to access various language models
from different providers such as Google, OpenAI, Cohere, Mistral and more.
To get started, make sure you have the environment variable EDENAI_API_KEY
set with your API key, or pass it as a named parameter to the constructor.
Additionally, EdenAI provides the flexibility to choose from a variety of models,
including the ones like "gpt-4".
EverlyAI Chat large language models.
To use, you should have the openai python package installed, and the
environment variable EVERLYAI_API_KEY set with your API key.
Alternatively, you can use the everlyai_api_key keyword argument.
Any parameters that are valid to be passed to the openai.create call can be passed
in, even if not explicitly saved on this class.
Fake ChatModel for testing purposes.
Friendli LLM for chat.
friendli-client package should be installed with pip install friendli-client.
You must set FRIENDLI_TOKEN environment variable or provide the value of your
personal access token for the friendli_token argument.
Google PaLM Chat models API.
To use you must have the google.generativeai Python package installed and either:
1. The ``GOOGLE_API_KEY`` environment variable set with your API key, or
2. Pass your API key using the google_api_key kwarg to the ChatGoogle
constructor.
GPTRouter by Writesonic Inc.
For more information, see https://gpt-router.writesonic.com/docs
ChatModel which returns user input as the response.
Tencent Hunyuan chat models API by Tencent.
For more information, see https://cloud.tencent.com/document/product/1729
Javelin AI Gateway chat models API.
To use, you should have the javelin_sdk python package installed.
For more information, see https://docs.getjavelin.io
Jina AI Chat models API.
To use, you should have the openai python package installed, and the
environment variable JINACHAT_API_KEY set to your API key, which you
can generate at https://chat.jina.ai/api.
Any parameters that are valid to be passed to the openai.create call can be passed in, even if not explicitly saved on this class.
Kinetica LLM Chat Model API.
Prerequisites for using this API:
gpudb and typeguard packages installed.KINETICA_URLKINETICA_USER, and KINETICA_PASSWD.This API is intended to interact with the Kinetica SqlAssist LLM that supports generation of SQL from natural language.
In the Kinetica LLM workflow you create an LLM context in the database that provides
information needed for infefencing that includes tables, annotations, rules, and
samples. Invoking load_messages_from_context() will retrieve the contxt
information from the database so that it can be used to create a chat prompt.
The chat prompt consists of a SystemMessage and pairs of
HumanMessage/AIMessage that contain the samples which are question/SQL
pairs. You can append pairs samples to this list but it is not intended to
facilitate a typical natural language conversation.
When you create a chain from the chat prompt and execute it, the Kinetica LLM will
generate SQL from the input. Optionally you can use KineticaSqlOutputParser to
execute the SQL and return the result as a dataframe.
The following example creates an LLM using the environment variables for the Kinetica connection. This will fail if the API is unable to connect to the database.
ChatKonko Chat large language models API.
To use, you should have the konko python package installed, and the
environment variable KONKO_API_KEY and OPENAI_API_KEY set with your API key.
Any parameters that are valid to be passed to the konko.create call can be passed in, even if not explicitly saved on this class.
Chat with LLMs via llama-api-server
For the information about llama-api-server, visit https://github.com/second-state/LlamaEdge
llama.cpp model.
To use, you should have the llama-cpp-python library installed, and provide the path to the Llama model as a named parameter to the constructor. Check out: https://github.com/abetlen/llama-cpp-python
MariTalk Chat models API.
This class allows interacting with the MariTalk chatbot API. To use it, you must provide an API key either through the constructor.
MiniMax chat model integration.
MLflow chat models API.
To use, you should have the mlflow[genai] python package installed.
For more information, see https://mlflow.org/docs/latest/llms/deployments.
MLflow AI Gateway chat models API.
To use, you should have the mlflow[gateway] python package installed.
For more information, see https://mlflow.org/docs/latest/gateway/index.html.
MLX chat models.
Works with MLXPipeline LLM.
To use, you should have the mlx-lm python package installed.
Moonshot chat model integration.
NCP ClovaStudio Chat Completion API.
following environment variables set or passed in constructor in lower case:
NCP_CLOVASTUDIO_API_KEYNCP_APIGW_API_KEYOCI Data Science Model Deployment chat model integration.
Prerequisite
The OCI Model Deployment plugins are installable only on
python version 3.9 and above. If you're working inside the notebook,
try installing the python 3.10 based conda pack and running the
following setup.
Setup:
Install ``oracle-ads`` and ``langchain-openai``.
.. code-block:: bash
pip install -U oracle-ads langchain-openai
Use `ads.set_auth()` to configure authentication.
For example, to use OCI resource_principal for authentication:
.. code-block:: python
import ads
ads.set_auth("resource_principal")
For more details on authentication, see:
https://accelerated-data-science.readthedocs.io/en/latest/user_guide/cli/authentication.html
Make sure to have the required policies to access the OCI Data
Science Model Deployment endpoint. See:
https://docs.oracle.com/en-us/iaas/data-science/using/model-dep-policies-auth.htm
Key init args - completion params:
endpoint: str
The OCI model deployment endpoint.
temperature: float
Sampling temperature.
max_tokens: Optional[int]
Max number of tokens to generate.
Key init args — client params:
auth: dict
ADS auth dictionary for OCI authentication.
default_headers: Optional[Dict]
The headers to be added to the Model Deployment request.
Instantiate:
.. code-block:: python
from langchain_community.chat_models import ChatOCIModelDeployment
chat = ChatOCIModelDeployment(
endpoint="https://modeldeployment.<region>.oci.customer-oci.com/<ocid>/predict",
model="odsc-llm", # this is the default model name if deployed with AQUA
streaming=True,
max_retries=3,
model_kwargs={
"max_token": 512,
"temperature": 0.2,
# other model parameters ...
},
default_headers={
"route": "/v1/chat/completions",
# other request headers ...
},
)
Invocation:
.. code-block:: python
messages = [
("system", "Translate the user sentence to French."),
("human", "Hello World!"),
]
chat.invoke(messages)
.. code-block:: python
AIMessage(
content='Bonjour le monde!',
response_metadata={
'token_usage': {
'prompt_tokens': 40,
'total_tokens': 50,
'completion_tokens': 10
},
'model_name': 'odsc-llm',
'system_fingerprint': '',
'finish_reason': 'stop'
},
id='run-cbed62da-e1b3-4abd-9df3-ec89d69ca012-0'
)
Streaming:
.. code-block:: python
for chunk in chat.stream(messages):
print(chunk)
.. code-block:: python
content='' id='run-02c6-c43f-42de'
content='
' id='run-02c6-c43f-42de' content='B' id='run-02c6-c43f-42de' content='on' id='run-02c6-c43f-42de' content='j' id='run-02c6-c43f-42de' content='our' id='run-02c6-c43f-42de' content=' le' id='run-02c6-c43f-42de' content=' monde' id='run-02c6-c43f-42de' content='!' id='run-02c6-c43f-42de' content='' response_metadata={'finish_reason': 'stop'} id='run-02c6-c43f-42de'
Async:
.. code-block:: python
await chat.ainvoke(messages)
# stream:
# async for chunk in (await chat.astream(messages))
.. code-block:: python
AIMessage(
content='Bonjour le monde!',
response_metadata={'finish_reason': 'stop'},
id='run-8657a105-96b7-4bb6-b98e-b69ca420e5d1-0'
)
Structured output:
.. code-block:: python
from typing import Optional
from pydantic import BaseModel, Field
class Joke(BaseModel):
setup: str = Field(description="The setup of the joke")
punchline: str = Field(description="The punchline to the joke")
structured_llm = chat.with_structured_output(Joke, method="json_mode")
structured_llm.invoke(
"Tell me a joke about cats, "
"respond in JSON with `setup` and `punchline` keys"
)
.. code-block:: python
Joke(
setup='Why did the cat get stuck in the tree?',
punchline='Because it was chasing its tail!'
)
See ``ChatOCIModelDeployment.with_structured_output()`` for more.
Customized Usage:
You can inherit from base class and overwrite the `_process_response`,
`_process_stream_response`, `_construct_json_body` for customized usage.
.. code-block:: python
class MyChatModel(ChatOCIModelDeployment):
def _process_stream_response(self, response_json: dict) -> ChatGenerationChunk:
print("My customized streaming result handler.")
return GenerationChunk(...)
def _process_response(self, response_json:dict) -> ChatResult:
print("My customized output handler.")
return ChatResult(...)
def _construct_json_body(self, messages: list, params: dict) -> dict:
print("My customized payload handler.")
return {
"messages": messages,
**params,
}
chat = MyChatModel(
endpoint=f"https://modeldeployment.<region>.oci.customer-oci.com/{ocid}/predict",
model="odsc-llm",
}
chat.invoke("tell me a joke")
Response metadata
.. code-block:: python
ai_msg = chat.invoke(messages)
ai_msg.response_metadata
.. code-block:: python
{
'token_usage': {
'prompt_tokens': 40,
'total_tokens': 50,
'completion_tokens': 10
},
'model_name': 'odsc-llm',
'system_fingerprint': '',
'finish_reason': 'stop'
}
OCI large language chat models deployed with Text Generation Inference.
To use, you must provide the model HTTP endpoint from your deployed
model, e.g. https://modeldeployment.us-ashburn-1.oci.customer-oci.com/
To authenticate, oracle-ads has been used to automatically load
credentials: https://accelerated-data-science.readthedocs.io/en/latest/user_guide/cli/authentication.html
Make sure to have the required policies to access the OCI Data Science Model Deployment endpoint. See: https://docs.oracle.com/en-us/iaas/data-science/using/model-dep-policies-auth.htm#model_dep_policies_auth__predict-endpoint
Example:
.. code-block:: python
from langchain_community.chat_models import ChatOCIModelDeploymentTGI
chat = ChatOCIModelDeploymentTGI(
endpoint="https://modeldeployment.us-ashburn-1.oci.customer-oci.com/<ocid>/predict",
max_token=512,
temperature=0.2,
frequency_penalty=0.1,
seed=42,
# other model parameters...
)
OCI large language chat models deployed with vLLM.
To use, you must provide the model HTTP endpoint from your deployed
model, e.g. https://modeldeployment.us-ashburn-1.oci.customer-oci.com/
To authenticate, oracle-ads has been used to automatically load
credentials: https://accelerated-data-science.readthedocs.io/en/latest/user_guide/cli/authentication.html
Make sure to have the required policies to access the OCI Data Science Model Deployment endpoint. See: https://docs.oracle.com/en-us/iaas/data-science/using/model-dep-policies-auth.htm#model_dep_policies_auth__predict-endpoint
Example:
.. code-block:: python
from langchain_community.chat_models import ChatOCIModelDeploymentVLLM
chat = ChatOCIModelDeploymentVLLM(
endpoint="https://modeldeployment.us-ashburn-1.oci.customer-oci.com/<ocid>/predict",
frequency_penalty=0.1,
max_tokens=512,
temperature=0.2,
top_p=1.0,
# other model parameters...
)
ChatOCIGenAI chat model integration.
OctoAI Chat large language models.
See https://octo.ai/ for information about OctoAI.
To use, you should have the openai python package installed and the
environment variable OCTOAI_API_TOKEN set with your API token.
Alternatively, you can use the octoai_api_token keyword argument.
Any parameters that are valid to be passed to the openai.create call can be passed
in, even if not explicitly saved on this class.
Outlines chat model integration.
Alibaba Cloud PAI-EAS LLM Service chat model API.
To use, must have a deployed eas chat llm service on AliCloud. One can set the
environment variable eas_service_url and eas_service_token set with your eas
service url and service token.
PremAI Chat models.
To use, you will need to have an API key. You can find your existing API Key or generate a new one here: https://app.premai.io/api_keys/
PromptLayer and OpenAI Chat large language models API.
To use, you should have the openai and promptlayer python
package installed, and the environment variable OPENAI_API_KEY
and PROMPTLAYER_API_KEY set with your openAI API key and
promptlayer key respectively.
All parameters that can be passed to the OpenAI LLM can also be passed here. The PromptLayerChatOpenAI adds to optional
Reka chat large language models.
Snowflake Cortex based Chat model
To use the chat model, you must have the snowflake-snowpark-python Python
package installed and either:
1. environment variables set with your snowflake credentials or
2. directly passed in as kwargs to the ChatSnowflakeCortex constructor.
IFlyTek Spark chat model integration.
Nebula chat large language model - https://docs.symbl.ai/docs/nebula-llm
API Reference: https://docs.symbl.ai/reference/nebula-chat
To use, set the environment variable NEBULA_API_KEY,
or pass it as a named parameter to the constructor.
To request an API key, visit https://platform.symbl.ai/#/login
Example:
.. code-block:: python
from langchain_community.chat_models import ChatNebula
from langchain_core.messages import SystemMessage, HumanMessage
chat = ChatNebula(max_new_tokens=1024, temperature=0.5)
messages = [
SystemMessage(
content="You are a helpful assistant."
),
HumanMessage(
"Answer the following question. How can I help save the world."
),
]
chat.invoke(messages)
Alibaba Tongyi Qwen chat model integration.
Volc Engine Maas hosts a plethora of models.
You can utilize these models through this class.
To use, you should have the volcengine python package installed.
and set access key and secret key by environment variable or direct pass those
to this class.
access key, secret key are required parameters which you could get help
https://www.volcengine.com/docs/6291/65568
In order to use them, it is necessary to install the 'volcengine' Python package. The access key and secret key must be set either via environment variables or passed directly to this class. access key and secret key are mandatory parameters for which assistance can be sought at https://www.volcengine.com/docs/6291/65568.
The two methods are as follows:
Environment Variable Set the environment variables 'VOLC_ACCESSKEY' and 'VOLC_SECRETKEY' with your access key and secret key.
Pass Directly to Class Example: .. code-block:: python
from langchain_community.llms import VolcEngineMaasLLM
model = VolcEngineMaasChat(model="skylark-lite-public",
volc_engine_maas_ak="your_ak",
volc_engine_maas_sk="your_sk")
YandexGPT large language models.
There are two authentication options for the service account
with the ai.languageModels.user role:
- You can specify the token in a constructor parameter iam_token
or in an environment variable YC_IAM_TOKEN.
- You can specify the key in a constructor parameter api_key
or in an environment variable YC_API_KEY.
Yi chat models API.
Yuan2.0 Chat models API.
To use, you should have the openai-python package installed, if package
not installed, using pip install openai to install it. The
environment variable YUAN2_API_KEY set to your API key, if not set,
everyone can access apis.
Any parameters that are valid to be passed to the openai.create call can be passed in, even if not explicitly saved on this class.
ZhipuAI chat model integration.
ERNIE-Bot large language model.
ERNIE-Bot is a large language model developed by Baidu, covering a huge amount of Chinese data.
To use, you should have the ernie_client_id and ernie_client_secret set,
or set the environment variable ERNIE_CLIENT_ID and ERNIE_CLIENT_SECRET.
Note: access_token will be automatically generated based on client_id and client_secret, and will be regenerated after expiration (30 days).
Default model is ERNIE-Bot-turbo,
currently supported models are ERNIE-Bot-turbo, ERNIE-Bot, ERNIE-Bot-8K,
ERNIE-Bot-4, ERNIE-Bot-turbo-AI.
Writer chat wrapper.
Fake ChatModel for testing purposes.
Wrapper around Google's PaLM Chat API.
Wrapper around Minimax chat models.
Wrapper around Moonshot chat models.
DO NOT USE; KEPT FOR BACKWARDS COMPAT.
THIS MAY BE DELETED AT ANY POINT.
Kinetica SQL generation LLM API.
Anyscale Endpoints chat wrapper. Relies heavily on ChatOpenAI.
Wrapper around Prem's Chat API.
EverlyAI Endpoints chat wrapper. Relies heavily on ChatOpenAI.
ChatModel wrapper which returns user input as the response..
OctoAI Endpoints chat wrapper. Relies heavily on ChatOpenAI.
KonkoAI chat wrapper.
ZhipuAI chat models wrapper.
Wrapper around YandexGPT chat models.
Chat model for OCI data science model deployment endpoint.
JinaChat wrapper.
MLX Chat Wrapper.
deepinfra.com chat models wrapper
ChatYuan2 wrapper.
PromptLayer wrapper.