GigaChat large language models API.
To use, you should pass login and password to access GigaChat API or use token.
Base class of Friendli.
Friendli LLM.
friendli-client package should be installed with pip install friendli-client.
You must set FRIENDLI_TOKEN environment variable or provide the value of your
personal access tok
Parameters for the Javelin AI Gateway LLM.
Javelin AI Gateway LLMs.
To use, you should have the javelin_sdk python package installed.
For more information, see https://docs.getjavelin.io
Writer large language models.
To use, you should have the writer-sdk Python package installed, and the
environment variable WRITER_API_KEY set with your API key.
Parameters for AI21 penalty data.
AI21 large language models.
To use, you should have the environment variable AI21_API_KEY
set with your API key or pass it as a named parameter to the constructor.
Fake LLM for testing purposes.
Fake streaming list LLM for testing purposes.
Banana large language models.
To use, you should have the banana-dev python package installed,
and the environment variable BANANA_API_KEY set with your API key.
This is the team API key avai
HazyResearch's Manifest library.
Wrapper around You.com's conversational Smart and Research APIs.
Each API endpoint is designed to generate conversational responses to a variety of query types, including inline citations and web res
Weight only quantized model.
To use, you should have the intel-extension-for-transformers packabge and
transformers package installed.
intel-extension-for-transformers:
https://github.com
The device to use for inference, cuda or cpu
Configuration for the reader to be deployed in Titan Takeoff API.
Titan Takeoff API LLMs.
Titan Takeoff is a wrapper to interface with Takeoff Inference API for generative text to text language models.
You can use this wrapper to send requests to a generative lang
Yi large language models.
Parse the byte stream input.
The output of the model will be in the following format:
b'{"outputs": [" a"]}
' b'{"outputs": [" challenging"]} ' b'{"outputs": [" problem"]} ' ...
Handler class to transform input from LLM to a format that SageMaker endpoint expects.
Similarly, the class handles transforming output from the SageMaker endpoint to a format that LLM class expects.
Content handler for LLM class.
Use your Predibase models with Langchain.
To use, you should have the predibase python package installed,
and have your Predibase API key.
The model parameter is the Predibase "serverless" bas
Aphrodite language model.
MLX Pipeline API.
To use, you should have the mlx-lm python package installed.
Parameters for the MLflow AI Gateway LLM.
MLflow AI Gateway LLMs.
To use, you should have the mlflow[gateway] python package installed.
For more information, see https://mlflow.org/docs/latest/gateway/index.html.
Nebula Service models.
To use, you should have the environment variable NEBULA_SERVICE_URL,
NEBULA_SERVICE_PATH and NEBULA_API_KEY set with your Nebula
Service, or pass it as a named para
OCI authentication types as enumerator.
Base class for OCI GenAI models
OCI large language models.
To authenticate, the OCI client uses the methods described in https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdk_authentication_methods.htm
The authentifcation me
AzureML Managed Endpoint client.
Azure ML endpoints API types. Use dedicated for models deployed in hosted
infrastructure (also known as Online Endpoints in Azure Machine Learning),
or serverless for models deployed as a service
Transform request and response of AzureML endpoint to match with required schema.
Content handler for GPT2
Deprecated: Kept for backwards compatibility
Content handler for LLMs from the OSS catalog.
Content handler for LLMs from the HuggingFace catalog.
Content handler for the Dolly-v2-12b model
Content formatter for models that use the OpenAI like API scheme.
Deprecated: Kept for backwards compatibility
Content formatter for Llama.
Azure ML Online Endpoint models.
Azure ML Online Endpoint models.
Common parameters for Minimax large language models.
Minimax large language models.
To use, you should have the environment variable
MINIMAX_API_KEY and MINIMAX_GROUP_ID set with your API key,
or pass them as a named parameter to the constructo
Common parameters for Moonshot LLMs.
Moonshot large language models.
To use, you should have the environment variable MOONSHOT_API_KEY set with your
API key. Referenced from https://platform.moonshot.cn/docs
Base OpenAI large language model class.
Langchain LLM class to help to access eass llm service.
To use this endpoint, must have a deployed eas chat llm service on PAI AliCloud.
One can set the environment variable eas_service_url a
Baichuan large language models.
iFlyTek Spark completion model integration.
EdenAI models.
To use, you should have
the environment variable EDENAI_API_KEY set with your API token.
You can find your token here: https://app.edenai.run/admin/account/settings
feature and
SambaStudio large language models.
SambaNova Cloud large language models.
NLPCloud large language models.
To use, you should have the nlpcloud python package installed, and the
environment variable NLPCLOUD_API_KEY set with your API key.
MosaicML LLM service.
To use, you should have the
environment variable MOSAICML_API_TOKEN set with your API token, or pass
it as a named parameter to the constructor.
GooseAI large language models.
To use, you should have the openai python package installed, and the
environment variable GOOSEAI_API_KEY set with your API key.
Any parameters that are valid
Replicate models.
To use, you should have the replicate python package installed,
and the environment variable REPLICATE_API_TOKEN set with your API token.
You can find your token here: https
MLflow LLM service.
To use, you should have the mlflow[genai] python package installed.
For more information, see https://mlflow.org/docs/latest/llms/deployments.
LLM that uses OpaquePrompts to sanitize prompts.
Wraps another LLM and sanitizes prompts before passing it to the LLM, then de-sanitizes the response.
To use, you should have the ``opaqueprompts
llama.cpp model.
To use, you should have the llama-cpp-python library installed, and provide the path to the Llama model as a named parameter to the constructor. Check out: https://github.com/abetlen
Anyscale large language models.
To use, you should have the environment variable ANYSCALE_API_KEYset with your
Anyscale Endpoint, or pass it as a named parameter to the constructor.
To use with A
User input as the response.
Petals Bloom models.
To use, you should have the petals python package installed, and the
environment variable HUGGINGFACE_API_KEY set with your API key.
Any parameters that are valid to be
Aviary backend.
Aviary hosted models.
Aviary is a backend for hosted models. You can find out more about aviary at http://github.com/ray-project/aviary
To get a list of the models supported on an aviary, follow the
ForefrontAI large language models.
To use, you should have the environment variable FOREFRONTAI_API_KEY
set with your API key.
Kobold API language model.
It includes several fields that can be used to control the text generation process.
To use this class, instantiate it with the required parameters and call it with a promp
ExllamaV2 API.
To use, you should have the exllamav2 library installed, and provide the path to the Llama model as a nam
Modal large language models.
To use, you should have the modal-client python package installed.
Any parameters that are valid to be passed to the call can be passed in, even if not explicitly sa
NIBittensor LLMs
NIBittensorLLM is created by Neural Internet (https://neuralinternet.ai/), powered by Bittensor, a decentralized network full of different AI models.
To analyze API_KEYS and logs of
Common configuration for Solar LLMs.
Solar large language models.
To use, you should have the environment variable
SOLAR_API_KEY set with your API key.
Referenced from https://console.upstage.ai/services/solar
Raised when the Ollama endpoint is not found.
ChatGLM3 LLM service.
Base class for VolcEngineMaas models.
volc engine maas hosts a plethora of models. You can utilize these models through this class.
To use, you should have the volcengine python package installed.
and set access key and secret key by
Neural Magic DeepSparse LLM interface.
To use, you should have the deepsparse or deepsparse-nightly
python package installed. See https://github.com/neuralmagic/deepsparse
This interface let's
HuggingFace Pipeline API to run on self-hosted remote hardware.
Supported hardware includes auto-launched instances on AWS, GCP, Azure, and Lambda, as well as servers specified by IP address and SSH
Adapter class to prepare the inputs from Langchain to a format that LLM model expects.
It also provides helper function to extract the generated text from the model response.
Base class for Bedrock models.
Model inference on self-hosted remote hardware.
Supported hardware includes auto-launched instances on AWS, GCP, Azure, and Lambda, as well as servers specified by IP address and SSH credentials (suc
Konko AI models.
To use, you'll need an API key. This can be passed in as init param
konko_api_key or set as environment variable KONKO_API_KEY.
Konko AI API reference: https://docs.konko.ai
CTranslate2 language model.
PipelineAI large language models.
To use, you should have the pipeline-ai python package installed,
and the environment variable PIPELINE_API_KEY set with your API key.
Any parameters that a
Tongyi completion model integration.
Cloudflare Workers AI service.
To use, you must provide an API token and account ID to access Cloudflare Workers AI, and pass it as a named parameter to the constructor.
Text generation models from WebUI.
To use, you should have the text-generation-webui installed, a model loaded, and --api added as a command-line option.
Suggested installation, use one-click instal
Yandex large language models.
To use, you should have the yandexcloud python package installed.
There are two authentication options for the service account
with the ai.languageModels.user r
GPT4All language models.
To use, you should have the gpt4all python package installed, the
pre-trained model file, and the model's config information.
Train result.
Gradient.ai LLM Endpoints.
GradientLLM is a class to interact with LLMs on gradient.ai
To use, set the environment variable GRADIENT_ACCESS_TOKEN with your
API token and ``GRADIENT_WORKSPACE_ID`
Llamafile lets you distribute and run large language models with a single file.
To get started, see: https://github.com/Mozilla-Ocho/llamafile
To use this class, you will need to first:
StochasticAI large language models.
To use, you should have the environment variable STOCHASTICAI_API_KEY
set with your API key.
OctoAI LLM Endpoints - OpenAI compatible.
OctoAIEndpoint is a class to interact with OctoAI Compute Service large language model endpoints.
To use, you should have the environment variable ``OCTOAI_
RWKV language models.
To use, you should have the rwkv python package installed, the
pre-trained model file, and the model's config information.
LLM wrapper for the Outlines library.
CerebriumAI large language models.
To use, you should have the cerebrium python package installed.
You should also have the environment variable CEREBRIUMAI_API_KEY
set with your API key or p
OpenLM models.
Wrapper around the BigdlLLM model
Adapter to prepare the inputs from Langchain to a format that LLM model expects.
It also provides helper function to extract the generated text from the model response.
Amazon API Gateway to access LLM models hosted on AWS.
Xinference large-scale model inference service.
To use, you should have the xinference library installed:
.. code-block:: bash
pip install "xinference[all]"
If you're simply using the service
Baidu Qianfan completion model integration.
VLLM language model.
vLLM OpenAI-compatible API client
Beam API for gpt2 large language model.
To use, you should have the beam-sdk python package installed,
and the environment variable BEAM_CLIENT_ID set with your client id
and ``BEAM_CLIENT_SE
DeepInfra models.
To use, you should have the environment variable DEEPINFRA_API_TOKEN
set with your API token, or pass it as a named parameter to the
constructor.
Only supports `text-generation
OpenAI's compatible API client for OpenLLM server
.. versionchanged:: 0.2.11
Changed in 0.2.11 to support OpenLLM 0.6. Now behaves similar to OpenAI wrapper.
Raises when token expired.
Raises when encounter server error when making inference.
Base class for LLM deployed on OCI Data Science Model Deployment.
LLM deployed on OCI Data Science Model Deployment.
To use, you must provide the model HTTP endpoint from your deployed
model, e.g. https://modeldeployment.
OCI Data Science Model Deployment TGI Endpoint.
To use, you must provide the model HTTP endpoint from your deployed
model, e.g. https://modeldeployment.
VLLM deployed on OCI Data Science Model Deployment
To use, you must provide the model HTTP endpoint from your deployed
model, e.g. https://modeldeployment.
ChatGLM LLM service.
Arcee's Domain Adapted Language Models (DALMs).
To use, set the ARCEE_API_KEY environment variable with your Arcee API key,
or pass arcee_api_key as a named parameter.
Yuan2.0 language models.
C Transformers LLM models.
To use, you should have the ctransformers python package installed.
See https://github.com/marella/ctransformers
Aleph Alpha large language models.
To use, you should have the aleph_alpha_client python package installed, and the
environment variable ALEPH_ALPHA_API_KEY set with your API key, or pass
it
Clarifai large language models.
To use, you should have an account on the Clarifai platform,
the clarifai python package installed, and the
environment variable CLARIFAI_PAT set with your PAT
PromptLayer OpenAI large language models.
To use, you should have the openai and promptlayer python
package installed, and the environment variable OPENAI_API_KEY
and ``PROMPTLAYER_API_KE
PromptLayer OpenAI large language models.
To use, you should have the openai and promptlayer python
package installed, and the environment variable OPENAI_API_KEY
and ``PROMPTLAYER_API_KE
IpexLLM model.
Base class for Cohere models.
Cohere large language models.
To use, you should have the cohere python package installed, and the
environment variable COHERE_API_KEY set with your API key, or pass
it as a named parameter t
Sagemaker Inference Endpoint models.
To use, you must supply the endpoint name from your deployed Sagemaker model & the region where it is deployed.
To authenticate, the AWS client uses the followin
DEPRECATED: Use langchain_google_genai.GoogleGenerativeAI instead.
Google PaLM models.
OpenAI large language models.
To use, you should have the openai python package installed, and the
environment variable OPENAI_API_KEY set with your API key.
Any parameters that are valid to
Azure-specific OpenAI large language models.
To use, you should have the openai python package installed, and the
environment variable OPENAI_API_KEY set with your API key.
Any parameters th
OpenAI Chat large language models.
To use, you should have the openai python package installed, and the
environment variable OPENAI_API_KEY set with your API key.
Any parameters that are val
HuggingFace text generation API. ! This class is deprecated, you should use HuggingFaceEndpoint instead !
To use, you should have the text-generation python package installed and
a text-generation
Fireworks models.
Google Vertex AI large language models.
Vertex AI Model Garden large language models.
Ollama locally runs large language models. To use, follow the instructions at https://ollama.ai/. Example: .. code-block:: python from langchain_community.llms import Ollama ollama
Prediction Guard large language models.
To use, you should have the predictionguard python package installed, and the
environment variable PREDICTIONGUARD_API_KEY set with your API key, or pa
HuggingFace Endpoint.
To use this class, you should have installed the huggingface_hub package, and
the environment variable HUGGINGFACEHUB_API_TOKEN set with your API token,
or given as a na
Bedrock models.
To authenticate, the AWS client uses the following methods to automatically load credentials: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html
If a spec
HuggingFaceHub models. ! This class is deprecated, you should use HuggingFaceEndpoint instead.
To use, you should have the huggingface_hub python package installed, and the
environment variable
Databricks serving endpoint or a cluster driver proxy app for LLM.
It supports two endpoint types:
HuggingFace Pipeline API.
To use, you should have the transformers python package installed.
Only supports text-generation, text2text-generation, summarization and
translation for now.
IBM watsonx.ai large language models.
To use, you should have ibm_watsonx_ai python package installed,
and the environment variable WATSONX_APIKEY set with your API key, or pass
it as a named
LLM models from Together.
To use, you'll need an API key which you can find here:
https://api.together.xyz/settings/api-keys. This can be passed in as init param
together_api_key or set as envi
Anthropic large language models.
To use, you should have the anthropic python package installed, and the
environment variable ANTHROPIC_API_KEY set with your API key, or pass
it as a named pa
Use tenacity to retry the completion call.
Use tenacity to retry the completion call.
Load LLM from Config Dict.
Load LLM from a file.
Use tenacity to retry the completion call.
Generate text from the model.
Use tenacity to retry the completion call.
Cut off the text as soon as any stop words occur.
Update token usage.
Use tenacity to retry the completion call.
Use tenacity to retry the async completion call.
Update token usage.
Create the LLMResult from the choices and prompts.
List available models
Get completions from Aviary models.
Conditionally apply a decorator.
Use tenacity to retry the completion call.
Use tenacity to retry the completion call.
Use tenacity to retry the completion call.
Use tenacity to retry the completion call.
Use tenacity to retry the completion call for streaming.
Remove trailing slash and /api from url if present.
Return True if the model name is a Codey model.
Return True if the model name is a Gemini model.
Use tenacity to retry the completion call.
Use tenacity to retry the completion call.
Get the notebook REPL context if running inside a Databricks notebook. Returns None otherwise.
Get the default Databricks workspace hostname. Raises an error if the hostname cannot be automatically determined.
Get the default Databricks personal access token. Raises an error if the token cannot be automatically determined.
Check the response from the completion call.
Use tenacity to retry the completion call.
Use tenacity to retry the completion call.
Async version of stream_generate_with_retry.
Because the dashscope SDK doesn't provide an async API,
we wrap stream_generate_with_retry with an async generator.
Generate elements from an iterable, and a boolean indicating if it is the last element.
Generate elements from an async iterable, and a boolean indicating if it is the last element.
Use tenacity to retry the completion call.
Use tenacity to retry the async completion call.