ChatOCIModelDeploymentTGI

OCI large language chat models deployed with Text Generation Inference.

To use, you must provide the model HTTP endpoint from your deployed model, e.g. https://modeldeployment.us-ashburn-1.oci.customer-oci.com//predict.

To authenticate, oracle-ads has been used to automatically load credentials: https://accelerated-data-science.readthedocs.io/en/latest/user_guide/cli/authentication.html

Make sure to have the required policies to access the OCI Data Science Model Deployment endpoint. See: https://docs.oracle.com/en-us/iaas/data-science/using/model-dep-policies-auth.htm#model_dep_policies_auth__predict-endpoint

Example:

.. code-block:: python

    from langchain_community.chat_models import ChatOCIModelDeploymentTGI

    chat = ChatOCIModelDeploymentTGI(
        endpoint="https://modeldeployment.us-ashburn-1.oci.customer-oci.com/<ocid>/predict",
        max_token=512,
        temperature=0.2,
        frequency_penalty=0.1,
        seed=42,
        # other model parameters...
    )