llms

To use the large language models, you need to have the langchain_ibm python package installed, and the environment variable WATSONX_API_KEY set with your API key or pass it as a named parameter api_key to the constructor.

pip install -U langchain-ibm

# or using uv
uv add langchain-ibm

export WATSONX_API_KEY="your-api-key"

from langchain_ibm import WatsonxLLM
from ibm_watsonx_ai.metanames import GenTextParamsMetaNames

parameters = {
    GenTextParamsMetaNames.DECODING_METHOD: "sample",
    GenTextParamsMetaNames.MAX_NEW_TOKENS: 100,
    GenTextParamsMetaNames.MIN_NEW_TOKENS: 1,
    GenTextParamsMetaNames.TEMPERATURE: 0.5,
    GenTextParamsMetaNames.TOP_K: 50,
    GenTextParamsMetaNames.TOP_P: 1,
}

model = WatsonxLLM(
    model_id="google/flan-t5-xl",
    url="https://us-south.ml.cloud.ibm.com",
    project_id="*****",
    params=parameters,
    # api_key="*****"
)

input_text = "The meaning of life is "
response = model.invoke(input_text)
print(response)

"42, but what was the question?
The answer to the ultimate question of life, the universe, and everything is 42.
But what was the question? This is a reference to Douglas Adams' science fiction
series "The Hitchhiker's Guide to the Galaxy."

for chunk in model.stream(input_text):
    print(chunk, end="")

"42, but what was the question?
The answer to the ultimate question of life, the universe, and everything is 42.
But what was the question? This is a reference to Douglas Adams' science fiction
series "The Hitchhiker's Guide to the Galaxy."

response = await model.ainvoke(input_text)

# stream:
# async for chunk in model.astream(input_text):
#     print(chunk, end="")

# batch:
# await model.abatch([input_text])

"42, but what was the question?
The answer to the ultimate question of life, the universe, and everything is 42.
But what was the question? This is a reference to Douglas Adams' science fiction
series "The Hitchhiker's Guide to the Galaxy."