Runnable that can fallback to other Runnable objects if it fails.
External APIs (e.g., APIs for a language model) may at times experience degraded performance or even downtime.
In these cases, it can be useful to have a fallback Runnable that can be
used in place of the original Runnable (e.g., fallback to another LLM provider).
Fallbacks can be defined at the level of a single Runnable, or at the level
of a chain of Runnables. Fallbacks are tried in order until one succeeds or
all fail.
While you can instantiate a RunnableWithFallbacks directly, it is usually
more convenient to use the with_fallbacks method on a Runnable.
Example:
from langchain_core.chat_models.openai import ChatOpenAI
from langchain_core.chat_models.anthropic import ChatAnthropic
model = ChatAnthropic(model="claude-3-haiku-20240307").with_fallbacks(
[ChatOpenAI(model="gpt-3.5-turbo-0125")]
)
# Will usually use ChatAnthropic, but fallback to ChatOpenAI
# if ChatAnthropic fails.
model.invoke("hello")
# And you can also use fallbacks at the level of a chain.
# Here if both LLM providers fail, we'll fallback to a good hardcoded
# response.
from langchain_core.prompts import PromptTemplate
from langchain_core.output_parser import StrOutputParser
from langchain_core.runnables import RunnableLambda
def when_all_is_lost(inputs):
return (
"Looks like our LLM providers are down. "
"Here's a nice 🦜️ emoji for you instead."
)
chain_with_fallback = (
PromptTemplate.from_template("Tell me a joke about {topic}")
| model
| StrOutputParser()
).with_fallbacks([RunnableLambda(when_all_is_lost)])The Runnable to run first.
A sequence of fallbacks to try.
The exceptions on which fallbacks should be tried.
Any exception that is not a subclass of these exceptions will be raised immediately.
If string is specified then handled exceptions will be passed to fallbacks as
part of the input under the specified key.
If None, exceptions will not be passed to fallbacks.
If used, the base Runnable and its fallbacks must accept a dictionary as input.
Iterator over the Runnable and its fallbacks.
Get a JSON schema that represents the input to the Runnable.
Get a JSON schema that represents the output of the Runnable.
The type of config this Runnable accepts specified as a Pydantic model.
Get a JSON schema that represents the config of the Runnable.
Return a list of prompts used by this Runnable.
Pipe Runnable objects.
Pick keys from the output dict of this Runnable.
Merge the Dict input with the output produced by the mapping argument.
Run invoke in parallel on a list of inputs.
Run ainvoke in parallel on a list of inputs.
Stream all output from a Runnable, as reported to the callback system.
Generate a stream of events.
Bind arguments to a Runnable, returning a new Runnable.
Bind lifecycle listeners to a Runnable, returning a new Runnable.
Bind async lifecycle listeners to a Runnable.
Bind input and output types to a Runnable, returning a new Runnable.
Create a new Runnable that retries the original Runnable on exceptions.
Map a function to multiple iterables.
Add fallbacks to a Runnable, returning a new Runnable.
Create a BaseTool from a Runnable.