ChatFriendliParams

Base endpoint url.

Whether to disable streaming.

If streaming is bypassed, then stream() will defer to invoke().

• If true, will always bypass streaming case.
• If false (default), will always use streaming case if available.

Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled, taking into account their frequency in the preceding text. This penalization diminishes the model's tendency to reproduce identical lines verbatim.

Friendli team ID to run as.

Friendli personal access token to run as.

The maximum number of concurrent calls that can be made. Defaults to Infinity, which means no limit.

The maximum number of retries that can be made for a single call, with an exponential backoff between each attempt. Defaults to 6.

Number between -2.0 and 2.0. Positive values penalizes tokens that have been sampled at least once in the existing text. presence_penalty: Optional[float] = None The maximum number of tokens to generate. The length of your input tokens plus max_tokens should not exceed the model's maximum length (e.g., 2048 for OpenAI GPT-3)

Model name to use.

Additional kwargs to pass to the model.

Custom handler to handle failed attempts. Takes the originally thrown error object as input, and should itself throw an error if the input error is not retryable.

Version of AIMessage output format to store in message content.

AIMessage.contentBlocks will lazily parse the contents of content into a standard format. This flag can be used to additionally store the standard format as the message content, e.g., for serialization purposes.

• "v0": provider-specific format in content (can lazily parse with .contentBlocks)
• "v1": standardized format in content (consistent with .contentBlocks)

You can also set LC_OUTPUT_VERSION as an environment variable to "v1" to enable this by default.

When one of the stop phrases appears in the generation result, the API will stop generation. The phrase is included in the generated result. If you are using beam search, all of the active beams should contain the stop phrase to terminate generation. Before checking whether a stop phrase is included in the result, the phrase is converted into tokens.

Sampling temperature. Smaller temperature makes the generation result closer to greedy, argmax (i.e., top_k = 1) sampling. If it is None, then 1.0 is used.

Tokens comprising the top top_p probability mass are kept for sampling. Numbers between 0.0 (exclusive) and 1.0 (inclusive) are allowed. If it is None, then 1.0 is used by default.