ChatLlamaCpp

The path to the Llama model file.

The path to the Llama LoRA base model.

The path to the Llama LoRA. If None, no LoRa is loaded.

Token context window.

Number of parts to split the model into. If -1, the number of parts is automatically determined.

Seed. If -1, a random seed is used.

Use half-precision for key/value cache.

Return logits for all tokens, not just the last token.

Only load the vocabulary, no weights.

Force system to keep model in RAM.

Number of threads to use. If None, the number of threads is automatically determined.

Number of tokens to process in parallel. Should be a number between 1 and n_ctx.

Number of layers to be loaded into gpu memory. Default None.

A suffix to append to the generated text. If None, no suffix is appended.

The maximum number of tokens to generate.

The temperature to use for sampling.

The top-p value to use for sampling.

The number of logprobs to return. If None, no logprobs are returned.

Whether to echo the prompt.

A list of strings to stop generation when encountered.

The penalty to apply to repeated tokens.

The top-k value to use for sampling.

The number of tokens to look back when applying the repeat_penalty.

Whether to keep the model loaded in RAM

Scale factor for rope sampling.

Base frequency for rope sampling.

Any additional parameters to pass to llama_cpp.Llama.

Whether to stream the results, token by token.

grammar_path: Path to the .gbnf file that defines formal grammars for constraining model outputs. For instance, the grammar can be used to force the model to generate valid JSON or to speak exclusively in emojis. At most one of grammar_path and grammar should be passed in.

grammar: formal grammar for constraining model outputs. For instance, the grammar can be used to force the model to generate valid JSON or to speak exclusively in emojis. At most one of grammar_path and grammar should be passed in.

Print verbose output to stderr.

Validate that llama-cpp-python library is installed.

Bind tool-like objects to this chat model

Model wrapper that returns outputs formatted to match the given schema.