Model Hyperparameters

💻 Practice Notebook

Master the concepts from this page with hands-on practice: 💻 VS Code | 🚀 Colab | 📥 Download Notebook When invoking Large Language Models, various hyperparameters control how the model selects the next token. Tuning these parameters is vital for tailoring responses to fit specific use cases (e.g., deterministic code generation vs. creative brainstorming).

1. Key Hyperparameters

1.1 Temperature

Controls the randomness of predictions.

Low Temperature (closer to 0): The model behaves deterministically, favoring the highest-probability tokens.
High Temperature (closer to 1 or higher): The model flattens token probability distributions, allowing less common words to be chosen, creating creative or diverse responses.

Example Scenario:

Given the following vocabulary probabilities:

cat: 0.70
dog: 0.20
tiger: 0.08
elephant: 0.02
Temperature = 0: Always outputs cat (deterministic).
Temperature = 0.2: Mostly outputs cat, occasionally dog.
Temperature = 1.0: Uses original probabilities as-is.
Temperature = 2.0: The probabilities flatten out, making even elephant highly possible.

Typical Values

Use Case	Suggested Temperature
Factual QA	`0.0` - `0.3`
Coding / Math	`0.0` - `0.2`
Summarization	`0.2` - `0.5`
Creative Writing	`0.8` - `1.2`

1.2 Max Tokens

Sets the maximum limit on the number of tokens the model is allowed to generate in a single request. This prevents excessive cost and runtime.

# Terminate generation after roughly 50 tokens
llm.invoke("Explain Artificial Intelligence", max_tokens=50)

1.3 Top-K Sampling

Limits token selection to the K most likely tokens. Unlikely tokens outside the top K are discarded entirely, preventing the model from generating random gibberish.

Top-K = 2: If the top tokens are cat (0.40), dog (0.30), and tiger (0.15), only cat and dog are kept. The rest are ignored.

1.4 Top-P (Nucleus Sampling)

Instead of keeping a static count like Top-K, Top-P selects enough tokens to reach a cumulative probability threshold P.

Top-P = 0.8: If cat (0.40), dog (0.30), and tiger (0.15) sum to 0.85, the model stops adding tokens and samples only from these three.
Top-P = 0.95: Includes a wider pool of less-likely tokens.

2. Summary Table

Parameter	Purpose	Typical Production Default
Temperature	Controls token randomness/creativity	`0.2` (Factual) or `0.7` (Conversational)
Max Tokens	Restricts generation length	`1000`
Top-K	Truncates choices to K tokens	Disabled or `40` - `50`
Top-P	Truncates choices to cumulative probability P	`0.9`

3. Practice Exercises

Practice 1: Configuring Parameters in LangChain

Configure a chat model using init_chat_model with a temperature of 0.0 and a max token limit of 100 to answer the question: "State the value of Pi to 10 decimal places."

Solution

from langchain.chat_models import init_chat_model

# Initialize with deterministic settings
llm = init_chat_model(
    "llama-3.3-70b-versatile",
    model_provider="groq",
    temperature=0.0,
    max_tokens=100
)

response = llm.invoke("State the value of Pi to 10 decimal places.")
print(response.content)

Foundations

LangChain Prompts

💻 Practice Notebook

1. Key Hyperparameters

1.1 Temperature

Example Scenario:

Typical Values

1.2 Max Tokens

1.3 Top-K Sampling

1.4 Top-P (Nucleus Sampling)

2. Summary Table

3. Practice Exercises

Practice 1: Configuring Parameters in LangChain

​💻 Practice Notebook

​1. Key Hyperparameters

​1.1 Temperature

​Example Scenario:

​Typical Values

​1.2 Max Tokens

​1.3 Top-K Sampling

​1.4 Top-P (Nucleus Sampling)

​2. Summary Table

​3. Practice Exercises

​Practice 1: Configuring Parameters in LangChain

💻 Practice Notebook

1. Key Hyperparameters

1.1 Temperature

Example Scenario:

Typical Values

1.2 Max Tokens

1.3 Top-K Sampling

1.4 Top-P (Nucleus Sampling)

2. Summary Table

3. Practice Exercises

Practice 1: Configuring Parameters in LangChain