vLLM | Ziyang Lin

LLM Hyperparameter Tuning Guide: A Comprehensive Analysis from Generation to Deployment

Fri, 27 Jun 2025 03:00:00 +0000

Introduction

Behind the powerful capabilities of large language models (LLMs) is a series of complex hyperparameters working silently. Whether you're deploying a local inference service like vLLM or calling OpenAI's API, precisely tuning these parameters is crucial for achieving ideal performance, cost, and output quality. This document provides a detailed analysis of two key categories of hyperparameters: Generation (Sampling) Parameters and Deployment (Serving) Parameters, helping you fully master their functions, values, impacts, and best practices across different scenarios.

Part 1: Generation (Sampling) Parameters — Controlling Model Creativity and Determinism

Generation parameters directly control the model's behavior when generating the next token. They primarily revolve around a core question: how to select from thousands of possible next words in the probability distribution provided by the model.

1. `temperature`

In one sentence: Controls the randomness of generated text. Higher temperature increases randomness, making responses more creative and diverse; lower temperature decreases randomness, making responses more deterministic and conservative.

Underlying Principle: When generating the next token, the model calculates logits (raw, unnormalized prediction scores) for all words in the vocabulary. Typically, we use the Softmax function to convert these logits into a probability distribution. The temperature parameter is introduced before the Softmax calculation, “smoothing” or “sharpening” this probability distribution.

The standard Softmax formula is: P(i) = exp(logit_i) / Σ_j(exp(logit_j))

With temperature (T) introduced, the formula becomes: P(i) = exp(logit_i / T) / Σ_j(exp(logit_j / T))
- When T -> 0, the differences in logit_i / T become dramatically amplified. The token with the highest logit approaches a probability of 1, while all other tokens approach 0. This causes the model to almost always choose the most likely word, behaving very deterministically and “greedily.”
- When T = 1, the formula reverts to standard Softmax, and the model behaves in its “original” state.
- When T > 1, the differences in logit_i / T are reduced. Tokens with originally lower probabilities get boosted, making the entire probability distribution “flatter.” This increases the chance of selecting less common words, introducing more randomness and creativity.
Value Range and Recommendations:
- Range: [0.0, 2.0] (theoretically can be higher, but OpenAI API typically limits to 2.0).
- temperature = 0.0: Suitable for scenarios requiring deterministic, reproducible, and highly accurate outputs. Examples: code generation, factual Q&A, text classification, data extraction. With identical inputs, outputs will be almost identical (unless the model itself is updated).
- Low temperature (e.g., 0.1 - 0.4): Suitable for semi-creative tasks requiring rigor and fidelity to source material. Examples: article summarization, translation, customer service bots. Outputs will vary slightly but remain faithful to core content.
- Medium temperature (e.g., 0.5 - 0.8): A good balance between creativity and consistency, recommended as the default for most applications. Examples: writing emails, marketing copy, brainstorming.
- High temperature (e.g., 0.9 - 1.5): Suitable for highly creative tasks. Examples: poetry writing, story creation, dialogue script generation. Outputs will be very diverse and sometimes surprising, but may occasionally produce meaningless or incoherent content.
Note:
- It's generally not recommended to modify both temperature and top_p simultaneously; it's better to adjust just one. OpenAI's documentation explicitly states that modifying only one is typically advised.

2. `top_p` (Nucleus Sampling)

In one sentence: Controls generation diversity by dynamically determining the sampling pool size through a cumulative probability threshold (p) of the highest probability tokens.

Underlying Principle: top_p is a more intelligent sampling strategy than temperature, also known as Nucleus Sampling. Instead of adjusting all token probabilities, it directly defines a “core” candidate set.

The specific steps are as follows:
1. The model calculates the probability distribution for all candidate tokens.
2. All tokens are sorted by probability from highest to lowest.
3. Starting from the highest probability token, their probabilities are cumulatively added until this sum exceeds the set top_p threshold.
4. All tokens included in this cumulative sum form the “nucleus” for sampling.
5. The model will only sample from this nucleus (typically renormalizing their probabilities), and all other tokens are ignored.
Example: Assume top_p = 0.9.
- If the highest probability token “the” has a probability of 0.95, then the nucleus will contain only “the”, and the model will choose it 100%.
- If “the” has a probability of 0.5, “a” has 0.3, and “an” has 0.1, then the cumulative probability of these three words is 0.9. The nucleus will contain {“the”, “a”, “an”}. The model will sample from these three words according to their (renormalized) probabilities.
Value Range and Recommendations:
- Range: (0.0, 1.0].
- top_p = 1.0: Means the model considers all tokens without any truncation (equivalent to no top_p).
- High top_p (e.g., 0.9 - 1.0): Allows for more diverse choices, suitable for creative tasks, similar in effect to higher temperature.
- Low top_p (e.g., 0.1 - 0.3): Greatly restricts the model's range of choices, making its output very deterministic and conservative, similar in effect to extremely low temperature.
- General Recommended Value: 0.9 is a very common default value as it maintains high quality while allowing for some diversity.
top_p vs temperature:
- top_p is more dynamic and adaptive. When the model is very confident about the next step (sharp probability distribution), top_p automatically narrows the candidate set, ensuring quality. When the model is less confident (flat distribution), it expands the candidate set, increasing diversity.
- temperature adjusts the entire distribution “equally,” regardless of whether the distribution itself is sharp or flat.
- Therefore, top_p is generally considered a safer and more robust method for controlling diversity than temperature.

3. `top_k`

In one sentence: Simply and directly samples only from the k tokens with the highest probabilities.

Underlying Principle: This is the simplest truncation sampling method. It directly selects the k tokens with the highest probabilities to form the candidate set, then samples from these k tokens. All other tokens are ignored.
Value Range and Recommendations:
- Range: Integers, such as 1, 10, 50.
- top_k = 1: Equivalent to greedy search, always choosing the most likely word.
- Recommendation: top_k is typically not the preferred sampling strategy because it's too “rigid.” In cases where the probability distribution is very flat, it might accidentally exclude many reasonable words; while in cases where the distribution is very sharp, it might include many extremely low-probability, useless words. top_p is usually a better choice.

4. `repetition_penalty`

In one sentence: Applies a penalty to tokens that have already appeared in the context, reducing their probability of being selected again, thereby reducing repetitive content.

Underlying Principle: After calculating logits but before Softmax, this parameter iterates through all candidate tokens. If a token has already appeared in the previous context, its logit value is reduced (typically divided by the value of repetition_penalty).

new_logit = logit / penalty (if token has appeared) new_logit = logit (if token has not appeared)

This way, the final probability of words that have already appeared decreases.
Value Range and Recommendations:
- Range: 1.0 to 2.0 is common.
- 1.0: No penalty applied (default value).
- 1.1 - 1.3: A relatively safe range that can effectively reduce unnecessary repetition without overly affecting normal language expression (such as necessary articles like “the”).
- Too High Values: May cause the model to deliberately avoid common words, producing unnatural or even strange sentences.

5. `frequency_penalty` & `presence_penalty`

These two parameters are more refined versions of repetition_penalty.

presence_penalty:
- Function: Applies a fixed penalty to all tokens that have appeared at least once in the context. It doesn't care how many times the token has appeared; as long as it has appeared, it gets penalized.
- Underlying Principle: new_logit = logit - presence_penalty (if token has appeared at least once).
- Scenario: This parameter is useful when you want to encourage the model to introduce entirely new concepts and vocabulary, rather than repeatedly discussing topics that have already been mentioned.
- Range: 0.0 to 2.0. Positive values penalize new tokens, negative values encourage them.
frequency_penalty:
- Function: The penalty is proportional to the frequency of the token in the context. The more times a word appears, the heavier the penalty it receives.
- Underlying Principle: new_logit = logit - count(token) * frequency_penalty.
- Scenario: This parameter is effective when you find the model tends to repeatedly use certain specific high-frequency words (even if they are necessary), leading to monotonous language.
- Range: 0.0 to 2.0.
Summary: presence_penalty addresses the question of “whether it has appeared,” while frequency_penalty addresses “how many times it has appeared.”

6. `seed`

In one sentence: By providing a fixed seed, you can make the model's output reproducible when other parameters (such as temperature) remain the same.

Function: In machine learning, many operations that seem random are actually “pseudo-random,” determined by an initial “seed.” Setting the same seed will produce the same sequence of random numbers. In LLMs, this means the sampling process will be completely deterministic.
Scenarios:
- Debugging and Testing: When you need to verify whether a change has affected the output, fixing the seed can eliminate randomness interference.
- Reproducible Research: Reproducibility is crucial in academic research.
- Generating Consistent Content: When you need the model to consistently produce outputs in the same style for the same input.
Note: For complete reproduction, all generation parameters (prompt, model, temperature, top_p, etc.) must be identical.

Part 2: Deployment (Serving) Parameters — Optimizing Service Performance and Capacity

Deployment parameters determine how an LLM inference service manages GPU resources, handles concurrent requests, and optimizes overall throughput and latency. These parameters are particularly important in high-performance inference engines like vLLM.

1. `gpu_memory_utilization`

In one sentence: Controls the proportion of GPU memory that vLLM can use, with the core purpose of reserving space for the KV Cache.

Underlying Principle (PagedAttention): The core of vLLM is the PagedAttention mechanism. Traditional attention mechanisms pre-allocate a continuous, maximum-length memory space for each request to store the Key-Value (KV) Cache. This leads to severe memory waste, as most requests are far shorter than the maximum length.

PagedAttention manages the KV Cache like virtual memory in an operating system:
1. It breaks down each sequence's KV Cache into many small, fixed-size “blocks.”
2. These blocks can be stored non-contiguously in GPU memory.
3. A central “Block Manager” is responsible for allocating and releasing these blocks.
gpu_memory_utilization tells vLLM: “You can use this much proportion of the total GPU memory for free management (mainly storing model weights and physical blocks of KV Cache).”
Value Range and Impact:
- Range: (0.0, 1.0].
- Default Value: 0.9 (i.e., 90%).
- Higher Values (e.g., 0.95):
  - Advantage: vLLM has more memory for KV Cache, supporting longer contexts and larger batch sizes, thereby increasing throughput.
  - Risk: If set too high, there might not be enough spare memory for CUDA kernels, drivers, or other system processes, easily leading to OOM (Out of Memory) errors.
- Lower Values (e.g., 0.8):
  - Advantage: Safer, less prone to OOM, reserves more memory for the system and other applications.
  - Disadvantage: Reduced available space for KV Cache, potentially causing vLLM to struggle with high concurrency or long sequence requests, degrading performance. When KV Cache is insufficient, vLLM triggers Preemption, swapping out some running sequences and waiting to swap them back in when there's enough space, severely affecting latency. vLLM's warning log "there is not enough KV cache space. This can affect the end-to-end performance." is reminding you of this issue.
Recommendations:
- Start with the default value of 0.9.
- If you encounter OOM, gradually lower this value.
- If you encounter many preemption warnings and confirm no other processes are occupying large amounts of GPU memory, you can gradually increase this value.

2. `max_num_seqs`

In one sentence: Limits the maximum number of sequences (requests) that the vLLM scheduler can process in one iteration (or one batch).

Underlying Principle: vLLM's scheduler selects a batch of requests from the waiting queue in each processing cycle. This parameter directly limits the size of this “batch.” Together with max_num_batched_tokens (which limits the total number of tokens across all sequences in a batch), it determines the scale of batch processing.
Value Range and Impact:
- Range: Positive integers, such as 16, 64, 256.
- Higher Values:
  - Advantage: Allows for higher concurrency, potentially improving GPU utilization and overall throughput.
  - Disadvantage: Requires more intermediate memory (e.g., for storing logits and sampling states) and may increase the latency of individual batches. If set too high, even if KV Cache still has space, OOM might occur due to insufficient temporary memory.
- Lower Values:
  - Advantage: More memory-friendly, potentially lower latency for individual batches.
  - Disadvantage: Limits concurrency capability, potentially leading to underutilization of GPU and decreased throughput.
Recommendations:
- This value needs to be adjusted based on your GPU memory size, model size, and expected concurrent load.
- For high-concurrency scenarios, try gradually increasing this value while monitoring GPU utilization and memory usage.
- For interactive, low-latency scenarios, consider setting this value lower.

3. `max_model_len`

In one sentence: Sets the maximum context length the model can process (including both prompt and generated tokens).

Underlying Principle: This parameter directly determines how much logical space vLLM needs to reserve for the KV Cache. For example, if max_model_len = 4096, vLLM must ensure its memory management mechanism can support storing KV pairs for up to 4096 tokens per sequence. This affects vLLM's memory planning at startup, such as the size of Position Embeddings.
Value Range and Impact:
- Range: Positive integers, cannot exceed the maximum length the model was originally trained on.
- Higher Values:
  - Advantage: Can handle longer documents and more complex contexts.
  - Disadvantage: Significantly increases memory consumption. Each token needs to store KV Cache; doubling the length roughly doubles the memory usage. Even if current requests are short, vLLM needs to prepare for potentially long requests, which occupies more KV Cache blocks.
- Lower Values:
  - Advantage: Significantly saves GPU memory. If you know your application scenario will never exceed 1024 tokens, setting this value to 1024 instead of the default 4096 or 8192 will free up a large amount of KV Cache space, supporting higher concurrency.
  - Disadvantage: Any requests exceeding this length will be rejected or truncated.
Recommendations:
- Set as needed! This is one of the most effective parameters for optimizing vLLM memory usage. Based on your actual application scenario, set this value to a reasonable maximum with some margin.

4. `tensor_parallel_size` & `pipeline_parallel_size`

These two parameters are used for deploying extremely large models across multiple GPUs or nodes.

tensor_parallel_size:
- Function: Divides each layer of the model (such as a large weight matrix) into N parts (N = tensor_parallel_size), placing them on N different GPUs. During computation, each GPU only processes its own portion of the data, then exchanges necessary results through high-speed interconnects (like NVLink) via All-Reduce operations, finally merging to get the complete output.
- Scenario: Used when a single model's volume exceeds the memory of a single GPU. For example, a 70B model cannot fit into a single 40GB A100, but can be deployed across two A100s by setting tensor_parallel_size=2.
- Impact:
  - Advantage: Achieves model parallelism, solving the problem of models not fitting on a single card.
  - Disadvantage: Introduces significant cross-GPU communication overhead, potentially affecting latency. Requires high-speed interconnects between GPUs.
pipeline_parallel_size:
- Function: Assigns different layers of the model to different GPUs or nodes. For example, placing layers 1-10 on GPU 1, layers 11-20 on GPU 2, and so on. Data flows through these GPUs like a pipeline.
- Scenario: Used when the model is extremely large and needs to be deployed across multiple nodes (machines).
- Impact:
  - Advantage: Can scale the model to any number of GPUs/nodes.
  - Disadvantage: Creates “pipeline bubbles” as additional overhead, where some GPUs are idle during the start and end phases of the pipeline, reducing utilization.
Combined Use: vLLM supports using both parallelism strategies simultaneously for efficient deployment of giant models on large clusters.

Summary and Best Practices

Scenario	`temperature`	`top_p`	`repetition_penalty`	`gpu_memory_utilization`	`max_num_seqs`	`max_model_len`
Code Generation/Factual Q&A	`0.0` - `0.2`	(Not recommended to modify)	`1.0`	`0.9` (Default)	Adjust based on concurrency	Set as needed
Article Summarization/Translation	`0.2` - `0.5`	(Not recommended to modify)	`1.1`	`0.9`	Adjust based on concurrency	Set to maximum possible document length
General Chat/Copywriting	`0.7` (Default)	`0.9` (Recommended)	`1.1` - `1.2`	`0.9`	Adjust based on concurrency	Set as needed, e.g., `4096`\|
Creative Writing/Brainstorming	`0.8` - `1.2`	`0.95`	`1.0`	`0.9`	Adjust based on concurrency	Set as needed
High Concurrency Throughput Optimization	(Task dependent)	(Task dependent)	(Task dependent)	Try `0.9` - `0.95`	Gradually increase	Set to the minimum value that meets business needs
Low Latency Interaction Optimization	(Task dependent)	(Task dependent)	(Task dependent)	`0.9` (Default)	Set to lower values (e.g., `16-64`)	Set as needed
Extremely Memory Constrained	(Task dependent)	(Task dependent)	(Task dependent)	Lower to `0.8`	Set to lower values	Set to the minimum value that meets business needs

Final Recommendations:

Start with Generation Parameters: First adjust temperature or top_p to achieve satisfactory output quality.
Set Deployment Parameters as Needed: When deploying, first set max_model_len to a reasonable minimum value based on your application scenario.
Monitor and Iterate: Start with the default gpu_memory_utilization=0.9 and a moderate max_num_seqs. Observe memory usage and preemption situations through monitoring tools (such as nvidia-smi and vLLM logs), then gradually adjust these values to find the optimal balance for your specific hardware and workload.

Model Quantization Guide: A Comprehensive Analysis from Theory to Practice

Fri, 27 Jun 2025 00:00:00 +0000

1. Introduction

As large language models (LLMs) continue to grow in scale and complexity, their deployment and inference costs have become increasingly expensive. Model quantization, as a key optimization technique, significantly reduces model storage requirements, memory consumption, and computational load by lowering the numerical precision of model weights and activation values, enabling efficient inference on resource-constrained devices such as mobile and edge devices.

This document aims to provide a clear and comprehensive introduction to the core concepts of deep learning model quantization, mainstream approaches, and specific implementations in two leading inference frameworks—llama.cpp and vLLM. We will explore in detail the quantization types they support, underlying principles, usage methods, and future trends in quantization technology.

2. Quantization Fundamentals

Before diving into specific frameworks, we need to understand some basic concepts of quantization.

2.1 What is Model Quantization?

Model quantization refers to the process of converting floating-point numbers in a model (typically 32-bit floating-point, or FP32) to integers with fewer bits (such as INT8, INT4) or lower-precision floating-point numbers (such as FP16, FP8). This process is essentially a form of information compression that attempts to significantly reduce model complexity while preserving model accuracy as much as possible.

2.2 Why is Quantization Needed?

Reduced Model Size: Lower bit-width numerical representations can significantly reduce the size of model files. For example, quantizing an FP32 model to INT8 can reduce the model size by approximately 4 times.
Lower Memory Bandwidth: Smaller data types mean less bandwidth is occupied when transferring data between memory and computational units, which is crucial for memory bandwidth-sensitive hardware.
Accelerated Computation: Many modern processors (CPUs, GPUs, TPUs) support integer operations more efficiently than floating-point operations, providing higher throughput and lower latency.
Reduced Power Consumption: Integer operations typically consume less energy than floating-point operations.

2.3 Quantization Principles: Mapping and Dequantization

The core of quantization is mapping a larger range of floating-point values to a smaller range of fixed-point integer values. This process is defined by the following formula:

Q(r) = round(r / S + Z)

Where:

r is the original floating-point value.
Q(r) is the quantized integer value.
S is the Scale factor, representing the floating-point value size corresponding to each quantized integer step.
Z is the Zero-point, representing the quantized integer value corresponding to floating-point zero.

When performing calculations, the quantized values need to be dequantized back to the floating-point domain:

r' = S * (Q(r) - Z)

r' is the dequantized floating-point number, which has some quantization error compared to the original value r.

2.4 Symmetric vs. Asymmetric Quantization

Based on the choice of zero-point, quantization can be divided into two modes:

Symmetric Quantization: Maps the floating-point range [-abs_max, abs_max] symmetrically to the integer range. In this mode, the zero-point Z is typically 0 (for signed integers) or 2^(bits-1) (for unsigned integer offset). Computation is relatively simple.
Asymmetric Quantization: Maps the complete floating-point range [min, max] to the integer range. In this mode, the zero-point Z is a floating-point number that can be adjusted according to data distribution. It can more accurately represent asymmetrically distributed data but is slightly more complex in computation.

2.5 Per-Layer vs. Per-Group/Per-Channel Quantization

The granularity of calculating scale factor S and zero-point Z also affects quantization accuracy:

Per-Layer/Per-Tensor: The entire weight tensor (or all weights in a layer) shares the same set of S and Z. This approach is the simplest, but if the value distribution within the tensor is uneven, it may lead to larger errors.
Per-Channel: For weights in convolutional layers, each output channel uses independent S and Z.
Grouped Quantization: The weight tensor is divided into several groups, with each group using independent S and Z. This is currently a very popular approach in LLM quantization as it achieves a good balance between accuracy and overhead. The group size is a key hyperparameter.

2.6 Common Quantization Paradigms

Post-Training Quantization (PTQ): This is the most commonly used and convenient quantization method. It is performed after the model has been fully trained, without requiring retraining. PTQ typically needs a small calibration dataset to calculate the optimal quantization parameters (S and Z) by analyzing the distribution of weights and activation values.
Quantization-Aware Training (QAT): This simulates the errors introduced by quantization during the model training process. By inserting pseudo-quantization nodes in the forward pass during training, it allows the model to adapt to the accuracy loss caused by quantization. QAT typically achieves higher accuracy than PTQ but requires a complete training process and dataset, making it more costly.

Now that we have the basic knowledge of quantization, let's delve into the specific implementations in llama.cpp and vLLM.

3. Quantization Schemes in llama.cpp

llama.cpp is an efficient LLM inference engine written in C/C++, renowned for its excellent cross-platform performance and support for resource-constrained devices. One of its core advantages is its powerful and flexible quantization support, which revolves around its self-developed GGUF (Georgi Gerganov Universal Format) file format.

3.1 GGUF Format and Quantization

GGUF is a binary format specifically designed for LLMs, used to store model metadata, vocabulary, and weights. A key feature is its native support for various quantized weights, allowing different precision tensors to be mixed within the same file. This enables llama.cpp to directly use quantized weights when loading models, without additional conversion steps.

3.2 Quantization Type Nomenclature in `llama.cpp`

llama.cpp defines a very specific quantization type naming convention, typically in the format Q<bits>_<type>. Understanding these names is key to mastering llama.cpp quantization.

Q: Represents quantization.
<bits>: Indicates the average number of bits per weight, such as 2, 3, 4, 5, 6, 8.
<type>: Indicates the specific quantization method or variant.

Below are some of the most common quantization types and their explanations:

3.2.1 Basic Quantization Types (Legacy)

These are earlier quantization methods, most of which have now been replaced by K-Quants, but are still retained for compatibility.

Q4_0, Q4_1: 4-bit quantization. Q4_1 uses higher precision scale factors than Q4_0, thus typically achieving higher accuracy.
Q5_0, Q5_1: 5-bit quantization.
Q8_0: 8-bit symmetric quantization using block-wise scale factors. This is one of the quantization types closest to the original FP16 precision and often serves as a benchmark for performance and quality.
Q2_K, Q3_K, Q4_K, Q5_K, Q6_K: These are the K-Quants series.

3.2.2 K-Quants (Recommended)

K-Quants is a more advanced and flexible quantization scheme introduced in llama.cpp. They achieve better precision preservation at extremely low bit rates through more refined block structures and the concept of super-blocks.

Block: Weights are divided into fixed-size blocks (typically 256 weights).
Super-block: Multiple blocks form a super-block. More detailed quantization parameters (such as min/max scale factors) are stored at the super-block level.

K-Quants naming typically includes a suffix like _S, _M, _L, indicating different sizes/complexities:

S (Small): The smallest version, typically with the lowest precision.
M (Medium): Medium size, balancing precision and size.
L (Large): The largest version, typically with the highest precision.

Common K-Quants Types:

Q4_K_M: 4-bit K-Quant, medium size. This is currently one of the most commonly used and recommended 4-bit quantization types, achieving a good balance between size and performance.
Q4_K_S: 4-bit K-Quant, small version.
Q5_K_M: 5-bit K-Quant, medium size. Provides better precision than 4-bit while being smaller than Q8_0.
Q6_K: 6-bit K-Quant. Provides very high precision, close to Q8_0, but with a smaller size.
IQ2_XS, IQ2_S, IQ2_XXS: 2-bit quantization variants, where IQ stands for “Inaccurate Quantization,” aimed at extreme model compression but with larger precision loss.

3.3 How to Use the `llama-quantize` Tool

llama.cpp provides a command-line tool called llama-quantize for converting FP32 or FP16 GGUF models to quantized GGUF models.

Basic Usage:

./llama-quantize <input-gguf-file> <output-gguf-file> <quantization-type>

Example: Quantizing an FP16 Model to Q4_K_M

# First, convert the original model (e.g., PyTorch format) to FP16 GGUF
python3 convert.py models/my-model/
# Then, use llama-quantize for quantization
./llama-quantize ./models/my-model/ggml-model-f16.gguf ./models/my-model/ggml-model-Q4_K_M.gguf Q4_K_M

3.4 Importance Matrix

To further reduce precision loss from quantization, llama.cpp introduced the concept of an importance matrix (imatrix). This matrix calculates the importance of each weight by running the model on a calibration dataset. During quantization, llama-quantize references this matrix to apply smaller quantization errors to more important weights, thereby protecting critical information in the model.

Using imatrix for Quantization:

# 1. Generate the importance matrix
./llama-imatrix -m model-f16.gguf -f calibration-data.txt -o imatrix.dat
# 2. Use imatrix for quantization
./llama-quantize --imatrix imatrix.dat model-f16.gguf model-Q4_K_M-imatrix.gguf Q4_K_M

3.5 Summary

llama.cpp's quantization scheme is centered around the GGUF format, providing a rich, efficient, and battle-tested set of quantization types. Its K-Quants series performs exceptionally well in low-bit quantization, and when combined with advanced techniques like importance matrices, it can maximize model performance while significantly compressing the model. For scenarios requiring LLM deployment on CPUs or resource-limited hardware, llama.cpp is an excellent choice.

4. vLLM's Quantization Ecosystem

Unlike llama.cpp's cohesive, self-contained quantization system, vLLM, as a service engine focused on high-performance, high-throughput GPU inference, adopts a “best of all worlds” quantization strategy. vLLM doesn't invent new quantization formats but instead embraces compatibility, supporting and integrating the most mainstream and cutting-edge quantization schemes and tool libraries from academia and industry.

4.1 Mainstream Quantization Schemes Supported by vLLM

vLLM supports directly loading models quantized by various popular algorithms and tool libraries:

4.1.1 GPTQ (General-purpose Post-Training Quantization)

GPTQ is one of the earliest widely applied LLM PTQ algorithms. It quantizes weights column by column and updates weights using Hessian matrix information to minimize quantization error.

Core Idea: Iteratively quantize each column of weights and update the remaining unquantized weights to compensate for errors introduced by already quantized columns.
vLLM Support: Can directly load GPTQ quantized models generated by libraries like AutoGPTQ.
Suitable Scenarios: Pursuing good 4-bit quantization performance with a large number of pre-quantized models available in the community.

4.1.2 AWQ (Activation-aware Weight Quantization)

AWQ observes that not all weights in a model are equally important, with a small portion of “significant weights” having a huge impact on model performance. Similar uneven distributions also exist in activation values.

Core Idea: By analyzing the scale of activation values, identify and protect those “significant weights” that multiply with large activation values, giving them higher precision during quantization. It doesn't quantize activation values but makes weights adapt to the distribution of activation values.
vLLM Support: Can directly load AWQ quantized models generated by the AutoAWQ library.
Suitable Scenarios: Seeking higher model precision than GPTQ at extremely low bits (such as 4-bit), especially when handling complex tasks.

4.1.3 FP8 (8-bit Floating Point)

FP8 is the latest low-precision floating-point format, pushed by hardware manufacturers like NVIDIA. It has a wider dynamic range than traditional INT8, making it more suitable for representing extremely unevenly distributed activation values in LLMs.

Core Idea: Use 8-bit floating-point numbers (typically in E4M3 or E5M2 format) to represent weights and/or activation values.
vLLM Support: Through integration with llm-compressor and AMD's Quark library, vLLM provides strong support for FP8, including both dynamic and static quantization.
Suitable Scenarios: Pursuing ultimate inference speed and throughput on modern GPUs (such as H100) that support FP8 acceleration.

4.1.4 FP8 KV Cache

This is a quantization technique specifically targeting the KV Cache, a major memory consumer during inference.

Core Idea: Quantize the Key-Value cache stored in GPU memory from FP16 or BF16 to FP8, thereby halving this portion of memory usage, allowing the model to support longer context windows or larger batch sizes.
vLLM Support: vLLM provides native support, which can be enabled at startup with the parameter --kv-cache-dtype fp8.

4.1.5 BitsAndBytes

This is a very popular quantization library, known for its ease of use and “on-the-fly” quantization.

Core Idea: Dynamically quantize during model loading, without needing pre-prepared quantized model files.
vLLM Support: vLLM integrates BitsAndBytes, allowing users to easily enable 4-bit quantization by setting the quantization="bitsandbytes" parameter.
Suitable Scenarios: Quick experimentation, user-friendly, avoiding complex offline quantization processes.

4.1.6 Other Schemes

SqueezeLLM: A non-uniform quantization method that believes weight importance is related to numerical size, thus using fewer bits for smaller weight values and more bits for larger weight values.
TorchAO: PyTorch's official quantization tool library, which vLLM is beginning to support.
BitBLAS: A low-level computation library aimed at accelerating low-bit (such as 1-bit, 2-bit, 4-bit) matrix operations through optimized kernel functions.

4.2 How to Use Quantized Models in vLLM

Using quantization in vLLM is very simple, typically just requiring specifying the quantization parameter in the LLM constructor. vLLM will automatically detect the quantization type from the model's configuration file (config.json).

Example: Loading an AWQ Quantized Model

from vllm import LLM
# vLLM will automatically recognize awq quantization from "TheBloke/My-Model-AWQ"'s config.json
llm = LLM(model="TheBloke/My-Model-AWQ", quantization="awq")

Example: Enabling FP8 KV Cache

from vllm import LLM
llm = LLM(model="meta-llama/Llama-2-7b-chat-hf",
kv_cache_dtype="fp8")

5. llama.cpp vs. vLLM: Comparison and Summary

Feature	llama.cpp	vLLM
Target Platform	CPU, Cross-platform, Edge devices	High-performance GPU servers
Core Philosophy	Cohesive, self-contained, extreme optimization	Open, integrated, high throughput
File Format	GGUF (custom format)	Standard Hugging Face format
Quantization Schemes	Built-in `K-Quants`, `IQ`, etc.	Integrates GPTQ, AWQ, FP8, BnB, etc.
Ease of Use	Requires `llama-quantize` conversion	Direct loading, automatic detection
Ecosystem	Self-contained ecosystem	Embraces the entire Python AI ecosystem
Latest Technology	Quickly follows up and implements own versions	Quickly integrates latest open-source libraries

6. Latest Quantization Trends and Outlook

The field of model quantization is still rapidly evolving. Here are some trends worth noting:

1-bit/Binary Neural Networks (BNNs): Ultimate model compression, restricting weights to +1 or -1. Although currently suffering significant precision loss in LLMs, its potential is enormous, with related research emerging constantly.
Non-uniform Quantization: Like SqueezeLLM, dynamically allocating bit numbers based on data distribution, theoretically superior to uniform quantization.
Hardware-Algorithm Co-design: New hardware (such as FP8, FP4, INT4 support) is driving the development of new quantization algorithms, while new algorithms are guiding future hardware design.
Combining Quantization with Sparsification: Combining quantization with sparsification techniques like pruning holds promise for achieving higher rates of model compression.

7. Conclusion

Model quantization is a key technology for addressing the challenges of the large model era. llama.cpp and vLLM represent two different quantization philosophies: llama.cpp provides ultimate local inference performance for resource-constrained devices through its elegant GGUF format and built-in K-Quants; while vLLM has become the king of GPU cloud inference services through its open ecosystem and integration of various cutting-edge quantization schemes.

Understanding the quantization implementations of these two frameworks not only helps us choose the right tool for specific scenarios but also gives us insight into the development trajectory and future directions of the entire LLM inference optimization field.

vLLM Technical Guide: High-Performance LLM Inference Engine

Thu, 26 Jun 2025 01:05:00 +0000

1. Introduction to vLLM

vLLM is an open-source inference and serving engine designed for large language models (LLMs), renowned for its high throughput and memory efficiency. In the field of LLM serving, vLLM addresses a core pain point: traditional inference systems are inefficient when handling the key-value cache (KV Cache) in Transformer models’ attention mechanism, resulting in significant memory waste and limited inference speed.

The memory bottleneck in LLM inference primarily stems from the KV Cache. This cache stores attention keys and values for each previous token in a sequence to accelerate the generation of subsequent tokens. However, the size of the KV Cache is dynamic and difficult to predict, creating enormous challenges for memory management. Traditional systems (like HuggingFace Transformers) typically pre-allocate a large continuous memory space to store the KV Cache, leading to severe memory fragmentation and waste.

vLLM fundamentally solves this problem by introducing its core innovation: the PagedAttention mechanism.

2. Core Features and Advantages

vLLM stands out among numerous LLM inference frameworks thanks to several key features:

Extremely High Throughput: Through PagedAttention and Continuous Batching, vLLM significantly improves GPU utilization. Its throughput is several times higher than HuggingFace Transformers and outperforms other mainstream inference libraries.
Efficient Memory Management: The PagedAttention mechanism divides the KV Cache into non-continuous memory blocks, greatly reducing internal and external memory fragmentation. According to official data, it can save up to 55% of memory, meaning you can load larger models or serve more concurrent requests with the same hardware.
Flexible Decoding Strategies: vLLM supports various complex decoding algorithms, including Parallel Sampling, Beam Search, and Top-K/Top-P sampling, meeting the needs of different application scenarios.
OpenAI API Compatibility: vLLM provides a service endpoint that is fully compatible with the OpenAI API. This means you can seamlessly integrate vLLM into existing application ecosystems built on the OpenAI API with just a few configuration changes.
Distributed Inference: For ultra-large models that cannot fit on a single GPU, vLLM supports Tensor Parallelism, distributing model weights and computational load across multiple GPUs for efficient distributed inference.
Streaming and Structured Output: Supports streaming of generated tokens and can produce structured outputs in specific formats (such as JSON Schema or regular expressions) through Guided Generation.

3. Core Architecture: Deep Dive into PagedAttention

PagedAttention is the soul of vLLM, with its design inspiration coming from the paging technique used in modern operating systems to manage virtual memory.

3.1 Working Principle

In traditional methods, the KV Cache for each sequence is stored in continuous memory space. While this approach seems simple, it leads to severe memory fragmentation due to the vast differences in sequence lengths.

PagedAttention divides each sequence's KV Cache into fixed-size blocks. Each block can store keys and values for a fixed number of tokens. During inference, vLLM's core scheduler dynamically allocates these blocks to sequences as needed.

The advantages of this design include:

Eliminating Internal Fragmentation: Since blocks are of fixed size, a sequence's last block may have some unused space, but this waste is far less than that caused by reserving continuous memory for the entire sequence.
Flexible Memory Allocation: Blocks are stored in non-continuous memory space, making memory management more flexible, similar to how operating systems manage physical memory pages.
Efficient Memory Sharing: PagedAttention makes sharing KV Cache between different sequences exceptionally simple and efficient. For example, in parallel sampling or beam search, multiple candidate sequences originate from the same prompt. vLLM allows these sequences to share KV blocks storing the prompt portion, only needing to allocate new, independent blocks for each sequence when generating new tokens. This “Copy-on-Write” mechanism greatly reduces the memory overhead of complex decoding algorithms.

Below is a Mermaid diagram that more intuitively illustrates PagedAttention's memory management approach:

graph TD
subgraph Physical_Memory [KV Cache Physical Memory]
direction LR
B1(Block 1)
B2(Block 2)
B3(Block 3)
B4(Block 4)
B5(Block 5)
B6(Block 6)
B7(Block 7)
B8(Block 8)
end
subgraph Logical_View [Sequence Logical View]
direction TB
subgraph Seq1 [Sequence 1]
P1(Prompt) --> T1(Token 1)
end
subgraph Seq2 [Sequence 2]
P2(Prompt) --> T2(Token 1) --> T3(Token 2)
end
subgraph Seq3 [Parallel Sampling]
P3(Prompt) --> T4(Token 1a)
P3 --> T5(Token 1b)
end
end
subgraph Block_Table [Block Table]
direction TB
Map1["Seq 1: [B1, B5]"]
Map2["Seq 2: [B2, B6, B8]"]
Map3["Seq 3a: [B3, B7]"]
Map4["Seq 3b: [B3, B4]"]
end
Seq1 --> Map1
Seq2 --> Map2
Seq3 --> Map3
Seq3 --> Map4
Map1 --> B1
Map1 --> B5
Map2 --> B2
Map2 --> B6
Map2 --> B8
Map3 --> B3
Map3 --> B7
Map4 --> B3
Map4 --> B4
style B3 fill:#f9f,stroke:#333,stroke-width:2px
linkStyle 8 stroke-width:2px,stroke:green,fill:none;
linkStyle 11 stroke-width:2px,stroke:green,fill:none;
linkStyle 12 stroke-width:2px,stroke:green,fill:none;

Diagram explanation:

KV Cache Physical Memory: Represents non-continuous physical memory blocks on the GPU.
Sequence Logical View: Represents multiple requests (sequences) being processed.
Block Table: vLLM's core component that maps logical token positions to physical memory blocks.
Memory Sharing: Note that the two branches in “Parallel Sampling” (3a and 3b) share the same Prompt block (B3), demonstrating PagedAttention's efficient memory sharing.

3.2 Continuous Batching

Based on PagedAttention, vLLM implements a more advanced batching strategy—continuous batching. Traditional static batching requires waiting for all sequences in a batch to complete generation before processing the next batch. Continuous batching, however, allows new requests to be inserted into the batch immediately after a sequence in the batch completes generation, avoiding GPU idle waiting and further improving throughput.

Below is a comparison of the two batching methods using a Mermaid sequence diagram:

sequenceDiagram
participant C as Client
participant S as Server
participant G as GPU
note over C, G: --- Static Batching ---
C->>S: Request [R1, R2, R3, R4]
S->>G: Process Batch 1 [R1, R2, R3, R4]
note right of G: All requests process in parallel
G-->>S: Batch 1 Finished
note right of S: Wait for the entire batch to complete
S-->>C: Response [O1, O2, O3, O4]
C->>S: Request [R5, R6]
S->>G: Process Batch 2 [R5, R6]
note over C, G: --- Continuous Batching ---
C->>S: Request [R1, R2, R3, R4]
S->>G: Process [R1, R2, R3, R4]
G-->>S: R2 Finished
S-->>C: Response O2
C->>S: New Request R5
S->>G: Add R5 to queue (GPU is not idle)
note right of G: R1, R3, R4, R5 are now processing
G-->>S: R4 Finished
S-->>C: Response O4

4. Quick Start Guide

Below, we'll demonstrate how to install and use vLLM through a few simple steps.

4.1 Installation

You can install vLLM using either pip or uv (a faster package installation tool). Using uv is recommended as it can automatically detect your CUDA version and install the matching PyTorch backend.

Using uv (recommended):

# Create and activate a virtual environment
uv venv
source .venv/bin/activate
# Install vLLM
uv pip install vllm --torch-backend=auto

Using pip:

pip install vllm

4.2 Offline Inference

The vllm.LLM class makes offline inference very convenient.

from vllm import LLM, SamplingParams
# Define input prompts
prompts = [
"Hello, my name is",
"The capital of France is",
"The future of AI is",
]
# Define sampling parameters
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
# Initialize the LLM engine (model will be automatically downloaded from Hugging Face)
llm = LLM(model="facebook/opt-125m")
# Generate text
outputs = llm.generate(prompts, sampling_params)
# Print results
for output in outputs:
prompt = output.prompt
generated_text = output.outputs[0].text
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

4.3 Launching an OpenAI-Compatible Server

One of vLLM's most powerful features is its built-in API server. With just one command, you can start a service compatible with the OpenAI API.

vllm serve Qwen/Qwen2.5-1.5B-Instruct

By default, the server will run on http://localhost:8000.

4.4 Interacting with the Server

You can interact with the server using curl or the openai Python client.

Using curl:

curl http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-1.5B-Instruct",
"prompt": "San Francisco is a",
"max_tokens": 7,
"temperature": 0
}'

Using the OpenAI Python client:

from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="not-used" # API key is not required
)
completion = client.chat.completions.create(
model="Qwen/Qwen2.5-1.5B-Instruct",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Who won the world series in 2020?"}
]
)
print(completion.choices[0].message)

5. Model Serving

5.1 Distributed Serving

If a model is too large to fit on a single GPU, you can distribute it across multiple GPUs using tensor parallelism.

# Start a service on 4 GPUs
vllm serve facebook/opt-13b --tensor-parallel-size 4

5.2 Docker Deployment

vLLM provides official Docker images for convenient containerized deployment.

docker run --runtime nvidia --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HUGGING_FACE_HUB_TOKEN=<your-hf-token>" \
-p 8000:8000 \
--ipc=host \
vllm/vllm-openai:latest \
--model mistralai/Mistral-7B-v0.1

6. Advanced Features

6.1 Structured Outputs

vLLM supports various ways to constrain the model's output format, which is crucial for applications requiring reliable, parsable outputs.

Generating JSON using Pydantic models:

from pydantic import BaseModel
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")
model = client.models.list().data[0].id
class People(BaseModel):
name: str
age: int
completion = client.chat.completions.create(
model=model,
messages=[
{"role": "user", "content": "Generate a JSON with the name and age of one random person."}
],
response_format={
"type": "json_schema",
"json_schema": {
"name": "people",
"schema": People.model_json_schema()
}
},
)
print(completion.choices[0].message.content)

6.2 LoRA Support

vLLM can efficiently serve multiple LoRA adapters on the same base model. This is particularly useful for scenarios requiring customized models for different customers or tasks.

Starting a server with LoRA support:

from vllm import LLM
llm = LLM(model="meta-llama/Llama-2-7b-hf", enable_lora=True)

Specifying a LoRA adapter in a request:

curl http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "sql-lora", # Specify the LoRA model ID
"prompt": "San Francisco is a",
"max_tokens": 7
}'

6.3 Quantization

Quantization is a technique to reduce model size and memory usage by lowering the precision of model weights. vLLM supports various quantization schemes, such as AWQ and FP8 KV cache.

Enabling FP8 KV cache:

from vllm import LLM
llm = LLM(
model="meta-llama/Llama-2-7b-chat-hf",
kv_cache_dtype="fp8",
calculate_kv_scales=True # Dynamically calculate quantization scales
)

7. Framework Integration

vLLM can be easily integrated with popular LLM application frameworks like Langchain and LlamaIndex for building complex systems such as Retrieval-Augmented Generation (RAG). Typically, vLLM serves as a backend providing fast LLM inference and embedding generation services.

Installing related dependencies:

pip install -U vllm langchain_openai langchain_community

Afterward, in Langchain, you can point the base_url of ChatOpenAI or OpenAIEmbeddings to your vLLM server's address to complete the integration.

8. Conclusion

Through its innovative PagedAttention architecture, vLLM successfully addresses memory management and performance bottlenecks in LLM inference, providing developers with an extremely efficient, flexible, and easy-to-use inference serving engine. Whether conducting quick offline experiments or deploying production-grade, high-concurrency LLM services, vLLM demonstrates excellent performance and powerful functionality. As the community continues to develop, vLLM is becoming one of the standard tools in the field of LLM serving.