Machine Learning | Ziyang Lin

CLIP Technology Analysis: Unified Representation Through Image-Text Contrastive Learning

Fri, 27 Jun 2025 05:00:00 +0000

1. Introduction

CLIP (Contrastive Language-Image Pre-training) is an advanced deep learning model developed by OpenAI, designed to understand the relationship between images and the text that describes them. Through pre-training on millions of (image, text) pairs, CLIP learns a shared multimodal embedding space that maps both images and text to vectors within this space.

The revolutionary aspect of CLIP lies in its powerful Zero-Shot Learning capabilities. Traditional image classification models typically require training for specific tasks and labels, whereas CLIP can classify images into categories it has never explicitly seen during training, greatly enhancing the model's generalization ability and flexibility.

2. Core Concepts

To understand CLIP, we first need to grasp several core concepts:

2.1 Multimodal Learning

Multimodal learning refers to the ability of models to process and associate information from different modalities (such as text, images, audio). Humans understand the world by combining visual, auditory, and linguistic information, and multimodal learning aims to give AI similar capabilities. CLIP is an outstanding example of multimodal learning in the domains of images and text.

2.2 Contrastive Learning

Contrastive learning is a self-supervised learning method. Its core idea is to bring similar samples closer together in the representation space while pushing dissimilar samples apart.

Imagine a large collection of “image-text” pairs. For a given image (e.g., a picture of a cat), its corresponding text description (“a photo of a cat”) is a positive sample, while all other text descriptions (e.g., “a photo of a dog”, “a photo of a car”) are negative samples. CLIP's goal is to learn an encoder that makes the representation of “a cat picture” and “a photo of a cat” very close in the vector space, while keeping representations of unrelated text descriptions far apart.

2.3 Zero-Shot Learning

Zero-shot learning refers to a model's ability to recognize and classify categories it has never seen during training. CLIP achieves this by transforming image classification into an image-text matching problem.

For example, to determine if an image is a “dog,” we don't need a model specifically trained to recognize “dogs.” We simply encode the image into a vector, encode the text “a photo of a dog” into another vector, and then calculate the similarity between these two vectors. If the similarity is high, we can consider the image to be a “dog.” This approach allows CLIP to identify objects of any category, as long as we can describe it in text.

3. Model Architecture

The CLIP model consists of two main components: an image encoder and a text encoder.

Image Encoder: Responsible for converting input images into feature vectors. CLIP uses two mainstream architectures:
- ResNet: A classic convolutional neural network.
- Vision Transformer (ViT): A model that applies the Transformer architecture to image recognition.
Text Encoder: Responsible for converting input text into feature vectors. CLIP uses the standard Transformer architecture.

These two encoders map images and text to the same multi-dimensional embedding space, allowing their vector representations to be directly compared.

graph TD
subgraph "CLIP Model"
direction LR
subgraph "Image Encoder (ViT or ResNet)"
I[Image] --> IE(Encoder) --> IV[Image Feature Vector]
end
subgraph "Text Encoder (Transformer)"
T[Text] --> TE(Encoder) --> TV[Text Feature Vector]
end
end
IV -- "Cosine Similarity" --> S(Similarity Score)
TV -- "Cosine Similarity" --> S

4. Workflow

CLIP's workflow is divided into training and inference phases.

4.1 Training Phase

During the training phase, CLIP learns from a dataset containing hundreds of millions of (image, text) pairs. For a batch of data containing N (image, text) pairs, CLIP performs the following operations:

Encoding: Pass N images through the image encoder to get N image feature vectors, and pass N texts through the text encoder to get N text feature vectors.
Calculate Similarity: Compute the cosine similarity between each of the N image feature vectors and each of the N text feature vectors, resulting in an N x N similarity matrix.
Contrastive Learning: In this matrix, the elements on the diagonal correspond to the correct (image, text) pairs, which we want to have high similarity. Elements off the diagonal represent mismatched pairs, which we want to have low similarity. The model is optimized through a contrastive loss function to achieve this goal.

graph TD
A["Input a batch of (image, text) pairs"] --> B{"Encoding"};
B --> C["Image Encoder"] --> D["Image Feature Vectors"];
B --> E["Text Encoder"] --> F["Text Feature Vectors"];
D & F --> G{"Calculate Cosine Similarity Matrix"};
G --> H["Contrastive Loss Function"];
H --> I["Optimize Model Parameters"];

4.2 Inference Phase (Zero-Shot Classification)

During the inference phase, CLIP can perform zero-shot image classification tasks:

Prepare Text Prompts: For all categories you want to classify (e.g., “cat”, “dog”, “car”), create a series of text prompts such as “a photo of a cat”, “a photo of a dog”, “a photo of a car”.
Encode Text: Convert these text prompts into a series of text feature vectors using the text encoder.
Encode Image: Convert the image to be classified into an image feature vector using the image encoder.
Calculate Similarity: Compute the cosine similarity between the image feature vector and all text feature vectors.
Prediction: The category corresponding to the text prompt with the highest similarity is CLIP's prediction result.

5. Applications

CLIP's powerful capabilities make it widely applicable in many fields:

Zero-Shot Image Classification: Classify images into arbitrary categories without additional training.
Image Retrieval: Search for matching images using natural language descriptions.
Content Moderation: Automatically identify and filter inappropriate image content.
Guiding Generative Models: CLIP's multimodal understanding ability can guide generative models (like DALL-E 2) to create images that match text descriptions.

6. Code Example

Here's a simple Python code example demonstrating how to use the clip library to load the model and obtain image feature vectors.

First, install the necessary libraries:

pip install torch clip

Then, you can use the following code:

import torch
import clip
from PIL import Image
# Load the model, can run on either CPU or GPU
device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)
# Load and preprocess the image
image_path = "cat.jpg" # Replace with your image path
image = preprocess(Image.open(image_path)).unsqueeze(0).to(device)
# Prepare text descriptions
text_descriptions = ["a photo of a cat", "a photo of a dog"]
text_tokens = clip.tokenize(text_descriptions).to(device)
with torch.no_grad():
# Encode images and text
image_features = model.encode_image(image)
text_features = model.encode_text(text_tokens)
# Calculate similarity
logits_per_image, logits_per_text = model(image, text_tokens)
probs = logits_per_image.softmax(dim=-1).cpu().numpy()
print("Label probs:", probs) # Output the matching probability between the image and each text description

7. Conclusion

Through its innovative contrastive learning method, CLIP successfully connects text and images in a shared representation space, demonstrating powerful zero-shot learning capabilities. It has not only achieved excellent results in multiple benchmark tests but has also opened new paths for the development of multimodal artificial intelligence.

Advantages:

Strong generalization ability and zero-shot performance.
No need for fine-tuning for specific tasks, saving significant annotation costs.
Can understand complex and abstract text descriptions.

Limitations:

May perform poorly on very fine-grained classification tasks (such as identifying specific bird species).
Limited understanding of abstract or systematic concepts (such as counting).
The model's performance is highly dependent on the quality and scale of pre-training data.

Despite some limitations, CLIP remains one of the most important breakthroughs in artificial intelligence in recent years and continues to push the boundaries of multimodal research.

Mixture of Experts (MoE): Sparse Activation Architecture for Large-Scale Neural Networks

Fri, 27 Jun 2025 04:02:00 +0000

1. Introduction

Mixture of Experts (MoE) is a neural network architecture that dramatically expands model capacity without significantly increasing computational costs by decomposing large models into multiple smaller “expert” networks and using a “gating” network to dynamically select the most appropriate subset of experts for each input.

This approach draws inspiration from expert systems in human society, where specific problems are directed to relevant specialists. In deep learning, this means the model can learn to route different inputs to expert networks specialized in processing that type of data, enabling more efficient and specialized learning.

2. Core Components: Macro and Micro Analysis

From a macro perspective, MoE layers typically serve as efficient alternatives to standard Feed-Forward Network (FFN) layers in Transformer models. While traditional FFN layers apply identical transformations to every token in a sequence, MoE layers introduce the concept of Conditional Computation: for each token, the model dynamically selects a small subset of “expert” networks to process it, rather than engaging the entire model's parameters. This mechanism allows models to maintain relatively constant computation costs despite having enormous parameter counts.

An MoE layer consists of two core components: Expert Networks and a Gating Network. Below is a visualization of the macro architecture of an MoE layer:

graph LR
A[Input Token] --> B{Gating Network};
B -- Routing Decision --> C1[Expert 1];
B -- Routing Decision --> C2[Expert 2];
B -- ... --> Cn[Expert n];
C1 --> D[Output];
C2 --> D;
Cn --> D;

An MoE layer consists of two core components: Expert Networks and a Gating Network.

2.1. Expert Networks: Specialized Processors

Underlying Structure and Variants

At the foundational level, each “expert” is typically an independent feed-forward neural network (FFN). In standard Transformer architectures, an FFN usually consists of two linear layers and a non-linear activation function (such as GeLU or SwiGLU).

Homogeneous Experts: In most MoE models, all experts share identical network structures. For example, in the Mixtral 8x7B model, each MoE layer contains 8 structurally identical expert FFNs. This design facilitates implementation and optimization.
Heterogeneous Experts: Though less common, experts can theoretically be heterogeneous, using different activation functions, hidden layer dimensions, or even more complex structures (like convolutional layers). This might allow the model to learn more diverse features but increases implementation complexity.

Functional Specialization: From General to Specialized

During training, although all experts start identical, the routing mechanism of the gating network guides them to develop different “specializations.” For example, in natural language processing tasks, after sufficient training, we might observe:

Grammar Experts: Specialized in processing tokens related to sentence structure, parts of speech, etc.
Semantic Experts: Focused on understanding word meanings and contextual relationships.
Domain-Specific Knowledge Experts: For instance, one expert might specialize in “legal” text, while another becomes more sensitive to “biomedical” domain knowledge.

This functional specialization is a key source of MoE models’ efficiency, as it allows the model to process specific types of information with dedicated subnetworks rather than using a single large, general network for all information.

2.2. Gating Network: Intelligent Routing and Dispatch Center

The gating network is the core decision-making unit of MoE, responsible for assigning the most appropriate experts to each input token.

Technical Details

The gating network implementation is typically concise and efficient. Its workflow is as follows:

Generate Logits: For the vector representation x of an input token (typically the output from a self-attention layer), the gating network calculates routing logits through a simple trainable linear layer W_g: logits = einsum("d,de->e", x, W_g), where d is the token dimension and e is the number of experts. This operation produces a vector of length e, with each element representing the “score” for the corresponding expert.
Top-K Routing Mechanism: To achieve sparse computation, tokens are not sent to all experts. The gating network selects the k highest scores from the logits vector. This k value is an important hyperparameter; in Mixtral 8x7B, k=2. This means each token is processed by only the two most relevant experts.
Calculate Gating Weights (Softmax): The selected k logits are normalized through a Softmax function, generating k gating weights that determine how to combine the outputs of these k experts. weights = softmax(top_k_logits)
Calculate Final Output: The input token x is sent to the selected k experts, producing k expert outputs. The final output is the weighted sum of these k expert outputs, with weights being the gating weights calculated in the previous step. output = sum(weights[i] * expert_i(x) for i in top_k_indices)

Below is a visualization of this workflow:

graph TD
A[Input Token x] --> B{Multiply by Gating Weight Matrix W_g};
B --> C{Calculate Logits};
C --> D{Top-K Selection};
D -- k highest scores --> E{Softmax};
E -- Normalized weights --> F[Weighted Sum];
A -- Send to Top-K experts --> G1["Expert i processes x"];
A -- Send to Top-K experts --> G2["Expert j processes x"];
G1 --> F;
G2 --> F;
F --> H[Final Output];

Key Challenge: Load Balancing

A critical challenge for the gating network is the “Matthew Effect”: some experts may receive more training opportunities due to slightly higher initial weights, becoming stronger and subsequently being selected more frequently, causing other experts to be “starved.” To address this issue, MoE introduces an Auxiliary Load Balancing Loss.

Principle: This loss function aims to encourage the gating network to distribute tokens as evenly as possible across all experts. It's typically implemented by calculating the square sum of the proportion of tokens assigned to each expert in a batch, multiplied by an adjustable hyperparameter α. The loss value increases as the distribution becomes more unbalanced.
Optimization: This auxiliary loss is added to the model's main task loss (such as cross-entropy loss for language models) to form the final total loss function. By optimizing both losses during backpropagation, the model is incentivized to maintain load balance among experts while completing its main task.

3. MoE Model Training Methods: Addressing Scale Challenges

Due to the enormous parameter count of MoE models (despite sparse computation), their training poses significant challenges to computational resources, especially memory. To effectively train MoE models, complex parallelization strategies must be employed.

3.1. Expert Parallelism

This is the core parallelization strategy for training MoE models.

Core Idea: Distribute different experts across different computing devices (such as GPUs). For example, in a scenario with an MoE layer containing 8 experts and 8 GPUs, each GPU is responsible for storing and computing one expert. Other parts of the model (such as self-attention layers) can be replicated on each GPU.
Workflow and Communication Overhead: In each forward pass, tokens from various GPUs, after being processed by the gating network, need to be sent to the GPUs storing the corresponding experts based on routing decisions. This process involves a global All-to-All communication operation, where each GPU needs to send and receive data to and from all other GPUs. After computation, results are sent back to the original GPUs through another All-to-All communication. This intensive communication is the main performance bottleneck in expert parallel mode.

3.2. Combining with Other Parallelism Strategies

To address different scales of models and hardware configurations, expert parallelism often needs to be combined with other parallelism strategies:

Data Parallelism: This is the most common parallelism approach. When the number of GPUs exceeds the number of experts, multiple GPUs can form a data parallel group, with each group containing a complete set of experts (distributed through expert parallelism). For example, with 64 GPUs and 8 experts, 8 data parallel groups can be created, each with 8 GPUs, with each GPU responsible for one expert.
Model Parallelism and Pipeline Parallelism: For ultra-large models where even a single expert or non-MoE layer cannot fit into a single GPU, tensor model parallelism and pipeline parallelism need to be introduced to further split the model.

In summary, training MoE is a complex multi-dimensional parallel engineering task that requires careful design of parallelism strategies based on factors such as model size, number of experts, number of GPUs, and network bandwidth.

4. Advantages of MoE

Enormous Model Capacity: MoE allows models to have massive parameters (e.g., trillions of parameters) without needing to compute all parameters in each forward pass. This enables the model to learn more complex and detailed knowledge.
Controllable Computational Cost: Due to the sparse activation strategy (activating only a few experts), the training and inference costs of MoE models are comparable to dense models with far fewer total parameters.
Faster Training and Inference: Under the same computational budget, MoE models typically converge faster and have faster inference speeds compared to dense models.

5. Challenges of MoE

Training Instability: The gating network may tend to always select a few “popular” experts, preventing other experts from being adequately trained. To address this issue, a “load balancing loss” is typically introduced to encourage the gating network to distribute inputs evenly across all experts.
High Communication Cost: In distributed training, since different experts may be distributed across different computing devices, routing input data from the gating network to selected experts incurs significant communication overhead.
Complex Implementation: Compared to standard dense models, MoE models are more complex to implement and deploy, requiring specialized parallel computing strategies and hardware support.
Memory Consumption: Although computation is sparse, all parameters of the model (all experts) need to be stored in memory, placing high demands on hardware.

6. Key Technologies and Recent Advances

Switch Transformers: This is a simplified MoE architecture proposed by Google that simplifies the top-k strategy to top-1, meaning each token is routed to only one expert. This design greatly simplifies routing logic and reduces communication costs.
GShard: This is a system for training MoE models on ultra-large-scale clusters. It effectively addresses the communication bottleneck in MoE training through clever data and model parallelism strategies.
Expert Capacity Factor: To handle load imbalance issues, a “capacity” can be set for each expert, defining the maximum number of tokens it can process in a batch. If an expert is selected more times than its capacity, excess tokens will be “dropped” or routed to other experts.
Latest Routing Strategies: Researchers are exploring more advanced routing strategies, such as allowing tokens to be routed to multiple experts with weighted combination of their outputs, or using more complex gating networks to make smarter routing decisions.
Applications in Computer Vision: MoE is not limited to NLP; it has also been successfully applied to computer vision tasks such as pose estimation, enhancing model performance by training specialized experts for different datasets or pose types.

7. Summary and Outlook

MoE models have successfully achieved massive model scaling at controllable computational costs by introducing sparsely activated expert networks, becoming a key technology for building ultra-large-scale language and vision models.

Despite challenges in training stability and communication overhead, with the continued maturation of technologies like Switch Transformers and GShard, as well as the emergence of new routing strategies and hardware optimizations, the application prospects for MoE are increasingly broad. In the future, we can expect to see more, larger, and more efficient MoE models playing important roles across various domains.

LLM Hyperparameter Tuning Guide: A Comprehensive Analysis from Generation to Deployment

Fri, 27 Jun 2025 03:00:00 +0000

Introduction

Behind the powerful capabilities of large language models (LLMs) is a series of complex hyperparameters working silently. Whether you're deploying a local inference service like vLLM or calling OpenAI's API, precisely tuning these parameters is crucial for achieving ideal performance, cost, and output quality. This document provides a detailed analysis of two key categories of hyperparameters: Generation (Sampling) Parameters and Deployment (Serving) Parameters, helping you fully master their functions, values, impacts, and best practices across different scenarios.

Part 1: Generation (Sampling) Parameters — Controlling Model Creativity and Determinism

Generation parameters directly control the model's behavior when generating the next token. They primarily revolve around a core question: how to select from thousands of possible next words in the probability distribution provided by the model.

1. `temperature`

In one sentence: Controls the randomness of generated text. Higher temperature increases randomness, making responses more creative and diverse; lower temperature decreases randomness, making responses more deterministic and conservative.

Underlying Principle: When generating the next token, the model calculates logits (raw, unnormalized prediction scores) for all words in the vocabulary. Typically, we use the Softmax function to convert these logits into a probability distribution. The temperature parameter is introduced before the Softmax calculation, “smoothing” or “sharpening” this probability distribution.

The standard Softmax formula is: P(i) = exp(logit_i) / Σ_j(exp(logit_j))

With temperature (T) introduced, the formula becomes: P(i) = exp(logit_i / T) / Σ_j(exp(logit_j / T))
- When T -> 0, the differences in logit_i / T become dramatically amplified. The token with the highest logit approaches a probability of 1, while all other tokens approach 0. This causes the model to almost always choose the most likely word, behaving very deterministically and “greedily.”
- When T = 1, the formula reverts to standard Softmax, and the model behaves in its “original” state.
- When T > 1, the differences in logit_i / T are reduced. Tokens with originally lower probabilities get boosted, making the entire probability distribution “flatter.” This increases the chance of selecting less common words, introducing more randomness and creativity.
Value Range and Recommendations:
- Range: [0.0, 2.0] (theoretically can be higher, but OpenAI API typically limits to 2.0).
- temperature = 0.0: Suitable for scenarios requiring deterministic, reproducible, and highly accurate outputs. Examples: code generation, factual Q&A, text classification, data extraction. With identical inputs, outputs will be almost identical (unless the model itself is updated).
- Low temperature (e.g., 0.1 - 0.4): Suitable for semi-creative tasks requiring rigor and fidelity to source material. Examples: article summarization, translation, customer service bots. Outputs will vary slightly but remain faithful to core content.
- Medium temperature (e.g., 0.5 - 0.8): A good balance between creativity and consistency, recommended as the default for most applications. Examples: writing emails, marketing copy, brainstorming.
- High temperature (e.g., 0.9 - 1.5): Suitable for highly creative tasks. Examples: poetry writing, story creation, dialogue script generation. Outputs will be very diverse and sometimes surprising, but may occasionally produce meaningless or incoherent content.
Note:
- It's generally not recommended to modify both temperature and top_p simultaneously; it's better to adjust just one. OpenAI's documentation explicitly states that modifying only one is typically advised.

2. `top_p` (Nucleus Sampling)

In one sentence: Controls generation diversity by dynamically determining the sampling pool size through a cumulative probability threshold (p) of the highest probability tokens.

Underlying Principle: top_p is a more intelligent sampling strategy than temperature, also known as Nucleus Sampling. Instead of adjusting all token probabilities, it directly defines a “core” candidate set.

The specific steps are as follows:
1. The model calculates the probability distribution for all candidate tokens.
2. All tokens are sorted by probability from highest to lowest.
3. Starting from the highest probability token, their probabilities are cumulatively added until this sum exceeds the set top_p threshold.
4. All tokens included in this cumulative sum form the “nucleus” for sampling.
5. The model will only sample from this nucleus (typically renormalizing their probabilities), and all other tokens are ignored.
Example: Assume top_p = 0.9.
- If the highest probability token “the” has a probability of 0.95, then the nucleus will contain only “the”, and the model will choose it 100%.
- If “the” has a probability of 0.5, “a” has 0.3, and “an” has 0.1, then the cumulative probability of these three words is 0.9. The nucleus will contain {“the”, “a”, “an”}. The model will sample from these three words according to their (renormalized) probabilities.
Value Range and Recommendations:
- Range: (0.0, 1.0].
- top_p = 1.0: Means the model considers all tokens without any truncation (equivalent to no top_p).
- High top_p (e.g., 0.9 - 1.0): Allows for more diverse choices, suitable for creative tasks, similar in effect to higher temperature.
- Low top_p (e.g., 0.1 - 0.3): Greatly restricts the model's range of choices, making its output very deterministic and conservative, similar in effect to extremely low temperature.
- General Recommended Value: 0.9 is a very common default value as it maintains high quality while allowing for some diversity.
top_p vs temperature:
- top_p is more dynamic and adaptive. When the model is very confident about the next step (sharp probability distribution), top_p automatically narrows the candidate set, ensuring quality. When the model is less confident (flat distribution), it expands the candidate set, increasing diversity.
- temperature adjusts the entire distribution “equally,” regardless of whether the distribution itself is sharp or flat.
- Therefore, top_p is generally considered a safer and more robust method for controlling diversity than temperature.

3. `top_k`

In one sentence: Simply and directly samples only from the k tokens with the highest probabilities.

Underlying Principle: This is the simplest truncation sampling method. It directly selects the k tokens with the highest probabilities to form the candidate set, then samples from these k tokens. All other tokens are ignored.
Value Range and Recommendations:
- Range: Integers, such as 1, 10, 50.
- top_k = 1: Equivalent to greedy search, always choosing the most likely word.
- Recommendation: top_k is typically not the preferred sampling strategy because it's too “rigid.” In cases where the probability distribution is very flat, it might accidentally exclude many reasonable words; while in cases where the distribution is very sharp, it might include many extremely low-probability, useless words. top_p is usually a better choice.

4. `repetition_penalty`

In one sentence: Applies a penalty to tokens that have already appeared in the context, reducing their probability of being selected again, thereby reducing repetitive content.

Underlying Principle: After calculating logits but before Softmax, this parameter iterates through all candidate tokens. If a token has already appeared in the previous context, its logit value is reduced (typically divided by the value of repetition_penalty).

new_logit = logit / penalty (if token has appeared) new_logit = logit (if token has not appeared)

This way, the final probability of words that have already appeared decreases.
Value Range and Recommendations:
- Range: 1.0 to 2.0 is common.
- 1.0: No penalty applied (default value).
- 1.1 - 1.3: A relatively safe range that can effectively reduce unnecessary repetition without overly affecting normal language expression (such as necessary articles like “the”).
- Too High Values: May cause the model to deliberately avoid common words, producing unnatural or even strange sentences.

5. `frequency_penalty` & `presence_penalty`

These two parameters are more refined versions of repetition_penalty.

presence_penalty:
- Function: Applies a fixed penalty to all tokens that have appeared at least once in the context. It doesn't care how many times the token has appeared; as long as it has appeared, it gets penalized.
- Underlying Principle: new_logit = logit - presence_penalty (if token has appeared at least once).
- Scenario: This parameter is useful when you want to encourage the model to introduce entirely new concepts and vocabulary, rather than repeatedly discussing topics that have already been mentioned.
- Range: 0.0 to 2.0. Positive values penalize new tokens, negative values encourage them.
frequency_penalty:
- Function: The penalty is proportional to the frequency of the token in the context. The more times a word appears, the heavier the penalty it receives.
- Underlying Principle: new_logit = logit - count(token) * frequency_penalty.
- Scenario: This parameter is effective when you find the model tends to repeatedly use certain specific high-frequency words (even if they are necessary), leading to monotonous language.
- Range: 0.0 to 2.0.
Summary: presence_penalty addresses the question of “whether it has appeared,” while frequency_penalty addresses “how many times it has appeared.”

6. `seed`

In one sentence: By providing a fixed seed, you can make the model's output reproducible when other parameters (such as temperature) remain the same.

Function: In machine learning, many operations that seem random are actually “pseudo-random,” determined by an initial “seed.” Setting the same seed will produce the same sequence of random numbers. In LLMs, this means the sampling process will be completely deterministic.
Scenarios:
- Debugging and Testing: When you need to verify whether a change has affected the output, fixing the seed can eliminate randomness interference.
- Reproducible Research: Reproducibility is crucial in academic research.
- Generating Consistent Content: When you need the model to consistently produce outputs in the same style for the same input.
Note: For complete reproduction, all generation parameters (prompt, model, temperature, top_p, etc.) must be identical.

Part 2: Deployment (Serving) Parameters — Optimizing Service Performance and Capacity

Deployment parameters determine how an LLM inference service manages GPU resources, handles concurrent requests, and optimizes overall throughput and latency. These parameters are particularly important in high-performance inference engines like vLLM.

1. `gpu_memory_utilization`

In one sentence: Controls the proportion of GPU memory that vLLM can use, with the core purpose of reserving space for the KV Cache.

Underlying Principle (PagedAttention): The core of vLLM is the PagedAttention mechanism. Traditional attention mechanisms pre-allocate a continuous, maximum-length memory space for each request to store the Key-Value (KV) Cache. This leads to severe memory waste, as most requests are far shorter than the maximum length.

PagedAttention manages the KV Cache like virtual memory in an operating system:
1. It breaks down each sequence's KV Cache into many small, fixed-size “blocks.”
2. These blocks can be stored non-contiguously in GPU memory.
3. A central “Block Manager” is responsible for allocating and releasing these blocks.
gpu_memory_utilization tells vLLM: “You can use this much proportion of the total GPU memory for free management (mainly storing model weights and physical blocks of KV Cache).”
Value Range and Impact:
- Range: (0.0, 1.0].
- Default Value: 0.9 (i.e., 90%).
- Higher Values (e.g., 0.95):
  - Advantage: vLLM has more memory for KV Cache, supporting longer contexts and larger batch sizes, thereby increasing throughput.
  - Risk: If set too high, there might not be enough spare memory for CUDA kernels, drivers, or other system processes, easily leading to OOM (Out of Memory) errors.
- Lower Values (e.g., 0.8):
  - Advantage: Safer, less prone to OOM, reserves more memory for the system and other applications.
  - Disadvantage: Reduced available space for KV Cache, potentially causing vLLM to struggle with high concurrency or long sequence requests, degrading performance. When KV Cache is insufficient, vLLM triggers Preemption, swapping out some running sequences and waiting to swap them back in when there's enough space, severely affecting latency. vLLM's warning log "there is not enough KV cache space. This can affect the end-to-end performance." is reminding you of this issue.
Recommendations:
- Start with the default value of 0.9.
- If you encounter OOM, gradually lower this value.
- If you encounter many preemption warnings and confirm no other processes are occupying large amounts of GPU memory, you can gradually increase this value.

2. `max_num_seqs`

In one sentence: Limits the maximum number of sequences (requests) that the vLLM scheduler can process in one iteration (or one batch).

Underlying Principle: vLLM's scheduler selects a batch of requests from the waiting queue in each processing cycle. This parameter directly limits the size of this “batch.” Together with max_num_batched_tokens (which limits the total number of tokens across all sequences in a batch), it determines the scale of batch processing.
Value Range and Impact:
- Range: Positive integers, such as 16, 64, 256.
- Higher Values:
  - Advantage: Allows for higher concurrency, potentially improving GPU utilization and overall throughput.
  - Disadvantage: Requires more intermediate memory (e.g., for storing logits and sampling states) and may increase the latency of individual batches. If set too high, even if KV Cache still has space, OOM might occur due to insufficient temporary memory.
- Lower Values:
  - Advantage: More memory-friendly, potentially lower latency for individual batches.
  - Disadvantage: Limits concurrency capability, potentially leading to underutilization of GPU and decreased throughput.
Recommendations:
- This value needs to be adjusted based on your GPU memory size, model size, and expected concurrent load.
- For high-concurrency scenarios, try gradually increasing this value while monitoring GPU utilization and memory usage.
- For interactive, low-latency scenarios, consider setting this value lower.

3. `max_model_len`

In one sentence: Sets the maximum context length the model can process (including both prompt and generated tokens).

Underlying Principle: This parameter directly determines how much logical space vLLM needs to reserve for the KV Cache. For example, if max_model_len = 4096, vLLM must ensure its memory management mechanism can support storing KV pairs for up to 4096 tokens per sequence. This affects vLLM's memory planning at startup, such as the size of Position Embeddings.
Value Range and Impact:
- Range: Positive integers, cannot exceed the maximum length the model was originally trained on.
- Higher Values:
  - Advantage: Can handle longer documents and more complex contexts.
  - Disadvantage: Significantly increases memory consumption. Each token needs to store KV Cache; doubling the length roughly doubles the memory usage. Even if current requests are short, vLLM needs to prepare for potentially long requests, which occupies more KV Cache blocks.
- Lower Values:
  - Advantage: Significantly saves GPU memory. If you know your application scenario will never exceed 1024 tokens, setting this value to 1024 instead of the default 4096 or 8192 will free up a large amount of KV Cache space, supporting higher concurrency.
  - Disadvantage: Any requests exceeding this length will be rejected or truncated.
Recommendations:
- Set as needed! This is one of the most effective parameters for optimizing vLLM memory usage. Based on your actual application scenario, set this value to a reasonable maximum with some margin.

4. `tensor_parallel_size` & `pipeline_parallel_size`

These two parameters are used for deploying extremely large models across multiple GPUs or nodes.

tensor_parallel_size:
- Function: Divides each layer of the model (such as a large weight matrix) into N parts (N = tensor_parallel_size), placing them on N different GPUs. During computation, each GPU only processes its own portion of the data, then exchanges necessary results through high-speed interconnects (like NVLink) via All-Reduce operations, finally merging to get the complete output.
- Scenario: Used when a single model's volume exceeds the memory of a single GPU. For example, a 70B model cannot fit into a single 40GB A100, but can be deployed across two A100s by setting tensor_parallel_size=2.
- Impact:
  - Advantage: Achieves model parallelism, solving the problem of models not fitting on a single card.
  - Disadvantage: Introduces significant cross-GPU communication overhead, potentially affecting latency. Requires high-speed interconnects between GPUs.
pipeline_parallel_size:
- Function: Assigns different layers of the model to different GPUs or nodes. For example, placing layers 1-10 on GPU 1, layers 11-20 on GPU 2, and so on. Data flows through these GPUs like a pipeline.
- Scenario: Used when the model is extremely large and needs to be deployed across multiple nodes (machines).
- Impact:
  - Advantage: Can scale the model to any number of GPUs/nodes.
  - Disadvantage: Creates “pipeline bubbles” as additional overhead, where some GPUs are idle during the start and end phases of the pipeline, reducing utilization.
Combined Use: vLLM supports using both parallelism strategies simultaneously for efficient deployment of giant models on large clusters.

Summary and Best Practices

Scenario	`temperature`	`top_p`	`repetition_penalty`	`gpu_memory_utilization`	`max_num_seqs`	`max_model_len`
Code Generation/Factual Q&A	`0.0` - `0.2`	(Not recommended to modify)	`1.0`	`0.9` (Default)	Adjust based on concurrency	Set as needed
Article Summarization/Translation	`0.2` - `0.5`	(Not recommended to modify)	`1.1`	`0.9`	Adjust based on concurrency	Set to maximum possible document length
General Chat/Copywriting	`0.7` (Default)	`0.9` (Recommended)	`1.1` - `1.2`	`0.9`	Adjust based on concurrency	Set as needed, e.g., `4096`\|
Creative Writing/Brainstorming	`0.8` - `1.2`	`0.95`	`1.0`	`0.9`	Adjust based on concurrency	Set as needed
High Concurrency Throughput Optimization	(Task dependent)	(Task dependent)	(Task dependent)	Try `0.9` - `0.95`	Gradually increase	Set to the minimum value that meets business needs
Low Latency Interaction Optimization	(Task dependent)	(Task dependent)	(Task dependent)	`0.9` (Default)	Set to lower values (e.g., `16-64`)	Set as needed
Extremely Memory Constrained	(Task dependent)	(Task dependent)	(Task dependent)	Lower to `0.8`	Set to lower values	Set to the minimum value that meets business needs

Final Recommendations:

Start with Generation Parameters: First adjust temperature or top_p to achieve satisfactory output quality.
Set Deployment Parameters as Needed: When deploying, first set max_model_len to a reasonable minimum value based on your application scenario.
Monitor and Iterate: Start with the default gpu_memory_utilization=0.9 and a moderate max_num_seqs. Observe memory usage and preemption situations through monitoring tools (such as nvidia-smi and vLLM logs), then gradually adjust these values to find the optimal balance for your specific hardware and workload.

Model Quantization Guide: A Comprehensive Analysis from Theory to Practice

Fri, 27 Jun 2025 00:00:00 +0000

1. Introduction

As large language models (LLMs) continue to grow in scale and complexity, their deployment and inference costs have become increasingly expensive. Model quantization, as a key optimization technique, significantly reduces model storage requirements, memory consumption, and computational load by lowering the numerical precision of model weights and activation values, enabling efficient inference on resource-constrained devices such as mobile and edge devices.

This document aims to provide a clear and comprehensive introduction to the core concepts of deep learning model quantization, mainstream approaches, and specific implementations in two leading inference frameworks—llama.cpp and vLLM. We will explore in detail the quantization types they support, underlying principles, usage methods, and future trends in quantization technology.

2. Quantization Fundamentals

Before diving into specific frameworks, we need to understand some basic concepts of quantization.

2.1 What is Model Quantization?

Model quantization refers to the process of converting floating-point numbers in a model (typically 32-bit floating-point, or FP32) to integers with fewer bits (such as INT8, INT4) or lower-precision floating-point numbers (such as FP16, FP8). This process is essentially a form of information compression that attempts to significantly reduce model complexity while preserving model accuracy as much as possible.

2.2 Why is Quantization Needed?

Reduced Model Size: Lower bit-width numerical representations can significantly reduce the size of model files. For example, quantizing an FP32 model to INT8 can reduce the model size by approximately 4 times.
Lower Memory Bandwidth: Smaller data types mean less bandwidth is occupied when transferring data between memory and computational units, which is crucial for memory bandwidth-sensitive hardware.
Accelerated Computation: Many modern processors (CPUs, GPUs, TPUs) support integer operations more efficiently than floating-point operations, providing higher throughput and lower latency.
Reduced Power Consumption: Integer operations typically consume less energy than floating-point operations.

2.3 Quantization Principles: Mapping and Dequantization

The core of quantization is mapping a larger range of floating-point values to a smaller range of fixed-point integer values. This process is defined by the following formula:

Q(r) = round(r / S + Z)

Where:

r is the original floating-point value.
Q(r) is the quantized integer value.
S is the Scale factor, representing the floating-point value size corresponding to each quantized integer step.
Z is the Zero-point, representing the quantized integer value corresponding to floating-point zero.

When performing calculations, the quantized values need to be dequantized back to the floating-point domain:

r' = S * (Q(r) - Z)

r' is the dequantized floating-point number, which has some quantization error compared to the original value r.

2.4 Symmetric vs. Asymmetric Quantization

Based on the choice of zero-point, quantization can be divided into two modes:

Symmetric Quantization: Maps the floating-point range [-abs_max, abs_max] symmetrically to the integer range. In this mode, the zero-point Z is typically 0 (for signed integers) or 2^(bits-1) (for unsigned integer offset). Computation is relatively simple.
Asymmetric Quantization: Maps the complete floating-point range [min, max] to the integer range. In this mode, the zero-point Z is a floating-point number that can be adjusted according to data distribution. It can more accurately represent asymmetrically distributed data but is slightly more complex in computation.

2.5 Per-Layer vs. Per-Group/Per-Channel Quantization

The granularity of calculating scale factor S and zero-point Z also affects quantization accuracy:

Per-Layer/Per-Tensor: The entire weight tensor (or all weights in a layer) shares the same set of S and Z. This approach is the simplest, but if the value distribution within the tensor is uneven, it may lead to larger errors.
Per-Channel: For weights in convolutional layers, each output channel uses independent S and Z.
Grouped Quantization: The weight tensor is divided into several groups, with each group using independent S and Z. This is currently a very popular approach in LLM quantization as it achieves a good balance between accuracy and overhead. The group size is a key hyperparameter.

2.6 Common Quantization Paradigms

Post-Training Quantization (PTQ): This is the most commonly used and convenient quantization method. It is performed after the model has been fully trained, without requiring retraining. PTQ typically needs a small calibration dataset to calculate the optimal quantization parameters (S and Z) by analyzing the distribution of weights and activation values.
Quantization-Aware Training (QAT): This simulates the errors introduced by quantization during the model training process. By inserting pseudo-quantization nodes in the forward pass during training, it allows the model to adapt to the accuracy loss caused by quantization. QAT typically achieves higher accuracy than PTQ but requires a complete training process and dataset, making it more costly.

Now that we have the basic knowledge of quantization, let's delve into the specific implementations in llama.cpp and vLLM.

3. Quantization Schemes in llama.cpp

llama.cpp is an efficient LLM inference engine written in C/C++, renowned for its excellent cross-platform performance and support for resource-constrained devices. One of its core advantages is its powerful and flexible quantization support, which revolves around its self-developed GGUF (Georgi Gerganov Universal Format) file format.

3.1 GGUF Format and Quantization

GGUF is a binary format specifically designed for LLMs, used to store model metadata, vocabulary, and weights. A key feature is its native support for various quantized weights, allowing different precision tensors to be mixed within the same file. This enables llama.cpp to directly use quantized weights when loading models, without additional conversion steps.

3.2 Quantization Type Nomenclature in `llama.cpp`

llama.cpp defines a very specific quantization type naming convention, typically in the format Q<bits>_<type>. Understanding these names is key to mastering llama.cpp quantization.

Q: Represents quantization.
<bits>: Indicates the average number of bits per weight, such as 2, 3, 4, 5, 6, 8.
<type>: Indicates the specific quantization method or variant.

Below are some of the most common quantization types and their explanations:

3.2.1 Basic Quantization Types (Legacy)

These are earlier quantization methods, most of which have now been replaced by K-Quants, but are still retained for compatibility.

Q4_0, Q4_1: 4-bit quantization. Q4_1 uses higher precision scale factors than Q4_0, thus typically achieving higher accuracy.
Q5_0, Q5_1: 5-bit quantization.
Q8_0: 8-bit symmetric quantization using block-wise scale factors. This is one of the quantization types closest to the original FP16 precision and often serves as a benchmark for performance and quality.
Q2_K, Q3_K, Q4_K, Q5_K, Q6_K: These are the K-Quants series.

3.2.2 K-Quants (Recommended)

K-Quants is a more advanced and flexible quantization scheme introduced in llama.cpp. They achieve better precision preservation at extremely low bit rates through more refined block structures and the concept of super-blocks.

Block: Weights are divided into fixed-size blocks (typically 256 weights).
Super-block: Multiple blocks form a super-block. More detailed quantization parameters (such as min/max scale factors) are stored at the super-block level.

K-Quants naming typically includes a suffix like _S, _M, _L, indicating different sizes/complexities:

S (Small): The smallest version, typically with the lowest precision.
M (Medium): Medium size, balancing precision and size.
L (Large): The largest version, typically with the highest precision.

Common K-Quants Types:

Q4_K_M: 4-bit K-Quant, medium size. This is currently one of the most commonly used and recommended 4-bit quantization types, achieving a good balance between size and performance.
Q4_K_S: 4-bit K-Quant, small version.
Q5_K_M: 5-bit K-Quant, medium size. Provides better precision than 4-bit while being smaller than Q8_0.
Q6_K: 6-bit K-Quant. Provides very high precision, close to Q8_0, but with a smaller size.
IQ2_XS, IQ2_S, IQ2_XXS: 2-bit quantization variants, where IQ stands for “Inaccurate Quantization,” aimed at extreme model compression but with larger precision loss.

3.3 How to Use the `llama-quantize` Tool

llama.cpp provides a command-line tool called llama-quantize for converting FP32 or FP16 GGUF models to quantized GGUF models.

Basic Usage:

./llama-quantize <input-gguf-file> <output-gguf-file> <quantization-type>

Example: Quantizing an FP16 Model to Q4_K_M

# First, convert the original model (e.g., PyTorch format) to FP16 GGUF
python3 convert.py models/my-model/
# Then, use llama-quantize for quantization
./llama-quantize ./models/my-model/ggml-model-f16.gguf ./models/my-model/ggml-model-Q4_K_M.gguf Q4_K_M

3.4 Importance Matrix

To further reduce precision loss from quantization, llama.cpp introduced the concept of an importance matrix (imatrix). This matrix calculates the importance of each weight by running the model on a calibration dataset. During quantization, llama-quantize references this matrix to apply smaller quantization errors to more important weights, thereby protecting critical information in the model.

Using imatrix for Quantization:

# 1. Generate the importance matrix
./llama-imatrix -m model-f16.gguf -f calibration-data.txt -o imatrix.dat
# 2. Use imatrix for quantization
./llama-quantize --imatrix imatrix.dat model-f16.gguf model-Q4_K_M-imatrix.gguf Q4_K_M

3.5 Summary

llama.cpp's quantization scheme is centered around the GGUF format, providing a rich, efficient, and battle-tested set of quantization types. Its K-Quants series performs exceptionally well in low-bit quantization, and when combined with advanced techniques like importance matrices, it can maximize model performance while significantly compressing the model. For scenarios requiring LLM deployment on CPUs or resource-limited hardware, llama.cpp is an excellent choice.

4. vLLM's Quantization Ecosystem

Unlike llama.cpp's cohesive, self-contained quantization system, vLLM, as a service engine focused on high-performance, high-throughput GPU inference, adopts a “best of all worlds” quantization strategy. vLLM doesn't invent new quantization formats but instead embraces compatibility, supporting and integrating the most mainstream and cutting-edge quantization schemes and tool libraries from academia and industry.

4.1 Mainstream Quantization Schemes Supported by vLLM

vLLM supports directly loading models quantized by various popular algorithms and tool libraries:

4.1.1 GPTQ (General-purpose Post-Training Quantization)

GPTQ is one of the earliest widely applied LLM PTQ algorithms. It quantizes weights column by column and updates weights using Hessian matrix information to minimize quantization error.

Core Idea: Iteratively quantize each column of weights and update the remaining unquantized weights to compensate for errors introduced by already quantized columns.
vLLM Support: Can directly load GPTQ quantized models generated by libraries like AutoGPTQ.
Suitable Scenarios: Pursuing good 4-bit quantization performance with a large number of pre-quantized models available in the community.

4.1.2 AWQ (Activation-aware Weight Quantization)

AWQ observes that not all weights in a model are equally important, with a small portion of “significant weights” having a huge impact on model performance. Similar uneven distributions also exist in activation values.

Core Idea: By analyzing the scale of activation values, identify and protect those “significant weights” that multiply with large activation values, giving them higher precision during quantization. It doesn't quantize activation values but makes weights adapt to the distribution of activation values.
vLLM Support: Can directly load AWQ quantized models generated by the AutoAWQ library.
Suitable Scenarios: Seeking higher model precision than GPTQ at extremely low bits (such as 4-bit), especially when handling complex tasks.

4.1.3 FP8 (8-bit Floating Point)

FP8 is the latest low-precision floating-point format, pushed by hardware manufacturers like NVIDIA. It has a wider dynamic range than traditional INT8, making it more suitable for representing extremely unevenly distributed activation values in LLMs.

Core Idea: Use 8-bit floating-point numbers (typically in E4M3 or E5M2 format) to represent weights and/or activation values.
vLLM Support: Through integration with llm-compressor and AMD's Quark library, vLLM provides strong support for FP8, including both dynamic and static quantization.
Suitable Scenarios: Pursuing ultimate inference speed and throughput on modern GPUs (such as H100) that support FP8 acceleration.

4.1.4 FP8 KV Cache

This is a quantization technique specifically targeting the KV Cache, a major memory consumer during inference.

Core Idea: Quantize the Key-Value cache stored in GPU memory from FP16 or BF16 to FP8, thereby halving this portion of memory usage, allowing the model to support longer context windows or larger batch sizes.
vLLM Support: vLLM provides native support, which can be enabled at startup with the parameter --kv-cache-dtype fp8.

4.1.5 BitsAndBytes

This is a very popular quantization library, known for its ease of use and “on-the-fly” quantization.

Core Idea: Dynamically quantize during model loading, without needing pre-prepared quantized model files.
vLLM Support: vLLM integrates BitsAndBytes, allowing users to easily enable 4-bit quantization by setting the quantization="bitsandbytes" parameter.
Suitable Scenarios: Quick experimentation, user-friendly, avoiding complex offline quantization processes.

4.1.6 Other Schemes

SqueezeLLM: A non-uniform quantization method that believes weight importance is related to numerical size, thus using fewer bits for smaller weight values and more bits for larger weight values.
TorchAO: PyTorch's official quantization tool library, which vLLM is beginning to support.
BitBLAS: A low-level computation library aimed at accelerating low-bit (such as 1-bit, 2-bit, 4-bit) matrix operations through optimized kernel functions.

4.2 How to Use Quantized Models in vLLM

Using quantization in vLLM is very simple, typically just requiring specifying the quantization parameter in the LLM constructor. vLLM will automatically detect the quantization type from the model's configuration file (config.json).

Example: Loading an AWQ Quantized Model

from vllm import LLM
# vLLM will automatically recognize awq quantization from "TheBloke/My-Model-AWQ"'s config.json
llm = LLM(model="TheBloke/My-Model-AWQ", quantization="awq")

Example: Enabling FP8 KV Cache

from vllm import LLM
llm = LLM(model="meta-llama/Llama-2-7b-chat-hf",
kv_cache_dtype="fp8")

5. llama.cpp vs. vLLM: Comparison and Summary

Feature	llama.cpp	vLLM
Target Platform	CPU, Cross-platform, Edge devices	High-performance GPU servers
Core Philosophy	Cohesive, self-contained, extreme optimization	Open, integrated, high throughput
File Format	GGUF (custom format)	Standard Hugging Face format
Quantization Schemes	Built-in `K-Quants`, `IQ`, etc.	Integrates GPTQ, AWQ, FP8, BnB, etc.
Ease of Use	Requires `llama-quantize` conversion	Direct loading, automatic detection
Ecosystem	Self-contained ecosystem	Embraces the entire Python AI ecosystem
Latest Technology	Quickly follows up and implements own versions	Quickly integrates latest open-source libraries

6. Latest Quantization Trends and Outlook

The field of model quantization is still rapidly evolving. Here are some trends worth noting:

1-bit/Binary Neural Networks (BNNs): Ultimate model compression, restricting weights to +1 or -1. Although currently suffering significant precision loss in LLMs, its potential is enormous, with related research emerging constantly.
Non-uniform Quantization: Like SqueezeLLM, dynamically allocating bit numbers based on data distribution, theoretically superior to uniform quantization.
Hardware-Algorithm Co-design: New hardware (such as FP8, FP4, INT4 support) is driving the development of new quantization algorithms, while new algorithms are guiding future hardware design.
Combining Quantization with Sparsification: Combining quantization with sparsification techniques like pruning holds promise for achieving higher rates of model compression.

7. Conclusion

Model quantization is a key technology for addressing the challenges of the large model era. llama.cpp and vLLM represent two different quantization philosophies: llama.cpp provides ultimate local inference performance for resource-constrained devices through its elegant GGUF format and built-in K-Quants; while vLLM has become the king of GPU cloud inference services through its open ecosystem and integration of various cutting-edge quantization schemes.

Understanding the quantization implementations of these two frameworks not only helps us choose the right tool for specific scenarios but also gives us insight into the development trajectory and future directions of the entire LLM inference optimization field.

SGLang Technical Guide: High-Performance Structured Generation Framework

Thu, 26 Jun 2025 01:07:00 +0000

1. SGLang Introduction

SGLang (Structured Generation Language) is a high-performance service framework designed for large language models (LLMs) and vision language models (VLMs). Its core goal is to address the challenges faced by complex LLM programs in real-world applications, maximizing inference performance while maintaining flexibility.

Traditional LLM service frameworks (like vLLM) excel at handling simple, one-shot prompting but face limitations in complex scenarios requiring multi-turn interactions, structured outputs, function calls, or control flow. SGLang effectively bridges this gap by introducing a novel frontend language and an efficient backend runtime.

Core advantages of SGLang include:

Exceptional Performance: SGLang introduces RadixAttention, an innovative attention mechanism that automatically and losslessly reuses key-value caches (KV Cache), significantly improving inference speed in scenarios with complex prompts (like CoT, ReAct) or multi-turn conversations. Compared to leading frameworks like vLLM, SGLang can achieve several times higher throughput in these scenarios.
Powerful Programming Capabilities: SGLang provides an intuitive domain-specific language (DSL) that allows developers to orchestrate complex generation tasks in a Pythonic way. You can easily define variables, use loops and conditional statements, call external tools, and seamlessly integrate these logic elements with the LLM's generation process. This makes building complex AI agents, multi-turn dialogue systems, and structured data extraction tasks unprecedentedly simple.
Unified Frontend-Backend Interface: SGLang decouples frontend programming logic from backend inference services. The frontend defines “what to generate,” while the backend handles “how to efficiently generate it.” This design not only simplifies the development process but also makes SGLang compatible with OpenAI's API standards, allowing users to easily migrate existing applications to SGLang and immediately benefit from performance gains.
Flexible Structured Output: SGLang provides powerful structured output constraint capabilities. Whether through regular expressions, EBNF grammar, or JSON Schema, you can precisely control the output format of the LLM, ensuring that the generated content conforms to the expected structure, which is crucial for applications requiring reliable data formats.

In summary, SGLang is not just an LLM inference acceleration engine but a complete programming and execution framework for complex generation tasks. It aims to enable developers to fully unleash the potential of large language models in an efficient and intuitive way.

2. Core Features

The power of SGLang lies in its unique design, which combines an intuitive frontend programming model with an efficient backend execution engine. Below are detailed introductions to several of its core features.

2.1 RadixAttention: KV Cache Optimization for Complex Prompts

When processing complex LLM programs, such as Chain-of-Thought, multi-turn dialogues, or agents that need to call tools, prompts often contain large shared prefixes. Traditional attention mechanisms produce redundant computation and storage when handling these shared prefixes.

SGLang introduces RadixAttention, a novel KV cache optimization technique. Its core idea is to organize prompts into a radix tree and perform attention calculations on this tree.

Automatic Sharing and Reuse: RadixAttention can automatically identify and share common prefixes between different requests, avoiding duplicate computation and storage. For example, in multi-turn dialogues, the conversation history of each turn can be losslessly reused by subsequent turns.
Performance Improvement: By maximizing KV cache reuse, RadixAttention significantly reduces memory usage and computational load, increasing throughput by 2 to 5 times, especially when handling long prompts or high-concurrency requests.

Below is a Mermaid diagram that visually demonstrates how RadixAttention handles requests with shared prefixes:

graph TD
subgraph "Traditional Method (No Sharing)"
req1["Request 1: 'A B C D'"]
req2["Request 2: 'A B E F'"]
kv1["KV Cache: [A, B, C, D]"]
kv2["KV Cache: [A, B, E, F]"]
req1 --> kv1
req2 --> kv2
end
subgraph "SGLang RadixAttention"
Root("Root") --> A("Token 'A'");
A --> B("Token 'B'");
B --> C("Token 'C'");
B --> E("Token 'E'");
C --> D("Token 'D'");
E --> F("Token 'F'");
style A fill:#9f9
style B fill:#9f9
end

In the diagram above, for two requests 'A B C D' and 'A B E F', the traditional method creates two independent KV caches. RadixAttention, however, organizes them into a tree, sharing the computation and storage of the common prefix 'A B' (green nodes), creating new branches only for the different parts (C, D, E, F). This greatly improves memory and computational efficiency.

2.2 Unified Frontend Programming Language (DSL)

SGLang provides an expressive domain-specific language (DSL) deeply integrated with Python, allowing developers to build complex generation logic in a natural and intuitive way.

SGLang Architecture Overview

To better understand how SGLang works, we can observe its core architecture through the following flowchart:

graph TD
subgraph User Side
A[Developer defines SGLang program<br>using function decorator] --> B{Call run method};
end
subgraph SGLang Frontend
B --> C[1. Parse Python AST<br>Separate deterministic logic and generation instructions];
C --> D[2. Build portable<br>SGLang IR intermediate representation];
end
subgraph Network Communication
D -- HTTP Request --> E[SGLang backend service SRT];
end
subgraph SGLang Backend SRT
E --> F[3. Receive IR and schedule];
F --> G{RadixAttention engine};
G --> H[4. Efficient execution<br>KV cache reuse];
H --> I[LLM/VLM model];
I --> J[5. Generate results];
end
subgraph Return Path
J -- HTTP Response --> K[Return results to frontend];
K --> L[6. Fill state object `s`];
L --> M[User gets final results];
end
style B fill:#f9f,stroke:#333,stroke-width:2px
style E fill:#ccf,stroke:#333,stroke-width:2px
style G fill:#9cf,stroke:#333,stroke-width:2px

This diagram clearly shows how SGLang decouples and combines the programming convenience of the frontend with the high-performance execution engine of the backend.

Pythonic Control Flow: You can directly use standard Python control flow statements like if/else and for loops in SGLang functions to dynamically build prompts.
Integration of Generation and Logic: Through the @function decorator and gen() instruction, SGLang seamlessly combines the LLM's generation process (the “non-deterministic” part) with the program's deterministic logic.

Example: Generating Different Content Based on Conditions

from sglang import function, system, user, assistant, gen
@function
def tool_use(s, question):
s += system("You are a helpful assistant.")
s += user(question)
s += assistant(
"To answer this question, I need to use a "
+ gen("tool", choices=["calculator", "search engine"])
+ ". "
)
if s["tool"] == "calculator":
s += assistant("The math expression is: " + gen("expression"))
elif s["tool"] == "search engine":
s += assistant("The key word to search is: " + gen("word"))
state = tool_use.run("What is the population of London?")
print(state["tool"])
# Output: search engine
print(state["word"])
# Output: population of London

In this example, the program first asks the LLM to choose between “calculator” and “search engine” as a tool, then executes different logic branches based on the LLM's choice, guiding the LLM to generate the next step of content.

2.3 Powerful Structured Output

To ensure that content generated by the LLM can be reliably parsed and used by downstream programs, SGLang provides multiple powerful structured output constraint mechanisms.

Regular Expressions (Regex): You can provide a regular expression to force the model's output to strictly match that pattern. This is useful for generating identifiers, numbers, or simple text fragments in specific formats.

response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
messages=[{"role": "assistant", "content": "What is the capital of France?"}],
extra_body={"regex": "(Paris|London)"},
)
# response.choices[0].message.content will necessarily be "Paris" or "London"

EBNF Grammar: For more complex grammatical structures, you can use Extended Backus-Naur Form (EBNF) to define a complete grammar. This allows you to generate code, DSLs, or other structured text that strictly adheres to specific syntax.

ebnf_grammar = """
root ::= city " is the capital of " country
city ::= "London" | "Paris" | "Berlin" | "Rome"
country ::= "England" | "France" | "Germany" | "Italy"
"""
response = client.chat.completions.create(
model="meta-llama/Meta-Llama-3.1-8B-Instruct",
messages=[{"role": "user", "content": "Give me the information of the capital of France."}],
extra_body={"ebnf": ebnf_grammar},
)
# response.choices[0].message.content will be "Paris is the capital of France"

JSON Schema: SGLang supports using JSON Schema to constrain the model to generate structured JSON objects. You can directly define a JSON Schema or use a Pydantic model to automatically generate one. This is crucial for APIs and data processing tasks that require reliable, verifiable JSON output.

from pydantic import BaseModel
class CapitalInfo(BaseModel):
name: str
population: int
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
messages=[{"role": "assistant", "content": "Give me the information and population of the capital of France in the JSON format."}],
response_format={
"type": "json_schema",
"json_schema": {
"name": "capital_info",
"schema": CapitalInfo.model_json_schema(),
},
},
)
# response.choices[0].message.content will be a JSON string conforming to the CapitalInfo structure

3. Quick Start

This section will guide you through installing SGLang, starting the service, and basic usage, allowing you to experience SGLang's powerful features in just a few minutes.

3.1 Installation

SGLang can be installed via pip or the faster uv. For the best experience and full functionality, it's recommended to install the all version.

Using pip:

pip install --upgrade pip
pip install "sglang[all]"

Using uv (recommended, faster):

pip install uv
uv pip install "sglang[all]"

Note: The installation process may require compiling CUDA kernels (such as flashinfer). Please ensure that the CUDA_HOME environment variable is correctly configured in your environment and that the CUDA version is compatible with your PyTorch version.

3.2 Starting the Backend Service (SRT)

After installation, the next step is to start SGLang's backend service (SRT, SGLang Runtime). This service will load the specified language model and provide an interface compatible with the OpenAI API.

Run the following command in your terminal:

python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000

Parameter Description:

--model-path: Specifies the path to the model to load. This can be a model name on the Hugging Face Hub (as shown in this example) or a local model path.
--host: The host address the service listens on. 0.0.0.0 means allowing access from any network interface.
--port: The port number the service listens on.

When the service starts successfully, you'll see output similar to the following, indicating that the model has been loaded and is ready to receive requests.

INFO: Uvicorn running on http://0.0.0.0:30000 (Press CTRL+C to quit)
INFO: Started server process [12345]
INFO: Waiting for application startup.
INFO: Application startup complete.

3.3 Sending Your First Request

With the service running, we can now interact with it using OpenAI's Python client library.

Create a Python file named test_sglang.py and fill it with the following content:

import openai
# Initialize the client, pointing to our locally started SGLang service
client = openai.Client(
base_url="http://127.0.0.1:30000/v1",
api_key="EMPTY" # SGLang service doesn't require an API Key
)
# Create a chat completion request
response = client.chat.completions.create(
model="meta-llama/Meta-Llama-3.1-8B-Instruct", # Must match the model loaded by the service
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France and why is it famous?"},
],
temperature=0.7,
max_tokens=150,
)
# Print the model's response
print(response.choices[0].message.content)

Run this script:

python test_sglang.py

You'll see the model's detailed answer about Paris. At this point, you've successfully completed the entire process from service deployment to inference request using SGLang!

4. Frontend Language (SGLang DSL)

SGLang's frontend language (DSL) is the core of its usability. It allows you to define complex generation processes in a declarative way, perfectly combining Python's flexibility with the generative capabilities of LLMs.

4.1 `@function` Decorator

All SGLang programs begin with a Python function decorated by @function. This decorator transforms an ordinary Python function into an executable SGLang program template.

State Management: The first parameter of the function (typically named s) represents the current generation state. It's a dictionary-like object used to store and pass all variables produced during the generation process.
Delayed Execution: Functions decorated with @function are not executed immediately when defined. Instead, they create a reusable template. The program only executes when the .run() or .run_batch() method is called.

Interaction Flow

The entire function call interaction flow can be represented by the following sequence diagram:

sequenceDiagram
participant User
participant App as Application (Python)
participant SGLang as SGLang Service
participant Tool as External Tool (e.g., Weather API)
User->>+App: "What's the weather like in Boston?"
App->>+SGLang: Send request with messages and tools
SGLang->>SGLang: Model decides to call get_current_weather
SGLang-->>-App: Return tool_calls with function name and parameters
App->>App: Parse tool_calls
App->>+Tool: Call get_current_weather(city="Boston", unit="fahrenheit")
Tool-->>-App: Return weather result: "68°F"
App->>+SGLang: Send new request with weather result
SGLang->>SGLang: Model generates final reply based on weather result
SGLang-->>-App: Return final natural language reply
App-->>-User: "It's currently 68°F in Boston."

This sequence diagram clearly shows the complete loop from user question to model decision, tool call, result integration, and final response.

4.2 Core Instructions

Within SGLang functions, you use a series of instructions to build prompts and control the generation flow.

Role Instructions: system(), user(), assistant() These instructions are used to define different parts of a conversation, conforming to the standard multi-turn dialogue format. You can pass strings directly to them.
Generation Instruction: gen() This is the most important instruction in SGLang. It tells the LLM to generate text at the current position.
- s += gen("variable_name", ...): The first parameter of gen() is required and specifies the variable name in which the generation result will be stored in the state s.
- max_tokens: Limits the maximum number of tokens to generate.
- stop: Defines one or more stop strings. When the model generates these strings, the generation process ends early.
- choices: Provides a list of strings, forcing the model to choose one of these options for generation.

Example: A Complete Frontend Function

from sglang import function, system, user, assistant, gen, set_default_backend, OpenAI
# Set the backend to the OpenAI-compatible service provided by SGLang
set_default_backend(OpenAI("meta-llama/Meta-Llama-3.1-8B-Instruct"))
@function
def multi_turn_qa(s, question1, question2):
s += system("You are a helpful assistant.")
s += user(question1)
s += assistant(gen("answer1", max_tokens=128))
s += user(question2)
s += assistant(gen("answer2", max_tokens=128))
# Execute the SGLang program
state = multi_turn_qa.run(
question1="What is the capital of the UK?",
question2="What is its population?",
temperature=0.1
)
print("Answer 1:", state["answer1"])
print("Answer 2:", state["answer2"])

4.3 Streaming Output

For applications requiring real-time feedback, SGLang supports streaming output. Simply set stream=True in the .run() method and iterate over the .text_iter() method of the returned state object.

state = multi_turn_qa.run(
question1="Write a short story about a robot.",
question2="Continue the story.",
stream=True
)
for out in state.text_iter("answer2"):
print(out, end="", flush=True)

5. Backend Service (SRT) and API Reference

SGLang's backend, the SGLang Runtime (SRT), is a high-performance inference server implemented in Python. It's responsible for loading models, managing KV caches (through RadixAttention), and handling requests from clients. SRT provides two main API endpoints.

5.1 Native API: `/generate`

This is a lower-level API that provides the finest control over the generation process.

Endpoint: POST /generate
Description: Generate text starting from a given text prompt.
Core Parameters:
- text (string, required): The input text prompt.
- sampling_params (object, optional): A JSON object containing sampling parameters.
  - temperature (float): Sampling temperature.
  - max_new_tokens (int): Maximum number of new tokens to generate.
  - stop (string or list[string]): Stop tokens.
  - json_schema (string): JSON Schema string for constraining output.
  - regex (string): Regular expression for constraining output.
  - ebnf (string): EBNF grammar for constraining output.
- stream (boolean, optional): Whether to use streaming.

Example (using requests):

import requests
import json
url = "http://127.0.0.1:30000/generate"
data = {
"text": "The capital of France is",
"sampling_params": {
"temperature": 0,
"max_new_tokens": 16,
}
}
response = requests.post(url, json=data)
print(response.json())
# {'text': ' Paris.\n\nThe capital of France is Paris. It is the most populous city in', 'meta': ...}

5.2 OpenAI Compatible API: `/v1/chat/completions`

For easy migration and integration, SGLang provides a chat completion API fully compatible with OpenAI. You can seamlessly use OpenAI's official client library.

Endpoint: POST /v1/chat/completions
Description: Perform chat-style text generation.
Core Parameters:
- model (string, required): The name of the model.
- messages (list[object], required): List of conversation messages.
- temperature, max_tokens, stream, etc.
- response_format (object, optional): For specifying structured output, such as {"type": "json_schema", "json_schema": ...}.
- extra_body (object, optional): SGLang-specific extension parameters, such as {"regex": "..."} or {"ebnf": "..."}.

Example (using the openai library):

import openai
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="meta-llama/Meta-Llama-3.1-8B-Instruct",
messages=[{"role": "user", "content": "List 3 countries and their capitals."}],
temperature=0,
max_tokens=64,
)
print(response.choices[0].message.content)

6. Advanced Usage: Function Calling/Tool Usage

SGLang's powerful programming model makes it very suitable for building AI agents capable of calling external tools. This is typically achieved through structured output, where the model is guided to generate text in a specific format (usually JSON) describing a function call.

Here are the steps to build a simple weather query agent:

1. Define Tool Schema

First, use JSON Schema to define your tool. This tells the model the name of the tool, its purpose, and what parameters it needs.

tools = [
{
"type": "function",
"function": {
"name": "get_current_weather",
"description": "Get the current weather in a given location",
"parameters": {
"type": "object",
"properties": {
"city": {"type": "string", "description": "The city name"},
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
},
"required": ["city", "unit"],
},
},
}
]

2. Guide the Model to Make Function Calls

In the messages sent to the model, include a system prompt indicating that the model can use these tools. Then, pass tools and tool_choice="auto" in the API call.

import json
messages = [
{"role": "system", "content": "You are a helpful assistant that can access external tools."},
{"role": "user", "content": "What's the weather like in Boston in fahrenheit?"}
]
response = client.chat.completions.create(
model="meta-llama/Meta-Llama-3.1-8B-Instruct",
messages=messages,
tools=tools,
tool_choice="auto",
)
# Check if the model decided to call a tool
response_message = response.choices[0].message
tool_calls = response_message.tool_calls
if tool_calls:
# Model decided to call a tool
for tool_call in tool_calls:
function_name = tool_call.function.name
function_args = json.loads(tool_call.function.arguments)
print(f"Function Call: {function_name}")
print(f"Arguments: {function_args}")
# Here, you could actually execute the function call
# e.g., result = get_current_weather(**function_args)

Output:

Function Call: get_current_weather
Arguments: {'city': 'Boston', 'unit': 'fahrenheit'}

In this way, you can build powerful AI applications capable of interacting with the external world.

Llama.cpp Technical Guide: Lightweight LLM Inference Engine

Thu, 26 Jun 2025 01:06:00 +0000

1. Introduction

Llama.cpp is a high-performance, lightweight inference framework for large language models (LLMs) written in C/C++. It focuses on efficiently running LLMs on consumer-grade hardware, making local inference possible on ordinary laptops and even smartphones.

Core Advantages:

High Performance: Achieves extremely fast inference speeds through optimized C/C++ code, quantization techniques, and hardware acceleration support (such as Apple Metal, CUDA, OpenCL, SYCL).
Lightweight: Extremely low memory and computational resource consumption, eliminating the need for expensive GPUs.
Cross-Platform: Supports multiple platforms including macOS, Linux, Windows, Docker, Android, and iOS.
Open Ecosystem: Features an active community and rich ecosystem, including Python bindings, UI tools, and OpenAI-compatible servers.
Continuous Innovation: Quickly follows and implements the latest model architectures and inference optimization techniques.

2. Core Concepts

2.1. GGUF Model Format

GGUF (Georgi Gerganov Universal Format) is the core model file format used by llama.cpp, an evolution of its predecessor GGML. GGUF is a binary format designed for fast loading and memory mapping.

Key Features:

Unified File: Packages model metadata, vocabulary, and all tensors (weights) in a single file.
Extensibility: Allows adding new metadata without breaking compatibility.
Backward Compatibility: Guarantees compatibility with older versions of GGUF models.
Memory Efficiency: Supports memory mapping (mmap), allowing multiple processes to share the same model weights, thereby saving memory.

2.2. Quantization

Quantization is one of the core advantages of llama.cpp. It is a technique that converts model weights from high-precision floating-point numbers (such as 32-bit or 16-bit) to low-precision integers (such as 4-bit, 5-bit, or 8-bit).

Main Benefits:

Reduced Model Size: Significantly reduces the size of model files, making them easier to distribute and store.
Lower Memory Usage: Reduces the RAM required to load the model into memory.
Faster Inference: Low-precision calculations are typically faster than high-precision ones, especially on CPUs.

llama.cpp supports various quantization methods, particularly k-quants, an advanced quantization technique that achieves extremely high compression rates while maintaining high model performance.

2.3. Multimodal Support

llama.cpp is not limited to text models; it has evolved into a powerful multimodal inference engine that supports processing text, images, and even audio simultaneously.

Supported Models: Supports various mainstream multimodal models such as LLaVA, MobileVLM, Granite, Qwen2.5 Omni, InternVL, SmolVLM, etc.
Working Principle: Typically converts images into embedding vectors through a vision encoder (such as CLIP), and then inputs these vectors along with text embedding vectors into the LLM.
Tools: llama-mtmd-cli and llama-server provide native support for multimodal models.

3. Usage Methods

3.1. Compilation

Compiling llama.cpp from source is very simple.

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
make

For specific hardware acceleration (such as CUDA or Metal), use the corresponding compilation options:

# For CUDA
make LLAMA_CUDA=1
# For Metal (on macOS)
make LLAMA_METAL=1

3.2. Basic Inference

After compilation, you can use the llama-cli tool for inference.

./llama-cli -m ./models/7B/ggml-model-q4_0.gguf -p "Building a website can be done in 10 simple steps:" -n 400

-m: Specifies the path to the GGUF model file.
-p: Specifies the prompt.
-n: Specifies the maximum number of tokens to generate.

3.3. OpenAI Compatible Server

llama.cpp provides a built-in HTTP server with an API compatible with OpenAI's API. This makes it easy to integrate with existing tools like LangChain and LlamaIndex.

Starting the server:

./llama-server -m models/7B/ggml-model-q4_0.gguf -c 4096

You can then send requests to http://localhost:8080/v1/chat/completions just like you would with the OpenAI API.

4. Advanced Features

4.1. Speculative Decoding

This is an advanced inference optimization technique that significantly accelerates generation speed by using a small “draft” model to predict the output of the main model.

Working Principle: The draft model quickly generates a draft token sequence, which is then validated all at once by the main model. If validated, it saves the time of generating tokens one by one.
Usage: Use the --draft-model parameter in llama-cli or llama-server to specify a small, fast draft model.

4.2. LoRA Support

LoRA (Low-Rank Adaptation) allows fine-tuning a model's behavior by training a small adapter without modifying the original model weights. llama.cpp supports loading one or more LoRA adapters during inference.

./llama-cli -m base-model.gguf --lora lora-adapter.gguf

You can even set different weights for different LoRA adapters:

./llama-cli -m base.gguf --lora-scaled lora_A.gguf 0.5 --lora-scaled lora_B.gguf 0.5

4.3. Grammars

Grammars are a very powerful feature that allows you to force the model's output to follow a specific format, such as a strict JSON schema.

Format: Uses a format called GBNF (GGML BNF) to define grammar rules.
Application: By providing GBNF rules through the grammar parameter in API requests, you can ensure that the model returns correctly formatted, directly parsable JSON data, avoiding output format errors and tedious post-processing.

Example: Using a Pydantic model to generate a JSON Schema, then converting it to GBNF to ensure the model output conforms to the expected Python object structure.

import json
from typing import List
from pydantic import BaseModel
class QAPair(BaseModel):
question: str
answer: str
class Summary(BaseModel):
key_facts: List[str]
qa_pairs: List[QAPair]
# Generate JSON Schema and print
schema = Summary.model_json_schema()
print(json.dumps(schema, indent=2))

5. Ecosystem

The success of llama.cpp has spawned a vibrant ecosystem:

llama-cpp-python: The most popular Python binding, providing interfaces to almost all features of llama.cpp and deeply integrated with frameworks like LangChain and LlamaIndex.
Ollama: A tool for packaging, distributing, and running models, using llama.cpp under the hood, greatly simplifying the process of running LLMs locally.
Numerous UI Tools: The community has developed a large number of graphical interface tools, allowing non-technical users to easily interact with local models.

6. Conclusion

llama.cpp is not just an inference engine; it has become a key force in driving the localization and popularization of LLMs. Through its excellent performance, highly optimized resource usage, and continuously expanding feature set (such as multimodality and grammar constraints), llama.cpp provides developers and researchers with a powerful and flexible platform, enabling them to explore and deploy AI applications on various devices, ushering in a new era of low-cost, privacy-protecting local AI.

vLLM Technical Guide: High-Performance LLM Inference Engine

Thu, 26 Jun 2025 01:05:00 +0000

1. Introduction to vLLM

vLLM is an open-source inference and serving engine designed for large language models (LLMs), renowned for its high throughput and memory efficiency. In the field of LLM serving, vLLM addresses a core pain point: traditional inference systems are inefficient when handling the key-value cache (KV Cache) in Transformer models’ attention mechanism, resulting in significant memory waste and limited inference speed.

The memory bottleneck in LLM inference primarily stems from the KV Cache. This cache stores attention keys and values for each previous token in a sequence to accelerate the generation of subsequent tokens. However, the size of the KV Cache is dynamic and difficult to predict, creating enormous challenges for memory management. Traditional systems (like HuggingFace Transformers) typically pre-allocate a large continuous memory space to store the KV Cache, leading to severe memory fragmentation and waste.

vLLM fundamentally solves this problem by introducing its core innovation: the PagedAttention mechanism.

2. Core Features and Advantages

vLLM stands out among numerous LLM inference frameworks thanks to several key features:

Extremely High Throughput: Through PagedAttention and Continuous Batching, vLLM significantly improves GPU utilization. Its throughput is several times higher than HuggingFace Transformers and outperforms other mainstream inference libraries.
Efficient Memory Management: The PagedAttention mechanism divides the KV Cache into non-continuous memory blocks, greatly reducing internal and external memory fragmentation. According to official data, it can save up to 55% of memory, meaning you can load larger models or serve more concurrent requests with the same hardware.
Flexible Decoding Strategies: vLLM supports various complex decoding algorithms, including Parallel Sampling, Beam Search, and Top-K/Top-P sampling, meeting the needs of different application scenarios.
OpenAI API Compatibility: vLLM provides a service endpoint that is fully compatible with the OpenAI API. This means you can seamlessly integrate vLLM into existing application ecosystems built on the OpenAI API with just a few configuration changes.
Distributed Inference: For ultra-large models that cannot fit on a single GPU, vLLM supports Tensor Parallelism, distributing model weights and computational load across multiple GPUs for efficient distributed inference.
Streaming and Structured Output: Supports streaming of generated tokens and can produce structured outputs in specific formats (such as JSON Schema or regular expressions) through Guided Generation.

3. Core Architecture: Deep Dive into PagedAttention

PagedAttention is the soul of vLLM, with its design inspiration coming from the paging technique used in modern operating systems to manage virtual memory.

3.1 Working Principle

In traditional methods, the KV Cache for each sequence is stored in continuous memory space. While this approach seems simple, it leads to severe memory fragmentation due to the vast differences in sequence lengths.

PagedAttention divides each sequence's KV Cache into fixed-size blocks. Each block can store keys and values for a fixed number of tokens. During inference, vLLM's core scheduler dynamically allocates these blocks to sequences as needed.

The advantages of this design include:

Eliminating Internal Fragmentation: Since blocks are of fixed size, a sequence's last block may have some unused space, but this waste is far less than that caused by reserving continuous memory for the entire sequence.
Flexible Memory Allocation: Blocks are stored in non-continuous memory space, making memory management more flexible, similar to how operating systems manage physical memory pages.
Efficient Memory Sharing: PagedAttention makes sharing KV Cache between different sequences exceptionally simple and efficient. For example, in parallel sampling or beam search, multiple candidate sequences originate from the same prompt. vLLM allows these sequences to share KV blocks storing the prompt portion, only needing to allocate new, independent blocks for each sequence when generating new tokens. This “Copy-on-Write” mechanism greatly reduces the memory overhead of complex decoding algorithms.

Below is a Mermaid diagram that more intuitively illustrates PagedAttention's memory management approach:

graph TD
subgraph Physical_Memory [KV Cache Physical Memory]
direction LR
B1(Block 1)
B2(Block 2)
B3(Block 3)
B4(Block 4)
B5(Block 5)
B6(Block 6)
B7(Block 7)
B8(Block 8)
end
subgraph Logical_View [Sequence Logical View]
direction TB
subgraph Seq1 [Sequence 1]
P1(Prompt) --> T1(Token 1)
end
subgraph Seq2 [Sequence 2]
P2(Prompt) --> T2(Token 1) --> T3(Token 2)
end
subgraph Seq3 [Parallel Sampling]
P3(Prompt) --> T4(Token 1a)
P3 --> T5(Token 1b)
end
end
subgraph Block_Table [Block Table]
direction TB
Map1["Seq 1: [B1, B5]"]
Map2["Seq 2: [B2, B6, B8]"]
Map3["Seq 3a: [B3, B7]"]
Map4["Seq 3b: [B3, B4]"]
end
Seq1 --> Map1
Seq2 --> Map2
Seq3 --> Map3
Seq3 --> Map4
Map1 --> B1
Map1 --> B5
Map2 --> B2
Map2 --> B6
Map2 --> B8
Map3 --> B3
Map3 --> B7
Map4 --> B3
Map4 --> B4
style B3 fill:#f9f,stroke:#333,stroke-width:2px
linkStyle 8 stroke-width:2px,stroke:green,fill:none;
linkStyle 11 stroke-width:2px,stroke:green,fill:none;
linkStyle 12 stroke-width:2px,stroke:green,fill:none;

Diagram explanation:

KV Cache Physical Memory: Represents non-continuous physical memory blocks on the GPU.
Sequence Logical View: Represents multiple requests (sequences) being processed.
Block Table: vLLM's core component that maps logical token positions to physical memory blocks.
Memory Sharing: Note that the two branches in “Parallel Sampling” (3a and 3b) share the same Prompt block (B3), demonstrating PagedAttention's efficient memory sharing.

3.2 Continuous Batching

Based on PagedAttention, vLLM implements a more advanced batching strategy—continuous batching. Traditional static batching requires waiting for all sequences in a batch to complete generation before processing the next batch. Continuous batching, however, allows new requests to be inserted into the batch immediately after a sequence in the batch completes generation, avoiding GPU idle waiting and further improving throughput.

Below is a comparison of the two batching methods using a Mermaid sequence diagram:

sequenceDiagram
participant C as Client
participant S as Server
participant G as GPU
note over C, G: --- Static Batching ---
C->>S: Request [R1, R2, R3, R4]
S->>G: Process Batch 1 [R1, R2, R3, R4]
note right of G: All requests process in parallel
G-->>S: Batch 1 Finished
note right of S: Wait for the entire batch to complete
S-->>C: Response [O1, O2, O3, O4]
C->>S: Request [R5, R6]
S->>G: Process Batch 2 [R5, R6]
note over C, G: --- Continuous Batching ---
C->>S: Request [R1, R2, R3, R4]
S->>G: Process [R1, R2, R3, R4]
G-->>S: R2 Finished
S-->>C: Response O2
C->>S: New Request R5
S->>G: Add R5 to queue (GPU is not idle)
note right of G: R1, R3, R4, R5 are now processing
G-->>S: R4 Finished
S-->>C: Response O4

4. Quick Start Guide

Below, we'll demonstrate how to install and use vLLM through a few simple steps.

4.1 Installation

You can install vLLM using either pip or uv (a faster package installation tool). Using uv is recommended as it can automatically detect your CUDA version and install the matching PyTorch backend.

Using uv (recommended):

# Create and activate a virtual environment
uv venv
source .venv/bin/activate
# Install vLLM
uv pip install vllm --torch-backend=auto

Using pip:

pip install vllm

4.2 Offline Inference

The vllm.LLM class makes offline inference very convenient.

from vllm import LLM, SamplingParams
# Define input prompts
prompts = [
"Hello, my name is",
"The capital of France is",
"The future of AI is",
]
# Define sampling parameters
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
# Initialize the LLM engine (model will be automatically downloaded from Hugging Face)
llm = LLM(model="facebook/opt-125m")
# Generate text
outputs = llm.generate(prompts, sampling_params)
# Print results
for output in outputs:
prompt = output.prompt
generated_text = output.outputs[0].text
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

4.3 Launching an OpenAI-Compatible Server

One of vLLM's most powerful features is its built-in API server. With just one command, you can start a service compatible with the OpenAI API.

vllm serve Qwen/Qwen2.5-1.5B-Instruct

By default, the server will run on http://localhost:8000.

4.4 Interacting with the Server

You can interact with the server using curl or the openai Python client.

Using curl:

curl http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-1.5B-Instruct",
"prompt": "San Francisco is a",
"max_tokens": 7,
"temperature": 0
}'

Using the OpenAI Python client:

from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="not-used" # API key is not required
)
completion = client.chat.completions.create(
model="Qwen/Qwen2.5-1.5B-Instruct",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Who won the world series in 2020?"}
]
)
print(completion.choices[0].message)

5. Model Serving

5.1 Distributed Serving

If a model is too large to fit on a single GPU, you can distribute it across multiple GPUs using tensor parallelism.

# Start a service on 4 GPUs
vllm serve facebook/opt-13b --tensor-parallel-size 4

5.2 Docker Deployment

vLLM provides official Docker images for convenient containerized deployment.

docker run --runtime nvidia --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HUGGING_FACE_HUB_TOKEN=<your-hf-token>" \
-p 8000:8000 \
--ipc=host \
vllm/vllm-openai:latest \
--model mistralai/Mistral-7B-v0.1

6. Advanced Features

6.1 Structured Outputs

vLLM supports various ways to constrain the model's output format, which is crucial for applications requiring reliable, parsable outputs.

Generating JSON using Pydantic models:

from pydantic import BaseModel
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")
model = client.models.list().data[0].id
class People(BaseModel):
name: str
age: int
completion = client.chat.completions.create(
model=model,
messages=[
{"role": "user", "content": "Generate a JSON with the name and age of one random person."}
],
response_format={
"type": "json_schema",
"json_schema": {
"name": "people",
"schema": People.model_json_schema()
}
},
)
print(completion.choices[0].message.content)

6.2 LoRA Support

vLLM can efficiently serve multiple LoRA adapters on the same base model. This is particularly useful for scenarios requiring customized models for different customers or tasks.

Starting a server with LoRA support:

from vllm import LLM
llm = LLM(model="meta-llama/Llama-2-7b-hf", enable_lora=True)

Specifying a LoRA adapter in a request:

curl http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "sql-lora", # Specify the LoRA model ID
"prompt": "San Francisco is a",
"max_tokens": 7
}'

6.3 Quantization

Quantization is a technique to reduce model size and memory usage by lowering the precision of model weights. vLLM supports various quantization schemes, such as AWQ and FP8 KV cache.

Enabling FP8 KV cache:

from vllm import LLM
llm = LLM(
model="meta-llama/Llama-2-7b-chat-hf",
kv_cache_dtype="fp8",
calculate_kv_scales=True # Dynamically calculate quantization scales
)

7. Framework Integration

vLLM can be easily integrated with popular LLM application frameworks like Langchain and LlamaIndex for building complex systems such as Retrieval-Augmented Generation (RAG). Typically, vLLM serves as a backend providing fast LLM inference and embedding generation services.

Installing related dependencies:

pip install -U vllm langchain_openai langchain_community

Afterward, in Langchain, you can point the base_url of ChatOpenAI or OpenAIEmbeddings to your vLLM server's address to complete the integration.

8. Conclusion

Through its innovative PagedAttention architecture, vLLM successfully addresses memory management and performance bottlenecks in LLM inference, providing developers with an extremely efficient, flexible, and easy-to-use inference serving engine. Whether conducting quick offline experiments or deploying production-grade, high-concurrency LLM services, vLLM demonstrates excellent performance and powerful functionality. As the community continues to develop, vLLM is becoming one of the standard tools in the field of LLM serving.

LoRA Technical Guide: Parameter-Efficient Fine-Tuning for Large Models

Thu, 26 Jun 2025 00:00:00 +0000

1. Introduction: Why LoRA?

In today's rapidly evolving landscape of Large Language Models (LLMs) and generative AI, we've witnessed an explosive growth in model sizes, ranging from hundreds of millions to trillions of parameters. These massive models demonstrate remarkable capabilities across various tasks. However, a significant challenge emerges: how can we fine-tune these models for specific downstream tasks?

The traditional Full Fine-Tuning approach, which updates all parameters of a model, faces severe challenges:

High computational cost: Fine-tuning a model with billions of parameters requires enormous computational resources and hundreds of GB of GPU memory, which is prohibitively expensive for most developers and small to medium-sized enterprises.
Massive storage requirements: Each fine-tuned model for a specific task requires storing a complete model copy, leading to rapidly escalating storage costs.
Deployment difficulties: Maintaining and switching between multiple massive model copies for different tasks in a production environment is a nightmare.

To address these pain points, Parameter-Efficient Fine-Tuning (PEFT) techniques have emerged. The core idea is to freeze most parameters of the pre-trained model during fine-tuning and only adjust a small portion (typically far less than 1% of the total) of new or specific parameters.

Among the various PEFT techniques, LoRA (Low-Rank Adaptation of Large Language Models) stands out for its excellent performance, efficiency, and implementation simplicity, becoming one of the most mainstream and widely applied solutions today. This document will provide an in-depth yet accessible introduction to the core principles of LoRA and offer detailed practical guidance.

2. Core Principles: The Magic of LoRA

LoRA's core assumption is that the weight changes in large language models when adapting to new tasks are low-rank. In other words, although the weight matrix W of the pre-trained model is very large (e.g., d x d dimensions), the weight change ΔW during fine-tuning has a very low “intrinsic rank.”

Based on this assumption, LoRA doesn't directly update W, but instead approximates ΔW by training two smaller, low-rank matrices B and A, such that ΔW ≈ BA.

W is the pre-trained, frozen weight matrix.
A is an r x d dimensional matrix, where r is a rank much smaller than d.
B is a d x r dimensional matrix.

During fine-tuning, only the parameters of matrices A and B are trainable. The forward propagation computation process is accordingly changed to:

h = Wx + BAx

Here's a diagram that illustrates this process more intuitively:

graph TD
A[Input x] --> B(Pre-trained weights W);
A --> C(Low-rank matrix A);
C --> D(Low-rank matrix B);
B --> E[Wx];
D --> F[BAx];
E --> G((Sum));
F --> G;
G --> H[Final output h];
style B fill:#eee,stroke:#333,stroke-width:2px,stroke-dasharray: 5, 5
style C fill:#9cf,stroke:#333,stroke-width:2px
style D fill:#9cf,stroke:#333,stroke-width:2px

Where x is the input and h is the output. This approach greatly reduces the number of parameters that need to be trained. For example, if d = 4096 and r = 8, the original matrix W has 4096 * 4096 ≈ 16.7M parameters, while A and B together have only 4096 * 8 + 8 * 4096 ≈ 65K parameters, reducing the parameter count by approximately 256 times!

Key parameter r: The rank r is the most important hyperparameter in LoRA. It controls the size of the low-rank matrices and directly determines the number of new parameters.

Smaller r: Fewer trainable parameters, faster training speed, lower memory usage, but may not fully capture complex features of the task.
Larger r: More trainable parameters, stronger model fitting capability, but increases computational cost and risk of overfitting. In practice, r is typically set to 8, 16, 32, or 64, which achieves a good balance between performance and efficiency.

3. Significant Advantages of LoRA

Compared to full fine-tuning, LoRA demonstrates overwhelming advantages in multiple aspects:

Extreme parameter efficiency: As mentioned above, LoRA only requires training a tiny fraction of parameters. We can see this intuitively through the print_trainable_parameters() function, where the proportion of trained parameters is typically less than 1%.
Faster training speed: With a significantly reduced number of parameters for gradient computation and updates, training time is also shortened, accelerating the iteration cycle.
Lower hardware requirements: LoRA significantly reduces GPU memory (VRAM) usage during training, making it possible to fine-tune models with tens of billions of parameters on consumer-grade GPUs (such as RTX 3090/4090).
Flexibility in deployment and management: This is one of LoRA's most attractive advantages. The pre-trained model remains unchanged and can be shared across all tasks. For each downstream task, we only need to save a lightweight (typically just a few MB to tens of MB) LoRA adapter (i.e., the weights of matrices A and B). During deployment, the appropriate adapter can be loaded dynamically according to needs, greatly simplifying model management and switching in multi-task scenarios.

4. Hands-on Practice: LoRA Training Methods

Below, we'll demonstrate a complete example of how to fine-tune a large model using LoRA with the transformers, peft, and trl libraries from the Hugging Face ecosystem.

Step 1: Environment Preparation

First, ensure you have installed the necessary Python libraries:

pip install transformers peft trl datasets torch

Step 2: Load Model, Tokenizer, and Dataset

We select a pre-trained model as the foundation and load the corresponding tokenizer. At the same time, we load a dataset from the Hugging Face Hub for fine-tuning.

from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from datasets import load_dataset
# Model ID, can be any supported Causal LM
model_id = "facebook/opt-350m"
# Load pre-trained model
model = AutoModelForCausalLM.from_pretrained(model_id)
# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id)
# Load dataset (using English quotes dataset as an example)
dataset = load_dataset("Abirate/english_quotes", split="train")

Step 3: Configure LoRA (`LoraConfig`)

This is the core step of LoRA fine-tuning. We need to create a LoraConfig object to define the behavior of the LoRA adapter.

from peft import LoraConfig
lora_config = LoraConfig(
r=16, # Rank of the low-rank matrices, recommended values are 8, 16, 32
lora_alpha=32, # Scaling factor, typically set to twice the value of r
target_modules=["q_proj", "v_proj"], # Specify which model layers to apply LoRA to. For Transformer models, typically q_proj and v_proj
lora_dropout=0.05, # Dropout probability for LoRA layers
bias="none", # Whether to train bias terms, "none" means not training
task_type="CAUSAL_LM" # Task type, here it's causal language modeling
)

target_modules: This parameter is crucial. It tells the PEFT library which modules (typically nn.Linear layers) in the model should have LoRA applied. For most Transformer models, applying it to the query and value projection layers in the Attention mechanism (i.e., q_proj and v_proj) is a common practice. You can print the model object to see the names of all its modules to determine which can be targeted.

Step 4: Apply LoRA and Train with `SFTTrainer`

The SFTTrainer (Supervised Fine-tuning Trainer) provided by the trl library greatly simplifies the fine-tuning process. It has built-in support for peft, so we just need to pass the model, tokenizer, dataset, and peft_config to it.

from trl import SFTTrainer
# Define training parameters
training_args = TrainingArguments(
output_dir="./lora_finetuned_model", # Model output directory
num_train_epochs=3, # Number of training epochs
per_device_train_batch_size=4, # Training batch size per device
logging_dir='./logs', # Logging directory
logging_steps=50, # Log every this many steps
learning_rate=2e-4, # Learning rate
)
# Initialize SFTTrainer
trainer = SFTTrainer(
model=model,
tokenizer=tokenizer,
args=training_args,
train_dataset=dataset,
peft_config=lora_config, # Pass in LoRA configuration
dataset_text_field="quote", # Field name containing text in the dataset
)
# Start training
trainer.train()
# Save the trained LoRA adapter
trainer.save_model()

After training is complete, an adapter_model.bin file and an adapter_config.json file will be generated in the output_dir directory. These are the lightweight LoRA adapter we've trained.

Step 5: Inference with the Trained LoRA Adapter

For inference, we first load the original pre-trained model, then load the trained LoRA adapter weights.

from peft import PeftModel
# Load the original, non-fine-tuned model
base_model = AutoModelForCausalLM.from_pretrained(model_id)
# Load the LoRA adapter
model_with_lora = PeftModel.from_pretrained(base_model, "./lora_finetuned_model")
# Now model_with_lora is a model with LoRA weights integrated, ready for inference
prompt = "The best way to predict the future is to"
inputs = tokenizer(prompt, return_tensors="pt")
# Generate text
outputs = model_with_lora.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

5. LoRA Model Deployment: From Static to Dynamic

After training, efficiently deploying LoRA models into production environments is the crucial next step. LoRA deployment strategies mainly fall into two categories: Weight Merging (Static Deployment) and Dynamic Adapter Loading (Dynamic Deployment). The following flowcharts illustrate these two paths:

Option 1: Weight Merging (Static Deployment)

graph TD
A[LoRA Training Complete] --> B[Base Model + LoRA Adapter];
B --> C["Call merge_and_unload()"];
C --> D[Generate standalone full model];
D --> E[Standard deployment];
style D fill:#c9f,stroke:#333,stroke-width:2px

Option 2: Dynamic Adapter Loading (Dynamic Deployment)

graph TD
A[LoRA Training Complete] --> B[vLLM / TGI server];
B --> C[Load Base Model];
C --> D[Load multiple LoRA Adapters];
D --> E[On-demand inference combinations];
style E fill:#9cf,stroke:#333,stroke-width:2px

Option 1: Weight Merging and Standard Deployment (Static)

This is the simplest and most direct deployment approach. The core idea is to merge the lightweight LoRA adapter weights into the original base model weights, generating a new, standalone full model.

Method: Using the merge_and_unload() method from the peft library, this process can be easily completed.

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
# Assuming model_id and lora_path are defined
base_model = AutoModelForCausalLM.from_pretrained(model_id)
model_with_lora = PeftModel.from_pretrained(base_model, "./lora_finetuned_model")
# Merge weights
merged_model = model_with_lora.merge_and_unload()
# Now merged_model is a standard Transformers model
# You can save it like any other model
merged_model.save_pretrained("./merged_lora_model")
tokenizer.save_pretrained("./merged_lora_model")

Afterward, you can load and use this merged_lora_model just like any regular Hugging Face model.

Advantages:
- Zero inference latency: After merging, the inference process is identical to a standard model, with no additional computational overhead.
- Simple deployment: No need for any additional inference framework support, can be used directly with standard libraries like transformers.
Disadvantages:
- Loss of flexibility: For each LoRA adapter, you need to save and load a complete model copy, defeating the lightweight purpose of LoRA.
- High storage cost: If you have multiple adapters, the storage overhead is enormous.

Option 2: High-Performance Dynamic Deployment with vLLM (Recommended)

For scenarios requiring simultaneous service of multiple LoRA adapters, vLLM is currently the industry-leading high-performance inference and serving engine. Through core technologies such as PagedAttention, it achieves efficient management and dynamic loading of multiple LoRA adapters, delivering extremely high throughput without significantly sacrificing performance.

Method:

Install vLLM:
```
pip install vllm
```
Start vLLM server: Use the vllm serve command to start an OpenAI-compatible API server. The key is to enable LoRA support with --enable-lora and optionally preload adapters with --lora-modules.
```
# lora_path points to your trained adapter directory
vllm serve meta-llama/Llama-2-7b-hf \
--enable-lora \
--lora-modules my_sql_lora=/path/to/your/sql_lora_adapter
```
Here, we've preloaded an adapter named my_sql_lora.
Send inference requests: You can send requests to the vLLM server using curl or any HTTP client. Just specify the model in the request body as the name of your loaded LoRA adapter.
```
curl http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "my_sql_lora",
"prompt": "Write a SQL query for all users.",
"max_tokens": 64
}'
```
vLLM will automatically route the request to the corresponding LoRA adapter for inference.

Using Python Client: vLLM also provides a Python API for direct calls in code.

from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest
# Initialize LLM engine with LoRA support
llm = LLM(model="meta-llama/Llama-2-7b-hf", enable_lora=True)
sampling_params = SamplingParams(max_tokens=64)
# In the generate call, specify which adapter to use via lora_request
outputs = llm.generate(
"Write a SQL query for all users.",
sampling_params,
lora_request=LoRARequest("my_sql_lora", 1, "/path/to/your/sql_lora_adapter")
)

Advantages:
- Extremely high throughput: Designed for large-scale concurrent inference.
- Dynamic flexibility: Can simultaneously serve hundreds or thousands of LoRA adapters, loading them on demand, perfect for multi-tenant scenarios.
- Memory efficient: PagedAttention mechanism effectively manages GPU memory, avoiding waste.
Disadvantages:
- Slightly more complex deployment: Requires additional learning and configuration of vLLM service.

Option 3: Other Dynamic Deployment Options (e.g., TGI)

Hugging Face's own Text Generation Inference (TGI) is another powerful production-grade inference server. Similar to vLLM, TGI also supports loading multiple LoRA adapters at startup and dynamically applying them based on incoming request headers. It integrates best with the Hugging Face ecosystem and is a strong competitor to vLLM.

Deployment Options Comparison Summary

Feature	Weight Merging (Static)	vLLM (Dynamic)	TGI (Dynamic)
Performance/Throughput	Highest (lowest single request latency)	Very High	High
Flexibility	Low (no dynamic capability)	Very High	High
Deployment Complexity	Low	Medium	Medium
Memory Usage	Very High (N adapters = N times memory)	Low (efficient sharing)	Low (efficient sharing)
Suitable Scenarios	Single, fixed tasks	Multi-tenant, high-concurrency, multi-task scenarios	Production deployment in Hugging Face ecosystem

6. Advanced Topics

Multi-adapter Management: PEFT supports dynamically adding, switching, and disabling multiple adapters on a single model using methods like model.add_adapter() and model.set_adapter(), providing great convenience for building flexible multi-task systems.

7. Conclusion

As a revolutionary parameter-efficient fine-tuning technique, LoRA successfully addresses the high cost challenges of fine-tuning in the era of large models. Through clever low-rank decomposition ideas, it greatly reduces computational resource and storage requirements while maintaining fine-tuning effectiveness. Combined with advanced inference engines like vLLM, LoRA deployment and service have become unprecedentedly efficient and flexible, driving the application of large models in more specific scenarios.

Machine Learning | Ziyang Lin

CLIP Technology Analysis: Unified Representation Through Image-Text Contrastive Learning

1. Introduction

2. Core Concepts

2.1 Multimodal Learning

2.2 Contrastive Learning

2.3 Zero-Shot Learning

3. Model Architecture

4. Workflow

4.1 Training Phase

4.2 Inference Phase (Zero-Shot Classification)

5. Applications

6. Code Example

7. Conclusion

Mixture of Experts (MoE): Sparse Activation Architecture for Large-Scale Neural Networks

1. Introduction

2. Core Components: Macro and Micro Analysis

2.1. Expert Networks: Specialized Processors

Underlying Structure and Variants

Functional Specialization: From General to Specialized

2.2. Gating Network: Intelligent Routing and Dispatch Center

Technical Details

Key Challenge: Load Balancing

3. MoE Model Training Methods: Addressing Scale Challenges

3.1. Expert Parallelism

3.2. Combining with Other Parallelism Strategies

4. Advantages of MoE

5. Challenges of MoE

6. Key Technologies and Recent Advances

7. Summary and Outlook

LLM Hyperparameter Tuning Guide: A Comprehensive Analysis from Generation to Deployment

Introduction

Part 1: Generation (Sampling) Parameters — Controlling Model Creativity and Determinism

1. temperature

2. top_p (Nucleus Sampling)

3. top_k

4. repetition_penalty

5. frequency_penalty & presence_penalty

6. seed

Part 2: Deployment (Serving) Parameters — Optimizing Service Performance and Capacity

1. gpu_memory_utilization

2. max_num_seqs

3. max_model_len

4. tensor_parallel_size & pipeline_parallel_size

Summary and Best Practices

Model Quantization Guide: A Comprehensive Analysis from Theory to Practice

1. Introduction

2. Quantization Fundamentals

2.1 What is Model Quantization?

2.2 Why is Quantization Needed?

2.3 Quantization Principles: Mapping and Dequantization

2.4 Symmetric vs. Asymmetric Quantization

2.5 Per-Layer vs. Per-Group/Per-Channel Quantization

2.6 Common Quantization Paradigms

3. Quantization Schemes in llama.cpp

3.1 GGUF Format and Quantization

3.2 Quantization Type Nomenclature in llama.cpp

3.2.1 Basic Quantization Types (Legacy)

3.2.2 K-Quants (Recommended)

3.3 How to Use the llama-quantize Tool

3.4 Importance Matrix

3.5 Summary

4. vLLM's Quantization Ecosystem

4.1 Mainstream Quantization Schemes Supported by vLLM

4.1.1 GPTQ (General-purpose Post-Training Quantization)

4.1.2 AWQ (Activation-aware Weight Quantization)

4.1.3 FP8 (8-bit Floating Point)

4.1.4 FP8 KV Cache

4.1.5 BitsAndBytes

4.1.6 Other Schemes

4.2 How to Use Quantized Models in vLLM

5. llama.cpp vs. vLLM: Comparison and Summary

6. Latest Quantization Trends and Outlook

7. Conclusion

SGLang Technical Guide: High-Performance Structured Generation Framework

1. SGLang Introduction

2. Core Features

2.1 RadixAttention: KV Cache Optimization for Complex Prompts

2.2 Unified Frontend Programming Language (DSL)

SGLang Architecture Overview

1. `temperature`

2. `top_p` (Nucleus Sampling)

3. `top_k`

4. `repetition_penalty`

5. `frequency_penalty` & `presence_penalty`

6. `seed`

1. `gpu_memory_utilization`

2. `max_num_seqs`

3. `max_model_len`

4. `tensor_parallel_size` & `pipeline_parallel_size`

3.2 Quantization Type Nomenclature in `llama.cpp`

3.3 How to Use the `llama-quantize` Tool

4.1 `@function` Decorator

5.1 Native API: `/generate`

5.2 OpenAI Compatible API: `/v1/chat/completions`

Step 3: Configure LoRA (`LoraConfig`)

Step 4: Apply LoRA and Train with `SFTTrainer`