Model Compression Beyond Quantization

Model Compression Beyond Quantization

Quantization changes how model values are represented. Other compression methods reduce the number of values or the size of model structures. Pruning removes weights or structures, while knowledge distillation trains a smaller student to reproduce a teacher’s behavior. Serving a smaller model is often an effective inference optimization. It reads fewer weights and may use less cache, depending on its layer count, KV heads, and head dimension. It may also allow larger batches or cheaper hardware. The difficult part is preserving the behavior the application needs and ensuring that the compressed structure is supported efficiently by the runtime. In this post, you will learn about: How pruning removes weights and when sparsity can improve execution How structured pruning changes model shapes and compute How distillation trains a smaller model for serving Let’s get started. Model Compression Beyond Quantization.Photo by Kutan Ural. Some rights reserved. Overview This post is divided into three parts; they are: Pruning and Sparsity Structured Pruning Knowledge Distillation Pruning and Sparsity Pruning sets selected model parameters to zero or removes larger structures. Magnitude pruning starts with a simple assumption: weights with small absolute values may have less effect on the output. 1234567891011121314151617181920212223 import torchtorch.manual_seed(0)def magnitude_mask(weight, sparsity): assert 0.0 thresholdweight = torch.randn(1024, 1024)mask = magnitude_mask(weight, sparsity=0.5)pruned_weight = weight * maskprint("remaining fraction:", mask.float().mean().item())# Convert the pruned matrix to compressed sparse row (CSR) format.sparse_weight = pruned_weight.to_sparse_csr()# Multiply a dense batch by the sparse weight matrix.x = torch.randn(8, 1024)sparse_output = torch.sparse.mm(sparse_weight, x.T).Tdense_output = x @ pruned_weight.Ttorch.testing.assert_close(sparse_output, dense_output, rtol=1e-4, atol=2e-4) Sparse and dense matrix multiplication can accumulate terms in a different order, so the script compares their outputs with a numerical tolerance. The sparsity argument represents the fraction of weights intended for removal. Setting sparsity=0.0 retains all weights. The upper bound excludes sparsity=1.0, which would remove every weight and make the layer unusable. After pruning, zeros can appear anywhere in pruned_weight, but that tensor is still stored densely. Calling to_sparse_csr() creates a compressed sparse row tensor that stores the nonzero values together with row and column metadata. The function torch.sparse.mm() then multiplies that sparse matrix by a dense matrix. A sparse representation saves space only when its metadata costs less than the removed data, and it improves speed only when the matrix shape, sparsity level, dtype, and hardware have an efficient sparse kernel. Structured Pruning Structured pruning removes complete units such as attention heads, MLP channels, layers, or blocks. The result can often be represented as a smaller dense model, which makes it easier to accelerate across hardware. A transformer MLP uses one projection to expand each hidden vector into an intermediate representation and another to project it back to the model’s hidden size: $$ \mathbf{y} = \mathbf{W}_{\text{down}} \phi(\mathbf{W}_{\text{up}} \mathbf{x}) $$ Each intermediate channel is produced by one row of $\mathbf{W}_{\text{up}}$ and consumed by the matching column of $\mathbf{W}_{\text{down}}$. Removing a row from the up projection therefore requires removing the corresponding column from the down projection. The following small example removes intermediate channels according to the norms of the first projection’s rows: 123456789101112131415161718192021222324252627282930313233343536 import torchimport torch.nn as nndef shrink_mlp(up_proj, down_proj, keep_channels): if not 1 <= keep_channels <= up_proj.out_features: raise ValueError("keep_channels must be within the MLP width") if up_proj.out_features != down_proj.in_features: raise ValueError("projection widths must match") scores = up_proj.weight.norm(dim=1) keep = scores.topk(keep_channels).indices.sort().values smaller_up = nn.Linear( up_proj.in_features, keep_channels, bias=up_proj.bias is not None, ).to(device=up_proj.weight.device, dtype=up_proj.weight.dtype) smaller_down = nn.Linear( keep_channels, down_proj.out_features, bias=down_proj.bias is not None, ).to(device=down_proj.weight.device, dtype=down_proj.weight.dtype) with torch.no_grad(): smaller_up.weight.copy_(up_proj.weight[keep]) smaller_down.weight.copy_(down_proj.weight[:, keep]) if up_proj.bias is not None: smaller_up.bias.copy_(up_proj.bias[keep]) if down_proj.bias is not None: smaller_down.bias.copy_(down_proj.bias) return smaller_up, smaller_downup = nn.Linear(16, 32)down = nn.Linear(32, 16)smaller_up, smaller_down = shrink_mlp(up, down, keep_channels=16)print(smaller_down(torch.relu(smaller_up(torch.randn(2, 16)))).shape) The parameter keep_channels is the number of intermediate MLP channels retained after pruning; it is a count, not a sparsity fraction. It must be an integer from 1 through up_proj.out_features, and the original projections must satisfy up_proj.out_features == down_proj.in_features. The code ranks the rows of up_proj.weight by their norms and keeps the keep_channels highest-scoring rows. It then keeps the matching columns of down_proj.weight, so the reduced output of smaller_up remains a valid input to smaller_down. For example, setting keep_channels=2048 reduces an original intermediate width of 8192 to 2048 channels. This code preserves compatible tensor shapes but does not determine whether the chosen channels are truly unimportant. A real pruning method evaluates layer outputs, interactions, and recovery fine-tuning. Removing attention heads requires similar coordinated edits to query, key, value, and output projections. With GQA, query heads share KV heads, so arbitrary head removal can break grouping. Removing transformer layers changes residual depth and may require learning mappings between the retained layers. These changes require coordinated edits and quality checks. Structured pruning can reduce parameter count and dense compute, but awkward hidden sizes may perform worse than slightly larger hardware-friendly sizes. Align channel counts with the tile requirements of the target matrix multiplication kernels. Knowledge Distillation A smaller model can run faster and use less memory. Knowledge distillation trains a smaller student model using signals from a larger teacher. The student may learn from the teacher’s output distribution (not just the highest-scoring token), intermediate representations, generated data, or a combination of them. For logits $\mathbf{z}_t$ and $\mathbf{z}_s$ from teacher and student, a common distillation loss function uses a temperature $T$: $$ \mathcal{L}_{KD} = T^2\operatorname{KL}\Big( \operatorname{softmax}\big(\frac{\mathbf{z}_t}{T}\big)\;\big\Vert \;\operatorname{softmax}\big(\frac{\mathbf{z}_s}{T}\big) \Big) $$ Here, $\operatorname{KL}(\mathbf{P}\Vert \mathbf{Q})$ is the Kullback-Leibler divergence from the teacher distribution $\mathbf{P}$ to the student distribution $\mathbf{Q}$; it is zero when they match and grows as their probabilities differ. The corresponding PyTorch code is: 123456789101112131415161718 import torchimport torch.nn.functional as Fdef distillation_loss(student_logits, teacher_logits, temperature=2.0): if temperature <= 0: raise ValueError("temperature must be positive") teacher_probs = F.softmax(teacher_logits / temperature, dim=-1) student_log_probs = F.log_softmax(student_logits / temperature, dim=-1) return ( F.kl_div(student_log_probs, teacher_probs, reduction="none") .sum(dim=-1) .mean() * temperature**2 )student_logits = torch.tensor([[1.0, 0.0]])teacher_logits = torch.tensor([[0.0, 1.0]])print("distillation loss:", distillation_loss(student_logits, teacher_logits).item()) For padded batches, replace .mean() with a masked average over valid token positions. This loss assumes teacher and student logits refer to the same vocabulary and token positions. The teacher runs without gradients, but producing its targets still costs time. The next example defines the same loss so it can run independently. The following local example uses two small, randomly initialized GPT-2 style models. It demonstrates one training step; it does not produce a useful language model. A real distillation experiment needs a pretrained teacher and representative text. 123456789101112131415161718192021222324252627282930313233343536373839 import torchimport torch.nn.functional as Ffrom transformers import GPT2Config, GPT2LMHeadModeldef distillation_loss(student_logits, teacher_logits, temperature=2.0): if temperature <= 0: raise ValueError("temperature must be positive") teacher_probs = F.softmax(teacher_logits / temperature, dim=-1) student_log_probs = F.log_softmax(student_logits / temperature, dim=-1) return ( F.kl_div(student_log_probs, teacher_probs, reduction="none") .sum(dim=-1) .mean() * temperature**2 )torch.manual_seed(0)vocab_size = 128teacher = GPT2LMHeadModel(GPT2Config( vocab_size=vocab_size, n_layer=4, n_head=4, n_embd=64,)).eval()student = GPT2LMHeadModel(GPT2Config( vocab_size=vocab_size, n_layer=2, n_head=4, n_embd=64,)).train()optimizer = torch.optim.AdamW(student.parameters(), lr=1e-3)tokens = torch.randint(0, vocab_size, (4, 24))with torch.no_grad(): teacher_logits = teacher(tokens).logits[:, :-1, :]student_logits = student(tokens).logits[:, :-1, :]hard_loss = F.cross_entropy( student_logits.reshape(-1, vocab_size), tokens[:, 1:].reshape(-1))soft_loss = distillation_loss(student_logits, teacher_logits)loss = 0.5 * hard_loss + 0.5 * soft_lossoptimizer.zero_grad()loss.backward()optimizer.step()print("training loss:", loss.item()) For a real comparison, train on a representative data split and report held-out task quality, parameter bytes, peak memory, prefill latency, and decode throughput for the same prompts and hardware. Pruning or distillation can change behavior even when parameter count falls substantially. Further Reading Below are some resources you may find useful: Jonathan Frankle, Michael Carbin (2018). The Lottery Ticket Hypothesis. This paper studies sparse subnetworks that can train effectively within larger networks. Elias Frantar, Dan Alistarh (2023). SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. This paper presents a post-training pruning method for large language models. Geoffrey Hinton, Oriol Vinyals, Jeff Dean (2015). Distilling the Knowledge in a Neural Network. This paper introduces the commonly used soft-target distillation formulation. Victor Sanh et al. (2019). DistilBERT. This paper applies knowledge distillation while pretraining a smaller transformer. Summary In this post, you learned that compression can change model structure, not only number formats. Unstructured pruning needs sparse storage and kernels to improve runtime, while structured pruning can produce smaller dense shapes. Distillation transfers behavior into a smaller student model. No comments yet.

Original Source

Read the full article at Machinelearningmastery →

KhanList aggregates and links to publicly available news content. We do not host full articles from third-party sources. Always verify important information with original sources.