Activation and Floating-Point Quantization

Activation and Floating-Point Quantization

Weight-Only Quantization focused on weights, which are fixed after training and can be quantized once before deployment. This post turns to tensors created during inference. Activations depend on the input token, sequence position, and previous layers. Their ranges can change from request to request, and a few channels may contain persistent outliers. This makes activation quantization harder than weight-only quantization. In this post, you will learn about: Why dynamic activations are harder to quantize than fixed weights How W8A8 methods handle activation ranges How floating-point quantization differs from integer quantization Let’s get started. Activation and Floating-Point Quantization.Photo by Bruno Martins. Some rights reserved. Overview This post is divided into three parts; they are: Why Activations Are Hard to Quantize W8A8 Inference and SmoothQuant Floating-Point Quantization Why Activations Are Hard to Quantize The post on weight-only quantization introduced scales, zero points, and quantization granularity for fixed weights. The same concepts apply to activations, but the quantization parameters may now need to follow values that do not exist until inference begins. Consider an activation tensor entering a linear layer. A static activation quantizer uses scales chosen during calibration. It is fast because the scales are already known, but a deployment input outside the calibration range may be clipped. A dynamic activation quantizer measures the tensor at runtime. It adapts to each input but adds reduction and scaling work to the inference path. An activation quantizer can be written as follows: import torchdef quantize_activation_per_token(x): qmax = 127 scale = x.abs().amax(dim=-1, keepdim=True) / qmax scale = scale.clamp_min(torch.finfo(x.dtype).eps) q = torch.round(x / scale).clamp(-qmax, qmax) return q.to(torch.int8), scalex = torch.randn(2, 32, 1024)q, scale = quantize_activation_per_token(x)x_hat = q.float() * scaleprint("mean absolute error:", (x - x_hat).abs().mean().item()) For an activation shaped [batch, sequence, hidden], each vector along the final dimension represents one token. Reducing over that dimension produces a scale shaped [batch, sequence, 1]: one scale shared by all hidden features of each token. A linear layer treats these token vectors as matrix rows, so this scheme is also called per-row quantization. Dynamic per-token scaling is a common implementation for transformer activations. It follows token-to-token range changes more closely than one scale for the entire tensor. The example uses signed symmetric INT8, so the zero point is fixed at zero instead of being calculated for every token. This simplifies the quantization and matrix-multiplication arithmetic. The trade-off is that the absolute-maximum reduction and scale calculation must run whenever a new activation tensor is produced. Transformer activations often have outlier values in the vector. Covering the entire range would make the step size coarse for ordinary values. Clipping improves resolution for ordinary values but destroys the outlier. The best choice depends on how the following weights use that vector. Activation distributions also differ by layer and token. Static activation quantization uses calibration data to predict the ranges of future runtime tensors, then select the most appropriate scale and zero point. Dynamic activation quantization avoids fixed calibration ranges but pays the runtime measurement cost. W8A8 Inference and SmoothQuant Here, W8A8 means INT8 weights and INT8 activations for matrix multiplication. When the hardware provides efficient integer matrix multiplication, this can reduce memory traffic and increase arithmetic throughput. Biases, normalization, residuals, or accumulators may remain at higher precision. W8A8 by itself names bit widths; another workflow could use an FP8 format. SmoothQuant addresses activation outliers by moving some quantization difficulty from activations into weights. For a linear layer $\mathbf{y}=\mathbf{x}\mathbf{W}^\top$, choose a positive scale vector $\mathbf{s}$ over input channels: $$ \mathbf{y} = (\mathbf{x} \operatorname{diag}(\mathbf{s})^{-1}) (\mathbf{W} \operatorname{diag}(\mathbf{s}))^\top $$ The transformation is mathematically equivalent before quantization. Large activation channels are divided by larger scales, making the activation range smoother. The corresponding weight columns are multiplied by the same scales. Weights are static and usually tolerate this uneven range better with per-channel quantization. This differs from AWQ in the previous post. AWQ uses calibration activations to protect important weights in a weight-only format such as W4A16; the runtime activations remain in a higher-precision format. SmoothQuant prepares both sides for W8A8 execution: it changes the stored weights and makes the runtime activations easier to quantize to INT8. The following code checks the equivalence: import torchimport torch.nn.functional as Fx = torch.randn(4, 128)weight = torch.randn(256, 128)smooth_scale = torch.rand(128) + 0.5reference = F.linear(x, weight)smoothed_x = x / smooth_scalesmoothed_weight = weight * smooth_scaletransformed = F.linear(smoothed_x, smoothed_weight)torch.testing.assert_close(transformed, reference, rtol=1e-5, atol=1e-5) This only demonstrates the exact rescaling. A SmoothQuant implementation derives smooth_scale from weight magnitudes and calibration activations, folds scales into adjacent operations where possible, and quantizes the transformed tensors. One practical way to create a reusable INT8 weight-and-activation checkpoint is LLM Compressor, a library in the vLLM project. The following workflow uses transformers to load the model, LLM Compressor to apply SmoothQuant and INT8 W8A8 quantization, and then saves the result for vLLM inference: 1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859 from datasets import load_datasetfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom llmcompressor import oneshotfrom llmcompressor.modifiers.gptq import GPTQModifierfrom llmcompressor.modifiers.transform.smoothquant import SmoothQuantModifiermodel_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"output_dir = "models/tinyllama-w8a8"num_calibration_samples = 128max_sequence_length = 512model = AutoModelForCausalLM.from_pretrained( model_id, device_map="auto", dtype="auto",)tokenizer = AutoTokenizer.from_pretrained(model_id)dataset = load_dataset( "HuggingFaceH4/ultrachat_200k", split="train_sft",)dataset = dataset.shuffle(seed=42).select(range(num_calibration_samples))def preprocess(example): text = tokenizer.apply_chat_template(example["messages"], tokenize=False) return {"text": text}def tokenize(example): return tokenizer( example["text"], max_length=max_sequence_length, truncation=True, add_special_tokens=False, )dataset = dataset.map(preprocess)dataset = dataset.map(tokenize, remove_columns=dataset.column_names)recipe = [ SmoothQuantModifier(smoothing_strength=0.8), GPTQModifier(targets="Linear", scheme="W8A8", ignore=["lm_head"]),]oneshot( model=model, dataset=dataset, recipe=recipe, max_seq_length=max_sequence_length, num_calibration_samples=num_calibration_samples,)model.save_pretrained(output_dir, save_compressed=True)tokenizer.save_pretrained(output_dir)print(f"Saved the INT8 checkpoint to {output_dir}") The saved checkpoint contains per-channel INT8 weights and instructions for dynamically quantizing activations during inference. The activations themselves are not stored in the checkpoint because they are produced and quantized per token at runtime. vLLM can load the local checkpoint and use its W8A8 kernels without a separate quantization argument: from vllm import LLM, SamplingParamsmodel = LLM(model="models/tinyllama-w8a8")sampling_params = SamplingParams(temperature=0, min_tokens=16, max_tokens=64)outputs = model.generate( ["Explain activation quantization in one paragraph."], sampling_params,)print(outputs[0].outputs[0].text) This short generation checks that the checkpoint loads and runs. The small model’s answer may be inaccurate; judge quantization quality against the original model on representative evaluation prompts. The checkpoint format, model architecture, vLLM version, and GPU must support the same W8A8 scheme. Current vLLM INT8 kernels target NVIDIA GPUs from Turing through Hopper; Blackwell users should choose a supported FP8 path instead. W8A8 is not guaranteed to be faster than W4A16 or BF16. W8A8 can use integer matrix hardware efficiently, especially for prefill and larger batches. W4A16 reads fewer weight bytes and can be attractive for small-batch decode. You should benchmark both phases. Floating-Point Quantization FP8 quantization does not simply crop bits from an FP32 value. In a common scaled FP8 scheme, the original tensor is divided by a scale and each result is rounded to a representable FP8 value: $$ \begin{aligned} q &= \operatorname{cast}_{\text{FP8}}(x/s), \\ \hat{x} &= s\operatorname{cast}_{\text{FP16 or FP32}}(q) \end{aligned} $$ The scale must accompany the FP8 tensor or block so the runtime can interpret its values. FP8 is therefore still a quantized representation rather than an ordinary unscaled cast to a smaller storage type. Unlike affine INT8, FP8 does not place representable values at one regular step size across the whole quantization group. Its exponent gives it nonuniform spacing: values near zero have finer steps, while larger values have wider steps. This covers a broader range of magnitudes, although rounding and overflow are still possible. Scaling places the source values in the most useful part of that FP8 range; it normally does not require an INT8-style zero-point shift. The word “FP8” does not specify which FP8 layout is used. For example, E4M3 has four exponent bits and three significand bits, while E5M2 has five exponent bits and two significand bits. E4M3 provides more precision, whereas E5M2 provides a wider range. Which one can be used depends on the hardware and inference engine. FP8 also does not specify whether scaling is per tensor or per block, or how scales are calculated and stored. FP4 is similar, but requires more aggressive scaling and appropriate hardware support. The vLLM project’s LLM Compressor can convert a Hugging Face checkpoint to an FP8 checkpoint. Its FP8_DYNAMIC scheme uses round-to-nearest quantization for the fixed weights and derives their scales directly from the weight tensors. Activations are not available during conversion, so vLLM measures them and calculates per-token scales during inference. Neither operation needs representative input samples, which is why this basic workflow does not provide calibration data: 123456789101112131415161718192021 from transformers import AutoModelForCausalLM, AutoTokenizerfrom llmcompressor import oneshotfrom llmcompressor.modifiers.quantization import QuantizationModifiermodel_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"output_dir = "models/tinyllama-w8a8-fp8"model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto")tokenizer = AutoTokenizer.from_pretrained(model_id)recipe = QuantizationModifier( targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"],)oneshot(model=model, recipe=recipe)model.save_pretrained(output_dir, save_compressed=True)tokenizer.save_pretrained(output_dir)print(f"Saved the FP8 checkpoint to {output_dir}") The saved compressed-tensors checkpoint includes the FP8 weights and their scales. Activations are not stored; vLLM calculates their scales and converts them to FP8 during each forward pass. SmoothQuant can be added when activation outliers cause too much error. This makes the conversion calibration-based. The following complete script loads the same model and calibration dataset as the INT8 example, then saves a separate FP8 checkpoint: 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657 from datasets import load_datasetfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom llmcompressor import oneshotfrom llmcompressor.modifiers.quantization import QuantizationModifierfrom llmcompressor.modifiers.transform.smoothquant import SmoothQuantModifiermodel_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"output_dir = "models/tinyllama-w8a8-fp8-smoothquant"num_calibration_samples = 128max_sequence_length = 512model = AutoModelForCausalLM.from_pretrained( model_id, device_map="auto", dtype="auto",)tokenizer = AutoTokenizer.from_pretrained(model_id)dataset = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")dataset = dataset.shuffle(seed=42).select(range(num_calibration_samples))def preprocess(example): text = tokenizer.apply_chat_template(example["messages"], tokenize=False) return {"text": text}def tokenize(example): return tokenizer( example["text"], max_length=max_sequence_length, truncation=True, add_special_tokens=False, )dataset = dataset.map(preprocess)dataset = dataset.map(tokenize, remove_columns=dataset.column_names)recipe = [ SmoothQuantModifier(smoothing_strength=0.8), QuantizationModifier( targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"], ),]oneshot( model=model, dataset=dataset, recipe=recipe, max_seq_length=max_sequence_length, num_calibration_samples=num_calibration_samples,)model.save_pretrained(output_dir, save_compressed=True)tokenizer.save_pretrained(output_dir)print(f"Saved the FP8 checkpoint to {output_dir}") In both cases, you can load the local FP8 checkpoint and run inference as follows: from vllm import LLM, SamplingParamsmodel_path = "models/tinyllama-w8a8-fp8"# Or use: models/tinyllama-w8a8-fp8-smoothquantmodel = LLM(model=model_path)sampling_params = SamplingParams(temperature=0, min_tokens=16, max_tokens=64)outputs = model.generate( ["Explain FP8 quantization in one paragraph."], sampling_params,)print(outputs[0].outputs[0].text) The generated text only confirms that the saved checkpoint runs. Compare accuracy and throughput with the unquantized model before choosing a deployment format. Current vLLM documentation lists NVIDIA Ada Lovelace, Hopper, and Blackwell GPUs, as well as some AMD accelerators, for accelerated FP8 weight-and-activation inference. The exact path depends on the GPU, FP8 layout, and quantization scheme, so verify the current compatibility table for the deployment hardware. Activation quantization may improve matrix-heavy prefill, but its runtime scaling can add overhead during small-batch decode. A combined weight-and-activation scheme can move the bottleneck to another operation, so profile the full request rather than adding isolated speedups. Further Reading Below are some resources you may find useful: PyTorch TorchAO documentation. Quantized Inference. This guide lists current per-token, per-row, per-tensor, and block-scaled activation configurations. Guangxuan Xiao et al. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. This paper introduces mathematically equivalent smoothing for W8A8 inference. vLLM documentation. INT8 W8A8 with LLM Compressor and vLLM. This guide covers calibration, conversion, compressed checkpoint saving, and inference. Paulius Micikevicius et al. (2022). FP8 Formats for Deep Learning. This paper describes FP8 formats and scaling conventions for deep learning. vLLM documentation. FP8 weight-and-activation quantization with LLM Compressor and vLLM. This guide covers dynamic FP8 conversion, checkpoint loading, and supported hardware. Summary In this post, you learned why activations are harder to quantize than weights: their ranges are input-dependent and often contain outliers. SmoothQuant moves part of this difficulty into static weights so W8A8 matrix multiplication becomes practical. FP8 provides floating-point range at eight bits, but still requires explicit format and scaling choices. You also learned how to create reusable INT8 W8A8 and FP8 weight-and-activation checkpoints with LLM Compressor and run them with vLLM. No comments yet.

Original Source

Read the full article at Machinelearningmastery →

KhanList aggregates and links to publicly available news content. We do not host full articles from third-party sources. Always verify important information with original sources.