How to Fine-Tune an LLM: An End-to-End Guide

? Let me provide a real, personal example. We fine-tuned a 7B parameter model which completely blows foundation models out of the water, but just for this very narrow subtask: Filling out synoptic reporting templates for breast cancer. This is a hellishly difficult task with complex input formats, branching logic, and fields that must appear in a strict order. The LLM needs to discern which of 40 different histologic subtypes trigger which branch of field subsets, perfectly, without hallucination. It’s near impossible to define in a conditional table. One wrong field, and the entire output is invalidated. When we used aggressive system prompts along with some light RAG, our accuracy (with Claude Opus 4.6) was ~35%. We had to include the entire body of the template in the context, along with a detailed guide when/where to use which field. Roughly 30k tokens, per call. The result? Omitted fields here, unnecessary subsections there, hallucinations, etc. which means that a human would need to manually read and edit the entire document. A no-go. After fine-tuning a Mistral 7B model (with QLoRA), our accuracy jumped to ~98%. I was stunned on how effective it actually was. Prompt + RAG ███████░░░░░░░░░░░░ 35% QLoRA ███████████████████░ 98% We got an improvement of 63 percentage points and completely eliminated our API costs (for this task). Our initial cost estimates for running this at the scale we needed (with the frontier model) would have been ~$320,000. We got there for free.* ** Not including the price for fine-tuning, running a local model (which we already do at scale), or measuring the energy usage per API call That’s why you fine-tune. Despite the common belief, RAG + System prompts will not solve fine-tuning problems, and they’re everywhere. In this article, I’ll cover: When to fine-tune (the RAG vs Fine-Tune debate) The mathematical intuition behind LoRA/QLoRA The technical implementation details Evaluating using a custom harness After reading, you’ll know when to fine-tune, why it works, and exactly how to implement it in practice When to fine-tune ❓❓Do I actually need to fine-tune?❓❓ Perhaps. Look for one of the following fine-tuning patterns: Rigid, Highly Specific Formatting Requirements You need the LLM to output specific formats which are very complex and unforgiving to the occasional hallucination, like a missed or added field. Some notable examples: Legacy Enterprise Documents: Large companies often have deeply ingrained, idiosyncratic templates with countless conditional branches. Court/Legal Documents: where each jurisdiction has its own format and template. These forms obviously weren’t part of the LLMs input data, and need to be introduced as new knowledge. Medical Forms: They’re complex, often contain redundant information and need to be just right. Cost constraints Thousands (or tens of thousands) of tokens in a system prompt which runs on every single API call for every customer inquiry. At scale, that’s real money and latency. A fine-tuned model that has internalized those patterns needs neither. Complex Instructions and Combinatorial Explosions System prompts work with simple constraints, but they often break down when rules overlap. If your task involves a massive decision tree (e.g., “If A, do B, but if C and A, do D, unless E is present…”), you may hit the limits of in-context learning. In our case, our combinatorial space exploded with rules that couldn’t reasonably be encoded into a table. Also, remember, context degrades with length. A system prompt with 50 different rules is likely to omit one here or there, invalidating the entire output. Custom Tone This isn’t relevant to us, but worth a mention. If you require a specific “brand voice” for your customer service agent, fine-tuning often works better than using system prompts. Also relevant: you need to add a system prompt to every single customer interaction to maintain a specific voice or tone. If your “brand voice” prompt is 2,000 tokens, the costs can add up quickly. When to use RAG A rule of thumb most practitioners use: RAG mostly augments the model’s knowledge. Fine-tuning mostly affects the output behavior. I say mostly, because full fine-tuning can definitely add new knowledge to an LLM, and RAG can (and is often used to) modify the default behavior of an LLM. It’s not black and white, so use your best judgement. Where RAG might be a better choice: Your knowledge base changes frequently You only need to augment the behavior rarely The model needs access to documents, policies, or facts that evolve over time You only have a few hundred high quality training examples You can reliably modify the behavior with a small system prompt Working solutions tend to end up as a mixture of both. We didn’t completely eliminate system prompts or RAG, we just greatly reduced our reliance on them. Now, let’s dive into the math so we can understand the mechanics of fine-tuning The mathematical intuition behind LoRA/QLoRA Before covering any of the mathematical detail of LoRA (Low Rank Adaptation), and its derivatives, conceptually grouped together as “Parameter Efficient Fine Tuning (PEFT)”, we need to understand what fine-tuning actually is doing under the hood. 🤔 Why take time to understand the math behind LoRA/QLoRA Understanding the math behind LoRA is critical to understand if its the right strategy for the task at hand. It’s the dividing factor between people who truly understand why/when to fine-tune vs why/when to RAG. I’d recommend not copy-pasting the training script and using the defaults provided, which might work for you right out of the box. Instead, develop a mathematical intuition for what’s happening here. That way, debugging becomes less guess work and more precision engineering. For example: Misunderstanding what “rank” is in LoRA will make it difficult to diagnose overfitting problems. E.g. Why chose rank 8 over rank 32? What does lowering the alpha parameter do to the residual stream? You won’t have to derive any formulae from scratch. I’ve described it in a way which is (hopefully) accessible if you have any understanding for how large transformer based models work. Fine-tuning is a continuation of the large pretraining task that LLMs undergo. In pretraining, the objective is typically predict the next token in a sequence of natural language. In supervised fine-tuning (SFT), the same causal next-token prediction objective is applied to specific examples: (prompt, completion) pairs. In many SFT setups, the loss is computed exclusively on the assistant/completion tokens rather than on the user’s prompt. The mathematical objective is approximately the same: Cross Entropy (Negative Log Likelihood) over tokens. ℒ(θ)=−𝔼(x,y)∼𝒟[∑t=1Tlog⁡Pθ(yt|x,y value 0000 -> -1.0 0001 -> -0.875 0010 -> -0.75 ... We can indeed take each one of a network’s possible FP16 weight values and map it to a 4bit code. However, this wastes capacity, as only a small percentage of the weights occupy the 0000 bit interval (the tail) and are more concentrated around zero. NF4 uses the expected distribution of neural networks weights to construct a more representative set of 16 quantization values. Since normalized weights are typically approximately Gaussian and centered around zero, the quantized values are placed more densely around zero and more sparsely in the tails. Thus, our naive bins (from the example above) become smarter and more representative of the actual weights. E.g. bit -> value 0000 -> -1.0000 0001 -> -0.6962 0010 -> -0.5251 ... We gain precision, as the actual weights can be reconstructed with less quantization error. Double Quantization 4bit quantizing works, but each layer may have a different distribution. If we created a global scaling factor, we’d lose significant precision. Thus, we need to divide all the weights into blocks (typically 64). Each block of weights has its own unique scaling factor, a 32bit floating point value that’s used to map the 4bit codes back to their full precision counterpart. What double quantization achieves is a quantization of the unique scaling factors, so that we don’t have to keep all of the full precision 32bit floats in VRAM. The first step is quantizing the weights into NF4: Divide all model weights into blocks of 64 Compute one FP32 scaling constant per block Quantize weights to 4bit NF4 using the process above Second level (the “double” part): Group 256 of those FP32 scaling constants together Quantize them down to 8bit floats (FP8) Store just one FP32 scaling constant per group of 256 Per the paper, you can save roughly 0.373 bits per parameter. It’s small, but across billions of parameters the VRAM savings are worth it (potentially 2-3GB of VRAM) Paged Optimizers Paged Optimizers are a nice NVIDIA unified memory hack. For long context sequences, occasionally, VRAM requirements spike, causing dreaded OOM states. Ultimately, paged optimizers are primarily a mechanism for handling memory spikes. They allow some of the optimizer states to reside outside GPU memory when GPU capacity becomes constrained. Otherwise, the LoRA implementation still holds. Instead of running forward/backward passes on a frozen, FP16 model, we can run our these passes on a highly quantized model, while maintaining LoRA adapters in full precision. The result is a set of clever memory management techniques that makes fine tuning very large models much more memory efficient. TLDR: LoRA makes fine tuning memory efficient. QLoRA makes LoRA more memory efficient. It’s an optimization of an optimization that retains performance with much less horsepower. Why QLoRA instead of full fine-tuning? Our model had 7B parameters. Full fine-tuning was possible, but it offered little benefit for this task relative to the additional memory and compute requirements. LoRA gave us another advantage: we could experiment with the adapter independently of the base model. That made QLoRA the obvious starting point. ApproachWhy we rejected/selected itPromptingToo inconsistentRAGDidn’t solve behavioral consistencyFull fine-tuningUnnecessary memory/computeLoRAGood parameter efficiencyQLoRASame LoRA approach with substantially lower base-model memory For us, QLoRA made financial sense. We don’t have an endless waterfall of compute resources, so we need to be smart. We’d spend less in compute and inference for a result which was (potentially) as accurate as a full fine-tune. If successful, we could build different adapters for different subtasks, allowing for even further specialization. We’re working on this now 😎 The technical implementation details Where the rubber meets the road. If you’re anything like us, you will spend: 70% of your time on generating high quality training data 20% of your time running evaluations and only 10% of your time actually running the fine-tune In fact, the actual fine tuning script is lightweight and very easy to run and understand. 😱You may be intimidated about running your own training pipeline for your fine tune. 😎 Don’t be. You’ll waste time and effort trying to customize some templated fine-tuning pipeline from X number of companies offering it. The script I offer below can be easily customized to your requirements. Unfortunately, I don’t know what your exact use case is. However, I can outline the requirements for our use case in the hope that you can form analogies where it makes sense. Either way, the data is the most painful part of this process. Generating high quality training data Case 1: You have 4,000–10,000+ real input/output pairs. This is an ideal situation, as the fine-tuning process really shines when you want to steer an LLM to answer in a very specific way. Customer service companies will likely live in this space. Case 2: Our situation. You have outputs, but no inputs. This is more difficult. How do you steer an LLM when there is nothing to steer with? It’s akin to trying to build a classifier with just targets, no features. Specifically, for our use case, we just had the raw corpus of finalized, structured synoptic reports. We needed the free hand notes the pathologist compiled before creating the structured report. We thought about running a full fine tune, hoping that the LLM would retain the formatting, structure and highly nuanced requirements of a finalized report. Ultimately, we decided against it. We needed a very specific solution: take an unstructured pathologist’s note and turn it into a structured synoptic report. Luckily, we found a solution. In “Self-Alignment with Instruction Backtranslation“, Li et al. reverse the problem. Instead of “given this input, generate the most likely output”, the problem is now “given this output, generate the most likely input.” We can use a language model to generate a plausible, synthetic input. Then, curate the pairs with the highest quality examples, then run a fine-tune on the curated dataset. Specifically, what we did was: Gathered 10,000 real synoptic reports (we have 200,000) Used a frontier model to generate (4) real, unstructured pathology narratives for each output (using 4 different system prompts) Curated the list of output pairs based on our initial error evaluations. Specifically, we wanted data that our initial foundation model trial failed on. Used (synthetic input, real output) as our training dataset. Why 4 different inputs per each output? We generated multiple candidate narratives so our model would learn diversity, that is, so that fine-tuned model can learn that the exact same output is reachable from a diverse set of inputs. We used different highly restrictive system prompts that match what clinician’s notes actually look like, along with restrictions on using field headers, so that the model learned how to map, not just how to reiterate facts from the note. All we then needed to do was format it for our training pipeline, each example was formatted as so: { "prompt_id": "xyz123", "prompt": "Received fresh, labeled with patient identifiers, a 3.2 x 2.8 x 2.1 cm...", "completion": "..." } Our final corpus consisted of ~8,000 input, output pairs with overlap. If you’re generating data in this method, I highly recommend generating multiple candidate inputs for each output. You want the LLM to conceptualize the fine-tuning data, so that minor variations in the input prompts don’t lead to hallucinations. The training pipeline Again, as I mentioned, running your own fine-tune is actually quite accessible. Paired with a thorough understanding of what’s going on underneath the hood (please read the math section), you will be well equipped to diagnose any sort of training errors. The training process is: Perform a hyperparameter search over relevant hyperparameters. Every hyperparameter isn’t a first class citizen, some LoRA hyperparameters have more influence than others Once a candidate set is found, run the fine-tune. The most important step, evaluate the outputs. Here, we’re not only concerned with the loss values, but also the quality and correctness of the output solutions. I provide the full script below, as well as a walk through some notable sections of the script, explaining what is happening and why. Hardware The goal with this article was to convey the reasonableness of training on a single, consumer size GPU. Thus, we want to use QLoRA and limit our VRAM requirements to 24 hours. For a safe margin, budget $50-100 in compute costs for a single QLoRA fine-tune. If you want to go ultra-budget, you could try an even smaller model and run this entirely on a T4, which makes Google Colab an option. Hyperparameters There are many hyperparameters to chose from. I selected reasonable defaults based on two notable sources, which are actually quite accurate. Source 1: Unsloth | LoRA Hyperparameters Guide This is a well researched and thorough guide to the hyperparameters which count for fine-tuning. We didn’t use Unsloth’s quantized models for our fine-tune, but it’s a reasonable option. Source 2: Thinking Machines Lab | LoRA without Regret Again another thoughtful review of the hyperparameters which matter in LoRA. Schulman exposes some “parameter invariances” which reduce the number of hyperparameters which are actually relevant. The big takeaway In the original LoRA paper, the authors only applied LoRA to the attention matrices of the LLM. However, Schulman shows that applying LoRA to all the weight matrices results in better performance. “Even in small data settings, LoRA performs better when applied to all weight matrices, especially MLP and MoE layers. Attention-only LoRA underperforms even when we match the number of trainable parameters by using higher rank for attention-only LoRA.” In the provided script, I don’t include automated hyperparameter searches. However, I’ll highlight the most relevant hyperparameters along with sensible defaults, cherry picked from the two sources above. If a sweep is recommended, the sweep values are in a list. Otherwise, just use the default value. HyperparameterWhat It ControlsRecommended Value(s)Learning RateStep size for adapter weight updatesDefault: 2e-4 Sweep: [5e-5, 1e-4, 2e-4, 4e-4, 8e-4].LoRA Rank (r)Adapter capacity / trainable-parameter count.Default: 16Sweep: [8, 16, 32, 64, 128].LoRA AlphaScales adapter output by α/r; interacts with LR rather than acting independently.Default: 2rEffective Batch Size (batch_size × grad_accum_steps)Trade-off between gradient stability and VRAM/training time.Default: 16 EpochsNumber of passes over the training set.Default: 1–3 epochsLoRA DropoutRegularization on adapter activations.Default: 0 Sweep (only if overfitting): [0, 0.05, 0.1].Weight DecayPenalty on weight magnitude.Default: 0.01LR Scheduler / WarmupShape of the LR curve over training.Default: a linear or cosine scheduler with warmup over the first 5–10% of total steps. This gives us reasonable defaults, selected based on empirical research. Now, let’s wire everything up in a PyTorch script. Step 1: Install prerequisites pip install torch transformers peft trl bitsandbytes datasets accelerate Step 2: Run the script ⚠️This is a very trimmed down, LLM edited, version of our training job for QLoRA. You’ll probably need to update it, especially if you’re going to do hyperparameter sweeps. I promise, I reviewed it for slop. Expand this block for the full script """ QLoRA fine-tuning for Mistral-7B-Instruct. Hyperparameters and defaults: Learning Rate 2e-4 Step size for adapter weight updates. LoRA Rank (r) 16 Adapter capacity / trainable-parameter count. LoRA Alpha 2*r Scales adapter output by alpha/r. Effective Batch Size 16 batch_size * grad_accum_steps. Epochs 3 Number of passes over the training set. LoRA Dropout 0 Regularization on adapter activations. Weight Decay 0.01 Penalty on weight magnitude. LR Scheduler cosine Linear or cosine, with 5-10% warmup. Usage python train_qlora.py --rank 32 --lr 1e-4 --epochs 2 python train_qlora.py --rank 32 --lr 1e-4 --epochs 2 --merge """ import argparse import os import torch from transformers import ( AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, EarlyStoppingCallback, TrainerCallback, ) from peft import LoraConfig, PeftModel, get_peft_model, prepare_model_for_kbit_training from trl import SFTTrainer, SFTConfig, DataCollatorForCompletionOnlyLM from datasets import load_dataset BASE_MODEL = "mistralai/Mistral-7B-Instruct-v0.3" DATA_FILE = # ADD YOUR JSON DATAFILE! RESPONSE_TEMPLATE = "[/INST]" # everything after this is the assistant turn OUTPUT_ROOT = "./mistral-lora" SYSTEM_PROMPT = ( # ADD A SYSTEM PROMPT HERE ) class PrintLossCallback(TrainerCallback): def on_log(self, args, state, control, logs=None, **kwargs): if logs: if "loss" in logs: print(f"Step {state.global_step} | Train Loss: {logs['loss']:.4f}") if "eval_loss" in logs: print(f"Step {state.global_step} | Eval Loss: {logs['eval_loss']:.4f}") def build_dataset(tokenizer, data_file=DATA_FILE, eval_fraction=0.1, seed=42): dataset = load_dataset("json", data_files=data_file, split="train") def to_chat_text(example): messages = [ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": example["prompt"]}, {"role": "assistant", "content": example["completion"]}, ] return {"text": tokenizer.apply_chat_template(messages, tokenize=False)} dataset = dataset.map(to_chat_text) split = dataset.train_test_split(test_size=eval_fraction, seed=seed) return split["train"], split["test"] def load_base_model(): bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True, ) model = AutoModelForCausalLM.from_pretrained( BASE_MODEL, quantization_config=bnb_config, device_map="auto", ) return prepare_model_for_kbit_training(model) def build_lora_model(rank, alpha, dropout): model = load_base_model() lora_config = LoraConfig( r=rank, lora_alpha=alpha, target_modules=[ "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", ], lora_dropout=dropout, bias="none", task_type="CAUSAL_LM", ) return get_peft_model(model, lora_config) def train(hp, train_ds, eval_ds, tokenizer, collator, output_dir): model = build_lora_model( rank=hp["rank"], alpha=hp["lora_alpha"], dropout=hp["lora_dropout"], ) model.print_trainable_parameters() sft_config = SFTConfig( output_dir=output_dir, per_device_train_batch_size=hp["per_device_train_batch_size"], gradient_accumulation_steps=hp["gradient_accumulation_steps"], num_train_epochs=hp["epochs"], learning_rate=hp["learning_rate"], lr_scheduler_type=hp["lr_scheduler_type"], warmup_ratio=hp["warmup_ratio"], weight_decay=hp["weight_decay"], bf16=True, gradient_checkpointing=True, logging_steps=10, eval_strategy="epoch", save_strategy="epoch", load_best_model_at_end=True, metric_for_best_model="eval_loss", greater_is_better=False, max_seq_length=32768, report_to="none", ) trainer = SFTTrainer( model=model, args=sft_config, train_dataset=train_ds, eval_dataset=eval_ds, data_collator=collator, callbacks=[ EarlyStoppingCallback(early_stopping_patience=3), PrintLossCallback(), ], ) trainer.train() metrics = trainer.evaluate() return trainer, metrics def save_adapter(trainer, tokenizer, output_dir): """Save just the LoRA adapter (a few hundred MB) plus the tokenizer.""" trainer.save_model(output_dir) tokenizer.save_pretrained(output_dir) print(f"Adapter saved to {output_dir}") def merge_and_save(base_model_name, adapter_dir, merged_dir): """Fold the LoRA adapter back into the base weights for standalone serving. Reloads the base model at full bf16 precision (not the 4-bit quantized copy used during training) so the merge itself doesn't compound quantization error, then writes out a standard dense model you can load with AutoModelForCausalLM like any other checkpoint -- no PEFT dependency needed at inference time. """ print(f"Loading base model for merge: {base_model_name}") base_model = AutoModelForCausalLM.from_pretrained( base_model_name, torch_dtype=torch.bfloat16, device_map="auto", ) merged_model = PeftModel.from_pretrained(base_model, adapter_dir) merged_model = merged_model.merge_and_unload() merged_model.save_pretrained(merged_dir, safe_serialization=True) tokenizer = AutoTokenizer.from_pretrained(adapter_dir) tokenizer.save_pretrained(merged_dir) print(f"Merged model saved to {merged_dir}") def main(): parser = argparse.ArgumentParser() parser.add_argument("--data_file", default=DATA_FILE) parser.add_argument("--eval_fraction", type=float, default=0.1) parser.add_argument("--lr", type=float, default=2e-4) parser.add_argument("--rank", type=int, default=16) parser.add_argument("--lora_alpha", type=int, default=None) parser.add_argument("--batch_size", type=int, default=4) parser.add_argument("--grad_accum", type=int, default=4) parser.add_argument("--epochs", type=int, default=3) parser.add_argument("--dropout", type=float, default=0.0) parser.add_argument("--weight_decay", type=float, default=0.01) parser.add_argument("--scheduler", choices=["linear", "cosine"], default="cosine") parser.add_argument("--warmup_ratio", type=float, default=0.05) parser.add_argument( "--merge", action="store_true", help="Merge the LoRA adapter into the base model weights after training.", ) parser.add_argument( "--merged_dir", default=None, help=f"Output directory for the merged model (default: {OUTPUT_ROOT}/merged). Only used with --merge.", ) args = parser.parse_args() os.makedirs(OUTPUT_ROOT, exist_ok=True) lora_alpha = args.lora_alpha if args.lora_alpha is not None else 2 * args.rank hp = { "learning_rate": args.lr, "rank": args.rank, "lora_alpha": lora_alpha, "per_device_train_batch_size": args.batch_size, "gradient_accumulation_steps": args.grad_accum, "epochs": args.epochs, "lora_dropout": args.dropout, "weight_decay": args.weight_decay, "lr_scheduler_type": args.scheduler, "warmup_ratio": args.warmup_ratio, } print("Hyperparameters:", hp) tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL) tokenizer.pad_token = tokenizer.eos_token train_ds, eval_ds = build_dataset( tokenizer, data_file=args.data_file, eval_fraction=args.eval_fraction ) collator = DataCollatorForCompletionOnlyLM( response_template=RESPONSE_TEMPLATE, tokenizer=tokenizer ) trainer, metrics = train( hp, train_ds, eval_ds, tokenizer, collator, output_dir=os.path.join(OUTPUT_ROOT, "run"), ) print("Final eval metrics:", metrics) adapter_dir = os.path.join(OUTPUT_ROOT, "final") save_adapter(trainer, tokenizer, adapter_dir) if args.merge: merged_dir = args.merged_dir or os.path.join(OUTPUT_ROOT, "merged") merge_and_save(BASE_MODEL, adapter_dir, merged_dir) if __name__ == "__main__": main() Part 1: Load in your data The script contains some placeholders for your dataset. Ensure the dataset is json, with both "prompt", "completion" fields. def build_dataset(tokenizer, data_file=DATA_FILE, eval_fraction=0.1, seed=42): dataset = load_dataset("json", data_files=data_file, split="train") def to_chat_text(example): messages = [ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": example["prompt"]}, {"role": "assistant", "content": example["completion"]}, ] return {"text": tokenizer.apply_chat_template(messages, tokenize=False)} dataset = dataset.map(to_chat_text) split = dataset.train_test_split(test_size=eval_fraction, seed=seed) return split["train"], split["test"] Here, we simply load in the data, format it as a text string and split it into train and test. We don’t need to worry about tokenization or tensorization, as we tokenize separately and SFTTrainer also has some internal hooks that tensorize. Part 2: Build the model def load_base_model(): bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True, ) model = AutoModelForCausalLM.from_pretrained( BASE_MODEL, quantization_config=bnb_config, device_map="auto", ) return prepare_model_for_kbit_training(model) def build_lora_model(rank, alpha, dropout): model = load_base_model() lora_config = LoraConfig( r=rank, lora_alpha=alpha, target_modules=[ "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", ], lora_dropout=dropout, bias="none", task_type="CAUSAL_LM", ) return get_peft_model(model, lora_config) From the original QLoRA paper, BitsAndBytes, handles quantization for us. Otherwise, we rely heavily on HuggingFace’s transfomers library, which is very standard in the industry for running inference and fine-tuning on LLMs. The other thing worth noting here is that in the build_lora_model function, we target all the modules (target_modules) from the base LLM, as noted from Schulman’s research above. It does work in practice!! Part 3: The training loop def train(hp, train_ds, eval_ds, tokenizer, collator, output_dir): model = build_lora_model( rank=hp["rank"], alpha=hp["lora_alpha"], dropout=hp["lora_dropout"], ) model.print_trainable_parameters() sft_config = SFTConfig( output_dir=output_dir, per_device_train_batch_size=hp["per_device_train_batch_size"], gradient_accumulation_steps=hp["gradient_accumulation_steps"], num_train_epochs=hp["epochs"], learning_rate=hp["learning_rate"], lr_scheduler_type=hp["lr_scheduler_type"], warmup_ratio=hp["warmup_ratio"], weight_decay=hp["weight_decay"], bf16=True, gradient_checkpointing=True, logging_steps=10, eval_strategy="epoch", save_strategy="epoch", load_best_model_at_end=True, metric_for_best_model="eval_loss", greater_is_better=False, max_seq_length=32768, report_to="none", ) Some of the more important fields in our SFTTrainer and SFTConfig GroupFieldsWhat it doesBatch shapeper_device_train_batch_size, gradient_accumulation_stepsEffective batch size = the two multiplied together. You process a small batch on-GPU, accumulate gradients over several steps, then apply one optimizer update.Schedulenum_train_epochs, learning_rate, lr_scheduler_type, warmup_ratioHow long to train and how the LR moves Precision & memorybf16=True, gradient_checkpointing=Truebf16 runs the trainable LoRA math and gradients in bfloat16Regularizationweight_decayStandard penalty on the adapter weights, discourages them from growing unboundedly large.Logginglogging_steps=10How often PrintLossCallback gets a logs dict to print from.Eval/checkpointingeval_strategy="epoch", save_strategy="epoch", load_best_model_at_end=True, metric_for_best_model="eval_loss", greater_is_better=FalseEvaluate and checkpoint once per epoch. Because both strategies are set to "epoch" (they have to match for load_best_model_at_end to work), at the end of training the Trainer automatically swaps back in whichever epoch’s checkpoint had the lowest eval_loss: even if it wasn’t the last one. Without this, you’d keep whatever the final epoch produced, which could already be overfitting.Sequence lengthmax_seq_length=32768Truncates the formatted chat text (system + note + report) beyond 32768 tokens. This is intentionally large for our use case because some pathology notes and structured reports are long. Part 4: Saving and merging the adapterOnce training finishes, you’ve got a LoRA adapter: a few hundred megabytes, not a full model. That’s great for storage (you can keep dozens of task-specific adapters against a single base model), but for production inference you usually want to merge the adapter back into the base weights, so there’s no extra matrix multiplication at serving time and you can deploy it behind a standard inference stack. def save_adapter(trainer, tokenizer, output_dir): trainer.save_model(output_dir) tokenizer.save_pretrained(output_dir) def merge_and_save(base_model_name, adapter_dir, merged_dir): base_model = AutoModelForCausalLM.from_pretrained( base_model_name, torch_dtype=torch.bfloat16, device_map="auto", ) merged_model = PeftModel.from_pretrained(base_model, adapter_dir) merged_model = merged_model.merge_and_unload() merged_model.save_pretrained(merged_dir, safe_serialization=True) tokenizer = AutoTokenizer.from_pretrained(adapter_dir) tokenizer.save_pretrained(merged_dir) Evaluating using a custom harness Again, I don’t know what your exact use case is, but I will provide some of the decisions we made. In a synoptic report template, field structure for each histologic subtype is fully determined by the template (that’s the whole reason this task is a fine-tuning candidate and not a RAG candidate). We could encode each subtype’s required field list and required order as a schema, then check a candidate document against it along four axes: Recall: are all required fields for the correct branch present? Precision: are there any hallucinated fields that don’t belong to that branch? Order: do the present fields appear in the required strict sequence? Value correctness: for categorical/numeric fields, does the value match; for free-text fields, is it semantically equivalent? What the harness actually found Running the harness against both systems on the same held out set is what turned “it felt better” into a number we could defend: MetricFoundation model (RAG + prompt)Fine-tuned Mistral 7BStrict document accuracy~35%~98%Docs with ≥1 hallucinated field~41%~2%Docs with ≥1 omitted field~52%~3%Docs with an order violation~23%<1%Mean field-level accuracy~71%~99.4% Lessons learned A few things we’d tell ourselves at the start of this project: Loss ≠ correctness. A plateauing eval_loss can hide a checkpoint that still hallucinates reliably on rare subtypes. Don’t pick your final checkpoint on loss alone. Oversample rare branches. Subtypes that show up rarely in your real reports get under represented in a naive split; upweight them, or the model will nail the common cases and quietly fail on the edge cases, which is exactly where a human reviewer is least likely to catch it. Watch for synthetic-input leakage. Because the synthetic notes were LLM generated, it’s easy for them to accidentally echo the template’s own field header phrasing, which lets the model shortcut on synthetic data and then stumble on real, messier clinician notes. Stripping header like phrasing from the generation prompts mattered more than we expected. Keep a human reviewed slice, permanently. Even at 98% strict accuracy, we kept a rolling human spot-check given the stakes. A custom eval harness catches structural errors, not everything a domain expert would catch! Wrapping up Fine-tuning is for behavior that needs to be exact, repeatable, and cheap at scale. Our task happened to be almost entirely this: a fixed, deeply branching output format where being 90% right on any given field is functionally the same as being wrong. We kept RAG and system prompts around for the parts of the pipeline that actually need up to date knowledge. We just stopped asking a 30k token system prompt to do a fine-tune’s job. I hope this helps you understand not only the mathematical intution behind LoRA/QLoRA, but how to implement it yourself. Enjoy! References [1] Aghajanyan, A., Gupta, S., & Zettlemoyer, L. (2020). Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. arXiv preprint arXiv:2012.13255. (Referenced regarding the “lower intrinsic dimension” of LLM parameter updates). [2] Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. arXiv preprint arXiv:2305.14314. (Referenced for 4-bit NormalFloat (NF4), Double Quantization, and Paged Optimizers). [3] Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv preprint arXiv:2106.09685. (Referenced for the core mathematical intuition of learning low-rank matrices for efficient adaptation). [4] Li, Xian, et al. (2024). Self-Alignment with Instruction Backtranslation. International Conference on Learning Representations (ICLR). [5] Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021). Finetuned Language Models are Zero-Shot Learners. arXiv preprint arXiv:2109.01652. (Referenced for the paradigm shift from next-token prediction to instruction tuning and formatting). [6] Schulman, J. / Thinking Machines Lab. LoRA without Regret. (Referenced for hyperparameter intuition, specifically the necessity of applying LoRA adapters to all dense layers rather than just attention matrices). [7] Unsloth. LoRA Hyperparameters Guide. (Referenced for baseline empirical defaults for rank, alpha, learning rate, and weight decay). [8] BitsAndBytes Foundation. bitsandbytes. GitHub Repository. https://github.com/bitsandbytes-foundation/bitsandbytes (Underlying library used for NF4 quantization and paged memory management). [9] Hugging Face. Transformers, PEFT (Parameter-Efficient Fine-Tuning), TRL (Transformer Reinforcement Learning), and Datasets. (Core ecosystem utilized in the training script pipeline). [10] Mistral AI. Mistral-7B-Instruct-v0.3. (The foundational base model utilized for the fine-tuning implementation script).