From API to GPU, Week 6 (Part 1): A Model That Predicts, and How Wrong It Is

Phase 2 of 8: Enough ML to understand inference. Week 6 of 32, part 1 of 2. Every week so far, the model already existed. I ran Phi-4, read Qwen's files, and measured tensors, but I never made a model learn anything. This week I build one from scratch and train it. That is a lot for one sitting, so I split the week into two posts and go slowly. The goal: the model starts with a random weight and bias, which are just wrong guesses. Training is what adjusts them, step by step, until they land on the values that actually convert Celsius to Fahrenheit: a weight of 1.8 and a bias of 32. Everything in these two posts is in service of that one idea. This first post builds the model and answers two questions, nothing more: how does the model make a prediction, and how do we measure how wrong that prediction is? No learning yet. Part 2 takes the error measured here and turns it into learning. The model is a one-neuron linear model: one output value computed from one weight and one bias. It is deliberately tiny, so nothing hides the machinery. It runs on the CPU through ssh spark, because the problem is far too small to need a GPU. What "learning" means here If you have never trained a model before, "learning" can sound like magic. It is not. It is trial and error run in a tight loop, the same way you would reverse-engineer an unknown function from its input and output logs. Picture a black box that turns numbers in into numbers out. You cannot see inside it, but you have a few examples of what it did: it turned -40 into -40, 0 into 32, and 100 into 212. You want a formula that reproduces those outputs. So you guess one, check how far each guess is from the real answer, nudge the guess in the direction that shrinks the error, and repeat until the answers line up. That loop is all that "training" is. A neural network does exactly this, with two differences. First, the "formula" is a set of adjustable numbers called parameters (the weights and biases from Week 4), and learning means finding good values for them. Second, the nudging is automatic: the network measures its own error and works out which way to move each parameter, so nobody tweaks anything by hand. That loop has four moves: make a guess, measure how wrong it is, work out which way to adjust, and take a small step, then repeat. This post covers only the first two, making a guess (the forward pass) and measuring how wrong it is (the loss). Part 2 covers the adjusting. Splitting it this way keeps each idea tied to a real number you can see, instead of a pile of new words up front. The black box in this post is Celsius to Fahrenheit. The true rule is F = C * 1.8 + 32. The model never sees that rule. I only use it to build the correct answers, then show the model input-output examples and let it discover on its own that the weight should be 1.8 and the bias should be 32. The model: one neuron Week 4 said a weight multiplies an input and a bias is added after. This model is exactly that, one weight and one bias: prediction=weight×Celsius+bias\text{prediction} = \text{weight} \times \text{Celsius} + \text{bias} That single weight-and-bias unit is a neuron, the smallest building block of a neural network. A neural network is just many neurons wired together, so the output of some becomes the input of others. Real networks stack thousands or millions of them; this week's has exactly one, which is why nothing hides the mechanics. In PyTorch that is a single linear layer, nn.Linear(1, 1): one input feature (the one input value) and one output value per example. A layer is one processing step, and a linear transformation is the multiply-and-add above. (Because there is a nonzero bias, a mathematician would call this an affine transformation, but PyTorch names the layer Linear, so I use that word.) If the weight lands on 1.8 and the bias on 32, the layer computes Fahrenheit exactly. Why is this called linear? Picture plotting the model's output against its input on a graph. A weight-times-input-plus-bias rule always traces a straight line. The weight sets the line's slope: how steeply the output climbs as the input grows. The bias slides the whole line up or down. Celsius to Fahrenheit is a straight line with slope 1.8, because every extra degree Celsius adds exactly 1.8 degrees Fahrenheit, the same amount at every temperature. That fixed, never-changing slope is what "linear" and "straight line" mean here, and a single linear layer can match it perfectly. This model has no activation function, the extra piece that would let it bend a straight line into a curve. Because Celsius to Fahrenheit is already straight, it does not need one. What an activation function is, and what a curve looks like, I unpack at the end, in Wait, why is there no activation function?, so the main thread stays on the model. Making a prediction: the forward pass A forward pass runs the inputs through the layer to get predictions. In PyTorch it is the single expression model(celsius). Before any training the weight and bias are random, so the predictions will be nonsense, which is exactly what I want to see first. The six Celsius inputs are paired with their correct Fahrenheit answers. Each correct answer is called a label or target, and the line fahrenheit = celsius * 1.8 + 32.0 builds all six at once. There is no loop here. celsius is a tensor, the small array type from Week 5, and multiplying a tensor by a single number multiplies every element by that number; adding 32 then adds it to every element. That is an element-wise operation, and it is why the result is another tensor of six answers, not one number. Note that celsius is the input data, not the model. The model is the nn.Linear layer built a few lines down. Each value in celsius is written with a trailing dot, so -40. means the floating-point number -40.0, not the integer -40. The dot keeps the tensor in decimals, which is what the model's math needs. And each value sits in its own brackets, like [-40.], so stacking the six of them makes a tensor shaped 6 rows by 1 column: six examples, one value each. To be clear, those six numbers are six separate Celsius inputs (-40, -10, 0, 20, 37, and 100 degrees). [-40.] and [-10.] are two different examples, not an input-and-answer pair. Each input's Fahrenheit answer lives at the matching position in the separate fahrenheit tensor, and the run below prints them side by side. The value -40 shows up as both the first input and its own answer only because -40 C equals -40 F, the one temperature where the Celsius and Fahrenheit scales meet, which is a coincidence of the scales, not the data layout. The command below builds the model, prints the device it runs on and its random starting weight and bias, then does one forward pass and lays each prediction next to its target: ssh spark '~/venvs/w1/bin/python - 6.1f} pred={p[0]:>8.3f} target={f[0]:>7.1f}") PY' Enter fullscreen mode Exit fullscreen mode device=cpu initial weight=-0.007487 bias=0.536444 initial predictions vs targets: C= -40.0 pred= 0.836 target= -40.0 C= -10.0 pred= 0.611 target= 14.0 C= 0.0 pred= 0.536 target= 32.0 C= 20.0 pred= 0.387 target= 68.0 C= 37.0 pred= 0.259 target= 98.6 C= 100.0 pred= -0.212 target= 212.0 Enter fullscreen mode Exit fullscreen mode Two things to notice. device=cpu confirms this runs on the CPU, the .device attribute from Week 5, and the problem is far too small to need a GPU. And the random start makes every prediction wrong: the weight is about -0.007 and the bias about 0.54, straight from torch.manual_seed(0) (a fixed seed, so you get these same numbers), so at 100 C the model guesses -0.212 instead of 212. The model has no idea what Fahrenheit is yet. So why are the predictions wrong when the correct answers are sitting right there in fahrenheit? Because the model never looks at those answers when it predicts. A prediction is only weight * celsius + bias, computed from the model's own weight and bias, which right now are the random -0.007 and 0.54. The fahrenheit values are a separate answer key: they are used to score the predictions, and in Part 2 to teach the model, but the model does not see them while it predicts. It is like a student taking a test with the answer key face down. Until they learn the rule, the answers are guesses, no matter that the key is in the room. The answers are not inside the input tensor at all. celsius and fahrenheit are two separate tensors, two different variables. The call model(celsius) passes only celsius, so fahrenheit never enters the model. Inside, the linear layer computes exactly weight * celsius + bias from its own two numbers. The one and only place fahrenheit is handed to anything is the loss, loss_fn(preds, fahrenheit), in the next section. You can prove the model uses only its weight, bias, and the input by recomputing its prediction by hand and comparing: ssh spark '~/venvs/w1/bin/python - 10.3f} squared={s:>12.3f}") # Step 3: the loss is the average of those six squared errors. by_hand = squared.mean().item() builtin = nn.MSELoss()(preds, fahrenheit).item() print(f"average of squared errors (by hand) = {by_hand:.4f}") print(f"nn.MSELoss() gives the same number = {builtin:.4f}") PY' Enter fullscreen mode Exit fullscreen mode per-example error and squared error: error= 40.836 squared= 1667.572 error= -13.389 squared= 179.257 error= -31.464 squared= 989.955 error= -67.613 squared= 4571.558 error= -98.341 squared= 9670.867 error= -212.212 squared= 45034.031 average of squared errors (by hand) = 10352.2070 nn.MSELoss() gives the same number = 10352.2070 Enter fullscreen mode Exit fullscreen mode Read the output top to bottom and the loss stops being a mystery number. Each row is one example's error and that error squared. The 100 C example dominates: its error of -212 squares to 45034, more than all the other five combined, which is exactly the effect of punishing big misses. Add the six squared errors and divide by six and you get 10352.2070, and PyTorch's nn.MSELoss returns the identical value. So the built-in loss really is just the average of the squared errors, nothing more. One consequence of squaring: the loss is in squared Fahrenheit units, not degrees. A loss of 10352.21 does not mean the model is off by 10352 F. The number is not a temperature to read; it is a score to drive down. Right now it is huge, which fits, because the predictions are still random. In Part 2, every training step pushes this number lower. This is the whole job of Part 1. The model can make a prediction (the forward pass), and we can score how wrong it is with one number (the loss). Part 2 uses that one number to improve the model. Results so far Item Verified value Task Celsius to Fahrenheit, F = C * 1.8 + 32 Model one linear layer, nn.Linear(1, 1), no activation Device CPU Initial weight / bias -0.007487 / 0.536444 Worst initial prediction 100 C predicts -0.212 (target 212) First loss (MSE) 10352.21 (squared Fahrenheit) What Part 2 covers Part 2 takes the loss from here and turns it into learning. It introduces the gradient (which direction to move each parameter), backpropagation (how PyTorch computes those directions), the optimizer and learning rate (how big a step to take), and epoch and batch (how the loop repeats). Then it runs the full training loop and watches the weight and bias climb from random noise to 1.8 and 32. Wait, why is there no activation function? Most neural networks include an activation function: a step that reshapes a layer's output so the network can learn relationships that are not straight lines. This model leaves it out, and seeing why is a good way to learn what one does. Many relationships are not straight. When the output climbs quickly in one place and slowly in another, plotting it draws a curve: a line whose slope keeps changing instead of staying fixed. A bend is just a point where the slope changes. A few everyday examples make it concrete: A savings account with compound interest. The balance grows slowly at first, then faster and faster, because the interest itself earns interest. Over time the line curves upward. A phone battery charging. It races from 0 to 80 percent, then crawls through the last stretch to 100. The rate keeps changing, so the line bends. A web server's response time as load rises. Lightly loaded, a few more requests barely change it. Near full capacity, each extra request adds much more delay and the response time shoots up. Nearly flat, then steep. Straight-line relationships, by contrast, hold one constant rate: total cost is the price per item times the number of items, distance is speed times time at a steady speed, and Celsius to Fahrenheit adds the same 1.8 for every degree. A single linear layer matches those exactly. The moment the rate has to change, a straight line cannot follow it, and that is the job an activation function does. The most common activation function is ReLU. The rule is simple: replace every negative value with zero, and leave positive values unchanged. Here it is applied to five sample numbers: ssh spark '~/venvs/w1/bin/python -c " import torch; from torch import nn # Five sample numbers, two negative, a zero, and two positive. x=torch.tensor([-2.0, -0.5, 0.0, 1.5, 3.0]) # ReLU replaces every negative with 0 and leaves the rest unchanged. print(nn.ReLU()(x))"' Enter fullscreen mode Exit fullscreen mode tensor([0.0000, 0.0000, 0.0000, 1.5000, 3.0000]) Enter fullscreen mode Exit fullscreen mode The two negative inputs came out as zero, the zero stayed zero, and the two positive ones passed through unchanged. That bend at zero is the ingredient that, stacked across many neurons and layers, lets a network trace curves like the ones above. Celsius to Fahrenheit is a straight line, though, so this model needs no activation function. Adding a ReLU here would actually break it: its Fahrenheit answers go negative (-40 C is -40 F), and ReLU would clamp those to zero. When a task does need curves, an activation function is what provides them, and a later week puts one to work. So what does a curve mean for a model in practice? It sets a ceiling on what the model can learn. A model built only from linear layers can draw straight lines and nothing else, no matter how its weights are set or how many layers you stack, because stacking straight-line layers just produces another straight line. If the real relationship bends, that model can never fit it well. An activation function is what lifts the ceiling: the bends let the network shape its output to follow curved data. That gives a simple rule for when to use one. If the relationship you are modeling really is a straight line, like this Celsius-to-Fahrenheit toy or a plain linear regression, you do not need an activation function. For almost everything else, recognizing an image, predicting a price from many features, understanding text, the relationship bends, so real networks add an activation function after every hidden layer by default. This model is the rare exception, which is exactly why it is a clean place to see what an activation function does by watching a model that works fine without one. Run it yourself The public Week 6 lab has the training script used in Part 2, the captured runs, observations, and troubleshooting notes.1 Week 6 companion lab: https://github.com/dramasamy/from-api-to-gpu/tree/main/week-06-neural-network-basics ↩