The Sigmoid Function: From 'e' to Neural Networks

Welcome back!We recently discussed backpropagation, and I hope you now have an idea of what backpropagation is and how it actually works.Let's continue the deep learning journey.Even though we apply the backpropagation algorithm to a neural network, we still have some problems, and vanishing gradients is one of them.While I was learning about vanishing gradients, I came across the sigmoid function.We all know that it is used in logistic regression, where we apply the sigmoid function to a value to obtain an output between 0 and 1.Now, here in neural networks, it can be used as an activation function.What I know about sigmoid is the equation we have and its usage in logistic regression and neural networks.I was curious about how we get this equation and the story behind it.In this blog, let's see how we get to the sigmoid equation.By the way, if you haven't read Part 3 of the backpropagation series, you can read it here.···How Do We Actually Use Sigmoid?We already know the equation of the sigmoid function.σ(x)=11+e−x\sigma(x) = \frac{1}{1 + e^{-x}}Before we proceed, let's see how we use it in logistic regression.For example, we want to predict whether a student will pass or fail based on the number of hours they studied.We are using the logistic regression model here.First, it calculates a scorez=wx+bz = wx + bLet's say the score for a student is:z=2z = 2This score is not a probability. It is just the linear combination of parameters.Now we pass it through the sigmoid function:σ(z)=11+e−z\sigma(z) = \frac{1}{1 + e^{-z}}we get,σ(2)=11+e−2≈0.88\sigma(2) = \frac{1}{1 + e^{-2}} \approx 0.88The sigmoid function always produces an output between 0 and 1.Here the output is approximately 0.88 or 88%.In logistic regression, this can be interpreted as an 88% probability of the student passing the exam.We can then use a threshold, such as 0.5, to make the final classification.In short, the flow can be likeScore→Sigmoid→Probability→Class\text{Score} \rightarrow \text{Sigmoid} \rightarrow \text{Probability} \rightarrow \text{Class}That's how we commonly use the sigmoid function in logistic regression.But What Is This “e”?Now, let's once again look at the sigmoid equation.σ(z)=11+e−z\sigma(z) = \frac{1}{1 + e^{-z}}The first thing we notice is the e.We know that it is a mathematical constant and its value ise≈2.71828e \approx 2.71828But what exactly is 'e'?Why is this number present in the sigmoid equation?Let's take a step back and understand where this number comes from.One thing is that here we are not trying to discover 'e', but the goal is to understand the significance of 'e' and see where it naturally appears.Now let's go to the bank and see what we can observe.Let's Start with a Simple Bank ExampleImagine we deposited Rs.100 into a bank account.Let's say the bank is giving us a 100% annual interest rate.If the bank adds the entire year's interest at the end of the year, we earn Rs.100 in interest.So after one year, we have100+100=200100 + 100 = 200We can also write it as100(1+1)=200100(1 + 1) = 200Rs.100 became Rs.200 after one year.But now let's change one thing.What if the bank doesn't wait until the end of the year to add the interest?What if it adds the interest twice a year?The annual interest rate is still 100%.But now the year is divided into two periods.So for each six-month period we get half of the annual interest rate:12=0.5=50%\frac{1}{2} = 0.5 = 50\%During the first six months, we get100(1+12)=150100\left(1 + \frac{1}{2}\right) = 150After six months, we have Rs.150.During the next six months, the interest is calculated on this new amount150(1+12)=225150\left(1 + \frac{1}{2}\right) = 225Then we have100(1+12)2=225100\left(1 + \frac{1}{2}\right)^2 = 225Why did we get Rs.225 instead of Rs.200?Because the interest earned during the first six months also earned interest during the second six months.In simple terms we can say'interest earns interest'This is the basic idea behind compound interest.What Happens When We Compound More Frequently?Now let's make the compounding more frequent.If we compound four times a year:100(1+14)4≈244.14100\left(1 + \frac{1}{4}\right)^4 \approx 244.14If we compound 12 times a year:100(1+112)12≈261.30100\left(1 + \frac{1}{12}\right)^{12} \approx 261.30If we compound every day:100(1+1365)365≈271.46100\left(1 + \frac{1}{365}\right)^{365} \approx 271.46Observe the pattern.As we increase the number of compounding periods, the final amount keeps increasing.The reason is that growth is being applied repeatedly to an amount that has already increased.Where Does e Come From?The Rs.100 is not the important part here.Let's remove it and look at the growth factor:(1+1n)n\left(1 + \frac{1}{n}\right)^nHere, 'n' represents the number of times we compound during the year.For example:(1+11)1=2\left(1 + \frac{1}{1}\right)^1 = 2(1+12)2=2.25\left(1 + \frac{1}{2}\right)^2 = 2.25(1+14)4≈2.4414\left(1 + \frac{1}{4}\right)^4 \approx 2.4414(1+112)12≈2.613\left(1 + \frac{1}{12}\right)^{12} \approx 2.613(1+1365)365≈2.7146\left(1 + \frac{1}{365}\right)^{365} \approx 2.7146As we make the compounding more and more frequent, the value gets closer and closer to2.71828…2.71828\ldotsThis number is called 'e'e≈2.71828e \approx 2.71828Mathematically, we can express this idea using a limite=lim⁡n→∞(1+1n)ne = \lim_{n \rightarrow \infty} \left(1 + \frac{1}{n}\right)^nThe notation may look complex, but the idea is simple.Here, we are asking:"What value does this expression approach as 'n' becomes larger and larger?"As 'n' increases:(1+1n)n\left(1 + \frac{1}{n}\right)^ngets closer and closer to:2.71828…2.71828\ldotsThat limiting value is 'e'.So, What Does the Bank Have to Do with Sigmoid?But why are we talking about this and what does this bank account have to do with sigmoid.This example isn't to explain compound interest, but it gives us an intuition for where 'e' naturally appears.The important idea here is repeated growth.When growth is repeatedly applied to an amount that has already grown, we get a compounding process.And when that process happens again and again more frequently, the number 'e' naturally appears.So instead of simply memorizing thate≈2.71828e \approx 2.71828we now have some intuition behind it.The Special Property of eFrom the bank example, we observed that 'e' naturally appears when we look at repeated growth and continuous compounding.But 'e' is more than just a number that appears in compound interest.It has a very special property when we look at it through calculus.Let's consider the exponential functiony=exy=e^xIf we differentiate this function, we getdydx=ex\frac{dy}{dx}=e^xThis formula we already know.But what does the derivative tell us?We already know that it tells us the rate of change of a function.For example, if we havey=x2y=x^2its derivative isdydx=2x\frac{dy}{dx}=2xThis means that the rate at which x2 changes depends on the value of x.At x=1:dydx=2(1)=2\frac{dy}{dx}=2(1)=2At x=3:dydx=2(3)=6\frac{dy}{dx}=2(3)=6So, for x2, the function and its rate of change are different.Now let's look at ex.Fory=exy=e^xwe havedydx=ex\frac{dy}{dx}=e^xThis means that the rate of change of ex is equal to its current value.Let's look at some values.When x=0e0=1e^0=1anddydx=1\frac{dy}{dx}=1When x=1e1≈2.718e^1\approx2.718anddydx≈2.718\frac{dy}{dx}\approx2.718When x=2e2≈7.389e^2\approx7.389anddydx≈7.389\frac{dy}{dx}\approx7.389So, here we can say thatRate of change = Current valueThis is one of the most important properties of the exponential function with base e.Why Is the Derivative of ex Equal to ex?We now have an idea of an important property of 'e' in calculus.ddxex=ex\frac{d}{dx}e^x=e^xWe just discussed what it is but let's see why does this happen?If you already know whyddxex=ex\frac{d}{dx}e^x=e^xthen use this section for quick revision as we connect it back to the sigmoid function.Starting with a General ExponentialFirst let's consider a general exponential function.y=zxy=z^xHere, z is the base and x is the exponent.2x,3x,5x,10x2^x,\qquad 3^x,\qquad 5^x,\qquad 10^xare all examples of this form.Now let's see what happens when we differentiate zxWe have,dydx=lim⁡h→0zx+h−zxh\frac{dy}{dx} = \lim_{h\to0} \frac{z^{x+h}-z^x}{h}Using the exponent rule we getzx+h=zxzhz^{x+h}=z^xz^hThereforedydx=lim⁡h→0zxzh−zxh\frac{dy}{dx} = \lim_{h\to0} \frac{z^xz^h-z^x}{h}Now notice that zx appears in both terms in the numerator.We can factor it outdydx=lim⁡h→0zxzh−1h\frac{dy}{dx} = \lim_{h\to0} z^x\frac{z^h-1}{h}Here zx does not depend on h, so we can take it outside the limitdydx=zxlim⁡h→0zh−1h\frac{dy}{dx} = z^x \lim_{h\to0} \frac{z^h-1}{h}And this is where things get interesting.Our result isdydx=zxlim⁡h→0zh−1h\frac{dy}{dx} = z^x \lim_{h\to0} \frac{z^h-1}{h}Look at the two parts separately.The first part iszxz^xThat is our original exponential function.The second part islim⁡h→0zh−1h\lim_{h\to0} \frac{z^h-1}{h}We can see that there is no 'x' in this expression.It depends on the base 'z', but not on 'x'.This means, for any value of 'z', this entire limit is just a constant.Let's call this constant 'C'.C=lim⁡h→0zh−1hC= \lim_{h\to0} \frac{z^h-1}{h}Therefore we can write it as,ddxzx=Czx\frac{d}{dx}z^x=Cz^xThis tells us something important.When we differentiate an exponential function, we get the original exponential function, multiplied by a constant.In other way,Derivative of zx=constant×zx\text{Derivative of }z^x = \text{constant}\times z^xThe Constant Depends on the BaseNow let's take an example of exponential function:y=3xy=3^xFrom our result, we haveddx3x=C3x\frac{d}{dx}3^x=C3^xFor z=3, the constant isC=lim⁡h→03h−1hC= \lim_{h\to0} \frac{3^h-1}{h}Now we need to find the value of this limit.Let's understand this in intuitive way.h=0.1h=0.1For the base 3, the value of the constant is approximatelyC≈1.0986C\approx1.0986Therefore,ddx3x≈1.0986(3x)\frac{d}{dx}3^x \approx 1.0986(3^x)Let's see what this tells us by using at different 'x' values.Whenx=0x=0we have30=13^0=1the rate of change here is approximately1.0986(1)=1.09861.0986(1)=1.0986Whenx=1x=1we get31=33^1=3The rate of change is1.0986(3)≈3.29581.0986(3)\approx3.2958And whenx=2x=2we have32=93^2=9The rate of change is approximately1.0986(9)≈9.88741.0986(9)\approx9.8874We can see that the derivative is not exactly equal to 3x.Instead, we gotddx3x≈1.0986(3x)\frac{d}{dx}3^x \approx 1.0986(3^x)The function and its rate of change have the same exponential shape, but the rate of change is scaled by a constant.Finding the Special BaseNow, we know thatddxzx=Czx\frac{d}{dx}z^x=Cz^xThe value of 'C' depended on the base.For 3x,C≈1.0986C\approx1.0986Ok but what if we could find a base for which C is exactly 1?Do we have any number? If yes, then we getC=1C=1Our derivative would becomeddxzx=zx\frac{d}{dx}z^x=z^xIn other words, we can say that the function would be exactly equal to its own derivative.So, now we are looking for a base z that satisfieslim⁡h→0zh−1h=1\lim_{h\to0} \frac{z^h-1}{h}=1There is one particular positive number that satisfies this condition and you all know what's that number is.We call this numbereeand its numerical value ise≈2.71828e\approx2.71828For this particular base, the constant becomesC=1C=1Therefore,ddxex=1⋅ex\frac{d}{dx}e^x = 1\cdot e^xwhich gives usddxex=ex\frac{d}{dx}e^x=e^x···So What Did We Actually Discover?We started with a general exponential functionzxz^xUsing the definition of a derivative, we foundddxzx=zxlim⁡h→0zh−1h\frac{d}{dx}z^x = z^x \lim_{h\to0} \frac{z^h-1}{h}We then observed that the limit is simply a constant that depends on the base.Then we have written it asddxzx=Czx\frac{d}{dx}z^x=Cz^xThen we asked:Is there a base for which C=1?The answer is yes.That special base is e.Therefore,ddxex=ex\frac{d}{dx}e^x=e^xNow we have an idea of how we got the derivative. In the earlier bank example, 'e' appeared through repeated growth and continuous compounding.Now, through calculus, we have seen another special property of the same numberddxex=ex\frac{d}{dx}e^x=e^xIn simple words, we can say that ex grows at a rate equal to its current value.Now, Let's Return to SigmoidLet's once again look at the sigmoid equation.σ(x)=11+e−x\sigma(x)=\frac{1}{1+e^{-x}}Now we have some idea of what 'e' actually is.Now we focus on the whole equation.The question here is why does the sigmoid function is in this particular form?To understand this we should go back to logistic regression.We started with a raw scorez=wx+bz=wx+b 'z' can be any real number.But for classification, we wanted to interpret the model's output as a probability.A probability must lie between 0 and 10<p<1So we want to transform any value of 'z' into a value between 0 and 1.In other words, we want something that can receivez∈(−∞,∞)z\in(-\infty,\infty)and produce:p∈(0,1)p\in(0,1)Building a Function That Outputs Between 0 and 1Now, the task is to construct such transformation.But how can we do that?Let's start with a very simple observation.Suppose we have a number greater than 1.For example55If we take its reciprocal, we get15=0.2\frac{1}{5}=0.2which is between 0 and 1.The same idea works for any numbers greater than 112=0.5\frac{1}{2}=0.5110=0.1\frac{1}{10}=0.11100=0.01\frac{1}{100}=0.01Here we can notice thatIfA>1A > 1then00e^x > 0for every real value of x.For example:e−2≈0.1353e^{-2}\approx0.1353e0=1e^0=1e2≈7.389e^2\approx7.389Whether the x is negative, zero, or positive, ex never becomes negative or zero.But the sigmoid equation contains e-x. Till here we only discussed about ex.So let's first see what a negative exponent means.We already know what a positive exponent means.For example:e2=e×ee^2=e\times eand:e3=e×e×ee^3=e\times e\times eA negative exponent represents the reciprocal of the corresponding positive exponent.For example:e−1=1ee^{-1}=\frac{1}{e}Similarlye−2=1e2e^{-2}=\frac{1}{e^2}ande−3=1e3e^{-3}=\frac{1}{e^3}In general, we can write ase−x=1exe^{-x}=\frac{1}{e^x}So, e-x is not a completely different function.It is simply the reciprocal of ex.Now we can use what we already know about ex.Since:ex>0e^x > 0its reciprocal is also positive1ex>0\frac{1}{e^x}>0and because e−x=1exe^{-x}=\frac{1}{e^x}we gete−x>0e^{-x}>0for every real value of x.This is important because it gives us exactly the kind of quantity we need.If e-x is always positive, then adding 1 gives us a quantity that is always greater than 11+e−x>11+e^{-x}>1And now we can use our reciprocal idea.If a number is greater than 1, its reciprocal lies between 0 and 1011+e^{-x}>1Taking its reciprocal gave usσ(x)=11+e−x\sigma(x)=\frac{1}{1+e^{-x}}This gave us the range we wanted.But does this equation actually behave the way we expected it to do?Here, our goal is to understand the intuition behind the structure of the sigmoid function. There are other functions that can map values to the range 0 to 1, and why logistic regression uses sigmoid is related to odds and log-odds, a topic which we will explore in future blogs.Does the Sigmoid Behave the Way We Expected?Let's test on few values.First, let's considerx=0x=0Substituting into the sigmoid function:σ(0)=11+e−0\sigma(0)=\frac{1}{1+e^{-0}}ase0=1e^0=1we getσ(0)=11+1=0.5\sigma(0)=\frac{1}{1+1}=0.5When the input is 0, the sigmoid gives us exactly 0.5.Now let's take a positive numberx=2x=2thenσ(2)=11+e−2\sigma(2)=\frac{1}{1+e^{-2}}We already seene−2≈0.1353e^{-2}\approx0.1353which gives σ(2)=11+0.1353=11.1353≈0.881\sigma(2) = \frac{1}{1+0.1353} = \frac{1}{1.1353} \approx 0.881The sigmoid converted the input 2 into approximately 0.881 or 88.1%.Now let's see what happens when the input is a negative number.Considerx=−2x=-2Thenσ(−2)=11+e−(−2)\sigma(-2) = \frac{1}{1+e^{-(-2)}}σ(−2)=11+e2\sigma(-2) = \frac{1}{1+e^2}We knowe2≈7.389e^2\approx7.389Finally we getσ(−2)=11+7.389=18.389≈0.119\begin{aligned} \sigma(-2) &=\frac{1}{1+7.389}\\ &=\frac{1}{8.389}\\ &\approx0.119 \end{aligned}So the sigmoid converted the input -2 into approximately 0.119 or 11.9%.Now we can see how the sigmoid behaves.For a negative input:x=−2⟶σ(x)≈0.119x=-2 \quad\longrightarrow\quad \sigma(x)\approx0.119For zero:x=0⟶σ(x)=0.5x=0 \quad\longrightarrow\quad \sigma(x)=0.5For a positive input:x=2⟶σ(x)≈0.881x=2 \quad\longrightarrow\quad \sigma(x)\approx0.881So as x increases, the sigmoid output moves from values close to 0, passes through 0.5 and moves toward 1.In the extreme cases:x→−∞⟹σ(x)→0x\rightarrow-\infty \quad\Longrightarrow\quad \sigma(x)\rightarrow0andx→+∞⟹σ(x)→1x\rightarrow+\infty \quad\Longrightarrow\quad \sigma(x)\rightarrow1This is exactly the behavior we wanted from a function that transforms any real number into something between 0 and 1.Image by AuthorNow we have an idea of how we got the equation of the sigmoid function.If you remember, in my recent blogs, when we discussed backpropagation and neural networks in general, we talked about activation functions and why they are important.We used the ReLU activation function to understand those concepts.Now, we can also use sigmoid as an activation function.But if we use sigmoid as an activation function, there is one more thing we need to know.During the backward pass, we already know that the network calculates gradients using derivatives. So, if sigmoid is part of the network, we need to differentiate it as well.Now let's focus only on deriving the derivative of the sigmoid function step by step.σ(x)=11+e−x\sigma(x)=\frac{1}{1+e^{-x}}Instead of carrying the exponential term throughout calculations, we can simply use the sigmoid output itself.This is the derivative we use whenever sigmoid appears in the gradient calculations of a neural network.···SummaryIn the upcoming blogs, we are going to discuss topics like vanishing gradients and exploding gradients.As we explore these topics, we will come across the sigmoid function, and we will also need its derivative.If we derive the sigmoid function and its derivative in those blogs, the discussion could become long, and we may lose focus on the actual concept that we are trying to understand. It would also be better to have an idea of where the sigmoid function and its derivative come from before using them in further concepts.We first started with the bank example to see how e appears. We then learned about its important property in calculus and, using these ideas, gradually built the sigmoid equation.We saw how this equation is used in logistic regression and neural networks, and we also derived its derivative.Now, when we move on to the upcoming topics, we already have this foundation which will be useful for us. I hope you found this blog helpful in understanding a concept that we frequently use.If you have any questions or suggestions for improvement, feel free to share them in the comments on LinkedIn.And if you haven't read my latest blog series on backpropagation yet, you can read it here.Sometimes, moving forward means going back and understanding the basics.Thanks for reading!···