{"id":1103,"date":"2017-12-09T17:45:10","date_gmt":"2017-12-09T17:45:10","guid":{"rendered":"http:\/\/www.philippeadjiman.com\/blog\/?p=1103"},"modified":"2025-11-11T06:32:22","modified_gmt":"2025-11-11T06:32:22","slug":"deep-dive-into-logistic-regression-part-1","status":"publish","type":"post","link":"https:\/\/philippeadjiman.com\/blog\/2017\/12\/09\/deep-dive-into-logistic-regression-part-1\/","title":{"rendered":"Deep Dive Into Logistic Regression: Part 1"},"content":{"rendered":"

Logistic regression is arguably the most widely used machine learning algorithm in production systems when it comes to classify or predict the likelihood of some events to happen, often\u00a0 in the context of modelling online users behaviour like e.g. the likelihood of a user clicking (a.k.a CTR estimation) or buying something (well, factorization machines<\/a> are getting some serious momentum as well, to be discussed in some future posts). There is a reason for that: logistic regression is incredibly powerful, scalable, simple to implement and blazing fast to apply online once the model was trained offline.<\/p>\n

In this post, we’ll deep dive into the theory behind logistic regression, giving the intuition behind its core concepts and its multiple faces across various fields of statistics and computer science. This will involve some maths, but nothing too deep assuming you have some notions of calculus and core statistics.<\/p>\n

In the second part<\/a> of this series, we’ll be much more concrete and deep dive into the implementation details of logistic regression, and go over some tricks like the hashing trick and the per coordinate adaptive learning rate which are making logistic regression works very well in practice on real (big) data sets. In that second post we’ll also go over a\u00a0beautifully simple and elegant implementation of online logistic regression including all those tricks. In the third part<\/a> of this series we’ll demonstrate the usage of a very powerful and popular library implementing logistic regression (and more) at scale: Vowpal Wabbit<\/a>.<\/p>\n

For now, let’s start with the theory \ud83d\ude42<\/p>\n

A classical derivation of logistic regression<\/h2>\n

We’ll start by introducing a standalone description of logistic regression, similar to what you can find in any classical introduction to machine learning course (e.g. that one<\/a> to cite the most popular of them all).<\/p>\n

So you have a training set\u00a0of N examples\u00a0 where is a sparse binary feature vector in a d-dimensional space, a.k.a. the signals or features of the th training example (more on that signals representation later, especially in part 2<\/a> of that series), and is the label associated to that example (which could represent a click\/non click, spam\/not spam, malignant\/benign, …):<\/p>\n

$$\\left\\{
\n\\begin{array}{ll}
\nx_1^{(1)},…,x_d^{(1)} & y_1 \\\\
\n… \\\\
\nx_1^{(N)},…,x_d^{(N)} & y_N
\n\\end{array}
\n\\right. $$<\/p>\n

To make a prediction for a given signal vector\u00a0 ,\u00a0 the logistic regression model proposes to take a linear combination\u00a0\u00a0 where is a vector of parameters (weights) , and to project it into the \u00a0range by applying the logistic<\/a> (or sigmoid) function directly to that linear product,\u00a0 giving the following model representation:<\/p>\n

$$h_{\\theta}(x) = logistic( \\theta^Tx ) = \\frac{1}{1+e^{-\\theta^Tx}} $$<\/p>\n

The usual interpretation of \u00a0is that it represents\u00a0the estimated probability that\u00a0\u00a0 on input\u00a0, in other words:\u00a0\u00a0 .\u00a0 Then, if you have to use that number to predict weather or , some threshold is picked , either simply 0.5 (i.e. predicting\u00a0 when\u00a0\u00a0 and 0 otherwise) or any other threshold empirically chosen using the classifier’s ROC curve (c.f. my older\u00a0other post<\/a>\u00a0for more details on that).<\/p>\n

Note that logistic regression is a linear classifier<\/a>\u00a0given that its decision boundary is a linear combination of the input. Indeed, if your threshold is e.g. 0.5, then you have\u00a0 when\u00a0\u00a0 . If you draw the sigmoid function , you can see that it is >= 0.5 when . Thus\u00a0\u00a0 \u00a0when\u00a0 , which is a linear decision boundary.<\/p>\n

Let’s now talk about the cost function which is the most important part when building a model given that it is what need to be minimised on the training data to learn the optimal weight vector. Given the model representation, we cannot take a standard cost function based on MSE<\/a>\u00a0because it would make it non convex. All the power\u00a0 of logistic regression is in its cost function which looks as follow:<\/p>\n

$$
\nCost(h_{\\theta}(x),y) = \\left\\{
\n\\begin{array}{ll}
\n-log(h_{\\theta}(x)) & \\textrm{if} \\quad y =1 \\\\
\n-log(1-h_{\\theta}(x)) & \\textrm{if} \\quad y =0
\n\\end{array}
\n\\right.
\n$$<\/p>\n

The beauty behind that cost function is first that it is very intuitive, because when you predict 0 instead of 1 (or 1 instead of 0), then your cost tends to infinity (and thus you penalize the learning algorithm by a very large cost), but most importantly, this cost function is convex (check here<\/a> for a proof), thus allowing to use any standard gradient descent<\/a> based optimization algorithm .<\/p>\n

Note that this function can be written \u00a0 (just replace \u00a0 by 0 or 1 to be convinced). We’ll denote the average cost on the whole training set , which is defined as:<\/p>\n

$$Cost(\\theta) = -\\frac{1}{N}\\sum\\limits_{i=1}^{N} [y_i\\thinspace log(h_{\\theta}(x^{(i)}))\u00a0\u00a0+(1-y_i)\u00a0log(1-h_{\\theta}(x^{(i)}))] $$<\/p>\n

This is also sometimes called the logarithmic loss. You can define a multi-class version of it (when your output can take more than 2 values) , see e.g. here<\/a>\u00a0or here<\/a> for some intuitive explanations.<\/p>\n

So, bottom line, we need to find the optimal weight vector by solving . To do so, gradient descent is the natural tool. We simply need to compute the partial derivative of\u00a0 according to each weight of\u00a0 ,\u00a0 i.e. . We won’t go into the details of the actual derivative calculation (you can find it e.g. here<\/a> ) but just remember the notations:\u00a0 the training example \u00a0 is\u00a0 a vector , and\u00a0 and thus, for instance, . The result of the calculation of the partial derivative gives:<\/p>\n

$$\u00a0\\frac{\\partial }{\\partial \\theta_j} Cost(\\theta)\u00a0 = \\sum\\limits_{i=1}^{N} (\u00a0h_{\\theta}(x^{(i)}) – y_i )x_j^{(i)}\u00a0 $$<\/p>\n

This concludes all what is needed to solve\u00a0 to find\u00a0 the optimal weight vector from our training data. Indeed,\u00a0 assuming some learning rate , we simply have to iterate enough times over updating all the weights of\u00a0\u00a0 using the gradient step below, until we observe that the cost is not reducing anymore :<\/p>\n

$$ \\theta_j =\u00a0\\theta_j \\thinspace\u00a0 – \\alpha\u00a0 \\sum\\limits_{i=1}^{N} (\u00a0h_{\\theta}(x^{(i)}) – y_i )x_j^{(i)}\u00a0 $$<\/p>\n

How to interpret the learned weights?<\/h2>\n

At the end of your learning procedure via gradient descent as described above, you end with an “optimal”\u00a0 weight vector , with \u00a0 \u00a0the weight associated with the input signal\u00a0 . In a simple linear regression model, the interpretation for that weight would be that if the corresponding signal\u00a0\u00a0 increases by one unit, then the predicted output increases by\u00a0\u00a0 units. In logistic regression it cannot be really interpreted that way given that we’re dealing with the sigmoid function and probabilities.<\/p>\n

To understand how to interpret the learned weights in logistic regression, we first need to define and understand the notion of odds ratio. Let’s say that the probability of some event to happen (e.g. a basketball team winning a game) is\u00a0 . The probability of them loosing is\u00a0 . The odds ratio is simply defined as the ratio between probability of success\u00a0 and probability of failure,\u00a0\u00a0 i.e. 0.8 \/ 0.2 = 4 in our example. The interpretation is that the odds for the basketball team to win are 4 to 1.<\/p>\n

How does that relate to logistic regression? To answer, you just need to know that the inverse function of the logistic function is the logit function<\/a>\u00a0. We thus have:<\/p>\n

When it comes to glass you have to consider what tadalafil 10mg uk<\/a> type you should use. Stay careful not to intake more than one published here<\/a> viagra from canada dosage in 24 hours as this may create very unfavorable critical health conditions. The activity of spermatozoids of men-volunteers was analyzed before and after taking http:\/\/amerikabulteni.com\/2011\/10\/19\/cumhuriyetci-adaylarin-las-vegas-bulusmasi-secim-yarisinda-tansiyonu-yukseltti\/<\/a> buy viagra italy. This amerikabulteni.com<\/a> cialis online canada implies a remedy from the erectile dysfunction as well as those with an enlarged prostate. <\/span>$$ logit(h_{\\theta}(x)) =\u00a0 logit(logistic(\\theta^Tx)) = \\theta^Tx $$<\/p>\n

Let’s remind that\u00a0 represents the probability of the outcome being 1 (given a signal vector\u00a0\u00a0 ). Let’s denote that probability p. We thus have:<\/p>\n

$$ logit(p) = \\theta^Tx $$<\/p>\n

Now the interesting part is that\u00a0 . Noticed ?\u00a0Yep, that’s the odds ratio defined above \ud83d\ude42 . In other words, logistic regression is a model relating the log odds probability of the outcome as a linear combination of the input signals:<\/p>\n

$$log(\\frac{p}{1-p}) = \\theta_0 + \\theta_1x_1 + … +\u00a0\\theta_dx_d\u00a0 $$<\/p>\n

We can now interpret the meaning of a weight : if the signal increases by one unit (or if it is present in case it is a boolean signal), then it increases by\u00a0 the log odds of the outcome. Even more interpretable, if you take the exponent of both sides in the expression above you get:<\/p>\n

$$\\frac{p}{1-p} = e^{ \\theta^T x} =\u00a0\\prod\\limits_{j=0}^{d}e^{ \\theta_j x_j}\u00a0 $$<\/p>\n

which gives a direct relation with the odds and thus an even more simple interpretation of the weight \u00a0 : the value\u00a0 directly gives you the increase in the odds of the outcome if the value\u00a0signal increases by one unit (or if it is present in case it is a boolean signal) . Example: if one of your signal is a boolean “already won NBA finals” for your predicting probability of a basketball team to win, and that it gets a weight of say\u00a0 , the interpretation would be: if the team already won an NBA finals, then it increases its odds of winning by \u00a0 , meaning an increase of 232% (i.e. ) in the odds of winning.<\/p>\n

Bottom line<\/strong>: If a signal ends up with\u00a0 a weight in logistic regression, it means that if the signal increases by one unit (or just if it is equal to 1 in case of boolean signal), then it increases the odds of the outcome to be 1 (e.g. a click happening) by\u00a0%.<\/p>\n

Log Loss vs. Cross Entropy vs. Negative Log Likelihood??<\/h2>\n

The concept behind logistic regression is so remarkable and efficient that it arose from\u00a0 various different fields, including different branches of computer science and statistics, and often, you stumble upon different ways of deriving it, including various different names for the cost function or what needs to be maximised or minimised etc.., which might make the whole thing quiet confusing. For instance, in NLP, logistic regression (more precisely the multi-class version of it) is often called Maximum Entropy (or MaxEnt), first defined in that paper\u00a0<\/a>.\u00a0 In this section, i’ll just recall the probabilistic view of logistic regression and connect the dots between cross-entropy, MLE, negative log likelihood, and logLoss .<\/p>\n

First, entropy is a powerful concept invented by Claude Shanon<\/a>\u00a0who basically set the ground for information theory (if you want to get the gist of it from scratch, check this very nice vulgarization video<\/a>). Cross-entropy<\/a>\u00a0is often used as a way to measure the difference between two probability vectors in the context of multinomial classification (a generalisation of the binary classification problem we’re interested in ), c.f. e.g that short\u00a0video<\/a> .\u00a0 The “binary” version of cross entropy (i.e. its particular case when you have only two output classes like in our setting) is defined over the two vectors and where is the observed true value and\u00a0\u00a0 is the prediction:<\/p>\n

$$\u00a0 H(p,q) =\u00a0-\\sum_{i=1} p_i log q_i \\\\ = -ylog(\\hat{y}) – (1-y)log(1-\\hat{y})$$<\/p>\n

This gives you a measure of “disorder” between the two vectors (the true one and the predicted one). In our case, , so the average cross entropy on the whole training set is:<\/p>\n

$$ -\\frac{1}{N}\\sum\\limits_{i=1}^{N} [y_i\\thinspace log(h_{\\theta}(x^{(i)}))\u00a0\u00a0+(1-y_i)\u00a0log(1-h_{\\theta}(x^{(i)}))] \u00a0$$<\/p>\n

Wait, did you notice? This is exactly the log loss cost function we had in the first section!!<\/p>\n

And there is more.<\/p>\n

Let’s move to another very popular concept in machine learning called Maximum Likelihood Estimation (MLE) . MLE is a simple yet very powerful tool to estimate a (set of) parameter(s) based on observed data (if you have never heard about it and need an explanation “for dummies” then you can check this video<\/a>\u00a0for the high level idea and that one<\/a> for a specific example). When you want to use MLE, the first step is to write down the probability of observing the data (in our case the ) given the input signals\u00a0 and the vector of parameters :<\/p>\n

$$ Pr(y_1, …, y_N\u00a0 |x^{(1)}, …, x^{(N)} \u00a0, \\theta) =\u00a0\u00a0\\prod\\limits_{i=1}^{N}Pr(y_i| x^{(i)}, \\theta)\u00a0 $$<\/p>\n

Given that in our case\u00a0 is either 0 or 1, a common trick is to write that:<\/p>\n

$$Pr(y_i| x^{(i)}, \\theta)\u00a0 = \\\\\u00a0 Pr(y_i=1 |x^{(i)}, \\theta)^{y_i}\u00a0\\thinspace Pr(y_i=0 |x^{(i)}, \\theta)^{1-y_i} $$<\/p>\n

The actual likelihood function always inverse the parameters in the notation to make clear that we are looking for an optimal given the fixed observations of the training set:<\/p>\n

$$ L(\\theta ,\u00a0x^{(1)}, …, x^{(N)} |\u00a0y_1, …, y_N\u00a0) = \\\\ \\prod\\limits_{i=1}^{N}\u00a0Pr(y_i=1 |x^{(i)}, \\theta)^{y_i}\u00a0\\thinspace Pr(y_i=0 |x^{(i)}, \\theta)^{1-y_i} \u00a0$$<\/p>\n

 <\/p>\n

Note that the same form could have been obtained without the need for the previous trick by simply noticing that in the case of binary classification, the proper likelihood function is Bernoulli .\u00a0 \u00a0Now, we denote \u00a0 as\u00a0 (exact same notation as in the first section). We’ll also denote the likelihood function for convenience. MLE thus suggest we find the\u00a0\u00a0 maximizing that likelihood function (hence the name maximum likelihood), in other words:<\/p>\n

$$\u00a0\\underset{\\theta}{\\arg\\max}\u00a0 \u00a0L(\\theta) =\u00a0\\underset{\\theta}{\\arg\\max}\u00a0\\prod\\limits_{i=1}^{N}h_{\\theta}(x^{(i)}) ^{y_i}\u00a0\\thinspace (1-h_{\\theta}(x^{(i)}))^{1-y_i} $$<\/p>\n

Since the next step is always to find a derivative of the likelihood, you almost always\u00a0take the log of the likelihood\u00a0 since it transforms the product into a sum (on which it is much easier to apply derivatives), and that the\u00a0logarithm function is monotonic (strictly increasing), and thus maximizing the log likelihood is equivalent to maximizing the likelihood, as well\u00a0as minimizing the negative log likelihood. So applying a log on the above product gives:<\/p>\n

$$\u00a0\\underset{\\theta}{\\arg\\max}\u00a0\u00a0\\thinspace\u00a0 log\u00a0\\thinspace\u00a0 L(\\theta) = \\\\ \\underset{\\theta}{\\arg\\max}\u00a0 \u00a0\\sum\\limits_{i=1}^{N}\u00a0 y_i log(h_{\\theta}(x^{(i)})) (1-y_i)log(1-h_{\\theta}(x^{(i)}))\u00a0 \u00a0$$<\/p>\n

Instead of looking for the maximum of the log likelihood, you can equivalently look for the minimum of the negative log likelihood. If you take the average negative log likelihood on the training set, what do you obtain? you guessed it, once again, the exact same log loss cost function we found both in the first section and also via cross entropy!!!<\/p>\n

As a final link between logistic regression and other well known concepts in ML or statistics, logistic regression is often compared with Naive Bayes, see here<\/a>\u00a0(wikipedia), here\u00a0<\/a>(more detailed book chapter) and here<\/a>\u00a0(high level Quora answer). But the point is that naive bayes can be seen as a generative version of logistic regression (which is a discriminative model, here<\/a> is a nice Quora discussion if you want to understand the difference between generative and discriminative models ).<\/p>\n

Bottom line<\/strong>: in the context of logistic regression, when you’ll hear about log loss or cross entropy or negative log likelihood, you’ll now know why and how they are so closely related.<\/p>\n

I hope you enjoyed\u00a0 that post. If you want to get to the details allowing to make this work at scale and actually see an implementation connecting it all in 30 lines of python, continue to part 2<\/a> of this series \ud83d\ude42 .