{"id":1215,"date":"2018-04-03T07:05:45","date_gmt":"2018-04-03T07:05:45","guid":{"rendered":"http:\/\/www.philippeadjiman.com\/blog\/?p=1215"},"modified":"2025-11-11T06:32:48","modified_gmt":"2025-11-11T06:32:48","slug":"deep-dive-into-logistic-regression-part-3","status":"publish","type":"post","link":"https:\/\/philippeadjiman.com\/blog\/2018\/04\/03\/deep-dive-into-logistic-regression-part-3\/","title":{"rendered":"Deep Dive Into Logistic Regression: Part 3"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n

In part 1<\/a> and part 2<\/a>\u00a0of this series, we set\u00a0both the theoretical and practical foundation of logistic regression and saw how a state of the art implementation can all be implemented in roughly 30 lines of code. In this third (and last) post of this series, we’ll demonstrate the use of a very effective and powerful library to build logistic regression models in practice: Vowpal Wabbit.<\/p>\n

What is Vowpal Wabbit<\/h2>\n

Vowpal Wabbit<\/a>\u00a0(VW) is a general purpose machine learning library which is implementing, among other things, logistic regression with the same ideas we presented in our previous post<\/a> like the hashing trick and per-coordinate adaptive learning rates\u00a0 (in fact, the hashing trick was made popular by that library). A big advantage\u00a0 of Vowpal Wabbit is that it is blazing fast. Not only because its underlying implementation is in C++, but also because it is using the L-BFGS optimization method.\u00a0L-BGFS\u00a0 stand for\u00a0 “Limited-memory Broyden\u2013Fletcher\u2013Goldfarb\u2013Shanno” and basically approximates the Broyden\u2013Fletcher\u2013Goldfarb\u2013Shanno (BFGS<\/a>) method using a limited amount of memory.\u00a0 This method is much more complex to implement than Stochastic Gradient descent (which can be implemented in few lines of code as we saw in our previous post), but is supposedly converging faster (in less iterations). If you want to read more about L-BFGS and\/or understand its difference with other optimisation methods, you can check\u00a0this<\/a>\u00a0 (doc from Vowpal Wabbit) or this<\/a>\u00a0(nice blog post). Note that L-BFGS was empirically observed to be superior to SGD in many cases, in particular in deep learning settings (check out that paper<\/a>\u00a0on that topic).<\/p>\n

Input format, Namespaces and more<\/h2>\n

Many times, i’ve heard people giving up on Vowpal Wabbit because of its input format, even after going quickly over its\u00a0documentation<\/a>\u00a0. So let’s try to present it through a toy (yet real) example that will be used throughout this post to illustrate the main concepts of Vowpal Wabbit. On top of it, i’ll provide an helper tool (in next section) allowing to transform your tabular dataset into the VW input format easily.<\/p>\n

So, the dataset we’ll use can be found here<\/a>\u00a0and represents the attempt of a bank trying to predict if a marketing phone call will end up in a bank term deposit by the customer, based on a bunch of signals like socio-economic factors of the customer like “does he have a loan?”, etc..<\/p>\n

The traditional way to represent such datasets is to have a tsv or csv file, with the header being the name of the signals and each line representing the value of the training example on each signal. Each line of the training set has thus a fixed size, and missing values are just a blank cell or some specific value to indicate that it’s missing. Typically, for that dataset, the header looks like that:<\/p>\n

age;job;marital;education;default;balance;housing;loan;contact;day;month;duration;campaign;pdays;previous;poutcome;y<\/span><\/p>\n

With y being the actual supervision (i.e. did the call ended up in bank term deposit). And a typical training example looks like that:<\/p>\n

58;management;married;tertiary;no;2143;yes;no;unknown;5;may;261;1;-1;0;unknown;no<\/span><\/p>\n

In Vowpal Wabbit, there is no header, and each signal name is embedded in the training example itself. For example, the training example above can look like that in Vowpal Wabbit format:<\/p>\n

-1 |i age:58 balance:2143 duration:261 campaign:1 pdays:-1 previous:0 |c job=management marital=married education=tertiary default=no housing=yes contact=unknown day=5 month=may poutcome=unknown<\/span><\/p>\n

Let’s discuss multiple important things there:<\/p>\n