{"id":1084,"date":"2010-12-30T09:56:25","date_gmt":"2010-12-30T14:56:25","guid":{"rendered":"http:\/\/philippeadjiman.com\/blog\/?p=1084"},"modified":"2025-07-18T13:47:34","modified_gmt":"2025-07-18T13:47:34","slug":"how-to-easily-build-and-observe-tf-idf-weight-vectors-with-lucene-and-mahout","status":"publish","type":"post","link":"https:\/\/philippeadjiman.com\/blog\/2010\/12\/30\/how-to-easily-build-and-observe-tf-idf-weight-vectors-with-lucene-and-mahout\/","title":{"rendered":"How To Easily Build And Observe TF-IDF Weight Vectors With Lucene And Mahout"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n

You have a collection of text documents, and you want to build their TF-IDF<\/a> weight vectors, probably before doing some clustering<\/a> on the collection or other related tasks.<\/p>\n

You would like to be able for instance to see what are the tokens with the biggest TF-IDF weights in any given document of the collection.<\/p>\n

Lucene<\/a> and\u00a0 Mahout<\/a> can help you to do that almost in a snap.<\/p>\n

Step 1 : Build a Lucene Index out of your document collection<\/strong><\/h3>\n

If you don’t know how to build a Lucene index, check the links at the end of the post.<\/p>\n

The two only important things in that step are to have in your index a field that can serve as a document id and to enable term vectors on the text field representing the content of your documents.<\/p>\n

So your indexing code should contains at least two lines similar to:<\/p>\n

doc.add(new Field(\"documentId\", documentId, Field.Store.YES, Field.Index.NOT_ANALYZED));\ndoc.add(new Field(\"content\", content, Field.Store.YES, Field.Index.ANALYZED,TermVector.YES));<\/pre>\n
Step 2 : Use Mahout lucene.vector driver to generate weighted vectors from your lucene index <\/strong><\/h3>\n
That step is well described here<\/a>. It also explains how to generate the vectors from a directory of text documents. I used lucene because my documents were in a data store and building the lucene index out of it was just much more flexible and convenient.<\/p>\n
You then should end up executing a command similar to:<\/p>\n
 .\/mahout lucene.vector --dir \"myLucenIndexDirectory\" --output \"outputVectorPathAndFilename\" --dictOut \"outputDictionnaryPathAndFilename\" -f content -i documentId -w TFIDF<\/pre>\n
Mahout will generate for you:<\/p>\n