Asymmetric LDA Priors, Christmas Edition

The end of the year is proving crazy busy as usual, but gensim acquired a cool new feature that I just had to blog about. Ben Trahan sent a patch that allows automatic tuning of Latent Dirichlet Allocation (LDA) hyperparameters in gensim. This means that an optimal, asymmetric alpha can now be trained directly from your data.

Asymmetric Dirichlet Priors

What is that good for? Asymmetric priors are reported to lead to better fitting models (sounds naughtier than it is!).

This means more natural looking LDA topics, because we expect certain groups of words (topics) to be occurring more frequently in documents, and the model can now account for that by learning a non-uniform Dirichlet prior (base measure + concentration). Now this is the prior for “topics in document” distributions, called alpha — there’s another Dirichlet prior in LDA, called eta, which controls the “words in topic” distributions. Using asymmetric eta is reported to give worse results though, so it’s not part of this patch and gensim uses a simple uniform base measure there.

Well, that’s basically it. I would have announced this on the mailing list, except I also want to include an example that will be easier to do in HTML & Javascript:

If you hover over a topic, it will highlight the closest topic from the other model in red (closest = topic with the smallest Hellinger distance). So placing your mouse over a topic from symmetric alpha (columns 1 & 3) will highlight the closest topic from asymmetric alpha model (columns 2 or 4). I tried to fit as many topics on screen as possible, because some highlighted hits may be quite far apart. There’s probably a better way to visualize this 🙂

For many topics there is a 1:1 correspondence between the two models, but for others, a single topic may be “the nearest” for several topics from the other model, and vice versa. In other words, topic A may be the nearest for topic B, but the nearest for topic B may be a different topic C (=the “nearest” relationship is not symmetric). I highlighted these “unusually matching” topics in light red in the table above. Other than that, skimming the table, I don’t see much difference in terms of topic quality on this particular dataset, although the perplexity was slightly lower for the asymmetric model.

I used the bag-of-words Wiki corpus of ~3.7 million documents generated in the previous blog post for training, except I restricted its vocabulary to the 7,702
words from Matt Hoffman’s original onlinelda code:

lda = gensim.models.LdaModel(vocab_filter[mm], id2word=dictionary, alpha='auto')

As you can see, this vocabulary filtering was streamed online, one vector at a time, so I didn’t have to recreate/store the entire Wiki corpus again. Check out the FAQ if you want to do similar transformation tricks in gensim.

Alpha values fitted on the Wikipedia corpus using 100 topics.

LDA perplexity estimates

I also improved reporting of model perplexity while I was at it. In the log, you’ll see lines like

INFO : -6.778 per-word bound, 109.7 perplexity estimate based on a held-out corpus of 386 documents with 18342 words

after every ten mini-batch updates (configurable).

This means that gensim took the current mini-batch and estimated perplexity based on the adjusted variational Evidence Lower BOund (ELBO), on this held-out corpus. The “adjusted” bit means that likelihood is re-weighted as if the held-out corpus was the entire training corpus. This is done to make comparisons between models using a different number of topics/model settings easier, as ELBO also includes model regularization parts that can otherwise dominate the score.

This held-out perplexity is an estimate, so different mini-batches will give different scores. But in general you’ll see the perplexity value decrease as training progresses.

The patch has been merged into gensim’s develop branch and will be part of the next stable release. Big thanks to Ben & happy holidays to everyone!