L21: Unsupervised Learning: Topic Models

Latent Dirichlet Allocation (LDA) has been a common approach

• it is easy to implement
• it performs well on small data

However:

• on large datasets with nuanced topics, LDA might struggle to capture the complexity of the text
• when dealing with sparse data, LDA might struggle
• the alternative would be word-embedding techniques such as BERT

When discussing democracy:

• Frequently use the words democracy, elections, and government.
• Infrequently use the words economy, growth, and market.

When discussing the economy:

• Frequently use the words economy, growth, and market.
• Infrequently use the words democracy, elections, and government.

We now compute the topic word probabilities to: 1) understand the document content; 2) identify patterns across a corpus; 3) facilitate human interpretation

Topic Word Probabilities Table

What is the probability of observing Document A’s word counts under the “Democracy” topic?

This following equation shows how likely a document’s words fit under a specific topic based on predefined probabilities. \[ \begin{equation} \begin{aligned} P(W_A|\mu_{\text{Democracy}}) &= \frac{M_{i}!}{\Pi_{j=1}^{J} W_{Aj!}}\Pi^{J}_{j=1}\mu^{W_{Aj}}_{\text{Democracy}j}\\ &= \frac{(4+3+2+1+1+0)!}{4! \cdot 3! \cdot 2! \cdot 1! \cdot 0!} \cdot 0.4^4 \cdot 0.3^3 \cdot 0.25^2 \cdot 0.02^1 \cdot 0.02^1 \cdot 0.01^0\\ &= \frac{11!}{288} \cdot 0.0000001728\\ &= \frac{39916800}{288} \cdot 0.0000001728\\ &=0.02393 \end{aligned} \end{equation} \]

\[ \begin{equation} \begin{aligned} P(W_A|\mu_{\text{Economy}}) &= \frac{M_{i}!}{\Pi_{j=1}^{J} W_{Aj!}}\Pi^{J}_{j=1}\mu^{W_{Aj}}_{\text{Economy}j}\\ &= \frac{(4+3+2+1+1+0)!}{4! \cdot 3! \cdot 2! \cdot 1! \cdot 0!} \cdot 0.02^4 \cdot 0.01^3 \cdot 0.02^2 \cdot 0.3^1 \cdot 0.35^1 \cdot 0.3^0\\ &= \frac{11!}{288} \cdot 0.00000000000672\\ &=\frac{39916800}{288} \cdot 0.00000000000672\\ &=0.000000931392 \end{aligned} \end{equation} \]

\[ \begin{equation} \begin{aligned} P(W_B|\mu_{\text{Democracy}}) &= \frac{M_{i}!}{\Pi_{j=1}^{J} W_{Bj!}}\Pi^{J}_{j=1}\mu^{W_{Bj}}_{\text{Democracy}j}\\ &= \frac{(1+1+1+3+4+3)!}{1! \cdot 1! \cdot 1! \cdot 3! \cdot 4! \cdot 3!} \cdot 0.4^1 \cdot 0.3^1 \cdot 0.25^1 \cdot 0.02^3 \cdot 0.02^4 \cdot 0.01^3\\ &= \frac{13!}{1 \cdot 1 \cdot 1 \cdot 6 \cdot 24 \cdot 6} \cdot 0.000000000000768\\ &= \frac{6227020800}{864} \cdot 0.000000000000768\\ &= 0.0000055300352 \end{aligned} \end{equation} \]

\[ \begin{equation} \begin{aligned} P(W_B|\mu_{\text{Economy}}) &= \frac{M_{i}!}{\Pi_{j=1}^{J} W_{Bj!}}\Pi^{J}_{j=1}\mu^{W_{Bj}}_{\text{Economy}j}\\ &= \frac{(1+1+1+3+4+3)!}{1! \cdot 1! \cdot 1! \cdot 3! \cdot 4! \cdot 3!} \cdot 0.02^1 \cdot 0.01^1 \cdot 0.02^1 \cdot 0.3^3 \cdot 0.35^4 \cdot 0.3^3\\ &= \frac{13!}{1 \cdot 1 \cdot 1 \cdot 6 \cdot 24 \cdot 6} \cdot 0.000000000001296\\ &= \frac{6227020800}{864} \cdot 0.000000000001296\\ &= 0.0000093402048 \end{aligned} \end{equation} \]

\[ \begin{equation} \begin{aligned} P(W_A|\mu_{\text{Democracy + Economy}}) &= \frac{M_{i}!}{\Pi_{j=1}^{J} W_{Aj!}}\Pi^{J}_{j=1}\mu^{W_{Aj}}_{\text{Democracy + Economy}j}\\ &= \frac{(4+3+2+1+1+0)!}{4! \cdot 3! \cdot 2! \cdot 1! \cdot 1! \cdot 0!} \cdot [(0.4+0.02)/2]^4 \cdot 0.155^3 \cdot 0.135^2 \cdot 0.16^1 \cdot 0.185^1 \cdot 0.155^0\\ &= \frac{11!}{24 \cdot 6 \cdot 2 \cdot 1 \cdot 1 \cdot 1} \cdot 0.000019246\\ &= \frac{39916800}{288} \cdot 0.000019246\\ &= 138600 \cdot 0.000019246\\ &= 0.002668524 \end{aligned} \end{equation} \]

\[ \begin{equation} \begin{aligned} P(W_B|\mu_{\text{Democracy + Economy}}) &= \frac{M_{i}!}{\Pi_{j=1}^{J} W_{Bj!}}\Pi^{J}_{j=1}\mu^{W_{Bj}}_{\text{Democracy + Economy}j}\\ &= \frac{(1+1+1+3+4+3)!}{1! \cdot 1! \cdot 1! \cdot 3! \cdot 4! \cdot 3!} \cdot 0.21^1 \cdot 0.155^1 \cdot 0.135^1 \cdot 0.16^3 \cdot 0.185^4 \cdot 0.155^3\\ &= \frac{13!}{1 \cdot 1 \cdot 1 \cdot 6 \cdot 24 \cdot 6} \cdot 0.00000000862176\\ &= \frac{6227020800}{864} \cdot 0.00000000862176\\ &= 7200480 \cdot 0.00000000862176\\ &= 0.00006206069 \end{aligned} \end{equation} \]

A topic estimates two sets of probabilities:

• probability of observing each word for each topic
• probability of observing each topic in each document

These quantities can then be used to organise documents by topic, assess how topics vary across documents, etc.

Each document \(d\) in the corpus is generated as follows:

• A set of \(K\) topics exists before the data
  • Each topic \(k\) is a probability distribution over words (\(\beta\))
  • \(\beta\) tells us how likely each word in the vocabulary is to belong to a given topic.

• A specific mix of those topics is randomly extracted to generate a document
  • this mix is a specific probability distribution over topics (\(\theta\))
  • \(\theta\) tells us the proportion of each topic present in a given document. Unlike \(\beta\), which focuses on words, \(\theta\) focuses on how much of each topic is represented in a document.

• Each word in a document is generated by:
  • First, choosing a topic \(k\) at random from the probability distribution over topics \(\theta\)
  • Then, choosing a word \(w\) at random from the topic-specific probability distribition over documents (\(\beta_k\))

The goal of LDA is to estimate hidden parameter (\(\beta\) and \(\theta\)) starting from \(w\).

• The researcher picks a number of topics, \(K\)

• Each topic (\(k\)) is a distribution over words

• Each document (\(d\)) is a mixture of corpus-wide topics

• Each word (\(w\)) is drawn from one of the topics

\[\begin{equation} \theta = \underbrace{\begin{pmatrix} \theta_{1,1} & \theta_{1,2} & \theta_{1,3}\\ \theta_{2,1} & \theta_{2,2} & \theta_{2,3}\\ ... & ... & ...\\ \theta_{D,1} & \theta_{D,2} & \theta_{D,3}\\ \end{pmatrix}}_{D\times K} = \underbrace{\begin{pmatrix} 0.7 & 0.2 & 0.1\\ 0.1 & 0.8 & 0.1\\ ... & ... & ...\\ 0.3 & 0.3 & 0.4\\ \end{pmatrix}}_{1000 \times 3} \end{equation}\]

\[\begin{equation} \beta = \underbrace{\begin{pmatrix} \beta_{1,1} & \beta_{1,2} & ... & \beta_{1,J}\\ \beta_{2,1} & \beta_{2,2} & ... & \beta_{2,J}\\ \beta_{3,1} & \beta_{3,2} & ... & \beta_{3,J}\\ \end{pmatrix}}_{K\times J} = \underbrace{\begin{pmatrix} 0.04 & 0.0001 & ... & 0.003\\ 0.0004 & 0.001 & ... & 0.00005\\ 0.002 & 0.0003 & ... & 0.0008\\ \end{pmatrix}}_{3 \times 10,000} \end{equation}\]

Johnson’s speech - 3900, features under the following top 5 topics: 15, 0, 17, 13, 6.

The topic with the highest score - 0.3026066 is 15: industri, unit, british, trade, uk

This speech reads as follows:

Disadvantages

• Bag-of-words assumption: Ignores word order and context, losing semantic nuance.
• Sensitive to preprocessing: Requires careful tokenization, stopword removal, and lemmatization.
• Fixed number of topics: Must predefine the number of topics, which may not align with the data.
• Performance limitations: Struggles with short texts and highly overlapping topics.
• Hyperparameter tuning: Requires tuning (e.g., α, β) to improve coherence and quality.

The Trade-Off: More Coherence → Less Exclusivity

• Topics overlap as similar words appear in multiple themes.
• Example: “Growth” might appear in both “Economy” and “Healthcare” topics.

The Trade-Off: More Exclusivity → Less Coherence

• Topics become overly specific and less interpretable.
• Example: A topic with rare words like “NASDAQ” and “dividends” may not clearly reflect “Economy.”