11  Topic Models

Author

Send comments to: Tony T (tthrall)

Published

06:32 Tue 30-Dec-2025

Show the code 
library(here)
library(knitr)
library(tidytext)
library(tidyverse)
library(tm)
library(topicmodels)
Show the code 
library(eda4mlr)
Show the code 
source(here("code", "eda4ml_set_seed.R"))
source(here("code", "text_helpers.R"))

11.1 Introduction

Learning objectives
  1. Describe the generative model underlying Latent Dirichlet Allocation (LDA).
  2. Interpret beta (word distributions per topic) and gamma (topic distributions per document).
  3. Fit an LDA model in R and extract topic-word and document-topic distributions.
  4. Select and justify the number of topics (K) using coherence metrics or domain knowledge.
  5. Evaluate topic quality through inspection of top words and representative documents.
  6. Distinguish Latent Dirichlet Allocation (topic model) from Linear Discriminant Analysis (classification method).
  7. Describe the role of the Dirichlet distribution and its concentration parameter.

Chapter 10 established that text can be represented as high-dimensional, sparse data. We learned to tokenize documents, construct term-document matrices, and use tf-idf to identify distinctive words. We also introduced topic models as methods for discovering latent themes, but deferred the question of how they work.

This chapter answers that question. We develop the intuition behind Latent Dirichlet Allocation (LDA), the most widely used topic model, through examples ranging from a toy corpus of two books to thousands of news articles. Along the way, we encounter the Dirichlet distribution: a probability distribution over probability vectors that gives LDA its name and its power.

11.1.1 The Core Insight

A topic model rests on two assumptions:

  1. Each document is a mixture of topics. A news article might be 70% business and 30% politics; a quarterly report might be 40% financial performance, 35% strategic outlook, and 25% risk factors.

  2. Each topic is a mixture of words. The “business” topic assigns high probability to words like “market”, “profit”, and “growth”; the “politics” topic assigns high probability to “congress”, “vote”, and “campaign”.

Given a corpus of documents, a topic model discovers both the set of topics (each a probability distribution over words) and the mixture of topics in each document. The model knows nothing about business or politics a priori; it infers topics from patterns of word co-occurrence.

11.1.2 Corpus Requirements for Topic Modeling

Topic modeling places specific demands on the corpus. Single-author literary corpora like janeaustenr can work—one might discover topics corresponding to dialogue, description, and narration—but richer thematic structure emerges from multi-author or multi-domain collections. The Associated Press corpus (topicmodels::AssociatedPress) provides 2,246 news articles spanning diverse subjects, making it well-suited for discovering interpretable topics. For students who wish to compare discovered topics against known categories, textdata::dataset_ag_news() provides news articles with category labels. Chapter 10 discusses corpus selection in greater detail.

11.2 The Generative Story

The key to understanding LDA is the generative story: a fictional account of how each document was created. Imagine generating a document as follows:

Step 1: Choose a topic mixture for this document.

Before writing anything, decide what proportion of the document will be devoted to each topic. For instance: “This document will be 70% Topic A and 30% Topic B.” This mixture is represented as a probability vector \(\gamma_{d,\bullet} = (\gamma_{d,1}, \ldots, \gamma_{d,K})\) summing to 1.

Step 2: For each word position in the document:

  1. Roll the topic dice according to \(\gamma_{d,\bullet}\) to select a topic.
  2. From that topic’s word distribution, generate a word.

Repeat until the document is complete.

This process is called a bag-of-words model because word order does not matter—only which words appear and how often. The generative story is fictional; no author actually writes this way. But if documents were generated this way, the resulting word patterns would resemble what we observe in real corpora.

LDA inverts this process. Given observed documents, LDA infers the topics and mixtures that best explain the data. The algorithm asks: “What topics, and what document-specific mixtures of those topics, would make these documents likely?”

11.3 Toy Example: Two Books

To build intuition, we begin with a corpus where we know the ground truth. Can LDA distinguish two very different books without being told which paragraphs came from which?

11.3.1 The Data

Animal Farm (George Orwell, 1945) tells of farm animals who overthrow their human masters, only to see the pigs become tyrants—an allegory for Soviet communism. The Butter Battle Book (Dr. Seuss, 1984) depicts Yooks and Zooks in escalating conflict over which side to butter bread—an allegory for the Cold War nuclear arms race.

We use Wikipedia’s plot summaries of these books, treating each paragraph as a separate “document.” This yields a small corpus where we know each document’s true source.

Show the code 
# Animal Farm: story plot from Wikipedia
af_pst_lst <- para2token(
  file = here::here("data", "retain", "Animal_Farm_summary.txt")
)

af_token_tbl <- af_pst_lst$token_tbl |>
  dplyr::mutate(src = "AF") |>
  dplyr::select(src, dplyr::everything())

# The Butter Battle Book: story plot from Wikipedia
bbb_pst_lst <- para2token(
  file = here::here("data", "retain", "Butter_Battle_summary.txt")
)

bbb_token_tbl <- bbb_pst_lst$token_tbl |>
  dplyr::mutate(src = "BBB") |>
  dplyr::select(src, dplyr::everything())

# Combine
af_bbb_token_tbl <- dplyr::bind_rows(af_token_tbl, bbb_token_tbl)
Show the code 
corpus_summary <- af_bbb_token_tbl |>
  dplyr::summarise(
    .by = src,
    n_paragraphs = dplyr::n_distinct(pdx),
    n_words = dplyr::n()
  ) |>
  dplyr::mutate(
    book = dplyr::case_when(
      src == "AF" ~ "Animal Farm",
      src == "BBB" ~ "The Butter Battle Book"
    )
  ) |>
  dplyr::select(book, n_paragraphs, n_words)
Show the code 
corpus_summary |>
  knitr::kable(col.names = c("Book", "Paragraphs", "Words (excl. stop words)"))
Table 11.1: Mini-corpus: paragraphs from two book summaries
Book Paragraphs Words (excl. stop words)
Animal Farm 5 553
The Butter Battle Book 4 172

11.3.2 Fitting LDA with K = 2

We construct a document-term matrix and fit LDA requesting \(K = 2\) topics—matching the number of source books.

Show the code 
# Count words per paragraph
af_bbb_counts <- af_bbb_token_tbl |>
  dplyr::mutate(doc_id = paste(src, pdx, sep = "_")) |>
  dplyr::count(doc_id, word)

# Convert to document-term matrix
af_bbb_dtm <- af_bbb_counts |>
  tidytext::cast_dtm(document = doc_id, term = word, value = n)
Show the code 
af_bbb_lda <- topicmodels::LDA(

  x       = af_bbb_dtm,
  k       = 2,
  method  = "Gibbs",
  control = list(seed = 42)
)

The LDA() function from the topicmodels package fits the model using Gibbs sampling, an iterative algorithm that infers topic assignments. Setting a seed ensures reproducibility.

11.3.3 Examining the Results: Word Distributions (β)

Each topic is a probability distribution over words. We extract these distributions and examine the highest-probability words for each topic:

Show the code 
af_bbb_topics <- tidytext::tidy(af_bbb_lda, matrix = "beta")
Show the code 
af_bbb_top_terms <- af_bbb_topics |>
  dplyr::group_by(topic) |>
  dplyr::slice_max(beta, n = 10, with_ties = FALSE) |>
  dplyr::ungroup()

af_bbb_top_terms |>
  dplyr::mutate(
    term = tidytext::reorder_within(term, beta, topic),
    topic = paste("Topic", topic)
  ) |>
  ggplot2::ggplot(ggplot2::aes(x = beta, y = term, fill = topic)) +
  ggplot2::geom_col(show.legend = FALSE) +
  ggplot2::facet_wrap(~topic, scales = "free_y") +
  tidytext::scale_y_reordered() +
  ggplot2::labs(
    x = expression(paste("Word probability (", beta, ")")),
    y = NULL
  ) +
  ggplot2::theme_minimal(base_size = 12)
Figure 11.1: Top words by probability in each topic (Animal Farm vs Butter Battle corpus)

Topic 1’s top words include “animals”, “napoleon”, “farm”, and “snowball”—clearly Animal Farm. Topic 2’s top words include “yooks”, “zooks”, “wall”, and “battle”—clearly The Butter Battle Book. LDA successfully separated the two sources without being told which paragraphs came from which book.

11.3.4 Log-Ratio Visualization

Another way to compare topics is to examine which words most strongly distinguish them. For words appearing in both topics, we compute the log-ratio of probabilities:

\[ \log_2 \left( \frac{\beta_2}{\beta_1} \right) \qquad(11.1)\]

Positive values indicate words more probable in Topic 2; negative values indicate words more probable in Topic 1.

Show the code 
af_bbb_log_ratio <- af_bbb_topics |>
  dplyr::mutate(topic = paste0("beta_", topic)) |>
  tidyr::pivot_wider(names_from = topic, values_from = beta) |>
  dplyr::filter(
    pmin(beta_1, beta_2) > 0,
    pmax(beta_1, beta_2) > 0.001
  ) |>
  dplyr::mutate(log_ratio = log2(beta_2 / beta_1))
Show the code 
af_bbb_log_ratio |>
  dplyr::group_by(direction = log_ratio > 0) |>
  dplyr::slice_max(abs(log_ratio), n = 8) |>
  dplyr::ungroup() |>
  dplyr::mutate(term = stats::reorder(term, log_ratio)) |>
  ggplot2::ggplot(ggplot2::aes(x = log_ratio, y = term)) +
  ggplot2::geom_col(ggplot2::aes(fill = log_ratio > 0), show.legend = FALSE) +
  ggplot2::scale_fill_manual(values = c("steelblue", "coral")) +
  ggplot2::geom_vline(xintercept = 0, linetype = "dashed") +
  ggplot2::labs(
    x = expression(log[2](beta[2] / beta[1])),
    y = NULL
  ) +
  ggplot2::theme_minimal(base_size = 12)
Figure 11.2: Words distinguishing Topic 2 (Butter Battle) from Topic 1 (Animal Farm)

The character names “yooks” and “zooks” strongly distinguish Topic 2, while “napoleon”, “snowball”, and “animals” strongly distinguish Topic 1. This visualization complements the top-words bar chart by highlighting contrast rather than absolute probability.

11.3.5 Toy Example: Takeaways

This toy example demonstrates that LDA works—it recovered the two sources without supervision. But interpretation requires judgment: “yooks” means nothing without context. And with only 9 documents (paragraphs), this is far from a realistic application. Real corpora have thousands of documents, and the topics discovered may not correspond to obvious categories.

11.4 Scaling Up: AP News Articles

We now turn to a larger corpus: 2,246 Associated Press articles from around 1988, provided by the topicmodels package. In Chapter 10, we fit \(K = 2\) topics and found one resembling business/finance and another resembling politics/government. What happens with more topics?

11.4.1 Fitting LDA with K = 4

Show the code 
ap_dtm <- eda4mlr::ap_tidy |>
  tidytext::cast_dtm(document = document, term = term, value = count)
Show the code 
ap_lda_k4 <- topicmodels::LDA(
  x       = ap_dtm,
  k       = 4,
  method  = "Gibbs",
  control = list(seed = 42, iter = 1000)
)
Show the code 
ap_topics_k4 <- tidytext::tidy(ap_lda_k4, matrix = "beta")
Show the code 
ap_top_terms_k4 <- ap_topics_k4 |>
  dplyr::group_by(topic) |>
  dplyr::slice_max(beta, n = 8, with_ties = FALSE) |>
  dplyr::ungroup()

ap_top_terms_k4 |>
  dplyr::mutate(
    term = tidytext::reorder_within(term, beta, topic),
    topic = paste("Topic", topic)
  ) |>
  ggplot2::ggplot(ggplot2::aes(x = beta, y = term, fill = topic)) +
  ggplot2::geom_col(show.legend = FALSE) +
  ggplot2::facet_wrap(~topic, scales = "free_y", ncol = 2) +
  tidytext::scale_y_reordered() +
  ggplot2::labs(
    x = expression(paste("Word probability (", beta, ")")),
    y = NULL
  ) +
  ggplot2::theme_minimal(base_size = 11)
Figure 11.3: Top words in each of four topics discovered in AP articles
Show the code 
# ap_top_terms_k4 |> 
#   summarise(
#     .by = topic, terms = str_flatten(term, collapse = ", ") )
# # A tibble: 4 × 2
#   topic terms                                                            
#   <int> <chr>                                                            
# 1     1 i, court, years, time, state, case, says, get                    
# 2     2 police, two, people, officials, three, city, air, wednesday      
# 3     3 percent, million, year, new, billion, last, company, market      
# 4     4 president, government, states, united, soviet, bush, party, house

ap_top_terms_k4_smy <- 
  ap_top_terms_k4 |> 
  dplyr::summarise(
    .by = topic, terms = str_flatten(term, collapse = ", ") ) |> 
  dplyr::mutate(
    theme = c(
      "Legal", 
      "Local News/Crime", 
      "Business/Finance", 
      "International/Cold War" ) )

With four topics, we see finer distinctions emerge:

Show the code 
ap_top_terms_k4_smy |> 
  knitr::kable(
    col.names = c("Topic", "Top Words", "Interpretation")
  )
Table 11.2
Topic Top Words Interpretation
1 i, court, years, time, state, case, says, get Legal
2 police, two, people, officials, three, city, air, wednesday Local News/Crime
3 percent, million, year, new, billion, last, company, market Business/Finance
4 president, government, states, united, soviet, bush, party, house International/Cold War

These interpretations require domain knowledge. The algorithm provides word distributions; the analyst provides labels. Note that the 1988 publication date explains the prominence of Cold War vocabulary in Topic 4.

11.4.2 Topic Mixtures per Document \((\gamma)\)

LDA also estimates each document’s topic mixture. We extract these as the \(\gamma\) matrix:

Show the code 
ap_gamma_k4 <- tidytext::tidy(ap_lda_k4, matrix = "gamma")
Show the code 
set.seed(123)
sample_docs <- sample(unique(ap_gamma_k4$document), 4)

ap_gamma_k4 |>
  dplyr::filter(document %in% sample_docs) |>
  tidyr::pivot_wider(names_from = topic, values_from = gamma) |>
  knitr::kable(
    digits = 2,
    col.names = c("Doc ID", "Topic 1", "Topic 2", "Topic 3", "Topic 4")
  )
Table 11.3: Topic mixtures for selected AP articles
Doc ID Topic 1 Topic 2 Topic 3 Topic 4
195 0.53 0.16 0.14 0.18
526 0.18 0.44 0.15 0.23
1842 0.12 0.62 0.13 0.13
2227 0.10 0.09 0.72 0.09

Some documents lean heavily toward one topic; others are more evenly mixed. This “soft clustering” is a key feature of topic models: unlike \(K\)-means, which assigns each observation to exactly one cluster, LDA allows documents to belong partially to multiple topics.

Show the code 
ap_gamma_k4 |>
  dplyr::group_by(document) |>
  dplyr::slice_max(gamma, n = 1) |>
  dplyr::ungroup() |>
  ggplot2::ggplot(ggplot2::aes(x = gamma)) +
  ggplot2::geom_histogram(binwidth = 0.05, fill = "steelblue", color = "white") +
  ggplot2::labs(
    x = expression(paste("Dominant topic proportion (", gamma, ")")),
    y = "Number of documents"
  ) +
  ggplot2::theme_minimal(base_size = 12)
Figure 11.4: Distribution of dominant topic proportions across AP articles

The histogram shows that while some articles are dominated by a single topic (\(\gamma > 0.8\)), many are mixtures. This reflects the reality that news articles often cover multiple themes—a story about government economic policy might blend politics and business.

11.5 The Dirichlet Distribution

Why is this method called Latent Dirichlet Allocation? The Dirichlet distribution is the mathematical engine that makes LDA work. Understanding it conceptually—without diving into the full mathematical treatment—illuminates the model’s behavior.

11.5.1 A Distribution Over Mixtures

Recall that each document has a topic mixture: a probability vector like \((0.7, 0.2, 0.1)\) indicating proportions devoted to each topic. Where do these mixtures come from?

LDA assumes topic mixtures are drawn from a Dirichlet distribution—a probability distribution whose outcomes are themselves probability vectors. Just as a normal distribution generates real numbers, the Dirichlet distribution generates vectors of positive numbers that sum to 1.

11.5.2 The Concentration Parameter

The Dirichlet distribution has a concentration parameter \(\alpha\) that controls how “spread out” or “concentrated” the generated mixtures are:

Small \(\alpha\) (< 1): Generates sparse mixtures. Documents tend toward one dominant topic. “This article is 95% politics.”

Large \(\alpha\) (> 1): Generates uniform mixtures. Documents spread across many topics. “This article is 30% politics, 35% business, 35% international.”

Show the code 
set.seed(42)
n_samples <- 500

# Helper function to sample from Dirichlet
rdirichlet <- function(n, alpha) {
  k <- length(alpha)
  x <- matrix(
    stats::rgamma(n * k, shape = rep(alpha, each = n)),
    nrow = n,
    byrow = FALSE
  )
  x / rowSums(x)
}

# Low concentration (sparse mixtures)
low_conc <- rdirichlet(n_samples, c(0.1, 0.1, 0.1)) |>
 tibble::as_tibble(.name_repair = ~c("Topic1", "Topic2", "Topic3")) |>
  dplyr::mutate(concentration = "α = 0.1 (sparse)")

# High concentration (uniform mixtures)
high_conc <- rdirichlet(n_samples, c(5, 5, 5)) |>
 tibble::as_tibble(.name_repair = ~c("Topic1", "Topic2", "Topic3")) |>
  dplyr::mutate(concentration = "α = 5 (uniform)")

dplyr::bind_rows(low_conc, high_conc) |>
  ggplot2::ggplot(ggplot2::aes(x = Topic1, y = Topic2, color = concentration)) +
  ggplot2::geom_point(alpha = 0.3, size = 1) +
  ggplot2::facet_wrap(~concentration) +
  ggplot2::coord_fixed() +
  ggplot2::labs(
    x = "Proportion Topic 1",
    y = "Proportion Topic 2"
  ) +
  ggplot2::theme_minimal(base_size = 12) +
  ggplot2::theme(legend.position = "none")

Samples from Dirichlet distributions with different concentration parameters

With \(\alpha = 0.1\), most samples cluster near the corners of the simplex—indicating dominance by one topic. With \(\alpha = 5\), samples cluster near the center—indicating balanced mixtures. LDA typically uses small \(\alpha\), reflecting the intuition that documents are usually about something specific.

11.5.3 Two Dirichlet Distributions in LDA

LDA actually uses two Dirichlet distributions:

  1. The one just noted, having concentration parameter \(\alpha\), controls topic mixtures per document. Small \(\alpha\)-values mean documents focus on few topics.

  2. The other, having concentration parameter \(\eta\), controls word mixtures per topic. Small \(\eta\)-values mean topics focus on specific words rather than spreading probability across the vocabulary.

These are hyper-parameters that can be tuned, though default values often work well. For a fuller mathematical treatment of the Dirichlet distribution, including its density function and conjugacy to the multinomial distribution, see Appendix 11.A.

11.6 Choosing K

Like \(K\)-means clustering, LDA requires specifying the number of topics \(K\). This choice significantly affects the results.

Too few topics: Themes are too broad. Business and finance might be lumped together; important distinctions are lost.

Too many topics: Themes are too narrow. Redundant topics emerge; interpretation becomes difficult.

11.6.1 Comparing K Values

Show the code 
ap_lda_k2 <- topicmodels::LDA(
  ap_dtm, k = 2, method = "Gibbs",
  control = list(seed = 42, iter = 500)
)

ap_lda_k8 <- topicmodels::LDA(
  ap_dtm, k = 8, method = "Gibbs",
  control = list(seed = 42, iter = 500)
)
Show the code 
get_top_words <- function(lda_model, k_val) {
  tidytext::tidy(lda_model, matrix = "beta") |>
    dplyr::group_by(topic) |>
    dplyr::slice_max(beta, n = 5, with_ties = FALSE) |>
    dplyr::summarise(top_words = paste(term, collapse = ", ")) |>
    dplyr::mutate(K = k_val)
}

k_comparison <- dplyr::bind_rows(
  get_top_words(ap_lda_k2, 2),
  get_top_words(ap_lda_k4, 4),
  get_top_words(ap_lda_k8, 8)
)

With \(K = 2\), we get broad themes (business vs. politics). With \(K = 4\), we see finer distinctions (domestic politics vs. international affairs). With \(K = 8\), we might discover additional themes like sports, legal affairs, or technology—or we might see redundant, hard-to-interpret topics.

Show the code 
k_comparison |>
  dplyr::arrange(K, topic) |>
  dplyr::mutate(topic = paste("Topic", topic)) |>
  tidyr::pivot_wider(names_from = K, values_from = top_words) |>
  knitr::kable(col.names = c("Topic", "K = 2", "K = 4", "K = 8"))
Table 11.4: Top words by topic for different values of K
Topic K = 2 K = 4 K = 8
Topic 1 i, people, two, president, government i, court, years, time, state soviet, government, party, union, president
Topic 2 percent, new, year, million, last police, two, people, officials, three bush, president, house, i, dukakis
Topic 3 NA percent, million, year, new, billion i, years, time, like, first
Topic 4 NA president, government, states, united, soviet police, people, city, two, miles
Topic 5 NA NA percent, million, year, billion, new
Topic 6 NA NA court, case, federal, state, attorney
Topic 7 NA NA military, united, two, force, air
Topic 8 NA NA year, workers, program, new, report

11.6.2 Practical Guidance

There is no universally reliable method for choosing \(K\). Practical approaches include:

  1. Domain knowledge. How many themes do you expect? For news articles, 4–10 might be reasonable; for academic papers in a narrow field, 3–5 might suffice.

  2. Interpretability. Can you name each topic? Do the top words cohere? If topics are redundant or incoherent, try fewer.

  3. Iteration. Fit models with \(K = 2, 3, 5, 10\). Compare results. Look for the value where topics are distinct and interpretable.

  4. Coherence metrics. Statistical measures of topic quality exist (e.g., based on word co-occurrence), but they do not always align with human judgment.

  5. Consider your goal. For exploration (getting a broad sense of themes), fewer topics may suffice. For fine-grained analysis, more topics may be warranted.

11.7 Summary

This chapter developed the intuition and mechanics of topic models, focusing on Latent Dirichlet Allocation.

Key concepts:

  • Topic models assume documents are mixtures of topics, and topics are mixtures of words.
  • The generative story imagines documents created by first choosing a topic mixture, then generating each word from a topic-specific distribution.
  • LDA inverts this process, inferring topics from observed word patterns.
  • The Dirichlet distribution generates probability vectors; its concentration parameter controls whether mixtures are sparse (one dominant topic) or uniform (many topics).
  • Choosing \(K\) requires domain knowledge and iterative exploration; there is no automatic method.

Key notation:

Symbol Meaning
\(K\) Number of topics (analyst’s choice)
\(\alpha_\bullet\) Dirichlet parameter for topic mixture per document
\(\eta_\bullet\) Dirichlet parameter for word mixture per topic
\(\beta_{k,v}\) Probability of term \(v\) in topic \(k\)
\(\gamma_{d,k}\) Proportion of topic \(k\) in document \(d\)

Constraints:

  • For each topic \(k\): \(\sum_v \beta_{k,v} = 1\)
  • For each document \(d\): \(\sum_k \gamma_{d,k} = 1\)

Connections to earlier material:

Earlier Concept Topic Model Analog
\(K\)-means clustering LDA (documents → topics)
Cluster centroids Word mixture (\(\beta_{k,\bullet}\)) per topic
Cluster assignments Topic mixture (\(\gamma_{d,\bullet}\)) per document
Hard vs. soft clustering \(K\)-means (hard) vs. LDA (soft)
PCA loadings Word weights per topic

Topic models are “soft clustering”—documents belong partially to multiple topics, just as factor analysis allows observations to load on multiple factors.

The Two LDAs:

Chapter 9 introduced Linear Discriminant Analysis—a supervised method that finds directions separating known classes. This chapter presented Latent Dirichlet Allocation—an unsupervised method that discovers topics from word co-occurrence. Both “allocate” observations to categories, but with very different goals and methods.

11.8 Exercises

  1. Interpret Topics. Using the AP News \(K = 4\) results: (a) Propose alternative labels for each topic, justifying with top words. (b) Which topic is most coherent? Least coherent? Why? (c) Find a word that appears prominently in multiple topics. What does this suggest about that word?

  2. Document Mixtures. For a document with \(\gamma_{d,\bullet} = (0.4, 0.3, 0.2, 0.1)\): (a) What does this mixture tell you about the document’s content? (b) How would you explain this to a non-technical colleague? (c) Would you classify this document into one topic or multiple? What are the tradeoffs?

  3. Choosing K. You have a corpus of 5,000 customer support tickets. (a) What value of \(K\) would you start with? Why? (b) Design an evaluation process to compare different values of \(K\). (c) How would you explain your final choice to stakeholders?

  4. LDA vs. LLMs. For theme discovery in a large corpus: (a) When would you prefer LDA over asking an LLM to identify themes? (b) When would you prefer an LLM? (c) Design a hybrid workflow that combines both approaches, specifying which tool you would use at each stage and why.

11.9 Resources

Foundational papers:

  1. Latent Dirichlet Allocation. Journal of Machine Learning Research. The original LDA paper.

  2. Probabilistic Topic Models. In Handbook of Latent Semantic Analysis. Accessible introduction with details on Gibbs sampling.

Practical guidance:

  1. Text Mining with R, Chapter 6. Tidy approach to topic modeling in R.

  2. LDA in Practice. Blog post with practical advice on hyperparameters and evaluation.

R functions reference:

Function Package Purpose
LDA() topicmodels Fit LDA model
tidy() tidytext Extract β (word-topic) or γ (document-topic) matrices
cast_dtm() tidytext Convert tidy data to document-term matrix
perplexity() topicmodels Compute model fit metric

11.10 Appendix A: The Dirichlet Distribution

This appendix provides the mathematical foundation for the Dirichlet distribution, which underlies the “mixture” structure in LDA. We begin with the multinomial distribution, introduce Bayesian inference with conjugate priors, and then present the Dirichlet distribution as the natural generalization.

11.10.1 The Multinomial Distribution

Consider a categorical random variable \(X\) that selects one of \(K\) distinct categories \(\{c_1, \ldots, c_K\}\) with probabilities \(p_\bullet = (p_1, \ldots, p_K)\) where \(\sum_k p_k = 1\) and each \(p_k > 0\).

If we observe \(n\) independent draws from this distribution, the counts \(S_\bullet = (S_1, \ldots, S_K)\) follow a multinomial distribution:

\[ P(S_\bullet = s_\bullet \mid p_\bullet) = \binom{n}{s_\bullet} \prod_{k=1}^{K} p_k^{s_k} \qquad(11.2)\]

where \(\binom{n}{s_\bullet} = \frac{n!}{s_1! \cdots s_K!}\) is the multinomial coefficient.

For \(K = 2\), this reduces to the binomial distribution.

11.10.2 Bayesian Inference and Conjugate Priors

In Bayesian inference, we represent uncertainty about the probability vector \(p_\bullet\) with a prior distribution. After observing data, we update to a posterior distribution.

A prior is conjugate to a likelihood if the posterior belongs to the same family as the prior. This property simplifies computation enormously.

For the binomial distribution (i.e., \(K = 2\)), the conjugate prior is the beta distribution:

\[ P(p) = \frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\Gamma(\beta)} p^{\alpha - 1}(1-p)^{\beta - 1} \qquad(11.3)\]

where \(\alpha, \beta > 0\) are shape parameters and \(\Gamma(\cdot)\) is the gamma function.

If we observe \(s_1\) successes and \(s_2 = n - s_1\) failures, the posterior is \(\text{Beta}(\alpha + s_1, \beta + s_2)\): the same family, with updated parameters.

11.10.3 The Dirichlet Distribution

The Dirichlet distribution generalizes the beta distribution to \(K \geq 2\) categories. It is the conjugate prior for the multinomial distribution.

The Dirichlet distribution with parameter vector \(\alpha_\bullet = (\alpha_1, \ldots, \alpha_K)\) has density:

\[ P(p_\bullet) = \frac{\Gamma(\alpha_1 + \cdots + \alpha_K)}{\Gamma(\alpha_1) \times \cdots \times \Gamma(\alpha_K)} \prod_{k=1}^{K} p_k^{\alpha_k - 1} \qquad(11.4)\]

where each \(\alpha_k > 0\) and the probability vector \(p_\bullet\) lies on the \((K-1)\)-dimensional simplex (i.e., \(p_k \geq 0\) and \(\sum_k p_k = 1\)).

11.10.4 The Concentration Parameter

In the symmetric case where all \(\alpha_k = \bar{\alpha}\), the sum \(\alpha_+ = K\bar{\alpha}\) is called the concentration parameter. Its value controls the distribution’s shape:

  • \(\bar{\alpha} < 1\): The distribution concentrates near the corners of the simplex, generating sparse probability vectors dominated by one or few categories.

  • \(\bar{\alpha} = 1\): The distribution is uniform over the simplex.

  • \(\bar{\alpha} > 1\): The distribution concentrates near the center of the simplex, generating probability vectors spread more evenly across categories.

Different technical communities parameterize the Dirichlet differently. Some use \(\alpha_+\) directly; others use \(\bar{\alpha} = \alpha_+/K\). The topicmodels package uses a symmetric Dirichlet with a single concentration value. When reading the literature, check which convention is in use.

11.10.5 Conjugacy and Posterior Updating

If the prior on \(p_\bullet\) is \(\text{Dir}(\alpha_\bullet)\) and we observe multinomial counts \(s_\bullet = (s_1, \ldots, s_K)\), the posterior is:

\[ p_\bullet \mid s_\bullet \sim \text{Dir}(\alpha_\bullet + s_\bullet) \qquad(11.5)\]

That is, we simply add the observed counts to the prior parameters. This elegant updating rule is why the Dirichlet distribution is central to LDA: it makes the Gibbs sampling updates computationally tractable.

11.10.6 Role in LDA

In LDA, Dirichlet distributions serve as priors in two places:

  1. Topic mixture per document: Each document’s topic proportions \(\gamma_{d,\bullet}\) are drawn from \(\text{Dir}(\alpha_\bullet)\). A small \(\alpha\) encourages documents to focus on few topics.

  2. Word mixture per topic: Each topic’s word probabilities \(\beta_{k,\bullet}\) are drawn from \(\text{Dir}(\eta_\bullet)\). A small \(\eta\) encourages topics to assign high probability to few words.

The conjugacy property means that as we observe words in documents and update our beliefs about topic assignments, the posterior distributions remain Dirichlet, enabling efficient iterative inference.

11.10.7 References for Appendix A

See Dirichlet distribution on Wikipedia for an accessible introduction, Blei, Ng, and Jordan (2003) for the original application to topic modeling, and Steyvers and Griffiths (2007) for details on the Gibbs sampling algorithm.

11.11 Appendix B: Mathematical Framework

For readers seeking a more formal treatment, this appendix presents the mathematical notation and generative model underlying Latent Dirichlet Allocation (LDA) in greater detail.

11.11.1 Notation

Let \(\mathcal{D}\) denote a corpus of documents. Each document \(d \in \mathcal{D}\) is a sequence of words:

\[ d = (w_1, w_2, \ldots, w_{|d|}) \]

where \(|d|\) denotes the number of words in document \(d\). The vocabulary \(V\) is the set of distinct terms appearing in the corpus, with \(|V| = N\) terms indexed as \(v_1, \ldots, v_N\).

We specify \(K\) topics. Each topic \(k\) is a probability distribution over the vocabulary, represented as a vector \(\beta_{k,\bullet} = (\beta_{k,1}, \ldots, \beta_{k,N})\) where \(\beta_{k,v}\) is the probability of term \(v\) under topic \(k\), and \(\sum_v \beta_{k,v} = 1\).

Each document \(d\) has a topic mixture \(\gamma_{d,\bullet} = (\gamma_{d,1}, \ldots, \gamma_{d,K})\) where \(\gamma_{d,k}\) is the proportion of document \(d\) devoted to topic \(k\), and \(\sum_k \gamma_{d,k} = 1\).

11.11.2 The Generative Model

LDA posits that documents are generated as follows:

  1. For each topic \(k \in \{1, \ldots, K\}\): Draw word distribution \(\beta_{k,\bullet} \sim \text{Dir}(\eta_\bullet)\).

  2. For each document \(d \in \mathcal{D}\):

    1. Draw topic mixture \(\gamma_{d,\bullet} \sim \text{Dir}(\alpha_\bullet)\).
    2. For each word position \(j \in \{1, \ldots, |d|\}\):
      • Draw topic assignment \(z_{d,j} \sim \text{Multinomial}(\gamma_{d,\bullet})\).
      • Draw word \(w_{d,j} \sim \text{Multinomial}(\beta_{z_{d,j},\bullet})\).

Here \(\text{Dir}(\cdot)\) denotes the Dirichlet distribution and \(\alpha_\bullet\), \(\eta_\bullet\) are hyperparameters controlling the concentration of topic and word mixtures respectively.

11.11.3 Inference

Given observed documents, we wish to infer the latent variables: topic distributions \(\beta_{k,\bullet}\), document mixtures \(\gamma_{d,\bullet}\), and word-level topic assignments \(z_{d,j}\). Exact inference is intractable, so practical implementations use approximate methods.

The topicmodels package offers two approaches: Variational Expectation-Maximization (VEM, the default) and Gibbs sampling. Gibbs sampling iteratively updates each word’s topic assignment conditional on all other assignments, eventually converging to a sample from the posterior distribution. The Dirichlet distribution’s conjugacy to the multinomial makes these updates computationally efficient.

For further details on inference algorithms, see 2 and 1.

11.11.4 References for Appendix B

See Blei, Ng, and Jordan (2003) for the original LDA paper with full mathematical details, and Steyvers and Griffiths (2007) for an accessible treatment of the Gibbs sampling algorithm.