Package 'faSTM'

Description

Maps a new corpus's terms onto the term indices of a fitted faSTM model, dropping out-of-vocabulary terms — the preprocessing needed before inferring topics for new documents (cf. stm::alignCorpus).

Usage

align_corpus(newdata, model)

Arguments

Value

A list with per-document ids (0-based indices into model$vocab) and counts, plus dropped (count of out-of-vocabulary term tokens).

Description

stm-shaped counterpart to align_corpus(): reindexes new's documents onto old.vocab, dropping out-of-vocabulary terms (and empty documents).

Usage

alignCorpus(new, old.vocab, verbose = TRUE)

Arguments

Value

list(documents, vocab, docs.removed, words.removed).

Description

The average expected change in a topic's proportion per unit of a covariate (continuous: average derivative; factor: average level-vs-reference contrast), averaged over the observed data. Cleaner than reading raw coefficients, especially with splines/interactions (cf. the margins package; stm #271).

Usage

ame(object, covariate, topics = object$topics, h = NULL, ci = 0.95)

Arguments

Value

A data.frame: topic, term, ame, se, lower, upper.

Description

faSTM does not do its own tokenization — it reads an already-prepared document-term representation from the tools the field already uses (quanteda, tidytext) or a plain sparse matrix. as_corpus() normalizes any of these into the structure stm() consumes, dropping empty documents and re-indexing the vocabulary, with metadata kept aligned.

Usage

as_corpus(x, meta = NULL, ...)

Arguments

Value

A faSTM_corpus: a list with documents (named list of 2×n integer matrices: row 1 = 1-based term id, row 2 = count), vocab (character), meta (data.frame or NULL), and word_counts (corpus term frequencies).

Description

Returns a long data.frame (K, metric, value) ready for ggplot2 — ggplot(as.data.frame(res), aes(K, value)) + geom_line() + facet_wrap(~metric, scales = "free_y").

Usage

## S3 method for class 'faSTM_searchk'
as.data.frame(x, ...)

Arguments

Description

Port of stm::asSTMCorpus's role: accepts a faSTM_corpus, quanteda dfm, or document-term matrix and returns list(documents, vocab, data) in stm format.

Usage

asSTMCorpus(documents, vocab = NULL, data = NULL, ...)

Arguments

Value

list(documents, vocab, data).

Description

Assigns each (document, term) cell to the topic maximizing theta[doc, k] * beta[k, term] (cf. tidytext::augment.STM).

Usage

## S3 method for class 'faSTM'
augment(x, data = NULL, ...)

Arguments

Value

A data.frame: document, term, count, .topic.

Description

Ports of stm:::calcfrex/calclift/calcscore. Each takes a K x V logbeta (log topic-word matrix) and returns a V x K matrix whose columns are the word indices ordered most- to least-characteristic for each topic.

Usage

calcfrex(logbeta, w = 0.5, wordcounts = NULL)

calclift(logbeta, wordcounts)

calcscore(logbeta)

Arguments

Value

A V x K matrix of ordered word indices.

Description

Multinomial residual dispersion (Taddy 2012; port of stm::checkResiduals). A dispersion well above 1 suggests too few topics.

Usage

check_residuals(model, tol = 0.01)

Arguments

Value

A list with dispersion, pvalue, and df.

Description

Port of stm:::checkBeta: finds (topic, word) cells whose exp(logbeta) exceeds 1 - tolerance — words that are nearly exclusive to a single topic, which can destabilize estimation.

Usage

checkBeta(stmobject, tolerance = 0.01)

Arguments

Value

A list with problemTopics, problemWords, and error counts per content group.

Description

Coherence scores for each topic's top-M words, computed from the fit's stored document-term matrix. "mimno" is the UMass-style score of semantic_coherence(); "npmi" averages pairwise normalized PMI; "c_v" is the Roeder et al. (2015) measure (one-set segmentation, NPMI confirmation, cosine aggregation). NPMI/c_v use document co-occurrence as the probability estimator. Higher is more coherent (npmi/c_v are roughly in -1, 1).

Usage

coherence(model, measure = c("mimno", "npmi", "c_v"), M = 10L)

Arguments

Value

A numeric vector, one coherence score per topic.

Description

A balanced sample of 1,679 floor speeches from the U.S. House and Senate, Congresses 100-111 (1987-2011). Speeches are sampled evenly across party x chamber x congress so covariate effects are estimable, then lowercased and pruned of stop words and rare terms. Metadata: party (Democrat/Republican), chamber (House/Senate), congress (100-111). Built to showcase multiple (crossed) content covariates and over-time prevalence.

Usage

congress

Format

A faSTM_corpus with 1,679 documents and a 4,110-term vocabulary.

Source

Congressional Record, Hein-bound edition (Gentzkow, Shapiro & Taddy), congresses 100-111. The underlying floor speeches are U.S. government works (public domain).

Examples

data(congress)
fit <- stm(congress, K = 12, prevalence = ~ party + s(congress),
           content = ~ party + chamber)

Description

For a multi-covariate (crossed) content model, recovers the topic-word labels for each level of a single content covariate, averaging the crossed topic-word distributions over the other covariate(s). Lets you read off how topics' vocabulary shifts with one covariate while marginalizing the rest.

Usage

content_topics(model, by = NULL, n = 7L, type = c("prob", "lift", "frex"))

Arguments

Value

A named list (one entry per level of by) of K x n word matrices.

Description

Port of stm:::convertCorpus. "Matrix" returns a documents x V sparse dgCMatrix; "lda" returns the documents list (the lda/stm format).

Usage

convertCorpus(documents, vocab, type = c("Matrix", "lda", "slam"))

Arguments

Value

The corpus in the requested format.

Description

Returns the point estimates, standard errors and confidence bounds that plot.faSTM_effect() would draw, so you can build a custom plot or table (stm issue #83). Same arguments as the plot method.

Usage

effect_estimates(
  x,
  covariate,
  method = c("pointestimate", "continuous", "difference"),
  topics = x$topics,
  cov.value1 = NULL,
  cov.value2 = NULL,
  values = NULL,
  moderator = NULL,
  moderator.value = NULL,
  npoints = 50L,
  ci = 0.95
)

Arguments

Value

A data.frame with topic, value, est, se, lower, upper.

Description

A drop-in for stm::estimateEffect() that propagates per-document topic-estimation uncertainty: it regresses each posterior draw of topic proportions on the covariates and pools the per-draw fits by Rubin's rules. Propagating that uncertainty is the reason faSTM ships its own estimator rather than inheriting stm's.

Usage

estimateEffect(
  formula,
  stmobj,
  metadata = meta,
  uncertainty = c("Global", "None", "Local"),
  nsims = 100L,
  seed = NULL,
  meta = NULL,
  documents = NULL,
  combine = NULL,
  weights = NULL,
  cluster = NULL,
  ...
)

Arguments

Value

An object of class c("faSTM_effect", "estimateEffect") with a summary() method, holding pooled coefficients and standard errors per topic.

Description

Evaluate held-out log-likelihood of a fit on a held-out set

Usage

eval_heldout(model, heldout)

Arguments

Value

Mean per-document held-out log-likelihood per token.

Description

Topic exclusivity (FREX-summary, frexw default 0.7)

Usage

exclusivity(model, M = 10L, frexw = 0.7)

Arguments

Value

A numeric vector, one exclusivity value per topic.

Description

Representative documents for each topic

Usage

find_thoughts(model, texts = NULL, topics = NULL, n = 3L)

Arguments

Value

A list with index (per-topic document indices) and, if texts is given, docs (the texts).

Description

Find topics whose top words include given words

Usage

find_topic(model, words, n = 20L, type = c("prob", "frex", "lift", "score"))

Arguments

Value

Integer vector of matching topics.

Description

Runs the variational E-step for each new document against the fitted model's fixed global parameters (topic-word matrix, prior mean and covariance), giving out-of-sample topic proportions (cf. stm::fitNewDocuments). The model's topics are held fixed; only each new document's proportions are estimated.

Usage

fit_new_documents(model, newdata)

Arguments

Value

A new-documents × K matrix of topic proportions.

Description

Inputs are pre-converted in R/stm.R:

• docs_flat / doc_lens: documents as one concatenated 0-based token-id stream plus per-document lengths (counts already expanded). Reassembled here into ⁠Vec<Vec<u32>>⁠, the shape fit_ctm wants.

• prevalence: row-major D×P design matrix flattened (NULL if none).

• content_groups: per-doc 0-based group id (NULL if none); num_groups.

Usage

fit_stm(
  docs_flat,
  doc_lens,
  num_types,
  num_topics,
  em_iters,
  em_tol,
  sigma_shrink,
  prevalence,
  num_features,
  content_groups,
  num_groups,
  init_spectral,
  init_beta,
  gamma_l1_alpha,
  diagonal,
  seed,
  inference,
  batch_size,
  tau,
  kappa,
  num_threads
)

Details

inference: "batch" -> fit_ctm (parity-validated). "svi" -> fit_ctm_svi once topica #231 PR B (STM-SVI) is in the pinned revision; the R layer gates the prevalence/content + svi combination until then.

Description

Drop-in for stm::fitNewDocuments(). Holds the fitted topics fixed and runs the variational E-step for each new document. Supports stm's prior modes and posterior return.

Usage

fitNewDocuments(
  model,
  documents,
  newData = NULL,
  origData = NULL,
  prevalence = NULL,
  betaIndex = NULL,
  prevalencePrior = c("Average", "Covariate", "None"),
  contentPrior = c("Covariate", "Average"),
  returnPosterior = FALSE,
  verbose = TRUE,
  ...
)

Arguments

Value

A theta matrix, or a posterior list when returnPosterior = TRUE.

Description

FREX balances word frequency and exclusivity (Bischof & Airoldi 2012; Roberts et al.). Unlike stm's labelTopics(), this returns the full numeric FREX matrix, not just the ranked words (addresses a long-standing stm request, bstewart/stm#265).

Usage

frex_scores(model, w = 0.5)

Arguments

Value

A topics × vocabulary matrix of FREX scores (columns named by vocab).

Description

For tidytext-style data: one row per (document, term) with a count.

Usage

from_tidy(data, document = "document", term = "term", count = "n", meta = NULL)

Arguments

Value

A faSTM_corpus.

Description

One-row model summary for a faSTM fit

Usage

## S3 method for class 'faSTM'
glance(x, ...)

Arguments

Value

A one-row data.frame.

Description

Out-of-sample topic inference: for each new document, run the variational E-step against fixed globals (β, μ, Σ⁻¹) and return θ. Documents are passed sparse — words are 0-based ids into the fitted model's vocabulary (out-of-vocabulary terms dropped by the R caller) with their counts, concatenated, plus per-document term counts doc_nterms.

Usage

infer_theta_new(
  beta_flat,
  num_topics,
  num_types,
  mu,
  siginv,
  words,
  counts,
  doc_nterms
)

Description

Label topics by top words (prob, FREX, lift, score)

Usage

label_topics(model, n = 7L, frexweight = 0.5)

Arguments

Value

A faSTM_labels object: per-metric top-word matrices (prob, frex, lift, score), each topics × n.

Description

LDA topic-word matrix via topica's CVB0 (deterministic collapsed variational Bayes), to seed a "replicate stm's LDA init" STM fit. Mirrors stm's collapsed-Gibbs LDA initialization; the result is fed back as init_beta. Returns K*V row-major topic-word probabilities.

Usage

lda_init_beta(
  docs_flat,
  doc_lens,
  num_types,
  num_topics,
  iters,
  alpha,
  beta,
  seed
)

Description

Document-topic proportions as a data frame

Usage

make_dt(model, meta = NULL)

Arguments

Value

A data.frame with document and Topic1..TopicK columns (+ meta).

Description

Create a held-out version of a corpus for document-completion validation

Usage

make_heldout(
  corpus,
  N = floor(0.1 * length(corpus$documents)),
  proportion = 0.5,
  seed = NULL
)

Arguments

Value

A list with corpus (training corpus, held-out tokens removed) and missing (per-document held-out terms + counts), class faSTM_heldout.

Description

Port of stm:::makeDesignMatrix: builds the model matrix for newData using the term structure and factor levels of origData.

Usage

makeDesignMatrix(formula, origData, newData, sparse = TRUE, ...)

Arguments

Value

A (sparse) design matrix.

Description

Runs select_model() for each K and returns the chosen model per K.

Usage

many_topics(
  corpus,
  K,
  N = 10L,
  prevalence = NULL,
  content = NULL,
  by = "sum",
  cores = 1L,
  seed = 1L,
  ...
)

Arguments

Value

A faSTM_manytopics: models (best per K) and a summary data.frame.

Description

Aligns every model from a select_model() run to the first and reports how stable each topic's top words are across runs (cf. stm::multiSTM).

Usage

multi_stm(x, n = 10L)

Arguments

Value

A faSTM_multistm with a per-topic mean top-word agreement.

Description

Port of stm:::optimizeDocument's interface: infers one document's topic proportions against fixed globals and returns its variational mean lambda (eta), Laplace covariance nu, and theta.

Usage

optimizeDocument(document, eta, mu, beta, sigma = NULL, sigmainv = NULL, ...)

Arguments

Value

A list with lambda, nu, and theta.

Description

Refits the model many times with the treatment labels permuted, aligning topics across refits, to build a null distribution for the treatment effect on each topic (cf. stm::permutationTest). Fast because each refit is cheap.

Usage

permutation_test(
  formula,
  model,
  treatment,
  corpus,
  nruns = 100L,
  seed = NULL,
  ...
)

Arguments

Value

A faSTM_permtest with ref (observed per-topic effects) and null ((nruns-1) × K permuted effects).

Description

Nodes are topics (sized by prevalence, labelled by top words); edges join topics whose proportions are positively correlated above cutoff. Uses a lightweight circular layout — no graph-library dependency.

Usage

plot_topic_network(model, cutoff = 0.03, n = 3L, labeltype = "frex")

Arguments

Value

A ggplot object.

Description

Plot a fitted model

Usage

## S3 method for class 'faSTM'
plot(
  x,
  type = c("summary", "labels", "perspectives", "hist"),
  topics = NULL,
  n = 5L,
  labeltype = "frex",
  ...
)

Arguments

Value

A ggplot object.

Description

Plot estimated covariate effects on topic prevalence

Usage

## S3 method for class 'faSTM_effect'
plot(
  x,
  covariate,
  method = c("pointestimate", "continuous", "difference"),
  topics = x$topics,
  model = NULL,
  cov.value1 = NULL,
  cov.value2 = NULL,
  values = NULL,
  moderator = NULL,
  moderator.value = NULL,
  npoints = 50L,
  ci = 0.95,
  labeltype = NULL,
  custom.labels = NULL,
  xlab = NULL,
  main = NULL,
  ...
)

Arguments

Value

A ggplot object.

Description

Faceted held-out likelihood, semantic coherence, exclusivity and bound vs K.

Usage

## S3 method for class 'faSTM_searchk'
plot(x, ...)

Arguments

Value

A ggplot object.

Description

5,000 political blog posts from the 2008 U.S. election (the stm vignette example) as a ready-to-use faSTM_corpus. Metadata: rating (Conservative/Liberal), day (1-365), blog, text.

Usage

data(poliblog)

Format

A faSTM_corpus: 5,000 documents, 2,632-term vocabulary.

Source

Eisenstein & Xing (2010), via the stm package.

Description

The variational (Laplace) posterior of each document's logit-topic vector is eta_d ~ N(lambda_d, nu_d), both stored on a faSTM fit. This draws nsims samples of theta per document by sampling eta and applying the softmax (with the reference topic appended as 0). This is the pure-R equivalent of topica's posterior_theta_samples; no Rust call is needed because eta + nu fully describe the posterior. Feeds estimateEffect()'s method of composition.

Usage

posterior_theta_samples(model, nsims = 100L, seed = NULL)

Arguments

Value

A nsims-length list of ⁠D x K⁠ theta matrices.

Description

Predict topic proportions for new documents

Usage

## S3 method for class 'faSTM'
predict(object, newdata, ...)

Arguments

Value

A new-documents x K matrix of topic proportions.

Description

Each line is ⁠M term:count term:count ...⁠ with 0-based term ids.

Usage

read_ldac(file)

write_ldac(documents, file)

Arguments

Value

read_ldac returns a list of 2×n integer matrices (faSTM/stm document format, 1-based ids); write_ldac returns the path invisibly.

Description

A b-spline basis for smooth covariate effects, e.g. prevalence = ~ s(day). Matches stm::s() exactly — including the df = min(10, nval - 1) default — so spline-term coefficients agree with stm. (You can also use splines::bs()/splines::ns() directly.)

Usage

s(x, df, ...)

Arguments

Value

A spline basis matrix (with class "s").

Description

For models fit with a content covariate, reports each topic's marginal top words plus, for every content group, the words most distinctive to that group within the topic (group-vs-marginal log-ratio — the SAGE deviation).

Usage

sage_labels(model, n = 7L, frexweight = NULL)

Arguments

Value

A faSTM_sagelabels object.

Description

Fits the model across a range of K and reports diagnostics for choosing it: held-out likelihood (document completion), semantic coherence, exclusivity, and the variational bound. Unlike stm::searchK, the per-K fits parallelize across K (a long-standing request, bstewart/stm#262) and each fit is itself fast (Rust), so a sweep that took minutes takes seconds.

Usage

search_k(
  corpus,
  K,
  prevalence = NULL,
  content = NULL,
  heldout = TRUE,
  proportion = 0.5,
  residuals = FALSE,
  cores = 1L,
  M = 10L,
  seed = 1L,
  measure = c("mimno", "npmi", "c_v"),
  verbose = FALSE,
  ...
)

Arguments

Value

A faSTM_searchk object wrapping a tidy data.frame results with one row per K (K, heldout, semcoh, exclusivity, bound).

Description

Pick one model from a select_model run

Usage

select_best(x, by = c("sum", "semcoh", "exclusivity"))

Arguments

Value

A single faSTM fit.

Description

With random initialization the variational objective is multimodal, so the standard workflow (cf. stm::selectModel) is to fit many models and keep those on the semantic-coherence / exclusivity frontier, then choose among them. faSTM fits the candidates in parallel.

Usage

select_model(
  corpus,
  K,
  N = 10L,
  prevalence = NULL,
  content = NULL,
  init.type = "Random",
  cores = 1L,
  M = 10L,
  frexw = 0.7,
  seed = 1L,
  ...
)

Arguments

Value

A faSTM_selectmodel: models (the fits), semcoh, exclusivity, and frontier (indices of non-dominated models).

Description

Sum over the top-M words of each topic of log((D(w_i,w_j)+1)/D(w_j)), using document co-occurrence counts. Higher (less negative) is more coherent.

Usage

semantic_coherence(model, M = 10L)

Arguments

Value

A numeric vector, one coherence value per topic.

Description

A drop-in replacement for stm::stm()'s fitting step. Accepts the same documents / vocab / prevalence / content inputs, fits with topica's Rust core, and returns an object compatible with the stm package so that stm::labelTopics(), stm::plot.STM(), stm::findThoughts(), stm::sageLabels(), and stm::toLDAvis() work unmodified. Use estimateEffect() from this package for covariate effects that propagate topic-estimation uncertainty.

Usage

stm(
  documents,
  vocab,
  K,
  prevalence = NULL,
  content = NULL,
  data = NULL,
  max.em.its = 500L,
  emtol = 1e-05,
  init.type = c("Spectral", "Random", "LDA", "Custom"),
  init.beta = NULL,
  model = NULL,
  gamma.prior = c("Pooled", "L1"),
  gamma.l1.alpha = 0.001,
  sigma.prior = 0,
  seed = 1L,
  inference = c("batch", "svi"),
  batch_size = 256L,
  tau = 64,
  kappa = 0.7,
  num_threads = 0L,
  verbose = TRUE,
  ...
)

Arguments

Value

An object of class c("faSTM", "STM") — an stm-compatible fit.

Description

Tidy a faSTM fit (topic-term or document-topic distributions)

Usage

## S3 method for class 'faSTM'
tidy(x, matrix = c("beta", "gamma", "frex"), ...)

Arguments

Value

A tidy data.frame.

Description

Tidy an estimateEffect fit (one row per term per topic)

Usage

## S3 method for class 'faSTM_effect'
tidy(x, ...)

Arguments

Value

A data.frame: topic, term, estimate, std.error, statistic, p.value.

Description

Exports the positive-correlation topic network as an igraph object (stm issue #242), with topic prevalence and FREX labels as vertex attributes and the positive correlations as edge weights — ready for igraph/ggraph layouts.

Usage

topic_corr_graph(x, model = NULL, nlabel = 3L)

Arguments

Value

An undirected igraph graph.

Description

Topic correlation graph (positive correlations of topic proportions)

Usage

topic_correlation(model, cutoff = 0.01)

Arguments

Value

A list with cor (the K×K correlation matrix) and posadj (the thresholded positive adjacency).

Description

Cross-validated lasso (glmnet) of an outcome on the topic-proportion matrix (cf. stm::topicLasso). Identifies which topics predict the outcome.

Usage

topic_lasso(
  formula,
  model,
  data,
  family = "gaussian",
  nfolds = 10L,
  seed = 2138L,
  ...
)

Arguments

Value

A faSTM_topiclasso with selected per-topic coefficients.

Description

Returns the corpus-level expected topic proportions — the mean of theta per topic — as a numeric table, so you can read off the values stm's plot(type = "summary") displays (stm issue #269).

Usage

topic_proportions(model, nlabel = 3L)

Arguments

Value

A data.frame with topic, proportion, label, sorted by proportion.

Description

Like label_topics() but returns the values behind the ranking, not just the words — e.g. the numeric FREX score per top term (stm issue #265).

Usage

topic_terms(
  model,
  n = 7L,
  by = c("prob", "frex", "lift", "score"),
  frexweight = 0.5
)

Arguments

Value

A tidy data.frame with topic, rank, term, score, measure.