Corpus Analysis and Visualization

10:30 - 12:00

• Recap: Frequency Based Analysis and the DTM
• Dictionary Analysis with AmCAT and R

13:30 - 15:00

• Simple Natural Language Processing
• Corpus Analysis and Visualization
• Topic Modeling and Visualization

15:15 - 17:00

• Sentiment Analysis with dictionaries
• Sentiment Analysis with proximity

• Preprocess documents to get more information
• Relatively fast and accurate
  • Lemmatizing
  • Part-of-Speech (POS) tagging
  • Named Entity Recognition
• Unfortunately, not within R

• nlpipe: simple NLP processing based on stanford corenlp, others

docker run --name corenlp -dp 9000:9000 chilland/corenlp-docker

docker run --name nlpipe --link corenlp:corenlp -e "CORENLP_HOST=http://corenlp:9000" -dp 5001:5001 vanatteveldt/nlpipe

devtools::install_github("vanatteveldt/nlpiper")

library(nlpiper)
process("test_upper", "test")

[1] "TEST"

library(nlpiper)
text = "Donald trump was elected president of the United States"
process("corenlp_lemmatize", text, format="csv")

• Exploratory Analysis
• Term statistics
• Corpus comparison

The corpustools package

• Useful functions for corpus analysis
• Not fully integrated with quanteda
  • (we're working on it :) )

devtools::install_github("kasperwelbers/corpus-tools")

• quanteda uses 'dfm' objects
  • document-feature matrix
• tm uses 'dtm' objects
  • document-term matrix
• Both are sparse matrices

library(quanteda)
dfm = dfm(c("a text", "and another text"))
dtm = convert(dfm, "tm")
class(dtm)

[1] "DocumentTermMatrix"    "simple_triplet_matrix"

library(corpustools)
dfm = dtm.to.dfm(dtm)
class(dfm)

[1] "dfmSparse"
attr(,"package")
[1] "quanteda"

dtm = dtm.create(tokens$id, tokens$lemma)
dtm.names = with(subset(tokens, POS1=="R"), 
                 dtm.create(id, lemma))
dtm.adj = with(subset(tokens, POS1=="G"), 
               dtm.create(id, lemma))
dtm.persons = with(subset(tokens, ner=="PERSON"), 
                   dtm.create(id, lemma))

stats = term.statistics(dtm.persons)
stats = plyr::arrange(stats, -termfreq)
Head(stats)

meta = readRDS("meta.rds")
nyt = meta$medium == "The New York Times"
nyt1 = meta$id[nyt & meta$date < "2016-08-01"]
nyt2 = meta$id[nyt & meta$date >= "2016-08-01"]
dtm1 = with(subset(tokens, id %in% nyt1 & POS1=="G"), 
            dtm.create(id, lemma))
dtm2 = with(subset(tokens, id %in% nyt2 & POS1=="G"), 
            dtm.create(id, lemma))
cmp = corpora.compare(dtm1, dtm2)
cmp = plyr::arrange(cmp, -chi)
Head(cmp)

• Word clouds waste most information
• corpustools::plotWords
  • specify x, y, colour, size, etc.
• Use any analytics you have to determine characteristics
• See also http://vanatteveldt.com/lse-text-visualization/

dtm = with(subset(tokens, POS1 %in% c("G", "N", "V")), 
                 dtm.create(id, lemma, minfreq = 10))
dtm = dtm[, !colnames(dtm) %in% tm::stopwords()]
set.seed(1234)
m = lda.fit(dtm, K = 10, num.iterations = 50, alpha=0.5)
Head(terms(m, 10))

compare.topics <- function(m, cmp_topics) {
  docs = factor(m@wordassignments$i, labels=m@documents)
  terms = factor(m@wordassignments$j, labels=m@terms)
  assignments = data.frame(doc=docs, term=terms, freq=m@wordassignments$v)
  terms = dcast(assignments, term ~ freq, value.var = "doc", fun.aggregate = length)
  terms = terms[, c(1, cmp_topics+1)]
  terms$freq = rowSums(terms[-1])
  terms = terms[terms$freq > 0,]
  terms$prop = terms[[2]] / terms$freq
  terms$col = hsv(rescale(terms$prop, c(1, .6666)), .5, .5)
  terms[order(-terms$freq), ]
}

tpd = topics.per.document(m)
colnames(tpd)[-1] = topics
tpd = merge(meta, tpd)
Head(tpd)

• Corpus analysis of Election campaign
  • (or your own data…)
• Which words, adjectives, verbs, etc are frequent?
• How do they differ over time, by medium, subcorpus
• What topics can we find?
• Can we visualize topics, contrasts, etc.