LexVars is a Python package for generating lexical predictors for psycholinguistic studies, with a focus on morphological and contextual variables. This tutorial showcases much of the functionality of the package; please read the individual functions' docstring for comprehensive documentation.
CELEX is a lexical database that provides rich morphological and syntactic information (Baayen & Piepenbrock, 1995). LexVars provides a convenient interface to access two CELEX databases (for English only): the lemma database, which collapses different inflected forms (e.g., running and runs), and the wordform database, which has a seperate entry for each form. Let's start by looking at the lemmas. A particular string, e.g. wind, is often mapped to multiple lemmas, which correspond to the word's possible parts of speech or meanings:
import lexvars.celex
c = lexvars.celex.Celex(celex_path)
wind = c.lemma_lookup('wind')
wind
To see a complex word's morphological decomposition:
lightbulb = c.lemma_lookup('lightbulb')
lightbulb[0].Parses
CELEX includes a wide range of annotations for each word. Let's list the first three (ordered alphabetically), and check out the description of one of them:
dir(lightbulb[0])[:3]
lightbulb[0].help('Attr_A')
Clearly, lightbulb is not an adjective that can only be used attributively:
lightbulb[0].Attr_A
In contrast with lemmas, wordforms are annotated for their inflectional status:
windows = c.wordform_lookup('windows')
windows[0].FlectType
CELEX includes the frequency of each wordform in the COBUILD corpus:
windows[0].Cob
Let's list all of the wordforms that are associated with the verb lemma for build. The same string often has multiple inflectional analyses, and therefore multiple wordforms; for example, built can be a past form or a participle.
wfs = c.lemma_to_wordforms(c.lemma_lookup('build')[1])
wfs
wfs[-1].FlectType
wfs[-2].FlectType
The derivational family of a lemma consists of all of the words that include have that lemma as a morpheme. There are multiple ways to define this family, and each are useful for a different task. Adding the flag right=True accesses the word's right derivational family, which only includes words that have the lemma as their leftmost morpheme. By default, multiword lemmas such as think up are not included in the morphological family, but can be added to it using the flag multiword=True.
import lexvars.lexvars
lv = lexvars.lexvars.LexVars(c)
think_family = lv.derivational_family('think')
think_family
lv.derivational_family('think', right=True)
lv.derivational_family('think', right=True, include_multiword=True)
We can now extract the lemma frequencies of each of the lemmas in the word's family, and calculate the entropy of the probability distribution defined by those frequencies (Moscoso del Prado Martín et al., 2004). The lemma think is much more frequent than all of the lemmas derived from it; its derivational entropy is therefore fairly low:
think_family[0].help('Cob')
[x.Cob for x in think_family]
lv.derivational_entropy('think')
Finally, we can calculate the entropy of the distribution of inflected forms of think, collapsing over all of the lemmas for think. There are multiple possible ways to group together inflected forms of the lemma for the purposes of calculating entropy; see the function's documentation for details.
lv.inflectional_entropy('think')
By analogy to the derivational family of a stem, we can define a verb's subcategorization family as the set of frames that a verb can occur in (Linzen et al., 2013). LexVars provides an interface to VALEX (Korhonen et al., 2006). Let's load their lexicon 5 and examine three of the frames for squash:
import lexvars.valex
vlx = lexvars.valex.Valex(valex_lex5_path)
vlx.load_all_verbs(progress=False)
vlx.verbs['squash'][:3]
We can also calcualte the Kullback-Leibler divergence between individual verbs' subcategorization distribution and the average subcategorization distribution in the language (i.e., averaged across all verbs, weighted by the verbs' frequency; see again Linzen et al., 2013).
vre = ValexRelativeEntropy(c, vlx)
vre.build_reference_distribution()
vre.calculate_relative_entropies()
vre.relative_entropies['squash']
Baayen, R. H., & Piepenbrock, R. (1995). The CELEX lexical database (Release 2) [CD-ROM]. Philadelphia, PA: Linguistic Data Consortium, University of Pennsylvania [Distributor].
Korhonen, A., Krymolowski, Y., & Briscoe, T. (2006). A large subcategorization lexicon for natural language processing applications. In Proceedings of the 5th international conference on language resources and evaluation. Genova, Italy.
Linzen, T. Marantz, A., & Pylkkanen, L. Syntactic context effects in visual word recognition: An MEG study. The Mental Lexicon 8(2), 117-139.
Moscoso del Prado Martín, F. M., Kostić, A., & Baayen, R. H. (2004). Putting the bits together: An information theoretical perspective on morphological processing. Cognition, 94(1), 1-18.