#LyX 2.0 created this file. For more info see http://www.lyx.org/
\lyxformat 413
\begin_document
\begin_header
\textclass article
\begin_preamble
\usepackage{url}
\end_preamble
\use_default_options false
\maintain_unincluded_children false
\language english
\language_package default
\inputencoding utf8
\fontencoding global
\font_roman times
\font_sans helvet
\font_typewriter courier
\font_default_family default
\use_non_tex_fonts false
\font_sc false
\font_osf false
\font_sf_scale 100
\font_tt_scale 100
\graphics default
\default_output_format default
\output_sync 0
\bibtex_command default
\index_command default
\paperfontsize default
\spacing single
\use_hyperref true
\pdf_bookmarks true
\pdf_bookmarksnumbered false
\pdf_bookmarksopen false
\pdf_bookmarksopenlevel 1
\pdf_breaklinks true
\pdf_pdfborder true
\pdf_colorlinks true
\pdf_backref false
\pdf_pdfusetitle true
\papersize default
\use_geometry false
\use_amsmath 2
\use_esint 0
\use_mhchem 0
\use_mathdots 1
\cite_engine basic
\use_bibtopic false
\use_indices false
\paperorientation portrait
\suppress_date false
\use_refstyle 0
\index Index
\shortcut idx
\color #008000
\end_index
\secnumdepth 3
\tocdepth 3
\paragraph_separation indent
\paragraph_indentation default
\quotes_language english
\papercolumns 1
\papersides 1
\paperpagestyle default
\tracking_changes false
\output_changes false
\html_math_output 0
\html_css_as_file 0
\html_be_strict false
\end_header
\begin_body
\begin_layout Title
Connector Sets
\end_layout
\begin_layout Author
Linas Vepstas
\end_layout
\begin_layout Abstract
Extract from the language-learning diary, reporting on the first small dataset
containing connector sets.
This is the 11 June 2017 update of the original 11 May 2017 report.
It includes more data and figures, and updates the figures for readability
(legibility).
It also revises notation slightly.
\end_layout
\begin_layout Section*
Introduction
\end_layout
\begin_layout Standard
This is a report on a small dataset of disjuncts and connector sets, extracted
from MST parses of a batch of sentences.
First, a recap of what these are, then a characterization of the database
contents, and finally, a short report on the grammatical similarity of
words in the dataset.
\end_layout
\begin_layout Subsection*
Summary of results
\end_layout
\begin_layout Standard
The primary results reported below are these:
\end_layout
\begin_layout Standard
* Most scores and metrics that can be assigned to connector sets give a
(scale-free) Zipfian ranking distribution, and are thus fairly boring.
\end_layout
\begin_layout Standard
* The greater the average number of observations per disjunct, the more
grammatically acceptable (accurate) the disjunct seems to be.
This is good news: it means that the general technique is not generating
ungrammatical garbage.
\end_layout
\begin_layout Standard
* Connector sets can be given a mutual information score.
The ranking distribution for this is not at all Zipfian.
The MI score seems to be quite good at identifying words that participate
in idioms, set phrases and institutional phrases.
\end_layout
\begin_layout Standard
* The average number of connectors per disjunct, which should have indicated
the part-of-speech that the word belongs to, fails to do this.
This seems to be due to the small size of the dataset, the fact that it
is polluted with lists and tables, masquerading as sentences (its a Wikipedia
sample), and that there seem the be very few verbs in the sample (Wikipedia
articles describe concepts and events, using the copula to describe them:
\begin_inset Quotes eld
\end_inset
is
\begin_inset Quotes erd
\end_inset
,
\begin_inset Quotes eld
\end_inset
has
\begin_inset Quotes erd
\end_inset
,
\begin_inset Quotes eld
\end_inset
was
\begin_inset Quotes erd
\end_inset
, and is almost devoid of narrative verbs:
\begin_inset Quotes eld
\end_inset
ran
\begin_inset Quotes erd
\end_inset
\begin_inset Quotes eld
\end_inset
jumped
\begin_inset Quotes erd
\end_inset
\begin_inset Quotes eld
\end_inset
hit
\begin_inset Quotes erd
\end_inset
,
\begin_inset Quotes eld
\end_inset
ate
\begin_inset Quotes erd
\end_inset
\begin_inset Quotes eld
\end_inset
thought
\begin_inset Quotes erd
\end_inset
\begin_inset Quotes eld
\end_inset
took
\begin_inset Quotes erd
\end_inset
.) The WP sources need to be supplemented with narrative texts, ideally from
young-adult literature.
\end_layout
\begin_layout Standard
* Cosine similarity applied to connector sets seems to be an effective way
of determining the grammatical similarity of words.
This is despite the fact that closer examination reveals that the dataset
is quite thin and poor, and the similarity is based on scant evidence.
Despite the scant evidence, the similarity scores do seem to distinguish
different kinds of nouns quite well.
A big surprise is that it also seems to cluster prepositions quite well.
There seem to be few or no verbs in the dataset, as already noted.
It seems that even small datasets are sufficient to get started with grammatica
l similarity clustering, but much larger samples are needed for discovering
more sophisticated grammar.
\end_layout
\begin_layout Subsection*
Recap
\end_layout
\begin_layout Standard
The story so far: Starting from a large text corpus, the mutual information
(MI) of word-pairs are counted.
This MI is used to perform a maximum spanning-tree (MST) parse (of a different
subset of) the corpus.
From each parse, a pseudo-disjunct is extracted for each word.
The pseudo-disjunct is like a real LG disjunct, except that each connector
in the disjunct is the word at the far end of the link.
\end_layout
\begin_layout Standard
So, for example, in in idealized world, the MST parse of the sentence "Ben
ate pizza" would produce the parse Ben <--> ate <--> pizza and from this,
we can extract the pseudo-disjunct (Ben- pizza+) on the word "ate".
Similarly, the sentence "Ben puked pizza" should produce the disjunct (Ben-
pizza+) on the word "puked".
Since these two disjuncts are the same, we can conclude that the two words
"ate" and "puked" are very similar to each other.
Considering all of the other disjuncts that arise in this example, we can
conclude that these are the only two words that are similar.
\end_layout
\begin_layout Standard
Any given word will have many pseudo-disjuncts attached to it.
Each disjunct has a count of the number of times it has been observed.
Thus, this set of disjuncts can be imagined to be a vector in a high-dimensiona
l vector space, which each disjunct being a single basis element.
The similarity of two words can be taken to be the cosine-similarity between
the disjunct-vectors.
\end_layout
\begin_layout Standard
Equivalently, the set of disjuncts can be thought of as a weighted set:
each disjunct has a weight, corresponding to the number of times it has
been observed.
A weighted set is more or less the same thing as a vector, and these two
are treated as the same, in what follows.
Note that the disjunct vectors are sparse: for any given word, almost all
coefficients will have a count of zero.
For example, the dataset that will be examined next has over a quarter
of a million different pseudo-disjuncts in it; most words have fewer than
a hundred disjuncts on them.
\end_layout
\begin_layout Standard
Some terminology and notation are introduced next, followed by a characterizatio
n of the dataset.
This is followed by a statistical analysis of the word-disjunct pairs,
and is followed by an analysis of the resulting word-similarity.
\end_layout
\begin_layout Subsection*
Terminology
\end_layout
\begin_layout Standard
It is useful to introduce some notation for counting words, disjuncts, and
connectors.
Let
\begin_inset Formula $N(w)$
\end_inset
be the number of times that the word
\begin_inset Formula $w$
\end_inset
has been observed, in the dataset.
Let
\begin_inset Formula $N(w,d)$
\end_inset
be the number of times that the disjunct
\begin_inset Formula $d$
\end_inset
has been observed on word
\begin_inset Formula $w$
\end_inset
.
The pair
\begin_inset Formula $(w,d)$
\end_inset
is referred to as a
\begin_inset Quotes eld
\end_inset
connector set
\begin_inset Quotes erd
\end_inset
or
\begin_inset Quotes eld
\end_inset
cset
\begin_inset Quotes erd
\end_inset
in the text below.
Thus, for a word
\begin_inset Formula $w$
\end_inset
, there is a set
\begin_inset Formula $(w,*)=\left\{ (w,d)|N(w,d)>0\right\} $
\end_inset
of associated csets, called the
\begin_inset Quotes eld
\end_inset
support
\begin_inset Quotes erd
\end_inset
of the word.
The size of this set can be written using the standard notation for set-sizes
as
\begin_inset Formula $\left|(w,*)\right|$
\end_inset
.
Similarly, a disjunct
\begin_inset Formula $d$
\end_inset
, is supported by the set
\begin_inset Formula $(*,d)=\left\{ (w,d)|N(w,d)>0\right\} $
\end_inset
of associated csets.
\end_layout
\begin_layout Standard
The primary contents of the database are the counts
\begin_inset Formula $N(w,d)$
\end_inset
and everything else of interest in this section can be obtained from this.
Note that
\begin_inset Formula $N(w,d)$
\end_inset
can be understood as a matrix, where the disjuncts identify columns, and
the words identify rows.
In general, this is a very sparse matrix: the number of non-zero entries
\begin_inset Formula $\left|(*,*)\right|$
\end_inset
is far less than the number of rows times the number of columns.
\end_layout
\begin_layout Standard
Every time a word is observed in an MST parse, a disjunct is extracted for
it; thus, word observations and disjunct observations are on one-to-one
correspondence.
In notation:
\begin_inset Formula
\[
\sum_{d}N(w,d)=N(w,*)=N(w)
\]
\end_inset
Similarly, the total number of times that a disjunct was observed is just
\begin_inset Formula
\[
N(*,d)=\sum_{w}N(w,d)
\]
\end_inset
\end_layout
\begin_layout Standard
Frequencies can be obtained by dividing by the total number of observations,
so that
\begin_inset Formula $p(w,d)=N(w,d)/N(*)$
\end_inset
and
\begin_inset Formula $p(w)=N(w)/N(*)$
\end_inset
with
\begin_inset Formula $N(*)=\sum_{w}N(w)$
\end_inset
the total number of observations of words.
\end_layout
\begin_layout Standard
A single disjunct is always composed of a fixed number of connectors, independen
tly of any observations; let
\begin_inset Formula $C(d,c)$
\end_inset
be the number of times that connector
\begin_inset Formula $c$
\end_inset
appears in disjunct
\begin_inset Formula $d$
\end_inset
.
Note that
\begin_inset Formula $C(d,c)$
\end_inset
is almost always either zero or one; however, a connector can appear more
than once in a disjunct, so this count can rise to 2 or 3 or very rarely
higher.
The wild-card sum
\begin_inset Formula $C(d,*)=\sum_{c}C(d,c)$
\end_inset
is the total number of connectors in the disjunct; it is the vertex degree
of all edges connecting to that disjunct.
It is also useful to define
\begin_inset Formula $C(d,+)$
\end_inset
and
\begin_inset Formula $C(d,-)$
\end_inset
as the total number of right-linking and left-linking connectors.
\end_layout
\begin_layout Subsection*
Dataset characterization
\end_layout
\begin_layout Standard
The following charts and analyses are derived from a single dataset, a relativel
y small dataset, taken as a snapshot during counting.
Its called 'en_pairs_sim'.
It contains 37413 words that have disjuncts attached to them.
These words have been observed a total of 661104 times, for an average
of 661104/37413 = 17.6 observations per word.
This dataset contains 291637 different, unique disjuncts, for an average
of 661104/291637 = 2.27 observations per disjunct.
The period appears 32536 times, suggesting that this many sentences were
observed.
Each sentence thus has an average of 661104/32536 = 20.3 words per sentence.
The dataset contains 446204 unique connector-sets, for an average of 661104/446
204 = 1.48 observations per cset.
It is this last number that makes this dataset feel thin and sparse.
\end_layout
\begin_layout Standard
The dataset is sparse in a completely different sense: viewing
\begin_inset Formula $N(w,d)$
\end_inset
as a matrix whose size is
\begin_inset Formula $37413\times291637$
\end_inset
, but only a very small number of these is non-zero: this is
\begin_inset Formula $446204/(37413\times291637)=4.089\times10^{-5}$
\end_inset
.
The sparsity of this matrix can be defined as
\begin_inset Formula $-\log_{2}$
\end_inset
of this number, which is 14.58.
\end_layout
\begin_layout Standard
The total word-entropy for the dataset is defined as
\begin_inset Formula
\[
H_{word}=-\sum_{w}p(w)\log_{2}p(w)
\]
\end_inset
and was measured to be
\begin_inset Formula $H_{word}=10.28$
\end_inset
bits.
\begin_inset Foot
status collapsed
\begin_layout Plain Layout
This and the following entropies were measured with the word-entropy-bits,
disjunct-entropy-bits, etc.
functions in disjunct-stats.scm
\end_layout
\end_inset
The connector-set entropy is much higher.
It is defined as
\begin_inset Formula
\[
H_{cset}=-\sum_{w,d}p(w,d)\log_{2}p(w,d)
\]
\end_inset
and is measured to be
\begin_inset Formula $H_{cset}=18.30$
\end_inset
bits.
The disjunct entropy is dual to the word entropy:
\begin_inset Formula
\[
H_{disjunct}=-\sum_{d}p(*,d)\log_{2}p(*,d)
\]
\end_inset
and is measured to be
\begin_inset Formula $H_{disjunct}=16.01$
\end_inset
bits.
The total mutual information between the words and disjuncts is then
\begin_inset Formula
\[
MI_{cset}=-\sum_{w,d}p(w,d)\log_{2}\frac{p(w,d)}{p(*,d)p(w,*)}=H_{cset}-H_{word}-H_{disjunct}
\]
\end_inset
and is measured to be
\begin_inset Formula $MI_{cset}=-7.985$
\end_inset
bits.
\end_layout
\begin_layout Subsection*
Connector-set distribution
\end_layout
\begin_layout Standard
Some connector-sets will be observed far more often than others.
Likewise for the two sides of the connector-set: some words will have far
more observations, and some disjuncts will be seen more often.
\end_layout
\begin_layout Standard
Two graphs, dual to one-another.
The one on the left shows
\begin_inset Formula $N(w,*)$
\end_inset
, ranked by count.
The one on the right shows
\begin_inset Formula $N(*,d)$
\end_inset
, also ranked.
\begin_inset Foot
status collapsed
\begin_layout Plain Layout
Obtained by running (print-ts-rank sorted-word-obs outport) from the disjunct-st
ats.scm file, on the en_pairs_sim database.
The second one prints sorted-dj-obs.
\end_layout
\end_inset
.
The first follows the canonical Zipf distribution.
The green line is an eyeballed, approximate fit, of exponent -1.1.
The second has an exponent of -0.85.
\end_layout
\begin_layout Standard
\align center
\begin_inset Graphics
filename ../images/ranked-word-obs.eps
width 50text%
\end_inset
\begin_inset Graphics
filename ../images/ranked-dj-obs.eps
width 50text%
\end_inset
\end_layout
\begin_layout Standard
The first ten words in the word ranking are: "the" "," "." "of" "and" "in"
"to" "a" "was".
This is the ranking of how often these words appear, overall, in the MST-parsed
corpus.
The number of periods should be equal to the number of sentences in the
corpus; commas and the word
\begin_inset Quotes eld
\end_inset
the
\begin_inset Quotes erd
\end_inset
can appear more than once in a sentence.
This list is repeated in the table below.
The frequency is just the count divided by 661104.
\end_layout
\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="10" columns="4">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
word
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
count
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
frequency
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
\begin_inset Formula $-\log_{2}$
\end_inset
frequency
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
the
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
38977
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
0.0589
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
4.084
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
,
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
37524
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
0.0568
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
4.139
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
.
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
32536
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
0.0489
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
4.353
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
of
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
22654
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
0.0343
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
4.867
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
and
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
17708
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
0.0268
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
5.222
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
in
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
14900
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
0.0225
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
5.471
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
to
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
12825
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
0.0194
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
5.688
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
a
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
11882
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
0.0180
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
5.798
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
was
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
6970
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
0.0105
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
6.568
\end_layout
\end_inset
</cell>
</row>
</lyxtabular>
\end_inset
\end_layout
\begin_layout Standard
The main point of this table is to demonstrate the log-likelihood column.
At this point, these numbers won't seem to have much meaning; however,
they provide an overall scale that will be seen, repeatedly, in the analysis
below.
The range of magnitudes -- 4 to 7 -- is no accident, and similar ranges
will be seen later.
\end_layout
\begin_layout Standard
The first ten pseudo-disjuncts in the disjunct-ranking are ".+ " "the- "
"the+ " ",+ " ",- " "of+ " "of- " "a- " "and+ " "and- ".
The meaning of the plus and minus signs was explained above; but to recap:
the disjunct
\begin_inset Quotes eld
\end_inset
.+
\begin_inset Quotes erd
\end_inset
means that there are many words that expect to be followed by a period
(on the right).
The disjunct
\begin_inset Quotes eld
\end_inset
the-
\begin_inset Quotes erd
\end_inset
means that there are many words that want to link to the word
\begin_inset Quotes eld
\end_inset
the
\begin_inset Quotes erd
\end_inset
on the left.
This is grammatically correct:
\begin_inset Quotes eld
\end_inset
the
\begin_inset Quotes erd
\end_inset
is a determiner, and it is always the dependent of some noun.
The disjunct
\begin_inset Quotes eld
\end_inset
the+
\begin_inset Quotes erd
\end_inset
is grammatical garbage/nonsense: it states that there are many words that
want to link to the word
\begin_inset Quotes eld
\end_inset
the
\begin_inset Quotes erd
\end_inset
on the right.
This is never correct for English; determiners always precede the noun
that they modify.
The rest of the disjunct look reasonable: it's legitimate to link to commas
on both the left and right, and likewise to the preposition
\begin_inset Quotes eld
\end_inset
of
\begin_inset Quotes erd
\end_inset
and conjunction
\begin_inset Quotes eld
\end_inset
and
\begin_inset Quotes erd
\end_inset
.
The disjunct
\begin_inset Quotes eld
\end_inset
a-
\begin_inset Quotes erd
\end_inset
is correct, and the insane disjunct
\begin_inset Quotes eld
\end_inset
a+
\begin_inset Quotes erd
\end_inset
doesn't appear until the 14th slot.
\end_layout
\begin_layout Standard
It's not clear at this point why the insane disjuncts
\begin_inset Quotes eld
\end_inset
the+
\begin_inset Quotes erd
\end_inset
and
\begin_inset Quotes eld
\end_inset
a+
\begin_inset Quotes erd
\end_inset
are so highly ranked.
The hope is that these appear due to the poor quality of the dataset (as
will become more clear, below), rather than a deficiency in the theory.
Based on decades-old research on MST parsing, its reasonable to hope that
these disjuncts will disappear from a larger, better corpus.
\end_layout
\begin_layout Standard
The ranking of connector sets is shown below.
It's a graph of the ranked counts
\begin_inset Formula $N(w,d)$
\end_inset
.
\end_layout
\begin_layout Standard
\align center
\begin_inset Graphics
filename ../images/ranked-cset-obs.eps
width 70text%
\end_inset
\end_layout
\begin_layout Standard
The top-ten connector-sets are ".: )-" "and: ,-" "in: the+" "(: )+" ",: and+"
"the: of-" "of: the+" "): .+" "He: was+".
These are hard to read, so, decoded: the first is a parenthesis followed
by a period.
The second is a comma, followed by
\begin_inset Quotes eld
\end_inset
and
\begin_inset Quotes erd
\end_inset
.
The third is the word-pair
\begin_inset Quotes eld
\end_inset
in the
\begin_inset Quotes erd
\end_inset
.
All the others are pairs, also.
Written in order:
\begin_inset Quotes eld
\end_inset
()
\begin_inset Quotes erd
\end_inset
(presumably with other text between the parens)
\begin_inset Quotes eld
\end_inset
, and
\begin_inset Quotes erd
\end_inset
\begin_inset Quotes eld
\end_inset
of the
\begin_inset Quotes erd
\end_inset
\begin_inset Quotes eld
\end_inset
of the
\begin_inset Quotes erd
\end_inset
\begin_inset Quotes eld
\end_inset
) .
\begin_inset Quotes erd
\end_inset
and
\begin_inset Quotes eld
\end_inset
He was
\begin_inset Quotes erd
\end_inset
.
Curiously the 1st and 8th give the same pair, as do 2nd and 5th, and the
6th and 7th.
This makes sense: if one word-disjunct pair is observed a lot, the converse
should be also.
This duality might be behind the slope of slightly less than -0.6 in the
graph.
\end_layout
\begin_layout Standard
It is also interesting to turn this graph
\begin_inset Quotes eld
\end_inset
on it's side
\begin_inset Quotes erd
\end_inset
.
So, in this dataset, there were 15270 words observed exactly once (out
of a total of 661104 observations of 37413 words).
This is quite something: of all the words observed, almost half were seen
only once.
More than half were seen twice, or less.
These are presumably rare typos, foreign words, IPA pronunciation guides:
any word that appears only once must be unusual; and yet, there are a lot
of them! There are 5790 words that appear twice, 3093 that appear exactly
3 times,
\emph on
etc.
\emph default
These counts are graphed below.
\begin_inset Foot
status collapsed
\begin_layout Plain Layout
graph of binned-word-counts.dat, generated in disjunct-stats.scm
\end_layout
\end_inset
\end_layout
\begin_layout Standard
\align center
\begin_inset Graphics
filename ../images/binned-word-counts.eps
width 70text%
\end_inset
\end_layout
\begin_layout Standard
This graph indicates that most (almost all) words were observed less than
100 times.
In this dataset, there were only 630 words that were observed more than
100 times, and 267 words that were observed 200 or more times.
\end_layout
\begin_layout Standard
Writing
\begin_inset Formula $N$
\end_inset
for the number of times that some word was observed, it appears that there
are approximately
\begin_inset Formula $15270\times N^{-3/2}$
\end_inset
words observed that many times.
In formulas, the size of the set of words
\begin_inset Formula $\left\{ w|N(w)=N\right\} $
\end_inset
is given by
\begin_inset Formula
\[
\left|\left\{ w|N(w)=N\right\} \right|\sim N^{-3/2}
\]
\end_inset
where
\begin_inset Formula $\left\{ w|\mbox{cond}\right\} $
\end_inset
is a set of words (subject to the condition cond) and
\begin_inset Formula $\left|\left\{ w|\mbox{cond}\right\} \right|$
\end_inset
denotes the size of that set of words.
\end_layout
\begin_layout Standard
The next graph belabors the point, and yet it's important.
\begin_inset Foot
status collapsed
\begin_layout Plain Layout
Generated from binned-word-logli.dat
\end_layout
\end_inset
It shows nothing new, but it does show it in a format that will be recur
frequently, later.
Thus, its worth understanding now.
This graph shows exactly the same data as the previous graph: it
\emph on
is
\emph default
the same graph, except that the x-axis is now labeled differently, and
some of the counts have been binned together.
So first: note that
\begin_inset Formula $-\log_{2}1/661104=19.334$
\end_inset
and so this is the location of the first spike on the far-right.
Next,
\begin_inset Formula $-\log_{2}2/661104=18.334$
\end_inset
and
\begin_inset Formula $-\log_{2}3/661104=17.750$
\end_inset
are the locations of the second and third spikes: these correspond to words
that have been observed 1,2 and 3 times.
The height of the spikes are
\begin_inset Formula
\[
\left|\left\{ w|N(w)=N\right\} \right|\sim2^{-3/2\times\log_{2}N}
\]
\end_inset
as clearly demonstrated by the straight green line.
\end_layout
\begin_layout Standard
\align center
\begin_inset Graphics
filename ../images/binned-word-logli.eps
width 70text%
\end_inset
\begin_inset CommandInset label
LatexCommand label
name "logli distribution graph"
\end_inset
\end_layout
\begin_layout Standard
One peculiar difference between this, and the previous graph is the use
of binning.
Notice here that the observed, red line bounces
\emph on
above
\emph default
the straight green line, while in the previous graph, it was strictly below.
If these both show the same data, how can this be? This seems to be a bit
of a paradox.
The difference can be understood as follows (and is important to keep in
mind).
This graph was generated by bin-counting.
The x-axis was divided into 200 equal-sized bins, and whenever the log-likeliho
od of a word landed within a particular bin, the count was accumulated into
that bin.
Both this and the previous graph are distributions, but they use different
measures.
The two measures are related by the Jacobian determinant
\begin_inset Foot
status collapsed
\begin_layout Plain Layout
https://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant
\end_layout
\end_inset
, which is given by the derivative of the log.
This is a curious effect, to be kept in mind when comparing different graphs.
\end_layout
\begin_layout Subsection*
Ranked average observations per disjunct
\end_layout
\begin_layout Standard
A more interesting distribution arises by looking at the average number
of observations per disjunct (per word).
That is, a single word may have hundreds of disjuncts, observed thousands
of times; what is the average number of times that a disjunct is observed?
By
\begin_inset Quotes eld
\end_inset
average
\begin_inset Quotes erd
\end_inset
, it is explicitly meant
\begin_inset Formula $N(w,*)/\left|(w,*)\right|$
\end_inset
, the number of observations divided by the support for those observations.
\end_layout
\begin_layout Standard
This number gives a hint of how
\begin_inset Quotes eld
\end_inset
narrow
\begin_inset Quotes erd
\end_inset
the grammatical usage of a word is.
If the average is high, it means that the word just does not have very
many disjuncts on it; the few that it does have are observed a lot.
Recall that these disjuncts (pseudo-disjuncts) connect to individual words,
and not to word-classes.
Thus, if a disjunct is seen a lot, it probably connects to another word,
forming a high-MI pair.
This can be explicitly seen in the example further below.
\end_layout
\begin_layout Standard
A graph of the ranked average number of observations, per disjunct, per
word, is shown on the left, below.
\begin_inset Foot
status collapsed
\begin_layout Plain Layout
Computed with the sorted-avg list in disjunct-stats.scm
\end_layout
\end_inset
Words with fewer than 100 observations have been excluded, so as to minimize
the spurious noise.
Note that although the line is straight in this log-log graph, the Zipf
exponent is approximately -1/4.
The graph on the right expresses an alternate view of the same idea: this
one shows a bin-count of all of the words.
Reaffirming the graph on the left, it indicates that almost all words have
an average disjunct observation count of less than three.
Both of these graphs emphasize just how thin and weak this dataset is:
to be certain of a disjunct, it sure would be nice to observe it, in actual
use, more than half-a-dozen times!
\end_layout
\begin_layout Standard
\align center
\begin_inset Graphics
filename ../images/ranked-avg.eps
width 50text%
\end_inset
\begin_inset Graphics
filename ../images/binned-avg.eps
width 50text%
\end_inset
\end_layout
\begin_layout Standard
The spike on the far right on the graph on the right suggests that there
are about 30 words that have an average number of observations of greater
than 6! Since they are
\emph on
not
\emph default
showing up in the graph on the left, this shows that the underlying word
have fewer than 100 observations.
\end_layout
\begin_layout Standard
The first ten on the ranked list are "According" "Gestalt" "Cao" "Berger"
"2015" "It" "U.S" "United" "However" "y".
Unsurprisingly, most of these are all capitalized, suggesting that they
are either part of a proper name, or that perhaps they begin a sentence.
\end_layout
\begin_layout Standard
For example, the appearance of
\begin_inset Quotes eld
\end_inset
United
\begin_inset Quotes erd
\end_inset
is almost surely due to the high-MI pair
\begin_inset Quotes eld
\end_inset
United States
\begin_inset Quotes erd
\end_inset
, and so this probably dominates the disjunct count.
This is confirmed by manual inspection
\begin_inset Foot
status collapsed
\begin_layout Plain Layout
View disjuncts by saying (filter (lambda (cset) (< 10 (get-count cset)))
(cog-incoming-by-type (Word "foo") 'LgWordCset)) where 10 is the minimum
number of counts.
\end_layout
\end_inset
, per table below.
\end_layout
\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="6" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
disjunct
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
number of observations
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
the- States+
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
108
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
in- the- States+
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
41
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
States+
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
35
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
the- Kingdom+
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
16
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
the- the- States+
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
14
\end_layout
\end_inset
</cell>
</row>
</lyxtabular>
\end_inset
\end_layout
\begin_layout Standard
Only the first four disjuncts seem to be grammatically acceptable.
In the first, second and fourth usage,
\begin_inset Quotes eld
\end_inset
States
\begin_inset Quotes erd
\end_inset
seems to be a head word, with
\begin_inset Quotes eld
\end_inset
United
\begin_inset Quotes erd
\end_inset
being the dependent.
In the third case, it would appears that
\begin_inset Quotes eld
\end_inset
United
\begin_inset Quotes erd
\end_inset
is the head-word, and
\begin_inset Quotes eld
\end_inset
States
\begin_inset Quotes erd
\end_inset
the dependent.
\end_layout
\begin_layout Standard
The fifth most frequent disjunct is grammatical garbage: noise from bad
MST parses.
There are even more disjuncts, which occur with lower frequency: there
are 156 in all.
Almost all of these are presumably junk of some kind.
We conclude that the signal/noise ratio here is about
\begin_inset Formula $10\log_{10}(108/14)$
\end_inset
or about 9 dB.
These first four disjuncts really pull up the average, which is quite low,
given this skew: the average number of observations per disjunct for
\begin_inset Quotes eld
\end_inset
United
\begin_inset Quotes erd
\end_inset
is 2.96.
\end_layout
\begin_layout Standard
The corresponding table for
\begin_inset Quotes eld
\end_inset
It
\begin_inset Quotes erd
\end_inset
is
\end_layout
\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="6" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
disjunct
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
number of observations
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
is+
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
216
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
was+
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
197
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
has+
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
55
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
was+ was+
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
12
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
is+ was+
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
10
\end_layout
\end_inset
</cell>
</row>
</lyxtabular>
\end_inset
\end_layout
\begin_layout Standard
This also is much as expected:
\begin_inset Quotes eld
\end_inset
It
\begin_inset Quotes erd
\end_inset
is a sentence opener, and the three sentence types it opens are
\begin_inset Quotes eld
\end_inset
It is
\begin_inset Quotes erd
\end_inset
,
\begin_inset Quotes eld
\end_inset
It was
\begin_inset Quotes erd
\end_inset
and
\begin_inset Quotes eld
\end_inset
It has
\begin_inset Quotes erd
\end_inset
.
The fourth and fifth disjuncts are grammatical garbage.
The average number of disjunct observations is 3.03, almost identical to
that for
\begin_inset Quotes eld
\end_inset
United
\begin_inset Quotes erd
\end_inset
.
\end_layout
\begin_layout Standard
The skewness appears to be very sharp.
This suggests that we should not waste time looking at mean-square variations
in the average, although we'll do this anyway.
But first, its worth graphing the skewness directly.
Again, this is done on a log-log graph, in a Zipfian way.
\end_layout
\begin_layout Subsection*
Disjunct count distribution
\end_layout
\begin_layout Standard
The graph below shows the distribution of the disjunct observations on the
two words
\begin_inset Quotes eld
\end_inset
United
\begin_inset Quotes erd
\end_inset
and
\begin_inset Quotes eld
\end_inset
It
\begin_inset Quotes erd
\end_inset
.
Indeed, it looks Zipfian; since we know that all but the first three or
four disjuncts are noise, this graph illustrates
\begin_inset Quotes eld
\end_inset
pink noise
\begin_inset Quotes erd
\end_inset
or
\begin_inset Quotes eld
\end_inset
1/f noise
\begin_inset Quotes erd
\end_inset
.
\begin_inset Foot
status collapsed
\begin_layout Plain Layout
Computed with the dj-united and dj-it arrays indisjunct-stats.scm
\end_layout
\end_inset
\end_layout
\begin_layout Standard
\align center
\begin_inset Graphics
filename ../images/ranked-dj-united.eps
width 70text%
\end_inset
\end_layout
\begin_layout Standard
Can one get a smoother distribution by summing together these two graphs?
Sure...
and one can sum together not just these two words, but all words (that
have been observed at least 100 times).
That graph is shown below.
\begin_inset Foot
status collapsed
\begin_layout Plain Layout
Computed with the accum-dj-all function in disjunct-stats.scm
\end_layout
\end_inset
\end_layout
\begin_layout Standard
\align center
\begin_inset Graphics
filename ../images/ranked-dj-counts.eps
width 70text%
\end_inset
\end_layout
\begin_layout Standard
Its kind of a strange graph.
Yes, the x-axis of this graph does imply that there are dozens, if not
hundreds of words with more than a thousand disjuncts on them, and maybe
half-a-dozen that have more than ten-thousand (unique, different) disjuncts
on them! Exactly what does this mean? This is covered in the next section.
\end_layout
\begin_layout Subsection*
Disjunct Support Distribution
\end_layout
\begin_layout Standard
Is it possible that some words have a large number of disjuncts on them?
Yes, it is.
For example, the period was observed to have 18200 unique, different disjuncts
associated with it.
The word
\begin_inset Quotes eld
\end_inset
the
\begin_inset Quotes erd
\end_inset
has 15420 unique, different disjuncts, the comma has 14510.
Rounding out this list are "of" "and" "in" "a" "to" "was".
Its not clear what fraction of these disjuncts are grammatically valid,
and what fraction are junk.
\end_layout
\begin_layout Standard
The graph below shows the distribution of the size of the support: the ranking
of
\begin_inset Formula $\left|(w,*)\right|$
\end_inset
.
Again, the graph appears to be approximately Zipfian.
\begin_inset Foot
status collapsed
\begin_layout Plain Layout
Generated by sorted-support in disjunct-stats.scm
\end_layout
\end_inset
\end_layout
\begin_layout Standard
\align center
\begin_inset Graphics
filename ../images/ranked-support.eps
width 60text%
\end_inset
\end_layout
\begin_layout Standard
Terminology: the
\begin_inset Quotes eld
\end_inset
support
\begin_inset Quotes erd
\end_inset
of a vector is the number of basis elements that have a non-zero coefficient.
This is the set
\begin_inset Formula $(w,*)$
\end_inset
defined earlier.
Equivalently, this is the size of the set of disjuncts associated with
a word, when counted
\emph on
without
\emph default
multiplicity.
\end_layout
\begin_layout Subsection*
Ranked Euclidean length (RMS Size)
\end_layout
\begin_layout Standard
A different distribution arises by looking at the ranked RMS sizes of the
disjunct sets
\begin_inset Foot
status collapsed
\begin_layout Plain Layout
Obtained by running (print-ts-rank sorted-lengths outport) from the disjunct-sta
ts.scm file on the en_pairs_sim database.
\end_layout
\end_inset
.
Here, the RMS size
\begin_inset Foot
status collapsed
\begin_layout Plain Layout
The word
\begin_inset Quotes eld
\end_inset
length
\begin_inset Quotes erd
\end_inset
can be used to describe the root-mean-square size of the set of disjuncts
associated with a word.
That is, each element of the set is a disjunct, and that disjunct has a
count, the number of times it has been observed.
The root-mean-square of these counts can be taken as the set-size.
But this set can also be interpreted as a vector, and so the RMS size is
the same thing as the Euclidean length of the vector.
Thus, the word
\begin_inset Quotes eld
\end_inset
length
\begin_inset Quotes erd
\end_inset
is sometimes used for the RMS size; they're the same thing.
\end_layout
\end_inset
is computed by taking the root-mean-square of the counts on each disjunct
in the set, that is, by computing
\begin_inset Formula $\sqrt{\sum_{d}N(w,d)^{2}}$
\end_inset
for each word
\begin_inset Formula $w$
\end_inset
and then ranking.
Interpreting
\begin_inset Formula $d$
\end_inset
as a basis element of a vector space, this can be recognized as the Euclidean
length of the count-vector.
\end_layout
\begin_layout Standard
The RMS size of the set is thus larger not only when more disjuncts have
been observed, but also when most of the observations are made of only
a small handful of disjuncts.
That is, the RMS size should be relatively larger, if the word is less
grammatically flexible.
So for example, prepositions tend to be very flexible; adjectives, not
so much.
Thus, we expect adjectives to appear higher-up on this ranking list, than
the observation-based list.
And this might be true, relatively, but certainly not true absolutely.
\end_layout
\begin_layout Standard
\align center
\begin_inset Graphics
filename ../images/ranked-lengths.eps
width 70text%
\end_inset
\end_layout
\begin_layout Standard
The first ten words of the RMS-size-list are: "." "and" "," "in" "the" ")"
"(" "of" "as" "to".
Not that interesting: these are all words that were observed a lot in the
text.
The RMS size is dominated by the total number of observations of a word
in text.
In and of itself, its insufficient to indicate how
\begin_inset Quotes eld
\end_inset
concentrated
\begin_inset Quotes erd
\end_inset
the disjuncts are, how grammatically narrow a word is.
For this, some other quantity is needed.
\end_layout
\begin_layout Standard
Note that the graph above is Zipfian, but the slope is approximately -0.65.
The green line is an
\begin_inset Quotes eld
\end_inset
eyeball
\begin_inset Quotes erd
\end_inset
fit.
Why the slope is approximately 2/3rds of the canonical slope is not clear.
\end_layout
\begin_layout Subsection*
Mean-square to size ratio
\end_layout
\begin_layout Standard
More interesting is the ratio of mean-square size to the total size.
In formulas, by ranking according to
\begin_inset Formula
\[
\frac{\sqrt{\sum_{d}N^{2}(w,d)}}{N(w,*)}=\frac{\sqrt{\sum_{d}p^{2}(w,d)}}{p(w,*)}
\]
\end_inset
This seems like the interesting ratio, because the Zipf exponent of -0.65
would be doubled, when working with mean-square sizes, thus making the
two rankings comparable.
\end_layout
\begin_layout Standard
Words high in this score will be words that have relatively few disjuncts
on them, or at least, few that matter much, that rise above the level of
noise.
The first ten on this list are (after excluding all words with fewer than
100 observations): "It" "According" "He" ")" "and" "(" "United" "as" "."
"well".
Note that most are capitalized: that means that these words appeared at
the beginning of sentences; the way in which the can link to what follows
is fairly constrained, and thus it is no surprise that these have only
a few disjuncts on them.
\end_layout
\begin_layout Standard
Excluding the capitalized words, and punctuation, in this list what remains
is quite surprising.
It is shown in the table below.
\begin_inset Foot
status collapsed
\begin_layout Plain Layout
Extracted from ranked-sqlen-norm.dat
\end_layout
\end_inset
\end_layout
\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="11" columns="3">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
rank
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
score
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
word
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
5
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
41.4
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
and
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
8
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
31.6
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
as
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
10
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
30.1
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
well
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
11
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
29.9
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
in
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
18
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
18.8
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
first
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
21
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
17.2
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
is
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
22
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
16.7
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
also
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
23
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
15.3
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
can
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
24
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
15.3
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
on
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
25
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
14.9
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
at
\end_layout
\end_inset
</cell>
</row>
</lyxtabular>
\end_inset
\begin_inset space \qquad{}
\end_inset
\begin_inset Tabular
<lyxtabular version="3" rows="11" columns="3">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
rank
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
score
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
word
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
26
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
14.4
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
been
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
31
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
13.2
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
but
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
32
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
13.1
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
part
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
34
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
12.8
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
with
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
35
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
12.8
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
old
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
36
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
12.7
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
same
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
38
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
11.9
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
one
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
39
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
11.6
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
which
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
43
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
10.8
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
not
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
49
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
10.2
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
the
\end_layout
\end_inset
</cell>
</row>
</lyxtabular>
\end_inset
\end_layout
\begin_layout Standard
It is surprising because, grammatically, we expect most of these words to
have a large number of varied disjuncts attached to them.
We expect them to be diffuse, not sharp: we expect that these would have
a large number of observations smeared over a large variety of disjuncts,
instead of having their weight concentrated in only one or two disjuncts,
the way that
\begin_inset Quotes eld
\end_inset
United
\begin_inset Quotes erd
\end_inset
was, above.
So what is going on, here?
\end_layout
\begin_layout Standard
The distribution is unusual; it is shown below.
Note that the x-axis is drawn with a log-scale.
It appears to be linear, with an exponent of -4.
\end_layout
\begin_layout Standard
\align center
\begin_inset Graphics
filename ../images/binned-sqlen-norm.eps
width 70text%
\end_inset
\end_layout
\begin_layout Standard
Are there other interesting measures? One could contemplate the ratio of
the mean-square size to the support
\begin_inset Formula $\sum_{d}N(w,d)^{2}/\left|(w,*)\right|$
\end_inset
.
Another possibility would be this, minus the average-squared, which would
give the second moment, aka, the mean-square deviation from the average,
specifically
\begin_inset Formula
\[
\frac{\sum_{d}N(w,d)^{2}}{\left|(w,*)\right|}-\left[\frac{N(w,*)}{\left|(w,*)\right|}\right]^{2}=\frac{1}{\left|(w,*)\right|}\sum_{d}\left[N(w,d)-\frac{N(w,*)}{\left|(w,*)\right|}\right]^{2}
\]
\end_inset
Neither of these variations seem promising; they seem to offer up more of
the same, at least on this dataset.
A larger and more refined dataset might reveal otherwise.
\end_layout
\begin_layout Subsection*
Mutual information
\end_layout
\begin_layout Standard
The concept of the
\begin_inset Quotes eld
\end_inset
fractional mutual information
\begin_inset Quotes erd
\end_inset
for a pair is interesting to explore.
Define this as
\begin_inset Formula
\begin{equation}
MI_{pair}(w,d)=-\log_{2}\frac{p(w,d)}{p(w,*)p(*,d)}\label{eq:mi-word-disjunct}
\end{equation}
\end_inset
This is
\begin_inset Quotes eld
\end_inset
fractional
\begin_inset Quotes erd
\end_inset
in the sense that the total MI for the set of all pairs can be written
as
\begin_inset Formula
\[
MI_{cset}=\sum_{w,d}p(w,d)MI_{pair}(w,d)
\]
\end_inset
Fractional MI is interesting because it usually has a reasonably nice distributi
on.
For this particular dataset, it ranges from about -20 to +8.
The distribution is shown in the graphs below.
\begin_inset Foot
status collapsed
\begin_layout Plain Layout
These are graphed by binned-cset-mi and weighted-cset-mi in the disjunct-stats.sc
m file.
\end_layout
\end_inset
\end_layout
\begin_layout Standard
\align center
\begin_inset Graphics
filename ../images/binned-cset-mi.eps
width 50col%
\end_inset
\begin_inset Graphics
filename ../images/weighted-cset-mi.eps
width 50col%
\end_inset
\end_layout
\begin_layout Standard
These graphs are generated by computing the value for
\begin_inset Formula $MI_{pair}(w,d)$
\end_inset
for each of the 446204
\begin_inset Formula $(w,d)$
\end_inset
pairs (aka 'connector sets'), and approximating it's distribution by bin
counting.
In each graph above, there are 200 bins, each of width of about 30/200,
and each pair is assigned to one of the binds, according to it's MI value.
The graph on the left then shows how many pairs there are in each bin.
The graph on the left is similar, but not the same: it sums the frequencies
for all the pairs in each bin.
In formulas: the graph on the left shows the value of
\begin_inset Formula
\[
\mbox{sizeof }\left\{ (w,d)|MI_{pair}(w,d)\mbox{ is in bin}\right\}
\]
\end_inset
while the graph on the right shows
\begin_inset Formula
\[
\sum_{MI_{pair}(w,d)\mbox{ is in bin}}p(w,d)
\]
\end_inset
where '
\begin_inset Formula $x\mbox{ is in bin}$
\end_inset
' simply means
\begin_inset Formula $lo\le x<hi$
\end_inset
with the bin being the interval
\begin_inset Formula $[lo,hi)$
\end_inset
.
\end_layout
\begin_layout Standard
Both of these graphs show
\begin_inset Quotes eld
\end_inset
combs
\begin_inset Quotes erd
\end_inset
in the left side.
These combs are exactly the same combs as noted in the last figure in section
\begin_inset CommandInset ref
LatexCommand nameref
reference "logli distribution graph"
\end_inset
\begin_inset CommandInset ref
LatexCommand vref
reference "logli distribution graph"
\end_inset
.
The combs are due to the large number of words that have been observed
only a small handful of times.
In essence, the combs attest that the bulk of the high-MI pairs have been
observed more than just a few times;
\emph on
i.e.
\emph default
the high MI values are meaningful.
\end_layout
\begin_layout Subsection*
Mutual Information of words
\end_layout
\begin_layout Standard
The mutual information of a single word can be defined by summing the (fractiona
l) mutual information between a word, and all of it's disjuncts:
\begin_inset Formula
\[
MI_{word}(w)=\frac{1}{p(w)}\sum_{d}p(w,d)MI_{pair}(w,d)
\]
\end_inset
This is also written in the
\begin_inset Quotes eld
\end_inset
fractional
\begin_inset Quotes erd
\end_inset
style, so that, again, the total MI of the entire dataset can be written
as
\begin_inset Formula
\[
MI_{cset}=\sum_{w}p(w)MI_{word}(w)
\]
\end_inset
That is,
\begin_inset Formula $MI_{word}(w)$
\end_inset
is the fractional contribution of the word to the total MI.
The fractional MI is very convenient for comparing different words, since
it factors out the frequency of how often a word is observed: the MI of
two words with two very different frequencies can be directly compared.
As can be seen from the graph below, the fractional MI ranges between +3
and +19 for this dataset.
\end_layout
\begin_layout Standard
The total MI for the dataset is measured to be 7.985 bits.
\end_layout
\begin_layout Standard
The distribution can be visualized in two different ways.
The graph on the left, below
\begin_inset Foot
status collapsed
\begin_layout Plain Layout
This is graphed by sorted-word-mi-hi-p in disjunct-stats.scm
\end_layout
\end_inset
shows the ranked MI of the 630 words that have been observed more than
100 times in this dataset.
Note that it is a semi-log plot; it is NOT Zipfian.
Note that the MI seems to decay logarithmically, for a good long ways,
and then drops off a cliff.
\end_layout
\begin_layout Standard
The graph on the right shows the distribution, for all 37413 words, bin-counted
into 100 bins.
Unfortunately, this sampling remains small and noisy, so the suggestion
of a log-linear decay to both left and right is hidden by rather noisy
spikes.
The combs on the far right are again the same combs as noted in the last
figure in section
\begin_inset CommandInset ref
LatexCommand nameref
reference "logli distribution graph"
\end_inset
\begin_inset CommandInset ref
LatexCommand vref
reference "logli distribution graph"
\end_inset
.
\end_layout
\begin_layout Standard
\align center
\begin_inset Graphics
filename ../images/ranked-word-mi-hi-p.eps
width 50text%
\end_inset
\begin_inset Graphics
filename ../images/binned-word-mi.eps
width 50col%
\end_inset
\end_layout
\begin_layout Standard
\begin_inset VSpace defskip
\end_inset
\end_layout
\begin_layout Standard
What re the disjuncts like at either end of this distribution? The first
ten words in the ranking are: "Cao" "Award" "x" "y" "Prime" "we" "per"
"Division" "League" "Game".
The word
\begin_inset Quotes eld
\end_inset
United
\begin_inset Quotes erd
\end_inset
, examined previously, is 86th in this rank.
Lets look at some of these.
\end_layout
\begin_layout Standard
The word
\begin_inset Quotes eld
\end_inset
Award
\begin_inset Quotes erd
\end_inset
has only two disjuncts that were observed more than twice:
\end_layout
\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="3" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
count
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
disjunct
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
8
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
Grammy- for+ Best+
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
7
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
Academy- for+ Best+
\end_layout
\end_inset
</cell>
</row>
</lyxtabular>
\end_inset
\end_layout
\begin_layout Standard
\begin_inset VSpace defskip
\end_inset
\end_layout
\begin_layout Standard
It is already clear from this one example that the hi-MI words will be those
that take part in idioms,
\begin_inset Quotes eld
\end_inset
set phrases
\begin_inset Quotes erd
\end_inset
or
\begin_inset Quotes eld
\end_inset
institutional phrases
\begin_inset Quotes erd
\end_inset
, and that the disjunct identifies the words taking part in the setting.
\end_layout
\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="5" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
count
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
disjunct for
\begin_inset Quotes eld
\end_inset
Prime
\begin_inset Quotes erd
\end_inset
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
23
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
Minister+
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
5
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
the- Minister+
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
3
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
as- Minister+
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
3
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
Governor-General- Minister+
\end_layout
\end_inset
</cell>
</row>
</lyxtabular>
\end_inset
\end_layout
\begin_layout Standard
\begin_inset VSpace defskip
\end_inset
\end_layout
\begin_layout Standard
Only one disjunct for
\begin_inset Quotes eld
\end_inset
Division
\begin_inset Quotes erd
\end_inset
was observed 4 times; those with 3 observations or less suggest that they
were extracted from tables listing WWII military battles and equipment.
\end_layout
\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="2" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
count
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
disjunct for
\begin_inset Quotes eld
\end_inset
Division
\begin_inset Quotes erd
\end_inset
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
4
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
NCAA-
\end_layout
\end_inset
</cell>
</row>
</lyxtabular>
\end_inset
\end_layout
\begin_layout Standard
\begin_inset VSpace defskip
\end_inset
\end_layout
\begin_layout Standard
The disjuncts for
\begin_inset Quotes eld
\end_inset
League
\begin_inset Quotes erd
\end_inset
correspond to word-pairs, and these will clearly be high-MI word-pairs.
\end_layout
\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="4" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
count
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
disjunct for
\begin_inset Quotes eld
\end_inset
League
\begin_inset Quotes erd
\end_inset
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
8
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
Baseball+
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
5
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
Premier-
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
4
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
Justice-
\end_layout
\end_inset
</cell>
</row>
</lyxtabular>
\end_inset
\end_layout
\begin_layout Standard
This ranking also captures a fair amount of the
\begin_inset Quotes eld
\end_inset
garbage
\begin_inset Quotes erd
\end_inset
in Wikipedia (which is the source for the dataset): the disjuncts on
\begin_inset Quotes eld
\end_inset
x
\begin_inset Quotes erd
\end_inset
and
\begin_inset Quotes eld
\end_inset
y
\begin_inset Quotes erd
\end_inset
indicate that these occur almost exclusively in mathematical formulas of
some sort, involving single-letter variables, plus signs and equal signs.
The most frequent disjunct on
\begin_inset Quotes eld
\end_inset
per
\begin_inset Quotes erd
\end_inset
was observed 5 times, and was
\begin_inset Quotes eld
\end_inset
100,000+ 100,000+ 100,000+ 100,000+ 100,000+
\begin_inset Quotes erd
\end_inset
suggesting that it arose in some parse of a table listing.
None of the disjuncts on
\begin_inset Quotes eld
\end_inset
we
\begin_inset Quotes erd
\end_inset
were observed more than 4 times, and seemed to consist of some sort of
grammatical word-salad.
This is not surprising: Wikipedia has a severe shortage of pronouns; style
guidelines prohibit their use.
\end_layout
\begin_layout Standard
At the other extreme, the ten words with the lowest MI are: "The" "a" "to"
"in" "of" "and" "the" "," ".".
These are already familiar from previous rankings: they occur with very
high frequency; the disjunct lists on them will be lengthly variable, diffuse.
\end_layout
\begin_layout Subsection*
Mutual information of disjuncts
\end_layout
\begin_layout Standard
Symmetrically, one also has the mutual information of a disjunct, in comparison
to all of the words it connects to:
\begin_inset Formula
\[
MI_{disjunct}(d)=\frac{1}{p(*,d)}\sum_{w}p(w,d)MI_{pair}(w,d)
\]
\end_inset
Again, this is presented in the
\begin_inset Quotes eld
\end_inset
fractional
\begin_inset Quotes erd
\end_inset
style, so that the total MI of the entire dataset can be written as before:
\begin_inset Formula
\[
MI=\sum_{d}p(*,d)MI_{disjunct}(d)
\]
\end_inset
The distribution is shown below.
\begin_inset Foot
status collapsed
\begin_layout Plain Layout
Use binned-dj-mi to get this.
\end_layout
\end_inset
The combs on the far left are again the same combs as noted in the last
figure in section
\begin_inset CommandInset ref
LatexCommand nameref
reference "logli distribution graph"
\end_inset
\begin_inset CommandInset ref
LatexCommand vref
reference "logli distribution graph"
\end_inset
.
\end_layout
\begin_layout Standard
\align center
\begin_inset Graphics
filename ../images/binned-dj-mi.eps
width 60col%
\end_inset
\end_layout
\begin_layout Standard
So, at first, this looks much like the word-MI graph, until one realizes
its really almost a mirror-image.
Also very notable is that the MI is negative, in all cases! What does this
mean? In this case, it means that entropy is dominating over any
\begin_inset Quotes eld
\end_inset
attractive
\begin_inset Quotes erd
\end_inset
effect: basically, that the various disjuncts are getting used for a very
large and widespread set of words, and that any one disjunct is not favoring
any particular word or word-set; indeed, any given disjunct is mostly dis-favor
ing any word or small set of words, and is instead appearing with just about
all words.
This is explored a bit more in the next section.
\end_layout
\begin_layout Subsection*
Fractional Entropy
\end_layout
\begin_layout Standard
There is a simpler variant than the mutual information, that is also worth
understanding: the fractional contribution to the total entropy.
This is given by the sum
\begin_inset Formula
\[
H_{word}(w)=-\frac{1}{p(w)}\sum_{d}p(w,d)\log_{2}p(w,d)
\]
\end_inset
This is written in the
\begin_inset Quotes eld
\end_inset
fractional
\begin_inset Quotes erd
\end_inset
style, so that the total entropy of the entire dataset can be written as
\begin_inset Formula
\[
H_{cset}=\sum_{w}p(w)H_{word}(w)
\]
\end_inset
Analogously, one also has the fractional contribution of the disjuncts:
\end_layout
\begin_layout Standard
\begin_inset Formula
\[
H_{disjunct}(d)=-\frac{1}{p(*,d)}\sum_{w}p(w,d)\log_{2}p(w,d)
\]
\end_inset
where, again, one has that
\begin_inset Formula
\[
H_{cset}=\sum_{d}p(d)H_{disjunct}(d)
\]
\end_inset
\end_layout
\begin_layout Standard
The ranked fractional entropy is shown in the left graph below.
\begin_inset Foot
status collapsed
\begin_layout Plain Layout
Generated from sorted-word-ent in disjunct-stats.scm.
\end_layout
\end_inset
It only shows those words that have been observed 100 times, or more.
It resembles the graph for the ranked fractional MI, above.
The graph on the right shows the distribution of the entropy, for all of
the words.
This affirms (or explains?) the sharp knee in the graph on the left: the
knee occurs because almost all words have a large disjunct-entropy.
This is explained below.
\end_layout
\begin_layout Standard
\align center
\begin_inset Graphics
filename ../images/ranked-word-ent.eps
width 50text%
\end_inset
\begin_inset Graphics
filename ../images/binned-word-ent.eps
width 50text%
\end_inset
\end_layout
\begin_layout Standard
The top-ranked words (which have been observed 100 times or more) are these:
\end_layout
\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="11" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
Entropy
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
Word
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
19.22
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
possible
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
19.18
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
education
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
19.18
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
lost
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
19.15
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
days
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
19.15
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
children
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
19.14
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
players
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
19.14
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
available
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
19.12
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
every
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
19.12
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
remained
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
19.10
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
above
\end_layout
\end_inset
</cell>
</row>
</lyxtabular>
\end_inset
\end_layout
\begin_layout Standard
This is not a list that has been exposed with other statistics.
These words are more
\begin_inset Quotes eld
\end_inset
interesting
\begin_inset Quotes erd
\end_inset
than any of the previous lists, which seemed to be filled with quirks associate
d with the dataset.
By contrast, these are
\begin_inset Quotes eld
\end_inset
normal
\begin_inset Quotes erd
\end_inset
words: mostly nouns, some modifiers, a verb, a preposition.
Why these?
\end_layout
\begin_layout Standard
The answer lies in looking at the disjunct set, shown in the table below.
All of these words have a large support, but were observed very infrequently.
For example,
\begin_inset Quotes eld
\end_inset
possible
\begin_inset Quotes erd
\end_inset
has only one disjunct that was observed 3 times, 4 that were observed twice,
and 101 that were observed only once.
A quick examination shows that many, maybe most, are grammatically reasonable,
for example: (investigate- routes+) (is- also- for+) (is- only- to+) (made-
through+) (many- as-) and so on.
But all of these were observed precisely once.
The table below illustrates this.
\end_layout
\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="6" columns="5">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
Entropy
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
Word
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
\begin_inset Formula $\left|(w,*)\right|_{2}$
\end_inset
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
\begin_inset Formula $\left|(w,*)\right|_{1}$
\end_inset
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
\begin_inset Formula $\left|(w,*)\right|_{0}$
\end_inset
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
19.22
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
possible
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
1
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
5
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
106
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
19.18
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
education
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
1
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
7
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
100
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
19.18
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
lost
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
1
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
7
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
100
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
19.15
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
days
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
2
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
7
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
98
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
19.15
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
children
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
1
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
9
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
101
\end_layout
\end_inset
</cell>
</row>
</lyxtabular>
\end_inset
\end_layout
\begin_layout Standard
\begin_inset VSpace defskip
\end_inset
\end_layout
\begin_layout Standard
A bit of notation.
Recall the definition for the set that supports a word:
\begin_inset Formula
\[
(w,*)=\left\{ (w,d)|N(w,d)>0\right\}
\]
\end_inset
together with the notation
\begin_inset Formula $\left|(w,*)\right|$
\end_inset
to indicate the size of this set.
Extend this notation as
\begin_inset Formula
\[
\left|(w,*)\right|_{k}=\mbox{sizeof }\left\{ (w,d)|N(w,d)>k\right\}
\]
\end_inset
That is,
\begin_inset Formula $\left|(w,*)\right|_{k}$
\end_inset
is the size of the support for when the disjuncts on word
\begin_inset Formula $w$
\end_inset
have been seen more than
\begin_inset Formula $k$
\end_inset
times.
\end_layout
\begin_layout Standard
From this table, it is now clear why
\begin_inset Quotes eld
\end_inset
large entropy
\begin_inset Quotes erd
\end_inset
can be intuitively understood to mean
\begin_inset Quotes eld
\end_inset
many possibilities
\begin_inset Quotes erd
\end_inset
.
Each of these words was seen in a very broad setting of possibilities:
in a sense, the broadest possible.
Each of these words was observed with a vary large set of different disjuncts,
and this set was as spread-out as possible: the vast majority of disjuncts
were observed exactly once.
\end_layout
\begin_layout Subsection*
More terms
\end_layout
\begin_layout Standard
Some curious terms show up in relating the fractional mutual information
to the fractional entropy.
Expanding out the above summations, one obtains
\begin_inset Formula
\[
MI_{word}(w)=H_{word}(w)+\log_{2}p(w,*)+\frac{1}{p(w)}\sum_{d}p(w,d)\log_{2}p(*,d)
\]
\end_inset
The last term is bizarre...
\end_layout
\begin_layout Subsection*
Vertex degrees and hubiness
\end_layout
\begin_layout Standard
Vertex degrees can be defined as the average number of connectors per disjunct.
In principle, the vertex degree is an excellent indicator of the part of
speech.
For example, determiners, adjectives and adverbs typically have a degree
of one: they have one connector, which is modifying the noun (or verb)
that they act on.
By contrast, nouns typically have a degree of two: one connector to attach
to a verb, another to a modifier, and that's it.
Verbs have a degree of three: one connector to a subject, one to a direct
object, a third to an indirect object or a modifier.
Of course, nouns might have two or more modifiers, or maybe zero modifiers;
verbs are also quite variable, but the general concept of vertex degree
is appealing.
Closely related to this is the idea of
\begin_inset Quotes eld
\end_inset
hubiness
\begin_inset Quotes erd
\end_inset
, which can be defined as the second moment of the degree.
\end_layout
\begin_layout Standard
Thus, its worth looking at this.
Define the average degree as
\begin_inset Formula
\[
K(w)=\frac{\sum_{d,c}N(w,d)C(d,c)}{N(w,*)}
\]
\end_inset
This is graphed, below, for all words that have at least 100 observations.
\begin_inset Foot
status collapsed
\begin_layout Plain Layout
Computed with sorted-avg-connectors in disjunct-stats.scm
\end_layout
\end_inset
\end_layout
\begin_layout Standard
\align center
\begin_inset Graphics
filename ../images/ranked-avg-connectors.eps
width 70text%
\end_inset
\end_layout
\begin_layout Standard
This graph is unexpected; in part, having an average number of connectors
that exceed 5 or 6 is intuitively surprising.
What's happening here? The first ten items in the ranking are: "-" "de"
"y" ":" "(" ")" "General" "Department" "x" "Act".
\end_layout
\begin_layout Standard
Consider
\begin_inset Quotes eld
\end_inset
de
\begin_inset Quotes erd
\end_inset
.
There are 12 observations of the disjunct
\begin_inset Quotes eld
\end_inset
Janeiro+
\begin_inset Quotes erd
\end_inset
.
There are 9 observations of the disjunct
\begin_inset Quotes eld
\end_inset
la+
\begin_inset Quotes erd
\end_inset
.
There are 51 observations of a disjunct that has 117 connectors on it!!
This starts out as
\begin_inset Quotes eld
\end_inset
Diego- Francisco- Francisco- Alonso- Carlos- Fernández- Carlos-
\begin_inset Quotes erd
\end_inset
and ends with
\begin_inset Quotes eld
\end_inset
Figueroa+ (+ (+ y+
\begin_inset Quotes erd
\end_inset
suggesting that there were possibly 51 really bad parses of a very long
table of Spanish kings, which was mistaken for being a single sentence.
Clearly, its junk; its frequently-occurring junk, which suggests that the
table was repeatedly included in maybe 51 different Wikipedia pages.
\end_layout
\begin_layout Standard
Similarly,
\begin_inset Quotes eld
\end_inset
Department
\begin_inset Quotes erd
\end_inset
has 18 observations of a disjunct with 41 connectors on it.
It starts with
\begin_inset Quotes eld
\end_inset
Education- Education- Health- Services- Services- Immigration-
\begin_inset Quotes erd
\end_inset
and ends with
\begin_inset Quotes eld
\end_inset
Veterans+ of+ Treasury+ Treasury+
\begin_inset Quotes erd
\end_inset
, again suggesting a bad parse of a table mistaken for a sentence, and included
in 18 different Wikipedia pages.
\end_layout
\begin_layout Standard
The list continues in a similar way, for quite a while.
The green line suggests that if some 30 or so pathological cases are ignored,
the system settles down to a more respectable behavior.
Entries 30 through 50 in the rankings are "Bay" "Street" "Island" "of"
"century" "right" "Game" "Georgian" "or" "a" ";" "near" "Party" "team"
"law" "Australia" "her" "research" "Church" "east" "Government".
Notable is a preponderance of capitalized words, suggesting more tables
of various sorts, and a complete lack of verbs.
A spot-check of words like
\begin_inset Quotes eld
\end_inset
team
\begin_inset Quotes erd
\end_inset
and
\begin_inset Quotes eld
\end_inset
law
\begin_inset Quotes erd
\end_inset
shows that the pathological behavior continues.
Several conclusions are possible.
\end_layout
\begin_layout Standard
One conclusion is that there is a severe shortage of verbs in Wikipedia
articles, and this makes sense: its primarily descriptive, rather than
active: running, jumping, hitting, putting, mixing, giving, setting are
not the kinds of verbs that are required to describe a typical encyclopedia
topic.
\end_layout
\begin_layout Standard
Another conclusion is that perhaps the number of observations of pairs are
insufficient to get deep, reliable MST parsing.
Junk links get used because there were not enough appropriate word-pairs
seen to give a good-quality MST parse.
A related conclusion is that the connector-set dataset is also too thin:
The grammatically reasonable connectors are observed not even a few dozen
times, barely pushing them out of the noise-floor of onesie-twosie observations
of junk.
\end_layout
\begin_layout Standard
So: bigger datasets, and an urgent need for non-Wikipedia content.
Fiction, and presumably teen fiction should be filled with the kinds of
active verbs describing human motions and actions, and should be absent
of tables and lists masquerading as sentences.
\end_layout
\begin_layout Subsection*
Hubiness
\end_layout
\begin_layout Standard
Similar to the above, hubiness can be defined as the second moment of the
connector count:
\end_layout
\begin_layout Standard
\begin_inset Formula
\[
hub(w)=\frac{\sum_{d,c}N(w,d)C^{2}(d,c)}{N(w,*)}-K^{2}(w)
\]
\end_inset
Given the earlier Zipfian results on the average degree
\begin_inset Formula $K(w)$
\end_inset
, it should be no surprise that a ranked listing of words by hubiness is
very nearly identical to the listing for average degree.
This is, after all, what the scale-free nature of the Zipfian distribution
really means.
Not only is the ranking nearly the same, but one also has the approximate
equality
\begin_inset Formula $hub(w)\approx2K(w)$
\end_inset
to some ten or twenty percent.
\end_layout
\begin_layout Subsection*
Disjunct Cosine Similarity
\end_layout
\begin_layout Standard
The cosine similarity between two vectors is simply their inner product.
In this case, given two words
\begin_inset Formula $w_{1}$
\end_inset
and
\begin_inset Formula $w_{2}$
\end_inset
, it is given by
\begin_inset Formula
\[
\mbox{sim}(w_{1},w_{2})=\frac{\sum_{d}N(w_{1},d)N(w_{2},d)}{\mbox{len}(w_{1})\mbox{len}(w_{2})}
\]
\end_inset
where
\begin_inset Formula $\mbox{len}(w)$
\end_inset
is the root-mean-square length (Euclidean length) of the connector-set
vector:
\begin_inset Formula
\[
\mbox{len}(w)=\sqrt{\sum_{d}N^{2}(w,d)}
\]
\end_inset
\end_layout
\begin_layout Standard
The current dataset being analyzed contains 1985 words whose length is greater
than 8; the ranking-by-length was already shown up above, in a previous
graph.
The similarity between all pairs of these was computed; this resulted in
15808 pairs with a cosine similarity of greater than 0.5.
These can be sorted and ranked.
\begin_inset Foot
status collapsed
\begin_layout Plain Layout
See the good-sims and ranked-sims in disjunct-stats.scm file.
\end_layout
\end_inset
They are shown below.
\end_layout
\begin_layout Standard
\align center
\begin_inset Graphics
filename ../images/ranked-sims.eps
width 70text%
\end_inset
\end_layout
\begin_layout Standard
Well, that's new! The similarity ranking is very well fit by a cubic! In
the above, the green line is given by
\begin_inset Formula $1-\log^{3}(rank)/1800$
\end_inset
and it fits the tailing end of the distribution so well, that the red data
line is almost completely hidden under the green line!
\end_layout
\begin_layout Standard
The graph below shows the distribution of cosine similarity.
\begin_inset Foot
status collapsed
\begin_layout Plain Layout
Produced by binned-good-sims.
\end_layout
\end_inset
There are 5544 words for which
\begin_inset Formula $4<\mbox{len}(w)$
\end_inset
.
This shows the distribution of the 2.17 million word-pairs formed from these
words, and for which
\begin_inset Formula $0.1\le\mbox{sim}(w_{1},w_{2})$
\end_inset
.
The eyeballed fit is for
\begin_inset Formula $\exp(-8\mbox{sim})$
\end_inset
.
\end_layout
\begin_layout Standard
\align center
\begin_inset Graphics
filename ../images/binned-sims.eps
width 70text%
\end_inset
\end_layout
\begin_layout Standard
The top-ten similarity pairs are: 'Poir ..
Perls' 'Poir ..
Sevan' 'Sevan ..
Perls' 'Sevan ..
Ruins' 'Ruins ..
Perls' 'Poir ..
Ruins' 'Gongora ..
Perls' 'Poir ..
Gongora' 'Gongora ..
Sevan' 'Gongora ..
Ruins' 'Poir ..
Gag'.
The first three all have a cosine of 1.0, and the rest in the 0.999 range.
The three words
\begin_inset Quotes eld
\end_inset
Poir
\begin_inset Quotes erd
\end_inset
\begin_inset Quotes eld
\end_inset
Perls
\begin_inset Quotes erd
\end_inset
and
\begin_inset Quotes eld
\end_inset
Sevan
\begin_inset Quotes erd
\end_inset
all have exactly one disjunct, containing exactly one connector, linking
them to a period, presumably at the end of a sentence.
Its observed a few dozen times, and that's it.
\begin_inset Quotes eld
\end_inset
Gongora
\begin_inset Quotes erd
\end_inset
has three disjuncts; (.+) is observed 56 times, (,+) is observed twice and
(,+ .+) is observed twice.
So, yes, all these words have been discovered to behave in a grammatically
similar fashion, however, this behavior seems quite boring: links to a
period.
\end_layout
\begin_layout Standard
The next 300 entries are all links between capitalized words, most of them
to the words above.
After that, the list is just barely slightly more varied, with e.g.
\begin_inset Quotes eld
\end_inset
tonight
\begin_inset Quotes erd
\end_inset
showing up occasionally, presumably because there are many sentences that
end with the word
\begin_inset Quotes eld
\end_inset
tonight
\begin_inset Quotes erd
\end_inset
, followed by a period.
There are very few links to lower-case words, but the ones that show up
are..
interesting.
The table below gives some hand-picked examples.
\end_layout
\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="14" columns="3">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
rank
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
cosine
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
word-pair
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
504
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
0.868
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
'in ..
from'
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
633
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
0.855
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
'in ..
at'
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
637
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
0.854
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
'In ..
By'
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
643
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
0.854
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
'in ..
After'
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
740
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
0.846
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
'canal ..
swamp'
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
754
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
0.843
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
'at ..
At'
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
757
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
0.843
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
'in ..
In
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
818
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
0.838
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
'in ..
on'
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
840
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
0.836
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
'creation ..
existence'
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
878
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
0.833
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
'e.g ..
i.e'
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
965
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
0.826
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
'and ..
but'
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
969
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
0.826
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
'in ..
With'
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
971
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
0.826
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
'in ..
inside'
\end_layout
\end_inset
</cell>
</row>
</lyxtabular>
\end_inset
\end_layout
\begin_layout Standard
This table is surprising in several different ways.
First, that the first entry in which two words are both not capitalized
does not occur until about 3% into the list.
Next, that this entry, and most of the others, involve prepositions! Not
adjectives, adverbs, verbs; although we noted already that the source text
is verb-poor.
It is also pleasantly surprising that all of these equivalences appear
to be correct, including the equivalence of upper-case prepositions that
start sentences, with lower-case ones that appear mid-sentence.
The identification of e.g.
with i.e.
seems like a bonus, as does the 'and ..
but' equivalence.
\end_layout
\begin_layout Standard
Lets look at this gift-horse in the mouth.
The equivalence 'canal ..
swamp' is rather pathetic.
So, 'canal' was observed with 68 different, unique disjuncts, but most
have a count of only 1 or 2.
The top-ranked disjunct on 'canal' is (the-) which is seen 20 times, followed
by (the- .+) 8 times.
Similarly, 'swamp' has 18 distinct disjuncts attached, most with counts
of 1 or 2; the top-ranked disjunct is (the-) with 14 counts, and (the-
.+) 3 times.
So, yes, these are both nouns, evidenced by the connection to the word
\begin_inset Quotes eld
\end_inset
the
\begin_inset Quotes erd
\end_inset
, but the evidence is otherwise very weak.
\end_layout
\begin_layout Standard
Likewise, 'creation' gets the disjunct (the- of+) 18 times, and no other
disjuncts observed more than twice.
'existence' gets the same disjunct 10 times, and no other is observed more
than 3 times.
In total, 'creation' had 22 disjuncts on it, while 'exstence' had 26, with
almost all having counts of 1 or 2.
Again, the evidence is thin, with one notable point: 'creation' is not
at all like 'canal', in that 'canal' has zero counts of (the- of+) while
'creation' has only one count of (the-).
As nouns, they are actually quite different, even based on the thin evidence
available.
\end_layout
\begin_layout Standard
The equivalence of 'in ..
from' is much more interesting.
The preposition 'in' has a total of 8338 different unique disjuncts --
this is huge, compared to the nouns above.
Likewise, 'from' has 2185 different disjuncts observed.
The top-ranked disjuncts for each are shown in the tables below.
\end_layout
\begin_layout Standard
\align center
\begin_inset Tabular
<lyxtabular version="3" rows="8" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
count
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
'in' disjunct
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
627
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
the+
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
75
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
a+
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
61
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
which+
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
49
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
the+ the+
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
36
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
used-
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
33
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
his+
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
32
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
order+
\end_layout
\end_inset
</cell>
</row>
</lyxtabular>
\end_inset
\begin_inset space \qquad{}
\end_inset
\begin_inset Tabular
<lyxtabular version="3" rows="8" columns="2">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="0">
<column alignment="center" valignment="top" width="0">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
count
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
'from' disjunct
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
155
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
the+
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
20
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
:+ :+ :+ ..
0+ 0+ 0+
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
18
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
a+
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
9
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
derived-
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
9
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
until+
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
9
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
degree-
\end_layout
\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
8
\end_layout
\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text
\begin_layout Plain Layout
his+
\end_layout
\end_inset
</cell>
</row>
</lyxtabular>
\end_inset
\end_layout
\begin_layout Standard
The similarity of these two are driven primarily by the word-pairs
\begin_inset Quotes eld
\end_inset
in the
\begin_inset Quotes erd
\end_inset
and
\begin_inset Quotes eld
\end_inset
from the
\begin_inset Quotes erd
\end_inset
, followed by
\begin_inset Quotes eld
\end_inset
in a
\begin_inset Quotes erd
\end_inset
and
\begin_inset Quotes eld
\end_inset
from a
\begin_inset Quotes erd
\end_inset
.
Notable also is the common appearance of the pronoun
\begin_inset Quotes eld
\end_inset
his
\begin_inset Quotes erd
\end_inset
, with
\begin_inset Quotes eld
\end_inset
in his
\begin_inset Quotes erd
\end_inset
and
\begin_inset Quotes eld
\end_inset
from his
\begin_inset Quotes erd
\end_inset
.
The second disjunct on 'from' appears to be some sort of garbage, presumably
due to the parse of a list of numeric values of some kind.
\end_layout
\begin_layout Standard
Just as the case with nouns, the evidence for the similarity for these two
prepositions seems quite thin.
The surprise is that, despite this, the identification does seem correct.
\end_layout
\begin_layout Standard
At any rate, the dataset seems a bit thin, but we already knew this; its
based on a fairly small sample.
\end_layout
\begin_layout Subsection*
Pair Cosine Similarity
\end_layout
\begin_layout Standard
To correctly gauge the disjunct-based cosine similarity, it should be contrasted
against the simpler pair-based cosine similarity.
This is given by
\begin_inset Formula
\[
\mbox{sim}_{pair}(w_{1},w_{2})=\frac{\sum_{w}N_{pair}(w_{1},w)N_{pair}(w,w_{2})}{\mbox{len}(w_{1})\mbox{len}(w_{2})}
\]
\end_inset
where
\begin_inset Formula $\mbox{len}(w)$
\end_inset
is the root-mean-square length (Euclidean length) of the pair vector:
\begin_inset Formula
\[
\mbox{len}(w)=\sqrt{\sum_{v}N_{pair}(w,v)N_{pair}(v,w)}
\]
\end_inset
and
\begin_inset Formula $N_{pair}(w,v)$
\end_inset
is the count of having observed the ordered word-pair
\begin_inset Formula $(w,v)$
\end_inset
.
Equivalently, writing the normalized frequency of observing a word pair
as
\begin_inset Formula $p(w,v)=N(w,v)/N(*,*)$
\end_inset
, this similarity can be written in the form
\begin_inset Formula
\[
\mbox{sim}_{pair}(w_{1},w_{2})=\frac{\sum_{w}p(w_{1},w)p(w,w_{2})}{\sqrt{\sum_{v}p(w_{1},v)p(v,w_{1})}\sqrt{\sum_{v}p(w_{2},v)p(v,w_{2})}}
\]
\end_inset
Note that this similarity measure is NOT symmetric:
\begin_inset Formula $\mbox{sim}(w_{1},w_{2})\ne\mbox{sim}(w_{2},w_{1})$
\end_inset
.
This is because it's built out of a manifestly non-symmetric count:
\begin_inset Formula $p(w,v)\ne p(v,w)$
\end_inset
and should really be written as
\begin_inset Formula $p(w,v)=p(R;w,v)$
\end_inset
with the relation
\begin_inset Formula $R$
\end_inset
encompassing all of the constraints of pair-wise word relationships (including,
for example, that the pair might have been extracted from a random planar
tree parse).
Of course, one could construct a symmetrized similarity measure.
\end_layout
\begin_layout Standard
The point here is that this measure treats words as similar with they link,
pair-wise, to the same kinds of words, with the same kinds of frequencies.
This is not unlike the similarity that the disjunct-cosine is measuring,
except that the disjunct carries additional grammatical information with
it: it captures more complex relationships between the words in a sentence.
\end_layout
\begin_layout Subsection*
Cosine Information
\end_layout
\begin_layout Standard
The cosine similarity was defined as
\begin_inset Formula
\[
\mbox{sim}(w_{1},w_{2})=\frac{\sum_{d}N(w_{1},d)N(w_{2},d)}{\sqrt{\sum_{d}N^{2}(w_{1},d)}\sqrt{\sum_{d}N^{2}(w_{2},d)}}
\]
\end_inset
which, after dividing by
\begin_inset Formula $N(*,*)$
\end_inset
so that
\begin_inset Formula $p(w,d)=N(w,d)/N(*,*)$
\end_inset
, gives the equivalent expression
\begin_inset Formula
\[
\mbox{sim}(w_{1},w_{2})=\frac{\sum_{d}p(w_{1},d)p(w_{2},d)}{\sqrt{\sum_{d}p^{2}(w_{1},d)}\sqrt{\sum_{d}p^{2}(w_{2},d)}}
\]
\end_inset
Comparing this to the expression for mutual information suggests that using
the vector support, instead of the vector length, could be interesting.
In particular, these might be interesting:
\begin_inset Formula
\[
\mbox{com}(w_{1},w_{2})=-\log_{2}\frac{\sum_{d}p(w_{1},d)p(w_{2},d)}{p(w_{1},*)p(w_{2},*)}
\]
\end_inset
and
\begin_inset Formula
\[
\mbox{mim}(w,d)=-\log_{2}\frac{p(w,d)}{\sum_{w,d}p^{2}(w,d)}
\]
\end_inset
\end_layout
\begin_layout Standard
I don't know what to call these; the first seems to be some kind of
\begin_inset Quotes eld
\end_inset
cosine information
\begin_inset Quotes erd
\end_inset
, the second, some sort of
\begin_inset Quotes eld
\end_inset
mutual length
\begin_inset Quotes erd
\end_inset
device.
\end_layout
\begin_layout Subsection*
Quality Cosine
\end_layout
\begin_layout Standard
More possibilities exist.
The motivation for using cosine similarity is to find pairs of words that
act in a grammatically similar fashion: they are used in the same way,
with the same kinds of disjuncts.
However, observational counts are subject to the vicissitudes of the input
text: perhaps, instead of using vectors where the components are given
by the frequency, once could instead use components based on, say, the
\begin_inset Quotes eld
\end_inset
quality
\begin_inset Quotes erd
\end_inset
of the disjunct itself.
A disjunct could be judged as being
\begin_inset Quotes eld
\end_inset
high quality
\begin_inset Quotes erd
\end_inset
if
\begin_inset Formula $mi(w,d)=MI_{pair}(w,d)$
\end_inset
as defined in equation
\begin_inset CommandInset ref
LatexCommand vref
reference "eq:mi-word-disjunct"
\end_inset
is high.
This motivates a
\begin_inset Quotes eld
\end_inset
quality cosine
\begin_inset Quotes erd
\end_inset
:
\begin_inset Formula
\[
\mbox{qim}(w_{1},w_{2})=\frac{\sum_{d}mi(w_{1},d)mi(w_{2},d)}{\sqrt{\sum_{d}mi^{2}(w_{1},d)}\sqrt{\sum_{d}mi^{2}(w_{2},d)}}
\]
\end_inset
But what if the high quality is based on a pathetically small number of
observations? Perhaps there should be some observational weighting.
One can contemplate defining
\begin_inset Formula $pi(w,d)=p(w,d)mi(w,d)$
\end_inset
and so the cosine
\begin_inset Formula
\[
\mbox{pim}(w_{1},w_{2})=\frac{\sum_{d}pi(w_{1},d)pi(w_{2},d)}{\sqrt{\sum_{d}pi^{2}(w_{1},d)}\sqrt{\sum_{d}pi^{2}(w_{2},d)}}
\]
\end_inset
The suitability of these different means of judging similarity is not clear.
\end_layout
\begin_layout Section*
The End
\end_layout
\end_body
\end_document