#LyX 2.2 created this file. For more info see http://www.lyx.org/
\lyxformat 508
\begin_document
\begin_header
\save_transient_properties true
\origin unavailable
\textclass article
\begin_preamble
\usepackage{url}
\end_preamble
\use_default_options false
\maintain_unincluded_children false
\language english
\language_package default
\inputencoding utf8
\fontencoding global
\font_roman "times" "default"
\font_sans "helvet" "default"
\font_typewriter "courier" "default"
\font_math "auto" "auto"
\font_default_family default
\use_non_tex_fonts false
\font_sc false
\font_osf false
\font_sf_scale 100 100
\font_tt_scale 100 100
\graphics default
\default_output_format default
\output_sync 0
\bibtex_command default
\index_command default
\paperfontsize default
\spacing single
\use_hyperref true
\pdf_bookmarks true
\pdf_bookmarksnumbered false
\pdf_bookmarksopen false
\pdf_bookmarksopenlevel 1
\pdf_breaklinks true
\pdf_pdfborder true
\pdf_colorlinks true
\pdf_backref false
\pdf_pdfusetitle true
\papersize default
\use_geometry false
\use_package amsmath 2
\use_package amssymb 2
\use_package cancel 1
\use_package esint 0
\use_package mathdots 1
\use_package mathtools 1
\use_package mhchem 0
\use_package stackrel 1
\use_package stmaryrd 1
\use_package undertilde 1
\cite_engine basic
\cite_engine_type default
\biblio_style plain
\use_bibtopic false
\use_indices false
\paperorientation portrait
\suppress_date false
\justification true
\use_refstyle 0
\index Index
\shortcut idx
\color #008000
\end_index
\secnumdepth 3
\tocdepth 3
\paragraph_separation indent
\paragraph_indentation default
\quotes_language english
\papercolumns 1
\papersides 1
\paperpagestyle default
\tracking_changes false
\output_changes false
\html_math_output 0
\html_css_as_file 0
\html_be_strict false
\end_header
\begin_body
\begin_layout Title
Stitching Together Vector Spaces
\end_layout
\begin_layout Author
Linas Vepštas
\end_layout
\begin_layout Date
Draft of 17 June 2018 - First Draft
\end_layout
\begin_layout Abstract
Applying machine learning to linguistics to extract both syntax and semantics,
ideally via unsupervised algorithms is the goal of an incresingly popular
quest.
At this time, vector-space based approaches seem to be the best suited
for this, and are thus increasingly commonplace and quite popular.
This includes systems such as word2vec and and GloVe as exemplars.
These systems fall short, however, as they fail to expose the syntactic
structure of natural language.
The reason for this, and how to move beyond this state of affairs, is discussed
in this text.
\end_layout
\begin_layout Abstract
The key observation made here is that there is more than just one vector
space onto which words can be mapped.
The different vector spaces are necessaarily related to one-another through
syntactic relations.
When a set of words are clustered into a grammatical category, this clustering
must be made in a consistent fashion, for each of the vector spaces.
The vector spaces must be
\begin_inset Quotes eld
\end_inset
stitched together
\begin_inset Quotes erd
\end_inset
in a consistent fashion; this consistency condition is exactly the syntax
of the grammar.
The stitching obeys a certain set of axioms, which are the sheaf axioms.
The best possible stitching together can be found by minimizing an overall
cost function.
\end_layout
\begin_layout Section*
Introduction
\end_layout
\begin_layout Standard
Not written.
See next section.
\end_layout
\begin_layout Standard
Also, the last half of this text is not written.
\end_layout
\begin_layout Section*
Meaning
\end_layout
\begin_layout Standard
So here's one approach to meaning.
It is already clear that disjuncts are correlated with meaning, so one
provisional approach might be to assign each disjunct a unique meaning.
Alternately, this can be used as a doorway to the intentional meaning of
a word.
\end_layout
\begin_layout Standard
Consider the phrases
\begin_inset Quotes eld
\end_inset
the big balloon
\begin_inset Quotes erd
\end_inset
,
\begin_inset Quotes eld
\end_inset
the red balloon
\begin_inset Quotes erd
\end_inset
,
\begin_inset Quotes eld
\end_inset
the small ballon
\begin_inset Quotes erd
\end_inset
...
The pseudo-disjuncts on balloon in these three cases would be
\begin_inset Quotes eld
\end_inset
the- big-
\begin_inset Quotes erd
\end_inset
\begin_inset Quotes eld
\end_inset
the - red-
\begin_inset Quotes erd
\end_inset
and
\begin_inset Quotes eld
\end_inset
the- small-
\begin_inset Quotes erd
\end_inset
(plus an additional connector to the verb).
Examining this connector-by-connector, we expect that the MI for the word
pair (the, balloon) to be small, while the MI for the word-pairs (big,
balloon), (red, balloon) and (small, balloon) to be large(r).
Its thus tempting to identify the set {big, red, small} as the set of intention
al attributes associated with
\begin_inset Quotes eld
\end_inset
balloon
\begin_inset Quotes erd
\end_inset
.
The strength of the MI values to each of the connectors might be taken
as a judgement of how much that attribute is prototypical of the object
(see other section on
\begin_inset Quotes eld
\end_inset
prototype theory
\begin_inset Quotes erd
\end_inset
).
\end_layout
\begin_layout Standard
The disjuncts associated with
\begin_inset Quotes eld
\end_inset
balloon
\begin_inset Quotes erd
\end_inset
will also connect to a verb.
These verb connectors may be taken as another set of intentional attributes,
for example {floats, drifts, rose, popped}.
It should be possible to distinguish these as an orthogonal set of attributes,
in that one might observe
\begin_inset Quotes eld
\end_inset
the- red- floats+
\begin_inset Quotes erd
\end_inset
and
\begin_inset Quotes eld
\end_inset
the- red- drifts+
\begin_inset Quotes erd
\end_inset
but never observe
\begin_inset Quotes eld
\end_inset
floats- drifts+
\begin_inset Quotes erd
\end_inset
.
\end_layout
\begin_layout Standard
Meaning bibliography:
\end_layout
\begin_layout Itemize
\begin_inset Quotes eld
\end_inset
The Molecular Level of Lexical Semantics
\begin_inset Quotes erd
\end_inset
, EA Nida, (1997) International Journal of Lexicography, 10(4): 265–274.
https://www.academia.edu/36534355/The_Molecular_Level_of_Lexical_Semantics_by_EA_
Nida
\end_layout
\begin_layout Section*
Vector Algebra
\end_layout
\begin_layout Standard
Here's the deal ...
vectors.
\end_layout
\begin_layout Subsection*
Word vectors
\end_layout
\begin_layout Standard
The last section introduces the idea of decomposable basis vectors.
Let try to make this more precise (a cleaned-up version of this and above
belongs in the (unfinished) sheaf paper.).
First, review the notion of a vector, and then show how to make vectors
from word-disjuncts, N-grams, and skip-grams.
Then review the idea of tensoring.
\end_layout
\begin_layout Standard
So, the textbook standard definition of a vector
\begin_inset Formula $\vec{v}$
\end_inset
is
\begin_inset Formula
\[
\vec{v}=a_{1}\widehat{e}_{1}+a_{2}\widehat{e}_{2}+\cdots+a_{n}\widehat{e}_{n}
\]
\end_inset
The
\begin_inset Formula $a_{k}$
\end_inset
are conventionally taken to be numbers; here, real numbers.
The
\begin_inset Formula $\widehat{e}_{k}$
\end_inset
are called
\begin_inset Quotes eld
\end_inset
basis vectors
\begin_inset Quotes erd
\end_inset
.
They have the property of having unit length; that is
\begin_inset Formula $\left\Vert \widehat{e}_{k}\right\Vert =1$
\end_inset
.
For every word
\begin_inset Formula $w$
\end_inset
, one can create several kinds of vectors.
A widely-known vector is the
\begin_inset Quotes eld
\end_inset
N-gram
\begin_inset Quotes erd
\end_inset
, where each
\begin_inset Formula $\widehat{e}_{k}$
\end_inset
corresponds to the N-word context in which the word
\begin_inset Formula $w$
\end_inset
was observed, and
\begin_inset Formula $a_{k}$
\end_inset
is the count of the number of observations.
Thus, for example, given the toy corpus: "
\emph on
John knew the bird was there.
John heard the bird.
John saw the bird.
Susan saw the bird too.
Mary saw it also
\emph default
".
Setting
\begin_inset Formula $w=bird$
\end_inset
and N=3, and the requirement that the word be in the middle of the N-gram,
one has
\begin_inset Formula
\begin{align*}
\widehat{e}_{1} & =\left[the\;*\;was\right]\\
\widehat{e}_{2} & =\left[the\;*\;.\right]\\
\widehat{e}_{3} & =\left[the\;*\;too\right]
\end{align*}
\end_inset
and observation counts
\begin_inset Formula $a_{1}=1$
\end_inset
,
\begin_inset Formula $a_{2}=2$
\end_inset
,
\begin_inset Formula $a_{3}=1$
\end_inset
.
Here,
\begin_inset Formula $a_{2}=2$
\end_inset
because there are two sentences containing the word-sequence
\begin_inset Quotes eld
\end_inset
the bird.
\begin_inset Quotes erd
\end_inset
In more compact form, one may write
\begin_inset Formula
\[
\overrightarrow{bird}=1\left[the\;*\;was\right]+2\left[the\;*\;.\right]+1\left[the\;*\;too\right]
\]
\end_inset
This can be extended in an obvious way to a larger corpus, to a larger N
and one can drop the requirement that the word be in the middle.
Skip-grams are similar, but allow some words to be skipped, and have a
complex algorithm to determine when and how words can be skipped.
\end_layout
\begin_layout Standard
Disjuncts behave in a very similar manner.
The word-pair counting and the MST pipeline creates disjuncts and the associate
d counts.
For the above corpus, the disjuncts might possibly resemble these:
\begin_inset Formula
\begin{align*}
\widehat{e}_{1} & =the\negthinspace-\;\&\;was\negthinspace+\\
\widehat{e}_{2} & =the\negthinspace-\;\&\;.\negthinspace+\\
\widehat{e}_{3} & =the\negthinspace-\;\&\;too\negthinspace+
\end{align*}
\end_inset
These disjuncts are essentially identical to the N-gram example above; only
the notation is different.
One might hope that the disjuncts coming out from the MST pipeline might
actually be of higher quality.
The support for this hope is several decades of research results on MST
parsing of natural language.
The point here is that N-grams, skip-grams and word-disjunct vectors are
quite similar to one another, and that, perhaps, word-disjunct vectors
are just a tiny bit more linguistically accurate.
\end_layout
\begin_layout Standard
What does one do with vectors? Well, one can apply any vector-based algorithm
you can wrap your mind around.
Some classics are clustering; SVM or K-means.
Here, one commonly starts with a vector dot-product, leading to a cosine-distan
ce metric.
Given two vectors
\begin_inset Formula $\vec{a}=a_{1}\widehat{e}_{1}+a_{2}\widehat{e}_{2}+\cdots+a_{n}\widehat{e}_{n}$
\end_inset
and
\begin_inset Formula $\vec{b}=b_{1}\widehat{e}_{1}+b_{2}\widehat{e}_{2}+\cdots+b_{n}\widehat{e}_{n}$
\end_inset
, the dot-product is given by
\begin_inset Formula
\[
\vec{a}\cdot\vec{b}=a_{1}b_{1}+a_{2}b_{2}+\cdots+a_{n}b_{n}
\]
\end_inset
The cosine angle between these is given by
\begin_inset Formula
\[
\left\Vert \vec{a}\right\Vert \left\Vert \vec{b}\right\Vert \cos\theta=\vec{a}\cdot\vec{b}
\]
\end_inset
where, as usual, the
\begin_inset Formula $l_{2}$
\end_inset
Hilbert-space norm is used:
\begin_inset Formula $\left\Vert \vec{a}\right\Vert =\sqrt{\vec{a}\cdot\vec{a}}$
\end_inset
.
The closer that
\begin_inset Formula $\theta$
\end_inset
is to zero, the more parallel the two vectors are.
There are plenty of other interesting ways of measuring the similarity
of vectors.
The point here is that if the corpus included the sentences
\begin_inset Quotes eld
\end_inset
\emph on
John heard the crow.
John saw the crow.
\emph default
\begin_inset Quotes erd
\end_inset
and one obtained a vector for
\begin_inset Formula $\overrightarrow{crow}$
\end_inset
, one might typically discover that
\begin_inset Formula $\overrightarrow{crow}$
\end_inset
is similar to
\begin_inset Formula $\overrightarrow{bird}$
\end_inset
: the cosine angle gives a concrete technology for measuring similarity.
\end_layout
\begin_layout Standard
Suppose one has a large number of vectors for a large number of words.
Similarity metrics that assign a number to the informal idea of similarity
then opens the door to automatically classifying words into bins, according
to their similarity.
A common favorite is the K-means clustering algorithm, which can take a
large number of vectors, and assign them to one of K different bins, with
all words in the same bin being quite 'similar'.
One can then make general hand-waving arguments that these clusters correspond
to 'grammatical classes'.
The actual data, the actual results make this obvious: a quick skim of
the clusters makes it clear that some clusters are dominated by nous, and
others by verbs.
\end_layout
\begin_layout Standard
There are other approaches, too.
Popular ones are the various deep learning approaches applied to neural
nets.
Word2Vec is a particularly famous one; its now a tensorflow tutorial.
The algorithmic details are interesting, and sophisticated.
However, conceptually, its not really all that different from what is described
above: one obtains a collection of vectors, and then one pumps the vectors
through a neural-net classifier, to obtain word-classes.
\end_layout
\begin_layout Standard
The entire point of this section is to drive home the idea that, whatever
one can do with N-grams or with skip-grams, one can also do with word-disjunct
vectors.
At the appropriate conceptual level, they are really very, very similar
to one-another.
\end_layout
\begin_layout Standard
As a minor side-point, one can hold out some hope that perhaps word-disjunct
vectors are of (slightly) higher quality than skip-grams.
It seems like they could be or should be, as reinforced by decades of MST-lingu
istics results.
If one takes some stock, off-the-shelf linguistics-K-means algorithm, or
some stock, off-the-shelf Word2vec algorithm, and instead plugs in word-disjunc
t vectors wherever the skip-grams or N-grams might go, then what happens?
Maybe, possibly, one might get results that are an itsy-bitsy teensy-weensy
bit better.
Maybe.
This is not known: there are no published results comparing these.
It is reasonable (to me) to expect that the results would be quite similar,
with perhaps word-disjunct vectors being just a smidgen higher-quality.
\end_layout
\begin_layout Standard
Due diligence suggests that such head-to-head comparisons should be performed
and published.
\end_layout
\begin_layout Standard
A later sections describes how one can do much better than naive clustering,
whether Word2vec or K-means.
But first, some additional basic issues need to be reviewed.
\end_layout
\begin_layout Subsection*
Merging vectors
\end_layout
\begin_layout Standard
In the above example, it seems obvious that
\begin_inset Quotes eld
\end_inset
\emph on
bird
\emph default
\begin_inset Quotes erd
\end_inset
and
\begin_inset Quotes eld
\end_inset
\emph on
crow
\emph default
\begin_inset Quotes erd
\end_inset
should be merged: they are grammatically similar, given their vectors.
So lets merge them into a class of
\begin_inset Quotes eld
\end_inset
\emph on
THINGS
\emph default
\begin_inset Quotes erd
\end_inset
.
How should that class be created, formally? This is a non-trivial question,
since additional merges affect the class.
Suppose, for example, the corupus contained the sentence
\begin_inset Quotes eld
\end_inset
\emph on
John saw the book
\emph default
\begin_inset Quotes erd
\end_inset
.
Is it plausible to merge
\begin_inset Quotes eld
\end_inset
book
\begin_inset Quotes erd
\end_inset
into
\begin_inset Quotes eld
\end_inset
THINGS
\begin_inset Quotes erd
\end_inset
? If so, then how? One way is to compute the cosine similarity
\begin_inset Formula $\cos\left(\overrightarrow{book},\overrightarrow{bird}\right)$
\end_inset
and
\begin_inset Formula $\cos\left(\overrightarrow{book},\overrightarrow{crow}\right)$
\end_inset
and merge only if both are large enough.
The other is to compute
\begin_inset Formula $\cos\left(\overrightarrow{book},\overrightarrow{THINGS}\right)$
\end_inset
but this requires the vector
\begin_inset Formula $\overrightarrow{THINGS}$
\end_inset
.
\end_layout
\begin_layout Standard
There are several ways to create this vector.
One way is by explicit vector addition
\begin_inset Formula
\[
\overrightarrow{THINGS}_{sum}=\overrightarrow{bird}+\overrightarrow{crow}
\]
\end_inset
This seems to be an adequate way of creating this grammatical class.
Recall that
\begin_inset Formula
\[
\overrightarrow{bird}=1\left|the\negthinspace-\;\&\;was\negthinspace+\right\rangle +2\left|the\negthinspace-\;\&\;.\negthinspace+\right\rangle +1\left|the\negthinspace-\;\&\;too\negthinspace+\right\rangle
\]
\end_inset
where we switched to disjunct notation.
The vector for
\begin_inset Formula $\overrightarrow{crow}$
\end_inset
is
\end_layout
\begin_layout Standard
\begin_inset Formula
\[
\overrightarrow{crow}=2\left|the\negthinspace-\;\&\;.\negthinspace+\right\rangle
\]
\end_inset
Comparing these two, directly like this, suggests that other merge strategies
are possible.
For example, since
\begin_inset Formula $\overrightarrow{crow}$
\end_inset
has neither
\begin_inset Formula $\left|the-\&was+\right\rangle $
\end_inset
nor
\begin_inset Formula $\left|the-\&too+\right\rangle $
\end_inset
on it, maybe these don't belong on the merged class: perhaps an intersection
of the basis vectors should be used, so that the combined vector is non-zero
only on the intersection of the basis elements.
Thus, perhaps a better definition would be
\begin_inset Formula
\[
\overrightarrow{THINGS}_{intersect}=4\left|the\negthinspace-\;\&\;.\negthinspace+\right\rangle
\]
\end_inset
This is narrow definition is nice, because we have that
\begin_inset Formula
\[
\overrightarrow{book}=1\left|the\negthinspace-\;\&\;.\negthinspace+\right\rangle
\]
\end_inset
and so clearly the cosine between
\begin_inset Formula $\overrightarrow{THINGS}_{intersect}$
\end_inset
and
\begin_inset Formula $\overrightarrow{book}$
\end_inset
is one, and they're mergeable.
By contrast, the cosine between
\begin_inset Formula $\overrightarrow{THINGS}_{sum}$
\end_inset
and
\begin_inset Formula $\overrightarrow{book}$
\end_inset
is much less than one - which also makes sense, because books are not very
bird-like.
\end_layout
\begin_layout Standard
Another possibility is to combine some fraction
\begin_inset Formula $0\le\alpha\le1$
\end_inset
of the basis elements that don't overlap.
This suggests a grammatical class of
\begin_inset Formula
\[
\overrightarrow{THINGS}_{\alpha}=\alpha\left|the\negthinspace-\;\&\;was\negthinspace+\right\rangle +4\left|the\negthinspace-\;\&\;.\negthinspace+\right\rangle +\alpha\left|the\negthinspace-\;\&\;too\negthinspace+\right\rangle
\]
\end_inset
so that
\begin_inset Formula
\[
\overrightarrow{THINGS}_{\alpha=0}=\overrightarrow{THINGS}_{intersect}
\]
\end_inset
while
\begin_inset Formula
\[
\overrightarrow{THINGS}_{\alpha=1}=\overrightarrow{THINGS}_{sum}
\]
\end_inset
That is, for
\begin_inset Formula $\alpha=1$
\end_inset
,
\begin_inset Formula $\overrightarrow{THINGS}_{\alpha}$
\end_inset
is rather bird-like, whereas for
\begin_inset Formula $\alpha=0$
\end_inset
, its more generic.
\end_layout
\begin_layout Standard
Note that
\begin_inset Formula $\overrightarrow{THINGS}_{\alpha}$
\end_inset
is not a linear combination of
\begin_inset Formula $\overrightarrow{bird}$
\end_inset
and
\begin_inset Formula $\overrightarrow{crow}$
\end_inset
.
It is non-linear, and, depending on the actual corpus, could be highly
non-linear.
\end_layout
\begin_layout Standard
The choice of merge strategy alters the contents of the grammatical clusters.
This is not a surprise; the goal here is to illustrate that a number of
different merge strategies are possible.
it is not
\emph on
a priori
\emph default
obvious which is the best.
\end_layout
\begin_layout Subsection*
Merge Strategies and Semantics
\end_layout
\begin_layout Standard
The different strategies for merging vectors lead in a natural way to the
automated discovery of word-meaning.
This can be illustrated by example; although a different example from the
above is needed.
\end_layout
\begin_layout Standard
Consider the word
\begin_inset Quotes eld
\end_inset
\emph on
saw
\emph default
\begin_inset Quotes erd
\end_inset
.
After observing a sufficient amount of text, one will find a vector
\begin_inset Formula $\overrightarrow{saw}$
\end_inset
that contains disjuncts for appropriate for usages as a "cutting tool",
"the verb cut", and "the past tense of to see".
Suppose that there is an existing class
\begin_inset Formula $\overrightarrow{TOOLS}$
\end_inset
, consisting of nouns, and that the similarity measure judges that
\begin_inset Formula $\overrightarrow{saw}$
\end_inset
and
\begin_inset Formula $\overrightarrow{TOOLS}$
\end_inset
are similar.
Clearly, a merge strategy of expanding an existing cluster by linear addition
is incorrect.
The expanded class
\begin_inset Formula
\[
\overrightarrow{TOOLS}_{linear-expand}=\overrightarrow{TOOLS}+\overrightarrow{saw}
\]
\end_inset
is incorrect, or at least, not very good, as it now contains disjuncts for
"the verb cut" and "the past tense of to see".
Two bad things happen: (a) the noun cluster is polluted with verb-vector
components, and (b) the vector has not been factorized, and so
\begin_inset Quotes eld
\end_inset
\emph on
saw
\emph default
\begin_inset Quotes erd
\end_inset
cannot also be placed into other clusters as well.
One looses the ability to distinguish different semantic meanings based
on grammatical usage.
\end_layout
\begin_layout Standard
We know that,
\emph on
a priori
\emph default
, the correct way of thinking about
\begin_inset Formula $\overrightarrow{saw}$
\end_inset
is that it has the components
\end_layout
\begin_layout Standard
\begin_inset Formula
\[
\overrightarrow{saw}=\vec{v}_{tool}+\vec{v}_{past-tense}+\vec{v}_{cutting}
\]
\end_inset
However, we do not know (
\emph on
a priori
\emph default
) what the components
\begin_inset Formula $\vec{v}_{tool}$
\end_inset
,
\begin_inset Formula $\vec{v}_{past-tense}$
\end_inset
and
\begin_inset Formula $\vec{v}_{cutting}$
\end_inset
are.
A good merge strategy would be able to factorize them out, to discover
these automatically.
\end_layout
\begin_layout Standard
The best merge strategy is hardly obvious.
The three vectors
\begin_inset Formula $\vec{v}_{tool}$
\end_inset
,
\begin_inset Formula $\vec{v}_{past-tense}$
\end_inset
and
\begin_inset Formula $\vec{v}_{cutting}$
\end_inset
are not orthogonal (or rather, in general, won't be), so even if there
were pre-existing grammatical classes for these three, an orthogonal decompose
ion of
\begin_inset Formula $\overrightarrow{saw}$
\end_inset
into components is not unique.
What's more, in early stages, a class for
\begin_inset Formula $\overrightarrow{TOOLS}$
\end_inset
might exist, but not yet one for
\begin_inset Formula $\overrightarrow{CUTTING}$
\end_inset
; its not even clear that a class of the form
\begin_inset Formula $\overrightarrow{CUTTING}$
\end_inset
might even be formable.
\end_layout
\begin_layout Standard
There are several possible non-linear merge strategies that are possible.
One is given in the next section.
However, the point here is not to propose the best-posible merge strategy,
but rather to point out that this is where word-sense disambiguation comes
from, and that this is how it can be done.
Different word-senses are already encoded in how words are used in sentences.
Word-senses can be distinguished by paying attention to the disjuncts attached
on a word.
Picking the correct merge strategy picks out the word-senses.
Picking the wrong strategy blurs them all together.
\end_layout
\begin_layout Subsection*
An example non-linear merge strategy
\end_layout
\begin_layout Standard
The earlier section illustrated a non-linear merge, using
\begin_inset Formula $\overrightarrow{THINGS}_{\alpha}$
\end_inset
as an example.
This section formalizes that example.
The result is a maybe-OK merge strategy, but not an obviously great one.
Its worth writing down only because its simple.
\end_layout
\begin_layout Standard
The idea starts with the computation of the intersection and the union of
the support for the vectors to be merged, and then taking a fraction
\begin_inset Formula $0\le\alpha\le1$
\end_inset
on the union of the support when merging.
This merge strategy seems to at least partly overcome the problem of the
erasure of word-senses noted above.
\end_layout
\begin_layout Standard
Suppose
\begin_inset Formula $\overrightarrow{CLASS}$
\end_inset
is some existing word-class, and
\begin_inset Formula $\overrightarrow{word}$
\end_inset
is a word that was judged
\begin_inset Quotes eld
\end_inset
mergeable
\begin_inset Quotes erd
\end_inset
into
\begin_inset Formula $\overrightarrow{CLASS}$
\end_inset
.
Define the support of a vector
\begin_inset Formula $\vec{v}$
\end_inset
as
\begin_inset Formula
\[
\mathcal{I}_{\vec{v}}=\left\{ \widehat{e}_{k}\mbox{ such that }\vec{v}=a_{1}\widehat{e}_{1}+\cdots+a_{n}\widehat{e}_{n}\mbox{ and }a_{k}\ne0\right\}
\]
\end_inset
The common support for
\begin_inset Formula $\overrightarrow{CLASS}$
\end_inset
and
\begin_inset Formula $\overrightarrow{word}$
\end_inset
would then be
\begin_inset Formula
\[
\mathcal{I}_{\overrightarrow{CLASS}}\,\cap\,\mathcal{I}_{\overrightarrow{word}}
\]
\end_inset
while the union would be
\end_layout
\begin_layout Standard
\begin_inset Formula
\[
\mathcal{I}_{\overrightarrow{CLASS}}\,\cup\,\mathcal{I}_{\overrightarrow{word}}
\]
\end_inset
with
\begin_inset Formula $\cap$
\end_inset
and
\begin_inset Formula $\cup$
\end_inset
denoting set-intersection and set-union, respectively.
A plausible merge strategy for
\begin_inset Formula $\vec{v}=a_{1}\widehat{e}_{1}+\cdots+a_{n}\widehat{e}_{n}$
\end_inset
and
\begin_inset Formula $\vec{w}=b_{1}\widehat{e}_{1}+\cdots+b_{n}\widehat{e}_{n}$
\end_inset
might be
\begin_inset Formula
\[
\mbox{merge}\left(\vec{v},\vec{w},\alpha\right)=c_{1}\widehat{e}_{1}+\cdots+c_{n}\widehat{e}_{n}
\]
\end_inset
with
\begin_inset Formula
\[
c_{k}=\begin{cases}
a_{k}+b_{k} & \mbox{if }\widehat{e}_{k}\in\mathcal{I}_{\overrightarrow{v}}\\
\alpha b_{k} & \mbox{if }\widehat{e}_{k}\notin\mathcal{I}_{\overrightarrow{v}}
\end{cases}
\]
\end_inset
This is designed so that, if
\begin_inset Formula $\vec{v}=\overrightarrow{CLASS}$
\end_inset
and
\begin_inset Formula $\vec{w}=\overrightarrow{word}$
\end_inset
then those disjuncts in
\begin_inset Formula $\overrightarrow{word}$
\end_inset
that are already in
\begin_inset Formula $\overrightarrow{CLASS}$
\end_inset
are folded in, in full, while disjuncts from
\begin_inset Formula $\overrightarrow{word}$
\end_inset
that are not yet in
\begin_inset Formula $\overrightarrow{CLASS}$
\end_inset
are only folded in a little bit, thus expanding the support for
\begin_inset Formula $\overrightarrow{CLASS}$
\end_inset
, but not putting a lot of weight into the expanded support.
\end_layout
\begin_layout Standard
If
\begin_inset Formula $\vec{v}$
\end_inset
is a word, and not a class, and one wishes to for a class for the first
time, out of two words, one might instead choose
\begin_inset Formula
\[
c_{k}=\begin{cases}
a_{k}+b_{k} & \mbox{if }\widehat{e}_{k}\in\mathcal{I}_{\overrightarrow{v}}\cap\mathcal{I}_{\vec{w}}\\
\alpha\left(a_{k}+b_{k}\right) & \mbox{otherwise}
\end{cases}
\]
\end_inset
What should the 'constant'
\begin_inset Formula $\alpha$
\end_inset
be? Certainly, a small
\begin_inset Formula $\alpha$
\end_inset
is very conservative, not expanding the meaning very much beyond the overlappin
g set.
Perhaps a variable
\begin_inset Formula $\alpha$
\end_inset
would be better; for example, taking
\begin_inset Formula $\alpha=\left(\cos\left(\vec{v},\vec{w}\right)-\beta\right)/\left(1-\beta\right)$
\end_inset
where
\begin_inset Formula $\beta$
\end_inset
is the smallest cosine that allows merging.
This makes the merges more conservative, the more different the words are,
but is very accepting when the words are similar.
\end_layout
\begin_layout Standard
To avoid expanding the support in unwise ways, perhaps it would be best
to assemble a set of mergeable words, first, and only then merge them all
together, in one shot, rather than merging pair-wise.
\end_layout
\begin_layout Standard
The merge strategy here seems OK, but probably not great.
A better merge strategy is proposed below.
The point here is that merge strategies that do perform word-sense disambiguati
on (that is, semantic extraction) are in principle possible.
\end_layout
\begin_layout Subsection*
Syntactic Broadening and Generalization
\end_layout
\begin_layout Standard
The act of merging together also broadens or generalizes the syntactic structure
of the language.
It extracts grammatical generalities in addition to semantic particulars.
This is again best illustrated by example.
\end_layout
\begin_layout Standard
Take up the previous example, of a nature scene with birds, and consider
the verbs
\begin_inset Quotes eld
\end_inset
\emph on
saw
\emph default
\begin_inset Quotes erd
\end_inset
and
\begin_inset Quotes eld
\end_inset
\emph on
heard
\emph default
\begin_inset Quotes erd
\end_inset
– John, Susan and others are seeing and hearing birds and crows.
The word-vectors for
\begin_inset Quotes eld
\end_inset
\emph on
saw
\emph default
\begin_inset Quotes erd
\end_inset
and
\begin_inset Quotes eld
\end_inset
\emph on
heard
\emph default
\begin_inset Quotes erd
\end_inset
will include the disjuncts:
\begin_inset Formula
\begin{align*}
\widehat{e}_{4} & =\left|John\negthinspace-\;\&\;bird\negthinspace+\right\rangle \\
\widehat{e}_{5} & =\left|Susan\negthinspace-\;\&\;bird\negthinspace+\right\rangle \\
\widehat{e}_{6} & =\left|John\negthinspace-\;\&\;crow\negthinspace+\right\rangle
\end{align*}
\end_inset
Working explicitly from this example, one would have the vector
\begin_inset Formula
\[
\overrightarrow{saw}=1\left|John\negthinspace-\;\&\;bird\negthinspace+\right\rangle +1\left|Susan\negthinspace-\;\&\;bird\negthinspace+\right\rangle +1\left|John\negthinspace-\;\&\;crow\negthinspace+\right\rangle
\]
\end_inset
\end_layout
\begin_layout Standard
If
\begin_inset Quotes eld
\end_inset
\emph on
bird
\emph default
\begin_inset Quotes erd
\end_inset
and
\begin_inset Quotes eld
\end_inset
\emph on
crow
\emph default
\begin_inset Quotes erd
\end_inset
are merged into the grammatical class
\begin_inset Quotes eld
\end_inset
\emph on
THINGS
\emph default
\begin_inset Quotes erd
\end_inset
, then none of the basis vectors
\begin_inset Formula $\widehat{e}_{4}$
\end_inset
,
\begin_inset Formula $\widehat{e}_{5}$
\end_inset
or
\begin_inset Formula $\widehat{e}_{6}$
\end_inset
are entirely appropriate any more.
They should be replaced by basis vectors made of the combined class; possibly,
for example,
\end_layout
\begin_layout Standard
\begin_inset Formula
\[
\overrightarrow{saw}=1\left|John\negthinspace-\;\&\;THINGS\negthinspace+\right\rangle +1\left|Susan\negthinspace-\;\&\;THINGS\negthinspace+\right\rangle
\]
\end_inset
But there's something odd in the above: Susan never saw a crow, in the sample
corpus; so the above has certainly enlarged or generalized the class of
things that ar see-able, at least with respect to Susan.
\end_layout
\begin_layout Standard
That is, given specific examples, the act of deducing grammatical categories
enlarges the generative size of the grammar.
This is a technical statement, and needs some exmplanation.
Formally, a 'grammar' is a collection of rules that define how sentences
can be parsed.
A 'language' is the set of all possible sentences that the grammar allows.
The original corpus is a sampling of the language; deduction of the class
of
\begin_inset Quotes eld
\end_inset
\emph on
THINGS
\emph default
\begin_inset Quotes erd
\end_inset
expands the language, because, in this expanded grammar, the sentence
\begin_inset Quotes eld
\end_inset
\emph on
Susan saw the crow
\emph default
\begin_inset Quotes erd
\end_inset
is now allowed, is part of the language, whereas previously it was not.
\end_layout
\begin_layout Subsection*
Parse Ranking
\end_layout
\begin_layout Standard
In the above, the expansion of the grammar from
\begin_inset Formula
\[
saw:\,Susan\negthinspace-\;\&\;bird\negthinspace+;
\]
\end_inset
to
\begin_inset Formula
\[
saw:\,Susan\negthinspace-\;\&\;THINGS\negthinspace+;
\]
\end_inset
was implied to be an all-or-nothing act.
In fact, one can apply a likilihood or probability to this expansion, with
the intent of keeping undesirable expansion at bay; of minimizing the risk
of incorrect expansion.
This is done by replacing the lexical set that is the grammatical class
with a proabilisitically-weighted set.
That is, instead of writing
\begin_inset Formula
\[
THINGS:\;\left\{ bird,\,crow\right\}
\]
\end_inset
as a named set, one could introduce a probability or cost:
\begin_inset Formula
\[
THINGS:\;\left\{ \left(bird,h_{bird}\right),\,\left(crow,h_{crow}\right)\right\}
\]
\end_inset
with
\begin_inset Formula $h_{crow}=-\log_{2}p\left(crow\right)$
\end_inset
and likewise for
\begin_inset Formula $h_{bird}$
\end_inset
.
The logarithm is used to keep the costs additive, instead of multiplicative;
this is a detail that tends to simplify things.
The value
\begin_inset Formula $p\left(crow\right)$
\end_inset
is a pseudo-probability; there are several plausible ways to define it.
One might consider, for example, setting
\begin_inset Formula
\[
p\left(crow\right)=\cos\left(\overrightarrow{THINGS},\overrightarrow{crow}\right)
\]
\end_inset
that indicates how well
\begin_inset Quotes eld
\end_inset
\emph on
crow
\emph default
\begin_inset Quotes erd
\end_inset
aligned with the class of
\begin_inset Quotes eld
\end_inset
\emph on
THINGS
\emph default
\begin_inset Quotes erd
\end_inset
.
\end_layout
\begin_layout Standard
But this is naive – too naive: the cosine product
\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
\begin_inset Formula $\cos\left(\overrightarrow{THINGS},\overrightarrow{crow}\right)$
\end_inset
was derived from vectors that used the grammatical context of
\begin_inset Quotes eld
\end_inset
\family default
\series default
\shape default
\size default
\emph on
\bar default
\strikeout default
\uuline default
\uwave default
\noun default
\color inherit
crow
\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
\begin_inset Quotes erd
\end_inset
and
\begin_inset Quotes eld
\end_inset
\family default
\series default
\shape default
\size default
\emph on
\bar default
\strikeout default
\uuline default
\uwave default
\noun default
\color inherit
bird
\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
\begin_inset Quotes erd
\end_inset
.
It did NOT take into account that, in the corpus, Susan had not yet seen
any birds.
Its as if the vector for
\begin_inset Formula $\overrightarrow{crow}$
\end_inset
is incorrect – it should be taking Susan into account, in some way, but
its not, because the word
\begin_inset Quotes eld
\end_inset
Susan
\begin_inset Quotes erd
\end_inset
is too far away in the sentence.
For the disjuncts,
\begin_inset Quotes eld
\end_inset
\family default
\series default
\shape default
\size default
\emph on
\bar default
\strikeout default
\uuline default
\uwave default
\noun default
\color inherit
Susan
\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
\begin_inset Quotes erd
\end_inset
is the subject of a verb,
\begin_inset Quotes eld
\end_inset
\family default
\series default
\shape default
\size default
\emph on
\bar default
\strikeout default
\uuline default
\uwave default
\noun default
\color inherit
crow
\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
\begin_inset Quotes erd
\end_inset
is the object, and there is no direct linkage of subject and object.
For N-grams,
\begin_inset Quotes eld
\end_inset
\family default
\series default
\shape default
\size default
\emph on
\bar default
\strikeout default
\uuline default
\uwave default
\noun default
\color inherit
Susan
\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
\begin_inset Quotes erd
\end_inset
is simply too far away from the object, unless N is raised to an obscenely
large value.
Skip-grams do not materially improve on this issue.
In all cases, the subject is too far away removed.
\end_layout
\begin_layout Standard
\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
Can this be remedied? Yes, it can.
The key observation will be that there are other (kinds of) vectors, besides
the vectors given above.
All of the vectors discussed so far are best considered to be 'naive' vectors.
They are 'naive' because they ignore the structure of the basis elements
themselves.
Acknowledging this structure allows one to move beyond the narrow constraints
of linear algebra.
Acknowledging this structure leads to the concept of a
\begin_inset Quotes eld
\end_inset
sheaf on a graph
\begin_inset Quotes erd
\end_inset
as a lingustically appropriate generalization of a vector space; it will
replace the vector spaces, and the linear algebra, by a more general concept.
This is, roughly, one of collections of vector spaces, sewn together at
the edges.
The word
\begin_inset Quotes eld
\end_inset
sheaf
\begin_inset Quotes erd
\end_inset
comes from the idea that the axioms defining how the sewing-together is
to be performed are the same axioms as those of
\begin_inset Quotes eld
\end_inset
sheaf theory
\begin_inset Quotes erd
\end_inset
.
But first, before we get to that, some more examples need to be developed.
\end_layout
\begin_layout Section*
Disjuncts are Tensors
\end_layout
\begin_layout Standard
The issue with
\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none
'naive' vectors is that they bring the story to a close.
\family default
\series default
\shape default
\size default
\emph default
\bar default
\strikeout default
\uuline default
\uwave default
\noun default
\color inherit
In ordinary linear algebra, the basis vectors
\begin_inset Formula $\widehat{e}_{k}$
\end_inset
are indecomposable, structure-less, atomic.
Nothing more can be said about them.
However, in linguistics, the basis vectors
\begin_inset Formula $\widehat{e}_{k}$
\end_inset
are not structure-less; they are made out of words! And that changes everything.
\end_layout
\begin_layout Standard
The concept of word-vectors, such as N-grams, skip-grams or word-disjunct
vectors, is just fine, but needs to be recognized as a flawed oversimplificatio
n for the structure of language.
They're reasonably OK concepts, for a first pass, giving OK results.
Clearly, word2vec and it's cousins have made a big impression on the industry.
That counts for something and should not be dismissed.
But (I believe that) one can do better by not ignoring the structure of
the basis vectors.
These ideas are developed next.
\end_layout
\begin_layout Standard
An earlier section wrote down the basis vector
\end_layout
\begin_layout Standard
\begin_inset Formula
\[
\widehat{e}_{4}=\left|John\negthinspace-\;\&\;crow\negthinspace+\right\rangle
\]
\end_inset
The notation used in writing this was modelled on the usual Link Grammar
notation for disjuncts.
A tensor-style notation would be to write
\begin_inset Formula
\[
\widehat{e}_{4}=\left|John\negthinspace-\right\rangle \otimes\left|crow\negthinspace+\right\rangle
\]
\end_inset
which makes it perhaps a bit more clear that
\begin_inset Formula $\widehat{e}_{4}$
\end_inset
consists of several parts.
When the 'naive' word-vectors were constructed, these parts were ignored.
In particular, their contribution to the similarity was ignored.
The can be remedied by defining a different kind of vector, the cross-connector
vector.
\end_layout
\begin_layout Subsection*
Cross-connector vectors
\end_layout
\begin_layout Standard
Returning to the nature-watching example, the corpus of seven sentences
produces a dataset of observed disjuncts that includes the following:
\begin_inset Formula
\begin{align*}
knew: & John\negthinspace-\;\&\;bird\negthinspace+;\\
heard: & John\negthinspace-\;\&\;bird\negthinspace+;\\
saw: & John\negthinspace-\;\&\;bird\negthinspace+;\\
saw: & Susan\negthinspace-\;\&\;bird\negthinspace+;\\
heard: & John\negthinspace-\;\&\;crow\negthinspace+;\\
saw: & John\negthinspace-\;\&\;crow\negthinspace+;
\end{align*}
\end_inset
All of the above are observed exactly once, each.
This is in addition to the observations
\begin_inset Formula
\begin{align*}
bird: & the\negthinspace-\;\&\;was\negthinspace+;\\
bird: & the\negthinspace-\;\&\;.\negthinspace+;\qquad\mbox{ (seen twice)}\\
bird: & the\negthinspace-\;\&\;too\negthinspace+;\\
crow: & the\negthinspace-\;\&\;.\negthinspace+;\qquad\mbox{ (seen twice)}
\end{align*}
\end_inset
\end_layout
\begin_layout Standard
We take up the question, again, can the words
\begin_inset Quotes eld
\end_inset
\emph on
bird
\emph default
\begin_inset Quotes erd
\end_inset
and
\begin_inset Quotes eld
\end_inset
\emph on
crow
\emph default
\begin_inset Quotes erd
\end_inset
be merged into a single common grammatical class? Previously, this determinatio
n was made, using the 'naive' word-disjunct vectors
\end_layout
\begin_layout Standard
\begin_inset Formula
\[
\overrightarrow{bird}_{naive}=1\left|the\negthinspace-\;\&\;was\negthinspace+\right\rangle +2\left|the\negthinspace-\;\&\;.\negthinspace+\right\rangle +1\left|the\negthinspace-\;\&\;too\negthinspace+\right\rangle
\]
\end_inset
and
\end_layout
\begin_layout Standard
\begin_inset Formula
\[
\overrightarrow{crow}_{naive}=2\left|the\negthinspace-\;\&\;.\negthinspace+\right\rangle
\]
\end_inset
The verb constructions allow a different kind of vector, that
\begin_inset Quotes eld
\end_inset
crosses over
\begin_inset Quotes erd
\end_inset
:
\begin_inset Formula
\begin{align*}
\overrightarrow{bird}_{obj}= & 1\left|knew:John\negthinspace-\;\&\;*\negthinspace+\right\rangle +1\left|heard:John\negthinspace-\;\&\;*\negthinspace+\right\rangle \\
& \quad+1\left|saw:John\negthinspace-\;\&\;*\negthinspace+\right\rangle +1\left|saw:Susan\negthinspace-\;\&\;*\negthinspace+\right\rangle
\end{align*}
\end_inset
The wild-card * is an explicit place-holder for the word in the disjunct.
The corresponding vector for
\begin_inset Quotes eld
\end_inset
\emph on
crow
\emph default
\begin_inset Quotes erd
\end_inset
would be
\begin_inset Formula
\[
\overrightarrow{crow}_{obj}=1\left|knew:John\negthinspace-\;\&\;*\negthinspace+\right\rangle +1\left|heard:John\negthinspace-\;\&\;*\negthinspace+\right\rangle
\]
\end_inset
Both of these are again vectors, but very different in form and shape from
before.
\end_layout
\begin_layout Standard
The mergability decision is going to be different, as a result.
Consider, for example, the cosines.
Computing explicitly:
\begin_inset Formula
\[
\cos\left(\overrightarrow{bird}_{naive},\overrightarrow{crow}_{naive}\right)=\frac{4}{\sqrt{6\cdot4}}=\frac{2}{\sqrt{6}}\approx0.8165
\]
\end_inset
while
\begin_inset Formula
\[
\cos\left(\overrightarrow{bird}_{obj},\overrightarrow{crow}_{obj}\right)=\frac{2}{\sqrt{4\cdot2}}=\frac{1}{\sqrt{2}}\approx0.7071
\]
\end_inset
So how similar are crows and birds? What should the similarity measure be
now? Should one take the average of these? Do something else? Perhaps one
might consider a total vector:
\begin_inset Formula
\[
\overrightarrow{bird}_{total}=\overrightarrow{bird}_{naive}+\overrightarrow{bird}_{obj}
\]
\end_inset
This is appealing, but for one important property: the subspaces
\begin_inset Formula $\overrightarrow{word}_{naive}$
\end_inset
and
\begin_inset Formula $\overrightarrow{word}_{obj}$
\end_inset
are always orthogonal to one-another, always, for any word.
These are really distinct vector spaces; they don't mix.
Also, there are more than just these two.
Consider the sentence
\begin_inset Quotes eld
\end_inset
\emph on
The bird flew away.
\emph default
\begin_inset Quotes erd
\end_inset
This suggests a vector
\begin_inset Formula
\[
\overrightarrow{bird}_{subj}=1\left|flew:*\negthinspace-\;\&\;away\negthinspace+\right\rangle
\]
\end_inset
where the wild-card is now in the first position, not the second.
Clearly
\begin_inset Formula $\overrightarrow{word}_{subj}$
\end_inset
is always orthgonal to
\begin_inset Formula $\overrightarrow{word}_{naive}$
\end_inset
and
\begin_inset Formula $\overrightarrow{word}_{obj}$
\end_inset
, for any word.
For any disjunct of length
\begin_inset Formula $N$
\end_inset
, there are at least
\begin_inset Formula $N$
\end_inset
distinct, orthogonal vector spaces, because the wild-card can occur in
any one of
\begin_inset Formula $N$
\end_inset
distinct locations in the disjunct.
The wild-card can also occur with a + attachment, or a - attachment, so
there are at least
\begin_inset Formula $2N$
\end_inset
distinct vector spaces.
Finally, if the disjunct has
\begin_inset Formula $k$
\end_inset
attachments that are -, and
\begin_inset Formula $N-k$
\end_inset
that are +, then the
\begin_inset Formula $*\negthinspace-$
\end_inset
can occur in any of
\begin_inset Formula $m$
\end_inset
locations, while
\begin_inset Formula $*\negthinspace+$
\end_inset
can occur in any of
\begin_inset Formula $N-k$
\end_inset
locations.
Adding up these possibilities, disjuncts of length
\begin_inset Formula $N$
\end_inset
span a total of
\begin_inset Formula $N\left(N+1\right)$
\end_inset
mututally pair-wise orthogonal subspaces.
That's a lot of different subspaces to consider.
\end_layout
\begin_layout Standard
Despite this, one expects an overall consistency in the grammatical classificati
on of a word: if one decides that birds are like crows, then the unified
grammatical class of THINGS that birds and crows belong to must behave
properly, when placed in any particular grammatical context.
Before, John heard and saw birds; now John can hear and see THINGS, and
this needs to hold true for all of the various possible grammatical relations:
\begin_inset Formula
\begin{align*}
heard: & John\negthinspace-\;\&\;THINGS\negthinspace+;\\
THINGS: & the\negthinspace-\;\&\;was\negthinspace+;
\end{align*}
\end_inset
It must necessarily be the same word-class
\begin_inset Quotes eld
\end_inset
\emph on
THINGS
\emph default
\begin_inset Quotes erd
\end_inset
in both of these locations.
It is grammatically inconsistent to have these being distinct from one-another.
\end_layout
\begin_layout Standard
Thus one concludes: (a) there is more than one vector space available, over
which similarity comparisons can be made; (b) the decision to merge must
be made consistently over all available vector spaces; (c) the merge itself
must still be non-linear, in order to differentiate between different word-sens
es attached to the same word.
How this may be accomplished is written up in the next section.
\end_layout
\begin_layout Standard
The important constraint here is that of (b) – that the resulting grammatical
classes must be consistent, in all of the syntactic roles that they can
occur in.
The various syntactic vector spaces are not independent of one-another,
but stitch together.
\end_layout
\begin_layout Subsection*
Merge decisions, redux
\end_layout
\begin_layout Standard
Should one take the average of these?
\end_layout
\begin_layout Subsection*
Replacing Cosines by Entropy?
\end_layout
\begin_layout Standard
Cosines are not additive.
And that's a big problem when trying to add together contributions from
cross-connectors.
\end_layout
\begin_layout Standard
Consider instead a different word-similarity measure.
As always, let
\begin_inset Formula $N(w,d)$
\end_inset
be the count of the number of times the disjunct
\begin_inset Formula $d$
\end_inset
was observed on word
\begin_inset Formula $w$
\end_inset
.
Define the right-product
\end_layout
\begin_layout Standard
\begin_inset Formula
\[
f(w,u)=\sum_{d}N(w,d)N(u,d)
\]
\end_inset
with the sum ranging over all disjuncts shared in common between the two
words.
This is just the dot-product for the two vectors
\begin_inset Formula $\vec{w}$
\end_inset
and
\begin_inset Formula $\vec{u}$
\end_inset
– that is,
\begin_inset Formula $f(w,u)=\vec{w}\cdot\vec{u}$
\end_inset
and so the cosine similarity of two word-disjunct vectors is just
\begin_inset Formula
\[
\cos\left(\vec{w},\vec{u}\right)=\frac{f\left(w,u\right)}{\sqrt{f\left(w,w\right)f\left(u,u\right)}}
\]
\end_inset
Consider instead a similar quantity
\begin_inset Formula
\[
p\left(w,u\right)=\frac{f\left(w,u\right)}{f\left(*,*\right)}
\]
\end_inset
where
\begin_inset Formula $f\left(*,*\right)=\sum_{w,u}f\left(w,u\right)$
\end_inset
is a normalization, a total count.
The quantity
\begin_inset Formula $p(w,u)$
\end_inset
can be interpreted as a probability: it clearly sums to one.
It is symmetric:
\begin_inset Formula $p(w,u)=p(u,w)$
\end_inset
and one can thus have traditional marginal probabilities:
\begin_inset Formula
\[
p\left(w\right)=p\left(w,*\right)=\sum_{u}p\left(w,u\right)
\]
\end_inset
This suggests a natural form for the mutual information between words:
\begin_inset Formula
\[
MI_{d}\left(w,u\right)=\log_{2}\frac{p\left(w,u\right)}{p\left(w\right)p\left(u\right)}
\]
\end_inset
The subscript
\begin_inset Formula $d$
\end_inset
on
\begin_inset Formula $MI_{d}$
\end_inset
serves to remind that this variant of mutual information is derived from
the disjunct product, and not from word-pair observations.
Unlike the word-pair observations, this value of MI is symmetric under
word-interchange: the word-order does not matter.
\end_layout
\begin_layout Subsection*
Replacing Cosines by Surprisingness?
\end_layout
\begin_layout Standard
Can this work?
\end_layout
\begin_layout Section*
Conclusion
\end_layout
\begin_layout Standard
Not yet written.
\end_layout
\begin_layout Section*
The End
\end_layout
\begin_layout Standard
\begin_inset CommandInset bibtex
LatexCommand bibtex
bibfiles "lang"
options "alpha"
\end_inset
\end_layout
\end_body
\end_document