#LyX 2.3 created this file. For more info see http://www.lyx.org/
\lyxformat 544
\begin_document
\begin_header
\save_transient_properties true
\origin unavailable
\textclass article
\use_default_options true
\maintain_unincluded_children false
\language english
\language_package default
\inputencoding auto
\fontencoding global
\font_roman "default" "default"
\font_sans "default" "default"
\font_typewriter "default" "default"
\font_math "auto" "auto"
\font_default_family default
\use_non_tex_fonts false
\font_sc false
\font_osf false
\font_sf_scale 100 100
\font_tt_scale 100 100
\use_microtype false
\use_dash_ligatures true
\graphics default
\default_output_format default
\output_sync 0
\bibtex_command default
\index_command default
\paperfontsize default
\use_hyperref false
\papersize default
\use_geometry false
\use_package amsmath 1
\use_package amssymb 1
\use_package cancel 1
\use_package esint 1
\use_package mathdots 0
\use_package mathtools 1
\use_package mhchem 1
\use_package stackrel 1
\use_package stmaryrd 1
\use_package undertilde 1
\cite_engine basic
\cite_engine_type default
\biblio_style plain
\use_bibtopic false
\use_indices false
\paperorientation portrait
\suppress_date false
\justification true
\use_refstyle 0
\use_minted 0
\index Index
\shortcut idx
\color #008000
\end_index
\secnumdepth 3
\tocdepth 3
\paragraph_separation indent
\paragraph_indentation default
\is_math_indent 0
\math_numbering_side default
\quotes_style english
\dynamic_quotes 0
\papercolumns 1
\papersides 1
\paperpagestyle default
\tracking_changes false
\output_changes false
\html_math_output 0
\html_css_as_file 0
\html_be_strict false
\end_header
\begin_body
\begin_layout Title
Not LSA
\end_layout
\begin_layout Date
July 2015
\end_layout
\begin_layout Subsection*
1 July 2015
\end_layout
\begin_layout Standard
NotLSA – a way to do LSA-like things without actually using LSA (Latent
Semantic Analysis).
Two very low-brow approaches, maybe well-known in the industry; I have
no idea.
Both of these approaches attempt to automatically extract keywords from
documents.
What cool about this is that its ...
unsupervised; requires no training, and is based on very simple, proven
ideas.
Obvious, even: compute the mutual information between pairs of things ...
between words and documents, between words and word-pairs, etc.
Heh.
\end_layout
\begin_layout Standard
But how do we do this? How do we compute the MI between a page of text,
and a word? No way to answer this without diving into the details.
\end_layout
\begin_layout Subsubsection*
Text-keyword correlation
\end_layout
\begin_layout Standard
Lets take a text, say – 1000 pages of ..
something.
Some corpus.
We want to compute the mutual information between the page itself, and
the words on the page.
We do this by analogy to MI of word pairs.
\end_layout
\begin_layout Standard
Call the
\begin_inset Formula $k$
\end_inset
'th page
\begin_inset Formula $g_{k}$
\end_inset
.
Count the number of times that word
\begin_inset Formula $w_{m}$
\end_inset
appears on this page; let this count be
\begin_inset Formula $N_{mk}$
\end_inset
.
Define
\begin_inset Formula $N_{m}=\sum_{k}N_{mk}$
\end_inset
be the total number of times that the work
\begin_inset Formula $w_{m}$
\end_inset
appear in the document, and let
\begin_inset Formula $N=\sum_{m}N_{m}$
\end_inset
be the total number of words in the document.
Then, as usual, define probabilities, so that
\begin_inset Formula
\[
p_{m}=P(w_{m})=N_{m}/N
\]
\end_inset
is the frequency of observing word
\begin_inset Formula $w_{m}$
\end_inset
in the entire corpus, and
\begin_inset Formula
\[
p_{mk}=P(w_{m}|g_{k})=N_{mk}/\sum_{m}N_{mk}
\]
\end_inset
be the (relative) frequency of the same word on page
\begin_inset Formula $g_{k}$
\end_inset
.
Notice that the definition of
\begin_inset Formula $p_{mk}$
\end_inset
is independent of the page size.
Pages do not all have to be of the same size.
Define the mutual information as
\begin_inset Formula
\[
\mbox{MI}(g_{k},w_{m})=-\log_{2}\frac{p_{mk}}{p_{m}}=-\log_{2}\frac{N_{mk}N}{\sum_{m}N_{mk}\sum_{k}N_{mk}}=-\log_{2}\frac{p(m,k)}{p(m,*)p(*,k)}
\]
\end_inset
This is essentially a measure of how much more often the word
\begin_inset Formula $w_{m}$
\end_inset
appears on page
\begin_inset Formula $g_{k}$
\end_inset
as compared to its usual frequency.
The highest-MI words are essentially the topic words for the page.
The right-most form introduces a new notation, to make it clear that it
resembles the traditional pair-MI expression.
The notation is
\begin_inset Formula
\[
p(m,k)=\frac{N_{mk}}{N}
\]
\end_inset
so that
\begin_inset Formula
\[
p(m,*)=\sum_{k}p(m,k)\qquad\mbox{and}\qquad p(*,k)=\sum_{m}p(m,k)
\]
\end_inset
are the traditional-looking pair-MI values.
\end_layout
\begin_layout Standard
TODO: – this does not have the feature-reduction/word-combing aspects of
LSA...
\end_layout
\begin_layout Subsubsection*
Variants
\end_layout
\begin_layout Standard
Instead of working with words, we could work with word-pairs, which is a
stand-in for working with (named) entities.
Thus, we can identify if a named entity occurs in a document more often
than average.
\end_layout
\end_body
\end_document