| … | … | @@ -0,0 +1,561 @@ |
| 1 | ||
| 2 | Correlation/Covariance Matrix Analysis Tools | |
| 3 | ============================================ | |
| 4 | [See also: key ideas presentation](docs/AtomSpace.pdf). | |
| 5 | ||
| 6 | In symbolic AI, there is a generic need to perform "reasoning" or | |
| 7 | "inference". There are more than a few ways this can be done. A | |
| 8 | traditional approach is "chaining", where one explores connected | |
| 9 | paths between a starting point and an end-point, connected by edges | |
| 10 | (one-dimensional links). Each edge has two end-points: two vertices | |
| 11 | define an edge. | |
| 12 | ||
| 13 | In conventional crisp-logic symbolic AI, these vertices are either | |
| 14 | connected, or not; either there is an edge connecting them, or there | |
| 15 | is not. One assigns true/false values to the edges in a connectivity | |
| 16 | diagram. In a probabilistic approach, the edges carry weights or | |
| 17 | "truth values". | |
| 18 | ||
| 19 | A collection of edges forms a graph. It can also be thought of forming | |
| 20 | a matrix `M(v,w)` where `v` and `w` are two vertices. An adjacency | |
| 21 | matrix is the matrix `M(v,w)=true` if the vertices `v` and `w` are | |
| 22 | connected by an edge; otherwise, `M(v,w)=false`. If the graph edges are | |
| 23 | weighted, then `M(v,w)` can be understood to be the weight on that edge. | |
| 24 | Thus, graphs can be represented as matrices, and vice-versa. | |
| 25 | ||
| 26 | The matrix representation of a graph simplifies certain types of | |
| 27 | reasoning and inference. For example, backward/forward chaining limited | |
| 28 | to two steps can be understood as just the square of the matrix. That | |
| 29 | is, two vertices `u` and `w` are connected if and only if | |
| 30 | ``` | |
| 31 | sum_v M(u,v) M(v,w) | |
| 32 | ``` | |
| 33 | is not zero. Three chaining steps requires three matrix products, and so | |
| 34 | on. | |
| 35 | ||
| 36 | A fundamental limitation of the matrix representation of graphs is that, | |
| 37 | for most graphs, the corresponding matrix is sparse, sometimes extremely | |
| 38 | sparse. Thus, storing an `m x k`-dimensional matrix as a block of | |
| 39 | numbers makes no sense, if 99.99% of those numbers are zero. Thus, in | |
| 40 | terms of RAM usage (or disk usage), a graph representation is much more | |
| 41 | compact than a matrix representation. In the AtomSpace, a vertex can be | |
| 42 | any Atom whatsoever; an edge is then just a pair of Atoms. | |
| 43 | ||
| 44 | The code here provides a set of tools to layer a matrix API on top of | |
| 45 | an arbitrary graph or collection of Atoms in the AtomSpace. One merely | |
| 46 | needs to define the types of the vertices, what the edges are, and how | |
| 47 | to find the associated weight on that edge. The toolkit then provides | |
| 48 | a decent set of matrix tools, skewing heavily towards probability-type | |
| 49 | functions, such as conditional probabilities, mutual information, and so | |
| 50 | on. | |
| 51 | ||
| 52 | A deeper abstraction of graphs, more suitable for complex reasoning, | |
| 53 | learning and inference tasks, can be found in the [sheaf](../sheaf) | |
| 54 | directory, as well as the [learn](https://github.com/opencog/learn) | |
| 55 | github repo. Those systems are heavily reliant on the code here. | |
| 56 | ||
| 57 | ||
| 58 | Pairs | |
| 59 | ----- | |
| 60 | More generally, there is a generic theme of "pairs of things" that | |
| 61 | are statistically related. These can be pairs of vertices, or the | |
| 62 | paring of (vertex, nearest-neighbors). For example, the vertices might | |
| 63 | be words of the English language. The nearest-neighbors might be | |
| 64 | N-grams or skip-grams. The vertices might be proteins or genes, | |
| 65 | and the pairing might be (gene, expressed-protein) or (gene, | |
| 66 | up/down-regulated-reaction). The pairs can be homogeneous, such as | |
| 67 | (event, event) pairs, or they can be inhomogeneous, such as (cause, | |
| 68 | effect) pairs. | |
| 69 | ||
| 70 | A recurring theme is to obtain and analyze statistics for such pairs, | |
| 71 | to correlate them, to classify them, to discriminate among them. In | |
| 72 | these cases, the pairs have some associated weight, frequency or count. | |
| 73 | ||
| 74 | A common theme in machine learning are "vectors of data": a vector | |
| 75 | of numbers can be associated to some object, and one is interested in | |
| 76 | classifying these vectors in some way; e.g. by using Principal Component | |
| 77 | Analysis (PCA), Singular Value Decomposition (SVD), K-means clustering, | |
| 78 | and so on. Vectors are buried in essentially all neural-net type | |
| 79 | algorithms. But - this is key: a collection of vectors can be viewed | |
| 80 | as a matrix. Entries in the matrix are (row, column) pairs. | |
| 81 | ||
| 82 | A prototypical example arising in graph theory is the (vertex, | |
| 83 | nearest-neighbors) pairing. Suppose one has a graph consisting of | |
| 84 | many vertices of type A (or having "feature" A), and many vertices of | |
| 85 | type B, C, ... The typical classification problem on such a graph is to | |
| 86 | determine if vertices of type A are similar to the vertices of type B. | |
| 87 | Each neighborhood of a vertex A is a single component of a vector; all | |
| 88 | such neighborhood defines the vector as a whole. To determine if A and | |
| 89 | B have similar neighborhoods, it suffices to compute either the | |
| 90 | dot-product (cosine distance) of the two vectors, or the mutual | |
| 91 | information between them. In this way, common subgraphs can be fished | |
| 92 | out. | |
| 93 | ||
| 94 | The code in this directory exposes portions of the AtomSpace as pairs, | |
| 95 | or as a matrix, or as a collection of vectors, depending on how you want | |
| 96 | to think about it. It implements the low-level code for this access, | |
| 97 | so that high-level algorithms can be implemented on top of it: so that | |
| 98 | they can grab that data, do things with it, and write it back. It | |
| 99 | provides a "window" onto a portion of the AtomSpace, and everything | |
| 100 | seen through that "window" looks like vectors, like one big matrix. | |
| 101 | ||
| 102 | That is, the structure of interest are ordered pairs `(x,y)` of atoms | |
| 103 | (that is, where `x` and `y` are atoms), along with any values attached | |
| 104 | to such pairs. The primary value of interest is an observation count | |
| 105 | `N(x,y)` of how often that particular pair was observed. From this | |
| 106 | observation count, one can compute various statistical measures: first, | |
| 107 | the normalized frequency `p(x,y)` (that is, the probability, likelihood) | |
| 108 | of how often the pair `(x,y)` occurred (the likelihood of observing the | |
| 109 | pair). These counts and frequencies can be viewed as a sparse correlation | |
| 110 | matrix, and the goal here is to do all the typical things that one | |
| 111 | might do with such a matrix. That's what the code in this directory | |
| 112 | does. | |
| 113 | ||
| 114 | The prototypical example is that of word-pairs. These are stored in the | |
| 115 | AtomSpace as | |
| 116 | ``` | |
| 117 | EdgeLink (count=6789) | |
| 118 | PredicateNode "word-pair" | |
| 119 | ListLink | |
| 120 | WordNode "foo" | |
| 121 | WordNode "bar" | |
| 122 | ``` | |
| 123 | which indicates that the word-pair (foo,bar) was observed 6789 times. | |
| 124 | In the general, generic case, we might want to observe not just these | |
| 125 | `PredicateNode "word-pair"` relations, but maybe some other kinds of | |
| 126 | predicates. We are interested, perhaps, not just in `WordNode`'s but | |
| 127 | in relations between, say, `ConceptNodes` and `ContextLink`s. For | |
| 128 | biology, the left side might be a GeneNode, and the right side a | |
| 129 | ProteinNode. | |
| 130 | ||
| 131 | Thus, generically, a "matrix" in the AtomSpace is simply a collection | |
| 132 | of Atoms, all of the same kind and general structure, with some | |
| 133 | identifiable sub-part, called "the rows", or the "left atoms", and some | |
| 134 | other identifiable sub-part, called "the columns", or the "right atoms". | |
| 135 | As long as one can easily identify all of the Atoms that belong to the | |
| 136 | collection, and one can identify two distinct sub-parts, one has | |
| 137 | everything one needs to identify pairs `(left, right)` aka | |
| 138 | `(row, column)` aka `(x,y)`. To get a matrix, one needs one more thing: | |
| 139 | a Value, some Value, any Value, holding a floating-point number for each | |
| 140 | pair. This number is called `N(x,y)`. This number can represent any | |
| 141 | numeric data at all. In the prototypical usage, this number is an | |
| 142 | observation count obtained by sensory data flowing in from the outside | |
| 143 | world. It doesn't have to be - the matrix toolset provided here is | |
| 144 | generic, intended to be useful for any AtomSpace data that meets the | |
| 145 | requirement of having a numeric value attached to a collection of pairs. | |
| 146 | ||
| 147 | The tools implemented here include: | |
| 148 | ||
| 149 | * Row and column subtotals (i.e. "marginals"). | |
| 150 | * Computing and caching frequencies from counts. | |
| 151 | * Computing and caching mutual information of pairs. | |
| 152 | * Computing and caching marginal mutual information of pairs. | |
| 153 | * Computing cosine and jaccard similarity between rows or columns. | |
| 154 | * Computing mutual information between rows or columns | |
| 155 | (as opposed to the MI of pairs). | |
| 156 | * Concatenating dissimilar matrices ("direct sum"). | |
| 157 | * Performing PCA (principal component analysis) in the matrix. | |
| 158 | * Performing cuts, to remove unwanted rows, columns and individual entries. | |
| 159 | This includes both filtering (to hide those entries) and removing | |
| 160 | (deleting) them from the AtomSpace. | |
| 161 | ||
| 162 | To use these tools, all you need to do is to specify a low-level | |
| 163 | object that describes the matrix. If the matrix is in the form of an | |
| 164 | `EdgeLink`, as shown above, then it is sufficient to use the | |
| 165 | `make-edge-pair-api` to describe the types of the left and right | |
| 166 | (row and column) atoms, and the `PredicateNode` identifying the pair; | |
| 167 | everything else is automated. If the matrix is not of the above form, | |
| 168 | then it is quite easy to write a small, custom matrix-access object | |
| 169 | that defines what the pairs are. See `object-api.scm` for details; | |
| 170 | in short, one provides some very simple methods: the `'left-type` and | |
| 171 | `'right-type` methods, that return the atom type of the rows and the | |
| 172 | columns; a `'pair-type` method that returns the atom-type of the pair, | |
| 173 | and a `'pair-count` method that returns the count, given the pair. | |
| 174 | ||
| 175 | ||
| 176 | FAQ | |
| 177 | --- | |
| 178 | **Q:** Why isn't this in C++? Surely, numerical computations would be | |
| 179 | a lot faster in C++, right? | |
| 180 | ||
| 181 | **A:** Yes, probably. But its a lot easier to write scheme code than | |
| 182 | it is to write C++ code, and so prototyping in scheme just made | |
| 183 | more sense. It was just-plain simpler, faster, easier (for me). | |
| 184 | You are invited to take the lessons learned, and re-implement | |
| 185 | in C++. | |
| 186 | ||
| 187 | **Q:** What were the lessons learned? | |
| 188 | ||
| 189 | **A:** The number #1 most important lesson is that the filter object | |
| 190 | is the most important object: it controls what data you want | |
| 191 | to keep, and what data you want to discard, and it needs to | |
| 192 | run "underneath", as a foundation to everything else. | |
| 193 | ||
| 194 | **Q:** Really, C++ is sooo fast... | |
| 195 | ||
| 196 | **A:** Yes, but since the data is stored in Values associated with | |
| 197 | Atoms in the AtomSpace, adding numbers together is NOT the bottleneck. | |
| 198 | Accessing Atom Values *is* the bottleneck. For this case, the overhead | |
| 199 | of using scheme, compared to C++, is not outrageous. | |
| 200 | ||
| 201 | **Q:** Why don't you just export all your data to SciPy or to Gnu R, or | |
| 202 | to Octave, or MatLab, for that matter, and just do your data analytics | |
| 203 | there? That way, you don't need to re-implement all these basic | |
| 204 | statistical algorithms! | |
| 205 | ||
| 206 | **A:** That sounds nice, but frankly, it's too much work for me. Maybe you | |
| 207 | can do this. Seriously, its just a lot easier (for me) to create | |
| 208 | and use the code here, than to struggle mightily with those packages. | |
| 209 | ||
| 210 | Also: realize that the end-goal of OpenCog is not to export data | |
| 211 | once, analyze it once, and be done. Rather, the goal is to constantly | |
| 212 | and continuously monitor external, real-world events, pull them into | |
| 213 | the AtomSpace, crunch it incessantly, and update knowledge of the | |
| 214 | external world as a result. This rules out GUI tools for data | |
| 215 | processing (because there's no GUI in a server) and it rules out | |
| 216 | popsicle-stick architectures as being a bit hokey. | |
| 217 | ||
| 218 | Also: many of these structures have millions of rows and columns, | |
| 219 | and ten-million or a hundred-million non-zero entries in the matrix. | |
| 220 | This can blow through huge amounts of RAM. Can SciPy actually deal | |
| 221 | with datasets this big? It gets worse, because typically, you want | |
| 222 | to work with different cuts and filters, where you discard much of | |
| 223 | the data, or average together different parts: can you really afford | |
| 224 | the RAM needed to export all of these different cut and filtered | |
| 225 | datasets? Maybe you can; its just not trivial. (In my datasets, | |
| 226 | petabytes of RAM would be needed for non-sparse representations. | |
| 227 | The AtomSpace is all about sparse representations of data.) | |
| 228 | ||
| 229 | **Q:** But if I did want to do it for Gnu R, how could I do it? | |
| 230 | ||
| 231 | **A:** You would use Rcpp at http://dirk.eddelbuettel.com/code/rcpp.html | |
| 232 | Quote: | |
| 233 | *The Rcpp package provides C++ classes that greatly facilitate | |
| 234 | interfacing C or C++ code in R packages using the `.Call()` interface | |
| 235 | provided by R. Rcpp provides matching C++ classes for a large | |
| 236 | number of basic R data types. Hence, a package author can keep his | |
| 237 | data in normal R data structures without having to worry about | |
| 238 | translation or transferring to C++. At the same time, the data | |
| 239 | structures can be accessed as easily at the C++ level, and used in | |
| 240 | the normal manner. The mapping of data types works in both | |
| 241 | directions. It is as straightforward to pass data from R to C++, | |
| 242 | as it is it return data from C++ to R.* | |
| 243 | ||
| 244 | **Q:** Any other design issues? | |
| 245 | ||
| 246 | **A:** Yes. As currently structured, all of these classes assume that | |
| 247 | your data is readily available in RAM. They will not work correctly, | |
| 248 | if your dataset is too big to fit into RAM. At the time of this | |
| 249 | writing, a single atom takes about 1.5KBytes or so, so a dataset | |
| 250 | consisting of 100M atoms will require about 150GBytes of RAM, plus | |
| 251 | a bit more for other processes (e.g. Postgres). Since most computers | |
| 252 | max out at about 256 GBytes RAM, this limits datasets to 100M atoms. | |
| 253 | Some language datasets can be considerably larger than this. | |
| 254 | At this time, the largest Amazon EC2 instances are 256 GBytes. | |
| 255 | ||
| 256 | ||
| 257 | Generic Programming / Traits | |
| 258 | ---------------------------- | |
| 259 | In order to perform some sort of generic analysis of the correlation | |
| 260 | of atoms, we need to somehow specify which pairs of atoms are | |
| 261 | the ones to be analyzed. This is done with a minimalist object-oriented | |
| 262 | API, where you, the user, get to provide a "low-level object" which | |
| 263 | indicates the types of the left and the right atoms, the type of the | |
| 264 | pair, and where the counts `N(x,y)` are located. This library then | |
| 265 | implements a collection of various objects that sit "on top" of this, | |
| 266 | and provide various reasonable default behaviors for computing the | |
| 267 | various interesting things. If you don't like some particular default, | |
| 268 | you can always overload that particular method to do something | |
| 269 | customized. This OO programming style is called "parametric | |
| 270 | polymorphism". It is also seems to go under the name of "traits". | |
| 271 | ||
| 272 | https://en.wikipedia.org/wiki/Parametric_polymorphism | |
| 273 | ||
| 274 | https://en.wikipedia.org/wiki/Trait_(computer_programming) | |
| 275 | ||
| 276 | https://en.wikipedia.org/wiki/Generic_programming | |
| 277 | ||
| 278 | This code is written in scheme. I know some of you want to use python | |
| 279 | instead, while others would prefer C++. More generally, it would | |
| 280 | probably be useful to set things up so that external, third-party | |
| 281 | software systems (such as SciPy or Gnu Octave or tensorflow) could | |
| 282 | be used to perform the analysis. Now that you've got the general | |
| 283 | idea... you can go do this! | |
| 284 | ||
| 285 | Anyway, if you are willing to use scheme, here's what we've got, | |
| 286 | ||
| 287 | Low-level API | |
| 288 | ------------- | |
| 289 | Every user needs to implement a "low-level API" object that describes | |
| 290 | the pairs, and provides methods to get them, and to get the associated | |
| 291 | count (or other numeric value). | |
| 292 | ||
| 293 | The methods that need to be implemented are described in | |
| 294 | `object-api.scm`; a working example is in `eval-pair.scm`. | |
| 295 | Additional working examples of the base classes can be found in | |
| 296 | https://github.com/opencog/learn/tree/master/scm/batch-word-pair.scm | |
| 297 | and in | |
| 298 | https://github.com/opencog/learn/tree/master/scm/pseudo-csets.scm | |
| 299 | ||
| 300 | Basic definitions | |
| 301 | ----------------- | |
| 302 | Some notation: | |
| 303 | ||
| 304 | Let `N(x,y)` be the observed count on the pair of atoms `(x,y)`. | |
| 305 | ||
| 306 | The `add-support-api` class provides an API to report the partial | |
| 307 | sums `N(x,*) = sum_y N(x,y)` and likewise `N(*,y)`. If you think of | |
| 308 | `N(x,y)` as a matrix, these are the totals for the entries in each | |
| 309 | row or column of the matrix. Likewise, `N(*,*) = sum_x sum_y N(x,y)`. | |
| 310 | In statistics, these are called "marginals", because you write the | |
| 311 | subtotals in the "margins" of your table. | |
| 312 | ||
| 313 | The `add-pair-freq-api` class provides an API to report the frequencies | |
| 314 | of pairs, and the partial sums over rows and columns. The frequency | |
| 315 | is defined, by default, to be `p(x,y) = N(x,y)/N(*,*)`. The row and | |
| 316 | column sums are `p(x,*) = sum_y p(x,y)`. By default, these total to | |
| 317 | one, as all good probabilities should: `1 = sum_x sum_y p(x,y)`. | |
| 318 | ||
| 319 | The `add-report-api` class provides an API to report summary information | |
| 320 | about the matrix, including the dimensions of the matrix (the number of | |
| 321 | rows and columns), the total number of non-zero entries (which is the | |
| 322 | same as the total number of unique pairs), the left, right and total | |
| 323 | entropies and mutual information. | |
| 324 | ||
| 325 | Computing basic statistics | |
| 326 | -------------------------- | |
| 327 | These `add-pair-*-api` classes simply provide methods to fetch these | |
| 328 | values, which are presumed to have been precomputed in some way. Several | |
| 329 | classes are provided to compute these. | |
| 330 | ||
| 331 | The `add-pair-stars` class provides methods to fetch the set | |
| 332 | `{x | the atom x occurs in some pair (x,y)}`. This is called the | |
| 333 | `left-support-set`. Likewise for the right-support. | |
| 334 | Another method returns the wild-card pairs: that is, the set | |
| 335 | `(x,*) = {(x,y) | x is fixed and y is any atom in the pair}` | |
| 336 | This is called the `right-star` because the right-side of the pair is | |
| 337 | a wild-card. These methods are useful for iterating over all rows and | |
| 338 | columns of the correlation matrix. | |
| 339 | ||
| 340 | The `add-loop-api` class provides methods to loop over all non-zero | |
| 341 | pairs in the matrix, and invoke a function on them. There are two | |
| 342 | methods: `for-each-pair`, which simply calls the function for each | |
| 343 | pair, and a `map-pair` method which returns a list of the results of | |
| 344 | calling the function on each pair. XXX Should be merged with the | |
| 345 | `batch-similarity` object, which does a similar thing, and also with | |
| 346 | `loop-upper-diagonal`, which does diagonal looping. | |
| 347 | ||
| 348 | The `add-support-compute` class provides methods to compute the | |
| 349 | partial sums `N(*,y)` and `N(x,*)`. It also provides methods that | |
| 350 | compute how many non-zero entries there are in each row or column. | |
| 351 | It also provides methods for the "length" of a column: that is, | |
| 352 | `len(y) = sqrt(sum_x N^2 (x,y))` and more generally the l_p norm. | |
| 353 | Because these computations can take a considerable amount of time, | |
| 354 | the partial sums (the "marginals") are cached. The cached values can | |
| 355 | be accessed with the `add-support-api` object. | |
| 356 | ||
| 357 | The `make-compute-freq` class provides methods to compute and cache | |
| 358 | the frequencies `P(x,y)`, `P(*,y)` and `P(x,*)`. These are cached | |
| 359 | exactly where the `add-pair-freq-api` class, above, can find them. | |
| 360 | ||
| 361 | The `make-batch-mi` class provides methods to compute the fractional | |
| 362 | mutual information of all pairs, namely the value | |
| 363 | ``` | |
| 364 | MI(x,y) = +log_2 P(x,y) / P(x,*) P(*,y) | |
| 365 | ``` | |
| 366 | ||
| 367 | The `batch-all-pair-mi` class is a convenience class that wraps up all | |
| 368 | three classes above, and unleashes them on a dataset: first computing | |
| 369 | the row and column partial counts, then computing the frequencies, and | |
| 370 | then computing and caching the mutual information. For large datasets, | |
| 371 | e.g. tens of millions of atoms, this can take hours to run. Thus, for | |
| 372 | this reason, the cached values are then saved to the currently-open | |
| 373 | database, so that these results become available later. | |
| 374 | ||
| 375 | Computing entropy | |
| 376 | ----------------- | |
| 377 | The `add-entropy-compute` class provides methods to compute the entropy | |
| 378 | and mutual information of rows and columns: for example, the column | |
| 379 | entropy (or `left-entropy`) `h_left(y) = -sum_x P(x,y) log_2 P(x,y)` | |
| 380 | It also returns the far more interesting 'fractional entropy', given | |
| 381 | by `H_left(y) = h_left(y) / P(*,y)`. Along similar lines, there is | |
| 382 | also the mutual information `mi_left(y) = sum_x P(x,y) log_2 MI(x,y)` | |
| 383 | where `MI(x,y) = +log_2 P(x,y) / P(x,*) P(*,y)` is the fractional pair | |
| 384 | MI. | |
| 385 | ||
| 386 | Methods are also provided to compute the total entropy: | |
| 387 | `H_tot = sum_x sum_y P(x,y) log_2 P(x,y)` as well as | |
| 388 | the left and right entropies: `H_left = sum_y P(*,y) log_2 P(*,y)` | |
| 389 | and v.v. | |
| 390 | ||
| 391 | Computing similarity measures | |
| 392 | ----------------------------- | |
| 393 | The `add-similarity-compute` class provides methods for taking the | |
| 394 | vector product of two different rows or columns, viz. the product | |
| 395 | `left-prod(y,z) = sum_x N(x,y) N(x,z)` and likewise for the right. | |
| 396 | The cosine similarity is | |
| 397 | ``` | |
| 398 | left-cosine(y,z) = left-prod(y,z) / (left-length(y) * left-length(z)) | |
| 399 | ``` | |
| 400 | where | |
| 401 | ``` | |
| 402 | left-length(y) = sqrt sum_x N(x,y) N(x,y) | |
| 403 | = sqrt left-prod(y,y) | |
| 404 | ```` | |
| 405 | The class also provides several other similarity measures, | |
| 406 | including the Jaccard similarity (Ruzicka distance) | |
| 407 | ``` | |
| 408 | left-jacc-sim(y,z) = sum_x min (N(x,y), N(x,z)) / | |
| 409 | sum_x max (N(x,y), N(x,z)) | |
| 410 | ``` | |
| 411 | and the Probability-Jaccard distance (a maximally consistent distance | |
| 412 | for probability distributions; see Wikiepdia for details). This class | |
| 413 | also provides a plain overlap similarity function. | |
| 414 | ||
| 415 | ||
| 416 | Direct sums | |
| 417 | ----------- | |
| 418 | If one has a collection of things that can be vectors in two different | |
| 419 | ways, then the direct sum can be used to create a single vector out of | |
| 420 | the two. It can be thought of as a concatenation of the two vectors, | |
| 421 | appending one to the other to create a single vector. | |
| 422 | ||
| 423 | For example, a word "foo" can be associated with a vector of disjuncts. | |
| 424 | It can also be associated with a vector of connectors that it appears | |
| 425 | in (a "shape"). These are two completely different kinds of vectors; | |
| 426 | all they have in common is that both are associated with the word "foo". | |
| 427 | As matrices, one consists of (word,disjunct) pairs, while the other | |
| 428 | consists of (word,shape) pairs. The left-side of the matrix (the rows | |
| 429 | of the matrix) all have the same type (i.e. words) while the right | |
| 430 | side (the columns) are two different types (and are completely disjoint, | |
| 431 | as sets). Concatenating these two matrices, placing one after the other, | |
| 432 | results in a single matrix where the rows are result of appending the | |
| 433 | rows of the second matrix to the first. If the dimensions of the two | |
| 434 | matrices are N x M and N x K, then the dimensions of the combined matrix | |
| 435 | would be N x (M + K). | |
| 436 | ||
| 437 | The `direct-sum` class will combine these two matrices to provide an | |
| 438 | object that behaves like a single matrix. The left and right basis | |
| 439 | elements will be the set-union of the left and right basis elts of each | |
| 440 | component matrix. By assumption, the types of either the columns or | |
| 441 | the rows are distinct, and so one of these sets is disjoint. Thus, the | |
| 442 | union is unambiguous; the matrices are either placed side-by-side by | |
| 443 | concatenating rows, or above-below, by concatenating columns. There is | |
| 444 | no "overlap". The total number of non-zero entries in the combined | |
| 445 | matrix is the sum of the number of non-zero entries in each component. | |
| 446 | ||
| 447 | (Caution: this is not the conventional definition of a direct sum, which | |
| 448 | results in block-diagonal matrix. The conventional definition takes the | |
| 449 | disjoint-union of the indexes (the basis), whereas here, we take the | |
| 450 | set-union of the basis. The set-union makes more sense in the current | |
| 451 | context, because the rows and columns have explicit labels, and it is | |
| 452 | the labels that are important. The set union is obtained from the | |
| 453 | disjoint union by means of a projection.) | |
| 454 | ||
| 455 | ||
| 456 | Working with rows and columns | |
| 457 | ----------------------------- | |
| 458 | The `add-tuple-math` class provides methods for applying arbitrary | |
| 459 | functions to arbitrary sets of rows or columns. The simplest examples | |
| 460 | include taking the sums and differences of columns, taking the | |
| 461 | element-by-element min or max of a set of columns, counting the number | |
| 462 | of entries that are simultaneously non-zero in sets of columns, etc. | |
| 463 | ||
| 464 | The class works by defining a new matrix, whose rows or columns | |
| 465 | are tuples (pairs, triples, etc) of the underlying matrix. For example, | |
| 466 | one can ask for the matrix entry `(x, [y,z,w])`. The value returned | |
| 467 | will be `(x, f((x,y), (x,z), (x,w)))` where the user-defined function | |
| 468 | `f` was used to create the `x` row. | |
| 469 | ||
| 470 | ||
| 471 | Gaussian Orthogonal Ensembles | |
| 472 | ----------------------------- | |
| 473 | The `add-gaussian-ortho-compute` object will take a (non-sparse) | |
| 474 | symmetric matrix as input, and normalize it such that the mean of all | |
| 475 | of the matrix entries is zero, and the standared deviation is one. | |
| 476 | This will result in a matrix that is (approximately) gaussian | |
| 477 | orthogonal. The rows and columns can then be understood to be | |
| 478 | (approximately) uniformly distributed about the origin, and when | |
| 479 | normalized to unit length, can be understood to lie on the surface | |
| 480 | of a sphere $S_{N-1}$ where the original matrix is N x N dimensional. | |
| 481 | ||
| 482 | ||
| 483 | Principal Component Analysis | |
| 484 | ---------------------------- | |
| 485 | A power iteration object is provided by the `make-power-iter-pca` | |
| 486 | object. Once a matrix `X` has been specified, this will iterate on | |
| 487 | either `X^T X` or on `XX^T` for `K` iterations. The result of this | |
| 488 | iteration is the principal component of `X` (equivalently, the | |
| 489 | Frobenius-Peron eigenvector of `X^T X` or `XX^T`). | |
| 490 | ||
| 491 | The code designed to work fast for sparse matrices, and can obtain | |
| 492 | eigenvectors in under 20 seconds or so for 15K by 15K matrices with | |
| 493 | 100K non-zero entries. | |
| 494 | ||
| 495 | The code is beta; only a bare minimum of function is provided. So far, | |
| 496 | PCA is not terribly interesting for the kinds of graph problems we are | |
| 497 | interested in solving. | |
| 498 | ||
| 499 | ||
| 500 | Data Cuts | |
| 501 | --------- | |
| 502 | Data will typically have lots of "noise" in it, which should be masked | |
| 503 | before data analysis starts. The tools in `filter.scm` can do this. A | |
| 504 | generic filter will mask out arbitrary rows, columns and individual | |
| 505 | entries, depending on the supplied predicates. Built on top of this is | |
| 506 | a filter that masks out rows, columns and entries that have counts | |
| 507 | below a threshold. Another filter can mask out explicitly-named rows | |
| 508 | and columns. | |
| 509 | ||
| 510 | Unfortunately, filtering can be slow, and loading a large matrix only | |
| 511 | to access small subsets of it is wasteful of RAM, and can add CPU | |
| 512 | overhead. Thus, a similar set of interfaces is provided in `trim.scm`, | |
| 513 | which will remove (delete) rows, columns and matrix entries from both | |
| 514 | the AtomSpace, and from attached storage. With appopropriate trimming, | |
| 515 | noisy datasets can be reduced in size by factors of two, ten or a | |
| 516 | hundred. This can have a huge impact on processing. | |
| 517 | ||
| 518 | ||
| 519 | Tensors, in general | |
| 520 | ------------------- | |
| 521 | Suppose you have more than just pairs. Suppose you have triples that | |
| 522 | you want to work with. Then what? Answer: use the network analysis | |
| 523 | tools in the `(opencog network)` module. | |
| 524 | ||
| 525 | Reading List | |
| 526 | ------------ | |
| 527 | If you choose to read the source code, read it in the following order: | |
| 528 | ||
| 529 | * object-api.scm | |
| 530 | * eval-pairs.scm | |
| 531 | * support.scm | |
| 532 | * transpose.scm | |
| 533 | * similarity-api.scm | |
| 534 | * cosine.scm | |
| 535 | * symmetric-mi.scm | |
| 536 | * group-similarity.scm | |
| 537 | * compute-mi.scm | |
| 538 | * trans-batch.scm | |
| 539 | * filter.scm | |
| 540 | * trim.scm | |
| 541 | * direct-sum.scm | |
| 542 | ||
| 543 | There are more files not mentioned in this list, but they are less relevant. | |
| 544 | ||
| 545 | TODO | |
| 546 | ---- | |
| 547 | To-do list items. | |
| 548 | * The "star" objects need to be redesigned. They fetch wild-card counts | |
| 549 | and other marginal values straight out of the AtomSpace. But those | |
| 550 | marginal values are correct only if no filtering is applied. If there | |
| 551 | are pre-filters, then the returned marginals are garbage. Yucko. Can | |
| 552 | we fail-safe this for now? | |
| 553 | ||
| 554 | * Need to support columns/rows that can be one of several types | |
| 555 | (e.g. can be WordNodes, or be WordClassNodes) This is currently | |
| 556 | handled in an ad hoc manner. | |
| 557 | ||
| 558 | * Need a more consistent/coherent API for interacting with storage. | |
| 559 | Currently, the `fetch` methods do this, and there is a scattering | |
| 560 | of other store methods provided in an ad hoc, as-needed basis. All | |
| 561 | this should be made prettier. |
| … | … | @@ -1,561 +0,0 @@ |
| 1 | ||
| 2 | Correlation/Covariance Matrix Analysis Tools | |
| 3 | ============================================ | |
| 4 | [See also: key ideas presentation](docs/AtomSpace.pdf). | |
| 5 | ||
| 6 | In symbolic AI, there is a generic need to perform "reasoning" or | |
| 7 | "inference". There are more than a few ways this can be done. A | |
| 8 | traditional approach is "chaining", where one explores connected | |
| 9 | paths between a starting point and an end-point, connected by edges | |
| 10 | (one-dimensional links). Each edge has two end-points: two vertices | |
| 11 | define an edge. | |
| 12 | ||
| 13 | In conventional crisp-logic symbolic AI, these vertices are either | |
| 14 | connected, or not; either there is an edge connecting them, or there | |
| 15 | is not. One assigns true/false values to the edges in a connectivity | |
| 16 | diagram. In a probabilistic approach, the edges carry weights or | |
| 17 | "truth values". | |
| 18 | ||
| 19 | A collection of edges forms a graph. It can also be thought of forming | |
| 20 | a matrix `M(v,w)` where `v` and `w` are two vertices. An adjacency | |
| 21 | matrix is the matrix `M(v,w)=true` if the vertices `v` and `w` are | |
| 22 | connected by an edge; otherwise, `M(v,w)=false`. If the graph edges are | |
| 23 | weighted, then `M(v,w)` can be understood to be the weight on that edge. | |
| 24 | Thus, graphs can be represented as matrices, and vice-versa. | |
| 25 | ||
| 26 | The matrix representation of a graph simplifies certain types of | |
| 27 | reasoning and inference. For example, backward/forward chaining limited | |
| 28 | to two steps can be understood as just the square of the matrix. That | |
| 29 | is, two vertices `u` and `w` are connected if and only if | |
| 30 | ``` | |
| 31 | sum_v M(u,v) M(v,w) | |
| 32 | ``` | |
| 33 | is not zero. Three chaining steps requires three matrix products, and so | |
| 34 | on. | |
| 35 | ||
| 36 | A fundamental limitation of the matrix representation of graphs is that, | |
| 37 | for most graphs, the corresponding matrix is sparse, sometimes extremely | |
| 38 | sparse. Thus, storing an `m x k`-dimensional matrix as a block of | |
| 39 | numbers makes no sense, if 99.99% of those numbers are zero. Thus, in | |
| 40 | terms of RAM usage (or disk usage), a graph representation is much more | |
| 41 | compact than a matrix representation. In the AtomSpace, a vertex can be | |
| 42 | any Atom whatsoever; an edge is then just a pair of Atoms. | |
| 43 | ||
| 44 | The code here provides a set of tools to layer a matrix API on top of | |
| 45 | an arbitrary graph or collection of Atoms in the AtomSpace. One merely | |
| 46 | needs to define the types of the vertices, what the edges are, and how | |
| 47 | to find the associated weight on that edge. The toolkit then provides | |
| 48 | a decent set of matrix tools, skewing heavily towards probability-type | |
| 49 | functions, such as conditional probabilities, mutual information, and so | |
| 50 | on. | |
| 51 | ||
| 52 | A deeper abstraction of graphs, more suitable for complex reasoning, | |
| 53 | learning and inference tasks, can be found in the [sheaf](../sheaf) | |
| 54 | directory, as well as the [learn](https://github.com/opencog/learn) | |
| 55 | github repo. Those systems are heavily reliant on the code here. | |
| 56 | ||
| 57 | ||
| 58 | Pairs | |
| 59 | ----- | |
| 60 | More generally, there is a generic theme of "pairs of things" that | |
| 61 | are statistically related. These can be pairs of vertices, or the | |
| 62 | paring of (vertex, nearest-neighbors). For example, the vertices might | |
| 63 | be words of the English language. The nearest-neighbors might be | |
| 64 | N-grams or skip-grams. The vertices might be proteins or genes, | |
| 65 | and the pairing might be (gene, expressed-protein) or (gene, | |
| 66 | up/down-regulated-reaction). The pairs can be homogeneous, such as | |
| 67 | (event, event) pairs, or they can be inhomogeneous, such as (cause, | |
| 68 | effect) pairs. | |
| 69 | ||
| 70 | A recurring theme is to obtain and analyze statistics for such pairs, | |
| 71 | to correlate them, to classify them, to discriminate among them. In | |
| 72 | these cases, the pairs have some associated weight, frequency or count. | |
| 73 | ||
| 74 | A common theme in machine learning are "vectors of data": a vector | |
| 75 | of numbers can be associated to some object, and one is interested in | |
| 76 | classifying these vectors in some way; e.g. by using Principal Component | |
| 77 | Analysis (PCA), Singular Value Decomposition (SVD), K-means clustering, | |
| 78 | and so on. Vectors are buried in essentially all neural-net type | |
| 79 | algorithms. But - this is key: a collection of vectors can be viewed | |
| 80 | as a matrix. Entries in the matrix are (row, column) pairs. | |
| 81 | ||
| 82 | A prototypical example arising in graph theory is the (vertex, | |
| 83 | nearest-neighbors) pairing. Suppose one has a graph consisting of | |
| 84 | many vertices of type A (or having "feature" A), and many vertices of | |
| 85 | type B, C, ... The typical classification problem on such a graph is to | |
| 86 | determine if vertices of type A are similar to the vertices of type B. | |
| 87 | Each neighborhood of a vertex A is a single component of a vector; all | |
| 88 | such neighborhood defines the vector as a whole. To determine if A and | |
| 89 | B have similar neighborhoods, it suffices to compute either the | |
| 90 | dot-product (cosine distance) of the two vectors, or the mutual | |
| 91 | information between them. In this way, common subgraphs can be fished | |
| 92 | out. | |
| 93 | ||
| 94 | The code in this directory exposes portions of the AtomSpace as pairs, | |
| 95 | or as a matrix, or as a collection of vectors, depending on how you want | |
| 96 | to think about it. It implements the low-level code for this access, | |
| 97 | so that high-level algorithms can be implemented on top of it: so that | |
| 98 | they can grab that data, do things with it, and write it back. It | |
| 99 | provides a "window" onto a portion of the AtomSpace, and everything | |
| 100 | seen through that "window" looks like vectors, like one big matrix. | |
| 101 | ||
| 102 | That is, the structure of interest are ordered pairs `(x,y)` of atoms | |
| 103 | (that is, where `x` and `y` are atoms), along with any values attached | |
| 104 | to such pairs. The primary value of interest is an observation count | |
| 105 | `N(x,y)` of how often that particular pair was observed. From this | |
| 106 | observation count, one can compute various statistical measures: first, | |
| 107 | the normalized frequency `p(x,y)` (that is, the probability, likelihood) | |
| 108 | of how often the pair `(x,y)` occurred (the likelihood of observing the | |
| 109 | pair). These counts and frequencies can be viewed as a sparse correlation | |
| 110 | matrix, and the goal here is to do all the typical things that one | |
| 111 | might do with such a matrix. That's what the code in this directory | |
| 112 | does. | |
| 113 | ||
| 114 | The prototypical example is that of word-pairs. These are stored in the | |
| 115 | AtomSpace as | |
| 116 | ``` | |
| 117 | EdgeLink (count=6789) | |
| 118 | PredicateNode "word-pair" | |
| 119 | ListLink | |
| 120 | WordNode "foo" | |
| 121 | WordNode "bar" | |
| 122 | ``` | |
| 123 | which indicates that the word-pair (foo,bar) was observed 6789 times. | |
| 124 | In the general, generic case, we might want to observe not just these | |
| 125 | `PredicateNode "word-pair"` relations, but maybe some other kinds of | |
| 126 | predicates. We are interested, perhaps, not just in `WordNode`'s but | |
| 127 | in relations between, say, `ConceptNodes` and `ContextLink`s. For | |
| 128 | biology, the left side might be a GeneNode, and the right side a | |
| 129 | ProteinNode. | |
| 130 | ||
| 131 | Thus, generically, a "matrix" in the AtomSpace is simply a collection | |
| 132 | of Atoms, all of the same kind and general structure, with some | |
| 133 | identifiable sub-part, called "the rows", or the "left atoms", and some | |
| 134 | other identifiable sub-part, called "the columns", or the "right atoms". | |
| 135 | As long as one can easily identify all of the Atoms that belong to the | |
| 136 | collection, and one can identify two distinct sub-parts, one has | |
| 137 | everything one needs to identify pairs `(left, right)` aka | |
| 138 | `(row, column)` aka `(x,y)`. To get a matrix, one needs one more thing: | |
| 139 | a Value, some Value, any Value, holding a floating-point number for each | |
| 140 | pair. This number is called `N(x,y)`. This number can represent any | |
| 141 | numeric data at all. In the prototypical usage, this number is an | |
| 142 | observation count obtained by sensory data flowing in from the outside | |
| 143 | world. It doesn't have to be - the matrix toolset provided here is | |
| 144 | generic, intended to be useful for any AtomSpace data that meets the | |
| 145 | requirement of having a numeric value attached to a collection of pairs. | |
| 146 | ||
| 147 | The tools implemented here include: | |
| 148 | ||
| 149 | * Row and column subtotals (i.e. "marginals"). | |
| 150 | * Computing and caching frequencies from counts. | |
| 151 | * Computing and caching mutual information of pairs. | |
| 152 | * Computing and caching marginal mutual information of pairs. | |
| 153 | * Computing cosine and jaccard similarity between rows or columns. | |
| 154 | * Computing mutual information between rows or columns | |
| 155 | (as opposed to the MI of pairs). | |
| 156 | * Concatenating dissimilar matrices ("direct sum"). | |
| 157 | * Performing PCA (principal component analysis) in the matrix. | |
| 158 | * Performing cuts, to remove unwanted rows, columns and individual entries. | |
| 159 | This includes both filtering (to hide those entries) and removing | |
| 160 | (deleting) them from the AtomSpace. | |
| 161 | ||
| 162 | To use these tools, all you need to do is to specify a low-level | |
| 163 | object that describes the matrix. If the matrix is in the form of an | |
| 164 | `EdgeLink`, as shown above, then it is sufficient to use the | |
| 165 | `make-edge-pair-api` to describe the types of the left and right | |
| 166 | (row and column) atoms, and the `PredicateNode` identifying the pair; | |
| 167 | everything else is automated. If the matrix is not of the above form, | |
| 168 | then it is quite easy to write a small, custom matrix-access object | |
| 169 | that defines what the pairs are. See `object-api.scm` for details; | |
| 170 | in short, one provides some very simple methods: the `'left-type` and | |
| 171 | `'right-type` methods, that return the atom type of the rows and the | |
| 172 | columns; a `'pair-type` method that returns the atom-type of the pair, | |
| 173 | and a `'pair-count` method that returns the count, given the pair. | |
| 174 | ||
| 175 | ||
| 176 | FAQ | |
| 177 | --- | |
| 178 | **Q:** Why isn't this in C++? Surely, numerical computations would be | |
| 179 | a lot faster in C++, right? | |
| 180 | ||
| 181 | **A:** Yes, probably. But its a lot easier to write scheme code than | |
| 182 | it is to write C++ code, and so prototyping in scheme just made | |
| 183 | more sense. It was just-plain simpler, faster, easier (for me). | |
| 184 | You are invited to take the lessons learned, and re-implement | |
| 185 | in C++. | |
| 186 | ||
| 187 | **Q:** What were the lessons learned? | |
| 188 | ||
| 189 | **A:** The number #1 most important lesson is that the filter object | |
| 190 | is the most important object: it controls what data you want | |
| 191 | to keep, and what data you want to discard, and it needs to | |
| 192 | run "underneath", as a foundation to everything else. | |
| 193 | ||
| 194 | **Q:** Really, C++ is sooo fast... | |
| 195 | ||
| 196 | **A:** Yes, but since the data is stored in Values associated with | |
| 197 | Atoms in the AtomSpace, adding numbers together is NOT the bottleneck. | |
| 198 | Accessing Atom Values *is* the bottleneck. For this case, the overhead | |
| 199 | of using scheme, compared to C++, is not outrageous. | |
| 200 | ||
| 201 | **Q:** Why don't you just export all your data to SciPy or to Gnu R, or | |
| 202 | to Octave, or MatLab, for that matter, and just do your data analytics | |
| 203 | there? That way, you don't need to re-implement all these basic | |
| 204 | statistical algorithms! | |
| 205 | ||
| 206 | **A:** That sounds nice, but frankly, it's too much work for me. Maybe you | |
| 207 | can do this. Seriously, its just a lot easier (for me) to create | |
| 208 | and use the code here, than to struggle mightily with those packages. | |
| 209 | ||
| 210 | Also: realize that the end-goal of OpenCog is not to export data | |
| 211 | once, analyze it once, and be done. Rather, the goal is to constantly | |
| 212 | and continuously monitor external, real-world events, pull them into | |
| 213 | the AtomSpace, crunch it incessantly, and update knowledge of the | |
| 214 | external world as a result. This rules out GUI tools for data | |
| 215 | processing (because there's no GUI in a server) and it rules out | |
| 216 | popsicle-stick architectures as being a bit hokey. | |
| 217 | ||
| 218 | Also: many of these structures have millions of rows and columns, | |
| 219 | and ten-million or a hundred-million non-zero entries in the matrix. | |
| 220 | This can blow through huge amounts of RAM. Can SciPy actually deal | |
| 221 | with datasets this big? It gets worse, because typically, you want | |
| 222 | to work with different cuts and filters, where you discard much of | |
| 223 | the data, or average together different parts: can you really afford | |
| 224 | the RAM needed to export all of these different cut and filtered | |
| 225 | datasets? Maybe you can; its just not trivial. (In my datasets, | |
| 226 | petabytes of RAM would be needed for non-sparse representations. | |
| 227 | The AtomSpace is all about sparse representations of data.) | |
| 228 | ||
| 229 | **Q:** But if I did want to do it for Gnu R, how could I do it? | |
| 230 | ||
| 231 | **A:** You would use Rcpp at http://dirk.eddelbuettel.com/code/rcpp.html | |
| 232 | Quote: | |
| 233 | *The Rcpp package provides C++ classes that greatly facilitate | |
| 234 | interfacing C or C++ code in R packages using the `.Call()` interface | |
| 235 | provided by R. Rcpp provides matching C++ classes for a large | |
| 236 | number of basic R data types. Hence, a package author can keep his | |
| 237 | data in normal R data structures without having to worry about | |
| 238 | translation or transferring to C++. At the same time, the data | |
| 239 | structures can be accessed as easily at the C++ level, and used in | |
| 240 | the normal manner. The mapping of data types works in both | |
| 241 | directions. It is as straightforward to pass data from R to C++, | |
| 242 | as it is it return data from C++ to R.* | |
| 243 | ||
| 244 | **Q:** Any other design issues? | |
| 245 | ||
| 246 | **A:** Yes. As currently structured, all of these classes assume that | |
| 247 | your data is readily available in RAM. They will not work correctly, | |
| 248 | if your dataset is too big to fit into RAM. At the time of this | |
| 249 | writing, a single atom takes about 1.5KBytes or so, so a dataset | |
| 250 | consisting of 100M atoms will require about 150GBytes of RAM, plus | |
| 251 | a bit more for other processes (e.g. Postgres). Since most computers | |
| 252 | max out at about 256 GBytes RAM, this limits datasets to 100M atoms. | |
| 253 | Some language datasets can be considerably larger than this. | |
| 254 | At this time, the largest Amazon EC2 instances are 256 GBytes. | |
| 255 | ||
| 256 | ||
| 257 | Generic Programming / Traits | |
| 258 | ---------------------------- | |
| 259 | In order to perform some sort of generic analysis of the correlation | |
| 260 | of atoms, we need to somehow specify which pairs of atoms are | |
| 261 | the ones to be analyzed. This is done with a minimalist object-oriented | |
| 262 | API, where you, the user, get to provide a "low-level object" which | |
| 263 | indicates the types of the left and the right atoms, the type of the | |
| 264 | pair, and where the counts `N(x,y)` are located. This library then | |
| 265 | implements a collection of various objects that sit "on top" of this, | |
| 266 | and provide various reasonable default behaviors for computing the | |
| 267 | various interesting things. If you don't like some particular default, | |
| 268 | you can always overload that particular method to do something | |
| 269 | customized. This OO programming style is called "parametric | |
| 270 | polymorphism". It is also seems to go under the name of "traits". | |
| 271 | ||
| 272 | https://en.wikipedia.org/wiki/Parametric_polymorphism | |
| 273 | ||
| 274 | https://en.wikipedia.org/wiki/Trait_(computer_programming) | |
| 275 | ||
| 276 | https://en.wikipedia.org/wiki/Generic_programming | |
| 277 | ||
| 278 | This code is written in scheme. I know some of you want to use python | |
| 279 | instead, while others would prefer C++. More generally, it would | |
| 280 | probably be useful to set things up so that external, third-party | |
| 281 | software systems (such as SciPy or Gnu Octave or tensorflow) could | |
| 282 | be used to perform the analysis. Now that you've got the general | |
| 283 | idea... you can go do this! | |
| 284 | ||
| 285 | Anyway, if you are willing to use scheme, here's what we've got, | |
| 286 | ||
| 287 | Low-level API | |
| 288 | ------------- | |
| 289 | Every user needs to implement a "low-level API" object that describes | |
| 290 | the pairs, and provides methods to get them, and to get the associated | |
| 291 | count (or other numeric value). | |
| 292 | ||
| 293 | The methods that need to be implemented are described in | |
| 294 | `object-api.scm`; a working example is in `eval-pair.scm`. | |
| 295 | Additional working examples of the base classes can be found in | |
| 296 | https://github.com/opencog/learn/tree/master/scm/batch-word-pair.scm | |
| 297 | and in | |
| 298 | https://github.com/opencog/learn/tree/master/scm/pseudo-csets.scm | |
| 299 | ||
| 300 | Basic definitions | |
| 301 | ----------------- | |
| 302 | Some notation: | |
| 303 | ||
| 304 | Let `N(x,y)` be the observed count on the pair of atoms `(x,y)`. | |
| 305 | ||
| 306 | The `add-support-api` class provides an API to report the partial | |
| 307 | sums `N(x,*) = sum_y N(x,y)` and likewise `N(*,y)`. If you think of | |
| 308 | `N(x,y)` as a matrix, these are the totals for the entries in each | |
| 309 | row or column of the matrix. Likewise, `N(*,*) = sum_x sum_y N(x,y)`. | |
| 310 | In statistics, these are called "marginals", because you write the | |
| 311 | subtotals in the "margins" of your table. | |
| 312 | ||
| 313 | The `add-pair-freq-api` class provides an API to report the frequencies | |
| 314 | of pairs, and the partial sums over rows and columns. The frequency | |
| 315 | is defined, by default, to be `p(x,y) = N(x,y)/N(*,*)`. The row and | |
| 316 | column sums are `p(x,*) = sum_y p(x,y)`. By default, these total to | |
| 317 | one, as all good probabilities should: `1 = sum_x sum_y p(x,y)`. | |
| 318 | ||
| 319 | The `add-report-api` class provides an API to report summary information | |
| 320 | about the matrix, including the dimensions of the matrix (the number of | |
| 321 | rows and columns), the total number of non-zero entries (which is the | |
| 322 | same as the total number of unique pairs), the left, right and total | |
| 323 | entropies and mutual information. | |
| 324 | ||
| 325 | Computing basic statistics | |
| 326 | -------------------------- | |
| 327 | These `add-pair-*-api` classes simply provide methods to fetch these | |
| 328 | values, which are presumed to have been precomputed in some way. Several | |
| 329 | classes are provided to compute these. | |
| 330 | ||
| 331 | The `add-pair-stars` class provides methods to fetch the set | |
| 332 | `{x | the atom x occurs in some pair (x,y)}`. This is called the | |
| 333 | `left-support-set`. Likewise for the right-support. | |
| 334 | Another method returns the wild-card pairs: that is, the set | |
| 335 | `(x,*) = {(x,y) | x is fixed and y is any atom in the pair}` | |
| 336 | This is called the `right-star` because the right-side of the pair is | |
| 337 | a wild-card. These methods are useful for iterating over all rows and | |
| 338 | columns of the correlation matrix. | |
| 339 | ||
| 340 | The `add-loop-api` class provides methods to loop over all non-zero | |
| 341 | pairs in the matrix, and invoke a function on them. There are two | |
| 342 | methods: `for-each-pair`, which simply calls the function for each | |
| 343 | pair, and a `map-pair` method which returns a list of the results of | |
| 344 | calling the function on each pair. XXX Should be merged with the | |
| 345 | `batch-similarity` object, which does a similar thing, and also with | |
| 346 | `loop-upper-diagonal`, which does diagonal looping. | |
| 347 | ||
| 348 | The `add-support-compute` class provides methods to compute the | |
| 349 | partial sums `N(*,y)` and `N(x,*)`. It also provides methods that | |
| 350 | compute how many non-zero entries there are in each row or column. | |
| 351 | It also provides methods for the "length" of a column: that is, | |
| 352 | `len(y) = sqrt(sum_x N^2 (x,y))` and more generally the l_p norm. | |
| 353 | Because these computations can take a considerable amount of time, | |
| 354 | the partial sums (the "marginals") are cached. The cached values can | |
| 355 | be accessed with the `add-support-api` object. | |
| 356 | ||
| 357 | The `make-compute-freq` class provides methods to compute and cache | |
| 358 | the frequencies `P(x,y)`, `P(*,y)` and `P(x,*)`. These are cached | |
| 359 | exactly where the `add-pair-freq-api` class, above, can find them. | |
| 360 | ||
| 361 | The `make-batch-mi` class provides methods to compute the fractional | |
| 362 | mutual information of all pairs, namely the value | |
| 363 | ``` | |
| 364 | MI(x,y) = +log_2 P(x,y) / P(x,*) P(*,y) | |
| 365 | ``` | |
| 366 | ||
| 367 | The `batch-all-pair-mi` class is a convenience class that wraps up all | |
| 368 | three classes above, and unleashes them on a dataset: first computing | |
| 369 | the row and column partial counts, then computing the frequencies, and | |
| 370 | then computing and caching the mutual information. For large datasets, | |
| 371 | e.g. tens of millions of atoms, this can take hours to run. Thus, for | |
| 372 | this reason, the cached values are then saved to the currently-open | |
| 373 | database, so that these results become available later. | |
| 374 | ||
| 375 | Computing entropy | |
| 376 | ----------------- | |
| 377 | The `add-entropy-compute` class provides methods to compute the entropy | |
| 378 | and mutual information of rows and columns: for example, the column | |
| 379 | entropy (or `left-entropy`) `h_left(y) = -sum_x P(x,y) log_2 P(x,y)` | |
| 380 | It also returns the far more interesting 'fractional entropy', given | |
| 381 | by `H_left(y) = h_left(y) / P(*,y)`. Along similar lines, there is | |
| 382 | also the mutual information `mi_left(y) = sum_x P(x,y) log_2 MI(x,y)` | |
| 383 | where `MI(x,y) = +log_2 P(x,y) / P(x,*) P(*,y)` is the fractional pair | |
| 384 | MI. | |
| 385 | ||
| 386 | Methods are also provided to compute the total entropy: | |
| 387 | `H_tot = sum_x sum_y P(x,y) log_2 P(x,y)` as well as | |
| 388 | the left and right entropies: `H_left = sum_y P(*,y) log_2 P(*,y)` | |
| 389 | and v.v. | |
| 390 | ||
| 391 | Computing similarity measures | |
| 392 | ----------------------------- | |
| 393 | The `add-similarity-compute` class provides methods for taking the | |
| 394 | vector product of two different rows or columns, viz. the product | |
| 395 | `left-prod(y,z) = sum_x N(x,y) N(x,z)` and likewise for the right. | |
| 396 | The cosine similarity is | |
| 397 | ``` | |
| 398 | left-cosine(y,z) = left-prod(y,z) / (left-length(y) * left-length(z)) | |
| 399 | ``` | |
| 400 | where | |
| 401 | ``` | |
| 402 | left-length(y) = sqrt sum_x N(x,y) N(x,y) | |
| 403 | = sqrt left-prod(y,y) | |
| 404 | ```` | |
| 405 | The class also provides several other similarity measures, | |
| 406 | including the Jaccard similarity (Ruzicka distance) | |
| 407 | ``` | |
| 408 | left-jacc-sim(y,z) = sum_x min (N(x,y), N(x,z)) / | |
| 409 | sum_x max (N(x,y), N(x,z)) | |
| 410 | ``` | |
| 411 | and the Probability-Jaccard distance (a maximally consistent distance | |
| 412 | for probability distributions; see Wikiepdia for details). This class | |
| 413 | also provides a plain overlap similarity function. | |
| 414 | ||
| 415 | ||
| 416 | Direct sums | |
| 417 | ----------- | |
| 418 | If one has a collection of things that can be vectors in two different | |
| 419 | ways, then the direct sum can be used to create a single vector out of | |
| 420 | the two. It can be thought of as a concatenation of the two vectors, | |
| 421 | appending one to the other to create a single vector. | |
| 422 | ||
| 423 | For example, a word "foo" can be associated with a vector of disjuncts. | |
| 424 | It can also be associated with a vector of connectors that it appears | |
| 425 | in (a "shape"). These are two completely different kinds of vectors; | |
| 426 | all they have in common is that both are associated with the word "foo". | |
| 427 | As matrices, one consists of (word,disjunct) pairs, while the other | |
| 428 | consists of (word,shape) pairs. The left-side of the matrix (the rows | |
| 429 | of the matrix) all have the same type (i.e. words) while the right | |
| 430 | side (the columns) are two different types (and are completely disjoint, | |
| 431 | as sets). Concatenating these two matrices, placing one after the other, | |
| 432 | results in a single matrix where the rows are result of appending the | |
| 433 | rows of the second matrix to the first. If the dimensions of the two | |
| 434 | matrices are N x M and N x K, then the dimensions of the combined matrix | |
| 435 | would be N x (M + K). | |
| 436 | ||
| 437 | The `direct-sum` class will combine these two matrices to provide an | |
| 438 | object that behaves like a single matrix. The left and right basis | |
| 439 | elements will be the set-union of the left and right basis elts of each | |
| 440 | component matrix. By assumption, the types of either the columns or | |
| 441 | the rows are distinct, and so one of these sets is disjoint. Thus, the | |
| 442 | union is unambiguous; the matrices are either placed side-by-side by | |
| 443 | concatenating rows, or above-below, by concatenating columns. There is | |
| 444 | no "overlap". The total number of non-zero entries in the combined | |
| 445 | matrix is the sum of the number of non-zero entries in each component. | |
| 446 | ||
| 447 | (Caution: this is not the conventional definition of a direct sum, which | |
| 448 | results in block-diagonal matrix. The conventional definition takes the | |
| 449 | disjoint-union of the indexes (the basis), whereas here, we take the | |
| 450 | set-union of the basis. The set-union makes more sense in the current | |
| 451 | context, because the rows and columns have explicit labels, and it is | |
| 452 | the labels that are important. The set union is obtained from the | |
| 453 | disjoint union by means of a projection.) | |
| 454 | ||
| 455 | ||
| 456 | Working with rows and columns | |
| 457 | ----------------------------- | |
| 458 | The `add-tuple-math` class provides methods for applying arbitrary | |
| 459 | functions to arbitrary sets of rows or columns. The simplest examples | |
| 460 | include taking the sums and differences of columns, taking the | |
| 461 | element-by-element min or max of a set of columns, counting the number | |
| 462 | of entries that are simultaneously non-zero in sets of columns, etc. | |
| 463 | ||
| 464 | The class works by defining a new matrix, whose rows or columns | |
| 465 | are tuples (pairs, triples, etc) of the underlying matrix. For example, | |
| 466 | one can ask for the matrix entry `(x, [y,z,w])`. The value returned | |
| 467 | will be `(x, f((x,y), (x,z), (x,w)))` where the user-defined function | |
| 468 | `f` was used to create the `x` row. | |
| 469 | ||
| 470 | ||
| 471 | Gaussian Orthogonal Ensembles | |
| 472 | ----------------------------- | |
| 473 | The `add-gaussian-ortho-compute` object will take a (non-sparse) | |
| 474 | symmetric matrix as input, and normalize it such that the mean of all | |
| 475 | of the matrix entries is zero, and the standared deviation is one. | |
| 476 | This will result in a matrix that is (approximately) gaussian | |
| 477 | orthogonal. The rows and columns can then be understood to be | |
| 478 | (approximately) uniformly distributed about the origin, and when | |
| 479 | normalized to unit length, can be understood to lie on the surface | |
| 480 | of a sphere $S_{N-1}$ where the original matrix is N x N dimensional. | |
| 481 | ||
| 482 | ||
| 483 | Principal Component Analysis | |
| 484 | ---------------------------- | |
| 485 | A power iteration object is provided by the `make-power-iter-pca` | |
| 486 | object. Once a matrix `X` has been specified, this will iterate on | |
| 487 | either `X^T X` or on `XX^T` for `K` iterations. The result of this | |
| 488 | iteration is the principal component of `X` (equivalently, the | |
| 489 | Frobenius-Peron eigenvector of `X^T X` or `XX^T`). | |
| 490 | ||
| 491 | The code designed to work fast for sparse matrices, and can obtain | |
| 492 | eigenvectors in under 20 seconds or so for 15K by 15K matrices with | |
| 493 | 100K non-zero entries. | |
| 494 | ||
| 495 | The code is beta; only a bare minimum of function is provided. So far, | |
| 496 | PCA is not terribly interesting for the kinds of graph problems we are | |
| 497 | interested in solving. | |
| 498 | ||
| 499 | ||
| 500 | Data Cuts | |
| 501 | --------- | |
| 502 | Data will typically have lots of "noise" in it, which should be masked | |
| 503 | before data analysis starts. The tools in `filter.scm` can do this. A | |
| 504 | generic filter will mask out arbitrary rows, columns and individual | |
| 505 | entries, depending on the supplied predicates. Built on top of this is | |
| 506 | a filter that masks out rows, columns and entries that have counts | |
| 507 | below a threshold. Another filter can mask out explicitly-named rows | |
| 508 | and columns. | |
| 509 | ||
| 510 | Unfortunately, filtering can be slow, and loading a large matrix only | |
| 511 | to access small subsets of it is wasteful of RAM, and can add CPU | |
| 512 | overhead. Thus, a similar set of interfaces is provided in `trim.scm`, | |
| 513 | which will remove (delete) rows, columns and matrix entries from both | |
| 514 | the AtomSpace, and from attached storage. With appopropriate trimming, | |
| 515 | noisy datasets can be reduced in size by factors of two, ten or a | |
| 516 | hundred. This can have a huge impact on processing. | |
| 517 | ||
| 518 | ||
| 519 | Tensors, in general | |
| 520 | ------------------- | |
| 521 | Suppose you have more than just pairs. Suppose you have triples that | |
| 522 | you want to work with. Then what? Answer: use the network analysis | |
| 523 | tools in the `(opencog network)` module. | |
| 524 | ||
| 525 | Reading List | |
| 526 | ------------ | |
| 527 | If you choose to read the source code, read it in the following order: | |
| 528 | ||
| 529 | * object-api.scm | |
| 530 | * eval-pairs.scm | |
| 531 | * support.scm | |
| 532 | * transpose.scm | |
| 533 | * similarity-api.scm | |
| 534 | * cosine.scm | |
| 535 | * symmetric-mi.scm | |
| 536 | * group-similarity.scm | |
| 537 | * compute-mi.scm | |
| 538 | * trans-batch.scm | |
| 539 | * filter.scm | |
| 540 | * trim.scm | |
| 541 | * direct-sum.scm | |
| 542 | ||
| 543 | There are more files not mentioned in this list, but they are less relevant. | |
| 544 | ||
| 545 | TODO | |
| 546 | ---- | |
| 547 | To-do list items. | |
| 548 | * The "star" objects need to be redesigned. They fetch wild-card counts | |
| 549 | and other marginal values straight out of the AtomSpace. But those | |
| 550 | marginal values are correct only if no filtering is applied. If there | |
| 551 | are pre-filters, then the returned marginals are garbage. Yucko. Can | |
| 552 | we fail-safe this for now? | |
| 553 | ||
| 554 | * Need to support columns/rows that can be one of several types | |
| 555 | (e.g. can be WordNodes, or be WordClassNodes) This is currently | |
| 556 | handled in an ad hoc manner. | |
| 557 | ||
| 558 | * Need a more consistent/coherent API for interacting with storage. | |
| 559 | Currently, the `fetch` methods do this, and there is a scattering | |
| 560 | of other store methods provided in an ad hoc, as-needed basis. All | |
| 561 | this should be made prettier. |