| … | … | @@ -10,12 +10,16 @@ similar "objects", much like all of the rows of a database table are |
| 10 | 10 | "similar". That collection of vertexes/edges can be thought of as |
| 11 | 11 | forming a vector. Such a vector is sparse, in that it is built out of |
| 12 | 12 | only some small fragment of the total graph. If the graph is big enough, |
| 13 | | there might be thousands (or millions) of such vectors, and then one is |
| 13 | there might be thousands (or millions) of such vectors. One is then |
| 14 | 14 | typically interested in doing conventional vector, matrix, tensor |
| 15 | 15 | analysis on that collection of (sparse) vectors. This repo contains |
| 16 | 16 | a library/toolset for performing this kind of analysis: correlation, |
| 17 | | covariance, mutual information, similarity measures of various kinds, |
| 18 | | and more. |
| 17 | covariance, mutual information, overlap, Hamming distance, similarity |
| 18 | measures of various kinds, and more. |
| 19 | |
| 20 | In essence, large graphs have many similar sub-patterns and structures |
| 21 | inside of them. This is a toolkit for discovering and working with such |
| 22 | similar subgraphs. |
| 19 | 23 | |
| 20 | 24 | ### Sparse and Dense Vectors |
| 21 | 25 | These are *not DL/NN vectors*. The vectors and weight matrices that |
| … | … | @@ -23,27 +27,55 @@ arise in deep learning neural nets (transformers, generative AI, etc.) |
| 23 | 27 | are dense, not sparse. If a typical DL/NN vector is one-million-dimensional, |
| 24 | 28 | then a dense vector will have all million basis coefficients be non-zero. |
| 25 | 29 | A sparse vector will have all except a small number (say, a thousand) be |
| 26 | | zero. |
| 30 | zero. So, if 1,000 of 1,000,000 entries are non-zero, this means that |
| 31 | 99.9% of the sparse vector is zero. |
| 27 | 32 | |
| 28 | | This has storage implications: if 99% of a vector is zero, you don't need |
| 29 | | to store those zeros! If 99% of a vector is zero, you don't need to pass |
| 30 | | it into your GPU's for processing, or send those zeros to some remote |
| 31 | | machine in a compute cluster. |
| 33 | This has all sorts of implications. If 99.9% of a vector is zero, you |
| 34 | don't need to store those zeros! (Who wants to fill up their RAM and |
| 35 | storage with 99.9% zeros???) If 99.9% of a vector is zero, you don't |
| 36 | need to pass those parts to a GPU for processing. Or send those zeros |
| 37 | to some remote machine in a compute cluster. Or even run a for-loop |
| 38 | over them. |
| 32 | 39 | |
| 33 | 40 | The [AtomSpace](https://github.com/opencog/atomspace) is a (hyper-)graph |
| 34 | 41 | database, and, as such, is highly optimized for sparse vector storage. |
| 35 | | Thus, all the code here is in reference to the AtomSpce (and not some |
| 36 | | other graph database.) |
| 42 | Thus, all the code here explicitly designed for the AtomSpace (and not |
| 43 | some other graph database.) |
| 37 | 44 | |
| 38 | | The code here is a library build from the ground-up. It does ***not*** |
| 45 | The code here is a library built from the ground-up. It does ***not*** |
| 39 | 46 | make use of pre-existing tools or libraries that can be found in R or |
| 40 | | SciPy. It would be nice if it was possible to use those systems: they |
| 47 | SciPy. It would be nice if it was possible to use those systems, as they |
| 41 | 48 | are very popular, with many many users. Unfortunately, both of them |
| 42 | 49 | choke on large vectors, and have no support for sparse vectors. If you |
| 43 | 50 | ask SciPy to create a million-dimensional vector, you'll run out of RAM |
| 44 | 51 | and then crash. The code in this directory routinely handles vectors of |
| 45 | 52 | this size. |
| 46 | 53 | |
| 54 | ### Results |
| 55 | Some notes to get your juices flowing, and get an idea of what this is |
| 56 | all about: |
| 57 | |
| 58 | * This library forms a central part of the Opencog |
| 59 | [Learn project](https://github.com/opencog/learn), where it is used to |
| 60 | extract structural similarities in natural language. |
| 61 | |
| 62 | * Some typical statistical results seen there resemble those of |
| 63 | traditional network analysis: vector lengths have a Zipfian |
| 64 | distribution, or sometimes a |
| 65 | [square-root Zipfian distribution](https://en.wikipedia.org/wiki/Wikipedia:Does_Wikipedia_traffic_obey_Zipf%27s_law%3F). |
| 66 | In such distributions, a handful of vectors are extremely long, |
| 67 | and almost all the rest are extremely short. This has been seen |
| 68 | in genetic, protein and reactome data (using this library) as |
| 69 | well as in natural language, and in Wikipedia reference pages |
| 70 | (using this library; see graph above). |
| 71 | |
| 72 | * When such vectors are classified according to pair-wise similarity, |
| 73 | they form a Gaussian Orthogonal Ensemble (GOE). This means that |
| 74 | traditional principles from physics and statistical mechanics can be |
| 75 | applied to them. Of course, the 2024 Nobel Prize was awarded for this |
| 76 | kind of work: but it is for *dense* vectors, as they appear in neural |
| 77 | nets. It turns out that such ideas hold for sparse vectors as well. |
| 78 | |
| 47 | 79 | Motivation |
| 48 | 80 | ---------- |
| 49 | 81 | In symbolic AI, there is a generic need to perform "reasoning" or |