| … | … | @@ -4,12 +4,14 @@ September 2025 |
| 4 | 4 | |
| 5 | 5 | Notes about implementing Atomese descriptions of RNN's an LSTM. |
| 6 | 6 | |
| 7 | | Embedding Atoms into RNN and LSTM |
| 8 | | ---------------------------------- |
| 7 | Describing NN's with Atomese |
| 8 | ---------------------------- |
| 9 | 9 | The task here is to be able to describe the structure of RNN's, LSTM and |
| 10 | 10 | similar networks in "pure Atomese", i.e. using PlusLink, TimesLink and |
| 11 | 11 | so on to describe the vector operations. |
| 12 | 12 | |
| 13 | Embedding Atoms into RNN and LSTM |
| 14 | ---------------------------------- |
| 13 | 15 | A subtrask is to create vector embeddings for Atoms. Several types of |
| 14 | 16 | vector embeddings are posible: |
| 15 | 17 | * Every Atom has a 64-bit hash value; this could be converted to a |
| … | … | @@ -19,3 +21,30 @@ vector embeddings are posible: |
| 19 | 21 | * Conventional RNN's and LSTM's use float vector embeddings. How would this |
| 20 | 22 | work? |
| 21 | 23 | |
| 24 | Vector Embeddings |
| 25 | ----------------- |
| 26 | A reasonable way of doing a vector embedding for some Atom is to have |
| 27 | ``` |
| 28 | (cog-set-value! (Atom...) (Predicate "some-key") (FloatValue...)) |
| 29 | ``` |
| 30 | where the `FloatValue` would be N floats long (for N=768 or 1024 or |
| 31 | whatever), and the initial value would be generated using a deterministic |
| 32 | algo that used the 64-bit atom hash value as an RNG seed, generating N |
| 33 | floats (deterministically) from this seed. |
| 34 | |
| 35 | Several issues: |
| 36 | * What should they key be? |
| 37 | * How is the float vector generated? |
| 38 | * What is being predicted/trained on? |
| 39 | |
| 40 | ### What should they key be? |
| 41 | We need to allow for (many) different embeddings, all available at the same |
| 42 | time. The specific set of Atoms to be embedded in some specific network |
| 43 | emerge as the result of running some QueryLink over the AtomSpace. This is |
| 44 | similar to how the "matrix" code works today: the vocabulary is limited to |
| 45 | a specific set of Atoms that respond to some specific Query (or other |
| 46 | identification.) |
| 47 | |
| 48 | Thus, it makes sense for the key to be the specific Atom, usually a |
| 49 | QueryLink, but also maybe any other atom that specifies the specific |
| 50 | vocabulary for the specific embedding. |