| … | … | @@ -241,7 +241,51 @@ Whatever. There are nicer clustering algos (which are more complex) that |
| 241 | 241 | can maybe do a better job. That's not the point. |
| 242 | 242 | |
| 243 | 243 | ### Axioms! |
| 244 | | |
| 245 | | |
| 244 | The word "axiom" is a fancy way of saying "we have abstracted down our |
| 245 | collection of data to a very small number of clusters, and these |
| 246 | clusters, still being jigsaw-like in nature, can still be assembled to |
| 247 | generate structures. |
| 248 | |
| 249 | For example, all of set theory can be reduced to 17 axioms. As the set |
| 250 | theory textbooks all point out: "Ooops! Actually 17 axioms plus an axiom |
| 251 | schema! Too bad, so sad, lets continue." |
| 252 | |
| 253 | So, K-means clustering, as described above, will fail to discover the |
| 254 | axioms of mathematics, at least in the naive first attempt. If we start |
| 255 | from raw 3D space and time, then the perceived sensory data is what |
| 256 | conventional robots peceive: sight, sound, radar, sonar. Whatever. |
| 257 | |
| 258 | We can certainly design robots that crawl filesystems, or crawl social |
| 259 | media posts, and the clustering there will be textual. Experimental |
| 260 | results of actually doing this gives, to first order in clustering, |
| 261 | results that are comparable to results from the early days of Word2Vec, |
| 262 | GloVE, and skipgrams, even though the algo I describe above is |
| 263 | ***completely different***. |
| 264 | |
| 265 | ### Recursion |
| 266 | The jigsaw-based approach above is resursive. Once I've created |
| 267 | clusters of jigsaws, I can create new overlay networks (out of the |
| 268 | jigsaw pieces, instead of pair-wise edges), and then repeat the process. |
| 269 | The resulting pieces are more complex, more abstract. This recursive |
| 270 | experiment is where things stopped last time; I want to restart, here, |
| 271 | and see if it is as easily as I glibly hand-wave here. |
| 272 | |
| 273 | It is easy to imagine that the recursive step will be difficult, and |
| 274 | fail. There is a counterpoint: each layer of a deep-learning system |
| 275 | is a kind-of-ish-like recursive step. We de facto know that LLM's work |
| 276 | really well. My personal belief is that a recursive jig-sawing of $M$ |
| 277 | steps will give capabilities and results comparable to a $M$-layer deep |
| 278 | neural net. |
| 279 | |
| 280 | But the algo is very very different. LLM's are all about gradient |
| 281 | descent and weight matrices. I'm talking about sheaves and jigsaws and |
| 282 | self-assembly. Critical points and thermodynamics. It is foundationally |
| 283 | completely different. |
| 284 | |
| 285 | (Who the heck am I writing this text for? This was going to be a think |
| 286 | piece for my own personal consumption, but its turned into an exposition |
| 287 | for the lay audience. ... Which is good for the audience, but bad for |
| 288 | me, because I am repeating things I already know and believe, rather |
| 289 | than moving forwards.) |
| 246 | 290 | |
| 247 | 291 | pairs. |