| … | … | @@ -13,15 +13,15 @@ there. However, I want to incorporate the new ideas here, and so, for |
| 13 | 13 | now, this chapter is here. |
| 14 | 14 | |
| 15 | 15 | The language learning started with computing pair-wise MI between words, |
| 16 | | then uusing this to create a network between words, then breaking up the |
| 16 | then using this to create a network between words, then breaking up the |
| 17 | 17 | network into jigsaws. The jigsaws were then clustered, and the clusters |
| 18 | 18 | then used to parse the same text, again. This all worked fine. The next |
| 19 | 19 | step of the plan was to then repeat: find correlations between these |
| 20 | | clustered jigsaws, then tile again, etc. ad ininfinitum. That step was |
| 20 | clustered jigsaws, then tile again, etc. ad infinitum. That step was |
| 21 | 21 | never taken, for two reasons: |
| 22 | 22 | |
| 23 | 23 | * Managing the batch processing pipeline became overwhelming. The |
| 24 | | propsoed solution to that was to create sensori-motor agents that |
| 24 | proposed solution to that was to create sensori-motor agents that |
| 25 | 25 | could crawl the data "of their own volition"; or rather, automated |
| 26 | 26 | in such a way that I could aim agents at the data and have them |
| 27 | 27 | process and digest without my explicit intervention at a low level. |
| … | … | @@ -38,7 +38,7 @@ never taken, for two reasons: |
| 38 | 38 | at all with a sensori-motor design. |
| 39 | 39 | |
| 40 | 40 | The solution to that stumbling block was to develop the concept of |
| 41 | | "flows" in Atomese: of flowng data that travels through networks |
| 41 | "flows" in Atomese: of flowing data that travels through networks |
| 42 | 42 | defined declaratively in Atomese. This required lots of low-level |
| 43 | 43 | grunt-work: defining Atoms that could add, multiply, accumulate, |
| 44 | 44 | take dot-products, mask bits, transpose columns and tows, but also |
| … | … | @@ -103,23 +103,23 @@ axioms are specific building blocks that are assembled. |
| 103 | 103 | The rules and axioms are jigsaw pieces; theorem proving is the assembly |
| 104 | 104 | of these. If it is free an unconstrained, then it is generation; if its |
| 105 | 105 | constrained, then some sort of satisfiability solver is needed: the |
| 106 | | jigsaw much assmble so as to build be bridge between constraints. The |
| 106 | jigsaw much assemble so as to build be bridge between constraints. The |
| 107 | 107 | satisfiability algorithm must be fast, but also the bridge that is build |
| 108 | | mst be minimal or optimal in some way. |
| 108 | must be minimal or optimal in some way. |
| 109 | 109 | |
| 110 | 110 | ### Minimization |
| 111 | 111 | The words "minimal" and "optimal" make me personally think of minimal |
| 112 | 112 | surfaces: soap bubbles, for instance. Geodesics in Riemannian geometry. |
| 113 | 113 | Assemblies of jigsaws do not have smooth differentiable structure, but |
| 114 | 114 | still, some of the ideas carry over analogously, because there are |
| 115 | | several natural ideas of "minimal" for jigaw assemblies. For example: |
| 115 | several natural ideas of "minimal" for jigsaw assemblies. For example: |
| 116 | 116 | the assembly that uses the minimal number of pieces. Or, if we weight |
| 117 | 117 | each piece with e.g. log probability (i.e. "entropy" or "information") |
| 118 | 118 | then minimize the sum of that. |
| 119 | 119 | |
| 120 | 120 | ### Homtopic deformation |
| 121 | | The general "feel" of continutity is taken up by the Scott topology, |
| 122 | | where we can talk about the homotopic deformation of jigsaw aassemblies. |
| 121 | The general "feel" of continuity is taken up by the Scott topology, |
| 122 | where we can talk about the homotopic deformation of jigsaw assemblies. |
| 123 | 123 | There is a very easy way of understanding this if you are a software |
| 124 | 124 | programmer: a "homotopic deformation" is a "refactoring of the |
| 125 | 125 | software". Any collection of small changes that can be made to a |
| … | … | @@ -138,7 +138,7 @@ work on bare assembly, but in abstract insn sets, the "intermediate |
| 138 | 138 | language" being a way-point. |
| 139 | 139 | |
| 140 | 140 | But this analogy can be pushed "all the way": A compiler is a homotopic |
| 141 | | defomration of a C/C++ program into assembly code; an assembler is a |
| 141 | deformation of a C/C++ program into assembly code; an assembler is a |
| 142 | 142 | further deformation (and linking!) into machine binary. A bridge between |
| 143 | 143 | two different representations that does not alter the function: Whatever |
| 144 | 144 | the programmer wrote the C/C++ code to do, the machine binary will do |
| … | … | @@ -158,9 +158,9 @@ equations (the solution of which defines geodesics)? I don't know. |
| 158 | 158 | Perhaps. People already run around claiming that one should maximize |
| 159 | 159 | free energy (Gibbs free energy) or perhaps the "integrated information"; |
| 160 | 160 | all of these ideas imply some kind of differentiable structure minimized |
| 161 | | or maximized via some Euler-Lagrange-style differentail or difference |
| 161 | or maximized via some Euler-Lagrange-style differential or difference |
| 162 | 162 | eqns. I don't want to get hung up on the details, but rather the |
| 163 | | artistic impression: when there is stuff to be minimized and maximzed, |
| 163 | artistic impression: when there is stuff to be minimized and maximized, |
| 164 | 164 | and when one has paths that can take you from here to there in some |
| 165 | 165 | quasi-continuous fashion, then one has some local difference-type |
| 166 | 166 | gradient that helps figure out how to move. |
| … | … | @@ -185,7 +185,7 @@ I want, and I think it is possible, to have a system that self-reifies: |
| 185 | 185 | a self-assembly of pieces that can figure out how to reason, deduce, |
| 186 | 186 | prove, use logic and discover axioms. A system that can perceive |
| 187 | 187 | structure, in a "sensori-" way, and can manipulate structure, in a |
| 188 | | "motor" kind of way. So a sensori-motor system that percives structure, |
| 188 | "motor" kind of way. So a sensori-motor system that perceives structure, |
| 189 | 189 | and acts on, manipulates that structure. |
| 190 | 190 | |
| 191 | 191 | ### Perceiving structure |
| … | … | @@ -215,7 +215,7 @@ $ |
| 215 | 215 | A . B = Sum_{p,q,r..} N(A | p,q,r...) N(B | p,q,r...) |
| 216 | 216 | $ |
| 217 | 217 | In practice, (i.e. experimentally measured) the dot products are |
| 218 | | gaussian distributed, i.e. the data is distributed uniformly on a very |
| 218 | Gaussian distributed, i.e. the data is distributed uniformly on a very |
| 219 | 219 | high dimensional sphere. |
| 220 | 220 | |
| 221 | 221 | ### Clustering |
| … | … | @@ -229,7 +229,7 @@ on it, and assign vectors to the closest one. Duh. That is, each |
| 229 | 229 | vertex label in the set {A, B, ...} is assigned to one of K different |
| 230 | 230 | clusters. |
| 231 | 231 | |
| 232 | | Unfortunately, this K-means clustering is horrble and ugly, because |
| 232 | Unfortunately, this K-means clustering is horrible and ugly, because |
| 233 | 233 | although it reduced the bras $(A|$ to a small number, it did not reduce |
| 234 | 234 | the kets $|p,q,r...)$, of which there are still a gazzilion. Well, |
| 235 | 235 | sort-of. Given that each p,q,r,... is a vertex indicator plus a sex |
| … | … | @@ -253,7 +253,7 @@ schema! Too bad, so sad, lets continue." |
| 253 | 253 | So, K-means clustering, as described above, will fail to discover the |
| 254 | 254 | axioms of mathematics, at least in the naive first attempt. If we start |
| 255 | 255 | from raw 3D space and time, then the perceived sensory data is what |
| 256 | | conventional robots peceive: sight, sound, radar, sonar. Whatever. |
| 256 | conventional robots perceive: sight, sound, radar, sonar. Whatever. |
| 257 | 257 | |
| 258 | 258 | We can certainly design robots that crawl filesystems, or crawl social |
| 259 | 259 | media posts, and the clustering there will be textual. Experimental |
| … | … | @@ -291,7 +291,7 @@ than moving forwards.) |
| 291 | 291 | Sound and Vision |
| 292 | 292 | ---------------- |
| 293 | 293 | Writing the above was procrastination for writing this section. But now |
| 294 | | I'm so bored, I don't feel like writing this sectiion. Alas. |
| 294 | I'm so bored, I don't feel like writing this section. Alas. |
| 295 | 295 | |
| 296 | 296 | The idea in a nutshell was to restart the jigsaw learning process, this |
| 297 | 297 | time not just for natural language, but also for vision and sound. |
| … | … | @@ -322,14 +322,14 @@ agent to experiment with blur filter size? What is it doing, except |
| 322 | 322 | being forced to dream? |
| 323 | 323 | |
| 324 | 324 | ### Rubiks |
| 325 | | Adam proposed somethng slightly more agentic: solving the rubiks cube. |
| 325 | Adam proposed something slightly more agentic: solving the rubiks cube. |
| 326 | 326 | The agent can "see" all the facets of the cube, and the agent can apply |
| 327 | 327 | any of the basic moves, and see the resulting changes. The learning task |
| 328 | 328 | is to see if the agent can learn to solve the cube. |
| 329 | 329 | |
| 330 | 330 | How should I think of this? "Solving" is finding a set of moves that |
| 331 | | take it from one (predeterimined, given) location in configuration |
| 332 | | space to another (predetermined, requested) location in confidguration |
| 331 | take it from one (predetermined, given) location in configuration |
| 332 | space to another (predetermined, requested) location in configuration |
| 333 | 333 | space. To build a bridge, a sequence of moves, from one to the other. To |
| 334 | 334 | solve a certain constraint satisfaction problem. |
| 335 | 335 | |
| … | … | @@ -338,7 +338,7 @@ possible moves, and search for shortest paths from one configuration to |
| 338 | 338 | another". It would need to remember things that it tried. How do I, as |
| 339 | 339 | a programmer, encode positions? How do I give it efficient high-speed |
| 340 | 340 | search of memory of prior configurations and moves to get there? This |
| 341 | | trns into an engineering task, where I do the difficult engineering, so |
| 341 | turns into an engineering task, where I do the difficult engineering, so |
| 342 | 342 | that a small and stupid agent can rapidly solve a Rubiks cube? I don't |
| 343 | 343 | think so. |
| 344 | 344 | |