| … | … | @@ -770,7 +770,271 @@ H\left(S\right)=\sum_{G\in\mathcal{G}}h\left(G\right) |
| 770 | 770 | \end_inset |
| 771 | 771 | |
| 772 | 772 | This is the appropriate form, because we want to consider all possible disjunct |
| 773 | | cost assignments, and minimize those |
| 773 | cost assignments, and minimize those. |
| 774 | Unlike the earlier case where multiple distinct (ambiguous) parses can |
| 775 | be tolerated and are even desirable, here, not so much. |
| 776 | Here, we do want to discover single unique, unambiguous collection of disjuncts. |
| 777 | It is hard to understand why more than one would be desired. |
| 778 | Perhaps some hand–waving is possible, but hard to see what that is. |
| 779 | \end_layout |
| 780 | |
| 781 | \begin_layout Standard |
| 782 | So lets try that. |
| 783 | Lets say that there is |
| 784 | \begin_inset Quotes eld |
| 785 | \end_inset |
| 786 | |
| 787 | one true set |
| 788 | \begin_inset Quotes erd |
| 789 | \end_inset |
| 790 | |
| 791 | of disjuncts that guide the parsing and understanding of the external world. |
| 792 | If they are correct and accurate, then all is well and we continue onwards. |
| 793 | Sometimes, model updates would be needed. |
| 794 | Most of these should be small, but it is easy to imagine avalanches in |
| 795 | |
| 796 | \begin_inset Quotes eld |
| 797 | \end_inset |
| 798 | |
| 799 | understanding |
| 800 | \begin_inset Quotes erd |
| 801 | \end_inset |
| 802 | |
| 803 | : if some lexis no longer parses reality, that lexis may need an update. |
| 804 | That update may be dramatic. |
| 805 | \end_layout |
| 806 | |
| 807 | \begin_layout Standard |
| 808 | So why would one want to keep around multiple different lexis? Well, to |
| 809 | try them out, maybe see how they differ. |
| 810 | In this model, each lexis is an |
| 811 | \begin_inset Quotes eld |
| 812 | \end_inset |
| 813 | |
| 814 | individual |
| 815 | \begin_inset Quotes erd |
| 816 | \end_inset |
| 817 | |
| 818 | , and that individual has it's own view of what the best kinds of parses |
| 819 | might be. |
| 820 | Most individuals agree, but clearly, if the training corpus for one individual |
| 821 | is a bit different from that of another individual, there will be differences. |
| 822 | \end_layout |
| 823 | |
| 824 | \begin_layout Standard |
| 825 | So I'm having fun with this: any particular collection of disjuncts is an |
| 826 | individual. |
| 827 | This can of course be taken much more broadly: different LLM weight–sets |
| 828 | are indeed |
| 829 | \begin_inset Quotes eld |
| 830 | \end_inset |
| 831 | |
| 832 | individuals |
| 833 | \begin_inset Quotes erd |
| 834 | \end_inset |
| 835 | |
| 836 | , but this diary is not about LLM's, so we won't go there. |
| 837 | \end_layout |
| 838 | |
| 839 | \begin_layout Section* |
| 840 | Disjuncts |
| 841 | \end_layout |
| 842 | |
| 843 | \begin_layout Standard |
| 844 | It's time to firm up what the disjunct discovery process will be. |
| 845 | For that, an expression for |
| 846 | \begin_inset Formula $h\left(G\right)$ |
| 847 | \end_inset |
| 848 | |
| 849 | is needed. |
| 850 | So, |
| 851 | \begin_inset Formula $G$ |
| 852 | \end_inset |
| 853 | |
| 854 | is a graph, having edges between some words in a sentence, but not others. |
| 855 | Talking about words keeps things concrete, but most of the below should |
| 856 | generalize just fine for any kind of network graph. |
| 857 | \end_layout |
| 858 | |
| 859 | \begin_layout Standard |
| 860 | For each word |
| 861 | \begin_inset Formula $w_{i}$ |
| 862 | \end_inset |
| 863 | |
| 864 | in the sentence, define the disjunct |
| 865 | \begin_inset Formula $D_{i}$ |
| 866 | \end_inset |
| 867 | |
| 868 | to be the set of edges attached to that word: so |
| 869 | \begin_inset Formula |
| 870 | \[ |
| 871 | D_{i}=\left\{ E_{ij}:E_{ij}\ne0\right\} =\left\{ E_{ij}:E_{ij}\in G\right\} |
| 872 | \] |
| 873 | |
| 874 | \end_inset |
| 875 | |
| 876 | which are two different notations for the same thing: the edge |
| 877 | \begin_inset Formula $E_{ij}$ |
| 878 | \end_inset |
| 879 | |
| 880 | is in the graph |
| 881 | \begin_inset Formula $G$ |
| 882 | \end_inset |
| 883 | |
| 884 | . |
| 885 | The colon notation : means |
| 886 | \begin_inset Quotes eld |
| 887 | \end_inset |
| 888 | |
| 889 | such that |
| 890 | \begin_inset Quotes erd |
| 891 | \end_inset |
| 892 | |
| 893 | : the edges such that they belong to the graph. |
| 894 | Properly, the disjunct is an ordered list. |
| 895 | The sequence of the edges matters (because word–order matters). |
| 896 | |
| 897 | \end_layout |
| 898 | |
| 899 | \begin_layout Standard |
| 900 | Thus, instead of using set–builder notation |
| 901 | \begin_inset Formula $\left\{ \cdot\right\} $ |
| 902 | \end_inset |
| 903 | |
| 904 | we could use other kids of notation. |
| 905 | Don't freak, but here are other notations commonly found in the literature: |
| 906 | \begin_inset Formula |
| 907 | \[ |
| 908 | D_{i}=E_{ij_{1}}\otimes E_{ij_{2}}\otimes\cdots\otimes E_{ij_{k}} |
| 909 | \] |
| 910 | |
| 911 | \end_inset |
| 912 | |
| 913 | where |
| 914 | \begin_inset Formula $\otimes$ |
| 915 | \end_inset |
| 916 | |
| 917 | is the tensor product. |
| 918 | The reason that the tensor product notation is correct can be found in |
| 919 | a variety of other places, so I won't try to explain the motivation for |
| 920 | it. |
| 921 | Link Grammar uses the & instead of |
| 922 | \begin_inset Formula $\otimes$ |
| 923 | \end_inset |
| 924 | |
| 925 | to write down disjuncts. |
| 926 | The above is a tensor of arity |
| 927 | \begin_inset Formula $k$ |
| 928 | \end_inset |
| 929 | |
| 930 | – it has |
| 931 | \begin_inset Formula $k$ |
| 932 | \end_inset |
| 933 | |
| 934 | different connectors. |
| 935 | |
| 936 | \end_layout |
| 937 | |
| 938 | \begin_layout Standard |
| 939 | Another notation is the bra–ket notation: |
| 940 | \begin_inset Formula |
| 941 | \[ |
| 942 | D_{i}=\left|E_{ij_{1}}\right\rangle \otimes\left|E_{ij_{2}}\right\rangle \otimes\cdots\left|E_{ij_{m}}\right\rangle \otimes\left\langle E_{ij_{m+1}}\right|\otimes\cdots\otimes\left\langle E_{ij_{k}}\right| |
| 943 | \] |
| 944 | |
| 945 | \end_inset |
| 946 | |
| 947 | where kets are used whenever |
| 948 | \begin_inset Formula $j_{n}<i$ |
| 949 | \end_inset |
| 950 | |
| 951 | and bras are used when |
| 952 | \begin_inset Formula $j_{n}>i$ |
| 953 | \end_inset |
| 954 | |
| 955 | . |
| 956 | That is, |
| 957 | \begin_inset Formula $j_{m}$ |
| 958 | \end_inset |
| 959 | |
| 960 | is the index such that |
| 961 | \begin_inset Formula $j_{m}<i<j_{m+1}$ |
| 962 | \end_inset |
| 963 | |
| 964 | . |
| 965 | Note the inequalities are strict: a word ca never link to itself. |
| 966 | The point of bra–ket notation is to say that words with |
| 967 | \begin_inset Formula $j_{n}<i$ |
| 968 | \end_inset |
| 969 | |
| 970 | link to the left, otherwise they link to the right. |
| 971 | This is fine for English but just makes things hard for other languages |
| 972 | and for general graphs; the bra–ket notation is problematic for a number |
| 973 | of reasons. |
| 974 | One is that natural language is not a symmetric monoidal category: the |
| 975 | is no dagger operator. |
| 976 | It is not true that |
| 977 | \begin_inset Formula $\left|\cdot\right\rangle ^{\dagger}=\left\langle \cdot\right|$ |
| 978 | \end_inset |
| 979 | |
| 980 | . |
| 981 | Another problem is that bra–ket notation is inherently |
| 982 | \begin_inset Quotes eld |
| 983 | \end_inset |
| 984 | |
| 985 | heterosexual |
| 986 | \begin_inset Quotes erd |
| 987 | \end_inset |
| 988 | |
| 989 | , in that inner products can only be formed with one bra and one ket, whereas |
| 990 | in |
| 991 | \begin_inset Quotes eld |
| 992 | \end_inset |
| 993 | |
| 994 | real life |
| 995 | \begin_inset Quotes erd |
| 996 | \end_inset |
| 997 | |
| 998 | , connectors may come with other kinds of sexes and mating rules. |
| 999 | For example, monosexual connectors, where there is only one sex, are appropriat |
| 1000 | e for graphs with undirected edges, for which left–right ordering does not |
| 1001 | matter. |
| 1002 | In other languages, such as Turkish, Finnish and Lithuanian, word order |
| 1003 | mostly does not matter, but head–dependent order matters very strongly. |
| 1004 | For those languages, there are four sexes: a product of the dominant head–depen |
| 1005 | dent dependency, and a much weaker left–right mating distinction for a small |
| 1006 | minority of words. |
| 1007 | Enough linguistics for now; this gives a flavor of some of the issues that |
| 1008 | come up in practice. |
| 1009 | \end_layout |
| 1010 | |
| 1011 | \begin_layout Standard |
| 1012 | Although the bra–ket notation is problematic, the tensor notation is not: |
| 1013 | the non–symmetric monoidal category really is the appropriate category |
| 1014 | for language. |
| 1015 | It really is a fragment of linear logic. |
| 1016 | See Wikipedia articles on pre–group grammars. |
| 1017 | See all my (Vepstas) paper on |
| 1018 | \begin_inset Quotes eld |
| 1019 | \end_inset |
| 1020 | |
| 1021 | Categorial Grammars are Equivalent to Link Grammar |
| 1022 | \begin_inset Quotes erd |
| 1023 | \end_inset |
| 1024 | |
| 1025 | ; I think search engines can find this. |
| 1026 | \end_layout |
| 1027 | |
| 1028 | \begin_layout Standard |
| 1029 | To conclude: a single disjunct is the same thing as a single–element tensor |
| 1030 | (a basis tensor, i.e. |
| 1031 | a product of basis vectors) and this is the same thing as a jigsaw piece. |
| 1032 | All three of these are the same thing. |
| 1033 | |
| 1034 | \end_layout |
| 1035 | |
| 1036 | \begin_layout Standard |
| 1037 | To each |
| 774 | 1038 | \end_layout |
| 775 | 1039 | |
| 776 | 1040 | \begin_layout Section* |