| … | … | @@ -666,9 +666,9 @@ Training vs. |
| 666 | 666 | \end_layout |
| 667 | 667 | |
| 668 | 668 | \begin_layout Standard |
| 669 | | So, where does that leave us? For now, lets ditch semantics: the original |
| 670 | | problem was to discover a collection of disjuncts that adequately describe |
| 671 | | possible syntactic parses of sentences. |
| 669 | So, where does that leave us? For now, lets ditch semantics: th In particular, |
| 670 | we wante original problem was to discover a collection of disjuncts that |
| 671 | adequately describe possible syntactic parses of sentences. |
| 672 | 672 | The goal was to do this with some sort of expression of aggregate satisfiabilit |
| 673 | 673 | y: some kind of function assigning weights to disjuncts, and maximizing |
| 674 | 674 | it. |
| … | … | @@ -1030,11 +1030,284 @@ To conclude: a single disjunct is the same thing as a single–element tensor |
| 1030 | 1030 | (a basis tensor, i.e. |
| 1031 | 1031 | a product of basis vectors) and this is the same thing as a jigsaw piece. |
| 1032 | 1032 | All three of these are the same thing. |
| 1033 | | |
| 1033 | \end_layout |
| 1034 | |
| 1035 | \begin_layout Section* |
| 1036 | Costs |
| 1037 | \end_layout |
| 1038 | |
| 1039 | \begin_layout Standard |
| 1040 | To each disjunct, we wish to assign a cost: a floating–point, real–valued |
| 1041 | number |
| 1042 | \begin_inset Formula $\alpha_{i}$ |
| 1043 | \end_inset |
| 1044 | |
| 1045 | for each |
| 1046 | \begin_inset Formula $D_{i}$ |
| 1047 | \end_inset |
| 1048 | |
| 1049 | . |
| 1050 | The total cost is then |
| 1051 | \begin_inset Formula |
| 1052 | \[ |
| 1053 | h\left(G\right)=\sum_{1\le i\le n}\alpha_{i} |
| 1054 | \] |
| 1055 | |
| 1056 | \end_inset |
| 1057 | |
| 1058 | Recall that |
| 1059 | \begin_inset Formula $G$ |
| 1060 | \end_inset |
| 1061 | |
| 1062 | is fixed, and that therefore the |
| 1063 | \begin_inset Formula $D_{i}$ |
| 1064 | \end_inset |
| 1065 | |
| 1066 | are fixed: they are uniquely determined by the graph. |
| 1067 | The task is to find an assignment of |
| 1068 | \begin_inset Formula $\alpha_{i}$ |
| 1069 | \end_inset |
| 1070 | |
| 1071 | that minimize the total cost over the entire training corpus. |
| 1072 | \begin_inset Formula |
| 1073 | \[ |
| 1074 | \mathcal{H}=\sum_{S\in\mathcal{C}}H\left(S\right)=\sum_{S\in\mathcal{C}}\sum_{G\in\mathcal{G}}h\left(G\right) |
| 1075 | \] |
| 1076 | |
| 1077 | \end_inset |
| 1078 | |
| 1079 | Of course, this can be met by setting |
| 1080 | \begin_inset Formula $\alpha_{i}=-\infty$ |
| 1081 | \end_inset |
| 1082 | |
| 1083 | which is pointless, and so a bound is needed: let the costs |
| 1084 | \begin_inset Formula $0\le\alpha_{i}$ |
| 1085 | \end_inset |
| 1086 | |
| 1087 | be always positive (or above some lower bound. |
| 1088 | It doesn't matter just right now.) |
| 1089 | \end_layout |
| 1090 | |
| 1091 | \begin_layout Standard |
| 1092 | The above is a general expression. |
| 1093 | Its correct, but awkward, in a way. |
| 1094 | For natural language, we want |
| 1095 | \begin_inset Formula $D_{i}$ |
| 1096 | \end_inset |
| 1097 | |
| 1098 | not to depend on the word–position |
| 1099 | \begin_inset Formula $i$ |
| 1100 | \end_inset |
| 1101 | |
| 1102 | but on the word |
| 1103 | \begin_inset Formula $w_{i}$ |
| 1104 | \end_inset |
| 1105 | |
| 1106 | at that position. |
| 1107 | That is, write not |
| 1108 | \begin_inset Formula $D_{i}$ |
| 1109 | \end_inset |
| 1110 | |
| 1111 | but |
| 1112 | \begin_inset Formula $D\left(w_{i}\right)$ |
| 1113 | \end_inset |
| 1114 | |
| 1115 | and since the disjunct should not depend on the word position, but only |
| 1116 | the word, then |
| 1117 | \begin_inset Formula $D\left(w\right)$ |
| 1118 | \end_inset |
| 1119 | |
| 1120 | should be written for the word |
| 1121 | \begin_inset Formula $w$ |
| 1122 | \end_inset |
| 1123 | |
| 1124 | . |
| 1125 | \end_layout |
| 1126 | |
| 1127 | \begin_layout Standard |
| 1128 | Likewise, for the edge |
| 1129 | \begin_inset Formula $E_{ij}$ |
| 1130 | \end_inset |
| 1131 | |
| 1132 | we don't want a dependency on the word–position |
| 1133 | \begin_inset Formula $j$ |
| 1134 | \end_inset |
| 1135 | |
| 1136 | but instead on the word |
| 1137 | \begin_inset Formula $w_{j}$ |
| 1138 | \end_inset |
| 1139 | |
| 1140 | at location |
| 1141 | \begin_inset Formula $j$ |
| 1142 | \end_inset |
| 1143 | |
| 1144 | . |
| 1145 | Thus, write |
| 1146 | \begin_inset Formula $E_{ij}=E\left(w_{i},w_{j}\right)$ |
| 1147 | \end_inset |
| 1148 | |
| 1149 | or, better yet, since |
| 1150 | \begin_inset Formula $w_{i}$ |
| 1151 | \end_inset |
| 1152 | |
| 1153 | is the same for all edges in the disjunct, simply write |
| 1154 | \begin_inset Formula $D\left(w\right)=u\otimes v\otimes\cdots$ |
| 1155 | \end_inset |
| 1156 | |
| 1157 | where |
| 1158 | \begin_inset Formula $u,v,\cdots$ |
| 1159 | \end_inset |
| 1160 | |
| 1161 | are other words. |
| 1162 | That is, the disjunct no longer depends on the sentence that it is on, |
| 1163 | nor does it depend on which location it has in the sentence: it only depends |
| 1164 | on the word, and it's connectors. |
| 1165 | \end_layout |
| 1166 | |
| 1167 | \begin_layout Standard |
| 1168 | So as not to get lost in abstraction, a practical example is in order. |
| 1169 | Consider the canonical parse of |
| 1170 | \begin_inset Quotes eld |
| 1171 | \end_inset |
| 1172 | |
| 1173 | the dog chased the cat |
| 1174 | \begin_inset Quotes erd |
| 1175 | \end_inset |
| 1176 | |
| 1177 | . |
| 1178 | This has |
| 1179 | \begin_inset Quotes eld |
| 1180 | \end_inset |
| 1181 | |
| 1182 | chased |
| 1183 | \begin_inset Quotes erd |
| 1184 | \end_inset |
| 1185 | |
| 1186 | as a transitive verb, |
| 1187 | \begin_inset Quotes eld |
| 1188 | \end_inset |
| 1189 | |
| 1190 | dog |
| 1191 | \begin_inset Quotes erd |
| 1192 | \end_inset |
| 1193 | |
| 1194 | and |
| 1195 | \begin_inset Quotes eld |
| 1196 | \end_inset |
| 1197 | |
| 1198 | cat |
| 1199 | \begin_inset Quotes erd |
| 1200 | \end_inset |
| 1201 | |
| 1202 | as nouns, and |
| 1203 | \begin_inset Quotes eld |
| 1204 | \end_inset |
| 1205 | |
| 1206 | the |
| 1207 | \begin_inset Quotes erd |
| 1208 | \end_inset |
| 1209 | |
| 1210 | as determiner. |
| 1211 | The disjuncts are: |
| 1212 | \end_layout |
| 1213 | |
| 1214 | \begin_layout Standard |
| 1215 | \begin_inset Box Frameless |
| 1216 | position "t" |
| 1217 | hor_pos "c" |
| 1218 | has_inner_box 1 |
| 1219 | inner_pos "t" |
| 1220 | use_parbox 0 |
| 1221 | use_makebox 0 |
| 1222 | width "100col%" |
| 1223 | special "none" |
| 1224 | height "1in" |
| 1225 | height_special "totalheight" |
| 1226 | thickness "0.4pt" |
| 1227 | separation "3pt" |
| 1228 | shadowsize "4pt" |
| 1229 | framecolor "black" |
| 1230 | backgroundcolor "none" |
| 1231 | status open |
| 1232 | |
| 1233 | \begin_layout Plain Layout |
| 1234 | \begin_inset Formula $D\left(\mbox{CHASED}\right)=\mathtt{DOG-\,\&\,CAT+}$ |
| 1235 | \end_inset |
| 1236 | |
| 1237 | |
| 1238 | \end_layout |
| 1239 | |
| 1240 | \begin_layout Plain Layout |
| 1241 | \begin_inset Formula $D\left(\mbox{DOG}\right)=\mathtt{THE-\,\&\,CHASED+}$ |
| 1242 | \end_inset |
| 1243 | |
| 1244 | |
| 1245 | \end_layout |
| 1246 | |
| 1247 | \begin_layout Plain Layout |
| 1248 | \begin_inset Formula $D\left(\mbox{CAT}\right)=\mathtt{THE-\,\&\,CHASED-}$ |
| 1249 | \end_inset |
| 1250 | |
| 1251 | |
| 1252 | \end_layout |
| 1253 | |
| 1254 | \begin_layout Plain Layout |
| 1255 | \begin_inset Formula $D\left(\mbox{THE}\right)=\mathtt{DOG+}\mbox{ or }\mathtt{CAT+}$ |
| 1256 | \end_inset |
| 1257 | |
| 1258 | |
| 1259 | \end_layout |
| 1260 | |
| 1261 | \end_inset |
| 1262 | |
| 1263 | |
| 1264 | \end_layout |
| 1265 | |
| 1266 | \begin_layout Standard |
| 1267 | Here, the plus and minus indicate a linkage to the left or to the right. |
| 1268 | The transitive verb links to a subject on the left and an object on the |
| 1269 | right. |
| 1270 | English is an SVO language, so we can tell apart the subject noun and the |
| 1271 | object noun from the directional indicators + and -. |
| 1272 | So |
| 1273 | \begin_inset Quotes eld |
| 1274 | \end_inset |
| 1275 | |
| 1276 | cat |
| 1277 | \begin_inset Quotes erd |
| 1278 | \end_inset |
| 1279 | |
| 1280 | is the object, because it has |
| 1281 | \begin_inset Quotes eld |
| 1282 | \end_inset |
| 1283 | |
| 1284 | chased- |
| 1285 | \begin_inset Quotes erd |
| 1286 | \end_inset |
| 1287 | |
| 1288 | on it, and not |
| 1289 | \begin_inset Quotes eld |
| 1290 | \end_inset |
| 1291 | |
| 1292 | chased+ |
| 1293 | \begin_inset Quotes erd |
| 1294 | \end_inset |
| 1295 | |
| 1296 | . |
| 1297 | \end_layout |
| 1298 | |
| 1299 | \begin_layout Standard |
| 1300 | Again, this is super–basic stuff; I'm repeating it here so that we can keep |
| 1301 | our feet on the ground. |
| 1034 | 1302 | \end_layout |
| 1035 | 1303 | |
| 1036 | 1304 | \begin_layout Standard |
| 1037 | | To each |
| 1305 | That's the good news. |
| 1306 | The bad news is that I am laboriously re–inventing the skip–gram. |
| 1307 | To some large degree, if we walk much farther down this path, we risk re–invent |
| 1308 | ing BERT. |
| 1309 | And that is not really the goal. |
| 1310 | Soo.... |
| 1038 | 1311 | \end_layout |
| 1039 | 1312 | |
| 1040 | 1313 | \begin_layout Section* |