═══ MVP Phase 2: NPMI Computation + MST Parsing ═══
Step 1: Opening database and setting up matrix API...
{"phase":2,"step":"open_db","elapsed_s":0}
Database opened.
Matrix API created.
{"phase":2,"step":"matrix_api_ready","elapsed_s":0}
Step 2: Loading word pairs from database...
This may take several minutes for 1M+ pairs.
{"phase":2,"step":"load_pairs_start","elapsed_s":0}
Elapsed time to load pairs: 80 secs
Loaded: 1184936 pairs, 28240 left-words, 29014 right-words
{"phase":2,"step":"load_pairs_done","elapsed_s":2600,"pairs":1184936,"left_words":28240,"right_words":29014}
Step 3: Computing MI/NPMI on all pairs...
This is the main computation - may take 30-60 minutes.
{"phase":2,"step":"compute_mi_start","elapsed_s":2600}
Start computing the basis
Support: found num left= 28240 num right= 29014 in 0 secs
Finished left norm marginals in 348 secs
Finished left totals in 1 secs
Finished right norm marginals in 354 secs
Finished right totals in 0 secs
Done with wild-card count N(x,*) and N(*,y) in 703 secs
Total count N(*,*) = 3328167.0 = 3328167.0
Finished column (left) norm averages in 2 secs
Finished row (right) norm averages in 1 secs
Going to do individual pair frequencies.
Done computing 1184936 pair frequencies in 363 secs
Start computing log P(*,y).
Done computing 29014 left-wild log frequencies in 0 secs.
Done with -log P(*,y), start -log P(x,*)
Done computing 28240 right-wild log frequencies in 0 secs.
Done storing 29014 left-wilds in 3 secs
Done storing 28240 right-wilds in 2 secs
Done saving -log P(x,*) and P(*,y)
Going to compute and store individual pair MI.
Done 10000 of 28240 outer loops in 62 secs, pairs=451350 (7279.8 pairs/sec)
Done 20000 of 28240 outer loops in 60 secs, pairs=862642 (6854.9 pairs/sec)
Done computing 1184936 pair MI's in 165 secs
Going to do column and row subtotals.
Finished left entropy subtotals in 353 secs
Finished right entropy subtotals in 362 secs
Finished left MI subtotals in 358 secs
Finished right MI subtotals in 364 secs
Going to compute the left, right and total entropy.
Done computing totals; start saving wildcards.
Done storing 29014 left-wilds in 2 secs
Done storing 28240 right-wilds in 3 secs
Finished with MI computations; this took 0.744 hours.
MI computation complete.
{"phase":2,"step":"compute_mi_done","elapsed_s":5279}
Matrix summary:
Summary Report for Correlation Matrix Link Grammar ANY link Word Pairs
Left type: WordNode Right Type: WordNode Pair Type: EdgeLink
Wildcard: (Edge
(Bond "ANY")
(List
(Any "left-word")
(Any "right-word")))
Rows: 28240 Columns: 29014 == log_2 14.7855 x 14.8245
Size: 1184936.0 log_2 size: 20.1764
Fraction non-zero: 1.4462E-3 Sparsity: 9.43354 Rarity: 5.37142
Total obs: 3328167.0 Avg obs/pair: 2.80873 log_2 avg: 1.48992
Entropy Tot: 17.8017 Left: 9.87434 Right: 9.70564 MI: 1.77826
No MM^T data present
No M^TM data present
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 3667. 3922.
Count (l_1) 2.9641E+4 3.0585E+4 8.084 7.798
Length (l_2) 2862. 2914. .7804 .7428
RMS Count 2839. 2890. .7742 .7369
Step 4: MST parsing to extract Sections...
Re-parsing corpus with MI-weighted MST parser.
{"phase":2,"step":"mst_parse_start","elapsed_s":5279}
Backtrace:
In ice-9/boot-9.scm:
1752:10 10 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
9 (apply-smob/0 #<thunk 7182f2f15300>)
In ice-9/boot-9.scm:
724:2 8 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
In ice-9/eval.scm:
619:8 7 (_ #(#(#<directory (guile-user) 7182f2f18c80>)))
In ice-9/boot-9.scm:
2836:4 6 (save-module-excursion _)
4388:12 5 (_)
In /storage/projects/code/hypercore-kb/opencog-kb/language-learning/mvp/run-phase2.scm:
218:4 4 (run-phase2)
In srfi/srfi-1.scm:
634:9 3 (for-each #<procedure 71825503c600 at /storage/project…> …)
650:11 2 (for-each #<procedure 7182f2ead340 at opencog/learn/sl…> …)
In opencog/learn/sliding-block.scm:
119:40 1 (_ _ _)
In opencog/learn/pipe-count.scm:
236:27 0 (disjunct-obs-text _)
opencog/learn/pipe-count.scm:236:27: In procedure disjunct-obs-text:
Unbound variable: ParseNode
Rocks: opened=data/pairs.rdb
Rocks: DB-version=2 multi-space=0 initial aid=2513684
Loading corpus...
alice-in-wonderland.txt: 147988 chars
great-expectations.txt: 1015024 chars
moby-dick.txt: 1240883 chars
pride-and-prejudice.txt: 743287 chars
sherlock-holmes.txt: 574145 chars
Loaded 5 text blocks.
Processing block 1/5 (147988 chars)...