Working Notes (Archived; Part A)
--------------------------------
This file contains a diary of random working notes, which I use to keep
track of what the heck it is that I'm doing. It is almost surely totally
useless to you, except maybe for some weird voyeuristic reasons.
Archived in 2021; this covers the time period from the start of the
project to Dec 2019. There was no activity on this project in 2020.
======================================================================
Jan-Feb 2014
Handy tools
-----------
Some handy SQL commands:
```
SELECT count(uuid) FROM Atoms;
select count(uuid) from atoms where type =123;
```
type 123 is `WordNode` for me; verify with
```
SELECT * FROM Typecodes;
```
The total count accumulated is
```
select sum(floatvalue[3]) from valuations where type=7;
```
where type 7 is `CountTruthValue`.
Pair-counting batch results
---------------------------
Example stats and performance:
current fr_pairs db has 16785 words and 177960 pairs.
This takes 17K + 2x 178K = 370K total atoms loaded.
These load up in 10-20 seconds-ish or so.
New fr_pairs has 225K words, 5M pairs (10.3M atoms):
Load 10.3M atoms, which takes about 10 minutes cpu time to load
20-30 minutes wall-clock time (500K atoms per minute, 9K/second
on an overloaded server).
RSS for cogserver: 436MB, holding approx 370K atoms
So this is about 1.2KB per atom, all included. Atoms are a bit fat...
... loading all pairs is very manageable even for modest-sized machines.
RSS for cogserver: 10GB, holding 10.3M atoms
So this is just under 1KB per atom.
(By comparison, direct measurement of atom size i.e. class Atom:
typical atom size: 4820384 / 35444 = 136 Bytes/atom
this is NOT counting indexes, etc.)
For dataset (fr_pairs) with 225K words, 5M pairs:
Current rate is 150 words/sec or 9K words/min.
After the single-word counts complete, and all-pair count is done.
This is fast, takes a couple of minutes.
Next: batch-logli takes 540 seconds for 225K words
Finally, an MI compute stage.
Current rate is 60 words/sec = 3.6K per minute.
This rate is per-word, not per word-pair .
Update Feb 2014: fr_pairs now contains 10.3M atoms
SELECT count(uuid) FROM Atoms; gives 10324863 (10.3M atoms)
select count(uuid) from atoms where type = 77; gives 226030 (226K words)
select count(uuid) from atoms where type = 8; gives 5050835 (5M pairs ListLink)
select count(uuid) from atoms where type = 27; gives 5050847 (5M pairs EvaluationLink)
Performance
-----------
Performance seems to suck:
-- two parsers, each takes maybe 4% cpu time total. Load avg of about 0.03
-- each parser runs 4 async write threads pushing atoms to postgres.
each one complains about it taking too long to flush the write queues.
-- postmaster is running 10 threads, load-avg of about 2.00 so about
2 cpu's at 100%
-- vmstat shows 500 blks per second written. This is low...
-- top shows maybe 0.2% wait state. So its not disk-bound.
-- what is taking so long?
So, take a tcpdump:
-- a typical tcpdump packet:
UPDATE Atoms SET tv_type = 2, stv_mean = 0 , stv_confidence = 0, stv_count = 54036 WHERE uuid = 367785;
its maybe 226 bytes long.
-- this gets one response from server, about 96 bytes long.
-- then one more req, one more repsonse, seems to be a 'were'done' mesg
or something ... which I guess is due to SQLFreeHandle(SQL_HANDLE_STMT ???
-- time delta in seconds, of tcpdump of traffic packets, between update, and
response from server:
0.0006 0.0002 0.0002 0.0002 0.028 (yow!!) 0.001 0.0002
-- so it looks like about every 8-10 packets are replied to fairly quick,
then there's one that takes 0.025 seconds to reply.... stair-steps in
response time like this all the way through the capture.
Wild guess:
-- Hmm ... this seems to be related to the commit delay in postgresql.conf
Change commit_delay to 1 second
change wal_bufers to 32MB since its mostly update traffic.
change checkpoint_segments to 32 (each one takes up 16MB of disk space.)
-- Making these changes has no obvious effect ... bummer.
I don't get it; performance sucks and I don't see why. Or rather: postmaster
is chewing up vast amounts of cpu time for no apparent reason...
select * from pg_stat_user_tables;
select * from pg_stat_all_tables;
select * from pg_statio_user_tables;
select * from pg_database;
pg_stat_user_indexes
pg_stat_all_indexes
select * from pg_catalog.pg_stat_activity;
select * from pg_catalog.pg_locks;
-- WOW!!! VACUUM ANALYZE; had a huge effect!!
-- vacuum tells em to do following:
change max_fsm_pages to 600K
chage max_fsm_relations to 10K
Anyway ... performance measured as of 27 Dec 2013:
Takes about 105 millisecs to clear 90 eval-links from the write-back
queues. This each eval-link is 5 atoms (eval, defind, list, word, word)
so this works out to 5*90 atoms /0.105 seconds = 4.3KAtoms/sec
which is still pretty pathetic...
gdb:
---
handle SIGPWR nostop noprint
handle SIGXCPU nostop noprint
How about using a reader-writer lock?
----------------------------------
boost::shared_lock for reading,
unique_lock for writing ...
upgrade_lock<shared_mutex> lock(workerAccess);
upgrade_to_unique_lock<shared_mutex> uniqueLock(lock);
shared_mutex
write uses: unique_lock<shared_mutex>
readers use shared_lock<shared_mutex>
writer does:
// get upgradable access
boost::upgrade_lock<boost::shared_mutex> lock(_access);
// get exclusive access
boost::upgrade_to_unique_lock<boost::shared_mutex> uniqueLock(lock);
// now we have exclusive access
}
am using boost-1.49 on cray
Some typical entropies for word-pairs
-------------------------------------
Three experiments:
1) Get H(de,*) H(*,de) H(en,*) H(*,en) and compare to
H(de+en,*) H(*, de+en)
2) H(vieux,*) H(* vieux) H(nouveaux, *) H(*, nouveaux)
3) H(vieille, *) etc + vieux
4) H(le, *) H(la,*) vs. H(le+la)
5) H(le,*) H(famille,*) which should fail ...!?
Some typical entropies for word-pairs
-------------------------------------
The below is arithmeticaly correct, but theoretically garbage.
(WordNode "famille") entropy H=11.185
H(*, famille) = 11.195548
H(famille, *) = 11.174561
MI(et, famille) = -5.2777815
H(et, *) = 5.5696678
P(et, *) = 0.021055372363972875
thus:
H(et, famille) = -MI(et, famille) + H(famille, *) + H(et, *) = 22.0220103
P(et, famille) = 2.348087815164205e-7
MI(de, famille) = 2.1422486
H(de, *) = 4.3749881
P(de, *) = 0.04819448582223504
H(de, famille) = -2.1422486 + 4.3749881 + 11.195548 = 13.4282875
P(de, famille) = 9.071574511509601e-5
P(de+et. *) = 0.06924985818620791
H(de+et, *) = 3.8520450730427047
P(de+et, famille) = 9.095055389661243e-5
H(de+et, famille) = 13.424558050397735
MI(de+et, famille) = 1.6230350226449701
So MI(et, famille) < MI(de+et, famille) < MI(de, famille)
-5.2777815 < 1.6230350226 < 2.1422486
By contrast, the arithmetic average is:
(MI(de, famille) * P(de, famille) + MI(et, famille) * P(et, famille)) /
(P(de, famille) + P(et, famille))
= 2.1230921666199825
Change in entropy:
MI(de, famille) * P(de, famille) + MI(et, famille) * P(et, famille) = 0.0012169
MI(de+et, famille) * P(de+et, famille) = 1.476159343e-4
Oh, wait ...
H(de, famille) * P(de, famille) + H(et, famille) * P(et, famille) = 0.001223328
H(de+et, famille) * P(de+et, famille) = 0.00122097099
Change in entropy = 0.00122097099 - 0.001223328 = -2.35701e-6
-------
H(de) = 4.3808608
H(et) = 5.5862331
P(de) = 0.04799870191172842
P(et) = 0.02081499323761464
P(de+et) = 0.06881369514934306
H(de+et) = 3.8611604742976153 = -log_2 (P(de)+P(et))
By contrast, the weighted average is
(P(de)*H(de) + P(et)*H(et)) /(P(de) + P(et)) = 4.745465784790553
Combinations:
P(de+et)*H(de+et) = 0.2657007
P(de)*H(de) + P(et)*H(et) = 0.32655303
The change in entropy, from forming a union, is:
P(de+et)*H(de+et) - P(de)*H(de) - P(et)*H(et) = -0.060852316
Recap: Delta(de+et) = -0.060852316
Delta(de+et, famille) = -2.35701e-6
Entropy increases (strongly) if word-pair merged, words are separated,
-------
MI(d'une, famille) = 5.230504
H(d'une, *) = 9.792551
H(la) = 5.6536283
H(la, *) = 5.5858526
sa
est
de
H(d'une) = 9.7960119
H(un) = 7.1578913
H(et) = 5.5862331
-----
repeat, for vielle+nouveaux
H(nouveaux) = 14.28815
P(nouveaux) = 4.998483553100357e-5
H(vieille) = 16.16037
P(vieille) = 1.365349e-5
P(nouveaux+vieille) = 6.363833e-5
H(nouveaux+vieille) = 13.93974
P(nouveaux+vieille)*H(nouveaux+vieille) = 8.87102088-4
P(nouveaux)*H(nouveaux) + P(vieille)*H(vieille) = 9.3483638e-4
Change in entropy is diff of the two: -4.7734297e-5
-----
repeat, for vielle+nouveaux
H(*, famille) = 11.195548
H(nouveaux, *) = 13.974219
P(nouveaux, *) = 6.213565989765264e-5
MI(nouveaux, famille) = 5.2966957
H(nouveaux, famille) = 19.8730713
P(nouveaux, famille) = 1.0413804797188067e-6
H(vieille, *) = 15.998603
P(vieille, *) = 1.5273571710064995e-5
MI(vieille, famille) = 10.195547
H(vieille, famille) = 16.998604
P(vieille, famille) = 7.636780561617735e-6
P(vieille+nouveaux, famille) = 8.678161041336542e-6
H(vieille+nouveaux, famille) = 16.814179210712517
P(vieille+nouveaux, *) = 7.740923160771763e-5
H(vieille+nouveaux, *) = 13.657134846045357
MI(vieille+nouveaux, famille) = 8.038503635332841
so MI(nouveaux, famille) < MI(vieille+nouveaux, famille) < MI(vieille, famille)
5.2966957 < 8.038503635332841 < 10.195547
Change in entropy:
P(nouveaux, famille)*H(nouveaux, famille) + P(vieille, famille)*H(vieille, famille)
= 1.505100371e-4
P(vieille+nouveaux, famille) * H(vieille+nouveaux, famille) = 1.459161549e-4
Change = 1.459161549e-4 - 1.505100371e-4 = -4.5938821e-6
To recap: Delta(vieille+nouveaux) = -4.7734297e-5
reduces the entropy more than
Delta(vieille+nouveaux, famille) = -4.5938821e-6
i.e. entropy increses if the word-pairs are merged, the words are separated.
======================================================================
Minimal morphology output
;; Lets say that there was one word in the sentence, it was 'foobar'
;; and the splitter split it into foo and bar
;; then the following should be generated:
;; for each sentence, create one of these, each with a distinct uuid:
(ParseLink (stv 1 1)
(ParseNode "sentence@fc98a97a-4753-45d9-be5b-1c752b5b21d9_parse_0")
(SentenceNode "sentence@fc98a97a-4753-45d9-be5b-1c752b5b21d9")
)
;; For each pair of morphemes, cereate the below:
(EvaluationLink (stv 1.0 1.0)
(LinkGrammarRelationshipNode "MOR")
(ListLink
(WordInstanceNode "foo@5e179119-3966-4bb9-8a38-ef2014b48f12")
(WordInstanceNode "bar@cb2443bb-fbec-472c-baee-36b822579861")
)
)
;; For each "word" aka morpheme, create these two clauses:
;; note that the UUID's match up exactly with the above.
;; the below shows only "foo", another pair is needed for "bar".
(ReferenceLink (stv 1.0 1.0)
(WordInstanceNode "foo@5e179119-3966-4bb9-8a38-ef2014b48f12")
(WordNode "foo")
)
(WordInstanceLink (stv 1.0 1.0)
(WordInstanceNode "foo@5e179119-3966-4bb9-8a38-ef2014b48f12")
(ParseNode "sentence@fc98a97a-4753-45d9-be5b-1c752b5b21d9_parse_0")
)
;; finally, at the very end:
;; again, the UUID must match with what was given above.
(ListLink (stv 1 1)
(AnchorNode "# New Parsed Sentence")
(SentenceNode "sentence@68e51cae-98bc-4102-b19c-78649c5f6cfb")
)
======================================================================
Tagalog status:
31 july 2014
4519631 = 4.5M morpehem pairs
204K morpehemes
======================================================================
Setup, July 2015
----------------
LXC container on gnucash.org
AtomSpace.cc line 303
LXC container on backlot
------------------------
nlp-base and nlp-server (currently used by rohit)
morf-server (currrently used by ainish)
LXC container on fanny
----------------------
ssh learn@10.0.3.182
cd src/learn
./run-all-servers.sh
tmux attach
psql en_pairs
======================================================================
======================================================================
31 Dec 2016
-----------
psql en_pairs
\dt
select count(*) from atoms;
18487291
loadmodule libPersistModule.so
sql-open learn-pairs linas asdf
sql-open en-pairs linas asdf
password authentication failed for user "linas"
sql-open opencog_test opencog_tester cheese
/etc/postgresql/9.6/main/pg_hba.conf looks OK...
So: I have en_pairs and
opencog_test | linas | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
but the owner of the tables is opencog_tester
/var/log/postgresql/postgresql-9.6-main.log
2017-01-03 16:15:50 CST [2856-1] linas@en_pairs FATAL: password authentication
failed for user "linas"
2017-01-03 16:15:50 CST [2856-2] linas@en_pairs DETAIL: User "linas" has no password
assigned.
2017-01-01 17:20:54 CST [4829-1] opencog_tester@opencog_test ERROR: insert or update
on table "atoms" violates foreign key constraint "atoms_space_fkey"
2017-01-01 17:20:54 CST [4829-2] opencog_tester@opencog_test DETAIL: Key (space)=(2)
is not present in table "spaces".
(use-modules (opencog persist-sql))
(sql-open "en-pairs" "learner" "asdf")
\du
alter user learner password 'asdf';
grant CONNECT ON DATABASE en_pairs to learner;
grant SELECT,INSERT,UPDATE on table atoms to learner;
sql-open en-pairs learner asdf
it worked!
(sql-load)
18 million atoms
Loaded 9045489 atoms at height 2
Finished loading 18487291 atoms in total
12:35 to load .. !? 18487291 atoms/755 secs = 24.5K atoms/sec
psql -h localhost -U ubuntu lt_pairs
ALTER USER ubuntu PASSWORD 'asdf';
========================================================
-----------------------------------------
lxc -- create an all-updated opencog-base
lxc-start -n opencog-base --daemon
time lxc-copy -n opencog-learn -N learn-lt
------------------------------------
4 Jan 2017
----------
https://dumps.wikimedia.org/zhwiki/20170101/
https://dumps.wikimedia.org/zh_yuewiki/20170101/
https://dumps.wikimedia.org/frwiki/20170101/
lynx https://dumps.wikimedia.org/ltwiki/20170101/ltwiki-20170101-pages-articles-multistream.xml.bz2
time cat ltwiki-20170101-pages-articles-multistream.xml.bz2 |bunzip2 |/home/ubuntu/src/relex/src/perl/wiki-scrub.pl
real 4m58.871s
user 5m35.652s
sys 0m11.700s
find |wc gives 209011 total articles
find |wc gives 178514 after cat/template removal
createdb lt_pairs
createdb lt_morph
cat opencog/persist/sql/odbc/atom.sql | psql lt_pairs
cat opencog/persist/sql/odbc/atom.sql | psql lt_morph
=============================================================
time cat zh_yuewiki-20170101-pages-articles.xml.bz2 |bunzip2 |/home/ubuntu/src/relex/src/perl/wiki-scrub.pl
about 48 seconds
find |wc gives 67363 total articles
find |wc gives 49170 after cat/template removal
apt-get install fonts-arphic-ukai fonts-arphic-uming fonts-babelstone-han
fonts-wqy-zenhei fonts-hanazono
fonts-arphic-bkai00mp
fonts-arphic-bsmi00lp
fonts-arphic-gbsn00lp
fonts-arphic-gkai00mp
Arghh. None of the above provide the Kangxi radicals for the terminal.
Which I think are coming from fonts-wqy-microhei
U+2F13 Kangxi Radicals,
U+42AA U+4401 CJK_Ext_A CJK-Ext.A
createdb yue_pairs
cd ~/src/atomspace
cat opencog/persist/sql/odbc/atom.sql | psql yue_pairs
。。
\p{Block: CJK}
\p{Block=CJK_Symbols_And_Punctuation}
\p{Punct}
\p{InCJK})
\p{Close_Punctuation} aka \p{Pe} (close parent)
\p{Final_Punctuation} aka \p{Pf}) more quote-close or open.. things
\p{Ps} open quote
full stop
relex-server-port relex-server-host
; -- count-all -- Return the total number of atoms in the atomspace.
; -- cog-get-atoms -- Return a list of all atoms of type 'atom-type'
; -- cog-prt-atomspace -- Prints all atoms in the atomspace
; -- cog-count-atoms -- Count of the number of atoms of given type.
; -- cog-report-counts -- Return an association list of counts.
wtf
(define (foo atom) (display "duude\n")(display atom) (newline) #f)
WARNING: No known abbreviations for language 'yue', attempting fall-back
to English version.. FIXED
odbc is still logging! FIXED
CommLog = No in /etc/odbcinst.ini
don't use "foo", it prints a warning .. better yet, don't warn! FIXED
below is due to bug opencog/relex#248 and is now fixed.
It needed a new link-grammar version
link-grammar: Error: EMPTY-WORD.zzz must be defined!
link-grammar: Error: Word 'EMPTY-WORDzzz': Internal error: NULL X_node
link-grammar: Error: sentence_split(): Internal error detected
Warning: No parses found for:
港 區 全 國 人 大 代 表 係 代 表 香 港 居 民 響 中 華 人 民 共 和 國 全 國 人 民 代 表 大 會 行 使 國 家 立 法 權 嘅 代 表 ， 名 額 36 人 （1997 年 香 港 主 權 移 交 之 後 ）。
link-grammar: Error: EMPTY-WORD.zzz must be defined!
link-grammar: Error: Word 'EMPTY-WORDzzz': Internal error: NULL X_node
link-grammar: Error: sentence_split(): Internal error detected
Warning: No parses found for:
深 圳 習 慣 叫 特 區 範 圍 做 「 關 內 」， 而 特 區 範 圍 之 外 嘅 ，
包 括 寶 安 區 、 龍 崗 區 同 光 明 新 區 、 坪 山 新 區 就 叫 「 關 外
」； 由 「 關 外 」 入 特 區 叫 「 入 關 」， 反 之 係 「 出 關 」。
<title>永利街</title> contains junk yest it does...
Started 5 Jan 2017 16:00 exactly.
ten minutes later: 5048 atoms -- so 500 atoms per minute...
after some halts and hiccups:
2211 articles after 1 hour = 37 articles/minute
18529 atoms after about 1 hour ...
or about 8.4 atoms per article...
There are only about 48K articles, so it should conclude in 24 hours
...!?
hours later... 219127 atoms 6345 articles done ...
Now its about 34.5 atoms per article.. whoa ...
java claims to have parsed 12424 sentences
11459 articles processed.
457009 atoms
29291 articles processed 19919 remaining
668042 atoms ...
; -- cog-report-counts -- Return an association list of counts.
(count-all)
(cog-report-counts)
(gc-stats)
... ram usage slowly increasing...
(gc-stats)
((gc-time-taken . 315428672862) (heap-size . 3479842816) (heap-free-size
. 1406132224) (heap-total-allocated . 132540491040)
(heap-allocated-since-gc . 906048240) (protected-objects . 500)
(gc-times . 414))
(gc-stats)
((gc-time-taken . 327615529234) (heap-size . 3582787584) (heap-free-size
. 1584795648) (heap-total-allocated . 138942040624)
(heap-allocated-since-gc . 1123554048) (protected-objects . 500)
(gc-times . 422))
((gc-time-taken . 491224665617) (heap-size . 4601581568) (heap-free-size
. 2257489920) (heap-total-allocated . 211782122608)
(heap-allocated-since-gc . 399939472) (protected-objects . 500)
(gc-times . 485))
if (number-of-cells-collected-recently < GUILE_MIN_YIELD_X)
then
allocate-new-heap
else
run-a-collection
`scm_i_gc_grow_heap_p ()' and `scm_gc_for_newcell ()'.)
(WordSequenceLink lots of these ...
gcprof procedure in the statprof library
https://www.gnu.org/software/guile/manual/html_node/Statprof.html
guile-yue> (statprof-display)
% cumulative self self total
time seconds seconds calls ms/call ms/call name
49.18 11506.88 11506.88 13516 851.35 851.35 inc
4.92 23397.33 1150.69 27 42618.08 866567.64 catch
4.92 12657.57 1150.69 215 5352.04 58872.42 cog-map-type
3.28 767.13 767.13 1420 540.23 540.23 char=?
3.28 767.13 767.13 619 1239.30 1239.30 write-char
3.28 767.13 767.13 372 2062.17 2062.17 memq
3.28 767.13 767.13 182 4214.97 4214.97 call-with-output-string
1.64 2684.94 383.56 182 2107.49 14752.41 tilde-dispatch
1.64 383.56 383.56 29 13226.30 13226.30 close-port
1.64 383.56 383.56 240 1598.18 1598.18 assv-ref
...
0.00 12657.57 0.00 215 0.00 58872.42 cog-count-atoms
above over about 24K seconds total, so accurate... ish
(use-modules (statprof))
(statprof-reset 0 50000 #t) ;
(statprof-start)
(do-something)
(statprof-stop)
(statprof-display)
(gcprof (λ () (observe-text "1769 年 ： 伊 萬 克 雷 洛 夫 ， 俄 國 寓 言 作 家 1910 年 ： 威 廉 肖 克 利 （William Shockley）， 美 國 物 理 學 家 ， 有 份 發 明 半 導 體 ，1956 年 諾 貝 爾 物 理 獎 得 主 1915 年 ： 昂 山 ， 緬 甸 國 父 1921 年 ： 趙 無 極 ， 法 國 華 裔 畫 家 1974 年 ：Robbie Williams， 英 國 歌 手 1974 年 ： 馬 國 明 ， 香 港 無 綫 電 視 演 員 1981 年 ： 何 紫 綸 ， 香 港 模 特 兒 1990 年 ： 西 藏 第 十 一 世 班 禪 額 爾 德 尼 金 瑞 瑤 ， 台 灣 音 樂 經 理 人 1993 年 ： 宋 希 濂 ， 抗 日 戰 爭 同 國 共 內 戰 時 期 中 國 國 民 黨 將 軍 2006 年 ： 王 選 ， 中 國 計 算 機 學 者 ， 發 明 漢 字 激 光 照 排 技 術")))
Maybe use it in "observe-text"? ...
total time is correct...
... its a thread thing. staprof with threads is borked.
See comments in ./module/statprof.scm ~ Implementation notes ~
compute-mi.scm: (for-each inc atom-list)
compute-mi.scm: (define (inc atom) (set! cnt (+ cnt (tv-count
(cog-tv atom)))))
Maybe lots and lots of threads ... ? Seems to get very backed-up.
No .. only 14 threads
(hash-map->list cons (module-obarray (current-module)))
(module-map (λ (sym var) sym) (resolve-interface '(guile)))
(module-map (λ (sym var) sym) (resolve-interface '(opencog)))
(module-map (λ (sym var) sym) (resolve-interface '(opencog learn)))
who is using a module?
(module-uses (resolve-module '(guile-user)))
>>>> excellent for modules!
http://git.net/ml/guile-user-gnu/2016-06/msg00040.html
101000 26627 256 11.9 15129736 11795708 pts/6 Sl+ 18:41 516:33 guile
-l pair-count-yue.scm
not being split: FIXED.
fix is
$text =~ s/([\.?!]) *(\p{InCJK})/$1\n$2/g;
呢度啲路順序係從南去到北嚟排列嘅，其中加粗咗嘅字係主幹道：美華北路.新河浦二橫路.新河浦五橫路.新慶路.煙墩路、寺右新馬路.寺貝通津.共和大街.松崗東.共和西路.中山一路.
2012年，《向前走向愛走》.第四十五屆金鐘獎個人獎戲劇節目女主角獎.郭采潔官方網站.
No database persistant storage configured! Use the STORAGE config
keyword to define.
Java gets slower and slower
=========================
replace call to scm_gc_register_collectable_memory by call to
scm_gc_register_allocation(size)
whoa ---
GC Warning: Repeated allocation of very large block (appr. size 27369472):
May lead to memory leak and poor performance.
GC Warning: Repeated allocation of very large block (appr. size 28766208):
May lead to memory leak and poor performance.
Loaded 280000 atoms.
GC Warning: Repeated allocation of very large block (appr. size 28766208):
May lead to memory leak and poor performance.
GC Warning: Repeated allocation of very large block (appr. size 28766208):
May lead to memory leak and poor performance.
Loaded 270000 atoms.
GC Warning: Repeated allocation of very large block (appr. size 28766208):
May lead to memory leak and poor performance.
GC Warning: Repeated allocation of very large block (appr. size 14385152):
May lead to memory leak and poor performance.
Loaded 260000 atoms.
================================
fresh, guile-2.0
(gc-stats)
$6 = ((gc-time-taken . 114568428) (heap-size . 14409728) (heap-free-size .
2711552) (heap-total-allocated . 18881904) (heap-allocated-since-gc .
1054528) (protected-objects . 137) (gc-times . 14))
after half-minute:
(gc-stats)
((gc-time-taken . 7534031063) (heap-size . 19734528) (heap-free-size
. 5259264) (heap-total-allocated . 939889168) (heap-allocated-since-gc .
1657120) (protected-objects . 143) (gc-times . 326))
guile> foo
(ConceptNode "foo" (ctv 0 0 2520410))
6861 101000 20 0 735404 42952 16680 R 109.9 0.0 14:33.40 guile
6861 101000 20 0 735404 42952 16680 R 114.5 0.0 44:35.68 guile
(gc-stats)
((gc-time-taken . 148069551756) (heap-size . 19734528) (heap-free-size .
2740224) (heap-total-allocated . 28219212944) (heap-allocated-since-gc .
790384) (protected-objects . 143) (gc-times . 4930))
guile> foo
(ConceptNode "foo" (ctv 0 0 54310643))
replace call to scm_gc_register_collectable_memory by call to
scm_gc_register_allocation(size)
static std::atomic<size_t> _tv_pend_cnt;
static std::atomic<size_t> _tv_total_cnt;
static std::atomic<size_t> _tv_pend_sz;
static std::atomic<size_t> _tv_total_sz;
(define (inc atom) (cog-set-tv! atom (cog-new-ctv 0 0 (+ 1 (tv-count (cog-tv
atom))))))
scheme@(guile-user)>
scheme@(guile-user)> (define foo (Concept "foo"))
scheme@(guile-user)> (define (loo) (inc foo) (loo))
scheme@(guile-user)> (loo)
duuude its pend cnt=11425 (274200) tot=1400000 (33600000)
duuude its pend cnt=14379 (345096) tot=1500000 (36000000)
duuude its pend cnt=15600 (374400) tot=2200000 (52800000)
duuude its pend cnt=8047 (193128) tot=48300000 (1159200000)
duuude its pend cnt=1230 (29520) tot=49300000 (1183200000)
duuude its pend cnt=12432 (298368) tot=49400000 (1185600000)
duuude its pend cnt=19857 (476568) tot=50000000 (1200000000)
(gc-stats)
((gc-time-taken . 54370538078) (heap-size . 10948608) (heap-free-size . 1925120)
(heap-total-allocated . 8532174704) (heap-allocated-since-gc . 10336)
(protected-objects . 7) (gc-times . 2051))
guile-yue> foo
(ConceptNode "foo" (ctv 0 0 26737610))
4617 linas 20 0 847596 50284 27072 R 133.2 0.1 5:26.89 guile
(ConceptNode "foo" (ctv 0 0 46416463))
duuude its pend cnt=20565 (493560) tot=94300000 (2263200000)
so -- 46M incrs but 94M take-tvs -- so two takes for each incr.
-- one to get the value, one to set the value.
4617 linas 20 0 847584 50612 27156 R 135.2 0.1 23:46.94 guile
(define (rate)
(define shu (Concept "shu"))
(define cnt 0)
(define start (- (current-time) 0.1))
(define (finc atom)
(if (eq? 0 (modulo cnt 100000))
(begin (display "rate=")
(display (/ cnt (- (current-time) start))) (newline)))
(set! cnt (+ cnt 1))
(cog-set-tv! atom (cog-new-ctv 0 0 (+ 1 (tv-count (cog-tv atom))))))
(define (floo) (finc shu) (floo))
(floo)
)
(define (inc atom) (cog-set-tv! atom (cog-new-ctv 0 0 (+ 1 (tv-count (cog-tv atom))))))
(define foo (Concept "foo"))
(define (loo) (inc foo) (loo))
(statprof-stop)
(statprof-display)
with the atomics: rate == about 145.5K/sec
without the atomics: about 103.2K/sec !!
again with atomics: rate == 125K/sec !! wtf .. why not as high as before?
stop restart, rate=130K ...
stop restart - rate= 107K ... wtf
stop, restart = 109K
stop restart = 107K glargle
again --- without atomics:
rate = 147K dafuq
stop restart = 151K
stop restart = 79K crazy shit
stop restart = 141K this is so not making sense, except as a
crazy cache-line issue.
clean start: without atomics
(gc-stats)
((gc-time-taken . 845338507) (heap-size . 5963776) (heap-free-size . 421888)
(heap-total-allocated . 73153424) (heap-allocated-since-gc . 902704)
(protected-objects . 7) (gc-times . 65))
(gc-stats)
((gc-time-taken . 605534396784) (heap-size . 6025216) (heap-free-size . 356352)
(heap-total-allocated . 29673578224) (heap-allocated-since-gc . 361904)
(protected-objects . 7) (gc-times . 32428))
no growth at all.
OK, so ... a leak in sql?
a leak in TLB!! ... no because that doesn't explain guile heap...
unless guile heap is confused...
on startup:
(gc-stats)
$1 = ((gc-time-taken . 170774772) (heap-size . 15364096) (heap-free-size .
3166208) (heap-total-allocated . 18421440) (heap-allocated-since-gc .
770768) (protected-objects . 149) (gc-times . 15))
wtf .. why no printing?
_tv_pend_cnt++;
_tv_pend_sz += sizeof(*tv);
// _tv_total_cnt++;
_tv_total_sz += sizeof(*tv);
if (0 == ((size_t) (_tv_total_cnt.fetch_add(1))) % 100000) {
printf("duuude its pend cnt=%lu (%lu) tot=%lu (%lu)\n",
(size_t) _tv_pend_cnt, (size_t) _tv_pend_sz, (size_t) _tv_total_cnt,
(size_t) _tv_total_sz);
logger().info("duuude its pend cnt=%lu (%lu) tot=%lu (%lu)",
(size_t) _tv_pend_cnt, (size_t) _tv_pend_sz, (size_t) _tv_total_cnt,
(size_t) _tv_total_sz);
}
OK, so its not the TV ... (not the TV in guile gc)
So maybe its prim environ?? Nooo not that either
Maybe handles?? (in guile) no its not that. (not in guile gc)
well, its not the TLB...
and not the atoms ... TLB has 400K entries, with 18MB of pairs
atoms allocated are 466286 for 63414896 = 63MBytes but guile is
1.6GB resident, 9.3GB virt.... wtf...
each atom is 136 MB ex tv.
2.9 gb resident, but 1.6M atoms for 215MB size, and 64MB of tlb contents
what about atomspace? only 19922 atoms in atomspace...
heap size is 2GB ...
Maybe stub out capture-stack? it was the cuplrit before...
Nope seems to make no difference.
are we leaking SCM values somwhere? How?
misc_to_string ? no, code audit.
scm_to_utf8_string no, code audit...
--------------------------------------------------------------
try guile-2.2 from git
Great. that seg-faults... maybe some other version doesn't ...
try 2.1.5 ? 2.1.4 ? No, because even though it segfaulted
it did seem to also grow.
Seg-faults twice in a row, within 10 minutes wall-clock time
(about 36 mins cpu time).
---------------------------------------------------------------
Try below. ... It does not leak.
(use-modules (opencog) (opencog cogserver))
(start-cogserver)
(define (slu)
(define cnt 0)
(define start (- (current-time) 0.1))
(define (mka)
(if (eq? 0 (modulo cnt 100000))
(begin (display "rate=")
(display (/ cnt (- (current-time) start))) (newline)
(cog-map-type (lambda (ato) (cog-extract ato) #f) 'ListLink)
(cog-map-type (lambda (ato) (cog-extract ato) #f) 'ConceptNode)
))
(set! cnt (+ cnt 1))
(ListLink
(ConceptNode (string-append "concepto " (number->string cnt )))
(ConceptNode (string-append "glorg " (number->string cnt )))))
(define (aloo) (mka) (aloo))
(aloo)
)
(count-all)
(cog-report-counts)
(gc-stats)
((gc-time-taken . 6660684289) (heap-size . 15646720) (heap-free-size . 3055616)
(heap-total-allocated . 861940880) (heap-allocated-since-gc . 5203440)
(protected-objects . 7) (gc-times . 298))
((gc-time-taken . 15414762477) (heap-size . 16101376) (heap-free-size . 2859008)
(heap-total-allocated . 2665977008) (heap-allocated-since-gc . 5064624)
(protected-objects . 7) (gc-times . 562))
rate=47.3K (concept only)
rate=15.7K (listlinks+concepts)
---------------------------------------------------------------
/tmp/bang.sh
#!/bin/bash
i=0
while true ; do
let i=$i+1
if [ "$(($i % 2000))" -eq "0" ] ; then
echo loop $i
fi
echo '(display ctr)' | nc localhost 17001
# echo '(NumberNode ctr)' | nc 10.0.3.239 17001
# echo '(NumberNode' $i ')' | nc 10.0.3.239 17001
# echo '(NumberNode 42)' | nc localhost 17001
echo '(ConceptNode "fooo ' $i $$ ' you too")' | nc localhost 17001 >> /dev/null
done
run 10 copies of above.
--- no leak ... and no crash... so this is very stable. wtf.
---------------------------------------------------------------
OK, so lets try the full pipeline.
but without updates
Whoops. Its blowing up
((gc-time-taken . 8772615322) (heap-size . 820801536) (heap-free-size . 90624000)
(heap-total-allocated . 6508583904) (heap-allocated-since-gc . 84464688)
(protected-objects . 313) (gc-times . 210))
((gc-time-taken . 8772615322) (heap-size . 1179750400) (heap-free-size . 27713536)
(heap-total-allocated . 16069320928) (heap-allocated-since-gc . 339494096)
(protected-objects . 313) (gc-times . 255))
what if we just do one article over and over?
./ss-one.sh yue beta-pages/A-Z/Zyu4 localhost 17006
It blows up.
what if we do one sentence over and over?
(observe-text "係 拉 丁 字 母 同 阿 剌 伯 數 字 串 字")
run it once, goes from (heap-size . 5963776) to (heap-size . 20996096)
run 10 times: (heap-size . 28332032)
It still blows up....
without bang.sh
(define (lo)
(define cnt 0)
(define start (- (current-time) 0.1))
(define (mke)
(if (eq? 0 (modulo cnt 20))
(begin (display "rate=")
(display (/ cnt (- (current-time) start))) (newline)
))
(set! cnt (+ cnt 1))
(observe-text "係 拉 丁 字 母 同 阿 剌 伯 數 字 串 字")
)
(define (aloo) (mke) (aloo))
(aloo)
)
Above is .. wow its stable :
(heap-size . 21184512) (gc-times . 70)
(heap-size . 40243200) (gc-times . 242)
(heap-size . 40243200) (gc-times . 1458) 14:19.43 cpu
(heap-size . 40243200) (gc-times . 2554) 23:48.49 guile
ssooo .. its something thread-related. But just having netcat did not do the
trick. ... using java on one thread doesn't trigger it. Using observe-text
does not trigger it. (ie. relex-parse, etc from one thread.)
do we have zombie threads?
(define x (call-with-new-thread (lambda () (observe-text " 係 拉 丁 字 母 同 阿 剌
伯 數 字 串 字"))))
(define (th)
(define cnt 0)
(define start (- (current-time) 0.1))
(define (thu v) (call-with-new-thread (lambda ()
(observe-text " 係 拉 丁 字 母 同 阿 剌 伯 數 字 串 字"))))
(define lst (list 1 2 3 4 5 6 7 8 9 0))
(define lst (make-list 10 42))
(define (mke)
(define thl (map thu lst))
(length (all-threads))
(map join-thread thl)
(set! cnt (+ cnt 1))
(display "rate=")
(display (/ cnt (- (current-time) start))) (newline)
)
(define (aloo) (mke) (aloo))
(aloo)
)
Hmmm ... above is blowing up! but also .. its not actually threading,
(the atoms stuff is not happening out of order...)
either. Oh, but it is blowing up ....
(heap-size . 87683072) (gc-times . 4295)
(heap-size . 104460288) (gc-times . 4513)
(heap-size . 255717376) (gc-times . 4884)
OK, full production system with the new guile:
(heap-size . 481013760) (gc-times . 301)
(heap-size . 755843072) (gc-times . 402) 11 minutes CPU
(heap-size . 797286400) (gc-times . 530) 15 mins cpu
(heap-size . 1285570560) (gc-times . 1713) 100 mins CPU
(heap-size . 1546678272) (gc-times . 2251) 140 mins CPU
but is this really new? It was recompiled, but the version number is
wrong...
Again, full prduction: with guile -v
guile (GNU Guile) 2.1.5.19-7e9395 (8 Jan 2017)
(heap-size . 652918784) (gc-times . 233) ; about 8 mins CPU
(heap-size . 737722368) (gc-times . 339) ; 12 mins CPU
(heap-size . 1332973568) (gc-times . 1797) ; 120 mins CPU
(heap-size . 1441443840) (gc-times . 2221) ; 151 min CPU
(heap-size . 1521213440) (gc-times . 2441) ; 168 min cpu
(heap-size . 1595101184) (gc-times . 3061) ; 218 min cpu
(heap-size . 1726119936) (gc-times . 3292) ; 237 min
(heap-size . 1960865792) (gc-times . 6698) ; 510 minn
(heap-size . 1960865792) (gc-times . 10383) ; 805 min
(heap-size . 2931556352) (gc-times . 14211) ; 1199 min
(heap-size . 3302985728) (gc-times . 23278) ; 2186
Much improved: an earlier result was:
(heap-size . 4601581568) (gc-times . 485) ; almost 5GB!
===========================================
Non-openccog test case showing problem: bug
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25386
(define junk 0)
(define halt #f)
(define (wtf-thr)
(define start (- (current-time) 0.1))
; Create thread that does junk and exits. Yes, the increment
; of `junk` is not protected, and its racey, but so what.
(define (mkthr v) (call-with-new-thread (lambda () (set! junk (+ junk 1)))))
; thread arguments
(define thrarg (make-list 10 0))
(define cnt 0)
(define (mke)
; Create a limited number of threads
(define thr-list (map mkthr thrarg))
; (display (length (all-threads)))
(map join-thread thr-list)
; Some handy debug printing.
(set! cnt (+ cnt 1))
(if (eq? 0 (modulo cnt 500))
(begin
(display "rate=")
(display (/ cnt (- (current-time) start))) (newline)
(display "num threads=")
(display (length (all-threads))) (newline)
(display (gc-stats)) (newline) (newline)
)))
; tail recursive infinite loop.
(define (aloop) (mke) (if (not halt) (aloop)))
; while forever.
(aloop)
)
(call-with-new-thread wtf-thr)
(set! halt #t)
Gahh seg-faults too
Thread 1 "guile" received signal SIGSEGV, Segmentation fault.
thread_mark (addr=0x55556a3f9e00, mark_stack_ptr=<optimized out>,
mark_stack_limit=0x7fffffffd350, env=<optimized out>)
at ../../libguile/threads.c:111
warning: Source file is more recent than executable.
111 /* No threads; we can just use GC_stackbottom. */
(gdb) bt
#0 thread_mark (addr=0x55556a3f9e00, mark_stack_ptr=<optimized out>,
mark_stack_limit=0x7fffffffd350, env=<optimized out>)
at ../../libguile/threads.c:111
bug-guile@gnu.org
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25386
----------------------------------------------------
(heap-size . 5939200) (gc-times . 70)
(heap-size . 45510656) (gc-times . 147)
(heap-size . 399896576) (gc-times . 1018)
(hash-map->list cons (module-obarray (current-module)))
(heap-size . 183734272) with gc inside the thread ... (gc-times . 402007)
(heap-size . 183734272) (gc-times . 976989)
(heap-size . 183734272) (gc-times . 1869962)
(heap-size . 183734272) (gc-times . 8526663)
stochastic gc:
(heap-size . 369917952) (gc-times . 40387)
(heap-size . 428638208) (gc-times . 511116)
(heap-size . 428638208) (gc-times . 1218770)
(heap-size . 428638208) (gc-times . 6515758)
every-17th gc:
(heap-size . 1875902464) (gc-times . 290105)
(heap-size . 2068840448) (gc-times . 2605562) -- 44325000 threads
every 10th-gc in main loop, not in thread:
(heap-size . 544063488) (gc-times . 113)
(heap-size . 5151375360) (gc-times . 1113)
(heap-size . 8322269184) (gc-times . 2113)
Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS
gc must be done at thread-exit, for this plan to work!
size=120MB+90MB * n
(define mtx (make-mutex))
(define (mkthr v) (call-with-new-thread (lambda ()
(lock-mutex mtx)
(if (eq? 0 (modulo junk 17)) (gc))
(set! junk (+ junk 1))
(unlock-mutex mtx)
)))
--------
OK new guile-2.2 doesn't blow up any more:
(heap-size . 7921664) (gc-times . 40)
(heap-size . 14344192) (gc-times . 953)
(heap-size . 14344192) (gc-times . 5219) ; after 4 minutes CPU
(heap-size . 26419200) (gc-times . 64975) ; after 77 minutes CPU
(heap-size . 26419200) (gc-times . 133346) ; after 154 mins CPU
(heap-size . 26419200) (gc-times . 170083) ; after 192 mins CPU
(heap-size . 26419200) (gc-times . 249102) ; after 283 mins cpu
(heap-size . 26419200) (gc-times . 420031) ; after 468 min cpu
(heap-size . 26419200) (gc-times . 557039) ; after 804 mins CPU
alt version: 510 threads:
(heap-size . 10604544) (gc-times . 32)
(heap-size . 19505152) (gc-times . 484)
(heap-size . 35926016) (gc-times . 1761)
(heap-size . 48238592) (gc-times . 4217) ; after 8 minutes cpu time
(heap-size . 48238592) (gc-times . 47902) ; after 76 mins CPU
(heap-size . 48238592) (gc-times . 73063) ; after 114 mins CPU
(heap-size . 65540096) (gc-times . 128094) ; after 209 mins cpu
(heap-size . 65540096) (gc-times . 248321) ; after 399 mins
(heap-size . 65540096) (gc-times . 344197) ; after 546 min
dffe495d0de1466f62a91a6d74cc0f388e0f4f3f
* libguile/threads.c (on_thread_exit): Lessen excess retention.
12eb7b8256f579fab60ebe0b38eb8788c1276eb8
scm_i_vm_free_stack
=================================================================
Also 25387
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25387
===================================================================
(make-dynamic-state (current-dynamic-state))
(define (ds)
(define cnt 0)
(define start (- (current-time) 0.1))
(define (mke)
(define x (make-dynamic-state (current-dynamic-state)))
(set! cnt (+ cnt 1))
(if (eq? 0 (modulo cnt 1000000))
(begin
(display "iter=") (display cnt) (newline)
(display "rate=")
(display (/ cnt (- (current-time) start))) (newline)
)))
(define (aloo) (mke) (aloo))
(aloo)
)
Above looks just fine.
(count-all)
(cog-report-counts)
(cog-prt-atomspace)
(length (all-threads))
(gc-stats)
====================================================================
Bug -- split needs to put whitespace around parent, before/after commas,
etc. and begin-end quotes FIXED
====================================================================
Bugg- the utf8 is fucked up. Fixed. Its a guile-2.1.5 regression
its ok in the containers ... select * from atoms where type=110;
(cog-get-atoms 'WordInstanceNode)
but guile-2.2 fucks up.
(string-append "a" "係" "c") is OK.
(Concept "係") is broken
scm_c_eval_string
scm_print_state
scm_puts now does scm_c_put_latin1_chars fuuuuuck,.
scm_lfwrite (s, strlen (s), port); --- again with latin1
scm_c_put_string
arggg
bug-guile@gnu.org -- bug#25397:
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25387
GUILE_VERSION_MINOR 1
====================================================================
call-with-new-thread
threads.c
data = scm_gc_typed_calloc (launch_data);
err = scm_i_pthread_create (&id, NULL, launch_thread, data);
scm_i_pthread_detach (scm_i_pthread_self ());
pthread-threads.h:#define scm_i_pthread_detach pthread_detach
#define scm_i_pthread_self pthread_self
join_thread_var libguile/threads.x uh no
(with-continuation-barrier
(set! (thread-join-data thread) (cons cv mutex))
scm_i_with_guile calls ...
GC_call_with_stack_base which is from boehm
calls with_guile
calls scm_i_init_thread_for_guile calls scm_i_init_guile
scm_c_with_continuation_barrier
scm_current_dynamic_state () -- save_dynamic_state ? maybe not freed?
what is in the currrent dynamic state?
everything looks kosher. best guess: copy of dyanmic state is not being gc'ed.
SCM_USE_PTHREAD_THREADS ?? defined ?? yes.
build/libguile/scmconfig.h:#define SCM_USE_PTHREAD_THREADS 1
GC_register_my_thread
on_thread_exit ??? thread_count--; seems to be called because throead count
is correct ... who sets scm_i_thread_key ? .. guile init does.
---------------------------------------------------------------
Ongoing failure:
If I gc after every thread, then hit this:
guile: hashtab.c:137: vacuum_weak_hash_table: Assertion `removed <= len'
failed.
Aborted
2x... once after a minute
3x ...
try 2.0.13
autoconf flex gettext libunistring-dev libffi-dev texinfo
apt-get purge guile-2.0 guile-2.0-libs
---------------------------------------------------------------
Bug --- java fucks up like this:
above: java is runnning at 900% so its getting stuff in paralel.
It seems to be responding slowly, though ... and guile is not running
at high cpu .. its under 100% wtf... ah haha -- java is spinning
on socket wait in some way...
Its messed up. Java is buggy.
------------------------------------------------------------------
Bug --- java always gives back exactly the same ... is the rand
num generator being reset each time??? I think it is, for
reproducibility? No .. it was a bug. Fixed. pull req #471 in LG
for morphemes, anysplit.c always sets seed=0 every call.
anysplit.c:
rng_uniform
sample_point
use_sampling
api.c: (rand_r(&sent->rand_state) %
parse_options_set_repeatable_rand
struct anysplit_params * anysplit;
set by
duuuude wtf 0 3b23e1b0 240 st=1840299789
duuuude wtf 1 78520aa1 33 st=-895367152
duuuude wtf 2 6372c222 322 st=1780636463
duuuude wtf 3 5c322e38 312 st=-231201766
duuuude wtf 4 303c3bea 146 st=-1008023999
duuuude wtf 5 2a463357 233 st=-480833580
Linkage lkg = &sent->lnkages[in];
Linkage_info *lifo = &lkg->lifo;
iindex
sentence_parse
63 70 248 277 -- birthday paradox
------------------------------------
LG: FIXED, will be in version 5.3.14 -- LG pull req #470
any is ignoring the last word
no its not, its just not displaying it. so its a minor bug.
linkage_print_diagram
l->r pass removed 1
LEFT-WALL[1] 港[2] 區[2] 全[2] 國[2] 人[2]
utf8_strlen
------------------------
BUGGGGGGGGGGG -- delete everything fails, cause still counting in the other
threads!! Aieee! ...
its also probably picking up the wrong sentence...
use .. parse-get-relex-outputs ? Uh no
(define foo
(let ((mtx (make-mutex)))
(lambda ()
(display mtx) (newline)
(format #t "duude is locked? ~A\n" (mutex-locked? mtx))
(format #t "duude owner ~A\n" (mutex-owner mtx))
(format #t "duude try ~A\n" (try-mutex mtx))
))
)
(use-modules (opencog))
(AnchorNode "foo")
(ListLink (AnchorNode "foo") (Concept "a"))
(ListLink (AnchorNode "foo") (Concept "b"))
(ListLink (AnchorNode "foo") (Concept "c"))
(SentenceSequenceLink
(WordSequenceLink
(cog-extract-recursive sent)
Delete (SentenceSequenceLink (sent numbernode))
(use-modules (ice-9 receive) (srfi srfi-1) (opencog) (opencog
atom-types))
(use-modules (opencog) (opencog persist) (opencog persist-sql))
(use-modules (opencog nlp learn))
(sql-open "opencog_test" "opencog_tester" "cheese")
(observe-text "this is a foo")
psql -h localhost -U opencog_tester opencog_test
PACKAGE_VERSION
config.h:#define PACKAGE_VERSION "2.1.5.19-7e9395"
Seem to have
libgc-dev
dpkg -l libgc1c2
Desired=Unknown/Install/Remove/Purge/Hold
|
Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-=================================
ii libgc1c2:amd64 1:7.4.2-8 amd64 conservative garbage
collector fo
---------------------------------------------------------------
May lead to memory leak and poor performance.
GC Warning: Repeated allocation of very large block (appr. size
21573632):
May lead to memory leak and poor performance.
GC Warning: Repeated allocation of very large block (appr. size
43147264):
May lead to memory leak and poor performance.
GC Warning: Repeated allocation of very large block (appr. size
9056256):
May lead to memory leak and poor performance.
GC Warning: Repeated allocation of very large block (appr. size
43147264):
May lead to memory leak and poor performance.
GC Warning: Repeated allocation of very large block (appr. size
43147264):
May lead to memory leak and poor performance.
GC Warning: Repeated allocation of very large block (appr. size
86290432):
May lead to memory leak and poor performance.
GC Warning: Repeated allocation of very large block (appr. size
7438336):
May lead to memory leak and poor performance.
GC Warning: Repeated allocation of very large block (appr. size
9588736):
May lead to memory leak and poor performance.
In conclusion:
(gc-stats)
Entering scheme shell; use ^D or a single . on a line by itself to exit.
guile-yue> ((gc-time-taken . 7077498787602) (heap-size . 15487754240)
(heap-free-size . 3580190720) (heap-total-allocated . 900893846432)
(heap-allocated-since-gc . 6264284448) (protected-objects . 288)
(gc-times . 1059))
15.5GB in total, 3.6GB free
guile-yue> (gc)
guile-yue> (gc-stats)
((gc-time-taken . 7122988472428) (heap-size . 15487754240)
(heap-free-size . 8725934080) (heap-total-allocated . 900895104976)
(heap-allocated-since-gc . 1142496) (protected-objects . 288) (gc-times
. 1060))
Now its 8.7GB free, leaving 7GB in use.
guile-yue> (cog-report-counts)
((ListLink . 331150) (EvaluationLink . 331150) (AnchorNode . 1)
(WordNode . 10720) (LinkGrammarRelationshipNode . 1) (LgConnectorNode .
1) (LgConnMultiNode . 1) (LgConnDirNode . 2) (LgConnector . 4) (LgAnd .
4))
guile-yue> (count-all)
673034
So why is guile so splurgy? This is just not that many atoms ...
is bdwgc too conservative, somehow finding the TLB stuff?
what If I close out the TLB, does guile free it all?
1244:00 start of sql-close cpu time at start
1244:36 at stop -- so about 40 cpu-sconds, OK.
((gc-time-taken . 7128743570856) (heap-size . 15487754240)
(heap-free-size . 14163222528) (heap-total-allocated . 900991577328)
(heap-allocated-since-gc . 172640) (protected-objects . 288) (gc-times .
1061))
whoa -- that was it:
fre is now 14.2 out of 15.5GB so actual use is only 1.3GB
... for 673K atoms, so about 2KBytes/atom which is still a lot, but
... plausible.
So bdwgc was confused about the TLB... how? why? what about atom
creation that leaves around this handle-confusion?
before sql close:
ubuntu 4335 217 29.4 32768196 29150768 pts/6 Sl+ Jan11 1243:04 guile -l pair-count-yue.scm
after:
ubuntu 4335 210 28.9 32164608 28616384 pts/6 Sl+ Jan11 1244:45 guile -l pair-count-yue.scm
so no shrinkage directly....
(cog-map-type (lambda (ato) (cog-extract-recursive ato) #f) 'WordNode)
(gc)
.. fails to shrink the heap!
stop the cogserver ... (gc) still no shrinkage
void fn (char*, GC_word)
GC_set_warn_proc (fn)
https://www.hboehm.info/gc/gcinterface.html
GC_warn_proc GC_set_warn_proc(GC_warn_proc p)
libcord
libgc.so -> libgc.so.1.0.3
GC_current_warn_proc
GC_default_warn_proc
WARN
select count(*) from atoms;
1480054
1522104
6: ??:0 GC_alloc_large()
7: ??:0 GC_generic_malloc()
8: ??:0 GC_core_malloc_atomic()
9: strings.c:165 make_wide_stringbuf()
10: strings.c:331 scm_i_make_wide_string()
11: strings.c:1165 scm_string()
12: vm-engine.c:763 vm_debug_engine()
SCM_STRING_LENGTH_HISTOGRAM
%stringbuf-hist
6: ??:0 GC_alloc_large()
7: ??:0 GC_generic_malloc()
8: weak-table.c:362 allocate_entries()
9: weak-table.c:432 is_acceptable_size_index()
10: weak-table.c:677 weak_table_put_x()
11: weak-table.c:893 scm_c_weak_table_put_x()
12: read.c:423 maybe_annotate_source()
13: gc.h:182 maybe_annotate_source()
14: read.c:725 scm_read_string()
15: read.c:476 scm_read_sexp()
16: read.c:1814 read_inner_expression()
17: read.c:476 scm_read_sexp()
18: read.c:1814 read_inner_expression()
19: read.c:1964 scm_read()
20: vm-engine.c:763 vm_debug_engine()
scm_sys_string_dump
scm_sys_symbol_dump
scm_ilength
#define SCM_CHAR(x) ((scm_t_wchar)SCM_ITAG8_DATA(x))
Ah hah ... very long strings are ... very long strings
from java.
---------------------------------------------------------------
local_id_cache
add_id_to_cache
get_ids
maybe_create_id
id_create_cache
WordInstanceNode
LgLinkInstanceNode
119 | WordInstanceNode
120 | WordInstanceLink
159 | LgLinkInstanceNode
160 | LgLinkInstanceLink
how are these getting in the TLB???
SQLBackingStore::getLink(Handle& h) const
{
AtomSpace::fetch_atom
persist/guile/PersistSCM.cc
fetch-atom
fetch-incoming-set
is id deadlocking on the resolve?
t1
delete
get table lock
rele table lock
now t2 gets the table lock.
get tlb lock << maybe halt here?
<<<< safe because nothing inside, so forward progress...
relese tlb lock
get table lock. << maybe halt here?
t2
add
table lock 1
tlb lock
do_res
table lock 2
<<< should wordwar prgress cause in ssame thread.
argh 201 theads
201 -- do_poll_result SchemeEval.cc:655
200 -- join from eval_loop GenericShell.cc:457
197 -- getNodeHandle AtomTable.cc:255
owenr 20950 whih is 44
44 -- TLB::removeAtom TLB.cc:143
owner is 21423 which is 179
179 -- AtomTable::getNodeHandl AtomTable.cc:255
owner is 20950 which is 44
so TLB can't get lock cause its held by 179
but 179 is holding lock and calling do_res while holding the
atomspace lock. called by tlb::add atom
solutions(s): TLB must drop lock while doing do_res
atomtable must fully unwindws recursive lock
so who in 44 has it?
AtomTable.cc:841 is hwere we are ...
called at line 762 with lock held at line 729 so we are at least
one-deep. maybe more??
std::unique_lock<std::recursive_mutex>
double frac = 100.0 * extra / ((double) all.size());
printf("sql-stats: Examined %lu atoms in atomspace; %lu extra in TLB
(%f percent) \n",
all.size(), extra, frac);
double frac = 100.0 * extra / ((double) all.size());
printf("sql-stats: Found %lu atoms in atomspace; %lu extra in TLB
(%f percent) \n",
all.size(), extra, frac);
frac = 100.0 * extra / ((double) _store->_tlbuf.size());
printf("sql-stats: tlbuf size=%lu extra=%f percent\n",
_store->_tlbuf.size(),
frac);
}
the __data.__owner member of the pthread_mutex_t
print mutex.__data.__owner
---------------------------------------------------------------
(near-startup)
sql-stats: Atomspace holds 55484 atoms
sql-stats: tlbuf holds 19928 atoms
sql-stats: tlbuf holds 0 atoms not in atomspace (0.000000 pct)
sql-stats: tlbuf holds 2031 unremapped atoms (6.600585 pct)
sql-stats: 211017858 of 211020413 uuids unused (99.998789 pct)
Gaaaaackk .. this implies that 99.999 percent of calls to sql
are wasted ! well, actually not, since the cache should stop
the bad acting.
sql-stats: Atomspace holds 420064 atoms
sql-stats: tlbuf holds 154316 atoms
sql-stats: tlbuf holds 0 atoms not in atomspace (0.000000 pct)
sql-stats: tlbuf holds 16398 unremapped atoms (6.627275 pct)
sql-stats: 211339442 of 211467072 uuids unused (99.939645 pct)
sql-stats: Atomspace holds 86550 atoms
sql-stats: tlbuf holds 80674 atoms
sql-stats: tlbuf holds 0 atoms not in atomspace (0.000000 pct)
sql-stats: tlbuf holds 59344 unremapped atoms (72.948089 pct)
sql-stats: 212784068 of 212861833 uuids unused (99.963467 pct)
====================================================
clean start:
sql-stats: Atomspace holds 13949 atoms
sql-stats: tlbuf holds 5750 atoms
num_get_nodes=18537 num_got_nodes=235 (1.267735 pct)
num_get_links=56472 num_got_links=1116 (1.976201 pct)
num_get_insets=0 num_get_inatoms=0
num_node_inserts=6 num_node_updates=31444
num_link_inserts=256 num_link_updates=20732
Remove 21485 of 64000 (33.570312 pct) calls tlbzz=9219
sql-stats: tlbuf holds 0 atoms not in atomspace (0.000000 pct)
sql-stats: tlbuf holds 1484 unremapped atoms (20.694464 pct)
sql-stats: 213462841 of 213464601 uuids unused (99.999176 pct)
sql-stats: Atomspace holds 31687 atoms
sql-stats: tlbuf holds 23769 atoms
num_get_nodes=253402 num_got_nodes=1131 (0.446326 pct)
num_get_links=779439 num_got_links=14328 (1.838245 pct)
num_get_insets=0 num_get_inatoms=0
num_node_inserts=70 num_node_updates=517152
num_link_inserts=4190 num_link_updates=340938
sql-stats: tlbuf holds 0 atoms not in atomspace (0.000000 pct)
sql-stats: tlbuf holds 15624 unremapped atoms (66.088575 pct)
sql-stats: 213763992 of 213783890 uuids unused (99.990692 pct)
sql-stats: AtomSpace not set
sql-stats: Atomspace holds 45225 atoms
sql-stats: tlbuf holds 45225 atoms
num_get_nodes=646388 num_got_nodes=1763 (0.272746 pct)
num_get_links=1988905 num_got_links=33444 (1.681528 pct)
num_get_insets=0 num_get_inatoms=0
num_node_inserts=107 num_node_updates=1339311
num_link_inserts=9898 num_link_updates=883771
sql-stats: tlbuf holds 0 atoms not in atomspace (0.000000 pct)
sql-stats: tlbuf holds 46231 unremapped atoms (52.940099 pct)
sql-stats: 214585523 of 214645132 uuids unused (99.972229 pct)
sql-stats: Atomspace holds 446035 atoms
sql-stats: tlbuf holds 212056 atoms
num_get_nodes=1572865 num_got_nodes=2519 (0.160154 pct)
num_get_links=4838305 num_got_links=68870 (1.423432 pct)
num_get_insets=0 num_get_inatoms=0
num_node_inserts=215 num_node_updates=3172207
num_link_inserts=22422 num_link_updates=2094051
sql-stats: tlbuf holds 0 atoms not in atomspace (0.000000 pct)
sql-stats: tlbuf holds 72188 unremapped atoms (28.542847 pct)
sql-stats: 215438536 of 215580711 uuids unused (99.934050 pct)
after 10 hours:
sql-stats: AtomSpace not set
sql-stats: Atomspace holds 731146 atoms
sql-stats: tlbuf holds 725447 atoms
num_get_nodes=20215212 num_got_nodes=6866 (0.033965 pct)
num_get_links=61851573 num_got_links=404021 (0.653211 pct)
num_get_insets=0 num_get_inatoms=0
num_node_inserts=4251 num_node_updates=42211029
num_link_inserts=307398 num_link_updates=27844955
sql-stats: tlbuf holds 0 atoms not in atomspace (0.000000 pct)
sql-stats: tlbuf holds 411009 unremapped atoms (56.626772 pct)
sql-stats: 240293860 of 241016540 uuids unused (99.700153 pct)
20 gb resident 81 M atoms removed.
Finally:
---------
sql-stats: Atomspace holds 726040 atoms
sql-stats: tlbuf holds 726040 atoms
num_get_nodes=20356005 num_got_nodes=6882 (0.033808 pct)
num_get_links=62281852 num_got_links=405421 (0.650946 pct)
num_get_insets=0 num_get_inatoms=0
num_node_inserts=4264 num_node_updates=42510570
num_link_inserts=309468 num_link_updates=28042704
sql-stats: tlbuf holds 0 atoms not in atomspace (0.000000 pct)
sql-stats: tlbuf holds 412287 unremapped atoms (56.785714 pct)
sql-stats: 240482670 of 241208711 uuids unused (99.698999 pct)
((gc-time-taken . 7898668634177) (heap-size . 17208950784)
(heap-free-size . 9361924096) (heap-total-allocated . 996784769424)
(heap-allocated-since-gc . 1247263344) (protected-objects . 292)
(gc-times . 1068))
heap is 17.2GB!! free is 9.4GB so 7.8GB in use
((gc-time-taken . 7910648760945) (heap-size . 17208950784)
(heap-free-size . 12845494272) (heap-total-allocated . 996787132704)
(heap-allocated-since-gc . 331136) (protected-objects . 292) (gc-times .
1069))
heap is 17.2GB!! free 12.8GB so actually only 4.4GB is needed.
guile-yue> (sql-close)
guile-yue> (gc)
guile-yue> (gc-stats)
((gc-time-taken . 7912464056925) (heap-size . 17208950784)
(heap-free-size . 16106553344) (heap-total-allocated . 996792364160)
(heap-allocated-since-gc . 164096) (protected-objects . 292) (gc-times .
1070))
free is 16.1GB so somehow the TLB was chewing up 3.3GB holding
412287 unremapped atoms!? How is this possible? Should I have run gc a
bunch more times?
-------------------------------------------------------------
New API:
Very fast: in 3 minutes:
sql-stats: Atomspace holds 113815 atoms
sql-stats: tlbuf holds 1195 atoms
num_get_nodes=26378 num_got_nodes=26361 (99.935552 pct)
num_get_links=8813 num_got_links=8251 (93.623057 pct)
num_get_insets=0 num_get_inatoms=0
num_node_inserts=133 num_node_updates=52697
num_link_inserts=1062 num_link_updates=16550
sql-stats: tlbuf holds 0 atoms not in atomspace (0.000000 pct)
sql-stats: tlbuf holds 0 unremapped atoms (0.000000 pct)
sql-stats: 0 of 1196 uuids unused (0.000000 pct)
wtf...
(cog-report-counts)
((NumberNode . 46505) (ListLink . 163927) (EvaluationLink . 215741)
(AnchorNode . 1) (WordNode . 5199) (ReferenceLink . 99275) (SentenceNode
. 55) (ParseNode . 837) (ParseLink . 835) (WordInstanceNode . 47312)
(WordInstanceLink . 47437) (WordSequenceLink . 47464)
(SentenceSequenceLink . 55) (LinkGrammarRelationshipNode . 1)
(LgConnectorNode . 1) (LgConnMultiNode . 1) (LgConnDirNode . 2)
(LgConnector . 4) (LgAnd . 4) (LgWordCset . 46306) (LgLinkInstanceNode .
52608) (LgLinkInstanceLink . 52715))
Runnning
sql-stats: Atomspace holds 687522 atoms
sql-stats: tlbuf holds 684142 atoms
num_get_nodes=39921257 num_got_nodes=40050951 (100.324875 pct)
num_get_links=13354387 num_got_links=13011199 (97.430148 pct)
num_get_insets=0 num_get_inatoms=0
num_node_inserts=10662 num_node_updates=80083214
num_link_inserts=673480 num_link_updates=26027646
sql-stats: tlbuf holds 0 atoms not in atomspace (0.000000 pct)
sql-stats: tlbuf holds 0 unremapped atoms (0.000000 pct)
sql-stats: 0 of 684143 uuids unused (0.000000 pct)
(gc-stats)
((gc-time-taken . 49802336967563) (heap-size . 14738780160)
(heap-free-size . 9885679616) (heap-total-allocated . 947281529744)
(heap-allocated-since-gc . 1685837392) (protected-objects . 352)
(gc-times . 1242))
So while running: 14.7GB heap, 9.9GB free
guile-yue> (cog-report-counts)
((NumberNode . 737) (ListLink . 337776) (EvaluationLink . 338599)
(AnchorNode . 1) (WordNode . 10670) (ReferenceLink . 1557) (SentenceNode
. 1) (ParseNode . 16) (ParseLink . 16) (WordInstanceNode . 736)
(WordInstanceLink . 736) (WordSequenceLink . 736) (SentenceSequenceLink
. 1) (LinkGrammarRelationshipNode . 1) (LgConnectorNode . 1)
(LgConnMultiNode . 1) (LgConnDirNode . 2) (LgConnector . 4) (LgAnd . 4)
(LgWordCset . 720) (LgLinkInstanceNode . 821) (LgLinkInstanceLink .
821))
while idle:
(cog-report-counts)
((ListLink . 339276) (EvaluationLink . 339276) (AnchorNode . 1)
(WordNode . 10750) (LinkGrammarRelationshipNode . 1) (LgConnectorNode .
1) (LgConnMultiNode . 1) (LgConnDirNode . 2) (LgConnector . 4) (LgAnd .
4))
So the cruuft goes away.
((gc-time-taken . 49831001288560) (heap-size . 14738780160)
(heap-free-size . 5885931520) (heap-total-allocated . 956146319472)
(heap-allocated-since-gc . 4515746160) (protected-objects . 296)
(gc-times . 1243))
Heap:free: 5.9GB wow -- lots smaller than while runing.
Because 4.5GB alloced since last gc
(heap-free-size . 9419513856)
(heap-free-size . 13306900480)
(heap-free-size . 13710024704)
(heap-free-size . 13740257280) wow
of
(heap-size . 14738780160) 14.7 so only 1 GB needed.
ten-word-java: 289105 bytes 2m18 backlog
20-word java: 550294 bytes 3m12.772s
40-word hava: 1101459 bytes: 3m15 delay
ignoreType is not being used
Wow .. the remove is completely un-needed! DONE
The "Loaded 370000 atoms" not needed at all, if not bulk-loading. DONE
this dead-locks the shell:
(observe-test "商 餘 質 數 分 解 質 數 定 理 相 對 質 數")
ERROR: In procedure apply-smob/1:
ERROR: Throw to key `decoding-error' with args `("scm_from_utf8_stringn"
"input locale conversion error" 29 #vu8(40 111 98 115 101 114 118 101 45
116 101 115 116 32 34 229 149 134 32 233 164 152 32 232 179 170 32 230
149 184 32 229 136 134 32 232 167 163 32 232 179 170 32 230 149 184 32
229 174 255 237 255 253 6 10))'.
ERROR: In procedure fetch-atom: Earlier version of atom has mis-matched
UUID! (/home/ubuntu/src/atomspace/opencog/atomspaceutils/TLB.cc:85)
---------------------------------------------------------------
-- fix TLB tables in sql - underway
(deadlock) DONE
-- fix sennas bug. DONE
-- publish new LG DONE
---------------------------------------------------------------
The fetch not needed if in the atomspace already. Well,
then don't fetch it!
There is no TV invalidation mechanism...
---------------------------------------------------------------
-- fetch not requied, unless updating the TV ... !!
(or getting the tv...
-- fix pgsql variant .. maybe
-- update all lxc's.
=======================================================================
20 Jan 2017 Friday
OK, all bugs above fixed, reinstalled the latest LG in yue,
erase all tables, start from scratch. At 4PM exactly, wall-clock time
to the second.
at start
$ find beta-pages |wc 49193 52687 1489227
$ find submitted-articles |wc 24 24 629
-------
After 40 minutes:
select count(*) from atoms; 111632
Java/do-count: 1064 sentences
find submitted-articles |wc 205
so, rate: 4.5 articles/minute, 5.9 sentences/article.
rate: 26.6 sents/minute
projected finish: 182 hours === 7.5 days
(do-count "report-stuff") gives alternate sentence count...
------
After 4.000 hours:
select count(*) from atoms; 454240
Java: cnt=6533 but 6610 from do-count !! wtf.
$ find submitted-articles |wc 919
38.9M node insert/updates 7164 total nodes!?
12.99 link insert/updates 447516 total links!?
== 51946920 total stores, so that cross-checks OK
total number of atoms also checks out (skew for measurement delay) OK
so, rate: 3.73 articles/minute, 7.4 sents/article
rate: 27.5 sents/minute
projected finish: 220 hours = 9.1 days
So the article-rate is bouncing around, but sentences/minute holds
steady. Numbers are checking out.
-------
After 8.00 hours:
select count(*) from atoms; 666606
Java: cnt=10787 but do-count says cnt=11464
$ find submitted-articles |wc 1634
66.1M node updates
22.0M link updates
So...
rate: 1610 / 8 hours = 201 articles/hour, 3.35 articles/minute
rate: 11464 / 480 min = 23.9 sents/min
22 minutes downtime for system upgrade....
4 concurrent writers:
after 15 mins:
write items=417737 drains=2532 fill_fraction=164.983017 concurrency=51.921801
after 20 mins cpu:
write items=616070 drains=3826 fill_fraction=161.021955 concurrency=53.232358
12 concurrent writers:
after 15 mins
write items=372257 drains=1805 fill_fraction=206.236565 concurrency=53.760111
after 20 mins:
write items=539921 drains=2802 fill_fraction=192.691292 concurrency=54.867595
... doesn't improve concurrency, and seems slower for total CPU, but
that may be fucked up java. ... XXX see below: because there were only
6 connections to postgres, so really of the 12 writers, 7 were blocked.
(assumiong one reader thread, or more...)
New code, where no atom get: (and 12 writer threads)
write items=21516264 drains=167609 fill_fraction=128.371770 concurrency=4.759196
avg drain time=0.345310 seconds; longest drain time=80.337000
New code, 12 writers, and C impl of count-increment:
write items=7817004 drains=61061 fill_fraction=128.019587 concurrency=3.714679
avg drain time=0.517914 seconds; longest drain time=29.733000
New code, 4 writers, and C impl of count-increment:
drains=7186 fill_fraction=136.883245 concurrency=3.445728
drains=25794 fill_fraction=124.202528 concurrency=3.613825
avg drain time=0.440724 seconds; longest drain time=5.136000
4 writers is not obvious worse and seems better than 12.
Ohh .. well, turns out there were only 6 odbc connections
available to work with.
--------------------------------------------------
21 January -- saturday -- start from scratch, again.
start at exactly 15:00
after 15 mins:
select count(*) from atoms; 66221
find submitted* |wc 127
sql-stats: total loads = 296 total stores = 4379768 ratio=14796.513514
num_get_nodes=4246 num_got_nodes=296 (6.971267 pct)
num_get_links=61974 num_got_links=0 (0.000000 pct)
num_get_insets=0 num_get_inatoms=0 ratio=-nan
num_node_inserts=2123 num_node_updates=3282677 ratio=1546.244465
num_link_inserts=61974 num_link_updates=1033002 ratio=16.668313
total stores for node=3284800 link=1094976 ratio=2.999883
write items=2189854 flushes=0 flush_ratio=inf
drains=17804 fill_fraction=122.997866 concurrency=3.866659
avg drain time=0.418640 seconds; longest drain time=4.090000
(monitor-rate "foo")
------------
18:15 -- 3hrs 15 mins later:
find submitted* |wc gives 1025 so
exactly 1000 articles in 195 minutes = 5.12 articles per minute
select count(*) from atoms; 482295
total loads = 803 total stores = 55843525
num_node_inserts=7427 num_node_updates=41875213 ratio=5638.240609
num_link_inserts=472298 num_link_updates=13488591 ratio=28.559492
total stores for node=41882640 link=13960889 ratio=2.999998
write items=27921766 flushes=0 flush_ratio=inf
drains=234012 fill_fraction=119.317667 concurrency=2.665735
avg drain time=0.348755 seconds; longest drain time=6.107000
why is write-items=27.9M but total stores=55.8M ?
write-items is exactly half of total stores ...
select sum(stv_count) from atoms; gives 28995086 shortly later.
they're close ... so write-items actually matches word-counting...
so we are double-storing somehow...
-------------------------------------------------
on shutdown, at 22:30 --
select sum(stv_count) from atoms; 62921108
select count(*) from atoms; 861867
find submitted* |wc 2314
total stores = 125995216
write items=62997608
write items=62997608 flushes=0 flush_ratio=inf
drains=532502 fill_fraction=118.304923 concurrency=2.559669
avg drain time=0.344836 seconds; longest drain time=6.107000
Rearrange; total stores still running at 2x write-items.
But ... 2 links, 3 nodes:
Eval link --
Pred node
List link
left node
right node
so the list link is never explicitly stored. which leaves 1 link, 3
nodes being explicit i.e. 4 explicit stores (for 5 atoms)
is write-items counting only the links? that would be only two, in tat
case... but the write items are counted in the queue.
--------------------------------------------
Bugs:
if there is a store in the queue, and there is is a get for something
in the queue, then the get will get stale data, and then try to write
back more stale data, ad infinitum... causing the counts to be bad.
well, its up to user to use the barrier, which should now work.
So .. need mechanism to fetch locally, until fully retired!
Better yet, in scheme, just don't .. don't what?
don't fetch, unless stv is the default stv ...!
-----------------------------------------------
23 Jan 2017:
------------
Holy cow -- postgres is sooo much faster!!
check this out:
after just a few minutes:
write items=94794 flushes=0 flush_ratio=inf
drains=70 fill_fraction=1354.200000 concurrency=4.385714
avg drain time=0.065023 seconds; longest drain time=0.623000
fill fraction used to be only about 150, before, and average drain time
was something like 0.3 seconds... this is a whole new ballgame!
after 1/2 hour wall clock
write items=1462482 flushes=0 flush_ratio=inf
drains=1830 fill_fraction=799.170492 concurrency=5.049180
avg drain time=0.080646 seconds; longest drain time=1.829000
after 100 minutes
write items=4580980 flushes=0 flush_ratio=inf
drains=5394 fill_fraction=849.273267 concurrency=4.693919
avg drain time=0.118629 seconds; longest drain time=70.343000
after 7 hours
write items=18375732 flushes=0 flush_ratio=inf
drains=21386 fill_fraction=859.241186 concurrency=4.406621
avg drain time=0.084344 seconds; longest drain time=70.343000
after 13 hours:
write items=32066762 flushes=0 flush_ratio=inf
drains=37044 fill_fraction=865.639834 concurrency=4.370316
avg drain time=0.077263 seconds; longest drain time=70.343000
after 24 hours:
write items=56233665 flushes=0 flush_ratio=inf
drains=67199 fill_fraction=836.822944 concurrency=4.378875
avg drain time=0.072986 seconds; longest drain time=70.343000
So this looks to be stable.
after 3 days:
write items=199535356 flushes=0 flush_ratio=inf
drains=310946 fill_fraction=641.704206 concurrency=5.116091
avg drain time=0.087041 seconds; longest drain time=70.343000
so its ogt a little worse... but atomspace is much much bigger, now.
---------------
and also: java is fixed: the magic fix was:
-- shut everything down every 500 sentences,
-- run gc five times in a row,
-- restart everything.
This unclogs the kitchen sink. -- java now stays up for 4+ hours,
without tanking.
progress:
as of 23 jan 01:50 AM have find submitted* |wc being 8198
2:15 AM 8299 and (monitor-rate "foo") cnt=2080 rate=0.740
i.e. 0.74 sents/sec = 44 sents/minute!
that's nearly double the previous rate.
9:15 AM 10711 -> 10711-8299=2412/(7*60+25) = 5.4 articles/min
-> 325 articles/hour = 7805 articles/day
cnt=18200 rate=0.65566 -> 39 sentences/minute
10:47 AM cnt=21430 rate=0.640945
12:43 PM cnt=27054 rate=0.668858
15:04 PM cnt=33874 rate=0.692012
24 jan 01:34 AM cnt=57497 rate=0.663148
18160 articles->(18160-8198)/24 = 415 articles/hour
10K articles/day
25 Jan 12:04 PM cnt=134017 rate=0.63543
27967 articles->(27967-8198)/59 = 335 articles/hour
26 Jan 15:26 PM cnt=196794 rate=0.63602
35736 articles
-----------------
CentOS 5
psql opencog_test -U opencog_tester -W -h localhost
PGPORT=5433
PGDATA=/home/linas/postgres/var/lib/pgsql/data
PGLOG=/tmp/pgstartup.log
postgres -c "/usr/bin/postmaster -p '$PGPORT' -D '$PGDATA' &" >> "$PGLOG" 2>&1
postgres -c "/usr/bin/postmaster -p 5432 -D /home/linas/postgres/var/lib/pgsq/data &" >> /home/linas/postgres/tmp/pgstartup.log 2>&1
alt: chroot /home/linas/postgres; /etc/init.d/postgresql start
nooo dudn't work.
just this:
/home/linas/postgres/etc/init.d/postgresql start
/usr/bin/postmaster -p 5432 -D /home/linas/postgres/var/lib/pgsql/data &
>> /home/linas/postgres/tmp/pgstartup.log 2>&1
psql opencog_test -U opencog_tester -W -h localhost
Nooooo
look at .odbc.ini
psql test-persist -p 6543
CREATE TABLE Spaces (
space BIGINT PRIMARY KEY,
parent BIGINT
);
INSERT INTO Spaces VALUES (0,0); -- no space defined; raw atoms.
INSERT INTO Spaces VALUES (1,1); -- default root
ALTER TABLE atoms ADD COLUMN space INT REFERENCES spaces(space);
select count(*) from atoms;
select sum(stv_count) from atoms;
all of these are only the ANY type:
fr_pairs -- learn-fr -- 11161199 (11M) atoms, 6842302577 (6.8G) count
ANY-count = 1200368900 = 1.2GB relations
lt_pairs -- learn-lt -- 9798103 (10M) atoms, 2873920459 (2.8G) count
ANY-count = 647356630 = 0.65G relations
pl_pairs -- learn-pl -- 16192701 (16M) atoms, 4331274070 (4.3G) count
ANY-count = 1061898400 = 1G relations
simple_pairs -- learn-simple -- empty
pair_data -- 532 atoms, junk
frwiki-20131220-pages-articles
find alpha-pages |wc -- 1452754 - 1.4M
find beta-pages |wc -- 1436278
find submitted-articles |wc -- 16505
plwiki-20131218-pages-articles
find alpha-pages | wc -- 1007645 -- 1.0M
find submitted-articles |wc -- 107190
ltwiki-20131216-pages-articles
find alpha-pages | wc -- 161488 -- 161K
submitted-articles |wc -- 18108
fr_ady : mophology pairs.
.odbc.ini -> fr-ady
relex is not spewing ady correctly -- FIXED
fr-ady on 17003, ady relex on 4446
Warning: Combinatorial explosion! nulls=1 cnt=2147483647
FIXED -- hacked on LG; works-ish, but slow for long sentences.
psql -p 6543 fr_ady
select type, name,stv_count from atoms where height=0;
Argh.
after a day: guile hung; but also, no articles moved from
split to submitted!
crash: string_set_add + 0x47
000000000003ad90 t string_set_add
e0: 84 c9 test %cl,%cl
e2: 0f 84 89 01 00 00 je 271 <string_set_add+0x1b1>
e8: 48 8b 36 mov (%rsi),%rsi
eb: 49 89 f8 mov %rdi,%r8
ee: 0f b6 d1 movzbl %cl,%edx
f1: 45 31 ed xor %r13d,%r13d
f4: 0f 1f 40 00 nopl 0x0(%rax)
f8: 41 c1 e5 08 shl $0x8,%r13d
fc: 49 83 c0 01 add $0x1,%r8
100: 41 8d 44 15 00 lea 0x0(%r13,%rdx,1),%eax
105: 31 d2 xor %edx,%edx
107: 48 f7 f6 div %rsi
but RSI is zero
liblink-grammar.so.5+0x3b6b3] int+0x3
000000000003b6b0 t is_capitalizable.isra.9
PrintWriter out -- done
outs OutputStream done
in_sock Socket -- done
send_sock Socket ... is null
23:30: 2159 articles
23:43: 2535 articles : 276/13 minutes = 21 articles/min
23:48 2540 articles
23:59 2872 articles : 332 aritcles/ 11 mins = 30 articles/min! waa
00:45 3651 articles
select sum(stv_count) from atoms; 1375507899 == 1.38 G counts
select count(*) from atoms; 7146891 == 7.1M atoms
# C [liblink-grammar.so.5+0x120d5] print_flat_constituents+0xbd5
# C [liblink-grammar.so.5+0x120d5] print_flat_constituents+0xbd5
from
C [liblink-grammar.so.5+0x12a25] linkage_print_constituent_tree+0x85
4180 + bd5 = 4d55
objdump -d -F -g -S
in build tree:
0000000000011500 t print_flat_constituents
00000000000120e0 t print_tree
in build tree so:
4ccd:
1207f: 31 c9 xor %ecx,%ecx
12081: 31 c0 xor %eax,%eax
12083: e8 78 2a 01 00 callq 24b00 <feature_enabled>
12088: 48 85 c0 test %rax,%rax
1208b: 0f 85 1f ff ff ff jne 11fb0
<print_flat_constituents+0xab0>
12091: eb d6 jmp 12069
<print_flat_constituents+0xb69>
12093: 48 89 c3 mov %rax,%rbx
12096: 48 8b 00 mov (%rax),%rax
12099: 48 8b b8 98 00 00 00 mov 0x98(%rax),%rdi
120a0: e8 9b 89 01 00 callq 2aa40 <post_process_new>
120a5: 48 89 c7 mov %rax,%rdi
120a8: 48 89 bb 88 00 00 00 mov %rdi,0x88(%rbx)
120af: e9 ff f7 ff ff jmpq 118b3
<print_flat_constituents+0x3b3>
120b4: e8 47 3e ff ff callq 5f00 <__stack_chk_fail@plt>
120b9: 48 8d 3d b0 00 05 00 lea 0x500b0(%rip),%rdi # 62170 <__func__.7560+0x8f0>
120c0: 31 c0 xor %eax,%eax
120c2: e8 e9 3b ff ff callq 5cb0 <prt_error@plt>
120c7: 48 8d 3d ac bf 05 00 lea 0x5bfac(%rip),%rdi # 6e07a <_ZTSN7Minisat6SolverE+0xca>
4d22:
120ce: 31 c0 xor %eax,%eax
120d0: e8 db 3b ff ff callq 5cb0 <prt_error@plt>
>>>>>>>>>>>>>>>>>>>> crash here.
120d5: 0f 0b ud2
120d7: 48 8d 3d 0a 00 05 00 lea 0x5000a(%rip),%rdi # 620e8 <__func__.7560+0x868>
120de: eb e0 jmp 120c0 <print_flat_constituents+0xbc0>
OK
00000000000120e0 <print_tree>:
120e0: 48 85 d2 test %rdx,%rdx
120e3: 0f 84 ab 08 00 00 je 12994 <print_tree+0x8b4>
120e9: 41 57 push %r15
120eb: 41 56 push %r14
120ed: 49 89 ff mov %rdi,%r15
120f0: 41 55 push %r13
So:
blank banshee
I just wasn't made for these times -- beach boys -- pet sounds
linkage_print_constituent_tree
getConstituentString
LinkGrammar.java -- wraps the C api w/static funcs
public static native String getConstituentString();
LGService.java -- where the struture is restored
if (config.isStoreConstituentString())
finish
Java_org_linkgrammar_LinkGrammar_close(JNIEnv *env, jclass cls)
public static native void close();
LGService.do_finalize()
allowSkippedWords
setAllowSkippedWords
linkgrammar_get_dict_version+0x0 <-- crash here twice ...
parser.parse(sentence);
parseSentence 159
parser.parse(sentence);
start again yue 30 Jan 22:19
at 22:49 -- 488 articles/30 minutes = 16 articles/minute! -- 3x faster than before.
at 23:48 -- 796 articles in 90 minutes = 9 artciles/min
at 00:19 -- 926 articles in 120 mins = 7.7 articles/min
4500 sents = 4.86 sents/article
22908000 stored, stv total = 18315330
stv/5 = 3.66M pairs or 814 pairs/sent(!)
at 11:55 -- 4275 articles in 816 mins = 5.24 articles/min - back to old rates.
restart java, no noticable diference in rate: its same cpu/loadavg
restart, and load all. Before stopping:
tlbuf holds 1381888 atoms Atomspace holds 1381901 atoms Hmm.
restart: explicit load: 1381888 atoms in 31 cpu-secs: 44.6K atoms/sec load speed.
at 12:13 -- restarted. Baseline: 4304 articles
at 18:13 -- 6094-4304 = 1790/6 hrs = 4.97 articles/min
at 00:26 -- 7547-6094 = 1453/373 mins = 3.90 articles/min
at 12:22 -- 12376-7547 = 4829/716 mins = 6.74 articles/min (!)
24-hour avg: 12376 - 4304 = 8072/1449min = 5.57 articles/min
1 Feb 2017:
change postgres shared_buffers = 24576MB from 1GB previously.
also -- change java to 2GB from 1GB
restart at 12:31 no explicit load.
at 14:14 -- 13926 - 12376 = 1550/103min = 15.0 articles/min(!)
at 15:40 - 12:31 = 189 mins, have
guile=637:07 mins, or 637/189 = 3.37 load
java = 50:25 = 50/189 = 0.267 load
postgres = 10974 secs = 182.9 mins = 182.9/189 = 0.968 load
at 21:27 -- 17014 - 13926 = 3088 / 433 mins = 7.13 articles/min
at 21:30 - 12:31 = 539 mins
guile = 1850:15 / 539 = 3.42 load
java = 148:04 / 539 = 0.274 load
postgres = 519 / 539 = 0.963 load
2 Feb 2017 -- guile crashed! no prints no dump.
restart at 12:05 -- 21174 articles
at 12:43 -- 21411 articles = 411-174/38 = 237/38 = 6.2 artciles/min
at 19:28 -- 23422 articles = 23422-21174/443 = 2248/443 = 5.07 arts/min
3 Feb at 07:46 -- 26828 articles = 26828-23422/738 = 3406/738 = 4.62 arts/min
11:45 -- 27691 articles = 27691-26828/239 = 863/239 = 3.61 arts/min
Hmm. the rate keeps dropping...
final stats for above:
sql-stats: total loads = 1228941 total stores = 286453870 ratio=233.090010
write items=57290774 flushes=0 flush_ratio=inf
drains=98324 fill_fraction=582.673345 concurrency=6.286054
avg drain time=0.109933 seconds; longest drain time=9.216000
scheme@(guile-user)> (gc-stats)
$9 = ((gc-time-taken . 19242586423) (heap-size . 5229101056) (heap-free-size .
5109686272) (heap-total-allocated . 3874090716576) (heap-allocated-since-gc . 509440)
(protected-objects . 301) (gc-times . 53424))
select sum(stv_count) from atoms; -- 801969742 -- 802M
select count(*) from atoms; -- 4786479 -- 4.8M
3 Feb 2017 -- restart. All-new atomspace, all-new link-grammar
at 12:18PM -- 27691 articles
at 12:30 -- crash of guile, restart.
at 14:11 -- 28246 articles, 28246-27691/100 = 555 / 100 = 5.5 arts/min
at 15:47 -- 28772 articles, 28772-28246/96 = 526 / 96 = 5.48 articles/min
at 20:03 -- 30166 articles, 30166-28772/256 = 1394/256 = 5.45 arts/min
4 Feb at 11:20 -- 35014 articles, 35014-30166/917 = 4848/917 = 5.29 artciles/min
5 Feb at 00:02 -- 38511 articles; there was accidental pause of system for hours.
at 11:25 -- 43905 articles; 43905-38511/683 = 5394/683 = 7.90 arts/min whoa!
6 Feb completed -- 49194 articles.
sql-stats: total loads = 2013739 total stores = 741785040 ratio=368.362057
write items=148357008 flushes=0 flush_ratio=inf
drains=256330 fill_fraction=578.773487 concurrency=5.138489
avg drain time=0.100850 seconds; longest drain time=10.537000
select count(*) from atoms; 6927244 -- 6.9M -- 6 Feb final
select sum(stv_count) from atoms; 1396889746 -- 1.40G -- 6 Feb final
cpu-ratio guile/java: 4846/425 = 11.4 at 11:20 4 Feb
6732/602 = 11.2 at 00:02 5 Feb
9001/786 = 11.5 at 11:30 5 Feb
12532/1127 = 11.1 final 6 Feb at completion
114022593 total stores in 1820 cpu-mins = 62.6K atoms/min = 1.04K sto/sec
292674070 total stores in 4846 cpu-mins = 60.4K atoms/min = 1.01K sto/sec
534084730 total stores in 8978 cpu-mins = 59.5K atoms/min = 991 stores/sec
741785040 total stores in 12532 cpu-mins = 59.2K atoms/min = 987 stores/sec
vs cx1 (fr-ady):
76688189 stores in 3325 cpu-mins = 23.1K stores/min = 384 stores/sec
193457228 stores in 6838 cpu-mins = 28.3K stores/min = 472 stores/sec
378934530 stores in 12396 cpu-mins = 30.6K stores/min = 509 stores/sec
cx1 has old, inefficient kernel and/or old-ineffcient postgres (and
crappy odbc)
postgres:
ps ax |grep postgres > x
cat x |cut -f 4 > y
EPOCH='jan 1 1970'
sum=")"
for i in 00:03:34 00:00:35 00:12:34
do
sum="$(date -u -d "$EPOCH $i" +%s) $sum"
done
echo "(+" $sum| guile
=====================================================================
French pairs
cx1: what to do.
/home/linas/postgres/etc/init.d/postgresql start
psql -p 6543
psql -p 6543 fr_ady
cd /storage/wikipedia/frwiki-20131220-pages-articles
./run-all-servers.sh
french pairs: start again: 22:36PM 30 Jan - find submitted* |wc 17053
12:25PM 2 Feb - find submitted* |wc 24563
11:43AM 4 Feb - find submitted* |wc 25361
11:34AM 5 Feb - find submitted* |wc 25830
19:19PM 10 Feb - find submitted* |wc 27763
20:16PM 14 Feb - find submitted* |wc 29720
2 Jul - find submitted* |wc 32526
19 Jul - 39077
disk: 206447916 used 41097304 free
disk 206574808 used 40970412 free 24 hours later... about 100MB more
disk 206888724 used 40656496 free 12:25 2 Feb
disk 208585480 used 38959740 free 11:42 4 Feb ... 2.1GB more
disk 208897572 used 38647648 free 00:00AM 5 feb
disk 211829000 used 35716220 free 19:20 10 Feb
disk 213607532 used 33937688 free 20:16 14 Feb
-
psql -p 6543 fr_ady:
select count(*) from atoms; 2090064 -- 2.1M -- 12:25PM 2 Feb
select sum(stv_count) from atoms; 115251031 -- 115M -- 12:25PM 2 Feb
select count(*) from atoms; 5676118 -- 5.7M -- 11:19AM 4 Feb
select sum(stv_count) from atoms; 351323640 -- 351M -- 11:19AM 4 Feb
select count(*) from atoms; 7714679 -- 7.7M -- 11:38AM 5 Feb
select sum(stv_count) from atoms; 502509756 -- 503M -- 11:38AM 5 Feb
select count(*) from atoms; 16945084 -- 16.9M -- 19:21PM 10 Feb
select sum(stv_count) from atoms; 1289249056 -- 1.29G -- 19:21PM 10 Feb
select count(*) from atoms; 23072988 -- 23.1M -- 20:20PM 14 Feb
select sum(stv_count) from atoms; 1856669388 -- 1.86G -- 20:20PM 14 Feb
select count(*) from atoms; 33519967 -- 33.5M - 2 Jul 2017
select sum(stv_count) from atoms; 2923838351 -- 2.92G -
select count(*) from atoms; 54337413 -- 54.3M - 19 Jul 2017
select sum(stv_count) from atoms; 5257433115 -- 5.26G
vs fr_pairs:
select count(*) from atoms; 11161199 -- 11M
select sum(stv_count) from atoms; 6842302577 -- 6.8G !!
guile/java ratio: 6781/683 = 9.93 -- 11:19AM 4 Feb
9401/1001 = 9.40 -- 00:00AM 5 Feb
12433/1304 = 9.53 -- 11:40AM 5 Feb
47250/4576 = 10.3 -- 19:23PM 10 Feb
22980/2340 = 9.82 -- 20:20PM 14 Feb
guile hangs:
cx1 - 30 jan hang after 4.4M stores
31 jan hang after 42M stores
1 Feb hang after 33M stores -- 2000:26 cpu
1 Feb hang after 6.9M stores
2 Feb hang
2 Feb hang after 20M stores -- 865:47 cpu
155 threads
155: pthread_cond_wait - multi-driver/SQLAtomStorage.cc:325
Ahhh ... too many open sockets, and one is deadlocked.
wait ... deadlocked in conn_pool.pop()
in SQLAtomStorage::get_conn() SQLAtomStorage.cc:325
so who is using all these conns?
154: ~GenericShell 152 150 148 .. 130 .. 127
153: same as 155 151 149 147 ... 131 .. 128
alternates like this till 131 or so
129: SchemeEval::do_poll_result SchemeEval.cc:655 86
90 76: nanosleep --SQLAtomStorage::storeAtom SQLAtomStorage.cc:754
waiting to drain ... so its not draining ... why?
25: GC25: GC
24: get_conn, do_store_single --- this is the write-back handler.
there are 8 of these .. through 17
so .. the 8 busy writers are blocking readers, cause there are only 8
conns total ...
works better if we double the pool...
no one seems to be waiting on postgres ... suggests that the
we've lost track of the number items in the queue ...
15: console list
14: logger
13-2: gc
12
So, its idle ... and ... we've leaked:
currently in_drain=0 num_busy=0 queue_size=0
current conn_pool free=4 of 20
ahhh its due to throws, which kill the put_conns ...
[2017-02-03 05:44:02:785] [WARN] (34) The # of binded parameters < the #
of parameter markers;
Unrecognized key passed to SQLGetInfo30.
[2017-02-03 05:44:02:785] [WARN] Query was: SELECT * FROM Atoms WHERE
type = 110 AND name = $ocp$?$ocp$ ;
[2017-02-03 05:44:02:785] [ERROR] Can't perform query rc=-1
(/home/linas/src/atomspace-git/opencog/persist/sql/multi-driver/odbcxx.cc:259)
Stack Trace:
Query was: SELECT * FROM Atoms WHERE type = 110 AND name =
$ocp$?,!ANY-PUNCT$ocp$ ;
... need to try-catch these blocks.
SELECT * FROM Atoms WHERE type = 110 AND name = $ocp$?$ocp$ ;
Seems to be an ODBC issue ...
(34) The # of binded parameters < the # of parameter markers;
Unrecognized key passed to SQLGetInfo30.
SQLBindParameter
glurg. ODBC needs this: replace ? by ? what a clusterfuck
Also:
[2017-02-08 08:16:43:154] [WARN] ODBC Driver: (1) ODBC escape convert
error
[2017-02-08 08:16:43:155] [WARN] ODCB Driver: Query was: SELECT * FROM
Atoms WHERE type = 110 AND name = $ocp${x$ocp$ ;
[2017-02-08 08:16:43:155] [ERROR] Can't perform query rc=-1
(/home/linas/src/atomspace-git/opencog/persist/sql/multi-driver/odbcxx.cc:340)
------------------
very late:
link-grammar: Info: Dictionary 'ady/4.0.dict': No locale definition -
"en_US.UTF-8" will be used.
link-grammar: Info: Dictionary 'ady/4.0.dict': No locale definition -
"en_US.UTF-8" will be used.
link-grammar: Error: Could not open dictionary "ady/4.0.dict"
restart not happen until connection... OK.
link-grammar not loaded until the next round!
re.setMaxParses
_is_in
dict=0x7f053c1bb0b0
dict=0x7f2e2c1f35a0
local-ptd=0x7f2e2c1f3550
checking for locale_t in locale.h... no
checking for locale_t in xlocale.h... no
__USE_GNU
semant = new AlgorithmApplier(
"relex.semalgpath", "relex-semantic.algs");
wordnet.configfile file_properties.xml
src/java/relex/morphy/MapMorphy.java:
src/java/relex/morphy/MorphyFactory.java:
getJWNLConfigFileStream(
WORDNET_PROPERTY,
JWNL_FILE_PROPERTIES_XML,
JWNL_DIR_PROPERTIES_XML
)
Info: hndlr=3 recv input: "- |3-1|25-23|20-25|25-17|25-11||95-76|couleur=gris}} - |3-0|28-26|25-19|25-23|||78-68}} - |3-1|25-21|25-21|13-25|25-12||88-79|couleur=gris}} - |3-1|21-25|33-31|40537|25-15||104-83}} - |3-2|25-23|22-25|25-22|17-25|17-15|106-110|couleur=gris}} - |3-0|25-19|25-13|25-23|||75-55}}"
Info: hndlr=8 recv input: "- |3-0|25-16|26-24|25-17|||76-57|couleur=gris}} - |3-1|25-22|23-25|25-20|25-20||98-87}} - |3-2|25-22|20-25|25-27|33-31|15-12|118-117|couleur=gris}} - |3-0|25-14|25-22|25-19|||75-55}} - |3-0|25-14|25-20|25-21|||75-55|couleur=gris}} - |3-0|26-24|25-18|33-31|||84-73}}"
then crash
"ISBN 0-380-72001-9. http://books.google.be/books?id=FMsNAG-gresC&pg=PA144&lpg=PA144&dq=vlakfontein+Dixon+kemp&source=web&ots=8cB7anqlpV&sig=REOVNA8sQO8iTZhI-0_IHVdhySQ&hl=en&sa=X&oi=book_result&resnum=3&ct=result."
crash again:
C [libc.so.6+0x7a97b] unsigned long+0xa5b
C [liblink-grammar.so.5+0x3dad4] short+0x164
3dad4 - 164 = 3d970
000000000003d970 t separate_word.isra.12
if (not dict->affix_table ) jump pst anysplit
3dab8: 74 1a je 3dad4 <separate_word.isra.12+0x164>
3daba: 48 83 78 58 00 cmpq $0x0,0x58(%rax)
if (not dict->affix_table->anysplit) jump past anysplit
3dabf: 74 13 je 3dad4 <separate_word.isra.12+0x164>
3dac1: 48 8b b5 30 ff ff ff mov -0xd0(%rbp),%rsi # arg2: unsplit_word
3dac8: 48 8b bd 20 ff ff ff mov -0xe0(%rbp),%rdi # arg1: sent
3dacf: e8 4c a9 fc ff callq 8420 <anysplit>
3dad4: 48 8b 95 70 ff ff ff mov -0x90(%rbp),%rdx # arg3: word
crash here
RBP=0x00002b9ecaf2e450, RDX=0x0000000000000000
call convention arg 1 2 3 in RDI RSI RDX
looks like RBP is pointing into the stack... and 0x00002b9ecaf2e3c0 is
null ...
0x00002b9ecaf2e3a8: 00002b9eeef04f10 000000004bb2c580
0x00002b9ecaf2e3b8: 00002b9e00000000 0000000000000000
0x00002b9ecaf2e3c8: 0000000000000000 00002b9eec065490
0x00002b9ecaf2e3d8: fffffffe00000026 00002b9ecaf2e310
0x00002b9ecaf2e3e8: 0000000000000001 00002b9e00737470
OK, this is fine, the crash is somewhere inside of anysplit.
unsplit word is.... at 0x00002b9ecaf2e450 - d0 = ...380
0x00002b9ecaf2e380: 000000004bb2c960 -- same as contents to rbx
3dadb: 48 8b b5 30 ff ff ff mov -0xd0(%rbp),%rsi # arg2: unsplit_word
3dae2: 48 8b bd 20 ff ff ff mov -0xe0(%rbp),%rdi # arg1: sent
3dae9: e8 a2 f8 ff ff callq 3d390 <morpheme_split>
3daee: 80 bd 1f ff ff ff 00 cmpb $0x0,-0xe1(%rbp)
3daf5: 89 c3 mov %eax,%ebx
3daf7: 75 2e jne 3db27 <separate_word.isra.12+0x1b7>
3daf9: 84 c0 test %al,%al
==================================
again
https://sourceware.org/bugzilla/show_bug.cgi?id=21104
Info: hndlr=1 sentence: "Chaussande (de) [http://gallica.bnf.fr/ark:/12148/bpt6k1120033/f177.image.r=], baron par bref pontifical de 1707, Comtat Venaissin, ANF-1969."
*** glibc detected *** java: free(): invalid pointer: 0x0000000059551940
***
======= Backtrace: =========
/lib64/libc.so.6[0x3e296714af]
/lib64/libc.so.6(cfree+0x4b)[0x3e296757ab]
/usr/local/lib/liblink-grammar.so.5[0x2b3a2392a7ce]
/usr/local/lib/liblink-grammar.so.5[0x2b3a2392b396]
/usr/local/lib/liblink-grammar.so.5(sentence_parse+0x99)[0x2b3a2392bb69]
/usr/local/lib/liblink-grammar-java.so.5.3.15(Java_org_linkgrammar_LinkGrammar_parse+0xf5)[0x2b3a236f8995]
[0x2b3a1690eeee]
#5 0x00002b3a2392a7ce in wordgraph_path_free (free_final_path=true, wp=0x59551860)
at ../../link-grammar/api.c:937
#6 sane_linkage_morphism (sent=sent@entry=0x586bb130, lkg=lkg@entry=0x5877b850,
opts=opts@entry=0x586baef0) at ../../link-grammar/api.c:1021
(gdb) print twp
$1 = (Wordgraph_pathpos *) 0x59551890
(gdb) print *twp
$3 = {word = 0x587d7f40, same_word = 56, next_ok = 74, used = 149, path = 0x59551940}
(gdb) x/16g 0x59551940
0x59551940: 0x00000000587d6ef0 0x0000000000000000
0x59551950: 0x0000000000001270 0x0000000000000021
0x59551960: 0x000000005894f650 0x0000003e29954a68
0x59551970: 0x0000000000000020 0x0000000000000040
0x59551980: 0x00000000595516f0 0x000000005a5c82a0
Hmm everything looks OK, so what's the problem???
struct Wordgraph_pathpos_s
{
Gword *word; /* Position in the Wordgraph */
/* Only for wordgraph_flatten(). */
bool same_word; /* Still the same word - issue an empty word */
bool next_ok; /* OK to proceed to the next Wordgraph word */
bool used; /* Debug - the word has been issued */
/* Only for sane_morphism(). */
const Gword **path; /* Linkage candidate wordgraph path */
};
wordgraph_pathpos_resize could take bigger jumps.
wordgraph_pathpos_len
len == len not counting the null.
e.g. n=1 insert = 0
memomve from 0 to 1 for 1 item
=========================================================
Arghh...
scheme@(guile-user)> GC Warning: Repeated allocation of very large block (appr. size
18894848):
May lead to memory leak and poor performance.
*** Error in `guile': double free or corruption (out): 0x00000000623e6010 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f3a6db117e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x7fe0a)[0x7f3a6db19e0a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f3a6db1d98c]
/usr/local/lib/libguile-2.2.so.0(scm_string_to_utf8+0xd7)[0x7f3a6e0ca067]
/usr/local/lib/libguile-2.2.so.0(scm_mkstrport+0x6b)[0x7f3a6e13d0ab]
/usr/local/lib/libguile-2.2.so.0(+0xbc5cd)[0x7f3a6e1465cd]
/usr/local/lib/libguile-2.2.so.0(scm_call_n+0x186)[0x7f3a6e154046]
/usr/local/lib/libguile-2.2.so.0(scm_call_3+0x2f)[0x7f3a6e0d2eaf]
/usr/local/lib/libguile-2.2.so.0(+0xbc5cd)[0x7f3a6e1465cd]
/usr/local/lib/libguile-2.2.so.0(scm_call_n+0x186)[0x7f3a6e154046]
/usr/local/lib/libguile-2.2.so.0(scm_call_3+0x2f)[0x7f3a6e0d2eaf]
/usr/local/lib/libguile-2.2.so.0(+0xbc5cd)[0x7f3a6e1465cd]
/usr/local/lib/libguile-2.2.so.0(scm_call_n+0x186)[0x7f3a6e154046]
/usr/local/lib/libguile-2.2.so.0(+0xb7fd2)[0x7f3a6e141fd2]
/usr/local/lib/opencog/libsmob.so(_ZN7opencog10SchemeEval7do_evalERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0xb9)[0x7f3a6539be09]
/usr/local/lib/opencog/libsmob.so(_ZN7opencog10SchemeEval11c_wrap_evalEPv+0x1a)[0x7f3a6539beba]
/usr/local/lib/libguile-2.2.so.0(+0x4324a)[0x7f3a6e0cd24a]
/usr/local/lib/libguile-2.2.so.0(+0xbc5cd)[0x7f3a6e1465cd]
/usr/local/lib/libguile-2.2.so.0(scm_call_n+0x186)[0x7f3a6e154046]
/usr/local/lib/libguile-2.2.so.0(+0xb7fd2)[0x7f3a6e141fd2]
/usr/local/lib/libguile-2.2.so.0(+0x43860)[0x7f3a6e0cd860]
/usr/local/lib/libguile-2.2.so.0(scm_c_with_continuation_barrier+0x45)[0x7f3a6e0cd945]
/usr/lib/x86_64-linux-gnu/libgc.so.1(GC_call_with_gc_active+0x77)[0x7f3a6d8438e7]
/usr/local/lib/libguile-2.2.so.0(+0xb6ad1)[0x7f3a6e140ad1]
/usr/lib/x86_64-linux-gnu/libgc.so.1(GC_call_with_stack_base+0x22)[0x7f3a6d83d952]
/usr/local/lib/libguile-2.2.so.0(scm_with_guile+0x38)[0x7f3a6e140e88]
/usr/local/lib/opencog/libsmob.so(_ZN7opencog10SchemeEval9eval_exprERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x2e)[0x7f3a6539bf3e]
/usr/local/lib/opencog/modules/libscheme-shell.so(_ZN7opencog12GenericShell9eval_loopEv+0x2ec)[0x7f3a582ea9cc]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80)[0x7f3a63f4ac80]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f3a6de716ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f3a6dba082d]
======= Memory map: ========
00400000-00401000 r-xp 00000000 fd:00 80300005 /usr/local/bin/guile
00600000-00601000 r--p 00000000 fd:00 80300005 /usr/local/bin/guile
00601000-00602000 rw-p 00001000 fd:00 80300005 /usr/local/bin/guile
01d52000-6258b000 rw-p 00000000 00:00 0 [heap]
7f3870000000-7f38764e3000 rw-p 00000000 00:00 0
7f38764e3000-7f3878000000 ---p 00000000 00:00 0
7f3878000000-7f387c000000 rw-p 00000000 00:00 0
7f387d7bb000-7f387d7bc000 ---p 00000000 00:00 0
7f387d7bc000-7f387dfbc000 rw-p 00000000 00:00 0
7f387dfbc000-7f387dfbd000 ---p 00000000 00:00 0
7f387dfbd000-7f387e7bd000 rw-p 00000000 00:00 0
7f387e7bd000-7f387e7be000 ---p 00000000 00:00 0
7f387e7be000-7f387efbe000 rw-p 00000000 00:00 0
7f387efbe000-7f387efbf000 ---p 00000000 00:00 0
7f387efbf000-7f387f7bf000 rw-p 00000000 00:00 0
7f387f7bf000-7f387f7c0000 ---p 00000000 00:00 0
7f387f7c0000-7f387ffc0000 rw-p 00000000 00:00 0
7f387ffc0000-7f387ffc1000 ---p 00000000 00:00 0
7f387ffc1000-7f38807c1000 rw-p 00000000 00:00 0
7f38807c1000-7f38807c2000 ---p 00000000 00:00 0
7f38807c2000-7f3880fc2000 rw-p 00000000 00:00 0
7f3880fc2000-7f3880fc3000 ---p 00000000 00:00 0
7f3880fc3000-7f38817c3000 rw-p 00000000 00:00 0
7f38817c3000-7f38817c4000 ---p 00000000 00:00 0
7f38817c4000-7f3881fc4000 rw-p 00000000 00:00 0
12 Feb 2017
-----------------
postgres: change shared_buffers = 64576MB # min 128kB
/etc/postgresql/9.5/main/postgresql.conf
apt-get update
apt-get upgrade
apt-get install autoconf-archive autoconf automake
apt-get install libgmp-dev libgc-dev libunistring-dev libffi-dev
apt-get install flex gettext texinfo
git clone git://git.sv.gnu.org/guile.git
cd guile; ./autogen.sh --no-configure;
mkdir build; cd build; ../configure; make -j
apt-get install libpq-dev
apt-get purge unixodbc-dev
apt-get autoremove
install link-grammar-5.3.15
cd relex; git pull; ant; ant install
update pair-count.scm to use URL
(sql-open "postgres:///yue_pairs?user=ubuntu&password=asdf")
lynx https://dumps.wikimedia.org/zhwiki/20170101/zhwiki-20170101-pages-articles.xml.bz2
time bzcat zhwiki-20170101-pages-articles.xml.bz2 |/home/ubuntu/src/relex/src/perl/wiki-scrub.pl
real 39m9.964s
user 39m15.964s
sys 1m14.940s
find wiki-stripped/* |wc 1211823
cd wiki-stripped
/home/ubuntu/src/relex/src/perl/wiki-clean-zh.sh
find |wc 1018923
cd ..
Portal:
Draft:
模块:
mkdir alpha-pages
cd alpha-pages
~/src/relex/src/perl/wiki-alpha-zh.sh
find |wc 1018946
cd ../..
cp -pr alpha-data/alpha-pages beta-pages
createdb zh_pairs
cat src/atomspace/opencog/persist/sql/multi-driver/atom.sql | psql zh_pairs
cp ~/src/relex/src/split-sentences/nonbreaking_prefixes/* ~/run/nonbreaking_prefixes
cp ~/src/relex/src/split-sentences/split-sentences.pl ~/run
.... etc.
===============================================================
Again:
sudo cgm create all linas
sudo cgm chown all linas $(id -u) $(id -g)
cgm movepid all linas $$
14 April status:
---------------
ZH:
select count(*) from atoms; 18941255
14 April 2017
-------------
Start from scratch with English.
time lxc-copy -n opencog-learn -N learn-en
real 6m43.362s
wget https://dumps.wikimedia.org/enwiki/20170320/enwiki-20170320-pages-articles-multistream.xml.bz2
2 hours 2 mins
time cat enwiki-20170320-pages-articles-multistream.xml.bz2 |bunzip2
|/home/ubuntu/src/relex/src/perl/wiki-count.pl
Counted 955001762 lines in 17398694 pages -- 17.4M articles
real 65m31.329s
time cat enwiki-20170320-pages-articles-multistream.xml.bz2 |bunzip2 |/home/ubuntu/src/relex/src/perl/wiki-scrub.pl
real 604m9.767s
user 413m17.996s
sys 13m8.680s
find |wc gives 7495026 total articles
real 267m1.295s
find |wc gives 5423537 after cat/template removal
createdb en_pairs
cat src/atomspace/opencog/persist/sql/multi-driver/atom.sql | psql en_pairs
----------------
Arghh more bugs:
* failure in LG to split punctuation
* failure to strip regex markup.
linkage_get_word
UBSCRIPT_MARK
SUBSCRIPT_DOT
linkage_print_diagram
DISPLAY_GUESS_MARKS
HIDE_MORPHO for any language.... except its set to rue for jni
where are the square bracekts removed??????
orig_str in java....
feature/LinkableView.java line 286
LocalLGParser.java circa line 150
-----------------------------
yue:
17006 -- 21876 submitted 27342 left
en: java crashed:
/home/ubuntu/run/hs_err_pid3206.log
# C [liblink-grammar.so.5+0x2aa00] linkage_set_domain_names+0xe0
-------------------------
en_pairs from 2016:
select count(*) from atoms; 18487291 atoms. == 18.5M
select sum(stv_count) from atoms; 1775388151 = 1.77 B
select sum(floatvalue[3]) from valuations where type=177; CountTru
select count(*) from atoms where type =73; 396255 this is words
select count(*) from atoms where type =47; 9045489 this is pairs
select count(*) from atoms where type =8; 9045489 this is list links
select count(*) from valuations where type=121; -- FloatValue
select * from valuations where type != 176 and type != 177 limit 50;
select count(*) from valuations where type != 176 and type != 177;
select count(*) from valuations where key =2440463582; -- 9672428
select count(*) from valuations where key =2440463583; -- 8880904
(use-modules (opencog) (opencog persist) (opencog persist-sql))
(use-modules (opencog nlp) (opencog nlp learn))
(sql-open "postgres:///en_pairs?user=linas")
(batch-all-pairs)
(load "compute-mi.scm")
(init-trace "/tmp/progress")
702
(load-atoms-of-type item-type)
(sql-stats)
; hangs... because it is walking the TLB which has 2.4GB of uuid
_store->print_stats();
(length (cog-get-atoms item-type))
(define lgr (LinkGrammarRelationshipNode "ANY"))
(fetch-incoming-set lgr)
; there aren't any yet....
(define word (car (cog-get-atoms item-type)))
(compute-pair-wildcard-counts word lgr)
delete all-pair-wildcard-counts
(par-for-each
(lambda (word)
(compute-pair-wildcard-counts word lgr)
(trace-msg-cnt "Wildcard-count did ")) al)
(cog-get-atoms item-type)
)
(define al (take (cog-get-atoms item-type) 1000))
scheme@(guile-user) [1]> ,bt
In ice-9/threads.scm:
289:22 5 (loop _)
In ice-9/futures.scm:
259:13 4 (touch #<future 55c59c4adb40 queued #<partial-continuation
55c5…>)
243:14 3 (work)
In unknown file:
2 (wait-condition-variable #t)
While executing meta-command:
ERROR: In procedure cdr: Wrong type argument in position 1 (expecting
pair): ()
(use-modules (srfi srfi-1))
(define al (list-tabulate 100000 values))
(define foo 0)
(par-for-each (lambda (x) (set! foo (+ x foo))) al)
^CERROR: In procedure scm-error:
ERROR: User interrupt
Entering a new prompt. Type `,bt' for a backtrace or `,q' to continue.
scheme@(guile-user) [1]> ,bt
In ice-9/threads.scm:
289:22 5 (loop _)
In ice-9/futures.scm:
265:11 4 (touch #<future 55e1984d3400 started #<procedure 55e198659f60 a…>)
243:14 3 (work)
In unknown file:
2 (wait-condition-variable #t)
While executing meta-command:
ERROR: In procedure cdr: Wrong type argument in position 1 (expecting pair): ()
scheme@(guile-user) [1]>
bug#26616: Acknowledgement (guile-2.2 par-for-each hangs for large lists)
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=26616
-------
(cog-new-value type list) string float link
(cog-set-value! atom key value)
(cog-value atom key)
(cog-value?
cog-value->list
cog-tv-count
confidence
---------
(use-modules (opencog) (opencog persist) (opencog persist-sql))
(use-modules (opencog nlp) (opencog nlp learn))
(sql-open "postgres:///en_pairs?user=linas")
(load-atoms-of-type 'WordNode)
(define lgr (LinkGrammarRelationshipNode "ANY"))
(fetch-incoming-set lgr)
(cog-incoming-set lgr)
(define word (car (cog-get-atoms 'WordNode)))
(fetch-incoming-set word)
(cog-incoming-set word)
(define lpr (car (cog-incoming-set word)))
(fetch-incoming-set lpr)
(length (cog-incoming-set lgr))
scheme@(guile-user)> (length (cog-incoming-set lgr))
Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS
Aborted
(load-atoms-of-type 'LinkGrammarRelationshipNode)
(fetch-atom lg_rel)
fetch_incoming_set_cb
SQLAtomStorage::getIncomingSet
tlb get uuid
getNod
h = as->fetch_atom(h);
Handle h(doGetNode(t, str));
Handle hg(doGetLink(t, hs));
getUUID INVALID_UUID
select * from atoms where type = 89;
en_pairs=> select * from atoms where type = 89;
uuid | space | type | height | name | outgoing
------------+-------+------+--------+------+----------
152 | 1 | 89 | 0 | ANY |
2439209701 | 1 | 89 | 0 | ANY |
2439209706 | 1 | 89 | 0 | ANY |
2439209711 | 1 | 89 | 0 | ANY |
2439209716 | 1 | 89 | 0 | ANY |
(5 rows)
OK, so that's a bug .. no unique constraint
8: SQLAtomStorage.cc:324
opencog::SQLAtomStorage::Response::get_all_values_cb()
"SELECT * FROM Valuations WHERE atom =
SELECT * FROM Valuations WHERE atom = 152;
1 | 152 | 177 | {1,0,402733330} | |
loading_typemap
(cog-new-ctv 0 0 right-total
set-count vs ctv, stv, confidence, count etc.
AnyN
any-left
any-right
mst-parser.scm
Going to batch-logli wildcards
Elapsed secs 53
Going to do individual word-pair mi
Noting for pair (When *) -- no frequency!
no frequency's anywhere...
(PredicateNode \"*-FrequencyKey-*\") was never stored!
select * from atoms where type=53; -- the predicateNodes
wtf line 646 ... sote the keyy....
Wildcard-count did 396255 compute-pair-wildcard-counts -- stores count
Elapsed secs 11
Done with wild-card count
Elapsed secs 0
Going to batch-count all-pairs
Start all-pair-count
Done with all-pair count -- again a count...
Elapsed secs 130
Going to batch-logli wildcards -- batch-all-pair-wildcard-logli
compute-atom-logli
Elapsed secs 53
Going to do individual word-pair mi
(define k (PredicateNode "*-FrequencyKey-*"))
(define w (WordNode "When"))
(define e (EvaluationLink
(LinkGrammarRelationshipNode "ANY")
(ListLink (WordNode "When") (AnyNode "right-word"))))
(cog-value e k)
well ... e has no counts ... and thus no logli
(length (cog-incoming-set (WordNode "When") ))
except there are lots so wtf?
(define lg_rel (LinkGrammarRelationshipNode "ANY"))
(define (get-total-atom-count atom-list)
(fold + 0.0 (map get-count atom-list))
(compute-pair-wildcard-counts w lg_rel
... there are no counts on any e's ... wtf ??
(EvaluationLink ; uuid 2434162536
(LinkGrammarRelationshipNode "ANY") ; uuid 152
(ListLink ; uuid 2434162535
(WordNode "Hear") ; uuid 17490475
(WordNode "When") ; uuiid 1537161
)
)
incoming set failed to load the values!!!
also cog-value is real slowww
// Get the correct UUID; its possible that we don't know it yet.
UUID uuid = _tlbuf.getUUID(h);
if (TLB::INVALID_UUID == uuid)
{
Handle hg(doGetAtom(h));
uuid = _tlbuf.getUUID(hg);
}
get_uuid
_vindex
_keyset
std::unordered_multimap<ContentHash, Handle>
std::unordered_map<
#include <unordered_map>
std::unordered_set<Handle, handle_hash> -- kill this -- done
HandlePair
-----------------------------------------
(use-modules (opencog) (opencog persist) (opencog persist-sql))
(use-modules (opencog nlp) (opencog nlp learn))
(sql-open "postgres:///en_pairs?user=linas")
(use-modules (opencog cogserver))
(start-cogserver)
(batch-all-pairs)
(define lg_rel (LinkGrammarRelationshipNode "ANY"))
(init-trace "/tmp/progress")
(load "common.scm")
(load "compute-mi.scm")
(load-atoms-of-type item-type)
(length (cog-get-atoms item-type))
(fetch-incoming-set lg_rel)
(compute-pair-wildcard-counts w lg_rel)
(EvaluationLink (ctv 0 0 90720) ; uuid = 143501595
(LinkGrammarRelationshipNode "ANY") ; uuid = 152
(ListLink ; uuid = 143501594
(AnyNode "left-word") ; uuid = 143427138
(WordNode "When") ; uuid = 1537161
)
)
elect * from valuations where atom=143501595;
(store-atom left-star) took forever...
WordNode load took 36s, 38s
9112060 in incoming set
serial inc fetch= 28m11 , 28m03
serial inc fetch w/GIN idx= 28m39
parallel inc fecth w/GIN idex = 38s, 28s, 29s
parallel inc fetch no idx = 32s, 30s, 33s
serial value fetch= 24m30
parallel value fetch= 1m36, 1m35
parallel resurse+value= 5m16, 6m47(!?), 6m47, 6m50, 7m6
atomspace add = 8m51 = 531 secs, 9m02, 7m56, 7m55, 8m00, 8m21
atsomapce add: added 9112060 atoms in 8m51 = 531 secs = 17.1K/sec
well, each atom is actually 5 atoms so this is 5x faster = 86K/sec
do a parallel type fetch...
as->fetch_all_atoms_of_type(t);
_backing_store->loadType(_atom_table, t);
o_store_single_ato
add_id_to_cache(uuid)
puts it in local_id_cache
erases from id_create_cach
maybe_create_id(uuid)
where is check for existance?????
well, get_uuid() does checking()
soo iif in tlb then works
rp.id_set ...!?
get_ids get all id's in
do_store_single_atom
do_store_atom
_write_q
does tlbuf???
I don'd see it being tlbuff....
UUID uuid = _tlbuf.getUUID(h);
if (TLB::INVALID_UUID != uuid) return uuid;
+CREATE INDEX ON Atoms USING GIN(outgoing);
TODO:
parallelize get_ids...
actually get_ids is not needed.....
XXX need to store the keys
do_store_atom
total stroes is miscounted!aactually, its right, values don't count.
num_get_nodes=3 num_got_nodes=1 (33.333333 pct)
num_get_links=0 num_got_links=0 (-nan pct)
sql-stats: total loads = 19741116 total stores = 2 ratio=0.000000
get_recursive is not counting what it got.
values and valuations are not counting....
Also: type load doesn't ...? grab the TV's!? FIXED
type load needs to be conservative with recursion! DONE
do-store_atom stores values recursively. It should have a flag for that.
Documented....
Also: ato->setValue() does not work for raw atoms! so incoming is
broken.
RuntimeException
move printing exceptiioon ot guile
fetch_incoming_set false
MultiThreadUTest
tv-conf
cog-tv
14:37
(use-modules (opencog) (opencog persist) (opencog persist-sql))
(use-modules (opencog nlp) (opencog nlp learn))
(sql-open "postgres:///en_pairs?user=linas")
(load-atoms-of-type 'WordNode) ;; approx 1 minutes?
(define lg_any (LinkGrammarRelationshipNode "ANY"))
(fetch-incoming-set lg_any) ;; approx 15 minutes ?
;
(mst-parse-text "this is a test")
_tlbuf.addAtom(node, p->uuid); can replace!?
line 1427
line 378
start-cost-pair is bogus!
and subsequent crash
(define e (EvaluationLink
(LinkGrammarRelationshipNode "ANY")
(ListLink (WordNode "this") (WordNode "test"))))
(cog-value e mi-key)
select * from atoms where name='this'; -- 953769
test -- 1712336
select * from atoms where outgoing='{953769,1712336}'; -- 29138136
select * from atoms where outgoing='{152,29138136}'; -- 29138137
select * from valuations where atom=29138137;
all count-tvs being stored as length 2...
fetch-atom aint getting the tv.
store-atom crashed. ... cause type=-1
store after fetch truncates
ctv print is fuckec
crap. Are teh datbases corrupt???
also .. reload the typemap
type_map_was_loaded = false
nameserver().getNumberOfClasses();
INSERT INTO Valuations
(SELECT 1 AS key, -- 1 here is the predicate node
uuid AS atom,
176 as type,
ARRAY[stv_mean, stv_confidence] as floatvalue
FROM Atoms_Backup WHERE tv_type = 1);
INSERT INTO Valuations
(SELECT 1 AS key, -- 1 here is the predicate node
uuid AS atom,
177 as type,
ARRAY[stv_mean, stv_confidence, stv_count] as floatvalue
FROM Atoms_Backup WHERE tv_type = 2);
wait .. link pipeline!???
observ-ext ... load???
the -FrequencyKey- shows {0,Infinity} ... wtf. MI is nan!!
Maybe 4 hours
(use-modules (opencog) (opencog persist) (opencog persist-sql))
(use-modules (opencog nlp) (opencog nlp learn))
(sql-open "postgres:///en_pairs?user=linas")
(load-atoms-of-type 'WordNode) ;; approx 3 minutes?
(define lg_any (LinkGrammarRelationshipNode "ANY"))
(fetch-incoming-set lg_any) ;; approx 15 minutes ?
;
(mst-parse-text "this is a test") ; OK
-----------------
Ben,
Just got out of surgery for my broken leg; this email attempts to prove
that the general anesthesia didn't kill too many brain cells. Its a report
on some the langauge-learning results.
Lets dive in.
(mst-parse-text "this is a test")
Raw hard-to-read result below. Easier-to-read versions later.
ctv holds the raw count of how many times the word was observed.
((2.862118287646645 ((2 (WordNode "is" (ctv 1 0 8165736))) (3 (WordNode "a" (ctv 1 0 14691104))
))) (2.1880378875282904 ((1 (WordNode "this" (ctv 1 0 1300681))) (2 (WordNode "is" (ctv 1 0 8165736))
))) (2.8103625339100944 ((1 (WordNode "this" (ctv 1 0 1300681))) (4 (WordNode "test" (ctv 1 0 60328))
))))
The floating point number above and below is the Yuret MI of the word pair.
I've ameneded
https://github.com/opencog/opencog/tree/master/opencog/nlp/learn/learn-lang-diary/learn-lang-diary.pdf
pages 2-5 so that its less confusing and the formulas are accurate.
Basically, it derives Yuret's formulas in a more rigorous
way; if I recall, his argument was scattered, and just asserted the
result without deriving it. So the PDF derives it.
((2.862118287646645 ((2 (WordNode "is" )) (3 (WordNode "a" ))))
(2.1880378875282904 ((1 (WordNode "this" )) (2 (WordNode "is" ))))
(2.8103625339100944 ((1 (WordNode "this" )) (4 (WordNode "test" )))))
Simplifying further:
(((2 (WordNode "is" )) (3 (WordNode "a" )))
((1 (WordNode "this" )) (2 (WordNode "is" )))
((1 (WordNode "this" )) (4 (WordNode "test" ))))
The integer is the ordinal of the word. Note that the linkage
"is-a" was selected over "a test" -- that's because "a test" has an
MI of 2.0935. This is not terribly surprising; any MI of less than
four is pretty crappy, and these four words occur so commonly that
the correlation between them really is quite qeak -- they're alomost
drowning in noise. Extracting disjuncts should strongly sharpen the
results. Next email.
Here's a "better" example:
(mst-parse-text "cats eat cat food")
$5 = ((7.329 ((2 (WordNode "eat" (ctv 20938))) (4 (WordNode "food" (ctv 73924)))))
(4.992 ((3 (WordNode "cat" (ctv 18408))) (4 (WordNode "food" (ctv 73924)))))
(-1000 ((1 (WordNode "cats" (ctv 5902))) (4 (WordNode "food" (ctv 73924))))))
So "eat food" has a decent MI, as expected. Also "cat food" is decent.
The minus-1000 means that the word pair "cats food" was never observed.
(get-pair-mi-str "cats" "eat") = -1000 means that "cats eat" was never
observed! Bummer! The word "cats" was observed 6000 times, and this was
not enough to discover a sentence that has "cats eat" in it. These
statistics are from some relatively smallish sample of WP articles, so
lack of such a sentence is maybe not surprising. Here, childrens &
young-adult lit may be better.
Anyway, clustering that reveals that cats, dogs, etc are similar should
help with this, or so goes the hypothesis.
The word-pair "cats cat" does occur and has an MI of 5 but is prevented
from linking by the link-crossing constraint. Have not attempted to
figure out if the Dick Hudson landmark transitivity idea can be mutated
to apply to this situation. I suppose I should think about things before
writing about them, but not thinking is faster.
Lets try again:
(mst-parse-text "dogs eat dog food")
((7.329 ((2 (WordNode "eat" (ctv 20938))) (4 (WordNode "food" (ctv 73924)))))
(7.047 ((3 (WordNode "dog" (ctv 41896))) (4 (WordNode "food" (ctv 73924)))))
(5.050 ((1 (WordNode "dogs" (ctv 14852))) (2 (WordNode "eat" (ctv 20938))
))))
Well, that's much better. Let's try something harder:
(mst-parse-text "It is not uncommon to discover strange things")
((7.515 ( (WordNode "not" ) (WordNode "uncommon" )))
(4.142 ( (WordNode "is" ) (WordNode "uncommon" )))
(4.412 ( (WordNode "It" ) (WordNode "is" )))
(2.739 ( (WordNode "uncommon" ) (WordNode "to" )))
(3.529 ( (WordNode "to" ) (WordNode "discover" )))
(0.822 ( (WordNode "to" ) (WordNode "things" )))
(6.171 ( (WordNode "strange" ) (WordNode "things" ))))
Almost right -- the stinker in there is "to things" and it has a
terrible MI. The correct link would have been "discover things"
but again, this word-pair was never ever observed.
That's it for now, more later.
p.s. The above is obtained with code that uses values in full
generality; so for example the normalized word frequency is stored as
(Valuation
(WordNode "foo")
(PredicateNode "*-Frequency Key-*")
(FloatValue 0.1234567 3.018))
Note that "Valuation" is like an EvaluationLink but different.
The first number is the normalized frequency of observation N(foo) / N(all words)
and the second number of log-base-2 of the first number (its easier to
read, than counting zeros in a frequency).
I had to fix a dozen bugs in brand-new SQL backend code to get this to
work right. It all seems stable, now.
==========================================
(EvaluationLink
(LinkGrammarRelationshipNode "ANY")
(ListLink
(AnyNode "left-word")
(AnyNode "right-word")))
fooo failed to store counts for lw, rw...
Hmm
total word pair counts = 4.1823518e+08
(define-public (count-all-words)
(define total-wordcnt 0)
(for-each
(lambda (word)
(set! total-wordcnt (+ total-wordcnt (get-count word))))
(cog-get-atoms item-type))
total-wordcnt
)
836420450.0
342:48 342:50
(define-public (count-all-words)
(get-total-atom-count (cog-get-atoms item-type))
)
(get-count (WordNode "the"))
$28 = 41658552.0
(define (mkhist)
(define nbins 100)
(define lo -0.7)
(define hi 0.7)
(define inc (/ nbins (- hi lo)))
(define hist (make-array 0 nbins))
(define (rat w) (/ (- (get-left-count-str w) (get-right-count-str w)) (get-word-count-str w)))
(define x lo)
(for-each (lambda (w)
(define bin (round (* inc (- (rat (cog-name w)) lo))))
(if (nan? bin) (set! bin 0))
(set! bin (inexact->exact bin))
(if (>= bin nbins) (set! bin (- nbins 1)))
(if (< bin 0) (set! bin 0))
(array-set! hist (+ 1 (array-ref hist bin)) bin))
(cog-get-atoms 'WordNode))
(array-for-each (lambda (cnt)
(format #t "~A ~A\n" x cnt)
(set! x (+ x (/ 1.0 inc))))
hist)
)
=============================================
(cog-incoming-by-type
(LinkGrammarRelationshipNode "ANY")
'EvaluationLink)
get_left_wildcard_count word lg_rel)
(get-total-atom-count (get-all-pairs))
(define (wtf)
(for-each
(lambda (pair)
(if (> 0.5 (get-count (get-left-word-of-pair pair)))
(format #t "wtf left of pair ~A" pair))
(if (> 0.5 (get-count (get-right-word-of-pair pair)))
(format #t "wtf right of pair ~A" pair))
)
(get-all-pairs)))
418235182.0
(get-total-atom-count (get-all-words))
836420450.0
(length (get-all-pairs))
(LgWordCset
(WordInstanceNode "adsf@62e9c582-1984-472b-a2f5-b61b2b0707e8")
(LgAnd
(LgConnector
(LgConnectorNode "ANY")
(LgConnDirNode "-")
(LgConnMultiNode "@")
)
(LgConnector
(LgConnectorNode "ANY")
(LgConnDirNode "+")
(LgConnMultiNode "@")
)
)
)
No cset for the last word!!
no DISJUNCT srcNode.get("DISJUNCT"); on last word .. line 139
no attr on the last word
load_senses
config.isStoreSense
ERROR: Throw to key `C++-EXCEPTION' with args `("fetch-atom" "Error:
get_uuid(): cannot find (WordNode \"abcccccccccc\" (ctv 1.000000
0.000000 4.000000)) ; [8608664669745395561][1]\n\n
(/home/linas/src/novamente/src/atomspace-git/opencog/persist/sql/multi-driver/SQLAtomStorage.cc:934)\nFunction
args:\n((EvaluationLink\n (LinkGrammarRelationshipNode \"ANY\")\n
(ListLink\n (WordNode \"###LEFT-WALL###\" (ctv 1 0 19431131))\n
(WordNode \"abcccccccccc\" (ctv 1 0 4))\n )\n)\n)")'.
(fetch-atom (Word "asdfasdfasdf"))
(fetch-atom (EvaluationLink (LinkGrammarRelationshipNode "ANY")
(ListLink (Word "###LEFT-WALL###") (Word "asdfasdfqewr"))))
(fetch-atom (ListLink (Word "###LEFT-WALL###") (Word "asdfasdfqewr")))
SQLAtomStorage::doGetLink
IOException
fixed in commit 327fc6c0a3c8b4b62ac80a61e2fdfc6971190c30
arghhh, last checkpoint: 24304265 24M distinct words
select sum(floatvalue[3]) from valuations where type=7;
gives 1914173302 == 1.9G observations
single-quote -- leading
semicolon -- leading
foobar
sql-stats should print name of the DB and the login user.<F4>
/home/ubuntu/run/hs_err_pid31730.log
but only when sock is dropped...
bug https://github.com/opencog/atomspace/issues/1054
Using bdwgc as of
0f4244e130541a591d638c98f820b6b49e915ac0
Author: Ivan Maidanski <ivmai@mail.ru>
Date: Thu Jan 12 00:58:41 2017 +0300
stack below seen twice 3 4
#0 0x00007ffff762c350 in pthread_kill ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007ffff785f490 in GC_suspend_all () at pthread_stop_world.c:640
#2 0x00007ffff785f564 in GC_stop_world () at pthread_stop_world.c:748
#3 0x00007ffff784b19e in GC_stopped_mark (
stop_func=stop_func@entry=0x7ffff784aca0 <GC_never_stop_func>)
at alloc.c:650
#4 0x00007ffff784bd19 in GC_try_to_collect_inner (
stop_func=0x7ffff784aca0 <GC_never_stop_func>) at alloc.c:488
#5 0x00007ffff784bf9a in GC_try_to_collect_general (
stop_func=stop_func@entry=0x0, force_unmap=force_unmap@entry=0)
at alloc.c:1064
#6 0x00007ffff784c06d in GC_gcollect () at alloc.c:1088
#7 0x00007ffff7b06529 in scm_i_gc (what=<synthetic pointer>)
at ../../libguile/gc.c:266
#8 scm_gc () at ../../libguile/gc.c:255
#9 0x00005555555551f2 in Eval::~Eval() ()
stack below seen twice.3
#0 0x00007ffff762c350 in pthread_kill ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007ffff785f6fb in GC_start_world () at pthread_stop_world.c:990
#2 0x00007ffff784b2d6 in GC_stopped_mark (
stop_func=stop_func@entry=0x7ffff784aca0 <GC_never_stop_func>)
at alloc.c:728
#3 0x00007ffff784bd19 in GC_try_to_collect_inner (
stop_func=0x7ffff784aca0 <GC_never_stop_func>) at alloc.c:488
#4 0x00007ffff784bf9a in GC_try_to_collect_general (
stop_func=stop_func@entry=0x0, force_unmap=force_unmap@entry=0)
at alloc.c:1064
#5 0x00007ffff784c06d in GC_gcollect () at alloc.c:1088
#6 0x00007ffff7b06529 in scm_i_gc (what=<synthetic pointer>)
at ../../libguile/gc.c:266
#7 scm_gc () at ../../libguile/gc.c:255
#8 0x00005555555551f2 in Eval::~Eval() ()
==============================================
latest bdwgc does this:
#0 GC_push_all_eager (bottom=<optimized out>,
top=top@entry=0x7ffff43f7000 <error: Cannot access memory at address
0x7ffff43f7000>) at mark.c:1599
#1 0x00007ffff7855625 in GC_push_all_stack (bottom=<optimized out>,
top=top@entry=0x7ffff43f7000 <error: Cannot access memory at address
0x7ffff43f7000>) at mark.c:1609
#2 0x00007ffff78568aa in GC_push_all_stack_sections (lo=<optimized
out>,
lo@entry=0x7ffff43f56c0 <error: Cannot access memory at address
0x7ffff43f56c0>,
hi=hi@entry=0x7ffff43f7000 <error: Cannot access memory at address
0x7ffff43f7000>, traced_stack_sect=<optimized out>) at mark_rts.c:565
#3 0x00007ffff785f2fd in GC_push_all_stacks () at
pthread_stop_world.c:557
#4 0x00007ffff7855bcf in GC_mark_some (
cold_gc_frame=0x7fffe43d5d00 "\274ǧ\367\377\177") at mark.c:343
#5 0x00007ffff784b20d in GC_stopped_mark (
stop_func=stop_func@entry=0x7ffff784aca0 <GC_never_stop_func>)
at alloc.c:702
#6 0x00007ffff784bd19 in GC_try_to_collect_inner (
stop_func=0x7ffff784aca0 <GC_never_stop_func>) at alloc.c:488
#7 0x00007ffff784bf9a in GC_try_to_collect_general (
stop_func=stop_func@entry=0x0, force_unmap=force_unmap@entry=0)
at alloc.c:1064
#8 0x00007ffff784c06d in GC_gcollect () at alloc.c:1088
#9 0x00007ffff7b06529 in scm_i_gc (what=<synthetic pointer>)
at ../../libguile/gc.c:266
#10 scm_gc () at ../../libguile/gc.c:255
#11 0x00005555555551f2 in Eval::~Eval() ()
apt-cache show libgc-dev
libgc-dev_7.4.2-8_amd64.deb
git tag
gc7_4_2
gc7_4_4
gc7_6_0 <--- try this one -- notpe still bad.
txt="Error: get_uuid(): cannot find (ListLink\n (WordNode \"Hume\"
(ctv 1.000000 0.000000 16.000000)) ; [1287745831931189139][1]\n
(WordNode \",\" (ctv 1.000000 0.000000 64.000000)) ;
[7731950238383222497][1]\n"...)
result = RAISE_SIGNAL(p, GC_sig_thr_restart);
Guess some of the threads are exiting when gc runs...
for (p = GC_threads[i]; p != 0; p = p -> next) {
p
#define RAISE_SIGNAL(t, sig) pthread_kill(THREAD_SYSTEM_ID(t), sig)
#define THREAD_SYSTEM_ID(t) (t)->id
ohhh 7.6 sometimes deadlocks...
------------------ done iteration 14
Done creating 120 threads
Thread 1917 "a.out" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffc639b700 (LWP 27872)]
0x00007ffff762d350 in pthread_kill ()
from /lib/x86_64-linux-gnu/libpthread.so.0
(gdb) bt
#0 0x00007ffff762d350 in pthread_kill ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007ffff785fe70 in GC_suspend_all () at pthread_stop_world.c:640
#2 0x00007ffff785ff44 in GC_stop_world () at pthread_stop_world.c:748
(igdb) up
#1 0x00007ffff785fe70 in GC_suspend_all () at pthread_stop_world.c:640
640 result = RAISE_SIGNAL(p, GC_sig_suspend);
(gdb) print p
$1 = (GC_thread) 0x7ffff7a7d380 <first_thread>
(gdb) print i
$2 = 0
(gdb) print p->id
$3 = 140737198868224
(gdb) print /x p->id
$4 = 0x7fffeebec700
Sooo.. p->id is not self.
(gdb) print /x p->next
$5 = 0x0
(gdb) print GC_threads[1]
$6 = (volatile GC_thread) 0x0
(gdb) print /x self
$10 = 0x7fffc639b700 -- yes this agreees with gdb message on top.
(gdb) info thr -- lists all the threads. ... 0x7fffeebec700 is not listed,
its not a valid thread.
sure looks like bdwgc is failing to keep GC_threads[0] accurate.!!
(gdb) print p->flags
$11 = 6 '\006'
include/private/pthread_support.h
# define FINISHED 1 /* Thread has exited.
# define DETACHED 2 /* Thread is treated as detached.
/* Thread may really be detached, or
/* it may have been explicitly
/* registered, in which case we can
/* deallocate its GC_Thread_Rep once
/* it unregisters itself, since it
/* may not return a GC pointer.
# define MAIN_THREAD 4 /* True for the original thread only.
but its not teh main thread::: .. p is neoither of the below:
(gdb) info thr
Id Target Id Frame
1 Thread 0x7ffff7fc7300 (LWP 24666) "a.out" 0x00007ffff762767d in pthread_join () from /lib/x86_64-linux-gnu/libpthread.so.0
* 1917 Thread 0x7fffc639b700 (LWP 27872) "a.out" 0x00007ffff762d350 in pthread_kill () from /lib/x86_64-linux-gnu/libpthread.so.0
(gdb) print *p
$12 = {next = 0x0, id = 140737198868224, stop_info = {last_stop_count = 1545,
stack_ptr = 0x7fffeebeb6c0 <error: Cannot access memory at address 0x7fffeebeb6c0>}, flags = 6 '\006', thread_blocked = 0 '\000', finalizer_skipped = 0,
finalizer_nested = 0 '\000', stack_end = 0x0, altstack = 0x0,
altstack_size = 0, stack = 0x0, stack_size = 0, traced_stack_sect = 0x0,
status = 0x0, tlfs = {_freelists = {{0x1, 0xbd, 0x555555818520,
0x55555591f5d0, 0x5555557ff500, 0x3d, 0x1, 0x1, 0x1, 0x65,
0x1 <repeats 14 times>, 0x1a}, {0x1, 0x5555559242d0, 0x55555587d820,
0x555555908540, 0x10, 0x3d, 0x5555558ff2a0, 0xb9, 0x1c, 0xb, 0x0, 0x1,
0x1, 0x1, 0x1, 0x1, 0x555555927d00, 0x1, 0x14, 0x1, 0x1, 0x1, 0x1,
0x1, 0x1}, {0x1 <repeats 25 times>}}, gcj_freelists = {
0xffffffffffffffff, 0x1 <repeats 24 times>}}}
new thread handled at line 538
threa delete on 572
GC_delete_thread line 550
but also ... GC_delete_gc_thread line 592
GC_delete_thread called from only one place:
GC_unregister_my_thread_inner
called from two places:
GC_unregister_my_thread
GC_thread_exit_proc -- not called by guile
pthread_start.c: pthread_cleanup_push(GC_thread_exit_proc, me);
scm_gc is NOT being called in guile mode.!! Example 1
but fixing this does not cure the bug.
threads.c: GC_unregister_my_thread ();
libguile/gc.c
static void on_thread_exit (void *v)
calls GC_unregister_my_thread() but only if t->needs_unregister) is set
called by init_thread_key
by scm_i_init_thread_for_guile
by scm_init_guile()
Hmm scm_i_with_guile no longer calls scm_init_guile()
but with_guile() does call scm_i_init_thread_for_guile
which does call init....
maybe the exit handler is still running.... when the gc is called.
except the thread should remain valid until all of the exit handlers
have returned. So maybe a thread handler did not run? ??
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f19e496f350 in pthread_kill ()
from /lib/x86_64-linux-gnu/libpthread.so.0
[Current thread is 1 (Thread 0x7f19aa6a7700 (LWP 7486))]
sconew not being called !
GC_register_my_thread
print notin
print nxt
x /150xg &board
[Current thread is 1 (Thread 0x7f7aa3a6d700 (LWP 25905))]
print p
$3 = (GC_thread) 0x7f7abb142380 <first_thread>
(gdb) print /x p->id
$5 = 0x7f7ab7a95700 this is the first thread....
Needs_unreg was not set!!
scm_init_guile ()
------------------ done iteration 540
Done creating 120 threads
argh this is broken:
cog-map-chase-links-chk
cog-par-chase-links-chk
thus all users of map-parses might be broken
map-parses
parallel-map-parses
map-word-instances
(use-modules (opencog) (opencog persist) (opencog persist-sql))
(use-modules (opencog nlp) (opencog nlp learn))
(sql-open "postgres:///en_pairs?user=linas")
(use-relex-server "127.0.0.1" 4445)
==========================================
again: using the rohit data:
select count(*) from atoms where type=73; gives 396K unique words.
select count(8) from atoms where type=47; gives number fo evluation
links which is 9672K but 2*396K of these are anynodes, so really only
9672-792=8880K unique word-pairs.
which were observed
en_pairs=> select * from atoms where type=74;
uuid | space | type | height | name | outgoing
-----------+-------+------+--------+------------+----------
143427138 | 1 | 74 | 0 | left-word |
143427163 | 1 | 74 | 0 | right-word |
en_pairs=> select * from atoms where outgoing='{143427138,143427163}';
uuid | space | type | height | name | outgoing
-----------+-------+------+--------+------+-----------------------
143670792 | 1 | 8 | 1 | | {143427138,143427163}
en_pairs=> select * from atoms where outgoing='{152,143670792}';
uuid | space | type | height | name | outgoing
-----------+-------+------+--------+------+-----------------
143670793 | 1 | 47 | 2 | | {152,143670792}
en_pairs=> select * from valuations where atom=143670793;
key | atom | type | floatvalue | stringvalue | linkvalue
-----+-----------+------+-----------------+-------------+-----------
1 | 143670793 | 177 | {0,0,418235182} | |
So 418M observations of word-pairs.
ALTER DATABASE en_pairs RENAME TO en_pairs_rohit;
=========================================================
(use-modules (opencog) (opencog persist) (opencog persist-sql))
(use-modules (opencog nlp) (opencog nlp learn))
(sql-open "postgres:///en_pairs?user=ubuntu")
(fetch-all-words)
(length (get-all-words)) 65021
(total-word-observations) 7181662.0 7.2M
(PredicateNode "*-Sentence Word Pair-*")
(SchemaNode "*-Pair Distance-*")o
(define pair-pred (PredicateNode "*-Sentence Word Pair-*"))
(define pair-dist (SchemaNode "*-Pair Distance-*"))
(length (cog-incoming-set pair-pred))
(get-total-atom-count (cog-incoming-set pair-pred))
get-any-left-wildcard-count
get-any-right-wildcard-count
set-any-pair-total
set-any-left-wildcard-count
set-any-right-wildcard-count
get-any-pair-link
batch-all-pair-wildcard-counts
get-all-words
batch-all-pair-wildcard-logli
get-pair-total
GET-LEFT-WILD
(store-atom
(set-count GET-LEFT-WILD TOTAL))
batch-all-pair-wildcard-logli
compute-pair-mi
get-any-pair-link-> GET-PAIR
get-word-pairs
cog-incoming-by-type
.scm: (par-map get-endpoint (cog-filter link-type
utilities.scm:
cog-map-chase-link dead
cog-map-chase-link-dbg used by ppars ranker
cog-map-chase-links-chk used by cog-chase-link-chk
cog-chase-link-chk used by nlp-utils
cog-map-apply-link dead used by cog-get-link
cog-get-link used by
cog-get-pred
(define bar (let ((done #f))
(lambda () (format #t "duude ~A\n" done) (set! done #t))))
(AnyNode \"left-word\") (WordNode \"###LEFT-WALL###\"
get-logli
(catch #t
(lambda () (xxxxx))
(lambda (key . args) #f)))
(start-trace msg)
get-any-pair get-clique-pair
wow 32 gigs to load all pairs!
tlb.clear()
reserve_upto(uuid)
getMaxUUID
(make-narrow-links (mst-parse-text "The game is played on a level playing field"))
(LgWordCset
(WordInstanceNode "foo@e5bfab2f-b3b1-4cf1-ae89-89aaa47bf871")
(LgAnd
(LgConnector
(LgConnectorNode "ANY")
(LgConnDirNode "-")
)
)
)
(unfold
(use-modules (opencog) (opencog persist) (opencog persist-sql))
(use-modules (opencog nlp) (opencog nlp learn))
(sql-open "postgres:///en_pairs_rohit?user=linas")
(fetch-all-words)
(fetch-any-pairs)
=========================================================
(sql-open "postgres:///en_snapshot?user=ubuntu")
Start counting, num words=106696
num words=106696
select count(*) from atoms where type=123; -- 123 is wordNode
85 | SchemaNode
88 | PredicateNode
123 | WordNode
142 | LinkGrammarRelationshipNode
uuid | space | type | height | name | outgoing
------+-------+------+--------+------------------------+----------
29 | 1 | 88 | 0 | *-Sentence Word Pair-* |
30 | 1 | 85 | 0 | *-Pair Distance-* |
141 | 1 | 142 | 0 | ANY |
select count(*) from atoms where outgoing[1]='29';
gives ... 12956699
30 gives ... 26174210 ... wtf!?
after loading clique only: 37925105 atoms = 38M/51G = 1.35KB/atom
(ListLink (WordNode "playing") (WordNode "field"))
zero
(ListLink (WordNode "day") (WordNode "field"))
1 and 1 at 30
Lincoln
the
which
(ListLink (Word "the" ) (Word "field" ))
(cog-chase-link
(cog-get-link 'EvaluationLink PAIR (Pred
(define cnt 0)
(for-each
(lambda (PAIR)
(set! cnt (+ cnt 1))
(if (not (eqv? (count-dist-pair PAIR) (count-clique-pair PAIR)))
(throw 'bad-count 'foobar PAIR)
(format #t "Its OK ~A\n" cnt)
))
(get-all-pairs))
115 | LgWordCset
2440463600
1 | 2440463600 | 177 | {1,0,12} | |
2440463720 {636276,2440463719}
2440463718 {107,2440463716}
2440463717 {101,2440463715}
2440463714 {1748714,2440463710}
2440463669 {2440463611,2440463599}
19741227 = 19.8M atoms / 30G = 1.52KB/atom
echo "" | ./submit-one.pl localhost 17001 observe-mst
(mst-parse-text "Area: total: 29,743 km² country comparison to the world: 149. land: 28,454 km² water: 1,289 km² Area comparative Australia comparative: about one third (33%) the size of Tasmania.")
freshie (0 ())
28 words
74 -- 34
108 -- 32
140 -- 30
170 -- 28
198 -- 26
224 -- 24
248 --
270
269
290
bad-pair in max-of-pair-list
(use-modules (opencog) (opencog persist) (opencog persist-sql))
(use-modules (opencog nlp) (opencog nlp learn))
(sql-open "postgres:///en_pairs_mst?user=linas")
(fetch-all-words)
(length (get-all-words))
396262
(fetch-pseudo-csets (get-all-words))
(length (filter-words-with-csets (get-all-words)))
30127
(cset-observations (get-all-words))
436595.0
425186
(cset-vec-support (Word "the"))
(cset-vec-len (Word "the"))
(
(length (cog-incoming-by-type WORD 'LgWordCset))
(define n 0)
(for-each
(lambda (cset)
(set! n (+ 1 n)))
(cog-incoming-by-type (Word "the") 'LgWordCset))
Ben, Ruiting,
For your enjoyment: I have some very preliminary results on word
similarity. They look pretty nice, even thogh based on a fairly
small number of observations.
If you've been watching TV instead of reading email, here's the story
so far: Starting from a large text corpus, the mutual information (MI)
of word-pairs are counted. This MI is used to perform a maximum
spanning-tree (MST) parse (of a different subset of) the corpus. From each
parse, a pseudo-disjunct is extracted for each word. The
pseudo-disjunct is like a real LG disjunct, except that each connector
in the disjunct is the word at the far end of the link.
So, for example, in in idealized world, the MST parse of the sentence
"Ben ate pizza" would prodouce the parse Ben <--> ate <--> pizza and
from this, we can extract the psuedo-disjunct (Ben- pizza+) on the
word "ate". Similarly, the sentence "Ben puked pizza" should produce
the disjunct (Ben- pizza+) on the word "puke". Since these two
disjuncts are the same, we can conclude that the two words "ate"
and "puke" are very similar to each other. Considering all of the
other disjuncts that arise in this example, we can conclude that these
are the only two words that are similar.
Note that a given word may have very many psuedo-disjuncts attached
to it. Each disjunct has a count of the number of times it has been
observed. Thus, this set of disjuncts can be imagined to be a vector
in a high-dimensional vector space, which each disjunct being a single
basis element. The similarity of two words can be taken to be the
cosine-similariy between the disjunct-vectors (or pick another,
different metric, as you please.)
Below are a set of examples, for English, on a somewhat smmall dataset.
Collected over a few days, it contains just under half-a-million
observations of disjuncts, distributed across about 30K words.
Thus, most words will have only a copule of disjuncts on them,
which may have been seen only a couple of times. its important, at
this stage, to limit oneself to only the most popular words.
We expect the determiners "the" and "a" to be similar, and they are:
(cset-vec-cosine (Word "the") (Word "a")) = 0.1554007744026141
Even more similar:
(cset-vec-cosine (Word "the") (Word "this")) = 0.3083725359820755
Not very similar at all:
(cset-vec-cosine (Word "the") (Word "that")) = 0.01981486048876119
Oh hey this and that are similar. Notice the triangle with "the".
(cset-vec-cosine (Word "this") (Word "that")) = 0.14342403062507977
Some more results
(cset-vec-cosine (Word "this") (Word "these")) = 0.23100101197144984
(cset-vec-cosine (Word "this") (Word "those")) = 0.1099725424243773
(cset-vec-cosine (Word "these") (Word "those")) = 0.13577971016706158
We expect that determiners, nouns and verbs to all be very differrent
from one-another. And they are:
(cset-vec-cosine (Word "the") (Word "ball")) = 2.3964597196461594e-4
(cset-vec-cosine (Word "the") (Word "jump")) = 0.0
(cset-vec-cosine (Word "ball") (Word "jump")) = 0.0
We expect verbs to be similar, and they sort-of are.
(cset-vec-cosine (Word "run") (Word "jump")) = 0.05184758473652128
(cset-vec-cosine (Word "run") (Word "look")) = 0.05283524652572603
Since this is a sampling from wikipedia, there will be very few
"action" verbs, unless the sample accidentally contains articles
about sports. A "common sense" corpus, or a corpus that talks about
what people do, could/should improve the above verbs. These are
very basic to human behavior, but are rare in most writing.
I'm thinking that a corpus of children's lit, and young-adult-lit
would be much better for these kinds of things.
An adjective.
(cset-vec-cosine (Word "wide") (Word "narrow")) = 0.06262242910851494
(cset-vec-cosine (Word "wide") (Word "look")) = 0.0
(cset-vec-cosine (Word "wide") (Word "ball")) = 0.02449979787750126
(cset-vec-cosine (Word "wide") (Word "the")) = 0.04718158900583385
(cset-vec-cosine (Word "heavy") (Word "wide")) = 0.05752237416355278
Here's a set of antonyms!
(cset-vec-cosine (Word "heavy") (Word "light")) = 0.16760038078849773
A pronoun
(cset-vec-cosine (Word "ball") (Word "it")) = 0.009201177048960233
(cset-vec-cosine (Word "wide") (Word "it")) = 0.005522960959398417
(cset-vec-cosine (Word "the") (Word "it")) = 0.01926824360790382
Wow!! In English, "it" is usually a male!
(cset-vec-cosine (Word "it") (Word "she")) = 0.1885493638629482
(cset-vec-cosine (Word "it") (Word "he")) = 0.4527656594627214
(cset-vec-cosine (Word "he") (Word "she")) = 0.1877589816088902
I can post the database on mondy, let me know when you're ready to
receive it.
above dataset has 200183 (200K) disjuncts in it
SIMILARITY_LINK
ac
scheme@(guile-user)> (batch-sim (Word "him") ac)
terminate called after throwing an instance of
'opencog::RuntimeException'
what(): Failed to execute!
(/home/linas/src/novamente/src/atomspace-git/opencog/persist/sql/multi-driver/ll-pg-cxx.cc:113)
Aborted
[2017-05-07 17:01:27:152] [WARN] PQresult message: ERROR: duplicate key
value violates unique constraint "atoms_uuid_idx"
DETAIL: Key (uuid)=(2441253852) already exists.
[2017-05-07 17:01:27:152] [WARN] PQ query was: INSERT INTO Atoms (uuid,
space, type, height, outgoing) VALUES (2441253852, 1, 43, 1, '{1888882,
127137759}');
dohhh Need to reserve some uuid's!
[2017-05-04 17:31:29:122] [ERROR] There is no value for key
(PredicateNode "link count") ; [8812231717367445517][1]
on atom (AnchorNode "MST data") ; [1539316140051770629][1]
createdb -T en_pairs_mst -e en_pairs_sim "forked from en_pairs_mst, holds similarity data"
COMMENT ON DATABASE en_pairs_mst IS 'forked from en_pairs_rohit, holds disjunct data';
COMMENT ON DATABASE en_pairs_rohit IS 'forked from en_pairs_2016, converted to modern format';
COMMENT ON DATABASE en_pairs_2016 IS 'Rohits original pair data; old format; missing correct word-counts';
(define ac (filter-words-with-csets (get-all-words)))
(cset-vec-len
(cset-vec-observations
(define lens (map (lambda (wrd) (cons (cset-vec-len wrd) wrd)) ac))
(define slens (sort lens (lambda (a b) (> (car a) (car b)))))
(define obs (map (lambda (wrd) (cons (cset-vec-observations wrd) wrd)) ac))
(define sobs (sort obs (lambda (a b) (> (car a) (car b)))))
(define firm (filter (lambda (wrd) (< 8.0 (cset-vec-len wrd))) ac))
*) count words during MST
err this can be reconstructed by taking:
counts of disjuncts based on counts of csets ....
so never mind....
what are teh most comon disjuncts???
(filter (lambda (cset) (< 10 (get-count cset)))
(cog-incoming-by-type (Word "United") 'LgWordCset))
156 ... 125.888
5 41.44827196747233 "and"
8 31.658516707416464 "as"
10 30.140583554376658 "well"
11 29.969530201342277 "in"
18 18.805788982259568 "first"
21 17.254613233923575 "is"
22 16.742203742203742 "also"
23 15.386776859504135 "can"
24 15.359470949456778 "on"
25 14.910157559503853 "at"
26 14.424603174603176 "been"
31 13.231375985977214 "but"
32 13.122743682310471 "part"
34 12.835568802781918 "with"
35 12.835051546391755 "old"
36 12.726962457337885 "same"
38 11.925403225806452 "one"
39 11.611694152923539 "which"
43 10.856905158069885 "not"
(filter (lambda (cset) (< 5 (get-count cset)))
(cog-incoming-by-type (Word "Gestalt") 'LgWordCset))
(.+) 34
(of -) 14
(is+) 216
was+ 197
(has+ 55
was+ was+ 12
is+ was+ 10
get-cset-vec
ad cset-vec-observations
(define disjunct-entropy
(define total-cset-count (cset-observations all-cset-words))
(define all-cset-words (filter-words-with-csets (get-all-words)))
(define total-cset-count 0)
(cog-map-type
(lambda (cset)
(set! total-cset-count (+ total-cset-count (get-count cset))) #f)
'LgWordCset)
7.984991833299495 bits total MI bits
(cset-vec-mi (Word "United")) 7.116097374166928
is the relative(?) MI to the disjuncts in the word.
(is how much MI this word contributes to the total MI)
(and its slightly below average)
(cset-vec-frequency (Word "United"))
(/ (log (cset-vec-frequency (Word "United"))) (log 2))
-10.48276867673862 is the entropy for that word, only.
"Cao" "Award" "x" "y" "Prime" "we" "per" "Division" "League" "Game"
(define cao (get-cset-vec (Word "Cao")))
34 (length cao)
(filter (lambda (cset) (< 10 (get-count cset))) cao)
18 ,- Cao- Cao+ garbage !!
8 Grammy- for+ Best+
7 Academy- for+ Best+
23 Minister+
5 the- Minister+
3 as- Minister+
3 Governor-General- Minister+
"The" "a" "to" "in" "of" "and" "the" "," "."
"Bay" "Street" "Island" "of" "century" "right" "Game" "Georgian" "or" "a" ";" "near" "Party" "team" "law" "Australia" "her" "research" "Church" "east" "Government"
--------------------------------------------------------------
1985 words of length > 8
all pairs of above with cosine > 0.5
23993 pairs, but some are junk
15809 good pairs.
store-sim
(sim-pair WORD-A WORD-B) cos-key
(FloatValue SIM (get-angle SIM)))
(define (sim-pair WORD-A WORD-B) (SimilarityLink WORD-A WORD-B))
(define cos-key (PredicateNode "*-Cosine Distance Key-*"))
(define all-sims '())
(cog-map-type
(lambda (sim) (set! all-sims (cons sim all-sims)) #f)
'SimilarityLink)
(define (cosine-of-sim SIM)
(car (cog-value->list (cog-value SIM cos-key))))
(define good-sims
(filter
(lambda (sim)
(< 0.5 (car (cog-value->list (cog-value sim cos-key)))))
all-sims))
(map (lambda (sim) (sim-cosine sim)) (take ranked-sims 20))
'Poir .. Perls'
'Poir .. Sevan'
'Sevan .. Perls'
'Sevan .. Ruins'
'Ruins .. Perls'
'Poir .. Ruins'
'Gongora .. Perls'
'Poir .. Gongora'
'Gongora .. Sevan'
'Gongora .. Ruins'
'Poir .. Gag'
(use-modules (opencog) (opencog persist) (opencog persist-sql))
(use-modules (opencog nlp) (opencog nlp learn))
(sql-open "postgres:///en_pairs_sim?user=linas")
(use-modules (opencog cogserver))
(start-cogserver "opencog2.conf")
(fetch-all-words)
(fetch-pseudo-csets (get-all-words))
70 74 128
95 0.953598053136202 'conjunction .. accordance'
504 0.868429815807705 'in .. from'
633 0.8549756852697713 'in .. at'
637 0.8545553489854567 'In .. By'
643 0.8540378312493768 'in .. After'
740 0.846034030453443 'canal .. swamp'
754 0.843837363738534 'at .. At'
757 0.8436223414042503 'in .. In'
818 0.838269351079866 'in .. on'
840 0.8366056647469058 'creation .. existence'
878 0.8332692184393486 'e.g .. i.e'
965 0.8265058609938521 'and .. but'
969 0.8264399296782371 'in .. With'
971 0.8263325988164274 'in .. inside'
canal -- 68 dj
(define (prt-sim sim port)
(format port "~A '~A .. ~A'\n" (sim-cosine sim)
(cog-name (gar sim)) (cog-name (gdr sim))))
get-disjunct-string
20.0 canal the-
8.0 canal the- .+
5.0 canal the- ,+
14.0 swamp the-
3.0 swamp the- .+
18.0 creation the- of+
10.0 existence the- of+
627 in the+
75 in a+
61 in which+
49 in the+ the+
36 in used-
33 in his+
32 in order+
31 in United+
27 in gallantry-
26 in born-
23 in role-
8338 -- in
2185 from
155 from the+
20 from :+ :+ :+ :+ :+ :+ :+ :+ :+ :+ 0+ 0+ 0+ 0+ 0+ 0+ 0+ 0+ 0+ 0+
18 from a+
9 from derived-
9 from until+
9 from degree-
8 from his+
8 from comes-
8 from away-
7.0 from time+
7.0 from the+ the+
get-pair-mi
get-pair-mi-str ("
MI(fast, horse) = 5.258979
MI(fast, car) = 3.805734
(get-pair-logli get-any-pair (Word "fast") (Word "horse"))
-log P(fast, horse) = 24.054777 P = 5.7383969102974027e-8
-log P(fast, car) = 23.832385 P = 6.694794440144226e-8
-log P(*, horse) = 14.383531 P = 4.6787056718009e-5
-log P(*, car) = 12.707894 P = 1.4946610743515038e-4
-log P(fast, *) = 14.930225 P = 3.2029815202410256e-5
MI(fast, {hosrse, car}) = linear interploation of the two.
[
p(fast,horse) log p(fast, horse) / p(fast,*) p(*,horse)
+
p(fast,car) log p(fast, car) / p(fast,*) p(*,car) ]
/ [p(fast,horse) + p(fast,car)]
= [ p(fast,horse) 5.258979 + p(fast,car) 3.805734] /
1.2433191350441628e-7
= 5.565671566878678e-7 / 1.2433191350441628e-7
= 4.476462567015012
(get-left-wild-prob GETTER WORD)
(get-left-wild-log-prob GETTER WORD)
(get-pair-prob GETTER WORD WORD)
(get-logli ATOM)
(define (get-left-wild-prob GETTER WORD)
(EvaluationLink any-pair-pred (ListLink any-left word))
(define (get-any-wild-wild)
(EvaluationLink any-pair-pred (ListLink any-left any-right))
)
(define (get-any-wild-wild)
(EvaluationLink any-pair-pred (ListLink any-left any-right))
)
get-any-pair PAIR -- public OK defined in common.scm
get-clique-pair PAIR --public OK
kill get-pair-mi-str
make get-pair-mi public. done
(define all-cset-words (filter-words-with-csets (get-all-words)))
(define all-disjuncts (get-all-disjuncts))
(define all-csets (get-all-csets))
(define total-cset-count (get-total-cset-count))
in the en_pairs sim: MI is 7.9849918332741225
(- (+ 10.280666542033556 16.00578269190252) 18.301457400661953)
in en_pairs_mst:
(length all-cset-words) 49423
(length all-disjuncts) 486824 + 73 more
(length all-csets) 749337 + 118 more
total-cset-count 1154809 + 235 more
word-entropy-bits 10.353641069558124
disjunct-entropy-bits 16.48346995957877
cset-entropy-bits 18.958440018522307
MI is 7.878671010614585
all connectors ...
(length (get-all-pseudo-connectors)) 81557
wtf:
(length all-disjuncts) 225396
wiki-mst-en.sh is running
why is get_nodes bad? because no nodes are being created. so that OK.
why is total stores for node =0? just like above. its OK.
postgres:///en_pairs_mst?user=linas is the ongoing accumulation of mst stats...
guile -l mst-count-en.scm
335 secs for words
1071 secs for pairs
results size is 32G for 19741168 atoms
1213 secs for csets
fix message -- done
add sql-clear-stats -- done
./mst-one.sh en gamma-pages/G/Gzhelian localhost 17001
move base-stats.scm to pair-stats.scm -- DONE
Cannot connect to database: FATAL: remaining connection slots are reserved for non-replication superuser connections
get-clique-pair get-any-pair
(display-backtrace (fluid-ref the-last-stack))
Throw to key `bad-summation' with args `(count-all-pairs "Error:
pair-counts unequal: 5974976.0 5754410.0\n")'.
pair-counts unequal: 5974600.0 5761584.0\n")'. afer redoing the left!
pair-counts unequal: 5974600.0 5761584.0\n")' afer doing left 2x
pair-counts unequal: 5974976.0 5761682.0\n")' after redoing right.
pair-counts unequal: 5974976.0 5761682.0\n")' after right 2x
pair-counts unequal: 5974600.0 5761584.0\n") after redoing left
(define aw (get-all-words))
(ob 'left-stars)
(define save-1 (map (lambda (w) (cons (ob 'left-wild-count w) w)) aw))
(length (cob 'cache-all-left-counts))
(for-each (lambda (a b) (if (not (equal? a b)) (error "aiiide ~A ~A" a b)))
save-1 save-2)
ERROR: aiiide ~A ~A (54.0 . (WordNode "d'Urville")
) (56.0 . (WordNode "d'Urville")
)
So... recomputing the right counts changed the left-count on above....
wtf.... (ob 'left-wildcard w) is empty....
set-pair-count calls make-pair
set-left-wild-count calls set-pair-count with 'left-wildcard which is null
foobar confused about which pari si which
Glurg. and now its this:
pair-counts unequal: 2133920.0 1168749.0\n")'
(define w (WordNode "d'Urville"))
(define ob (make-pair-wild (make-any-link)))
(define ob (make-pair-count-api ob))
(define cob (make-compute-count ob))
(ob 'left-support-size)
(length (ob 'left-stars))
(cob 'compute-total-count)
(length (cob 'cache-all-left-counts))
(cob 'compute-total-count)
(define save-1 (map (lambda (w) (cons (ob 'left-wild-count w) w)) aw))
(cob 'compute-left-count w)
(ob 'left-wildcard w)
(ob 'right-wildcard w)
(ob 'left-wild-count w)
(for-each (lambda (a b) (if (not (equal? a b)) (error "aiiide ~A ~A" a b)))
save-1 save-2)
(fold (lambda (p s) (+ s (car p))) 0 save-1) 1173656.0
(define rave-1 (map (lambda (w) (cons (ob 'right-wild-count w) w)) aw))
(fold (lambda (p s) (+ s (car p))) 0 rave-1) 2133920.0
(define (lsum w)
(
1) fetch failed
2) compute failed.
3) left-stars returns junk fixed
(define mave-1 (map (lambda (w) (cons (ob 'left-wild-count w) w)) aw))
pair-counts unequal: 34692424.0 34689203.0\n")'.
Elapsed time to load words: 45 secs
Done loading words, now loading any-pairs
Elapsed time to load ANY-link pairs: 322 secs
Finished loading any-word-pairs
Support: num left=106696 num right=106696
Done with wild-card count N(*,w) and N(w,*) in 222 secs << par-map vs.. 217
Done computing N(*,*) total-count=34688664.0 in 13 secs
Start computing log P(*,w)
Done computing 106696 left-wilds in 39 secs << 120% vs. 15
Done storing 106696 left-wilds in 458 secs << par-for vs 101 secs for non-par
Done with -log P(*,w), start -log P(w,*)
Done computing 106696 right-wilds in 45 secs vs. 15 now
Done storing 106696 right-wilds in 458 secs << par-for vs 103 now
Done computing -log P(w,*) and <-->
Going to do individual word-pair MI
Done computing 2360328 pair MI's in 962 secs (1124 secs under load)
Stored 100000 of 2360328 pairs in 191 secs (523.5602094240837 pairs/sec)
....
Stored 2000000 of 2360328 pairs in 317 secs (315.45741324921136 pairs/sec)
Stored 2100000 of 2360328 pairs in 251 secs (398.40637450199205 pairs/sec)
Stored 2200000 of 2360328 pairs in 254 secs (393.7007874015748 pairs/sec)
Stored 2300000 of 2360328 pairs in 300 secs (333.3333333333333 pairs/sec)
Done storing 2360328 pair MI's in 6964 secs <<< 6847 under load
Finished with MI computations
Sooo any-pair:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5180 linas 20 0 11.819g 8.935g 37080 S 0.0 3.5 117:00.53 guile
5525 linas 20 0 11.483g 8.931g 37616 S 0.0 3.5 129:07.73 guile
----------------
clique pairs:
Start loading words ...
Elapsed time to load words: 45 secs
Done loading words, now loading clique pairs
Elapsed time to load clique pairs: 1145 secs <<< vs 322 for any-pairs
Finished loading clique-word-pairs
Support: num left=106696 num right=106696
Done with wild-card count N(*,w) and N(w,*) in 737 secs <<< vs 215 for any
Done computing N(*,*) total-count=31628463.0 in 14 secs <<<!? almost the same??
Start computing log P(*,w)
Done computing 106696 left-wilds in 14 secs
Done storing 106696 left-wilds in 103 secs
Done with -log P(*,w), start -log P(w,*)
Done computing 106696 right-wilds in 15 secs
Done storing 106696 right-wilds in 103 secs
Done computing -log P(w,*) and <-->
Going to do individual word-pair MI
Done computing 4810 pair MI's in 37872 secs << 10.5 hours!!!
Done storing 4810 pair MI's in 7 secs << whhhat!????
Finished with MI computations
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17274 linas 20 0 26.656g 0.024t 37676 S 0.0 9.6 406:04.03 guile
=================================================================
=================================================================
disjuncts & csets:
Start loading words ...
Elapsed time to load words: 126 secs << now 112 ...814 ... 22 .. 20
Done loading words, now loading pairs
Elapsed time to load csets: 783 secs << now 367 on fresh..553...294..309
Finished loading any-word-pairs
Support: num left=396262 num right=291637 << in 18 13
Done with wild-card count N(*,w) and N(w,*) in 39 secs `<< now 203 !??? fluke
<< 85 after redesign ,, 72
Done computing N(*,*) total-count=661104.0 in 34 secs << uncahnged
<< 14 after redesign 15
Done computing 446204 pairs in 13 secs << 23 after red 24
Start computing log P(*,w)
Done computing 291637 left-wilds in 33 secs << 27
Done storing 291637 left-wilds in 670 secs << now 493 .. 711 (spin media)
<< dropped to 87 secs on SSD
Done with -log P(*,w), start -log P(w,*)
Done computing 396262 right-wilds in 26 secs now 4 secs
Done storing 396262 right-wilds in 549 secs <<< now 28 !! wtf ??? .. 47
Done computing -log P(w,*) and <-->
Going to do individual word-pair MI
Done computing 446204 pair MI's in 88 secs << 70 secs
Stored 100000 of 446204 pairs in 268 secs (373.13432835820896 pairs/sec)
Stored 200000 of 446204 pairs in 238 secs (420.16806722689074 pairs/sec)
Stored 300000 of 446204 pairs in 286 secs (349.65034965034965 pairs/sec)
Stored 400000 of 446204 pairs in 303 secs (330.03300330033005 pairs/sec)
Done storing 446204 pair MI's in 1215 secs ... 1345 (spin media)
<< 201 secs SSD
Going to do column and row subtotals
Finished left entropy subtotals in 46 secs << 43
Finished right entropy subtotals in 20 secs << 16
Finished left MI subtotals in 40 secs << 35
Finished right MI subtotals in 12 secs << 8
Finished with MI computations
(use-modules (opencog) (opencog persist) (opencog persist-sql))
(use-modules (opencog nlp) (opencog nlp learn))
(sql-open "postgres:///en_pairs_sim?user=linas")
(use-modules (opencog cogserver))
(start-cogserver "opencog2.conf")
(define pca (make-pseudo-cset-api))
(batch-pairs pca)
(pca 'fetch-pairs)
(print-matrix-summary-report pca #t)
(define psa (add-pair-stars pca))(define psa (add-pair-stars pca))
(define sup (add-support-compute psa))
(sup 'cache-all)
(define cca (make-central-compute psa))
(cca 'cache-all)
(define sto (make-store psa))
(sto 'store-wildcards)
(print-matrix-summary-report pca #t)
Summary Report for Correlation Matrix Word-Disjunct Pairs (aka Connector
Sets)
Left type: WordNode Right Type: LgAnd Pair Type: LgWordCset
Wildcard: (ListLink (ctv 0 0 661104)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 37413 Columns: 291637
Size: 446204 non-zero entries of 10911015081 possible
Fraction non-zero: 4.0895E-5 Sparsity (-log_2): 14.578
Total observations: 661104.0 Avg obs per pair: 1.4816
Entropy Total: 18.301 Left: 16.006 Right: 10.281
Total MI: -7.985
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 4122. 370.0
Count (l_1) 8763. 680.7 2.126 1.839
Length (l_2) 236.8 51.39 5.7452E-2 .1389
RMS Count 216.6 46.40 5.2550E-2 .1254
duuuu
(define pca (make-pseudo-cset-api))
(define wpca (add-pair-wildcards pca))
(define cpca (add-pair-count-api pca))
(define fpca (add-pair-freq-api pca))
(define w (car (wpca 'left-support)))
(define d (car (wpca 'right-support)))
(cpca 'left-wild-count d)
(define d (car (drop-while
(lambda (dj) (> 10 (cpca 'left-wild-count dj))) (wpca 'right-support))))
(define w (car (drop-while
(lambda (wrd) (> 10 (cpca 'right-wild-count wrd))) (wpca 'left-support))))
(cpca 'right-wild-count w)
(fpca 'left-wild-freq d)
(/ (cpca 'left-wild-count d) (cpca 'wild-wild-count))
(fpca 'right-wild-freq w)
set-right-wild-freq
(define cf (make-compute-freq (add-pair-freq-api (add-pair-wildcards cpca))))
(cf 'init-freq)
(cf 'compute-right-freq w)
(cf 'cache-right-freq w)
'compute-right-freq w
(define righties (cf 'cache-all-right-freqs))
(define okr (filter (lambda (x) (not (null? x))) righties))
(length okr)
etierh not iint or ot being computed. FIXED
ERROR: In procedure cog-value: There is no value for key (PredicateNode
"*-FrequencyKey-*") ; [6632538966455889482][1]
on atom (LgWordCset (ctv 1.000000 0.000000 1.000000)
(LgWordCset
(WordNode "dentist"
(LgAnd
(PseudoConnector
(WordNode "an")
(LgConnDirNode "-"))
(PseudoConnector
(WordNode ",")
(LgConnDirNode "+"))))
there's no compute-pair 'set-pair-freq
=========================================================================
2017-05-13 01:35:49 UTC [229-2879] LOG: checkpoints are occurring too frequently (29 seconds apart)
2017-05-13 01:35:49 UTC [229-2880] HINT: Consider increasing the configuration parameter "max_wal_size".
pre postgres 9.5:
checkpoint segments. 100 or so is not out of the ordinary.
increase your checkpoint_timeout to 1 hour, as well as look at
increasing your checkpoint_completion_target
------
2017-05-17 07:48:25 UTC [228-1] LOG: database system was shut down at 2017-05-17 05:02:36 UTC
2017-05-17 07:48:25 UTC [229-1] [unknown]@[unknown] LOG: incomplete startup packet
2017-05-17 07:48:25 UTC [228-2] LOG: invalid primary checkpoint record
2017-05-17 07:48:25 UTC [228-3] LOG: invalid secondary checkpoint record
2017-05-17 07:48:25 UTC [228-4] PANIC: could not locate a valid checkpoint record
2017-05-17 07:48:25 UTC [230-1] postgres@postgres FATAL: the database system is starting up
2017-05-17 07:48:25 UTC [203-1] LOG: startup process (PID 228) was terminated by signal 6: Aborted
2017-05-17 07:48:25 UTC [203-2] LOG: aborting startup due to startup process failure
2017-05-17 07:51:21 UTC [1430-1] LOG: database system was shut down at 2017-05-17 05:02:36 UTC
2017-05-17 07:51:21 UTC [1430-2] LOG: invalid primary checkpoint record
2017-05-17 07:51:21 UTC [1430-3] LOG: invalid secondary checkpoint record
2017-05-17 07:51:21 UTC [1430-4] PANIC: could not locate a valid checkpoint record
2017-05-17 07:51:21 UTC [1429-1] LOG: startup process (PID 1430) was terminated by signal 6: Aborted
2017-05-17 07:51:21 UTC [1429-2] LOG: aborting startup due to startup process failure
http://stackoverflow.com/questions/8799474/postgresql-error-panic-could-not-locate-a-valid-checkpoint-record
time cp -pr main main-fu <<< 23 minutes
du -sm << 81638 -- 82 gigabytes!
/usr/lib/postgresql/9.5/bin/pg_resetxlog /var/lib/postgresql/9.5/main
Transaction log reset
sudo service postgresql start
2017-05-17 13:17:08 UTC [15452-1] LOG: database system was shut down at 2017-05-17 13:05:10 UTC
2017-05-17 13:17:08 UTC [15452-2] FATAL: could not access status of transaction 2804419190
2017-05-17 13:17:08 UTC [15452-3] DETAIL: Could not open file "pg_clog/0A72": No such file or directory.
2017-05-17 13:17:08 UTC [15451-1] LOG: startup process (PID 15452) exited with exit code 1
2017-05-17 13:17:08 UTC [15451-2] LOG: aborting startup due to startup process failure
https://www.postgresql.org/message-id/AANLkTims48njGKkvkXHiTG6=19hQ31VFmFRPPLg+5w03@mail.gmail.com
dd if=/dev/zero of=Transaction_missing_file_of pg_clog bs=256K count=1
(To make the uncommitted record as they haven't been committed).
dd if=/dev/zero of=0A72 bs=256K count=1
Last entries in pg_clog are from 14 May, so 3 days of nothingness.
2017-05-17 13:27:53 UTC [24212-1] LOG: database system was interrupted; last known up at 2017-05-17 13:17:08 UTC
2017-05-17 13:27:53 UTC [24212-2] LOG: database system was not properly shut down; automatic recovery in progress
2017-05-17 13:27:53 UTC [24212-3] LOG: invalid record length at D62/6C000098
2017-05-17 13:27:53 UTC [24212-4] LOG: redo is not required
2017-05-17 13:27:53 UTC [24212-5] FATAL: could not access status of transaction 2124106599
2017-05-17 13:27:53 UTC [24212-6] DETAIL: Could not open file "pg_multixact/offsets/7E9B": No such file or directory.
dd if=/dev/zero of=7E9B bs=256K count=1
members/1FA5
psql
\l
en_pairs
en_pairs_old
en_pairs_partial
en_snapshot
pg_dump en_pairs
pg_dump -cC en_pairs
pg_dumpall
time pg_dump en_pairs >en_pairs.sql
pg_dump: Dumping the contents of table "valuations" failed: PQgetResult() failed.
pg_dump: Error message from server: ERROR: could not access status of transaction 2786849088
DETAIL: Could not open file "pg_clog/0A61": No such file or directory.
dd if=/dev/zero of=~/9.5/main/pg_clog/0A61 bs=256K count=1
pg_clog/0A51
pg_clog/0A70
pg_clog/0A59
pg_clog/0A5C
pg_clog/0A50
pg_clog/093A
everything from that to 0A5E
for ((i=0;i<293;i++))
do
b=$(echo "ibase=16;93A" | bc)
j=$(echo "$b+$i" |bc)
k=$(echo "obase=16;$j" | bc)
echo dd if=/dev/zero of=~/9.5/main/pg_clog/0$k bs=256K count=1
done
pg_clog/0923
pg_clog/08E4
pg_clog/085E
0524
pg_clog/0523
mv /var/lib/postgresql/9.5/main /var/lib/postgresql/9.5/main-semi-fu
/usr/lib/postgresql/9.5/bin/initdb /var/lib/postgresql/9.5/main
CREATE ROLE ubuntu WITH CREATEDB;
ALTER ROLE ubuntu WITH LOGIN;
createdb en_pairs
psql en_pairs < ~/src/atomspace/opencog/persist/sql/multi-driver/atom.sql
time psql en_pairs < /var/lib/postgresql/en_pairs.sql
real 59m16.468s
user 0m57.480s
sys 0m12.480s
psql en_pairs
select count(*) from atoms where type=123; 123 == wordnode
276715 << there used to be about a million, here
select count(*) from atoms where type=81; 81 == evaluationlink.
37199086 37 M. Hmmm
select sum(floatvalue[3]) from valuations where type=7;
select * from valuations limit 1;
select * from atoms, valuations where valuations.type=7 and valuations.atom=atoms.uuid and atoms.type = 81 limit 1;
Total counts on all evaluationlinks.
(this should be doubled, clique and any.)
select sum(floatvalue[3]) from atoms, valuations where valuations.type=7 and valuations.atom=atoms.uuid and atoms.type = 81;
gives: 327030163 == 327M observations of links! wow!
Nonetheless, restart from scratch.
ALTER DATABASE en_pairs RENAME TO en_pairs_semi_fu;
COMMENT ON DATABASE en_pairs_semi_fu IS
'rohit-pairs and some gutenberg, after the crash';
=======================================
and again, on fanny:
time cp -pr main main-fu
real 23m29.237s
user 0m0.464s
sys 2m54.152s
/usr/lib/postgresql/9.6/bin/pg_resetxlog /var/lib/postgresql/9.6/main
/usr/lib/postgresql/9.6/bin/pg_resetxlog -f /var/lib/postgresql/9.6/main
pg_clog/0351
dd if=/dev/zero of=~/9.6/main/pg_clog/0351 bs=256K count=1
pg_multixact/offsets/057A
pg_multixact/members/0E99
Those 3 fixed it.
time pg_dumpall > everything.sql
pg_clog/033C
create all from 02CE to pg_clog/0350
#!/bin/bash
# inclusive
start=$(echo "ibase=16;2CE" | bc)
end=$(echo "ibase=16;350" | bc)
len=$(echo "$end-$start+1" | bc)
echo $len
for ((i=0;i<$len;i++))
do
j=$(echo "$start+$i" |bc)
k=$(echo "obase=16;$j" | bc)
# echo $k
dd if=/dev/zero of=~/9.6/main/pg_clog/0$k bs=256K count=1
done
cd /home2/postgresql/data-lost-dumps
time pg_dumpall > everything.sql
real 13m5.740s
user 1m13.272s
sys 1m3.764s
-rw-r--r-- 1 postgres postgres 15190388055 May 17 15:59 everything.sql
so 15 gigabytes.
sudo service postgresql stop
mv /var/lib/postgresql/9.6/main /var/lib/postgresql/9.6/main-semi-fu
/usr/lib/postgresql/9.6/bin/initdb /var/lib/postgresql/9.6/main
sudo service postgresql start
time psql < /home2/postgresql/data-lost-dumps/everything.sql
real 55m43.405s
user 1m13.960s
sys 0m21.596s
=======================================
objdump -d post-process.o
linkage_set_domain_names +0xe0
linkage->pp_info[j].num_domains = k;
for (d = pp->d_type_array[j]; d != NULL; d = d->next) k++;
char buff[] = {d->type, '\0'}
string_set_add (buff, postprocessor->string_set);
build_type_array not needed if not printed.
linkage->pp_info[j].domain_name[k] =
=====================================================
en_pairs_2016
-------------
Rohit's original pair data; old format; missing correct word-counts.
All data is from Wikipedia article random planar-tree parses.
Contains word-pair counts only (and some MI calcs in obsolete format).
STATUS: obsolete, archived.
DATE: 17 May 2017
REASON: converted to newer format.
en_pairs_2016=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+-----------+-------+-------+------------+-------------
public | atoms | table | linas | 2198 MB |
public | global | table | linas | 8192 bytes |
public | spaces | table | linas | 8192 bytes |
public | typecodes | table | linas | 48 kB |
(4 rows)
select count(atoms.stv_count) from atoms, typecodes where
atoms.type=typecodes.type and typecodes.typename='EvaluationLink';
gives: 9045489 i.e. 9M different evlinks
gives 536234371 i.e. 536M observations when sum()
gives 396255 distinct wordnodes
866733953 bytes uncompressed.
141609207 bytes compressed
md5sum en_pairs_2016.sql.bz2
28577666fb51d3dc3658688a73e8bee6 en_pairs_2016.sql.bz2
==================================================================
en_pairs_rohit
--------------
forked from en_pairs_2016, converted to modern format
This includes 8.9M word-pairs with MI on them. (But the MI is
in the old format).
STATUS: obsolete, archived.
DATE: 18 May 2017
REASON: This is included in the newer en_pairs_mst, which contains
all data here, plus data from MST parsing. Also, since its word-pairs
from wikipedia only, the data quality is mediocre.
en_pairs_rohit=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+--------------+-------+-------+------------+-------------
public | atoms | table | linas | 1735 MB |
public | atoms_backup | table | linas | 2198 MB |
public | global | table | linas | 8192 bytes |
public | spaces | table | linas | 8192 bytes |
public | typecodes | table | linas | 48 kB |
public | valuations | table | linas | 2321 MB |
public | values | table | linas | 8192 bytes |
(7 rows)
atoms_backup is a copy of atoms from en_pairs_2016
SELECT count(*) FROM atoms; <<< 19741280 = 19M atoms total
SELECT count(*) FROM atoms WHERE height=0; <<< 396313 = 396K
SELECT count(*) FROM atoms WHERE height=1; <<< 9672482 = 9.6M
SELECT count(*) FROM atoms WHERE height=2; <<< 9672472 = 9.6M
SELECT count(*) FROM atoms WHERE height=3; <<< 13
select count(atoms.uuid) from atoms,typecodes where
atoms.type=typecodes.type and typecodes.typename='EvaluationLink';
gives 9672459 i.e. 9.6M atoms. This is a 6% more than en_pairs_2016,
(which had 9.0M) because maybe it includes the partial sums??
gives 396262 WordNodes -- as before, plus 7 new test words.
select sum(valuations.floatvalue[3]) from atoms,typecodes,valuations where
valuations.atom = atoms.uuid and
atoms.type=typecodes.type and typecodes.typename='EvaluationLink';
gives 1672941118 == 1.6GB of counts so this is triple the pairs, because
it includes: left and right sums, and the total sum. Hmm. Should be
quadruple!? Yes, see below.
select * from atoms where outgoing='{143427138,143427163}';
is the ListLink (Any Any)
select * from atoms where outgoing='{152,143670792}';
select * from valuations where atom=143670793;
gives 418235277 == 418M observations of ANY,ANY
2440463583 | 53 | *-Mutual Info Key-* | PredicateNode
select count(*) from valuations where key=2440463583;
8880914 so 8.9M pairs with mutual info on them.
2957364787 bytes uncompressed -- 2.9GB
448704179 bytes compressed -- 448 MBytes
md5sum en_pairs_rohit.sql
792a8830520c8cdddb5b723656ae82cc en_pairs_rohit.sql
md5sum en_pairs_rohit.sql.bz2
6f85e89787cbf20c54167868867cf41c en_pairs_rohit.sql.bz2
==================================================================
en_pairs_mst
------------
forked from en_pairs_rohit, holds disjunct data from MST parsing.
Both word-pairs and MST parses came from wikipedia articles;
this appears to give only mediocre results.
STATUS: archived. Total junk, should be deleted.
DATE: 18 May 2017
REASON: The wikipedia-based data is mediocre.
Also:
a) it fails to separate quotation marks.
b) The any-disjuncts are junk.
c) The LgWordCset typename needs to be changed to Section
d) FATAL ERROR: all disjunct created with minus-the-MI and so are junk.
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+-------+------------+-------------
public | atoms | table | linas | 1886 MB |
public | spaces | table | linas | 8192 bytes |
public | typecodes | table | linas | 48 kB |
public | valuations | table | linas | 2421 MB |
public | values | table | linas | 8192 bytes |
(5 rows)
SELECT count(*) FROM atoms; <<< 21381856 = 21M atoms
SELECT count(*) FROM atoms WHERE height=0; <<< 396313 = 396K == rohit.
SELECT count(*) FROM atoms WHERE height=1; <<< 9763966 = 9.8M almost rohit
SELECT count(*) FROM atoms WHERE height=2; <<< 10280485 = 10.2M = 6% larger
SELECT count(*) FROM atoms WHERE height=3; <<< 941092 = 941K
The height-3 are exactly the pseudo-disjuncts.
SELECT count(atoms.uuid) FROM atoms,typecodes WHERE
atoms.type=typecodes.type AND typecodes.typename='EvaluationLink';
gives 9672459 so exactly the same as en_pairs_rohit, that's good.
'WordNode' gives 396262 -- 400K words, exactly same as before, Good.
'LgWordCset' gives 941092 - almost a million connector sets.
SELECT sum(valuations.floatvalue[3]) FROM atoms,typecodes,valuations
WHERE valuations.atom = atoms.uuid AND
atoms.type=typecodes.type AND typecodes.typename='LgWordCset';
gives 1669981 -- 1.7M observations of csets or maybe 1.8 obs per cset.
A bit disappointing....
2205218439 bytes uncompressed (2.2GB)
325948435 bytes compressed (325 MByte)
md5sum en_pairs_mst.sql
955b7fe0578ded9a266f0874e8a02d6f en_pairs_mst.sql
md5sum en_pairs_mst.sql.bz2
ee81f6b11e4629ffa30efc0950af3595 en_pairs_mst.sql.bz2
==================================================================
simil_en
--------
The similarity database published on the web, forked from an earlier
version of en_pairs_mst. Contains pairs derived only from wikipedia.
plus some number of disjuncts from mst parses.
STATUS: Total junk, should be deleted.
REASON: FATAL ERROR: all disjunct created with minus-the-MI and so are junk.
Published on web 12 May 2017
322343968 bytes compressed (322MBytes)
md5sum simil_en.sql.bz2
b9bb221069867c5f4a307670a34387c8 simil_en.sql.bz2
==================================================================
en_pairs_sim
------------
Forked from earlier version of en_pairs_mst, contains less data.
(half as many disjuncts) This is the dataset analyzed in the diary.
The simil_en is a snapshot of this; the en_pairs_sim contains
additional caches of partial sums across disjuncts, etc.
STATUS: TOTAL JUNK, should be deleted.
DATE: 21 May 2017
REASON: FATAL ERROR: all disjuncts created with minus-the-MI,
and so are junk.
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+-------+------------+-------------
public | atoms | table | linas | 1840 MB |
public | spaces | table | linas | 8192 bytes |
public | typecodes | table | linas | 48 kB |
public | valuations | table | linas | 2702 MB |
public | values | table | linas | 8192 bytes |
(5 rows)
SELECT count(atoms.uuid) FROM atoms,typecodes WHERE
atoms.type=typecodes.type AND typecodes.typename='EvaluationLink';
gives 9672459 so exactly the same as en_pairs_rohit, that's good.
'LgWordCset' gives 446204 which is as expected, (as reported in diary)
We are not creating a backup of this yet, the simil_en is good enough
backup; this dataset marches on as a working dataset.
==================================================================
en_pairs_semi_fu
----------------
Forked from en_pairs_rohit, plus more WP data, plus some gutenberg.
Has some data loss (many missing words, missing LGLink-pairs), due to
server lockup + crash failure (and fsync was off, dohhhh).
Large database, bloated with ExecutationLinks, and the clique pairs,
which just are not useful at this time. In fact, 95% of the atoms are
these junk atoms. This dataset is just wayyy too bloated to be useful.
STATUS: obsolete, archived.
DATE: 18 May 2017
REASON: abandoned. Too bloated to be useful, per above.
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+-------+------------+-------------
public | atoms | table | linas | 12 GB |
public | spaces | table | linas | 8192 bytes |
public | typecodes | table | linas | 48 kB |
public | valuations | table | linas | 9633 MB |
public | values | table | linas | 8192 bytes |
(5 rows)
SELECT count(*) FROM atoms; <<< 132172780 = 132M atoms total
SELECT count(atoms.uuid) FROM atoms,typecodes WHERE
atoms.type=typecodes.type AND typecodes.typename='EvaluationLink';
gives 37199086 = 37M evlinks compared to rohits 9M
However, of these
SELECT count(*) FROM Atoms WHERE outgoing @> ARRAY[CAST(29 AS BIGINT)];
where PredicateNode "*-Sentence Word Pair-*" uuid=29;
gives 29637409 = 29.6M clique-pairs
vs LinkGrammarRelationshipNode uuid=141; <<< 7561677 7.5M
So many of rohit's pairs were lost, and most of the new data are the
cliique pairs.
'WordNode' gives 276715 which is 2/3rds of rohits 400K so where did
all the words go? More importantly, why aren't the foreign-key
constraints violated???
'ExecutionLink' has 62936410 = 62M counted pairs
SELECT count(*) FROM atoms WHERE height=0; <<< 276972 = 277K
SELECT count(*) FROM atoms WHERE height=1; <<< 30958666 = 30M
SELECT count(*) FROM atoms WHERE height=2; <<< 100135499 = 100M
SELECT count(*) FROM atoms WHERE height=3; <<< 801643 = 802K
SELECT count(*) FROM atoms WHERE height=4; <<< 0
wtf is height=3? Answer: some (many?) random "ANY" disjuncts
from the random tree parses. (!) These have low UUID's!
Max observed UUID is 135947882
Loaded 276972 atoms at height 0 <<< so a lot of words are missing.
also: there are 508 links with missing uuids in thier outgoing set.
7721725512 bytes uncompressed (7.7GB) 17 May 2017
1207333639 bytes compressed (1.2GB)
md5sum en_pairs_semi_fu.sql.bz2
06b0acfecf9f5d4b25cca903b8f3b2a7 en_pairs_semi_fu.sql.bz2
==================================================================
en_snap
-------
STATUS: deleted
DATE: 21 May 2017
REASON: bloated junk, looks like wikipedia.
select count(*) from atoms; <<< 41511869 = 41M
A total of 106696 = 106K WordNodes
SELECT count(*) FROM Atoms WHERE outgoing @> ARRAY[CAST(141 AS BIGINT)];
gives 2786640 == 2.7M LG links.
and 9587761 = 9.6M clique pairs, and 18M ExecutionLinks.
==================================================================
en_pairs_tone
-------------
Random-planar-tree parsed books from "tranche-1" of mostly project
gutenberg books. Contains counts for word-pairs, words and disjuncts.
STATUS: poor
DATE: 24 May 2017
REASON: Multiple issues:
a) it fails to separate quotation marks.
b) The any-disjuncts are junk.
c) Missing links to ###LEFT-WALL###
d) FATAL ERROR: all disjuncts created with minus-the-MI, and so are junk.
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 909 MB |
public | spaces | table | ubuntu | 40 kB |
public | typecodes | table | ubuntu | 56 kB |
public | valuations | table | ubuntu | 591 MB |
public | values | table | ubuntu | 8192 bytes |
(5 rows)
select count(*) from atoms; <<< 10317409 = 10M atoms total
WordNodes: 139790 -- 140K
EvaluationLink : 4886362 -- 4.9M
ListLink: exactly the same.
LgWordCset: 404882 = 405K Hmm Curious.
select count(*) from valuations; <<< 5431036 = 5.4M values
SELECT sum(floatvalue[3]) FROM valuations WHERE type=7;
gives 396713166 << 397M total obs.
SELECT sum(valuations.floatvalue[3]) FROM valuations, atoms, typecodes
WHERE valuations.atom=atoms.uuid AND atoms.type=typecodes.type AND
typecodes.typename='EvaluationLink';
gives 139806259 << 140M pair observations
.. WordNode: 125696239 << 125M obs
...LgWordCset: 125707119 << 126M disjuncts
SentenceNode: 330376 == 330K unique sentences
ParseNode: 5173173 == 5.1M parses
###LEFT-WALL### seen 5173095 = ... a few missing left-walls!?
464744628 bytes uncompressed == 464MB en_pairs_tone.sql 24 May 2017
81217915 bytes compressed == 81MB en_pairs_tone.sql.bz2
md5sum:
28562003d0567039571006803b716dc0 en_pairs_tone.sql
c4a981af336075f0121ac020887bfa47 en_pairs_tone.sql.bz2
==================================================================
en_pairs_tone_mst
-----------------
Disjuncts from MST-parsed books from "tranche-1" of mostly project
gutenberg books. Based off of the en_pairs_tone. Contains MI for tone,
but not the cosine angles.
STATUS: poor, obsolete. Replaced by en_pairs_tone_mst_rev
Should be deleted.
DATE: 27 May 2017 obsoleted on 9 June
REASON: Three issues:
a) fails to separate quoation marks (inherited from tone)
b) contains junk lg-any disjuncts
c) Uses LgWordCset for for connectors and pseudo-connectors.
Which screws up statistics.
d) FATAL ERROR: all disjuncts created with minus-the-MI, and so are junk.
en_pairs_tone_mst=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 1496 MB |
public | spaces | table | ubuntu | 40 kB |
public | typecodes | table | ubuntu | 56 kB |
public | valuations | table | ubuntu | 1871 MB |
public | values | table | ubuntu | 8192 bytes |
(5 rows)
select count(*) from atoms; << 16487725 = 16.5M (6M more than baseline)
select count(*) from valuations; <<< 19330366 = 19.3M (14M more than base)
SELECT sum(valuations.floatvalue[3]) FROM valuations, atoms, typecodes
WHERE valuations.atom=atoms.uuid AND atoms.type=typecodes.type AND
valuations.type=7 AND typecodes.typename='EvaluationLink';
EaluationLink: 559225036 = 559M almost 4X more than baseline.
WordNode: 125696239 = 126M = identical to baseline;
LgWordCset: 133363005 = 134M Hm. 9M more ob.
XXX Wait. Since 134M were ANY links, that means only 9M are MST links.
Oh wow. We need to fix this.
Key 10317416 *-FrequencyKey-*
Key 10872983 *-Mutual Info Key-*
1461271903 bytes uncompressed = 1.46G
208591406 bytes compressed = 209 MB
180dff2b3d683ad5559290338b5815fa en_pairs_tone_mst.sql
e7dddab32c92b0fd0e88adfe6be1a87b en_pairs_tone_mst.sql.bz2
==================================================================
en_pairs_tone_mst_rev
---------------------
Disjuncts from MST-parsed books from "tranche-1" of mostly project
gutenberg books. Based off of the en_pairs_tone. Contains MI for tone,
but not the cosine angles.
Uses the new PseudoAnd and PseudoWordCset types.
Kills the bogus lg-any disjuncts.
Contains table summaries for pairs and csets.
STATUS: Total junk, should be deleted.
DATE: 9 June 2017
REASON: Several issues:
a) Missing links to LEFT-WALL
b) fails to separate quotation marks (inherited from tone)
c) fails to separete various other punctuation.
d) The PseudoWordCset typename needs to be changed to Section, etc.
e) FATAL ERROR: all disjunct created with minus-the-MI and so are junk.
en_pairs_tone_mst=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 1998 MB |
public | spaces | table | ubuntu | 40 kB |
public | typecodes | table | ubuntu | 56 kB |
public | valuations | table | ubuntu | 5153 MB |
public | values | table | ubuntu | 8192 bytes |
select count(*) from atoms; --- 18083439 = 18M
select count(*) from valuations; -- 36903851 = 36M
(print-matrix-summary-report asa)
Summary Report for Correlation Matrix Link Grammar ANY link Word Pairs
Left type: WordNode Right Type: WordNode Pair Type: ListLink
Rows: 138014 Columns: 139770
Size: 4886362 non-zero entries of 19290216780 possible
Fraction non-zero: 2.5331E-4 Sparsity (-log_2): 11.947
Total observations: 139806259.0 Avg obs per pair: 28.612
Entropy Total: 17.725 Left: 9.9308 Right: 9.8248
Total MI: -2.030
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 1.6405E+4 1.6483E+4
Count (l_1) 1.9950E+6 1.8453E+6 121.6 112.0
Length (l_2) 2.2904E+5 1.9881E+5 13.96 12.06
RMS Count 2.2878E+5 1.9854E+5 13.95 12.05
(print-matrix-summary-report psa)
Summary Report for Correlation Matrix Word-Disjunct Pairs (aka Connector Sets)
Left type: WordNode Right Type: LgAnd Pair Type: LgWordCset
Wildcard: (ListLink (ctv 0 0 7655886)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 133022 Columns: 1860793
Size: 3571037 non-zero entries of 247526406446 possible
Fraction non-zero: 1.4427E-5 Sparsity (-log_2): 16.081
Total observations: 7655886.0 Avg obs per pair: 2.1439
Entropy Total: 20.558 Left: 14.711 Right: 10.004
Total MI: -4.156
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 3.4890E+4 6946.
Count (l_1) 9.1434E+4 3.4736E+4 2.621 5.001
Length (l_2) 2143. 1397. 6.1427E-2 .2011
RMS Count 2107. 1368. 6.0395E-2 .1970
2652501065 Bytes = 2.65 GB uncompressed
295210074 Bytes = 295M compressed
70494c6108a2fb952003f86264e5fd35 en_pairs_tone_mst_rev.sql
4176f8600d56867301e51462da2b8b6e en_pairs_tone_mst_rev.sql.bz2
==================================================================
en_pairs_ttwo
-------------
Random parses from tranche-1 and tranche-2.
STATUS: fair
DATE: 29 May 2017
REASON: Three issues:
a) fails to separate quoation marks (inherited from tone)
b) contains junk lg-any disjuncts
c) Missing links to LEFT-WALL
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 1483 MB |
public | spaces | table | ubuntu | 40 kB |
public | typecodes | table | ubuntu | 56 kB |
public | valuations | table | ubuntu | 956 MB |
public | values | table | ubuntu | 8192 bytes |
select count(*) from atoms; <<< 16817599 = 16.8M atoms (6M more than tone)
select count(*) from valuations; <<< 8770525 = 8.8m (3.4M more)
SELECT count(atoms.uuid) FROM atoms,typecodes WHERE
atoms.type=typecodes.type AND typecodes.typename='foobar';
WordNode: 186573 = 187K (47K more)
EvaluationLink: 8047063 = 8M (3.1M more)
SELECT sum(floatvalue[3]) FROM valuations WHERE type=7;
767377888 << 767M observations = 370M more
SELECT sum(valuations.floatvalue[3]) FROM valuations, atoms, typecodes
WHERE valuations.atom=atoms.uuid AND atoms.type=typecodes.type AND
typecodes.typename='EvaluationLink';
EvaluationLink: 267913561 << 268M pair obs, more than double of tone.
WordNode: 243085829 << 243 M -- almost double
LgWordCset: << 243114608 -- almost double
SentenceNode << 795835 = 796K more than double
ParseNode << 12468055 = 12.5M parses more than double.
772033746 Bytes = 772M uncompressed.
132694670 Bytes = 133M compressed
8bbfb8f0f4956f21ec1120b550890e0f en_pairs_ttwo.sql
0b1908dfc232de4ea4e20d616cbbe290 en_pairs_ttwo.sql.bz2
==================================================================
en_pairs_ttwo_mst
-----------------
Random parses from tranche-1 and tranche-2, together with MST parses from
both tranches. The bogus ANY-disjuncs got removed, and the correct
PseudoWordCsets are used. Contains the summary stats for csets, too.
STATUS: Total junk, should be deleted.
DATE: 9 Jun 2017
REASON: Multiple issues:
a) Missing links to LEFT-WALL
b) fails to separete various punctuation
c) The PseudoWordCset typename needs to be changed to Section, etc.
d) FATAL ERROR: all disjunct created with minus-the-MI and so are junk.
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 3616 MB |
public | spaces | table | ubuntu | 40 kB |
public | typecodes | table | ubuntu | 56 kB |
public | valuations | table | ubuntu | 9220 MB |
public | values | table | ubuntu | 8192 bytes |
select count(*) from atoms; --- 30689942 = 30.6M
select count(*) from valuations; --- 63432483 = 63M
Summary Report for Correlation Matrix Link Grammar ANY link Word Pairs
Left type: WordNode Right Type: WordNode Pair Type: ListLink
Wildcard: (EvaluationLink (ctv 0 0 2.6791356e+08)
(LinkGrammarRelationshipNode "ANY")
)
Rows: 183290 Columns: 186523
Size: 8047063 non-zero entries of 34187800670 possible
Fraction non-zero: 2.3538E-4 Sparsity (-log_2): 12.053
Total observations: 267913561.0 Avg obs per pair: 33.293
Entropy Total: 17.827 Left: 9.9014 Right: 9.7626
Total MI: -1.837
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 2.3078E+4 2.2456E+4
Count (l_1) 4.0365E+6 3.4647E+6 174.9 154.3
Length (l_2) 4.2779E+5 3.4953E+5 18.54 15.56
RMS Count 4.2741E+5 3.4915E+5 18.52 15.55
Summary Report for Correlation Matrix Word-Disjunct Pairs (aka Connector Sets)
Left type: WordNode Right Type: PseudoAnd Pair Type: PseudoWordCset
Wildcard: (ListLink (ctv 0 0 14382276)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 175559 Columns: 3401462
Size: 6438484 non-zero entries of 597157267258 possible
Fraction non-zero: 1.0782E-5 Sparsity (-log_2): 16.501
Total observations: 14382276.0 Avg obs per pair: 2.2338
Entropy Total: 21.006 Left: 14.906 Right: 10.007
Total MI: -3.906
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 6.1010E+4 9176.
Count (l_1) 1.5853E+5 6.4221E+4 2.598 6.999
Length (l_2) 5694. 3476. 9.3332E-2 .3788
RMS Count 5666. 3444. 9.2870E-2 .3753
4316209411 Bytes = 4.31 GB uncompressed
517697226 Bytes = 518 MB compressed
5103471bb355c863d8082dd4fcd99955 en_pairs_ttwo_mst.sql
8cfd247413d2429273657974a42a24be en_pairs_ttwo_mst.sql.bz2
==================================================================
en_pairs_tthree
---------------
Random parses from tranche-1, tranche-2 and tranche_3.
STATUS: good
DATE: 8 Jun 2017
REASON: I think it's good. Has some extra cruft: has the bogus disjuncts
that were collected.
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 2831 MB |
public | spaces | table | ubuntu | 40 kB |
public | typecodes | table | ubuntu | 56 kB |
public | valuations | table | ubuntu | 2099 MB |
public | values | table | ubuntu | 8192 bytes |
(5 rows)
select count(*) from atoms; <<< 32125825 = 32M atoms (15M more than before)
select count(*) from valuations; <<< 16901543 = 16.9M (8.1M more)
SELECT count(atoms.uuid) FROM atoms,typecodes WHERE
atoms.type=typecodes.type AND typecodes.typename='foobar';
WordNode: 431609 = 432K (245K more) !!! that's a lot. More than double.
EvaluationLink: 15224271 = 15.2 M (7.2M more, almost double)
SELECT sum(floatvalue[3]) FROM valuations WHERE type=7;
1588822019 << 1.59 G observations (832M more, more than double)
SELECT sum(valuations.floatvalue[3]) FROM valuations, atoms, typecodes
WHERE valuations.atom=atoms.uuid AND atoms.type=typecodes.type AND
typecodes.typename='EvaluationLink';
EvaluationLink: 557197874 = 557M pair obs, nearly double of before.
WordNode: 503467934 = 503M -- more than double.
LgWordCset: 503537347 = 503M -- double
SentenceNode: 1487169 = 1.5M -- double
ParseNode: 23131695 = 23M -- double
1506102265 Bytes = 1.51 GB uncompressed
256913537 Bytes = 257 GB compressed
1cb778a18a51244c7a5c919337a38557 en_pairs_tthree.sql
df1109d706bc5d088097713d7f85188e en_pairs_tthree.sql.bz2
==================================================================
en_pairs_tthree_mi
------------------
Random parses from tranche-1, tranche-2 and tranche_3, and pair MI.
Removed the junk disjuncts from ANY parsing.
STATUS: Misleading
DATE: 9 Jun 2017
REASON: Worthless: the sign on the MI is backwards.
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 2983 MB |
public | spaces | table | ubuntu | 40 kB |
public | typecodes | table | ubuntu | 56 kB |
public | valuations | table | ubuntu | 5852 MB |
public | values | table | ubuntu | 8192 bytes |
select count(*) from atoms; -- 32593133 = 32M atoms
select count(*) from valuations; -- 50386820 = 50M valuations
Summary Report for Correlation Matrix Link Grammar ANY link Word Pairs
Left type: WordNode Right Type: WordNode Pair Type: ListLink
Wildcard: (EvaluationLink (ctv 0 0 5.5719787e+08)
(LinkGrammarRelationshipNode "ANY")
(ListLink
(AnyNode "left-word")
(AnyNode "right-word")
)
)
Rows: 424940 Columns: 431538
Size: 15224271 non-zero entries of 183377757720 possible
Fraction non-zero: 8.3021E-5 Sparsity (-log_2): 13.556
Total observations: 557197874.0 Avg obs per pair: 36.599
Entropy Total: 18.317 Left: 10.191 Right: 10.061
Total MI: -1.935
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 4.4098E+4 4.3839E+4
Count (l_1) 8.3106E+6 7.4675E+6 188.5 170.3
Length (l_2) 9.1631E+5 7.7754E+5 20.78 17.74
RMS Count 9.1591E+5 7.7713E+5 20.77 17.73
3411596887 = 3.4GB uncompressed
470255603 = 470 MB compressed
209da5230c7ac24c6ae0b8182ddb2412 en_pairs_tthree_mi.sql
ea257e5ab807b70555d182caa6d712df en_pairs_tthree_mi.sql.bz2
==================================================================
en_pairs_tthree_mst
------------------
MST parses from tranche-1, tranche-2 and tranche_3, and summary stats.
STATUS: Total junk, should be deleted.
DATE: 19 Jun 2017
REASON: Several issues:
a) Missing many links to LEFT-WALL (on tranche-1,2, and some of 3)
b) fails to separete some of the punctuation
c) FATAL ERROR: all disjunct created with minus-the-MI and so are junk.
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 5557 MB |
public | spaces | table | ubuntu | 40 kB |
public | typecodes | table | ubuntu | 56 kB |
public | valuations | table | ubuntu | 14 GB |
public | values | table | ubuntu | 8192 bytes |
select count(*) from atoms; -- 61026822 = 61M
select count(*) from valuations; -- 127753161 = 128M
Summary Report for Correlation Matrix Word-Disjunct Pairs (aka Connector
Sets)
Left type: WordNode Right Type: ConnectorSeq Pair Type: Section
Wildcard: (ListLink (ctv 0 0 31507285)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 378412 Columns: 7179578
Size: 13176250 non-zero entries of 2716838470136 possible
Fraction non-zero: 4.8498E-6 Sparsity (-log_2): 17.654
Total observations: 31507285.0 Avg obs per pair: 2.3912
Entropy Total: 21.941 Left: 15.630 Right: 10.282
Total MI: -3.971
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 1.2948E+5 1.7353E+4
Count (l_1) 3.6454E+5 1.3127E+5 2.815 7.565
Length (l_2) 9336. 5603. 7.2108E-2 .3229
RMS Count 9298. 5557. 7.1812E-2 .3203
8158333695 Bytes = 8.16 GBytes uncompressed
1018619547 Bytes = 1.02 GBytes compressed
487fbe4f636fb40e75a802dcda315d05 en_pairs_tthree_mst.sql
086f9fdae0f27a2d0e6b857d1adc901b en_pairs_tthree_mst.sql.bz2
==================================================================
en_pairs_rone
-------------
Random-planar-tree parsed books from "tranche-1" of mostly project
gutenberg books. Contains counts for word-pairs, words and disjuncts.
This is a "clean" reparse, with correct quotation mark handling,
and no record of the broken ANY-disjuncts. This fixes the issues
in en_pairs_tone.
STATUS: good
DATE: 9 June 2017
REASON: Best so far, I think. One issue:
infix-marks: dashes, ellipsis, long-dashes are not split.
It now seems that this would have been the right thing to do.
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 1024 MB |
public | spaces | table | ubuntu | 40 kB |
public | typecodes | table | ubuntu | 56 kB |
public | valuations | table | ubuntu | 1390 MB |
public | values | table | ubuntu | 8192 bytes |
select count(*) from atoms; -- 11224348 = 11M
select count(*) from valuations; -- 5679764 = 5.7M
Does not contain MI info, but it if did, it would be this:
Left type: WordNode Right Type: WordNode Pair Type: ListLink
Wildcard: (EvaluationLink (ctv 0 0 1.7409991e+08)
(LinkGrammarRelationshipNode "ANY")
(ListLink
(AnyNode "left-word")
(AnyNode "right-word")
)
)
Rows: 133792 Columns: 135156
Size: 5544578 non-zero entries of 18082791552 possible
Fraction non-zero: 3.0662E-4 Sparsity (-log_2): 11.671
Total observations: 174099906.0 Avg obs per pair: 31.400
Entropy Total: 17.667 Left: 9.8592 Right: 9.7506
Total MI: 1.9426
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 1.7398E+4 1.7417E+4
Count (l_1) 2.4875E+6 2.2524E+6 143.0 129.3
Length (l_2) 2.8788E+5 2.4347E+5 16.55 13.98
RMS Count 2.8758E+5 2.4316E+5 16.53 13.96
502761623 Bytes = 503MB uncompressed
86534772 Bytes = 86MBytes compressed
29c23c43826e8aeb5037b99acdda02a3 en_pairs_rone.sql
37f9a6ef2415ba708ffbf42f9c5948de en_pairs_rone.sql.bz2
==================================================================
en_pairs_rone_mst
-----------------
The en_pairs_rone dataset used to perform MST parses of tranche-1
books. Contains both pair-MI and cset-MI data and summaries.
STATUS: Total junk, should be deleted.
DATE: 13 June 2017
REASON: Multiple issues:
a) Missing links to LEFT-WALL
b) fails to separete various punctuation
c) The PseudoWordCset typename needs to be changed to Section, etc.
d) MI values have the float truncation bug.
e) FATAL ERROR: all disjunct created with minus-the-MI and so are junk.
en_pairs_rone_mst=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 1890 MB |
public | spaces | table | ubuntu | 40 kB |
public | typecodes | table | ubuntu | 56 kB |
public | valuations | table | ubuntu | 4601 MB |
public | values | table | ubuntu | 8192 bytes |
select count(*) from atoms; 20840690 == 20.8 M atoms
select count(*) from valuations; 42809795 == 42.8M values
(print-matrix-summary-report asa)
Summary Report for Correlation Matrix Link Grammar ANY link Word Pairs
Left type: WordNode Right Type: WordNode Pair Type: ListLink
Wildcard: (EvaluationLink (ctv 0 0 1.7409991e+08)
(LinkGrammarRelationshipNode "ANY")
(ListLink
(AnyNode "left-word")
(AnyNode "right-word")
)
)
Rows: 133792 Columns: 135156
Size: 5544578 non-zero entries of 18082791552 possible
Fraction non-zero: 3.0662E-4 Sparsity (-log_2): 11.671
Total observations: 174099906.0 Avg obs per pair: 31.400
Entropy Total: 17.667 Left: 9.8592 Right: 9.7506
Total MI: -1.943
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 1.7398E+4 1.7417E+4
Count (l_1) 2.4875E+6 2.2524E+6 143.0 129.3
Length (l_2) 2.8788E+5 2.4347E+5 16.55 13.98
RMS Count 2.8758E+5 2.4316E+5 16.53 13.96
(print-matrix-summary-report pca)
Summary Report for Correlation Matrix Word-Disjunct Pairs (aka Connector Sets)
Left type: WordNode Right Type: PseudoAnd Pair Type:
PseudoWordCset
Wildcard: (ListLink (ctv 0 0 9326663)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 131247 Columns: 2242794
Size: 4275308 non-zero entries of 294359984118 possible
Fraction non-zero: 1.4524E-5 Sparsity (-log_2): 16.071
Total observations: 9326663.0 Avg obs per pair: 2.1815
Entropy Total: 20.715 Left: 14.926 Right: 9.9403
Total MI: -4.151
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 4.1304E+4 7307.
Count (l_1) 1.1025E+5 3.9464E+4 2.669 5.401
Length (l_2) 3011. 1778. 7.2909E-2 .2433
RMS Count 2975. 1748. 7.2030E-2 .2392
3059640344 Bytes = 3.1GB uncompressed
332027863 Bytes = 332 MB compressed
4ad62e404a1473f44173800c456a1934 en_pairs_rone_mst.sql
f90db044b2fae8baeffd1e7decbc3f45 en_pairs_rone_mst.sql.bz2
==================================================================
en_pairs_rtwo
-------------
Random-planar-tree parsed books from "tranche-1" and "tranche-2".
A continutation of en_pairs_rone.
STATUS: good
DATE: 12 June 2017
REASON: One issue, see en_pairs_rone for description.
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 1596 MB |
public | spaces | table | ubuntu | 40 kB |
public | typecodes | table | ubuntu | 56 kB |
public | valuations | table | ubuntu | 1390 MB |
public | values | table | ubuntu | 8192 bytes |
Strange: the above looks wrong. \l+ says total is 6608 MB so wtf?
Maybe the rest is indexes....
select count(*) from atoms; -- 18082606 = 18M (7M more than rone)
select count(*) from valuations; -- 9135195 = 9M (3.4M more)
825319194 Bytes = 825MB uncompressed
140784957 Bytes = 141MB compressed.
bzfc3a573593f60c6e2b546da22ce915f6 en_pairs_rtwo.sql
f86114cf3f342dcd574406890a3a7168 en_pairs_rtwo.sql.bz2
==================================================================
en_pairs_rtwo_mst
-----------------
en_pairs_rtwo, with tranche-1 and tranche-2 MST parses.
STATUS: Total junk, should be deleted.
DATE: 14 June 2017
REASON: Multiple issues:
a) Missing links to LEFT-WALL
b) fails to separete various punctuation
c) The PseudoWordCset typename needs to be changed to Section, etc.
d) MI values have the float truncation bug.
e) FATAL ERROR: all disjunct created with minus-the-MI and so are junk.
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 3150 MB |
public | spaces | table | ubuntu | 40 kB |
public | typecodes | table | ubuntu | 56 kB |
public | valuations | table | ubuntu | 7081 MB |
public | values | table | ubuntu | 8192 bytes |
select count(*) from atoms; -- 34720695 = 34.7M
select count(*) from valuations; -- 72196748 = 72.2M
> (print-matrix-summary-report ala)
Summary Report for Correlation Matrix Link Grammar ANY link Word Pairs
Left type: WordNode Right Type: WordNode Pair Type: ListLink
Wildcard: (EvaluationLink (ctv 0 0 3.2123033e+08)
(LinkGrammarRelationshipNode "ANY")
(ListLink
(AnyNode "left-word")
(AnyNode "right-word")
)
)
Rows: 184681 Columns: 187712
Size: 8947405 non-zero entries of 34666839872 possible
Fraction non-zero: 2.5810E-4 Sparsity (-log_2): 11.920
Total observations: 321230333.0 Avg obs per pair: 35.902
Entropy Total: 17.770 Left: 9.8499 Right: 9.7062
Total MI: -1.786
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 2.4698E+4 2.3820E+4
Count (l_1) 4.9158E+6 4.1130E+6 199.0 172.7
Length (l_2) 5.3153E+5 4.2463E+5 21.52 17.83
RMS Count 5.3111E+5 4.2421E+5 21.50 17.81
Summary Report for Correlation Matrix Word-Disjunct Pairs (aka Connector Sets)
Left type: WordNode Right Type: PseudoAnd Pair Type:
PseudoWordCset
Wildcard: (ListLink (ctv 0 0 17048845)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 179487 Columns: 3979599
Size: 7503114 non-zero entries of 714286285713 possible
Fraction non-zero: 1.0504E-5 Sparsity (-log_2): 16.539
Total observations: 17048845.0 Avg obs per pair: 2.2722
Entropy Total: 21.146 Left: 15.105 Right: 9.9507
Total MI: -3.910
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 7.0869E+4 9633.
Count (l_1) 1.9041E+5 7.1256E+4 2.687 7.397
Length (l_2) 7130. 4017. .1006 .4171
RMS Count 7100. 3985. .1002 .4137
5034299859 Bytes = 5.0 GB uncompressed.
560626052 Bytes = 561MB compressed
688143b9152f920f52d5cc4df317d9d3 en_pairs_rtwo_mst.sql
4f36561722a0689bf213c304b61334d4 en_pairs_rtwo_mst.sql.bz2
==================================================================
en_pairs_rthree
---------------
Random-planar-tree parsed books from tranche-1, 2 and 3. Includes
the pair-MI computations. A continutation of en_pairs_rtwo.
STATUS: good
DATE: 21 June 2017
REASON: Best so far, I think. Two issues:
1) infix-marks: dashes, ellipsis, long-dashes are not split.
It now seems that this would have been the right thing to do.
2) The pair-MI values are bad; they suffer from the float-point
truncation bug. (opencog/atomspace#1295) Solution: recompute
MI before using.
3) The pair-MI values have the wrong sign.
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 3090 MB |
public | spaces | table | ubuntu | 40 kB |
public | typecodes | table | ubuntu | 56 kB |
public | valuations | table | ubuntu | 6024 MB |
public | values | table | ubuntu | 8192 bytes |
select count(*) from atoms; -- 35044156 = 35M atoms
select count(*) from valuations; -- 54073127 = 54M values
(print-matrix-summary-report asa)
Summary Report for Correlation Matrix Link Grammar ANY link Word Pairs
Left type: WordNode Right Type: WordNode Pair Type: ListLink
Wildcard: (EvaluationLink (ctv 0 0 6.3884586e+08)
(LinkGrammarRelationshipNode "ANY")
(ListLink
(AnyNode "left-word")
(AnyNode "right-word")
)
)
Rows: 428185 Columns: 433810
Size: 16442999 non-zero entries of 185750934850 possible
Fraction non-zero: 8.8522E-5 Sparsity (-log_2): 13.464
Total observations: 638845863.0 Avg obs per pair: 38.852
Entropy Total: 18.270 Left: 10.149 Right: 10.017
Total MI: -1.896
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 4.5589E+4 4.5082E+4
Count (l_1) 9.4861E+6 8.4112E+6 208.1 186.6
Length (l_2) 1.0437E+6 8.7608E+5 22.89 19.43
RMS Count 1.0433E+6 8.7564E+5 22.88 19.42
3633274138 Bytes = 3.6GBytes uncompressed = 41 bytes per atom or valuation.
509100498 Bytes = 509MBytes compressed = 5.7 Bytes/atom
286f8b84338af370465d2face539b5a7 en_pairs_rthree.sql
4cd74f0093a99e1855d61ef896738fd3 en_pairs_rthree.sql.bz2
==================================================================
en_pairs_rfour
---------------
Random-planar-tree parsed books from tranche-1, 2, 3 and 4.
A continutation of en_pairs_rthree. Does NOT contain pair-MI values.
STATUS: good
DATE: 29 June 2017
REASON: Best so far, I think. One issue:
1) infix-marks: dashes, ellipsis, long-dashes are not split.
It now seems that this would have been the right thing to do.
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 3937 MB |
public | spaces | table | ubuntu | 40 kB |
public | typecodes | table | ubuntu | 56 kB |
public | valuations | table | ubuntu | 2619 MB |
public | values | table | ubuntu | 8192 bytes |
select count(*) from atoms; -- 44676827 = 45M atoms
select count(*) from valuations; -- 22638956 = 23M valations
2083700214 Bytes = 2.1 GBytes uncompressed
354780718 Bytes compressed = 355 MBytes
md5sum en_pairs_rfour.sql
98e873b7626d22e04f914b043c14d6ec en_pairs_rfour.sql
6726298084cb8b6ad92a63419c91b416 en_pairs_rfour.sql.bz2
==================================================================
en_pairs_rfive
---------------
Random-planar-tree parsed books from tranche-1, 2, 3, 4 and 5.
A continutation of en_pairs_rfour. Does NOT contain pair-MI values.
STATUS: good/fair
DATE: 12 July 2017
REASON: Best so far, I think. One issue:
1) infix-marks: dashes, ellipsis, long-dashes are not split.
It now seems that this would have been the right thing to do.
See `en_pairs_cfive` for a cleaned-up version of this data.
(i.e. a copy of this data, but with infixed words removed.)
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 5380 MB |
public | spaces | table | ubuntu | 40 kB |
public | typecodes | table | ubuntu | 56 kB |
public | valuations | table | ubuntu | 3365 MB |
public | values | table | ubuntu | 8192 bytes |
select count(*) from atoms; -- 61069212 = 61M atoms
select count(*) from valuations; -- 30960586 = 31M values
2860579197 bytes uncompressed = 2.86 GB
488196590 bytes compressed = 488MB
dd28f22ef60b1d7f3f5854bfba793116 en_pairs_rfive.sql
2bbe479fe0caa77c5a1438568a22a4c4 en_pairs_rfive.sql.bz2
==================================================================
en_pairs_rthree_mone
--------------------
MST-parsed books from tranche-1, using pairs from rthree.
STATUS: Total junk, should be deleted.
DATE: 2 July 2017
REASON: FATAL ERROR: all disjunct created with minus-the-MI, and so are junk.
select count(*) from atoms; -- 42237812 = 42.2M i.e. 7.2M more than rthree
select count(*) from valuations; -- 58709672 = 58.7M i.e. 4.6M more than rthree
5114130466 Bytes = 5.1 GBytes uncompressed.
870711778 Bytes = 871 MB compressed
564717bc7a950ede64218cf316459cd9 en_pairs_rthree_mone.sql
8d508e61dcc2ee0ee4a739273ec37033 en_pairs_rthree_mone.sql.bz2
==================================================================
en_pairs_rfive_mone
-------------------
Random-planar-tree parsed books from tranche-1, 2, 3, 4 and 5.
MST-parsed tranche-one. Contains pair-MI values; does NOT contain
any disjunct marginals.
STATUS: fair
DATE: 16 July 2017
REASON: Two issues:
* Contains words with infixed dashes (see en_pairs_rfive)
* Contains bad disjuncts from sentences with repeated words. #2252
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 6502 MB |
public | spaces | table | ubuntu | 40 kB |
public | typecodes | table | ubuntu | 56 kB |
public | valuations | table | ubuntu | 11 GB |
public | values | table | ubuntu | 8192 bytes |
select count(*) from atoms; -- 73314120 = 73M
select count(*) from valuations; -- 104615442 = 105M
Summary Report for Correlation Matrix Link Grammar ANY link Word Pairs
Rows: 839266 Columns: 850510
Size: 30108620 non-zero entries of 713804125660 possible
Fraction non-zero: 4.2181E-5 Sparsity (-log_2): 14.533
Total observations: 1352780591.0 Avg obs per pair: 44.930
Entropy Total: 18.535 Left: 10.247 Right: 10.125
Total MI: 1.8370
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 8.0633E+4 8.0657E+4
Count (l_1) 2.0090E+7 1.8514E+7 249.2 229.5
Length (l_2) 2.2769E+6 1.9732E+6 28.24 24.46
RMS Count 2.2764E+6 1.9726E+6 28.23 24.46
9088177187 bytes uncompressed = 9.1GB
1542005839 bytes compressed = 1.54GB
f226416e374414c0ef8decb19047b596 en_pairs_rfive_mone.sql
a4f8fbc42f802dd4ffc218296d20f703 en_pairs_rfive_mone.sql.bz2
==================================================================
en_pairs_rfive_mtwo
-------------------
Random-planar-tree parsed books from tranche-1, 2, 3, 4 and 5.
MST-parsed tranche-1 and 2. Contains pair-MI values; does NOT contain
any disjunct marginals.
STATUS: fair
DATE: 17 July 2017
ISSUES: Two issues:
* Contains words with infixed dashes (see en_pairs_rfive)
* Contains bad disjuncts from sentences with repeated words. #2252
ISSUES: Dump does not have
`CREATE INDEX incoming_idx on Atoms USING GIN(outgoing);` on it.
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 7082 MB |
public | spaces | table | ubuntu | 40 kB |
public | typecodes | table | ubuntu | 56 kB |
public | valuations | table | ubuntu | 11 GB |
public | values | table | ubuntu | 8192 bytes |
select count(*) from atoms; -- 79586531 = 79M = 6M more
select count(*) from valuations; -- 108255875 = 108M = 5M more
9455749625 Bytes uncompressed = 9.45 GB
1610511576 Bytes compressed = 1.61 GB
83fa68d57ec5db8604e3487862223efb en_pairs_rfive_mtwo.sql
de95af1f2ddb7c696b19a4cc3a3e443e en_pairs_rfive_mtwo.sql.bz2
==================================================================
en_pairs_rfive_mthree
---------------------
Random-planar-tree parsed books from tranche-1, 2, 3, 4 and 5.
MST-parsed tranche-1, 2 and 3. Contains pair-MI values; does NOT
contain any disjunct marginals.
STATUS: fair
DATE: 2 August 2017
ISSUES: Two issues:
* Contains words with infixed dashes (see en_pairs_rfive)
* Contains bad disjuncts from sentences with repeated words. #2252
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 8437 MB |
public | spaces | table | ubuntu | 40 kB |
public | typecodes | table | ubuntu | 56 kB |
public | valuations | table | ubuntu | 11 GB |
public | values | table | ubuntu | 8192 bytes |
select count(*) from atoms; -- 94097585 = 94M = 15M more
select count(*) from valuations; -- 116409279 = 116M = 8M more
10315553282 Bytes uncompressed = 10.3 GB
1767006478 Bytes compressed = 1.7GB
2ec9c3544f307c7f58a49fa22762724e en_pairs_rfive_mthree.sql
ca4bcdf8d041053ac0d4bc72624a2978 en_pairs_rfive_mthree.sql.bz2
==================================================================
en_pairs_rfive_mfour
--------------------
Random-planar-tree parsed books from tranche-1, 2, 3, 4 and 5.
MST-parsed tranche-1, 2, 3 and 4. Contains pair-MI values; does NOT
contain any disjunct marginals.
STATUS: fair
DATE: 8 August 2017
ISSUES: Two issues:
* Contains words with infixed dashes (see en_pairs_rfive)
* Contains bad disjuncts from sentences with repeated words. #2252
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 9598 MB |
public | spaces | table | ubuntu | 40 kB |
public | typecodes | table | ubuntu | 56 kB |
public | valuations | table | ubuntu | 12 GB |
public | values | table | ubuntu | 8192 bytes |
select count(*) from atoms; -- 106482665 = 106M = 12M more
select count(*) from valuations; -- 123419676 = 123M = 7M more
11069504769 bytes uncompressed = 11.1 GBytes
1901439821 bytes compressed = 1.90 GBytes
eff6127b82ab205502bc1f9e8b4e139d en_pairs_rfive_mfour.sql
763ffc8bf032127501692a5ee61538ec en_pairs_rfive_mfour.sql.bz2
==================================================================
en_pairs_rfive_mfive
--------------------
Random-planar-tree parsed books from tranche-1, 2, 3, 4 and 5.
MST-parsed tranche-1, 2, 3, 4 and 5. Contains pair-MI values; does NOT
contain any disjunct marginals.
STATUS: fair
DATE: 14 August 2017
ISSUES: Two issues:
* Contains words with infixed dashes (see en_pairs_rfive)
* Contains bad disjuncts from sentences with repeated words. #2252
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 11 GB |
public | spaces | table | ubuntu | 40 kB |
public | typecodes | table | ubuntu | 56 kB |
public | valuations | table | ubuntu | 13 GB |
public | values | table | ubuntu | 8192 bytes |
select count(*) from atoms; -- 124866072 = 125M = 19M more
select count(*) from valuations; -- 133848790 = 134M = 11M more
12210323166 bytes uncompressed = 12.2GBytes
2104485691 byttes compressed = 2.10 GBytes
988be345ccaffc9312db6446693a3625 en_pairs_rfive_mfive.sql
1bd09cb8c014ec2c8e33ee7d101f4357 en_pairs_rfive_mfive.sql.bz2
==================================================================
en_pairs_rfive_mst
------------------
Same as en_pairs_rfive_mfive, but with word-disjunct marginals in it.
STATUS: fair
DATE: 6 Sept 2017
ISSUES: It contains infixed punctuation. Which is a pointless waste of
space. And which throws off stats slightly, I suppose.
ISSUES:
Contains bad disjuncts from sentences with repeated words. #2252
ISSUES: Too big to fit into RAM. Just loading disjunct marginals
requires 75GBytes. (25M csets and 25M ListLinks and umpteen valuations)
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 13 GB |
public | valuations | table | ubuntu | 34 GB |
select count(*) from atoms; -- 150550752 = 150M (so +25M for marginals)
select count(*) from valuations; -- 330716326 = 331M (+200M for marginals!!!)
Not possible to load the whole dataset, so:
(fetch-incoming-set (AnyNode "cset-word"))
(print-matrix-summary-report psa)
Summary Report for Correlation Matrix Word-Disjunct Pairs (Connector Sets)
Left type: WordNode Right Type: ConnectorSeq Pair Type: Section
Wildcard: (ListLink (ctv 0 0 77798246)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 522243 Columns: 25162434
Size: 34222079 non-zero entries of 13140905019462 possible
Fraction non-zero: 2.6042E-6 Sparsity (-log_2): 18.551
Total observations: 77798246.0 Avg obs per pair: 2.2733
Entropy Total: 22.821 Left: 20.920 Right: 10.085
Total MI: 8.1836
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 2.2054E+5 392.2
Count (l_1) 1.0180E+6 1.0483E+4 4.616 26.73
Length (l_2) 4.0497E+4 4661. .1836 11.88
RMS Count 4.0447E+4 4596. .1834 11.72
26935688497 bytes uncomepressed = 26.9 GBytes
3389925594 bytes compressed = 3.4 GBytes
61bd65b782182bf98c66abc2bc2e4612 en_pairs_rfive_mst.sql
28cce911418ca8c82428cb1e6776aa2d en_pairs_rfive_mst.sql.bz2
==================================================================
en_pairs_cfive
---------------
Random-planar-tree parsed books from tranche-1, 2, 3, 4 and 5.
A cleaned-up version of en_pairs_rfive. Words with infix marks
were removed. However, hyphenated words and apostrophised words
remain. Does NOT contain pair-MI values. Use en_pairs_cfive_mi
for dataset with word-pair marginals.
Appears to contain 27892737 (27.9M) word-pairs.
Resulting AtomSpace occupies 60GB of RAM when loaded.
Loads in about 3000 seconds. Loads 56408360 Atoms (56M Atoms)
STATUS: good
DATE: 1 Sept 2017
REASON: Best so far, I think.
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 5380 MB |
public | valuations | table | ubuntu | 3365 MB |
select count(*) from atoms; -- 56409030 = 56M = 5M fewer than rfive.
select count(*) from valuations; -- 28516287 = 28.5M = 2.5M fewer
(cog-report-counts)
((PredicateNode . 2) (ListLink . 27892737) (EvaluationLink . 27892737)
(AnchorNode . 1) (SchemaNode . 1) (WordNode . 622858) (AnyNode . 2)
(LinkGrammarRelationshipNode . 1))
2638878848 bytes uncompressed = 2.64 GBytes
453916099 bytes compressed = 454 MBytes
3f9a82d6f5ab8547d15b8a32b27896a2 en_pairs_cfive.psql
ad6e0ad034855441be507344f201f624 en_pairs_cfive.psql.bz2
==================================================================
en_pairs_cfive_mst
------------------
Same as en_pairs_rfive_mst, but with infix-punctuation words removed.
The pair marginals have been recomputed, and the dj marginals also.
STATUS: fair
DATE: 12 Sept 2017
ISSUES:
* Contains bad disjuncts from sentences with repeated words. #2252
ISSUES: Too big to fit into RAM.
Loading word-pairs and marginals requires 72GBytes.
Just loading disjunct marginals requires 75GBytes.
(25M csets and 25M ListLinks and umpteen valuations)
Recomputing disjunct marginals uses 134GB RAM.
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 13 GB |
public | valuations | table | ubuntu | 57 GB |
select count(*) from atoms; -- 139638908 = 140 M atoms (10M less)
select count(*) from valuations; -- 307109283 = 307 M values (24M less)
Summary Report for Correlation Matrix Link Grammar ANY link Word Pairs
Rows: 619168 Columns: 580804
Size: 27892737 non-zero entries of 359615251072 possible
Fraction non-zero: 7.7563E-5 Sparsity (-log_2): 13.654
Total observations: 1247457239.0 Avg obs per pair: 44.723
Entropy Total: 18.645 Left: 10.053 Right: 10.390
Total MI: 1.7982
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 5.3758E+4 7.3126E+4
Count (l_1) 1.5140E+7 1.8578E+7 281.6 254.1
Length (l_2) 1.8653E+6 1.8923E+6 34.70 25.88
RMS Count 1.8647E+6 1.8917E+6 34.69 25.87
Summary Report for Correlation Matrix Word-Disjunct Pairs (Connector Sets)
Left type: WordNode Right Type: ConnectorSeq Pair Type: Section
Wildcard: (ListLink (ctv 0 0 69447009)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 444903 Columns: 23444293
Size: 31914870 non-zero entries of 10430436288579 possible
Fraction non-zero: 3.0598E-6 Sparsity (-log_2): 18.318
Total observations: 69447009.0 Avg obs per pair: 2.1760
Entropy Total: 23.092 Left: 21.137 Right: 10.112
Total MI: 8.1569
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 2.2742E+5 367.6
Count (l_1) 9.4430E+5 7917. 4.152 21.54
Length (l_2) 2.4549E+4 2397. .1079 6.520
RMS Count 2.4499E+4 2379. .1077 6.472
24893983049 bytes uncompressed = 24.9 GB
3197324260 bytes compressed = 3.2 GB
85a872a283314d5bc1900b20da35a017 en_pairs_cfive_mst.sql
8f5eab41474c92c1a084ce9fc60f243c en_pairs_cfive_mst.sql.bz2
==================================================================
get: en_pairs_cfive and compute MI
(wps 'fetch-pairs)
(sql-close)
createdb en_pairs_cfive_mi
cat atom.sql | psql en_pairs_cfive_mi
COMMENT ON DATABASE en_pairs_cfive_mi IS
'en_pairs_cfive plus marginals (i.e. clean word-pairs, only)';
(sql-open "postgres:///en_pairs_cfive_mi")
(batch-all-pair-mi wps)
(print-matrix-summary-report wps)
Support: found num left= 619168 num right= 580804 in 680 secs << as before. Good
Finished left norm marginals in 1239 secs
Finished right norm marginals in 1175 secs
Done with wild-card count N(x,*) and N(*,y) in 2478 secs
Total count N(*,*) = 1247457239.0 = 1247457239.0 << same as before. Good
Done computing 27892737 pairs in 1599 secs
Done storing 580804 left-wilds in 868 secs
Done storing 619168 right-wilds in 702 secs
Done computing 27892737 pair MI's in 31017 secs
Finished left entropy subtotals in 1212 secs
Finished right entropy subtotals in 1227 secs
Finished left MI subtotals in 1221 secs
Finished right MI subtotals in 1229 secs
Finished with MI computations; this took 12.0 hours
en_pairs_cfive_mi
-----------------
Same as en_pairs_cfive but with marginals. So: this is a large, clean
collection of word-pairs, together with thier MI and other marginals.
Contains 27892737 (27.9M) word-pairs.
Requires 72 GB RAM for the AtomSpace when loaded.
Loads 58808293 (59M) Atoms in under an hour.
STATUS: good
DATE: 7 July 2019
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 5203 MB |
public | valuations | table | ubuntu | 7807 MB |
select count(*) from atoms; -- 58808293 = 59M Atoms
select count(*) from valuations; -- 88478107 = 88M Valuations
(cog-report-counts)
((PredicateNode . 14) (ListLink . 29092710) (EvaluationLink . 29092710)
(TypeNode . 1) (AnchorNode . 1) (SchemaNode . 1) (WordNode . 622858)
(AnyNode . 2) (LinkGrammarRelationshipNode . 1))
Summary Report for Correlation Matrix Link Grammar ANY link Word Pairs
Left type: WordNode Right Type: WordNode Pair Type: EvaluationLink
Wildcard: (EvaluationLink (ctv 0 0 1.2474572e+09)
(LinkGrammarRelationshipNode "ANY")
(ListLink
(AnyNode "left-word")
(AnyNode "right-word")
)
)
Rows: 619168 Columns: 580804
Size: 27892737 non-zero entries of 359615251072 possible
Fraction non-zero: 7.7563E-5 Sparsity (-log_2): 13.654
Total observations: 1247457239.0 Avg obs per pair: 44.723
Entropy Total: 18.645 Left: 10.053 Right: 10.390
Total MI: 1.7982
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 7.3126E+4 5.3758E+4
Count (l_1) 1.8578E+7 1.5140E+7 254.1 281.6
Length (l_2) 1.8923E+6 1.8653E+6 25.88 34.70
RMS Count 1.8917E+6 1.8647E+6 25.87 34.69
No MM^T data present
No M^TM data present
7567839839 bytes uncompressed = 7.5 GBytes
1203314284 bytes compressed = 1.2 GBytes
faf3b6ab15c26eb965bce14a5ea873c8 en_pairs_cfive_mi.sql
eed7f70d3fbdfbd7154615512aa4d29f en_pairs_cfive_mi.sql.bz2
==================================================================
zh_pairs_one
------------
Mandarin individual character pairs. Only that, without any pair stats.
Wikipedia articles only!
STATUS: good
DATE: 7 July 2017
REASON: No known serious defects. Minor issue: its wikipedia and does
not contain any fiction or action or narrative text.
zh_pairs | 4235 MB | \l+
select count(*) from atoms; -- 12003950 = 12.0M
select count(*) from valuations; -- 6081472 = 6.08 M
537357532 bytes uncompressed = 537MB
96333834 bytes compressed = 96.3MB
0718a3670ac5ea3e3f6bca3a7fb560e9 zh_pairs_one.sql
cf45874b8d353ca9332040c9acd783ae zh_pairs_one.sql.bz2
==================================================================
zh_pairs_sone
-------------
Mandarin individual character pairs. Same as above, but includes the
pair-MI stats. (After the minus-sign fix!)
STATUS: good
DATE: 12 July 2017
REASON: No known defects. Minor issue: its wikipedia and does
not contain any fiction or action or narrative text.
select count(*) from atoms; -- 12638021 = 12.6M - 600K more than above.
select count(*) from valuations; -- 19511577 = 19.5M - 13.4M more
Conclude: 6M word-pairs, total three valuations per word-pair; two added
here. What are they? a) pair-count b) pair-freq c) pair-mi
(plus additional marginals.)
Rows: 158038 Columns: 158991
Size: 5922477 non-zero entries of 25126619658 possible
Fraction non-zero: 2.3571E-4 Sparsity (-log_2): 12.051
Total observations: 728947854.0 Avg obs per pair: 123.08
Entropy Total: 18.452 Left: 10.265 Right: 10.207
Total MI: 2.0195
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 6819. 6411.
Count (l_1) 3.7360E+6 3.1244E+6 547.9 487.4
Length (l_2) 2.8408E+5 2.4156E+5 41.66 37.68
RMS Count 2.8231E+5 2.3990E+5 41.40 37.42
1647543636 bytes uncompressed = 1.65 GB
290683964 bytes compressed = 291 GB
4ab27184db51048f362fc5d6134113c6 zh_pairs_sone
39748c2abc0be436f266dec021ee4402 zh_pairs_sone.bz2
==================================================================
zen_pairs
---------
Mandarin word-pairs. Segmentation into words done by Ruiting. Does
NOT include pair-MI stats. This is the alpha-segmented-1 texts only.
STATUS: good
DATE: 17 July 2017
REASON: No known defects. Except that the dataset is too small.
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 303 MB |
public | spaces | table | ubuntu | 8192 bytes |
public | typecodes | table | ubuntu | 48 kB |
public | valuations | table | ubuntu | 187 MB |
public | values | table | ubuntu | 8192 bytes |
select count(*) from atoms; -- 3419298 = 3.4M
select count(*) from valuations; -- 1739721 = 1.7M whoa
152263232 Bytes uncompressed = 152MB
25488421 Bytes compressed = 25MBytes
fe8a58ae4f66f61c57da90a92d5787c3 zen_pairs.sql
16e651cf0a719fe9e6d0883155e9a5b8 zen_pairs.sql.bz2
==================================================================
zen_pairs_mi
------------
Mandarin word-pairs. Segmentation into words done by Ruiting.
This is the alpha-segmented-1 texts only. Includes word-pair
MI statstics.
STATUS: good
DATE: 17 July 2017
REASON: No known defects.
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 325 MB |
public | spaces | table | ubuntu | 8192 bytes |
public | typecodes | table | ubuntu | 48 kB |
public | valuations | table | ubuntu | 622 MB |
public | values | table | ubuntu | 8192 bytes |
select count(*) from atoms; -- 3659843 = 3.6M
select count(*) from valuations; -- 5700207 = 5.7M
Rows: 60123 Columns: 60143
Size: 1679575 non-zero entries of 3615977589 possible
Fraction non-zero: 4.6449E-4 Sparsity (-log_2): 11.072
Total observations: 87843720.0 Avg obs per pair: 52.301
Entropy Total: 17.473 Left: 10.253 Right: 10.107
Total MI: 2.8871
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 8170. 8702.
Count (l_1) 1.5620E+6 1.3583E+6 191.2 156.1
Length (l_2) 1.4766E+5 1.3177E+5 18.07 15.14
RMS Count 1.4721E+5 1.3142E+5 18.02 15.10
472247960 Bytes uncompressed = 472 MB
84150547 Bytes compressed = 84 MB
b4e48f00a94fa8d3d1d08e8331d0d84a9 zen_pairs_mi.sql
2c6d2667a4ff2d9f54d89765a91c3e37 zen_pairs_mi.sql.bz2
==================================================================
zen_pairs_mst
-------------
Mandarin word-pairs. Segmentation into words done by Ruiting.
This is the alpha-segmented-1 texts only. Includes word-pair
MI statstics, disjuncts and disjunct-MI. MST texts were the
same alpha-segmented-1 texts.
STATUS: good
DATE: 17 July 2017
REASON: No known defects.
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 522 MB |
public | spaces | table | ubuntu | 40 kB |
public | typecodes | table | ubuntu | 56 kB |
public | valuations | table | ubuntu | 1219 MB |
public | values | table | ubuntu | 8192 bytes |
Summary Report for Correlation Matrix Word-Disjunct Pairs (Connector Sets)
Rows: 60143 Columns: 602124
Size: 800855 non-zero entries of 36213543732 possible
Fraction non-zero: 2.2115E-5 Sparsity (-log_2): 15.465
Total observations: 1188456.0 Avg obs per pair: 1.4840
Entropy Total: 18.859 Left: 17.994 Right: 10.125
Total MI: 9.2606
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 6327. 7.399
Count (l_1) 1.8066E+4 94.14 2.855 12.72
Length (l_2) 722.8 67.89 .1142 9.175
RMS Count 695.7 62.85 .1100 8.494
select count(*) from atoms; -- 5845332 = 5.85 M atoms
select count(*) from valuations; -- 11414113 = 11.4M values
901757880 Bytes uncompressed = 902 MB
125868179 Bytes compressed = 126 MB
c4df6f2da0c476a561cbea82981a6853 zen_pairs_mst.sql
ff7cb1628cb7aad0a59c71b98a626121 zen_pairs_mst.sql.bz2
==================================================================
zen_pairs_two
-------------
Mandarin word-pairs. Segmentation into words done by Ruiting. Does
NOT include pair-MI stats. This is includes alpha-segmented-1 and
alpha-segmented-2 texts.
STATUS: good
DATE: 10 Sept 2017
REASON: No known defects. Except that the dataset is too big.
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 8947 MB |
public | spaces | table | ubuntu | 40 kB |
public | typecodes | table | ubuntu | 56 kB |
public | valuations | table | ubuntu | 103 GB |
public | values | table | ubuntu | 8192 bytes |
select count(*) from atoms; -- 101322902 = 101 M atoms
select count(*) from valuations; -- 54922685 = 55 M valuations
5140909514 bytes uncompressed = 5.14 GBytes
863704716 bytes compressed = 864 MBytes
c8ec47ba53fedad6ce0871c8a7482410 zen_pairs_two.sql
c8dc76a1bd5e68c8a6d08689f6485610 zen_pairs_two.sql.bz2
==================================================================
==================================================================
==================================================================
==================================================================
==================================================================
dropdb en_pairs_tthree en_pairs_ttwo en_pairs_rone
Plan:
load en_pairs_mst
load en_pairs_semi_fu
recompute MI for word-pairs.
fork, and restart mst counting...
(use-modules (opencog) (opencog persist) (opencog persist-sql))
(sql-open "postgres:///en_pairs_mst?user=linas")
(use-modules (opencog cogserver))
(start-cogserver "opencog3.conf")
(sql-load)
sql-load has no docs
also missing type inheritance! (but what if inheritance conflicts!?)
SQLAtomStorage.cc:1583 thrown by makeAtom
but who calls makeAtom?
load_all_atoms_cb
QLAtomStorage.cc:1360 null pointer deref.
get_recursive_if_not_exists
from :1377 PseudoPtr po(petAtom(idu)); returned null ptr for 132204462
132204462
132272910
132300252
133895580
133896697
133884903
27 missing uuids so far. They're not in rohit, so wtf?
1673 should be omp-algo
#define OC_OMP 1
setting_omp(num_threads(), 3);
num_threads
SELECT uuid, name FROM atoms WHERE type = 73
EXCEPT SELECT atoms.uuid,atoms.name FROM atoms, atoms_wtf
WHERE atoms_wtf.uuid = atoms.uuid;
2293473835 | tail-piece
1621296002 | dendropark
2146998680 | Fellers
244689200 | PVC
1438975809 | Arntzenius
709360907 | server-based
2400387025 | Kihelkonna
799467208 | Crispe's
610089230 | Byssocallis
325547913 | leveraging
179977711 | amounted
492116248 | Kampf
209124496 | Barranca
1158151745 | Ilisl
1005621585 | Caravans
1926542305 | agonising
1593374046 | Urged
SELECT * FROM Atoms WHERE outgoing @> ARRAY[CAST(2293473835 AS BIGINT)];
2293477332 | 1 | 8 | 1 | | {570739,2293473835}
2293477368 | 1 | 8 | 1 | | {1537258,2293473835}
2293477336 | 1 | 8 | 1 | | {2293473835,241}
2293477334 | 1 | 8 | 1 | | {573730097,2293473835}
2293477354 | 1 | 8 | 1 | | {4369780,2293473835}
2439795736 | 1 | 8 | 1 | | {143427138,2293473835}
2439795738 | 1 | 8 | 1 | | {2293473835,143427163}
CREATE TABLE tmp_atoms AS
SELECT uuid, space, type, height, name, outgoing
FROM atoms WHERE height=0
EXCEPT SELECT atoms.uuid, atoms.space, atoms.type, atoms.height, atoms.name, atoms.outgoing
FROM atoms, atoms_wtf
WHERE atoms_wtf.uuid = atoms.uuid;
338354
UPDATE tmp_atoms SET type= x WHERE NAME='joe';
INSERT INTO atoms_wtf
SELECT uuid, space, type, height, name, outgoing
fuck it.
load rohit first, then load semi_fu then load mst
(use-modules (opencog) (opencog persist) (opencog persist-sql))
(use-modules (opencog nlp))
(use-modules (opencog cogserver))
(sql-open "postgres:///en_pairs_rohit?user=linas")
(start-cogserver "opencog2.conf")
(sql-load)
OK, that's loading in parallel yayyy! peaked over 17K/second
dropped down to 11.3K per sec
Need to load 19M for this one <<< 19741280
... in 1691 seconds (11674 per second)
... using 27GB RAM
... in approx 260 cpu-minutes.
OK, so can hit rates over 20K atoms/sec during loads, when the
DB is in RAM (or maybe SSD). Drops to 300 atoms/sec when DB is
on spinning disk. Ouch!
Hmmm .. 20K/sec if there are not values on the atom.
10K/sec if there are values ...
So when bulk loading....
calls load_all_atoms_cb
which does get values. get_atom_values
get_all_values_cb
(sql-close)
(sql-open "postgres:///en_pairs_semi_fu?user=linas")
(sql-load)
loaded 132172832 in 9938 seconds (13299 per second)
Atomspace holds 146629585 atoms in 171GBytes RAM. Ouch. 1159:08 CPU
again: 953:04.24 CPU again 949:04.10
(sql-close) <<< 1163:38 CPU
(sql-open "postgres:///en_pairs_semi_ok?user=linas")
(sql-store)
Store rate is 20K atoms/minute! 1.2M/hour --> 100 hours (4 days) to
store fu. Lets try again. parallelize too
So... store queues not being used. Lots of fetches, but these
seem pointless ? FIXED
Max UUID is 2 !!!!?? FIXED
OK, better: 1.3K/sec, about 100K/min - 1470 min = 24 hours
num_get_links and write queues not used. FIXED
... and max uuid is still 2. FIXED
should call storeAtom(h, false) FIXED
OK, store queues are cranking away, 100% i/o bound. 1.1K/sec
--> 38 hours to complete! Ouch.
... dropped to 603 per second after 15% complete.
Plan B:
copy en_pairs_semi_fu to en_pairs_wiki
load en_pairs_mst
close
open en_pairs_wiki
=================================
/home/ubuntu/src/atomspace/opencog/persist/sql/multi-driver/ll-pg-cxx.cc:113)
PQresult message: ERROR: could not read block 479803 in file
"base/16446/16452.3": read only 0 of 8192 bytes
SELECT * FROM Atoms WHERE type = 82 AND outgoing = '{23, 7830585, 9}';
SELECT * FROM Atoms WHERE type = 137 AND outgoing = '{232023, 14}';
SELECT * FROM Atoms WHERE type = 137 AND outgoing = '{2433080, 852}';
SELECT * FROM Atoms WHERE type = 137 AND outgoing = '{9196089, 2499}';
SELECT * FROM Atoms WHERE type = 137 AND outgoing = '{3250528, 10}';
SELECT * FROM Atoms WHERE type = 82 AND outgoing = '{23, 30218901, 9}';
/usr/lib/postgresql/9.5/bin/pg_resetxlog /var/lib/postgresql/9.5/main
Crap. Total data corruption.
Restart, again
====================================
/etc/postgresql/9.6/main/postgresql.conf
increase shared_buffers to 24GB
shared_buffers = 24GB
max_worker_processes = 24
fsync = on
effective_cache_size = 128GB
increse /etc/sysctl.d/30-postgresql-shm.conf too
grep ^VmPeak /proc/4170/status
hugpages may be too big... but anonhugepages shoud work fine
no commit delay: 300%, 70% idle 6% wait 10 loadavg 10m val/h 20a/h
10K commit
11:18
=====================================================
-- Restart counting. Done, May 21 06:15 UTC
As of: 2017-05-21 18:59:08 i.e. 12.7 hours later,
284 articles
select count(*) from atoms; <<< 1832992 = 1.8M
SELECT count(atoms.uuid) FROM atoms,typecodes WHERE
atoms.type=typecodes.type AND typecodes.typename='EvaluationLink';
gives 840285 = 840K and equal number of ListLink
39772 = 39K WordNode
SELECT sum(floatvalue[3]) FROM valuations WHERE type=7;
aka
SELECT sum(valuations.floatvalue[3]) FROM valuations,typecodes
WHERE valuations.type=typecodes.type AND typecodes.typename='CountTruthValue';
gives 39459145 = 39M counts = 3.1M counted per hour = 863 pairs/second
----------------
As of 2017-05-22 05:56:04 i.e. 24 hours later
497 articles .. really 500 a day, avg.
select count(*) from atoms; << 2829421 = 2.8M
1309922 = 1.3M EvaluationLinks
53781 = 54K words
67789172 = 68M counts = 2.8M/hour = 784 pairs/sec
----------------
As of 2017-05-22 23:58:10
888 articles
select count(*) from atoms; <<< 4239364 = 4.2M
1980561 = 2M eval links
71595 = 71K words
116752710 = 117M counts in 42 hours = 2.8M/hour
or (117M-68M) / 18 hours = 2.7M/hour
------------------
Restart from scratch 2017-05-23 00:10:00
at 01:51 have 129302108= 129M so 12M in 1.7 hours = 7M/hour
So now its more than 2x faster. almost 3x, with rounding errors
Yayyyy!!!
05:30 152806491 =153 -117 = 36M obs in 5.4 hours = 6.7M obs/hour
= 1854 obs/second
----------------------------------------------------
Why is it slowing?
---------------
1) why is valuation updates about 3.5x of sum(3)?
2) why is get-got almost 2x bigger than delta for nodes? A:
3) why is get-got almost 1.5x bigger than delta for links? A: LgAnd's
One sentence:
sum valuations=245 but 461 valuation updates!?
5 WordNodes
7 ListLinks
7 EvaluationLinks
----------------- expect 19 atoms
got 41 atoms... many are LgWordCsets, LgAnd's OK. total stores is perfect.
num_get_nodes=34 huh???
this time, 922 updates!!!! wtf
why are fecth and store counts same?? maybe cause not syn
cog-logger-set-sync!
There were 23 fetches logged:
1 sent
1 parse
5 word
-- total 7 nodes
?? where are the connectors? not fetched, cause not incr'ed so OK.
7 eval
9 lgwordcset
-- total 16 links
ListLinks, not fetched cause not incr'ed. so OK.
245 stores. again 922 updates of values...
SentenceNode got update twice...
ParseNode twice ...
2x for all the words ...
Then lots of updates, even when not teh main target of the store...
Now, only 490 updates. Still too many. should be 245 ... !?
barrier is broken.. wewell its not forcing a flush
does close have a barrier?
well, busy_writers is a bad idea... because it races.
but queue size was wrong.
OK, so 105 of 245 ... 123
why is fillfraction 55K ? Because drain is no longer being hit.
=====================================================
TODO:
-- verify integrity of the database.
-- verify that MST has all the pairs that rohit does. OK.
-- Update the dump status above; copy it over. OK, Done.
-- Create the missing status on en_pairs_sim from the paper.
(use-modules (opencog) (opencog persist) (opencog persist-sql))
(use-modules (opencog nlp) (opencog nlp learn))
(sql-open "postgres:///en_pairs_sim?user=linas")
(use-modules (opencog cogserver))
(start-cogserver "opencog2.conf")
(define pca (make-pseudo-cset-api))
(batch-pairs pca)
So now word-pair loading is out of control...
loadeed csets in 553 seconds, same as before, but at 2400% cpu
Crap. Below is a disaster.
commit_delay = 0
synchronous_commit = local
Stored 1000 of 291637 lefties in 270 secs (3.7037037037037037 stores/sec)
Stored 2000 of 291637 lefties in 320 secs (3.125 stores/sec)
Try again:
commit_delay = 0
synchronous_commit = off
Stored 20000 of 291637 lefties in 45 secs (444.44444444444446 stores/sec)
Stored 40000 of 291637 lefties in 47 secs (425.531914893617 stores/sec)
Stored 60000 of 291637 lefties in 62 secs (322.5806451612903 stores/sec)
OK, so we are back in business
======================================================================
(add-pair-mi-api pca) 'compute-left-fractional
right-wild gives dj
/tmp/ranked-word-ent.dat
(define cs (cog-incoming-by-type (Word "possible") 'LgWordCset))
(filter (lambda (cset) (< 10 (get-count cset))) cs)
possible N=3 has 1: a-
N=2 has 5: (is- that+) (it- to+)
N=1 = 106
education: 100 N=3 has 1 N=2 has 7
lost 100 1 and 7
days 98 2 7
children 101 1 9
(define bins (second binned-pair-mi))
(fold + 0 (array->list bins))
-----------------------------------------------------------
1 word was observed 38977 times
37502
32517
22632
differences:
there is a left and a right diff.
The wild-card support needs to be a union
overload 'left-stars 'right-stars 'pair-count 'item-pair
this can be a generic map function of any kind! not just a diff!
=======================================
There is no value for key (PredicateNode "*-FrequencyKey-*")
en_pairs_tone_mst: spinnning drives:
Done storing 4886362 pair MI's in 6903 secs
=======================================
OK, so:
pca is the pseudo-cset-api
pma adds special stars to it
pdi will provide utilities
pdi takes pma and wraps it with default stars
default stars should see the pma under it.
15265 pos only
15420 the all by itself
7168 for "a"
total 15420+7168 -256 = 22588 - 256 = 22332 which is what union reports...
unions 22260 < so wtf is this? oh 72 obse are identical!
256 interested! sum = 22260+256 = 22516
Hmmm.
- (filter!
- (lambda (wrd)
- (not (null? (get-cset-vec wrd))))
- WORD-LIST)
(cog-incoming-by-type ITEM 'LgWordCset)
Error: do_store_single_atom: Maxiumum Link size is 330.
(/home/ubuntu/src/atomspace/opencog/persist/sql/multi-driver/SQLAtomStorage.cc:1202)
Aborted
relex crash always on this:
“What the hell is happening here?!”
split-books/gneiosric-ab three out of three crashes
cat gneiosric-ab | ./split-sentences.pl > g-split
./ss-one.sh en gneiosric-ab localhost 17005
cat g-split | ./submit-one.pl localhost 17005 observe-text breaks
g2 breaks
g2b breaks
g2guess is OK
g2bb is OK
g2ba breaks
g2baa is OK
g2bab breaks
g2baba breaks
g2babaa is OK
g2babab breaks
g2bababa breaks
g3 is OK
cat g4 |nc localhost 4445 breaks!
echo 「...」 |nc localhost 4445 breks
link-parser works ... but eats the trailing character...!?
and generates zillions of parses!!??
of which most are identical...
relex definitely crashes...
linkage_set_domain_names+0xe0
6 links,
5th link crashes..
dtype array holds junk
dtsz supposed to be the actual size of valid entries
ppn->dtsz
chk_d_type reallocs but does not memset....
dtsz is too small.!!
chk_d_type
chk_d_type(pp->pp_node, numli...
do not set the pipinfo if domains is zero.
so do not free. either
*** START OF THIS PROJECT GUTENBERG EBOOK
***START OF THE PROJECT GUTENBERG EBOOK
END OF THIS PROJECT GUTENBERG EBOOK
*** START: FULL LICENSE ***
The Full Project Gutenberg License
sed -n '/PROJECT GUTENBERG EBOOK/,$p' filename
schweben
getp "is a"
54756-0.txt
grep -l 大
grep -l Anmerkungen *
purezza
quella
if
Langauge:
54637-0.txt
54640-0.txt
54652-0.txt
54655-0.txt
54660-0.txt
54665-0.txt
54667-0.txt
54668-0.txt
54672-0.txt
54675-0.txt
54676-0.txt
54679-0.txt
54681-0.txt
54706-0.txt
54711-0.txt
54714-0.txt
54719-0.txt
54721-0.txt
54724-0.txt
54729-0.txt
guile
(use-modules (opencog) (opencog persist) (opencog persist-sql))
(use-modules (opencog nlp) (opencog nlp learn))
(use-modules (opencog cogserver))
(start-cogserver "opencog2.conf")
(sql-open "postgres:///en_pairs_tone_mst?user=ubuntu")
(sql-load)
(sql-stats)
(sql-close)
Finished loading 16487725 atoms in total in 2212 seconds (7453 per second)
started at 20K/sec .. towards end, was 75% system cpu... why????
21GB ram = 1400Bytes/atom. as usual. Need a diet plan.
(sql-open "postgres:///en_pairs_ttwo?user=ubuntu")
(sql-load)
(sql-stats)
(sql-close)
Finished loading 16817599 atoms in total in 991 seconds (16970 per second)
Much faster because most atoms are in atomspace already.
sql-stats: Atomspace holds 22987920 atoms ... and is 26GB in size.
(sql-open "postgres:///en_pairs_ttwo_mst?user=ubuntu")
(sql-store)
(sql-stats)
(sql-close)
Initial rate is (3389 per second) on SSD which is 10x the spinning-disk
Peaked at (3546 per second) half-way throough
Dropped to (2537 per second) overall average.
version but is still pathetically slow.
iotop shows 3.2 MBytes/sec and about 770 tps
so where is the bottleneck?
Bottleneck is still disk i/o, even though its SSD:
how can I tell?
1) guile is under 90% cpu
2) 20+ postgres procs show 30% cpu, each
3) sql-stats shows 8 threads busy draining.
4) loadavg shows 6
5) top shows > 2.0% wait, (on 24 cpus!) 4% system 18% user 75% idle
viz about 0.5 cou wait, 1 cpu system, 4cpu's busy
and since guile is less than 1 cpu, the postgres take up the
remaining cpu. Since 18 cpus are idle, we must be stalled in i/o.
So, somehow still stuck in i/o waiting for postgres to flush to disk.
6) iotop shows 60 to 100 MBytes/sec, highly variable.
wait sometimes spikes to 15% (and io dropts to zero)
sometimes rate spikes to 300MB/sec! but rarely.
7) Launching any-parsing does not seem to affect iowait!
Nore does it change store performance at all!!!
but it does raise the write rates and the tps
?? Is it postgres indexing? Can we suspend that while storing?
--------- meanwhile continue pair-parsing
launch tranche-3 any-parsing Tue May 30 04:39:44 UTC 2017
Done
en_pairs_tthree
(sql-open "postgres:///en_pairs_tthree?user=ubuntu")
(fetch-all-words)
(define ala (make-any-link-api))
(define asa (add-pair-stars ala))
(asa 'fetch-pairs)
(batch-all-pair-mi asa)
Finished with MI computations; this took 11.9 hours
(print-matrix-summary-report asa)
Summary Report for Correlation Matrix Link Grammar ANY link Word Pairs
Left type: WordNode Right Type: WordNode Pair Type: ListLink
Wildcard: (EvaluationLink (ctv 0 0 5.5719787e+08)
(LinkGrammarRelationshipNode "ANY")
(ListLink
(AnyNode "left-word")
(AnyNode "right-word")
)
)
Rows: 424940 Columns: 431538
Size: 15224271 non-zero entries of 183377757720 possible
Fraction non-zero: 8.3021E-5 Sparsity (-log_2): 13.556
Total observations: 557197874.0 Avg obs per pair: 36.599
Entropy Total: 18.317 Left: 10.191 Right: 10.061
Total MI: -1.935
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 4.4098E+4 4.3839E+4
Count (l_1) 8.3106E+6 7.4675E+6 188.5 170.3
Length (l_2) 9.1631E+5 7.7754E+5 20.78 17.74
RMS Count 9.1591E+5 7.7713E+5 20.77 17.73
OK, So I think we are done with en_pairs_tthree.
----- anyway, recompute pair mi and resume mst-parsing
en_pairs_ttwo_mst
(define ala (make-any-link-api))
(define asa (add-pair-stars ala))
(asa 'fetch-pairs)
(batch-all-pair-mi asa)
Final store: about (1250.0 pairs/sec)
Done storing 8047063 pair MI's in 6905 secs
sql-stats: Atomspace holds 23178644 atoms = 23M = 35GBytes
Now can start tranche-2 MST parsing.
Done. Wow. That was fast! did anything actually happen?
Well, after a series of issues, yes, it looks like this run was good.
select count(*) from atoms; <<< 27638783 = 27M
get-all-words -- 188793 vs. tone: 139790 Seems OK.
all-cset-words -- 83131 vs tone 139790 ?? Crazy talk WTF? did we not
load correctly?
disjuncts -- 1670883 vs tone 1860796
wtf somethng broke....
(define (filter-words-with-csets WORD-LIST)
(filter! (lambda (wrd) (not (null? (get-cset-vec wrd)))) WORD-LIST))
Recompute the matrices again.
(sql-open "postgres:///en_pairs_ttwo_mst?user=ubuntu")
(fetch-all-words)
(define ala (make-any-link-api))
(define asa (add-pair-stars ala))
(asa 'fetch-pairs) << 1077 1154
(batch-all-pair-mi asa)
Support: found num left=183290 num right=186523 in 389 secs
Done with wild-card count N(x,*) and N(*,y) in 733 secs
Done computing N(*,*) total-count=267913561.0 in 23 secs
Done computing 8047063 pairs in 480 secs
Done computing 183290 left-wilds in 25 secs
Done storing 186523 left-wilds in 86 secs
Done storing 183290 right-wilds in 95 secs
....
Going to compute the left, right and total entropy
Finished left norm totals in 100 secs
Finished right norm totals in 106 secs
Done computing totals; start saving wildcards
terminate called after throwing an instance of
'opencog::RuntimeException'
what(): Failed to execute!
(/home/ubuntu/src/atomspace/opencog/persist/sql/multi-driver/ll-pg-cxx.cc:113)
arghhh. The valuation table was missing a lock.
Fixed in 32d8ae4be6164a9c3594eab05f2d5af5e2c9fec6
Finished with MI computations; this took 6.25 hours
Summary Report for Correlation Matrix Link Grammar ANY link Word Pairs
Left type: WordNode Right Type: WordNode Pair Type: ListLink
Wildcard: (EvaluationLink (ctv 0 0 2.6791356e+08)
(LinkGrammarRelationshipNode "ANY")
)
Rows: 183290 Columns: 186523
Size: 8047063 non-zero entries of 34187800670 possible
Fraction non-zero: 2.3538E-4 Sparsity (-log_2): 12.053
Total observations: 267913561.0 Avg obs per pair: 33.293
Entropy Total: 17.827 Left: 9.9014 Right: 9.7626
Total MI: -1.837
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 2.3078E+4 2.2456E+4
Count (l_1) 4.0365E+6 3.4647E+6 174.9 154.3
Length (l_2) 4.2779E+5 3.4953E+5 18.54 15.56
RMS Count 4.2741E+5 3.4915E+5 18.52 15.55
(define pca (make-pseudo-cset-api))
(define psa (add-pair-stars pca))
(psa 'fetch-pairs)
Elapsed time to load csets: 2454 secs
(batch-all-pair-mi psa)
Finished with MI computations; this took 4.55 hours
(print-matrix-summary-report psa)
Summary Report for Correlation Matrix Word-Disjunct Pairs (aka Connector Sets)
Left type: WordNode Right Type: LgAnd Pair Type: LgWordCset
Wildcard: (ListLink (ctv 0 0 2.5749688e+08)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 186573 Columns: 3401465
Size: 6975371 non-zero entries of 634621529445 possible
Fraction non-zero: 1.0991E-5 Sparsity (-log_2): 16.473
Total observations: 257496884.0 Avg obs per pair: 36.915
Entropy Total: 11.868 Left: 2.4858 Right: 9.8057
Total MI: -0.423
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 5.8326E+4 1.7086E+5
Count (l_1) 3.2094E+6 9.5093E+7 55.03 556.6
Length (l_2) 2.4075E+6 1.2274E+7 41.28 71.84
RMS Count 1.8036E+6 1.2272E+7 30.92 71.83
OK, but this is wrong, we have to redoit.
------
Hand-repair
INSERT INTO typecodes VALUES (229, 'PseudoAnd');
INSERT INTO typecodes VALUES (230, 'PseudoWordCset');
select count(*) from atoms where type = 174; -- 174==PseudoConnector
gives 241601
CREATE TEMP TABLE con AS SELECT uuid FROM atoms WHERE type=174;
select count(*) from atoms, con where atoms.outgoing[1]=con.uuid;
gives 3401462
-- 167 is LgAnd ....
select count(*) from atoms, con where atoms.outgoing[1]=con.uuid AND atoms.type=167;
give same result
-- change all of them to PseudoAnd
update atoms set type=229 FROM con where outgoing[1]=con.uuid and type=167;
How many LgAnds are left?
select count(*) from atoms where type=167; -- gives 3
select count(*) from atoms where type = 165; -- 165==LgConnector
gives 2
CREATE TEMP TABLE lgs AS SELECT uuid FROM atoms WHERE type=167;
select count(*) from atoms, lgs where atoms.outgoing[2]=lgs.uuid;
536890 -- which is what we well need to delete
-- now the rest.
CREATE TEMP TABLE pan AS SELECT uuid FROM atoms WHERE type=229;
-- 229 is the new PseudoAnd
select count(*) from pan; 3401462
select count(*) from atoms, pan where atoms.outgoing[2]=pan.uuid;
gives 9839946
-- 169 is LgWordCset
select count(*) from atoms, pan where atoms.outgoing[2]=pan.uuid AND atoms.type=169;
gives 6438484
select count(*) from atoms, pan where atoms.outgoing[2]=pan.uuid AND atoms.type=21;
gives 3401462 which is as before. == (- 9839946 6438484)
so that makes sense too.
update atoms set type=230 FROM pan where outgoing[2]=pan.uuid and type=169;
------
May as well drop the bogus connectors, too?? Yes, we should.
select count(*) from atoms where type=169;
536887 -- exactly as expected.
CREATE TEMP TABLE lws AS SELECT uuid FROM atoms WHERE type=169;
select count(*) from valuations, lws where valuations.atom=lws.uuid;
delete from valuations using lws where atom =lws.uuid;
delete from atoms where type=169;
DONE. Now redo the summaries.
redoing cset MI took
Finished with MI computations; this took 8.28 hours
(print-matrix-summary-report psa)
Summary Report for Correlation Matrix Word-Disjunct Pairs (aka Connector Sets)
Left type: WordNode Right Type: PseudoAnd Pair Type: PseudoWordCset
Wildcard: (ListLink (ctv 0 0 14382276)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 175559 Columns: 3401462
Size: 6438484 non-zero entries of 597157267258 possible
Fraction non-zero: 1.0782E-5 Sparsity (-log_2): 16.501
Total observations: 14382276.0 Avg obs per pair: 2.2338
Entropy Total: 21.006 Left: 14.906 Right: 10.007
Total MI: -3.906
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 6.1010E+4 9176.
Count (l_1) 1.5853E+5 6.4221E+4 2.598 6.999
Length (l_2) 5694. 3476. 9.3332E-2 .3788
RMS Count 5666. 3444. 9.2870E-2 .3753
OK,
so en_pairs_ttwo_mst seems to contain valid data.
So we are done, here, and its time to construct the next tranche.
---------- Try again. Here's what tone_mst says:
(use-modules (opencog) (opencog persist) (opencog persist-sql))
(use-modules (opencog nlp) (opencog nlp learn))
(use-modules (opencog analysis))
(sql-open "postgres:///en_pairs_tone_mst?user=ubuntu")
(fetch-all-words)
(define ala (make-any-link-api))
(define asa (add-pair-stars ala))
(asa 'fetch-pairs) << 672 secs
(print-matrix-summary-report asa) ; crash, of course. So rebuild .
(batch-all-pair-mi asa)
Start computing the basis
Support: found num left=138014 num right=139770 in 247 secs
Done with wild-card count N(x,*) and N(*,y) in 419 secs
Done computing N(*,*) total-count=139806259.0 in 17 secs
Going to do individual pair frequencies
Done computing 4886362 pairs in 268 secs
Start computing log P(*,y)
Done computing 138014 left-wilds in 18 secs
Done computing 139770 right-wilds in 17 secs
Done storing 139770 left-wilds in 67 secs
Done storing 138014 right-wilds in 64 secs
Done computing 4886362 pair MI's in 2051 secs
Done storing 4886362 pairs in 3378 secs
Finished left entropy subtotals in 191 secs
Finished right entropy subtotals in 184 secs
Finished left MI subtotals in 193 secs
Finished right MI subtotals in 181 secs
Finished left support subtotals in 479 secs
Finished right support subtotals in 456 secs
Finished left norm totals in 86 secs
Finished right norm totals in 85 secs
Done storing 139770 left-wilds in 126 secs
Done storing 138014 right-wilds in 132 secs
Finished with MI computations; this took 2.388 hours
(print-matrix-summary-report asa)
Summary Report for Correlation Matrix Link Grammar ANY link Word Pairs
Left type: WordNode Right Type: WordNode Pair Type: ListLink
Rows: 138014 Columns: 139770
Size: 4886362 non-zero entries of 19290216780 possible
Fraction non-zero: 2.5331E-4 Sparsity (-log_2): 11.947
Total observations: 139806259.0 Avg obs per pair: 28.612
Entropy Total: 17.725 Left: 9.9308 Right: 9.8248
Total MI: -2.030
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 1.6405E+4 1.6483E+4
Count (l_1) 1.9950E+6 1.8453E+6 121.6 112.0
Length (l_2) 2.2904E+5 1.9881E+5 13.96 12.06
RMS Count 2.2878E+5 1.9854E+5 13.95 12.05
Now do it again, for the disjuncts!
(sql-clear-stats)
(define pca (make-pseudo-cset-api))
(define psa (add-pair-stars pca))
(psa 'fetch-pairs)
Elapsed time to load csets: 1484 secs
(batch-all-pair-mi psa)
Finished with MI computations; this took 3.19 hours
(print-matrix-summary-report psa)
Summary Report for Correlation Matrix Word-Disjunct Pairs (aka Connector Sets)
Left type: WordNode Right Type: LgAnd Pair Type: LgWordCset
Wildcard: (ListLink (ctv 0 0 1.33363e+08)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 139790 Columns: 1860796
Size: 3975919 non-zero entries of 260120672840 possible
Fraction non-zero: 1.5285E-5 Sparsity (-log_2): 15.998
Total observations: 133363005.0 Avg obs per pair: 33.543
Entropy Total: 11.921 Left: 2.4678 Right: 9.8636
Total MI: -0.410
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 3.3665E+4 1.2870E+5
Count (l_1) 1.6849E+6 5.1241E+7 50.05 398.2
Length (l_2) 1.2139E+6 6.4911E+6 36.06 50.44
RMS Count 1.0132E+6 6.4895E+6 30.10 50.42
===============
Arghhhh Sanity check:
the LgWordCset's in these files are:
(LgWordCset
(WordNode "this")
(LgAnd
(LgConnector
(LgConnectorNode "ANY")
(LgConnDirNode "-")
(LgConnMultiNode "@")
)
vs.
(LgWordCset
(WordNode "playing")
(LgAnd
(PseudoConnector
Ugh. The load is polluted!
Double check tone_mst...
dd(use-modules (opencog) (opencog persist) (opencog persist-sql))
(use-modules (opencog nlp) (opencog nlp learn) (opencog analysis))
(sql-open "postgres:///en_pairs_tone_mst?user=ubuntu")
(fetch-all-words)
(define pca (make-pseudo-cset-api))
(define psa (add-pair-stars pca))
(psa 'fetch-pairs)
So, tone, by itself, in sql, has:
LgWordCset: 404882 = 405K unique csets and
LgWordCset: 125707119 << 126M disjunct observations.
tone_mst has via SQL:
3975919 = 3.9M csets
(cog-count-atoms 'LgWordCset) <<< 3975919 exactly same as above
(sql-close)
(cog-delete-recursive (LgConnMultiNode "@"))
(cog-count-atoms 'LgWordCset) <<< 3571037 == 3.57M atoms Hmmm!
difference = (- 3975919 3571037) = 404882 which is exactly tone by
itself. Excellent. Now recompute the cset stats....
(sql-open "postgres:///en_pairs_tone_mst?user=ubuntu")
(batch-all-pair-mi psa)
Finished with MI computations; this took 4.15 hours
(print-matrix-summary-report psa)
So this is very different than before. Fuck.
(print-matrix-summary-report psa)
Summary Report for Correlation Matrix Word-Disjunct Pairs (aka Connector Sets)
Left type: WordNode Right Type: LgAnd Pair Type: LgWordCset
Wildcard: (ListLink (ctv 0 0 7655886)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 133022 Columns: 1860793
Size: 3571037 non-zero entries of 247526406446 possible
Fraction non-zero: 1.4427E-5 Sparsity (-log_2): 16.081
Total observations: 7655886.0 Avg obs per pair: 2.1439
Entropy Total: 20.558 Left: 14.711 Right: 10.004
Total MI: -4.156
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 3.4890E+4 6946.
Count (l_1) 9.1434E+4 3.4736E+4 2.621 5.001
Length (l_2) 2143. 1397. 6.1427E-2 .2011
RMS Count 2107. 1368. 6.0395E-2 .1970
---------- repair:
INSERT INTO typecodes VALUES (229, 'PseudoAnd');
INSERT INTO typecodes VALUES (230, 'PseudoWordCset');
select count(*) from atoms where type = 174; -- 174==PseudoConnector
gives 182912
CREATE TEMP TABLE con AS SELECT uuid FROM atoms WHERE type=174;
select count(*) from atoms, con where atoms.outgoing[1]=con.uuid;
gives 1860793
-- 167 is LgAnd ....
select count(*) from atoms, con where atoms.outgoing[1]=con.uuid AND atoms.type=167;
give same result
-- change all of them to PseudoAnd
update atoms set type=229 FROM con where outgoing[1]=con.uuid and type=167;
How many LgAnds are left?
select count(*) from atoms where type=167; -- gives 3
wtf??? but this is exactly what is in tone, also!!!!!!!
XXX so any disjunct counting is borken! fuuck.
select count(*) from atoms where type = 165; -- 165==LgConnector
gives 2
wtf. Also same as tone.
CREATE TEMP TABLE lgs AS SELECT uuid FROM atoms WHERE type=167;
select count(*) from atoms, lgs where atoms.outgoing[2]=lgs.uuid;
404882 -- which is exactly what got deleted. Very good.
-- now the rest.
CREATE TEMP TABLE pan AS SELECT uuid FROM atoms WHERE type=229;
-- 229 is the new PseudoAnd
select count(*) from pan;
so its 1860793 which is same as before.
select count(*) from atoms, pan where atoms.outgoing[2]=pan.uuid;
gives 5431830
-- 169 is LgWordCset
select count(*) from atoms, pan where atoms.outgoing[2]=pan.uuid AND atoms.type=169;
gives 3571037
so what the heck has pseudoAnd as second, but is not an LgWordCset ????
its type=21 ... ListLink.
(ListLink ; 17184167 --- I guess we stored the counts here. Hmmm. why?
(AnyNode "cset-word")
(PseudoAnd ...)
select count(*) from atoms, pan where atoms.outgoing[2]=pan.uuid AND atoms.type=21;
gives 1860793 which is as before. == (- 5431830 3571037)
so that makes sense too.
update atoms set type=230 FROM pan where outgoing[2]=pan.uuid and type=169;
------
May as well drop the bogus connectors, too?? Yes, we should.
select count(*) from atoms where type=169;
404882 -- exactly as expected.
CREATE TEMP TABLE lws AS SELECT uuid FROM atoms WHERE type=169;
select count(*) from valuations, lws where valuations.atom=lws.uuid;
delete from valuations using lws where atom =lws.uuid;
delete from atoms where type=169;
DONE.
---------- meanwhile, construct en_pairs_tthree_mst DONE
but first, in en_pairs_tthree, kill the bogus disjuncts.
select count(*) from atoms where type=169; --- 1245661
Deleted. And take a new DB dump, while we are at it. DONE.
Computed the pair MI's they are in en_pairs_tthree now, see above. DONE.
Now merge in en_pairs_ttwo_mst
strategy:
-- load words and disjuncts from ttwo_mst
-- close db
-- load everything from en_pairs_tthree
-- save to en_pairs_tthree_mst
169 LgWordCset
167 | LgAnd
(use-modules (opencog) (opencog persist) (opencog persist-sql))
(use-modules (opencog nlp) (opencog nlp learn))
(use-modules (opencog analysis))
(sql-open "postgres:///en_pairs_ttwo_mst?user=ubuntu")
(fetch-all-words)
(define pca (make-pseudo-cset-api))
(define psa (add-pair-stars pca))
(psa 'fetch-pairs) <<< took 3449 secs
(sql-close)
(sql-open "postgres:///en_pairs_tthree?user=ubuntu")
(sql-load)
(sql-close)
(sql-open "postgres:///en_pairs_tthree_mst?user=ubuntu")
(sql-store) ; -- 46262710 = 46M atoms total
(sql-close)
---------------------------------------------------------------------
---------- meanwhile, finish stats work
(fetch-all-sims)
general idea: how many pairs are there out of N^2?
what is the distribution of pair memebership?
sims should be LLOBJ
Umm batch-all-pair-mi is where?
batch-sim should NOT store!
batch-sim-pairs
So: 1) cutoff for sim 2) cutoff for store.
start 2.4G and run 0.3 sim
again 0.1 sim...
possible: (37413 * 37412)/2 = 699847578 = 700M pairs
estimate: 7.5% of these, so = 52M pairs!? Ouch. .. 70-80 GB RAM
and insanely slow .. secs = 102 hrs = 4.2 days
running at 1.9K pairs/sec ouch.
4000 secs for 20x36K = 180/sec total collapse.
Try again:
(use-modules (opencog) (opencog persist) (opencog persist-sql))
(use-modules (opencog nlp) (opencog nlp learn))
; (sql-open "postgres:///en_pairs_supersim?user=linas")
(sql-open "postgres:///en_pairs_ultrasim?user=linas")
(use-modules (opencog cogserver))
(start-cogserver "opencog2.conf")
(use-modules (opencog analysis))
(fetch-all-words)
(define pca (make-pseudo-cset-api))
(define psa (add-pair-stars pca))
(psa 'fetch-pairs)
(print-matrix-summary-report psa)
(sql-clear-stats)
; 1524572 atoms in AS
(define ac (get-all-cset-words))
(length ac)
(load "gram-sim.scm")
(para-batch-sim-pairs ac 0.1)
oh fu... its running 3x because:
1) not_yet_stored
2) do_store_single_atom
FIXED. num_get_links - num_got_links == total stores
and num_get_links == valuation update Yayyy!!
OK, so: 1.5M atoms, 37413 words, now: 13M new atoms
so 13M / 37K = 343 atoms per incoming set.
OK. DONE. Now what?
binned-sims.dat
(define all-sims '())
(cog-map-type
(lambda (sim) (set! all-sims (cons sim all-sims)) #f)
'SimilarityLink)
(length all-sims) -- 44139852 = 44M wow
(define scored-sims (score sim-cosine all-sims)) ;; many hours
(define binned-sims (bin-count-simple scored-sims 300))
-------------------
FWIW, the crappy results from back-when: en_pairs_sim:
Summary Report for Correlation Matrix Word-Disjunct Pairs (aka Connector Sets)
Left type: WordNode Right Type: LgAnd Pair Type: LgWordCset
Wildcard: (ListLink (ctv 0 0 661104)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 37413 Columns: 291637
Size: 446204 non-zero entries of 10911015081 possible
Fraction non-zero: 4.0895E-5 Sparsity (-log_2): 14.578
Total observations: 661104.0 Avg obs per pair: 1.4816
Entropy Total: 18.301 Left: 16.006 Right: 10.281
Total MI: -7.985
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 4122. 370.0
Count (l_1) 8763. 680.7 2.126 1.839
Length (l_2) 236.8 51.39 5.7452E-2 .1389
RMS Count 216.6 46.40 5.2550E-2 .1254
Notice that tone has maybe 200x mor observations!!!! 133M instead of
660K ... wow. en_pairs_sim also has:
Left type: WordNode Right Type: WordNode Pair Type: ListLink
Wildcard: (EvaluationLink (ctv 0 0 4.1823528e+08)
(LinkGrammarRelationshipNode "ANY")
)
Rows: 395274 Columns: 396260
Size: 8880914 non-zero entries of 156631275240 possible
Fraction non-zero: 5.6699E-5 Sparsity (-log_2): 14.106
Total observations: 418235277.0 Avg obs per pair: 47.094
Entropy Total: 19.280 Left: 11.088 Right: 11.215
Total MI: -3.023
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 2.9904E+4 3.4828E+4
Count (l_1) 4.4813E+6 5.2946E+6 149.9 152.0
Length (l_2) 4.9537E+5 4.9524E+5 16.57 14.22
RMS Count 4.9502E+5 4.9482E+5 16.55 14.21
---------- meanwhile do the rone_mst DONE
Started gamma-guten-r port 19505 feeder-term-7 / guile-term-6
guile -l mst-count-en-r.scm on term-6
9 June 10PM 3028 articles
12 June 2017 8PM DONE
Now compute the summaries.
(define pca (make-pseudo-cset-api))
(batch-pairs pca)
COMMENT ON DATABASE en_pairs_rone_mst IS 'Clean MST-parse of tranche-1 and summary';
---------- meanwhile do the tthree_mst DONE
gamma-guten feeder-8 port 19005 ./guten-mst-en.sh
cog-term-1 guile -l mst-count-en.scm on term 1
Started 10 June 8AM -- 5712 articles
running at about 900 articles/day
---------- meanwhile do rtwo pairs DONE
Started. beta-guten-r 4500 articles. feeder-4 guile-3 port 17015
9 June 10PM done 12 June 1PM
now:
createdb -T en_pairs_r en_pairs_rtwo
COMMENT ON DATABASE en_pairs_rtwo IS 'Clean ANY-parse of tranche-1 and tranch-2';
pg_dump en_pairs_rtwo > en_pairs_rtwo.sql
DONE.
---------- meanwhile do rtwo_mst DONE
started
copy rtwo to rtwo_mst this gets pairs
open rone_mst, load cset pairs
close rone_mst
open rtwo_mst
write cset pairs only (not the words)
close rtwo_mst
$ createdb -T en_pairs_rtwo en_pairs_rtwo_mst
(use-modules (opencog analysis))
(sql-open "postgres:///en_pairs_rone_mst?user=ubuntu")
(fetch-all-words)
(define pca (make-pseudo-cset-api))
(define psa (add-pair-stars pca))
(psa 'fetch-pairs)
(print-matrix-summary-report psa)
(sql-close)
(sql-open "postgres:///en_pairs_rtwo_mst?user=ubuntu")
(cog-map-type (lambda (cset) (store-atom cset) #f) 'PseudoWordCset)
(sql-close)
Oh, wait; we also have to compute the pairMI's!. Ooops, almost forgot.
(define ala (make-any-link-api))
(batch-pairs ala) DONE
Started gamma-guten-r port 19505 feeder-term-7 / guile-term-6
guile -l mst-count-en-r.scm on term-6
4500 articles
Started 13 June 2017 6:30PM Done 6 hours later! wtf! last time, it was
too fast, also. ??? WTF. why? Bug?
(batch-pairs psa) DONE
Well, there's a huge pile of errors in the MST log file.
---------- meanwhile do rthree pairs
guile-3 port 17015 feeder-4 5713 articles
Started 12 June 2017 2PM
Paused 24 hours later, == 800 articles/24 hours
---------- meanwhile, compute & cache similarity DONE
; (sql-open "postgres:///en_pairs_tone_sim?user=ubuntu")
Wait, where?
Oh, gram-sim.
DONE.
then binned-sims
-----------
todo items:
* trim low observation counts ....
* implement sql delete function.
* see todo at bottom
* XXX r versions still fail to separate single-quotes
Except I am having trouble reproducing this... lg any seems to work OK
* Also ." is not split .. which I also cannot reproduce.
* Hyphenated words should be split, by default. These can be
reconstructed later. Split - dashes, long dashes, elipsis. DONE.
---------- debug hang:
latest bdgw from git does not solve it.
latest stable-2.2 guile does not solve it.
revert to distro bdwgc
#0 0x00007f362151c826 in __GI___sigsuspend (
set=set@entry=0x7f36214b1740 <suspend_handler_mask>)
at ../sysdeps/unix/sysv/linux/sigsuspend.c:30
#1 0x00007f3621293602 in GC_suspend_handler_inner
(dummy=dummy@entry=0x0,
context=context@entry=0x7fff6c807540) at pthread_stop_world.c:324
#2 0x00007f36212936cf in GC_suspend_handler (sig=30, info=<optimized
out>,
context=0x7fff6c807540) at pthread_stop_world.c:237
#3 <signal handler called>
thr 2: find_slot_map (
cache=0x7f361fe85ab0, ip=0x1) at ../../libguile/vm.c:936
#1 scm_i_vm_mark_stack (vp=0xbdca3120, mark_stack_ptr=0x7f361fe85ef0,
mark_stack_limit=0x7f361fe95eb0) at ../../libguile/vm.c:1011
#2 0x00007f36212883be in GC_mark_from (mark_stack_top=0x7f361fe85ea0,
mark_stack_top@entry=0x7f361fe85f00,
mark_stack=mark_stack@entry=0x7f361fe85eb0,
mark_stack_limit=mark_stack_limit@entry=0x7f361fe95eb0) at
mark.c:784
#3 0x00007f36212889de in GC_do_local_mark
(local_mark_stack=0x7f361fe85eb0,
local_top=0x7f361fe85f00) at mark.c:1048
#4 0x00007f3621288bf8 in GC_mark_local (
local_mark_stack=local_mark_stack@entry=0x7f361fe85eb0,
id=id@entry=1)
at mark.c:1181
#5 0x00007f3621288f0a in GC_help_marker
(my_mark_no=my_mark_no@entry=195743)
at mark.c:1249
#6 0x00007f3621292edc in GC_mark_thread (id=<optimized out>)
at pthread_support.c:380
single-stepping:
scm_i_vm_mark_stack (vp=0xbdca3120, mark_stack_ptr=0x7f361fe85ef0,
mark_stack_limit=0x7f361fe95eb0) at ../../libguile/vm.c:975
975 fp = SCM_FRAME_DYNAMIC_LINK (fp))
(gdb) print fp
$1 = (union scm_vm_stack_element *) 0x7f36183d5e38
(gdb) step
973 for (fp = vp->fp, sp = vp->sp;
(gdb)
979 for (slot = nlocals - 1; sp < fp; sp++, slot--)
(gdb)
1011 slot_map = find_slot_map (SCM_FRAME_RETURN_ADDRESS (fp),
&cache);
(gdb)
1006 sp = SCM_FRAME_PREVIOUS_SP (fp);
(gdb) print fp
$2 = (union scm_vm_stack_element *) 0x7f36183d5e38
(gdb) print sp
$3 = (union scm_vm_stack_element *) 0x7f36183d5e48
(gdb) step
1011 slot_map = find_slot_map (SCM_FRAME_RETURN_ADDRESS (fp),
&cache);
(gdb)
find_slot_map (cache=0x7f361fe85ab0, ip=0x1) at ../../libguile/vm.c:932
932 size_t slot = (((scm_t_uintptr) ip) >> 2) %
SLOT_MAP_CACHE_SIZE;
(gdb)
935 if (cache->entries[slot].ip == ip)
(gdb)
scm_i_vm_mark_stack (vp=0xbdca3120, mark_stack_ptr=0x7f361fe85ef0,
mark_stack_limit=0x7f361fe95eb0) at ../../libguile/vm.c:1011
1011 slot_map = find_slot_map (SCM_FRAME_RETURN_ADDRESS (fp),
&cache);
(gdb)
find_slot_map (cache=0x7f361fe85ab0, ip=0x1) at ../../libguile/vm.c:936
936 map = cache->entries[slot].map;
(gdb)
scm_i_vm_mark_stack (vp=0xbdca3120, mark_stack_ptr=0x7f361fe85ef0,
mark_stack_limit=0x7f361fe95eb0) at ../../libguile/vm.c:975
975 fp = SCM_FRAME_DYNAMIC_LINK (fp))
(gdb) print fp
$4 = (union scm_vm_stack_element *) 0x7f36183d5e38
git tag -l
Try guile-2.2 v2.2.0 Wed Mar 15 09:02:52 2017
Try guile-2.1.8 v2.1.8 Fri Mar 10 11:01:04 2017 tried it - spins. its broke
(breaks after about 300 cpu-min)
OK guile-2.0.11 on xenial crashes almost immediately with
guile: hashtab.c:137: vacuum_weak_hash_table: Assertion `removed <= len' failed.
Arguments with Curtis:
GC_register_my_thread
-------------------------------------------------------------------------
Bewlow has gone to
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27234
bug-guile@gnu.org.
Subject: Hang in GC, inf loop while walking frame pointers
what: guile-2.2-stable, from git.
I've got a large, complex, heavily multi-threaded guile program that
hangs during garbage collection; usually after running for half a day.
It hangs in a tight loop in scm_i_vm_mark_stack, spinning at 100% of CPU.
This is due to the for-loop line fp = SCM_FRAME_DYNAMIC_LINK (fp))
at libguile/vm.c line 975 failing to advance the frame pointer.
There's no "obvious" corruption in the stack; it simply looks like
the frame was incompletely set up, and so incrementing to the next
fp does not go anywhere.
I have recompiled guile with VM_ENABLE_ASSERTIONS and am trying to
reproduce the bug now. The rest of this email is a record of a long
debug session isolating the problem, and showing that, overall, the
thread and stack data look more-or-less correct and uncorrupoted,
except for the inability to walk forward in the frame.
-- linas
Try again.
same as above: thread 4: find_slot_map
187 threads total.
#0 find_slot_map (cache=0x7f3e9b783ab0, ip=0x1) at
../../libguile/vm.c:935
935 if (cache->entries[slot].ip == ip)
(gdb) print slot
$1 = 0
(gdb) print cache->entries[slot].ip
$2 = (scm_t_uint32 *) 0x1
(gdb) print cache->entries[slot].map
$4 = (const scm_t_uint8 *) 0x0
(gdb) step
scm_i_vm_mark_stack (vp=0x755c1bd0, mark_stack_ptr=0x7f3e9b783f40,
mark_stack_limit=0x7f3e9b793eb0) at ../../libguile/vm.c:1011
1011 slot_map = find_slot_map (SCM_FRAME_RETURN_ADDRESS (fp), &cache);
(gdb) print fp
$5 = (union scm_vm_stack_element *) 0x7f3e94cdee38
#define SCM_FRAME_RETURN_ADDRESS(fp) ((fp)[0].as_ip)
(gdb) print (fp)[0].as_ip
$6 = (scm_t_uint32 *) 0x1
OK that looks weird ... is this corrupted ?? but whatever,
because the returned slot_map is never used ... because ...
(gdb) step
scm_i_vm_mark_stack (vp=0x755c1bd0, mark_stack_ptr=0x7f3e9b783f40,
mark_stack_limit=0x7f3e9b793eb0) at ../../libguile/vm.c:975
975 fp = SCM_FRAME_DYNAMIC_LINK (fp))
(gdb) print fp
$7 = (union scm_vm_stack_element *) 0x7f3e94cdee38
frames.h:#define SCM_FRAME_DYNAMIC_LINK(fp) ((fp) + (fp)[1].as_uint)
(gdb) print (fp)[1].as_uint
$8 = 0
OK, that seems bad, because now fp never advances, it just repeats
over and over with this same value.
(gdb)
979 for (slot = nlocals - 1; sp < fp; sp++, slot--)
(gdb) print nlocals
$10 = -2
(gdb) print sp
$11 = (union scm_vm_stack_element *) 0x7f3e94cdee48
#define SCM_FRAME_NUM_LOCALS(fp, sp) ((fp) - (sp))
(gdb) print fp
$12 = (union scm_vm_stack_element *) 0x7f3e94cdee38
(gdb) print ((fp) - (sp))
$13 = -2
Ohh .. its not -16 because its -2 * sizeof (union scm_vm_stack_element *)
so that's OK.
So for loops is skipped, it should go to.
sp = SCM_FRAME_PREVIOUS_SP (fp);
frames.h:#define SCM_FRAME_PREVIOUS_SP(fp) ((fp) + 2)
and so now it loops around and repeats.
(gdb) print cache
$19 = {entries = {{ip = 0x1, map = 0x0}, {ip = 0x0,
map = 0x0} <repeats 23 times>, {ip = 0x1a1b5e0, map = 0x0}, {ip = 0x0,
map = 0x0}, {ip = 0x0, map = 0x0}, {ip = 0x0, map = 0x0}, {ip = 0x0,
map = 0x0}, {ip = 0x0, map = 0x0}, {ip = 0x0, map = 0x0}, {ip = 0x0,
map = 0x0}}}
(gdb) print &cache
$20 = (struct slot_map_cache *) 0x7f3e9b783ab0
and so the loop repeates forever, because
fp = SCM_FRAME_DYNAMIC_LINK (fp) never advances fp, because
(fp)[1].as_uint is zero.
So where is fp pointing to? recall fp == 0x7f3e94cdee38
(gdb) x/20x 0x7f3e94cdee00
0x7f3e94cdee00: 0x0000000000000904 0x000000000000200c
0x7f3e94cdee10: 0x0000000001a1b7e0 0x0000000001a1b5e0
0x7f3e94cdee20: 0x0000000000000004 0x000000000000200c
0x7f3e94cdee30: 0x0000000000169bd6 0x0000000000000001
0x7f3e94cdee40: 0x0000000000000000 0x000000000192acd0
0x7f3e94cdee50: 0x0000000001fa4bc0 0x0000000001a1c6d0
0x7f3e94cdee60: 0x0000000044507950 0x0000000000000002
0x7f3e94cdee70: 0x000000000005a6f5 0x00000000018febd0
0x7f3e94cdee80: 0x000000006d490d00 0x00007f3e9e97241c
0x7f3e94cdee90: 0x0000000000000002 0x00007f3e9e97241c
(gdb) x/s 0x7f3e94cdee60
0x7f3e94cdee60: "PyPD" <<< ?? is this a meaningul string?
(gdb) x/s 0x7f3e94cdee81
0x7f3e94cdee81: "\rIm" << how about this ???
So this does not look healthy. How did we get here?
(gdb) print vp
$21 = (struct scm_vm *) 0x755c1bd0
(gdb) print mark_stack_ptr
$22 = (struct GC_ms_entry *) 0x7f3e9b783f40
(gdb) print mark_stack_limit
$23 = (struct GC_ms_entry *) 0x7f3e9b793eb0
(gdb) print upper
$24 = (void *) 0x7ffdd986
(gdb) print GC_greatest_plausible_heap_addr
$25 = (void *) 0x7ffdd986
(gdb) print lower
$26 = (void *) 0x17a1ff8
(gdb) print vp->fp
$27 = (union scm_vm_stack_element *) 0x7f3e94cdee18
(gdb) print vp->sp
$28 = (union scm_vm_stack_element *) 0x7f3e94cdee00
(gdb) print vp->stack_top
$29 = (union scm_vm_stack_element *) 0x7f3e94cdf000
So lets walk manually.
frames.h:#define SCM_FRAME_DYNAMIC_LINK(fp) ((fp) + (fp)[1].as_uint)
(gdb) print (vp->fp) + (vp->fp)[1].as_uint
$31 = (union scm_vm_stack_element *) 0x7f3e94cdee38
So we are instanly in trouble! once through the loop, and we've got the broken fp
So how did we get here?
(gdb) print vp
$32 = (struct scm_vm *) 0x755c1bd0
(gdb) print *vp
$33 = {ip = 0x7f3e9e5a6378 <subr_stub_code+56>, sp = 0x7f3e94cdee00,
fp = 0x7f3e94cdee18, stack_limit = 0x7f3e94cde000, trace_level = 0,
sp_min_since_gc = 0x7f3e94cdee00, stack_size = 512,
stack_bottom = 0x7f3e94cde000, stack_top = 0x7f3e94cdf000,
overflow_handler_stack = 0x304, hooks = {0x4, 0x4, 0x4, 0x4, 0x4},
resumable_prompt_cookie = 0x7f3d397f9440, engine = 1}
Looks plausible, not corrupted. Why is engine=1 ??
#define SCM_VM_REGULAR_ENGINE 0
#define SCM_VM_DEBUG_ENGINE 1
#define SCM_VM_NUM_ENGINES 2
# define SCM_I_CURRENT_THREAD (scm_i_current_thread)
(gdb) print scm_i_current_thread
$34 = (scm_i_thread *) 0x0
OK, that's not excellent.
(gdb) print thread_count
$35 = 161
(gdb) bt
#0 scm_i_vm_mark_stack (vp=0x755c1bd0, mark_stack_ptr=0x7f3e9b783f40,
mark_stack_limit=0x7f3e9b793eb0) at ../../libguile/vm.c:1011
#1 0x00007f3e9db8835e in GC_mark_from (mark_stack_top=0x7f3e9b783ee0,
mark_stack_top@entry=0x7f3e9b783f00,
mark_stack=mark_stack@entry=0x7f3e9b783eb0,
mark_stack_limit=mark_stack_limit@entry=0x7f3e9b793eb0) at ../mark.c:772
#2 0x00007f3e9db8897e in GC_do_local_mark (local_mark_stack=0x7f3e9b783eb0,
local_top=0x7f3e9b783f00) at ../mark.c:1037
#3 0x00007f3e9db88b98 in GC_mark_local (
local_mark_stack=local_mark_stack@entry=0x7f3e9b783eb0, id=id@entry=4)
at ../mark.c:1170
#4 0x00007f3e9db88eaa in GC_help_marker (my_mark_no=my_mark_no@entry=80003)
at ../mark.c:1238
#5 0x00007f3e9db92e3c in GC_mark_thread (id=<optimized out>)
at ../pthread_support.c:380
#6 0x00007f3e9e2ae6ba in start_thread (arg=0x7f3e9b794700)
at pthread_create.c:333
#7 0x00007f3e9dfdd82d in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
so bdwgc is version gc7_6_0
So ../mark.c:772 is GET_HDR and GET_HDR is in
include/private/gc_hdrs.h
define GET_HDR(p, hhdr) ... ((hhdr) = HDR_INNER(p)
# define HDR_INNER(p) HDR_FROM_BI(BI(p),p)
#define HDR_FROM_BI(bi, p) \
((bi)->index[((word)(p) >> LOG_HBLKSIZE) & (BOTTOM_SZ - 1)])
This seems all wrong.. wtf ....
is it called by thread_mark ??
in libguile/thread.c:
thread_gc_kind =
GC_new_kind (GC_new_free_list (),
GC_MAKE_PROC (GC_new_proc (thread_mark), 0),
0, 1)
in bdwgc/misc.c GC_new_proc
GC_mark_procs[result] = proc
#define GC_mark_procs GC_arrays._mark_procs
#define PROC(descr) \
(GC_mark_procs[((descr) >> GC_DS_TAG_BITS) & (GC_MAX_MARK_PROCS-1)])
GC_obj_kinds[result].ok_descriptor = descr; <<< this is where MAKE_PROC goes.
anyway ... in mark.c line 733
case GC_DS_PROC:
mark_stack_top--;
credit -= GC_PROC_BYTES;
mark_stack_top = (*PROC(descr))((word *)current_p, mark_stack_top,
mark_stack_limit, ENV(descr));
continue;
But that's the wrong line number. line 772 is in
case GC_DS_PER_OBJECT:
if ((signed_word)descr >= 0) {
/* Descriptor is in the object. */
descr = *(word *)(current_p + descr - GC_DS_PER_OBJECT);
Oh, it goes to retry, which loops around and calls PROC which is thread_mark
which is called with args
mark_stack_top = (*PROC(descr))((word *)current_p, mark_stack_top,
mark_stack_limit, ENV(descr));
(gdb) print current_p
$39 = (ptr_t) 0x7225d380 "@\325%r"
so back in libguile/threads.c line 84:
const struct scm_i_thread *t = (struct scm_i_thread *) addr;
(gdb) print *(struct scm_i_thread *) current_p
$40 = {next_thread = 0x7225d540, handle = 0x5763f3b0, pthread = 139900934399744,
result = 0x4, exited = 0, guile_mode = 1, needs_unregister = 1, wake = 0x0,
sleep_cond = {__data = {__lock = 0, __futex = 0, __total_seq = 0,
__wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0, __nwaiters = 0,
__broadcast_seq = 0}, __size = '\000' <repeats 47 times>, __align = 0},
sleep_pipe = {151, 197}, freelists = 0x71cfe800,
pointerless_freelists = 0x71cfe780, dynamic_state = 0x71de7900, dynstack = {
base = 0x698c3800, top = 0x698c3940, limit = 0x698c3a00},
pending_asyncs = 0x304, block_asyncs = 0, continuation_root = 0x5763f340,
continuation_base = 0x7f3d397f9c90, vp = 0x755c1bd0, base = 0x7f3d397f9da0,
regs = {{__jmpbuf = {139905204221696, 139906923931021, 1, 139894281647232,
139906913276512, 0, 139906918327551, 8388608},
__mask_was_saved = 964664752, __saved_mask = {__val = {0, 139900934399744,
139900934399744, 139900934397344, 139906923929400, 139893936840720,
1804170656, 139906913276512, 139900934399744, 139900934397344,
139900934397504, 139893936840720, 139906910859134, 139906913276484,
139906910857112, 27401856}}}}}
(gdb) print ((struct scm_i_thread *) current_p)->vp
$41 = (struct scm_vm *) 0x755c1bd0
which is exactly the vp we had before, so this is the correct thread.
So ... all the values in *(struct scm_i_thread *) current_p look more or less OK;
its not corrupted. And all the values in *vp look more or less OK, its not corrrupted
except that engine=1 looks wrong. So what's up with the weird stack looping?
Again:
From before, we had:
(gdb) print *vp
$42 = {ip = 0x7f3e9e5a6378 <subr_stub_code+56>, sp = 0x7f3e94cdee00,
fp = 0x7f3e94cdee18, stack_limit = 0x7f3e94cde000, trace_level = 0,
sp_min_since_gc = 0x7f3e94cdee00, stack_size = 512,
stack_bottom = 0x7f3e94cde000, stack_top = 0x7f3e94cdf000,
overflow_handler_stack = 0x304, hooks = {0x4, 0x4, 0x4, 0x4, 0x4},
resumable_prompt_cookie = 0x7f3d397f9440, engine = 1}
so fp, sp, stack_bottom, stack_top all are self-consistent with each other.
The problem is that
fp = SCM_FRAME_DYNAMIC_LINK (fp)
goes nowhere, after the first iteration.
Is the frame corrupted? How? what does the rest of the stack look like?
(gdb) x/32x 0x7f3e94cdee00
0x7f3e94cdee00: 0x0000000000000904 0x000000000000200c
0x7f3e94cdee10: 0x0000000001a1b7e0 0x0000000001a1b5e0
0x7f3e94cdee20: 0x0000000000000004 0x000000000000200c <<< theres the 4
0x7f3e94cdee30: 0x0000000000169bd6 0x0000000000000001 <<< there's the new fp
0x7f3e94cdee40: 0x0000000000000000 0x000000000192acd0 <<< there's the zero.
0x7f3e94cdee50: 0x0000000001fa4bc0 0x0000000001a1c6d0
0x7f3e94cdee60: 0x0000000044507950 0x0000000000000002
0x7f3e94cdee70: 0x000000000005a6f5 0x00000000018febd0
0x7f3e94cdee80: 0x000000006d490d00 0x00007f3e9e97241c
0x7f3e94cdee90: 0x0000000000000002 0x00007f3e9e97241c
0x7f3e94cdeea0: 0x0000000000000002 0x00007f3e8d797630
0x7f3e94cdeeb0: 0x0000000000000002 0x00007f3e86effacc
0x7f3e94cdeec0: 0x0000000000000003 0x000000006d490d00
0x7f3e94cdeed0: 0x00007f3e86eb8698 0x0000000000000002
0x7f3e94cdeee0: 0x00007f3e8e51f450 0x0000000000000008
0x7f3e94cdeef0: 0x0000000001a37c10 0x000000000192acd0
So... who sets up thread->vp ?
/* #define VM_ENABLE_ASSERTIONS */
However, defining this breaks the build.
diff --git a/libguile/vm-engine.c b/libguile/vm-engine.c
index 6c88ebf..f0f8164 100644
--- a/libguile/vm-engine.c
+++ b/libguile/vm-engine.c
@@ -2050,11 +2050,11 @@ VM_NAME (scm_i_thread *thread, struct scm_vm *vp,
if (!scm_module_system_booted_p)
{
- ASSERT (scm_is_true
+ ASSERT (scm_is_true(
scm_equal_p (modname,
scm_list_2
(SCM_BOOL_T,
- scm_from_utf8_symbol ("guile"))));
+ scm_from_utf8_symbol ("guile")))));
var = scm_lookup (sym);
}
else if (scm_is_true (SCM_CAR (modname)))
diff --git a/libguile/vm.c b/libguile/vm.c
index 18f2192..c26066e 100644
--- a/libguile/vm.c
+++ b/libguile/vm.c
@@ -63,7 +63,7 @@ static size_t page_size;
/* The VM has a number of internal assertions that shouldn't normally be
necessary, but might be if you think you found a bug in the VM. */
-/* #define VM_ENABLE_ASSERTIONS */
+#define VM_ENABLE_ASSERTIONS
static void vm_expand_stack (struct scm_vm *vp,
union scm_vm_stack_element *new_sp) SCM_NOINLINE;
GEN guile-procedures.texi
/bin/bash: line 1: 24078 Broken pipe
./meta/guile
Aborted
./libguile/guile
Aborted
gdb libguile/.libs/lt-guile
#0 0x00007ffff74f7428 in __GI_raise (sig=sig@entry=6)
at ../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007ffff74f902a in __GI_abort () at abort.c:89
#2 0x00007ffff7b707c4 in vm_regular_engine (thread=0x7282, vp=0x806f30,
registers=0x6, resume=-145787864) at ../../libguile/vm-engine.c:2053
#3 0x00007ffff7b711d2 in scm_call_n (proc=0x832b30, argv=argv@entry=0x0,
nargs=nargs@entry=0) at ../../libguile/vm.c:1257
#4 0x00007ffff7af14e9 in scm_call_0 (proc=<optimized out>)
at ../../libguile/eval.c:481
#5 0x00007ffff7b10568 in scm_primitive_load_path (args=<optimized out>)
at ../../libguile/load.c:1248
#6 0x00007ffff7b6e976 in vm_regular_engine (thread=0x7282, vp=0x806f30,
registers=0x6, resume=-145787864) at ../../libguile/vm-engine.c:784
Oh. wtf.
(define avg-time-taken
(let ((last-gc (gc-stats))
(start-time (get-internal-real-time))
(run-time (get-internal-run-time)))
(lambda ()
(define now (get-internal-real-time))
(define run (get-internal-run-time))
(define cur (gc-stats))
(define gc-time-taken (* 1.0e-9 (- (cdar cur) (cdar last-gc))))
(define elapsed-time (* 1.0e-9 (- now start-time)))
(define cpu-time (* 1.0e-9 (- run run-time)))
(format #t "Elapsed time: ~5f secs. GC-time-taken: ~5f s or ~5f% cpu-usage: ~5f%\n"
elapsed-time gc-time-taken (* 100 (/ gc-time-taken elapsed-time))
(* 100 (/ cpu-time elapsed-time))
)
(set! last-gc cur)
(set! start-time now)
(set! run-time run))))
Anyway: it hangs without ever hitting any of the VM DEBUG asserts.
This time:
(gdb) print *vp
$1 = {ip = 0x7ffff7b87358 <subr_stub_code+56>, sp = 0x7fffdc617e00,
fp = 0x7fffdc617e18, stack_limit = 0x7fffdc617000, trace_level = 0,
sp_min_since_gc = 0x7fffdc617b58, stack_size = 512,
stack_bottom = 0x7fffdc617000, stack_top = 0x7fffdc618000,
overflow_handler_stack = 0x304, hooks = {0x4, 0x4, 0x4, 0x4, 0x4},
resumable_prompt_cookie = 0x7fff857f9440, engine = 1}
(gdb) x/32x 0x7fffdc617e00
0x7fffdc617e00: 0x0000000000000904 0x000000000000630c
0x7fffdc617e10: 0x000000000089f7e0 0x000000000089f5e0
0x7fffdc617e20: 0x0000000000000004 0x000000000000630c
0x7fffdc617e30: 0x00000000000377d6 0x0000000000000001
0x7fffdc617e40: 0x0000000000000000 0x00000000007aecd0
0x7fffdc617e50: 0x00000000010de660 0x00000000008a06d0
0x7fffdc617e60: 0x000000003f543a40 0x0000000000000002
0x7fffdc617e70: 0x000000000000ddf5 0x0000000000782bd0
0x7fffdc617e80: 0x00000000130cbe20 0x00007ffff7f4d41c
0x7fffdc617e90: 0x0000000000000002 0x00007ffff7f4d41c
0x7fffdc617ea0: 0x0000000000000002 0x00007fffe7099630
0x7fffdc617eb0: 0x0000000000000002 0x00007fffe08f3acc
0x7fffdc617ec0: 0x0000000000000003 0x00000000130cbe20
0x7fffdc617ed0: 0x00007fffe08be698 0x0000000000000002
0x7fffdc617ee0: 0x00007fffe7c8e450 0x0000000000000008
0x7fffdc617ef0: 0x00000000008bbbe0 0x00000000007aecd0
Wow. More or less IDENTICAL to last time!!
what is vp->ip pointing at??
(gdb) x/16 0x7ffff7b87320
0x7ffff7b87320 <subr_stub_code>: 0x0000000a00000115 0x00000009000000b7
0x7ffff7b87330 <subr_stub_code+16>: 0x0000000000000000 0x0000000a00000215
0x7ffff7b87340 <subr_stub_code+32>: 0x00000009000000b7 0x0000000000000000
0x7ffff7b87350 <subr_stub_code+48>: 0x0000021800000217 0x000000b70000000a
0x7ffff7b87360 <subr_stub_code+64>: 0x0000000000000009 0x0000000a00000120
0x7ffff7b87370 <subr_stub_code+80>: 0x00000009000000b7 0x0000000000000000
0x7ffff7b87380 <subr_stub_code+96>: 0x0000000a00000315 0x00000009000000b7
0x7ffff7b87390 <subr_stub_code+112>: 0x0000000000000000 0x0000031700000216
WHo sets t->vp ???
thread_vm (scm_i_thread *t)
t->vp = make_vm ();
So clearly, above is NOT a fresh stack, but is well-used.
(gdb) print vm_default_engine
$3 = 1
But ... #define SCM_VM_REGULAR_ENGINE 0
#define SCM_VM_DEBUG_ENGINE 1
So who set it to 1??
#define FP_SLOT(i) SCM_FRAME_SLOT (vp->fp, i)
#define SCM_FRAME_SLOT(fp,i) ((fp) - (i) - 1)
vp->fp = SCM_FRAME_SLOT (old_fp, proc - 1);
SCM_FRAME_SET_DYNAMIC_LINK (vp->fp, old_fp);
frames.h:#define SCM_FRAME_DYNAMIC_LINK(fp) ((fp) + (fp)[1].as_uint)
frames.h:#define SCM_FRAME_SET_DYNAMIC_LINK(fp, dl) ((fp)[1].as_uint = ((dl) - (fp)))
So I'm getting dl == fp ??
guile -l pair-count-en.scm
libguile/frames.h
@@ -104,7 +104,8 @@ union scm_vm_stack_element
#define SCM_FRAME_RETURN_ADDRESS(fp) ((fp)[0].as_ip)
#define SCM_FRAME_SET_RETURN_ADDRESS(fp, ra) ((fp)[0].as_ip = (ra))
#define SCM_FRAME_DYNAMIC_LINK(fp) ((fp) + (fp)[1].as_uint)
-#define SCM_FRAME_SET_DYNAMIC_LINK(fp, dl) ((fp)[1].as_uint = ((dl) - (fp)))
+#define SCM_FRAME_SET_DYNAMIC_LINK(fp, dl) { \
+((fp)[1].as_uint = ((dl) - (fp))) ; if ((dl) == (fp)) abort(); }
#define SCM_FRAME_SLOT(fp,i) ((fp) - (i) - 1)
#define SCM_FRAME_LOCAL(fp,i) (SCM_FRAME_SLOT (fp, i)->as_scm)
#define SCM_FRAME_NUM_LOCALS(fp, sp) ((fp) - (sp))
--- a/libguile/gsubr.c
+++ b/libguile/gsubr.c
@@ -169,7 +169,8 @@
(generate-bytecode i)
(setq i (1+ i)))))
*/
-static const scm_t_uint32 subr_stub_code[] = {
+// static const scm_t_uint32 subr_stub_code[] = {
+const scm_t_uint32 subr_stub_code[] = {
/* C-u 1 0 M-x generate-bytecodes RET */
/* 0 arguments */
A(0),
--- a/libguile/vm.c
+++ b/libguile/vm.c
@@ -63,7 +63,7 @@ static size_t page_size;
/* The VM has a number of internal assertions that shouldn't normally be
necessary, but might be if you think you found a bug in the VM. */
-/* #define VM_ENABLE_ASSERTIONS */
+#define VM_ENABLE_ASSERTIONS
static void vm_expand_stack (struct scm_vm *vp,
union scm_vm_stack_element *new_sp) SCM_NOINLINE;
@@ -952,6 +952,9 @@ enum slot_desc
SLOT_DESC_UNUSED = 3
};
+extern const scm_t_uint32 subr_stub_code[];
+scm_t_uint32 * pstub = (void*) &subr_stub_code;
+
/* Mark the active VM stack region. */
struct GC_ms_entry *
scm_i_vm_mark_stack (struct scm_vm *vp, struct GC_ms_entry *mark_stack_ptr,
@@ -967,6 +970,7 @@ scm_i_vm_mark_stack (struct scm_vm *vp, struct GC_ms_entry
*mark_stack_ptr,
void *upper = (void *) GC_greatest_plausible_heap_addr;
void *lower = (void *) GC_least_plausible_heap_addr;
struct slot_map_cache cache;
+ union scm_vm_stack_element *fprev = NULL;
memset (&cache, 0, sizeof (cache));
@@ -974,6 +978,14 @@ scm_i_vm_mark_stack (struct scm_vm *vp, struct GC_ms_entry
*mark_stack_ptr,
fp < vp->stack_top;
fp = SCM_FRAME_DYNAMIC_LINK (fp))
{
+ if (fprev == fp)
+ {
+ printf("Oh no Mr Bill!! stack_size=%lu fp-bot = %lu ip-subr_stub_code=%lu engine=%d\n",
+ vp->stack_size, fp - vp->stack_bottom, vp->ip - pstub, vp->engine);
+ break;
+ }
+ fprev = fp;
+
scm_t_ptrdiff nlocals = SCM_FRAME_NUM_LOCALS (fp, sp);
size_t slot = nlocals - 1;
for (slot = nlocals - 1; sp < fp; sp++, slot--)
Oh no Mr Bill!! stack_size=512 fp-bot = 455 ip-subr_stub_code=14 engine=1
Oh no Mr Bill!! stack_size=512 fp-bot = 455 ip-subr_stub_code=14 engine=1
after a wall-clock hour of runtime, it crashes:
#0 0x00007ffff7affb2a in scm_frame_instruction_pointer (frame=0x5bcae580)
at ../../libguile/frames.c:346
frames.h:#define SCM_VM_FRAME_IP(f) SCM_VM_FRAME_DATA (f)->ip
frames.h:#define SCM_VM_FRAME_DATA(x) ((struct scm_frame *)SCM_CELL_WORD_1 (x))
#define SCM_CELL_WORD_1(x) SCM_CELL_WORD ((x), 1)
#define SCM_CELL_WORD(x, n) SCM_GC_CELL_WORD ((x), (n))
gc.h:#define SCM_GC_CELL_WORD(x, n) (SCM_UNPACK (SCM_GC_CELL_OBJECT ((x), (n))))
gc.h:#define SCM_GC_CELL_OBJECT(x, n) (((SCM *)SCM2PTR (x)) [n])
gc.h:#define SCM2PTR(x) ((scm_t_cell *) (SCM_UNPACK_POINTER (x)))
etc.
(gdb) x/8x 0x5bcae580
0x5bcae580: 0xffffffffa49232af 0x0000000000000000
0x5bcae590: 0xffffffffa396e59f 0x0000000000000000
0x5bcae5a0: 0xffffffffa4924287 0x0000000000000000
0x5bcae5b0: 0xffffffffa39d15df 0x0000000000000000
All of these look bad, and will clearly crash if derefernced.
(gdb) x 0xa49232af
0xa49232af: Cannot access memory at address 0xa49232af
(gdb) x 0xffffffffa396e59f
0xffffffffa396e59f: Cannot access memory at address 0xffffffffa396e59f
#1 0x00007ffff7b67331 in vm_debug_engine (thread=0x5bcae580, vp=0x5b62c480,
registers=0x7ffff7b04330 <scm_apply_subr+176>, resume=-536784)
at ../../libguile/vm-engine.c:784
#2 0x00007ffff7b7128a in scm_call_n (proc=0x7fffede811e8,
argv=argv@entry=0x7ffe53ffddd8, nargs=nargs@entry=1)
at ../../libguile/vm.c:1269
#3 0x00007ffff7af1568 in scm_call_1 (proc=<optimized out>, arg1=0x5bcae580)
at ../../libguile/eval.c:487
#4 0x00007ffff7affd08 in scm_i_frame_print (frame=0x5bcae580, port=0x16f10160,
pstate=<optimized out>) at ../../libguile/frames.c:48
why are we printing???
#5 0x00007ffff7b33c4b in scm_prin1 (exp=exp@entry=0x5bcae580,
port=port@entry=0x16f10160, writingp=writingp@entry=1)
at ../../libguile/print.c:817
#6 0x00007ffff7b34b11 in scm_write (obj=0x5bcae580, port=0x16f10160)
at ../../libguile/print.c:1076
why are we writing?
#7 0x00007ffff7b67331 in vm_debug_engine (thread=0x5bcae580, vp=0x5b62c480,
registers=0x7ffff7b04330 <scm_apply_subr+176>, resume=-536784)
at ../../libguile/vm-engine.c:784
#8 0x00007ffff7b7128a in scm_call_n (proc=0x7fffeddbed70,
argv=argv@entry=0x7ffe53ffe080, nargs=nargs@entry=4)
at ../../libguile/vm.c:1269
#9 0x00007ffff7af1644 in scm_call_4 (proc=<optimized out>,
arg1=arg1@entry=0x59d5fee0, arg2=arg2@entry=0x59d5cf20,
arg3=arg3@entry=0x983520, arg4=arg4@entry=0x52) at ../../libguile/eval.c:508
#10 0x00007ffff7ae29a5 in display_backtrace_body (a=<optimized out>)
at ../../libguile/backtrace.c:244
Ohh, why is this being done?
#11 0x00007ffff7b67331 in vm_debug_engine (thread=0x5bcae580, vp=0x5b62c480,
registers=0x7ffff7b04330 <scm_apply_subr+176>, resume=-536784)
at ../../libguile/vm-engine.c:784
#12 0x00007ffff7b7128a in scm_call_n (proc=proc@entry=0x59d5cf00,
argv=argv@entry=0x0, nargs=nargs@entry=0) at ../../libguile/vm.c:1269
#13 0x00007ffff7af1549 in scm_call_0 (proc=proc@entry=0x59d5cf00)
at ../../libguile/eval.c:481
#14 0x00007ffff7b600f9 in catch (tag=tag@entry=0x404, thunk=0x59d5cf00,
handler=0x59d5cee0, pre_unwind_handler=0x4) at ../../libguile/throw.c:137
#15 0x00007ffff7b60465 in scm_catch_with_pre_unwind_handler (key=key@entry=0x404,
thunk=<optimized out>, handler=<optimized out>,
pre_unwind_handler=<optimized out>) at ../../libguile/throw.c:254
#16 0x00007ffff7b6061f in scm_c_catch (tag=tag@entry=0x404,
body=body@entry=0x7ffff7ae2880 <display_backtrace_body>,
body_data=body_data@entry=0x7ffe53ffe460,
handler=handler@entry=0x7ffff7ae2c70 <error_during_backtrace>,
handler_data=<optimized out>,
pre_unwind_handler=pre_unwind_handler@entry=0x0, pre_unwind_handler_data=0x0)
at ../../libguile/throw.c:377
#17 0x00007ffff7b6062e in scm_internal_catch (tag=tag@entry=0x404,
body=body@entry=0x7ffff7ae2880 <display_backtrace_body>,
body_data=body_data@entry=0x7ffe53ffe460,
handler=handler@entry=0x7ffff7ae2c70 <error_during_backtrace>,
handler_data=<optimized out>) at ../../libguile/throw.c:386
#18 0x00007ffff7ae2855 in scm_display_backtrace_with_highlights (
stack=<optimized out>, port=<optimized out>, first=<optimized out>,
depth=<optimized out>, highlights=<optimized out>)
at ../../libguile/backtrace.c:282
Yuck.
#19 0x00007fffecb62654 in opencog::SchemeEval::catch_handler(scm_unused_struct*,
scm_unused_struct*) (this=0x7ffe8c000980, tag=0x7a86e0, throw_args=0x5bcda980)
at /home/ubuntu/src/atomspace/opencog/guile/SchemeEval.cc:420
Oh. Ewww. Something that got sent caused an exception. But what?
Lets print to the log file.
#20 0x00007ffff7b67331 in vm_debug_engine (thread=0x5bcae580, vp=0x5b62c480,
registers=0x7ffff7b04330 <scm_apply_subr+176>, resume=-536784)
at ../../libguile/vm-engine.c:784
#21 0x00007ffff7b7128a in scm_call_n (proc=proc@entry=0x5bcae600,
argv=<optimized out>, nargs=5) at ../../libguile/vm.c:1269
#22 0x00007ffff7af1a3b in scm_apply_0 (proc=proc@entry=0x5bcae600, args=0x304)
at ../../libguile/eval.c:594
#23 0x00007ffff7b60177 in catch (tag=<optimized out>, thunk=0x5bcae620,
handler=0x5bcae600, pre_unwind_handler=0x5bcae5e0)
at ../../libguile/throw.c:134
#24 0x00007fffecb62d29 in opencog::SchemeEval::do_eval(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&) (this=0x7ffe8c000980,
expr=...) at /home/ubuntu/src/atomspace/opencog/guile/SchemeEval.cc:572
#25 0x00007fffecb62dda in opencog::SchemeEval::c_wrap_eval(void*) (
p=0x7ffe8c000980)
at /home/ubuntu/src/atomspace/opencog/guile/SchemeEval.cc:501
#26 0x00007ffff7aeb70a in c_body (d=0x7ffe53ffece0)
at ../../libguile/continuations.c:422
#27 0x00007ffff7b67331 in vm_debug_engine (thread=0x5bcae580, vp=0x5b62c480,
registers=0x7ffff7b04330 <scm_apply_subr+176>, resume=-536784)
at ../../libguile/vm-engine.c:784
<<<<<<<<<< first one is here.
#28 0x00007ffff7b7128a in scm_call_n (proc=proc@entry=0x5c6d3580,
argv=argv@entry=0x0, nargs=nargs@entry=0) at ../../libguile/vm.c:1269
#29 0x00007ffff7af1549 in scm_call_0 (proc=proc@entry=0x5c6d3580)
at ../../libguile/eval.c:481
#30 0x00007ffff7b600f9 in catch (tag=tag@entry=0x404, thunk=0x5c6d3580,
handler=0x5c6d3560, pre_unwind_handler=0x5c6d3540)
at ../../libguile/throw.c:137
#31 0x00007ffff7b60465 in scm_catch_with_pre_unwind_handler (key=key@entry=0x404,
thunk=<optimized out>, handler=<optimized out>,
pre_unwind_handler=<optimized out>) at ../../libguile/throw.c:254
#32 0x00007ffff7b6061f in scm_c_catch (tag=tag@entry=0x404,
body=body@entry=0x7ffff7aeb700 <c_body>,
body_data=body_data@entry=0x7ffe53ffece0,
handler=handler@entry=0x7ffff7aeb9a0 <c_handler>,
handler_data=handler_data@entry=0x7ffe53ffece0,
pre_unwind_handler=pre_unwind_handler@entry=0x7ffff7aeb800 <pre_unwind_handler>, pre_unwind_handler_data=0x7ff020) at ../../libguile/throw.c:377
#33 0x00007ffff7aebd10 in scm_i_with_continuation_barrier (
body=body@entry=0x7ffff7aeb700 <c_body>,
body_data=body_data@entry=0x7ffe53ffece0,
handler=handler@entry=0x7ffff7aeb9a0 <c_handler>,
handler_data=handler_data@entry=0x7ffe53ffece0,
pre_unwind_handler=pre_unwind_handler@entry=0x7ffff7aeb800 <pre_unwind_handler>, pre_unwind_handler_data=0x7ff020) at ../../libguile/continuations.c:360
#34 0x00007ffff7aebdf5 in scm_c_with_continuation_barrier (func=<optimized out>,
data=<optimized out>) at ../../libguile/continuations.c:456
#35 0x00007ffff717f127 in GC_call_with_gc_active (
fn=fn@entry=0x7ffff7b5e200 <with_guile_trampoline>,
client_data=client_data@entry=0x7ffe53ffedc0) at ../pthread_support.c:1343
#36 0x00007ffff7b5ec11 in with_guile (base=base@entry=0x7ffe53ffed90,
data=data@entry=0x7ffe53ffedc0) at ../../libguile/threads.c:688
#37 0x00007ffff7179132 in GC_call_with_stack_base (
fn=fn@entry=0x7ffff7b5eb80 <with_guile>, arg=arg@entry=0x7ffe53ffedc0)
at ../misc.c:1941
#38 0x00007ffff7b5efb8 in scm_i_with_guile (dynamic_state=<optimized out>,
data=<optimized out>, func=<optimized out>) at ../../libguile/threads.c:703
#39 scm_with_guile (func=<optimized out>, data=<optimized out>)
at ../../libguile/threads.c:709
#40 0x00007fffecb62e5e in opencog::SchemeEval::eval_expr(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (
this=0x7ffe8c000980, expr=...)
at /home/ubuntu/src/atomspace/opencog/guile/SchemeEval.cc:473
#41 0x00007fffdef6eb6c in opencog::GenericShell::eval_loop (this=0x7fff69581ac0)
at /home/ubuntu/src/opencog/opencog/cogserver/shell/GenericShell.cc:450
(observe-text "In the calmness of the evening, when the winds had sunk to rest, When no earthquake heaved its fury, burned no fire within my breast, Came a still small voice so tender, it the heart of Christ confessed: \"Unto thy God be true!\"")
Interesting. Unescaped quotes! FIXED
also - unescaped carriage returns FIXED.
also sentence spliter fails when there are footnotes:
.[571] FIXED
also at any time."[572] FIXED
but also...
ERROR: In procedure fport_read: Connection reset by peer
Java eats .\n as a string.... but does not eat \n by itself. FIXED
also it un-escapes the quotes. FIXED
Ohhh: right.\n[?].n FIXED
skip setWordAndPos or rather use simplified form of it.
setWordAndPos
Remove from Linkable...
add loop for englsih when? in the algs... in Morphy? No, too late
in RelationExtractor.java: in
FeatureNode fn = _parse.getLeft();
while (fn != null)
fn = fn.get("NEXT");
(use-modules (ice-9 textual-ports))
(define (fubar sent-txt)
(let ((s (socket PF_INET SOCK_STREAM 0)))
(connect s AF_INET (inet-pton AF_INET "127.0.0.1") 4445)
(set-port-encoding! s "utf-8")
(display sent-txt s) (display "\n" s)
(format #t "Its ~A\n" (get-string-all s))
(close-port s)))
Oh no Mr Bill!! stack_size=512 fp-bot = 455 ip-subr_stub_code=14 engine=1 Mon Jun 5 03:43:12 2017
Oh no Mr Bill!! stack_size=512 fp-bot = 455 ip-subr_stub_code=14 engine=1 Mon Jun 5 05:53:08 2017
Oh no Mr Bill!! stack_size=512 fp-bot = 455 ip-subr_stub_code=14 engine=1 Mon Jun 5 05:53:08 2017
Oh no Mr Bill!! stack_size=512 fp-bot = 455 ip-subr_stub_code=14 engine=1 Mon Jun 5 23:06:13 2017
This one has an exception half-an-hour earlier:
[2017-06-05 22:37:22:726] [INFO] Aiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiieee it
was this
: Backtrace: ...
In opencog/base/file-utils.scm:
79:27 5 (exec-scm-from-port _)
In ice-9/boot-9.scm:
849:4 4 (with-throw-handler _ _ _)
849:4 3 (with-throw-handler _ _ _)
In unknown file:
2 (string-concatenate-reverse (" (WordInstanceNode \"of@d7a23…" …) …)
In ice-9/boot-9.scm:
759:25 1 (dispatch-exception 2 wrong-type-arg ("string-append" "Wrong…" …))
In unknown file:
0 (apply-smob/1 #<catch-closure 2425880> wrong-type-arg "string-…" …)
ERROR: In procedure apply-smob/1:
ERROR: In procedure string-append: Wrong type (expecting string): (filename . #f)
ABORT: wrong-type-arg
But there are no files anywhere in this code...
[2017-06-05 23:06:14:154] [INFO] Aiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiieee it
was this: Backtrace:
In opencog/base/file-utils.scm:
79:27 5 (exec-scm-from-port _)
In ice-9/boot-9.scm:
849:4 4 (with-throw-handler _ _ _)
849:4 3 (with-throw-handler _ _ _)
In unknown file:
2 (string-concatenate-reverse ((#<tree-il (const "ANY")>)) "\n …" …)
In ice-9/boot-9.scm:
759:25 1 (dispatch-exception 2 wrong-type-arg ("string-append" "Wrong…" …))
In unknown file:
0 (apply-smob/1 #<catch-closure 276ce00> wrong-type-arg "string-…" …)
ERROR: In procedure apply-smob/1:
ERROR: In procedure string-append: Wrong type (expecting string):
(#<tree-il (const "ANY")>)
ABORT: wrong-type-arg
Oh no Mr Bill!! stack_size=512 fp-bot = 455 ip-subr_stub_code=14
engine=1 Tue Jun 6 02:39:21 2017
Oh no Mr Bill!! stack_size=512 fp-bot = 455 ip-subr_stub_code=14
engine=1 Tue Jun 6 02:39:21 2017
.... and then a crash:
the crash is related to a backtrace, again.
Thread 24232 "guile" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffee57fa700 (LWP 10775)]
0x00007ffff7b0d809 in scm_is_pair (x=0x0) at ../../libguile/pairs.h:159
159 return SCM_I_CONSP (x);
(gdb) bt
#0 0x00007ffff7b0d809 in scm_is_pair (x=0x0) at
../../libguile/pairs.h:159
#1 scm_ilength (sx=<optimized out>) at ../../libguile/list.c:190
#2 0x00007ffff7b0d859 in scm_list_p (x=<optimized out>)
at ../../libguile/list.c:168
#3 0x00007ffff7b66381 in vm_debug_engine (thread=0x54569480,
vp=0x1644e990,
registers=0x0, resume=-536832) at ../../libguile/vm-engine.c:784
#4 0x00007ffff7b702fa in scm_call_n (proc=0x7fffeddbdd70,
argv=argv@entry=0x7ffee57f9070, nargs=nargs@entry=4)
at ../../libguile/vm.c:1271
#5 0x00007ffff7af0694 in scm_call_4 (proc=<optimized out>,
arg1=arg1@entry=0x54c8fd90, arg2=arg2@entry=0x547a18c0,
arg3=arg3@entry=0x984520, arg4=arg4@entry=0x52)
at ../../libguile/eval.c:508
#6 0x00007ffff7ae19f5 in display_backtrace_body (a=<optimized out>)
at ../../libguile/backtrace.c:244
#7 0x00007ffff7b66381 in vm_debug_engine (thread=0x54569480,
vp=0x1644e990,
registers=0x0, resume=-536832) at ../../libguile/vm-engine.c:784
#8 0x00007ffff7b702fa in scm_call_n (proc=proc@entry=0x547a1700,
argv=argv@entry=0x0, nargs=nargs@entry=0) at ../../libguile/vm.c:1271
#9 0x00007ffff7af0599 in scm_call_0 (proc=proc@entry=0x547a1700)
at ../../libguile/eval.c:481
#10 0x00007ffff7b5f149 in catch (tag=tag@entry=0x404, thunk=0x547a1700,
---Type <return> to continue, or q <return> to quit---
handler=0x547a16e0, pre_unwind_handler=0x4) at
../../libguile/throw.c:137
#11 0x00007ffff7b5f4b5 in scm_catch_with_pre_unwind_handler (
key=key@entry=0x404, thunk=<optimized out>, handler=<optimized out>,
pre_unwind_handler=<optimized out>) at ../../libguile/throw.c:254
#12 0x00007ffff7b5f66f in scm_c_catch (tag=tag@entry=0x404,
body=body@entry=0x7ffff7ae18d0 <display_backtrace_body>,
body_data=body_data@entry=0x7ffee57f9450,
handler=handler@entry=0x7ffff7ae1cc0 <error_during_backtrace>,
handler_data=<optimized out>,
pre_unwind_handler=pre_unwind_handler@entry=0x0,
pre_unwind_handler_data=0x0) at ../../libguile/throw.c:377
#13 0x00007ffff7b5f67e in scm_internal_catch (tag=tag@entry=0x404,
body=body@entry=0x7ffff7ae18d0 <display_backtrace_body>,
body_data=body_data@entry=0x7ffee57f9450,
handler=handler@entry=0x7ffff7ae1cc0 <error_during_backtrace>,
handler_data=<optimized out>) at ../../libguile/throw.c:386
#14 0x00007ffff7ae18a5 in scm_display_backtrace_with_highlights (
stack=<optimized out>, port=<optimized out>, first=<optimized out>,
depth=<optimized out>, highlights=<optimized out>)
at ../../libguile/backtrace.c:282
#15 0x00007fffecb61684 in
opencog::SchemeEval::catch_handler(scm_unused_struct*,
scm_unused_struct*) (this=0x7fff98000980, tag=0x7a86e0,
throw_args=0x5148e850)
---Type <return> to continue, or q <return> to quit---
at /home/ubuntu/src/atomspace/opencog/guile/SchemeEval.cc:422
#16 0x00007ffff7b66381 in vm_debug_engine (thread=0x54569480,
vp=0x1644e990,
registers=0x0, resume=-536832) at ../../libguile/vm-engine.c:784
#17 0x00007ffff7b702fa in scm_call_n (proc=proc@entry=0x54569500,
argv=<optimized out>, nargs=5) at ../../libguile/vm.c:1271
#18 0x00007ffff7af0a8b in scm_apply_0 (proc=proc@entry=0x54569500,
args=0x304)
at ../../libguile/eval.c:594
#19 0x00007ffff7b5f1c7 in catch (tag=<optimized out>, thunk=0x54569520,
handler=0x54569500, pre_unwind_handler=0x545694e0)
at ../../libguile/throw.c:134
#20 0x00007fffecb61d89 in
opencog::SchemeEval::do_eval(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&) (
this=0x7fff98000980, expr=...)
at /home/ubuntu/src/atomspace/opencog/guile/SchemeEval.cc:576
#21 0x00007fffecb61e3a in opencog::SchemeEval::c_wrap_eval(void*) (
p=0x7fff98000980)
at /home/ubuntu/src/atomspace/opencog/guile/SchemeEval.cc:505
#22 0x00007ffff7aea75a in c_body (d=0x7ffee57f9ce0)
at ../../libguile/continuations.c:422
#23 0x00007ffff7b66381 in vm_debug_engine (thread=0x54569480,
vp=0x1644e990,
registers=0x0, resume=-536832) at ../../libguile/vm-engine.c:784
#24 0x00007ffff7b702fa in scm_call_n (proc=proc@entry=0x545695c0,
---Type <return> to continue, or q <return> to quit---
argv=argv@entry=0x0, nargs=nargs@entry=0) at ../../libguile/vm.c:1271
#25 0x00007ffff7af0599 in scm_call_0 (proc=proc@entry=0x545695c0)
at ../../libguile/eval.c:481
#26 0x00007ffff7b5f149 in catch (tag=tag@entry=0x404, thunk=0x545695c0,
handler=0x545695a0, pre_unwind_handler=0x54569580)
at ../../libguile/throw.c:137
#27 0x00007ffff7b5f4b5 in scm_catch_with_pre_unwind_handler (
key=key@entry=0x404, thunk=<optimized out>, handler=<optimized out>,
pre_unwind_handler=<optimized out>) at ../../libguile/throw.c:254
#28 0x00007ffff7b5f66f in scm_c_catch (tag=tag@entry=0x404,
body=body@entry=0x7ffff7aea750 <c_body>,
body_data=body_data@entry=0x7ffee57f9ce0,
handler=handler@entry=0x7ffff7aea9f0 <c_handler>,
handler_data=handler_data@entry=0x7ffee57f9ce0,
pre_unwind_handler=pre_unwind_handler@entry=0x7ffff7aea850
<pre_unwind_handler>, pre_unwind_handler_data=0x7ff020) at
../../libguile/throw.c:377
#29 0x00007ffff7aead60 in scm_i_with_continuation_barrier (
body=body@entry=0x7ffff7aea750 <c_body>,
body_data=body_data@entry=0x7ffee57f9ce0,
handler=handler@entry=0x7ffff7aea9f0 <c_handler>,
handler_data=handler_data@entry=0x7ffee57f9ce0,
pre_unwind_handler=pre_unwind_handler@entry=0x7ffff7aea850
<pre_unwind_handl---Type <return> to continue, or q <return> to quit---
er>, pre_unwind_handler_data=0x7ff020) at
../../libguile/continuations.c:360
#30 0x00007ffff7aeae45 in scm_c_with_continuation_barrier (
func=<optimized out>, data=<optimized out>)
at ../../libguile/continuations.c:456
#31 0x00007ffff717e127 in GC_call_with_gc_active (
fn=fn@entry=0x7ffff7b5d250 <with_guile_trampoline>,
client_data=client_data@entry=0x7ffee57f9dc0) at
../pthread_support.c:1343
#32 0x00007ffff7b5dc61 in with_guile (base=base@entry=0x7ffee57f9d90,
data=data@entry=0x7ffee57f9dc0) at ../../libguile/threads.c:688
#33 0x00007ffff7178132 in GC_call_with_stack_base (
fn=fn@entry=0x7ffff7b5dbd0 <with_guile>,
arg=arg@entry=0x7ffee57f9dc0)
at ../misc.c:1941
#34 0x00007ffff7b5e008 in scm_i_with_guile (dynamic_state=<optimized
out>,
data=<optimized out>, func=<optimized out>) at
../../libguile/threads.c:703
#35 scm_with_guile (func=<optimized out>, data=<optimized out>)
at ../../libguile/threads.c:709
#36 0x00007fffecb61ebe in
opencog::SchemeEval::eval_expr(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&) (
this=0x7fff98000980, expr=...)
at /home/ubuntu/src/atomspace/opencog/guile/SchemeEval.cc:477
#37 0x00007fffdef6def4 in opencog::GenericShell::eval_loop (
this=0x7ffee8253d50)
---Type <return> to continue, or q <return> to quit---
at
/home/ubuntu/src/opencog/opencog/cogserver/shell/GenericShell.cc:497
#38 0x00007fffeb71dc80 in ?? () from
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
#39 0x00007ffff78916ba in start_thread (arg=0x7ffee57fa700)
at pthread_create.c:333
#40 0x00007ffff75c782d in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb)
Oh no Mr Bill!! stack_size=512 fp-bot = 455 ip-subr_stub_code=14 engine=1 Tue Jun 6 10:16:44 2017
Oh no Mr Bill!! stack_size=512 fp-bot = 455 ip-subr_stub_code=14 engine=1 Tue Jun 6 10:16:44 2017
Oh no Mr Bill!! stack_size=512 fp-bot = 455 ip-subr_stub_code=14 engine=1 Tue Jun 6 18:27:01 2017
Oh no Mr Bill!! stack_size=512 fp-bot = 455 ip-subr_stub_code=14 engine=1 Tue Jun 6 18:27:01 2017
----
This time, no prints at all!! just hit abort:
scheme@(guile-user)>
Thread 451 "guile" received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffd25ffb700 (LWP 5037)]
(gdb) bt
#0 0x00007ffff74f6428 in __GI_raise (sig=sig@entry=6)
at ../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007ffff74f802a in __GI_abort () at abort.c:89
#2 0x00007ffff7af22f4 in compute_assigned (exp=<optimized out>,
assigned=assigned@entry=0x24ba1c0) at ../../libguile/expand.c:1268
#3 0x00007ffff7af2309 in compute_assigned (exp=<optimized out>,
assigned=assigned@entry=0x24ba1c0) at ../../libguile/expand.c:1185
#4 0x00007ffff7af2388 in compute_assigned (exp=<optimized out>,
assigned=assigned@entry=0x24ba1c0) at ../../libguile/expand.c:1227
#5 0x00007ffff7af2309 in compute_assigned (exp=<optimized out>,
assigned=assigned@entry=0x24ba1c0) at ../../libguile/expand.c:1185
#6 0x00007ffff7af2309 in compute_assigned (exp=exp@entry=0xd0366c0,
assigned=assigned@entry=0x24ba1c0) at ../../libguile/expand.c:1185
#7 0x00007ffff7af3e1e in scm_convert_assignment (exp=0xd0366c0)
at ../../libguile/expand.c:1570
#8 0x00007ffff7b12a45 in scm_memoize_expression (exp=<optimized out>)
at ../../libguile/memoize.c:641
#9 0x00007ffff7b66381 in vm_debug_engine (thread=0xabe, vp=0xc65240,
registers=0x6, resume=-145791960) at ../../libguile/vm-engine.c:784
#10 0x00007ffff7b702fa in scm_call_n (proc=0x7fffe7c9d4a8,
argv=argv@entry=0x7ffd25ffa300, nargs=nargs@entry=3)
at ../../libguile/vm.c:1271
#11 0x00007ffff7af063f in scm_call_3 (proc=<optimized out>,
arg1=<optimized out>, arg2=<optimized out>, arg3=<optimized out>)
at ../../libguile/eval.c:501
well, that's unexpected....
(gdb) print exp
$1 = <optimized out>
wtf....
another crazy:
Thread 11590 "guile" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff437fe700 (LWP 1316)]
0x00007ffff7b2b067 in encode_escape_sequence (ch=-134642584,
buf=buf@entry=0x7fff437fcd70 "\\x00e3c0") at
../../libguile/ports.c:3053
3053 buf[i++] = hex[ch / 16];
(gdb) bt
#0 0x00007ffff7b2b067 in encode_escape_sequence (ch=-134642584,
buf=buf@entry=0x7fff437fcd70 "\\x00e3c0") at
../../libguile/ports.c:3053
ch is -134642584
(gdb) print/x ch
$9 = 0xf7f98468
So clearly ports.c fails to check for positive.
Oh no Mr Bill!! stack_size=512 fp-bot = 455 ip-subr_stub_code=14 engine=1
Tue Jun 6 20:27:36 2017
Oh no Mr Bill!! stack_size=512 fp-bot = 455 ip-subr_stub_code=14 engine=1
Wed Jun 7 02:06:40 2017
Oh no Mr Bill!! stack_size=512 fp-bot = 455 ip-subr_stub_code=14 engine=1
Wed Jun 7 02:06:40 2017
Oh no Mr Bill!! stack_size=512 fp-bot = 455 ip-subr_stub_code=14 engine=1
Wed Jun 7 06:04:23 2017
Oh no Mr Bill!! stack_size=512 fp-bot = 455 ip-subr_stub_code=14 engine=1
Wed Jun 7 06:04:23 2017
Thread 326311 "guile" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffe457fa700 (LWP 13958)]
iprin1 (exp=0x0, port=0x85864500, pstate=0x71165c50)
at ../../libguile/print.c:611
611 switch (SCM_TYP7 (exp))
#0 iprin1 (exp=0x0, port=0x85864500, pstate=0x71165c50)
at ../../libguile/print.c:611
#1 0x00007ffff7b32ad0 in scm_iprin1 (exp=<optimized out>,
port=<optimized out>, pstate=<optimized out>) at
../../libguile/print.c:544
#2 0x00007ffff7b60787 in scm_i_variable_print (exp=0x86b77de0,
port=0x85864500, pstate=0x71165c50) at ../../libguile/variable.c:41
#3 0x00007ffff7b32c9b in scm_prin1 (exp=exp@entry=0x86b77de0,
port=port@entry=0x85864500, writingp=writingp@entry=1)
at ../../libguile/print.c:817
#4 0x00007ffff7b33b61 in scm_write (obj=0x86b77de0, port=0x85864500)
at ../../libguile/print.c:1076
#5 0x00007ffff7b66381 in vm_debug_engine (thread=0x0, vp=0x19a7c60,
registers=0xfffffffffffa7a74, resume=16) at
../../libguile/vm-engine.c:784
#6 0x00007ffff7b702fa in scm_call_n (proc=0x7fffeddbdd70,
argv=argv@entry=0x7ffe457f9070, nargs=nargs@entry=4)
at ../../libguile/vm.c:1271
#7 0x00007ffff7af0694 in scm_call_4 (proc=<optimized out>,
arg1=arg1@entry=0x8759cc40, arg2=arg2@entry=0x86101be0,
arg3=arg3@entry=0x983520, arg4=arg4@entry=0x52)
at ../../libguile/eval.c:508
#8 0x00007ffff7ae19f5 in display_backtrace_body (a=<optimized out>)
at ../../libguile/backtrace.c:244
----------------
anotehr:
Thread 213118 "guile" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff61ffb700 (LWP 11550)]
scm_ipruk (hdr=0x7ffff7b87ec8 "immediate", ptr=0xffffffff81480f9f,
port=0x7ed4f6a0) at ../../libguile/print.c:946
946 scm_uintprint (SCM_CELL_WORD_0 (ptr), 16, port);
(gdb) bt
#0 scm_ipruk (hdr=0x7ffff7b87ec8 "immediate", ptr=0xffffffff81480f9f,
port=0x7ed4f6a0) at ../../libguile/print.c:946
#1 0x00007ffff7b32b73 in print_vector_or_weak_vector (v=0x7feff9e0,
len=<optimized out>, ref=0x7ffff7b73ac0 <scm_c_weak_vector_ref>,
port=0x7ed4f6a0, pstate=0x847400) at ../../libguile/print.c:561
#2 0x00007ffff7b33245 in iprin1 (exp=0x7feff9e0, port=0x7ed4f6a0,
pstate=0x847400) at ../../libguile/print.c:743
#3 0x00007ffff7b32c9b in scm_prin1 (exp=exp@entry=0x7feff9e0,
port=port@entry=0x7ed4f6a0, writingp=writingp@entry=1)
at ../../libguile/print.c:817
#4 0x00007ffff7b33b61 in scm_write (obj=0x7feff9e0, port=0x7ed4f6a0)
at ../../libguile/print.c:1076
#5 0x00007ffff7b66381 in vm_debug_engine (thread=0x7e25ea80,
vp=0x160d2360,
registers=0x4d, resume=8) at ../../libguile/vm-engine.c:784
#6 0x00007ffff7b702fa in scm_call_n (proc=0x7fffeddbdd70,
argv=argv@entry=0x7fff61ffa070, nargs=nargs@entry=4)
at ../../libguile/vm.c:1271
#7 0x00007ffff7af0694 in scm_call_4 (proc=<optimized out>,
arg1=arg1@entry=0x7eabfee0, arg2=arg2@entry=0x7eabefa0,
arg3=arg3@entry=0x983520, arg4=arg4@entry=0x52)
at ../../libguile/eval.c:508
#8 0x00007ffff7ae19f5 in display_backtrace_body (a=<optimized out>)
-----------------
Another one:
Thread 230572 "guile" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffec77fe700 (LWP 3880)]
scm_reverse (lst=lst@entry=0x80003e30) at ../../libguile/list.c:346
#0 scm_reverse (lst=lst@entry=0x80003e30) at ../../libguile/list.c:346
#1 0x00007ffff7b4d1ea in scm_string_concatenate_reverse (ls=0x80003e30,
final_string=0x7fc19520, end=<optimized out>)
at ../../libguile/srfi-13.c:2465
#2 0x00007ffff7b66381 in vm_debug_engine (
thread=0x7ffff73cd260 <GC_allocate_ml>, vp=0x7c1fd3f0,
registers=0x7ffff73cd260 <GC_allocate_ml>, resume=1)
at ../../libguile/vm-engine.c:784
#3 0x00007ffff7b702fa in scm_call_n (proc=0x7fffe7c9d4a8,
argv=argv@entry=0x7ffec77fd540, nargs=nargs@entry=3)
at ../../libguile/vm.c:1271
#4 0x00007ffff7af063f in scm_call_3 (proc=<optimized out>,
arg1=<optimized out>, arg2=<optimized out>, arg3=<optimized out>)
at ../../libguile/eval.c:501
#5 0x00007ffff7b66381 in vm_debug_engine (
thread=0x7ffff73cd260 <GC_allocate_ml>, vp=0x7c1fd3f0,
registers=0x7ffff73cd260 <GC_allocate_ml>, resume=1)
at ../../libguile/vm-engine.c:784
#6 0x00007ffff7b702fa in scm_call_n (proc=proc@entry=0x7cc865a0,
argv=argv@entry=0x0, nargs=nargs@entry=0) at ../../libguile/vm.c:1271
#7 0x00007ffff7af0599 in scm_call_0 (proc=proc@entry=0x7cc865a0)
at ../../libguile/eval.c:481
#8 0x00007ffff7b5f149 in catch (tag=<optimized out>, thunk=0x7cc865a0,
handler=0x7cc86580, pre_unwind_handler=0x7cc86560)
at ../../libguile/throw.c:137
#9 0x00007fffecb61d89 in opencog::SchemeEval::do_eval(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (
this=0x7fff3c000980, expr=...)
at /home/ubuntu/src/atomspace/opencog/guile/SchemeEval.cc:576
Well, that's different: for the first time ever, it is NOT in an
exception context!!!
GC_MALLOC_ATOMIC GC_malloc_atomic
scm_gc_malloc_pointerless
scm_gc_strndup
=======================
other guile bugs:
(acos 1.0) can return small imaginaey number
(fold of reals can return small imaginary parts
=======================
total unique pairs
1) no way to get all pairs...
2) support not computed or stored.. DONE
3) stored support not fetchable FIXED with report-API
4) store left and right sizes DONE
cache-all-pair-freqs returs total
* sep out bin-counting code into own file DONE
add-subtotal-mi-compute DONE
call-only-once in sing
fetch-all-words in sing
add-pair-freq-api in mst
(use-modules (opencog) (opencog persist) (opencog persist-sql))
(use-modules (opencog nlp) (opencog nlp learn))
(use-modules (opencog analysis))
(sql-open "postgres:///en_pairs_sim?user=linas")
(fetch-all-words)
(define ala (make-any-link-api))
(ala 'fetch-pairs) << 1099 secs 1074 1028
(batch-short ala)
(batch-all-pair-mi ala)
Support: found num left=395274 num right=396260 in 426 secs << 395, ph:527
Done with wild-card count N(x,*) and N(*,y) in 1259 secs << ph:1300
Done computing N(*,*) total-count=418235277.0 in 51 secs << ph:46
Going to do individual pair frequencies
Done computing 8880914 pairs in 1074 secs << 1007 ph:872
Start computing log P(*,y)
Done computing 396260 left-wilds in 61 secs << ph:51
Done storing 396260 left-wilds in 121 secs << ph:115
Done computing 395274 right-wilds in 55 secs << ph:51
Done with -log P(*,y), start -log P(x,*)
Done storing 395274 right-wilds in 121 secs << ph:127
Done computing -log P(x,*) and P(*,y)
Going to do individual pair MI
Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS
Fuck.
alloc.c: ABORT("Too many heap sections: Increase MAXHINCR or
MAX_HEAP_SECTS");
MAX_HEAP_SECTS
include/private/gc_priv.h:# define MAXHINCR 4096
define HBLKSIZE 4096
include/private/gc_priv.h:#
define MAX_HEAP_SECTS 8192 if -DLARGE_CONFIG
--enable-large-config
--enable-parallel-mark didn't do this.
Done computing 8880914 pair MI's in 3590 secs
Done storing 8880914 pairs in 4190 secs
Going to do column and row subtotals
Finished left entropy subtotals in 580 secs
Finished right entropy subtotals in 517 secs
Finished left MI subtotals in 353 secs
Finished right MI subtotals in 330 secs
Finished left support subtotals in 1021 secs << 772 after star fix
Finished right support subtotals in 961 secs << 745
Going to compute the left, right and total entropy
Finished left norm totals in 210 secs << 232
Finished right norm totals in 210 secs << 237
Done computing totals in 2360 secs
Start saving left-wildcards
Done storing 396260 left-wilds in 224 secs
Done storing 395274 right-wilds in 230 secs
Finished with MI computations
========================================================
support stats
|(*,y)| = number of non-zero entries for column y
sum_y |(*,y)| = right-support
sqrt [ sum_x N^2 (x,*) / num-rows ]
left-length(y) = sqrt(sum_x N^2(x,y)) = col-length
so sum over rights
divide by num-rights == num cols
so this is right-legnth
<xun> = sum_y N(*,y) / num-cols
= N(*,*) / num-cols
<0w> = sum_y p(*,y) |(*,y)|
<xw> = s x p
= sum_y p(*,y) N(*,y)
but p(*,y) = N(*,y) / N(*,*)
<x^2w> = s x^2 p
= sum_y p(*,y) N^2(*,y)
s (x- <xw>)^2 p =
<x^2w> - 2<xw> s x p + <xw>^2
= <x^2w> - <xw>^2
anyway, its not clear that's what we want, anyway. Do we want count
variation? or length variation? length variation is trite if we've got
totals. But count-variation requires....
2d 2 obj, each = 1, avg=1
<x^2w> = 1/2 + 1/2 = 1 ...
so for lensq, needed to divide by support.
and for avg, need to divide by support too.
Argh. Sim has the old style. pair-entropy
(define fla (add-pair-freq-api ala))
(define rla (add-report-api ala))
(rla 'left-dim)
(rla 'right-dim)
(rla 'num_pairs)
(rla 'left-entropy)
(rla 'right-entropy)
(rla 'total-entropy)
(rla 'total-mi)
(define pca (make-pseudo-cset-api))
(pca 'fetch-pairs)
(batch-all-pair-mi pca)
(define fca (add-pair-freq-api pca))
(define ac (get-all-cset-words))
5205
(fca 'pair-mi xxxx)
So:
(EvaluationLink ; 388
(LinkGrammarRelationshipNode "ANY")
(ListLink (AnyNode "left-word") (WordNode "one-point")))
has no "*-Entropy Key-* why?
because not computeed ... why?
Going to do MI column and row subtotals
should be 'cache-all-subtotals
Going to compute the left, right and total entropy
(define wca (add-pair-stars pca))
(define fca (add-pair-freq-api wca))
(define rb (wca 'right-basis))
(define dj (car rb))
(define lwd (pca 'left-wildcard dj))
(fca 'left-wild-entropy dj)
(define (left-sum FN)
(fold
(lambda (right-item sum)
(+ sum (FN right-item)))
0 rb))
(define (right-sum FN)
(fold
(lambda (left-item sum)
(+ sum (FN left-item)))
0 (wca 'left-basis)))
argh. compute-left-entropy is wrong... FIXED
(define rca (add-report-api wca))
(rca 'set-entropy le re tent)
wild-wild-counta DONE
time for totlaa DONE
(define (an llobj)
(lambda (message . args)
(case message ((name) "the big old name")
(else (apply llobj (cons message args))))))
atom->getKeys()
std::set<Handle> getKeys() const;
todo: print times DONE
todo: create a report. DONE
todo: investigate exception DONE
todo: how many words, on average, go into a pair?
what about hubiness? in the graph con sense?
i.e avg incomiing avg outgoing (i.e. left, right support sizes)
avg left-support: non-zero / num-lefts(rows)
left-size: tot count /num-lefts<F8>x
avg length
( err... left-length 'rigt-basis
===============================================================
(use-modules (opencog) (opencog persist) (opencog persist-sql))
(use-modules (opencog nlp) (opencog nlp learn))
(use-modules (opencog matrix))
(sql-open "postgres:///en_pairs_sim?user=linas")
(sql-open "postgres:///en_pairs_ttwo_mst?user=linas")
(sql-open "postgres:///en_pairs_ttwo_sim?user=linas")
(fetch-all-words)
(define pca (make-pseudo-cset-api))
(define psa (add-pair-stars pca))
(psa 'fetch-pairs)
(define fsa (add-subtotal-filter psa 2 2 4))
(define fsi (add-subtotal-filter psa 50 30 10))
(define pna (add-pair-count-api psa))
(define pfa (add-pair-freq-api psa))
(define pta (make-thresh-pca pfa))
(define aw (get-all-cset-words))
(define w (car aw))
(define cset (cog-incoming-by-type w 'LgWordCset))
(define dj (gdr (car cset)))
(define feig (pta 'left-initial))
(define lm (pta 'left-mult feig))
(lm dj)
(define lit (pta 'left-iterate feig))
(lit w)
(pta 'left-norm feig)
without caching:
scheme@(guile-user)> (pta 'left-norm feig)
left-norm took 0 seconds
$22 = 1.0000000000001292 OK still good. This is sum over 37413 items
(define cfg (pta 'make-cache feig))
(pta 'left-norm cfg)
left-norm took 1 seconds <<< OK makes sense
(pta 'left-norm (pta 'make-cache feig)) << 461 seconds - 81 atoms/sec
ouch! this hashing sucks!
OK, problem: all atoms hashing to same value. OK fixed.
(define lm (pta 'left-mult feig))
(pta 'right-norm lm)
right-norm took 16 seconds <<
right-norm took 9 seconds << baseline, no caching
(define clm (pta 'left-mult cfg))
(pta 'right-norm clm)
right-norm took 11 seconds << with caching. Mkaes sense, since the
cache is a lot slower than feig.
$23 = 1.8948456865590222e-4
-----
(define cclm (pta 'make-cache clm))
(pta 'right-norm cclm)
right-norm took 14 seconds <<< first time makes sense
(pta 'right-norm cclm)
right-norm took 1 seconds << makes sense
------
(define lit (pta 'left-iterate feig))
(pta 'left-norm lit)
left-norm took 2699 seconds << yow
$24 = 1.8266495662334775e-7
(define clit (pta 'right-mult cclm))
(pta 'left-norm clit)
left-norm took 18 seconds << huh. weird.
left-norm took 10 seconds
left-norm took 10 seconds
left-norm took 10 seconds
------
So this is excellent. Lets try it front to back.
(define alit (pta 'right-mult (pta 'make-cache (pta 'left-mult feig))))
(pta 'left-norm alit)
left-norm took 22 seconds << excellent
left-norm took 11 seconds
left-norm took 12 seconds
OK, we are golden I think
(define pta (make-thresh-pca pfa))
(define lit (pta 'left-iterate feig 1))
(define lit2 (pta 'left-iterate lit 1))
(pta 'left-print lit 20)
Wow. Like 4 iterations and its converged already!
(define fsa (add-subtotal-filter psa 2 2 4))
(fsa 'right-stars w)
(fsa 'left-basis-size)
(fsa 'right-basis-size)
(WordNode ",")
(define dj (car (psa 'right-basis))) ; crazy
(define pna (add-pair-count-api psa))
(pna 'left-wild-count dj) ; gives 4.0
(length (psa 'left-stars dj)) gives 1
so its entirely in just one: (WordNode ",") which has 4 obs.
so filter should cut this entirely, but it doesn't...
(define cm (Word ","))
(length (psa 'right-stars cm)) ; 281776
(length (filter
(lambda (PAIR)
(< 30 (pna 'left-wild-count (gdr PAIR)))) for each OK
(psa 'right-stars cm))) loop over all djs
-------------
OK, so the cut to use is the last one:
(define fsi (add-subtotal-filter psa 50 30 10))
(define pfi (add-pair-freq-api fsi))
(define pti (make-thresh-pca pfi))
(define feig (pti 'left-initial))
(pti 'left-norm feig)
(define lm (pti 'left-mult feig))
(length (pti 'right-basis)) ; 14377
(define dj (car (pti 'right-basis)))
(lm dj) ; 8.730046003535126e-9
(define lit (pti 'left-iterate feig 1))
(define w (car (pti 'left-basis)))
(lit w) ; 5.29078468146764e-4 left-norm took 133 seconds
(pti 'right-norm lm) ; 4.12871026183e-4 right-norm took 39 seconds
(pti 'left-print lit 20)
(define lit2 (pti 'left-iterate lit 1))
(pti 'left-print lit2 20)
(define lit3 (pti 'left-iterate lit2 1))
(pti 'left-print lit3 20)
(define lit4 (pti 'left-iterate lit3 1))
(pti 'left-print lit4 20)
Wow. Like 4 iterations and its converged already!
--------------------
OK, knockout time. Kill the period.
(define fsj (add-knockout-filter fsi (list (Word ".")) '()))
(define pfj (add-pair-freq-api fsj))
(define ptj (make-thresh-pca pfj))
(define lit (ptj 'left-iterate feig 4))
Hmm.
(length (pti 'right-stars (Word "."))) ; 2031
the 1403 106378.0
and 1225 96276.0
to 1276 96308.0
” 506 45809.0
, 1703 111982.0
a 958 73760.0
in 750 56751.0
of 890 64753.0
his 691 48728.0
it 606 44211.0
with 480 33681.0
him 425 30345.0
that 729 49714.0
for 479 33092.0
(define psi (add-support-compute pti))
(psi 'right-count (Word ".")) 341112.0
--------------------
OK, cosine time. Overload the pair-freq method.
Need a left and right unit. DONE.
(define pci (make-cosine-matrix pfi))
(define pti (make-thresh-pca pci 'left-unit))
(define lit (pti 'left-iterate feig 4))
(pti 'left-print lit 20)
OK, that didn't work out very much. Why?
(WordNode "." . 0.9342344830920888)
(WordNode "," . 0.24827492849839902)
(WordNode "the" . 0.14147241205706643)
(WordNode "to" . 0.11305294927260087)
(WordNode "and" . 0.10935602770196484)
(WordNode "a" . 0.06263713832111492)
((WordNode "of" . 0.05345591470333568)
((WordNode "was" . 0.04159803271278564)
(cset-vec-cosine (WordNode ".") (WordNode ",")) = 0.5411980781755208
(cset-vec-cosine (WordNode "the") (WordNode ",")) = 0.6949870533623743
(cset-vec-cosine (WordNode "the") (WordNode ".")) = 0.7239559271562869
(cset-vec-cosine (WordNode "the") (WordNode "to")) = 0.7812122780640004
(cset-vec-cosine (WordNode "the") (WordNode "a")) = 0.8863119467021999
(define ac (pti 'left-basis))
(define top-cset-words
(filter (lambda (wrd) (< 200 (cset-vec-word-observations wrd))) ac))
(length top-cset-words)
4918
(sql-open "postgres:///en_pairs_ttwo_sim?user=linas")
(define poi (add-pair-cosine-compute pti))
(poi 'right-cosine WORD-A WORD-B)
(para-batch-sim-pairs top-cset-words 0.5)
--------------------------
Whoa, is this working as expected? Lets check.
(define fsi (add-subtotal-filter psa 50 30 10))
(define pfi (add-pair-freq-api fsi))
well, that's a problem: pfi is returning bogus values
for this filtered thing. Well, as long as no one uses them?
Crap. Wait, what?
(define pci (make-cosine-matrix pfi))
(define pti (make-power-iter-pca pci 'right-unit))
(define wv (pti 'make-left-unit (list (WordNode "the"))))
(wv (WordNode "the")) ; 1 OK
(define wm (pti 'left-iterate wv 1))
(wm (WordNode "the")) ;; ---- wtf Oh; renormalization does this.
(define psi (add-support-compute pti))
(psi 'right-support (Word "the")) ; 1403
(psi 'right-length (Word "the")) ; 9761.855971074354 OK correct for
counts...
(define asi (add-support-compute pci 'right-unit))
(asi 'right-support (Word "the")) ; 1403 OK
; OK, so here, asi should when asked for a pair, return
; the normalized vector component. So the length should be 1.0
(asi 'right-length (Word "the")) wtf!? count si crazy too. OK FIXED
; OK, so frq is too low. where is it from? its from pfi, which is giving
; us freq. so actually looks OK, and length is wrong, since its a count
; len, not a freq len.
(define cset (car (pci 'right-stars (Word "the"))))
(define acc (add-pair-cosine-compute pfi))
(acc 'right-cosine (Word "the") (Word "the"))
; 0.9807641990591952 because the partials are deficient. FIXED
; Can we fix this? FIXED
-- it uses support, to get lengths, which uses freq to get freq, which
is wrong but still ends up using add-support-api which is wrong. FIXED
-- it uses star-obj to get stars. which should be filtered stars.
So:
(acc 'right-cosine (Word "the") (Word "a")) ; 0.8971244127293886 XXX
Should be 0.8964505696268208
(sqrt (acc 'right-product (Word "the") (Word "the"))) ;
9761.855971074354 OK so that agrees with support, now.
Hmm But its off.
(acc 'right-cosine (Word "the") (Word "yes")) ;0.48396212182787846 XXX
(acc 'right-cosine (Word "yes") (Word "the")) ; 0.38874316832684397 OK
(wm (WordNode "yes")) ; 0.38488729600833205
wm symm: ; 0.38488729600833205 so symmetrical and little OFF!!
OK, wtf!
(define acn (add-pair-cosine-compute fsi))
(define acc (add-pair-cosine-compute pfi))
(psi 'right-length (Word "the")) ; 9761.855971074354 OK correct for
(psi 'right-length (Word "yes")) ; 187.6432785899884
(acn 'right-product (Word "the") (Word "yes")) ; 886496.0 was wrong, FIXED.
(acn 'right-product (Word "yes") (Word "the")) ; 712079.0 this was OK.
(/ 886496.0 (* 9761.855971074354 187.6432785899884)) = 0.48396212182787846
ohh. The stars .. but then the cutter is not preventing seeing bad
counts because it does not overload the get-count.
overload ... item-pair and pair-count Done, but this doesn' change pti
(define acc (add-pair-cosine-compute pfi))
(acc 'right-product (Word "the") (Word "yes")) ; 712079.0 wtf! actuall OK
(acc 'right-product (Word "yes") (Word "the")) ; 712079.0 also
(/ 712079.0 (* 9761.855971074354 187.6432785899884)) = 0.38874316832684397
Arghhh. wm is a little off from cosine!! why?
(define acf (add-pair-cosine-compute pfi 'pair-freq))
(acf 'right-cosine (Word "yes") (Word "the")) ; 0.384887296008332
--- which is wrong, cause it bypassed the cutoff
sooo ... pfi gives different results than the count API...
because maybe count not filtered correctly, still????
cosine creates fold, says "use pair-freq"
fold uses pair-freq and bypasses filtering!
solutions:
1) pair-freq should check w/ filter. All of freq-api should.
2) filter should provide pair-freq.
3) avoid cached freq, since its inaccurate in general.
4) freq-api should check for filter, and respond differently. How?
5) api's refuse to work with filters; filter blocks API.
OK, I think that's running reliably now. DONE.
(apply LLOBJ (cons message args)
-----------------------------------------------------------------
(define grate
(let ((lg 0.00001) (ls 0.00001) (tm (current-time)))
(lambda()
(define now (current-time))
(define fra (/ (- sent-cnt ls) (- gc-cnt lg) ))
(define dt (- now tm))
(format #t "delta-t=~A frac=~f gc-rate=~f\n"
dt fra (/ dt (- gc-cnt lg)))
(set! tm (- now 00001))
(set! lg gc-cnt)
(set! ls sent-cnt))))
-----------------------------------------------------------------
So again: stats.
Redo earlier work: but with fixed filtering. line 9305 above
(define fsi (add-subtotal-filter psa 50 30 10))
(define pti (make-power-iter-pca fsi))
(define feig (pti 'make-left-unit (fsi 'left-basis)))
(length (fsi 'left-basis)) ; 13122
(feig (Word "the")) ; = 1/sqrt (13122)
(length (pti 'right-basis)) ;
(define lit (pti 'left-iterate feig 1))
(define w (car (pti 'left-basis)))
(WordNode "cloud")
(lit w) ; 7.963841319884063e-4
left-norm took 582 seconds
(define lit2 (pti 'left-iterate lit 1))
(pti 'left-print lit2 20)
(define lit4 (pti 'left-iterate lit3 1))
(pti 'left-print lit4 20)
OK, this needs a few more steps to converge, but gives same results as
before. Yayy!
------------------------------------------
OK, cosine time. adapt from line 9364
(define pci (make-cosine-matrix fsi))
(define pti (make-power-iter-pca pci 'left-unit))
(define lit (pti 'left-iterate feig 4))
(define lit2 (pti 'left-iterate lit 4))
Substantially similar but not identical results:
the lit1 results are closer.
(pti 'left-print lit2 20)
((WordNode "." . 0.9762388252965614)
((WordNode "," . 0.14180449500019524)
((WordNode "the" . 0.08346874712036408)
((WordNode "to" . 0.07062641675278085)
((WordNode "and" . 0.07032844639726361)
((WordNode "a" . 0.04157464373827347)
((WordNode "of" . 0.03604121229169698)
((WordNode "”" . 0.029226311035184596)
((WordNode "in" . 0.028139776676391105)
((WordNode "that" . 0.026246992008664854)
((WordNode "was" . 0.02540012588054255)
((WordNode "his" . 0.024322426008004088)
((WordNode "he" . 0.023323167571591752)
cosines?
(define poi (add-pair-cosine-compute pti))
(poi 'right-cosine WORD-A WORD-B)
(poi 'right-cosine (Word ".") (Word ",")) ; 0.548721899627726
(poi 'right-cosine (Word ".") (Word ".")) = 0.9999999999999999
(poi 'right-cosine (Word ".") (Word "the")) = 0.7306741325101412
(poi 'right-cosine (Word ".") (Word "to")) = 0.6453504435435683
(poi 'right-cosine (Word ".") (Word "and")) = 0.6683619269954274
(poi 'right-cosine (Word ".") (Word "a")) = 0.6271733855216901
(poi 'right-cosine (Word ",") (Word "the")) = 0.7105031050865372
(poi 'right-cosine (Word ",") (Word "to")) = 0.8240420194068381
(poi 'right-cosine (Word ",") (Word "and")) = 0.8877149213148245
(poi 'right-cosine (Word ",") (Word "a")) = 0.7645779363007278
(poi 'right-cosine (Word "the") (Word "the")) = 0.9999999999999999
(poi 'right-cosine (Word "the") (Word "to")) = 0.7904174107721877
(poi 'right-cosine (Word "the") (Word "and")) = 0.7436842107977618
(poi 'right-cosine (Word "the") (Word "a")) = 0.8964505696268208
(poi 'right-cosine (Word "to") (Word "and")) = 0.9061808294998355
(poi 'right-cosine (Word "to") (Word "a")) = 0.8567320046759815
(poi 'right-cosine (Word "and") (Word "a")) = 0.7545793156905689
OK, substaintially similar to before, about 8% larger.
So we double-checked our data, fixed the bugs, and its good.
Now what?
-----------------------------------------------------------------
(define ac (pti 'left-basis))
(define top-cset-words
(filter (lambda (wrd) (< 1500 (cset-vec-word-observations wrd))) ac))
(length top-cset-words)
(length top-cset-words) = 803
(define (mkcos wa wb) (cons (cons wa wb) (poi 'right-cosine wa wb)))
(define (coprs wrd wrds)
(map (lambda (wb) (mkcos wrd wb)) wrds))
(define (allprs wrds lst)
(if (null? wrds) lst
(allprs (cdr wrds) (append (coprs (car wrds) (cdr wrds)) lst))))
(define asprs (allprs top-cset-words '()))
(length asprs) = 322003
(define also (sort asprs (lambda (a b) (> (cdr a) (cdr b)))))
(take also 200) ;; see below
(define lcprs
(filter
(lambda (scp)
(and (char-lower-case? (car (string->list (cog-name (caar scp)))))
(char-lower-case? (car (string->list (cog-name (cdar scp)))))))
also))
(length lcprs) = 239778
(use-modules (system repl server))
(spawn-server)
rlwrap telnet localhost 37146
The top 200 of the above:
(He, She) . 0.997345758
(Her, His) . 0.996268828
(nodded, sighed) . 0.99601656
(There, It) . 0.995809996
(He, It) . 0.995435813
(Dallas, Two-Bit) . 0.995062883
(Dallas, Jim) . 0.994875764
(But, When) . 0.994776669
(Two-Bit, Mai) . 0.994641695
(Aster, Mai) . 0.994363834
(He, There) . 0.994337552
(After, With) . 0.994303242
(Sandy, Bunny) . 0.994181214
(One, All) . 0.99409611
(Then, But) . 0.99408755
(She, It) . 0.994054139
(And, But) . 0.99404553
(Gene, Mai) . 0.99381092
(Then, When) . 0.993778539
(Jim, Mai) . 0.993727250
(For, With) . 0.993687969
(There, She) . 0.993678998
(It, They) . 0.993620758
(Two-Bit, Jim) . 0.993559994
(That, It) . 0.99348913
(His, All) . 0.993405770
(Ross, Jim) . 0.993276220
(Two-Bit, Gene) . 0.993190381
(It, This) . 0.993141274
(One, His) . 0.993081512
(He, His) . 0.993057520
(Now, Here) . 0.993019266
(For, But) . 0.992958261
(There, His) . 0.992797376
(At, With) . 0.992753993
(There, They) . 0.992725314
(Dallas, Mai) . 0.992627449
(But, That) . 0.992603728
(Richard, Anna) . 0.992601407
(There, This) . 0.992485293
(Gene, Jim) . 0.992427189
(For, After) . 0.992407193
(They, The) . 0.992382512
(She, His) . 0.991863215
(When, With) . 0.991836780
(Dallas, Gene) . 0.991791358
(smiled, sighed) . 0.991786864
(After, At) . 0.991678895
(For, At) . 0.991640239
(He, Her) . 0.991640030
(Now, But) . 0.99161025
(smiled, nodded) . 0.991579848
(After, When) . 0.991579720
(His, It) . 0.99155633
(That, This) . 0.991516495
(She, They) . 0.991503829
(laughed, asked) . 0.991476047
(One, There) . 0.991343749
(For, Then) . 0.991257404
(One, It) . 0.991256062
(His, The) . 0.991234256
(Now, So) . 0.991134393
(For, All) . 0.991128043
(There, All) . 0.99111722
(It, All) . 0.991093382
(Hiccup, Mai) . 0.991088746
(It, The) . 0.991087621
(For, When) . 0.991060842
(Then, After) . 0.99103343
(That, There) . 0.99100824
(He, They) . 0.990965962
(And, So) . 0.990895373
(But, All) . 0.990827920
(Jack, Demelza) . 0.990804758
(Her, She) . 0.990769217
(That, All) . 0.990720297
(Bunny, Anna) . 0.990713341
(Her, There) . 0.990708057
(He, This) . 0.990600114
(After, Even) . 0.990597163
(As, When) . 0.990560365
(Aster, Anna) . 0.990540130
(Margaret, Jim) . 0.990520388
(He, That) . 0.990497589
(Aster, Gene) . 0.99039301
(Sandy, Aster) . 0.990387947
(Then, With) . 0.990387390
(As, With) . 0.990361068
(Then, As) . 0.99034848
(She, This) . 0.990270109
(This, They) . 0.990171297
(Richard, John) . 0.990166796
(He, The) . 0.990001224
(There, The) . 0.989960300
(Aster, Hiccup) . 0.989888481
(His, They) . 0.989796499
(Then, That) . 0.989791187
(Her, The) . 0.989757803
(Ross, Margaret) . 0.98975517
(Bunny, Tooth) . 0.989753554
(Demelza, Anna) . 0.989723864
(But, With) . 0.989686041
(But, It) . 0.989666595
(That, His) . 0.989488364
(She, The) . 0.989417566
(He, But) . 0.989407999
(Richard, Ross) . 0.989276354
(He, Then) . 0.989261202
(At, Even) . 0.989197444
(Pitch, Demelza) . 0.989162716
(He, All) . 0.98912848
(For, That) . 0.989083210
(Margaret, Mai) . 0.988996494
(That, She) . 0.988974985
(Pitch, Ross) . 0.98891348
(Pitch, John) . 0.988908738
(All, This) . 0.988904006
(Sandy, Anna) . 0.988798684
(Ross, Mai) . 0.988786842
(Her, It) . 0.988778883
(Then, All) . 0.988676101
(Bunny, Hiccup) . 0.988607248
(Her, They) . 0.988558601
(And, That) . 0.988510273
(Sandy, Hiccup) . 0.988502544
(For, Here) . 0.988465756
(His, This) . 0.988433038
(That, When) . 0.988401578
(Then, At) . 0.988371519
(Then, It) . 0.988369524
(At, There) . 0.988350034
(So, But) . 0.988346421
(For, There) . 0.988324599
(Aster, Two-Bit) . 0.988296335
(Ross, Anna) . 0.988264218
(All, The) . 0.988254573
(This, The) . 0.988248820
(Ross, Demelza) . 0.988230903
(Richard, Demelza) . 0.988228650
(Anna, Mai) . 0.988222252
(One, Her) . 0.988219868
(He, One) . 0.98818965
(There, With) . 0.9881787
(Gene, Margaret) . 0.988139740
(But, After) . 0.988126
(But, This) . 0.988098723
(But, His) . 0.988057331
(Richard, Bunny) . 0.988050499
(Her, All) . 0.988048392
(Then, There) . 0.98801170
(Ross, John) . 0.988011020
(Bunny, Demelza) . 0.987989404
(And, Now) . 0.987989085
(One, The) . 0.987973231
(But, There) . 0.987836677
(One, That) . 0.987799175
(When, All) . 0.987739822
(Sandy, Demelza) . 0.987734391
(But, As) . 0.987733581
(nodded, Mai) . 0.987722642
(For, Now) . 0.987711666
(Richard, Margaret) . 0.987683845
(Richard, Sandy) . 0.987624417
(Then, And) . 0.987609330
(Then, His) . 0.987578038
(Richard, Pitch) . 0.987549191
(It, With) . 0.987521448
(Aster, Jim) . 0.987507205
(Even, With) . 0.987479957
(For, It) . 0.987479776
(Dallas, Ross) . 0.987453815
(laughed, sighed) . 0.987427226
(Aster, Bunny) . 0.987377192
(Richard, Francis) . 0.98734084
(But, Here) . 0.987333229
(All, With) . 0.98731580
(One, This) . 0.987267291
(For, This) . 0.987236319
(Aster, Ross) . 0.987205054
(John, Anna) . 0.987190883
(For, His) . 0.987144501
(That, They) . 0.987130294
(For, As) . 0.987098321
(Ross, Two-Bit) . 0.987032831
(And, When) . 0.987003352
(Then, She) . 0.986940328
(But, She) . 0.986857973
(Aster, Demelza) . 0.986831710
(Hiccup, Demelza) . 0.986821933
(When, It) . 0.986787674
(Yes, Oh) . 0.9867814
(Richard, Aster) . 0.986772251
(She, All) . 0.986771596
(Margaret, Anna) . 0.986743199
(For, And) . 0.986722258
(After, As) . 0.986701581
(Now, That) . 0.986653929
(Ross, Gene) . 0.98659473
(As, There) . 0.986564698
(Then, Now) . 0.986558599
----------
(nodded, sighed) . 0.99601656
(smiled, nodded) . 0.991579848
(laughed, asked) . 0.991476047
(laughed, sighed) . 0.987427226
The top 20 non-capitalized:
(((WordNode "nodded" . (WordNode "sighed" ) . 0.996016567043952)
(smiled, sighed) . 0.9917868640114925)
(smiled, nodded) . 0.9915798481917005)
(laughed, asked) . 0.9914760479256299)
(laughed, sighed) . 0.9874272267287744)
(smiled, laughed) . 0.9858859081908571)
(nodded, laughed) . 0.9826039675475831)
(she, he) . 0.981876892746973)
(on, with) . 0.9816995328349034)
(with, in) . 0.9792626952550025)
(around, over) . 0.9770136201691976)
(asked, sighed) . 0.9769922567018301)
(into, from) . 0.9769095700320374)
(off, down) . 0.9753902427048208)
(on, from) . 0.9747012785336783)
(on, in) . 0.9746021337920453)
(smiled, asked) . 0.97447306776569)
(into, through) . 0.972188886981962)
(nodded, asked) . 0.9721627408838814)
(on, for) . 0.9719692162741662))
The first non-preps, non-verbs
(would, could) . 0.9718788037224313)
(an, a) . 0.9716791745530755)
(night, day) . 0.9707165794699689)
(evening, morning) . 0.9702929323741435)
(water, fire) . 0.9670737755352151)
(black, white) . 0.9666152160018894)
(head, hands) . 0.9665686866713692)
(an, some) . 0.965654580040422)
(throat, chest) . 0.9647259995951039)
(floor, table) . 0.9643904394443693)
(did, really) . 0.9643071841661717)
(bed, window) . 0.9642793931193115)
(upon, through) . 0.9640223642503675)
(shoulders, chest) . 0.9636889297135423)
(child, question) . 0.9631387627819359)
(didn’t, didn't) . 0.9628278593174421)
(mind, life) . 0.9619200398737651)
(three, two) . 0.9618460862079484)
(truth, evening) . 0.9614428353471973)
(window, table) . 0.9613977436755163)
(night, morning) . 0.9612113069050774)
(quite, almost) . 0.9609898949120979)
-----------------------------------------------------------------
Repeat the above, but for the overlap
(define (mk-ovl wa wb) (cons (cons wa wb) (poi 'right-overlap wa wb)))
(define (ovl-prs wrd wrds)
(map (lambda (wb) (mk-ovl wrd wb)) wrds))
(define (all-ovl wrds lst)
(if (null? wrds) lst
(all-ovl (cdr wrds) (append (ovl-prs (car wrds) (cdr wrds)) lst))))
(define top-cset-words
(filter (lambda (wrd) (< 1500 (cset-vec-word-observations wrd))) ac))
(define ultra-cset-words
(filter (lambda (wrd) (< 3500 (cset-vec-word-observations wrd))) ac))
(length ultra-cset-words) = 383
(define ovprs (all-ovl top-cset-words '()))
(define ovuprs (all-ovl ultra-cset-words '()))
(length ovprs) ; = 322003
(length ovuprs) ; = 73153
(define ovso (sort ovprs (lambda (a b) (> (cdr a) (cdr b)))))
(define ovuso (sort ovuprs (lambda (a b) (> (cdr a) (cdr b)))))
(take ovso 200) ;; see below
(take ovuso 200) ;; see below
The below are ovso -- the 1500 cutoff.
Some look great, some look like stinkers.
(act . foot) . 0.875
(laid . pushed) . 0.846153846
(Sandy . Bunny) . 0.842105263
(start . act) . 0.84
(soft . control) . 0.838709677
(thoughts . heavy) . 0.833333333
(Mary . Sandy) . 0.833333333
(teeth . throat) . 0.826086956
(Sandy . it's) . 0.823529411
(wide . shoulder) . 0.821428571
(shut . King) . 0.818181818
(entered . drew) . 0.809523809
(act . walk) . 0.807692307
(wide . foot) . 0.807692307
(teeth . shoulder) . 0.807692307
(town . darkness) . 0.807692307
(became . pretty) . 0.806451612
(soft . touch) . 0.806451612
(wide . blue) . 0.8
(wide . evening) . 0.8
(warm . step) . 0.8
(warm . letter) . 0.8
(ran . across) . 0.8
(Mary . wait) . 0.8
(Jack’s . Pitch’s) . 0.8
(soft . closed) . 0.794117647
(wide . further) . 0.793103448
(shut . town) . 0.791666666
(Sandy . Anna) . 0.789473684
(move . control) . 0.787878787
(deep . heavy) . 0.787878787
(except . heavy) . 0.785714285
(legs . itself) . 0.785714285
(he’s . you’re) . 0.78260869
(teeth . hot) . 0.78260869
(sister . shut) . 0.78260869
(strong . pretty) . 0.78125
(thoughts . control) . 0.78125
(glass . chest) . 0.777777777
(wide . act) . 0.777777777
(wide . slightly) . 0.777777777
(Aster . Anna) . 0.777777777
(kiss . chest) . 0.777777777
(conversation . view) . 0.777777777
(Mary . it's) . 0.777777777
(control . further) . 0.774193548
(thoughts . shoulder) . 0.774193548
(evening . shoulder) . 0.774193548
(soft . leaving) . 0.774193548
(book . window) . 0.774193548
(Prince . usual) . 0.772727272
(step . five) . 0.769230769
(wide . throat) . 0.769230769
(warm . act) . 0.769230769
(teeth . wide) . 0.769230769
(teeth . chest) . 0.769230769
(wide . control) . 0.766666666
(wide . low) . 0.766666666
(loved . touch) . 0.766666666
(expression . breath) . 0.764705882
(Aster . Hiccup) . 0.764705882
(Sandy . Aster) . 0.764705882
(pain . fear) . 0.763157894
(Bunny . Anna) . 0.761904761
(friend . family) . 0.761904761
(space . food) . 0.76
(warm . presence) . 0.76
(town . presence) . 0.76
(spirit . group) . 0.76
(sister . view) . 0.76
(everyone . straight) . 0.758620689
(four . five) . 0.758620689
(talking . completely) . 0.758620689
(wide . outside) . 0.758620689
(wide . heavy) . 0.758620689
(town . control) . 0.758620689
(heavy . red) . 0.757575757
(control . seeing) . 0.757575757
(blue . evening) . 0.757575757
(raised . pretty) . 0.757575757
(wife . heavy) . 0.757575757
(talking . wife) . 0.757575757
(soft . thoughts) . 0.757575757
(blue . high) . 0.756756756
(quiet . fear) . 0.756756756
(away . off) . 0.755555555
(true . tears) . 0.75
(top . corner) . 0.75
(strong . heavy) . 0.75
(thoughts . further) . 0.75
(usual . Tooth) . 0.75
(closed . near) . 0.75
(foot . corner) . 0.75
(blue . further) . 0.75
(George . Francis) . 0.75
(George . North) . 0.75
(act . cried) . 0.75
(act . lady) . 0.75
(show . turn) . 0.75
(wide . except) . 0.75
(wide . watch) . 0.75
(he’s . it's) . 0.75
(dropped . placed) . 0.75
(soft . further) . 0.75
(position . act) . 0.75
(position . glass) . 0.75
(town . usual) . 0.75
(wait . Sandy) . 0.75
(Tom . Aster) . 0.75
(deep . strong) . 0.75
(shut . foot) . 0.75
(shut . step) . 0.75
(late . thoughts) . 0.75
(spirit . wide) . 0.75
(sister . presence) . 0.75
(year . move) . 0.742857142
(heavy . low) . 0.741935483
(control . heavy) . 0.741935483
(leaving . control) . 0.741935483
(shoulder . control) . 0.741935483
(shoulder . low) . 0.741935483
(outside . low) . 0.741935483
(holding . bring) . 0.741935483
(wide . thoughts) . 0.741935483
(soft . wide) . 0.741935483
(quiet . except) . 0.741935483
(spirit . strong) . 0.741935483
(slightly . walk) . 0.740740740
(moving . throat) . 0.740740740
(shoulder . throat) . 0.740740740
(except . shoulders) . 0.740740740
(fight . walk) . 0.740740740
(George . giving) . 0.740740740
(outside . throat) . 0.740740740
(talking . presence) . 0.740740740
(sit . walk) . 0.740740740
(surprise . kiss) . 0.740740740
(start . surprise) . 0.740740740
(conversation . letter) . 0.740740740
(spirit . town) . 0.740740740
(spirit . slightly) . 0.740740740
(spirit . shoulders) . 0.740740740
(sister . talking) . 0.740740740
(sister . lady) . 0.740740740
(living . shut) . 0.740740740
(inside . past) . 0.74
(replied . cried) . 0.739130434
(alone . round) . 0.739130434
(shut . hot) . 0.739130434
(shut . easy) . 0.739130434
(shut . corner) . 0.739130434
(yourself . shut) . 0.739130434
(yourself . King) . 0.739130434
(ran . near) . 0.738095238
(several . cut) . 0.736842105
(Aster . Bunny) . 0.736842105
(during . control) . 0.735294117
(live . stay) . 0.735294117
(Jack’s . thoughts) . 0.735294117
(above . inside) . 0.734693877
(Francis . North) . 0.733333333
(shoulder . heavy) . 0.733333333
(thoughts . space) . 0.733333333
(force . glass) . 0.733333333
(outside . shoulder) . 0.733333333
(outside . moving) . 0.733333333
(talking . heavy) . 0.733333333
(laid . bring) . 0.733333333
(laid . touch) . 0.733333333
(wide . touch) . 0.733333333
(warm . tears) . 0.733333333
(town . thoughts) . 0.733333333
(quiet . step) . 0.733333333
(loved . makes) . 0.733333333
(living . wide) . 0.733333333
(high . fear) . 0.731707317
(pulled . met) . 0.731707317
(five . food) . 0.730769230
(figure . war) . 0.730769230
(black . white) . 0.730769230
(foot . shoulders) . 0.730769230
(act . step) . 0.730769230
(act . letter) . 0.730769230
(certainly . carried) . 0.730769230
(warm . foot) . 0.730769230
(start . foot) . 0.730769230
(start . step) . 0.730769230
(town . foot) . 0.730769230
(town . war) . 0.730769230
(shut . wide) . 0.730769230
(shut . straight) . 0.730769230
(spirit . teeth) . 0.730769230
(spirit . staff) . 0.730769230
(expression . eye) . 0.729729729
(thoughts . blood) . 0.729729729
(thoughts . arm) . 0.729729729
(evening . fear) . 0.729729729
(blue . fear) . 0.729729729
(wife . person) . 0.729729729
(wife . red) . 0.729729729729
====================================================================
How about the 3500 cutoff?
Meh.
(away . off) . 0.755555555
(black . white) . 0.730769230
(off . over) . 0.71856287
(whole . end) . 0.701754385
(herself . myself) . 0.701754385
(both . three) . 0.696629213
(quite . almost) . 0.695121951
(each . every) . 0.694736842
(night . place) . 0.693181818
(held . stood) . 0.692307692
(used . hard) . 0.692307692
(water . end) . 0.689655172
(off . through) . 0.687943262
(turned . went) . 0.686868686
(air . end) . 0.685185185
(side . behind) . 0.68493150
(called . end) . 0.684210526
(stood . started) . 0.683333333
(didn’t . didn't) . 0.682926829
(off . around) . 0.681159420
(mother . father) . 0.680851063
(away . through) . 0.676691729
(people . last) . 0.675675675
(these . both) . 0.673076923
(end . reason) . 0.672413793
(off . very) . 0.670886075
(away . around) . 0.669230769
(down . over) . 0.666666666
(whole . feeling) . 0.666666666
(woman . boy) . 0.666666666
(doing . point) . 0.666666666
(night . day) . 0.666666666
(into . over) . 0.665116279
(around . through) . 0.664122137
(away . things) . 0.663934426
(before . over) . 0.661691542
(face . around) . 0.661417322
(own . its) . 0.661290322
(end . along) . 0.661016949
(given . reason) . 0.661016949
(days . end) . 0.660714285
(held . started) . 0.660714285
(mother . course) . 0.66037735
(arms . half) . 0.66037735
(much . good) . 0.660130718
(mother . woman) . 0.66
(never . has) . 0.659574468
(come . go) . 0.659420289
(upon . through) . 0.659090909
(away . own) . 0.658914728
(off . too) . 0.658064516
(between . behind) . 0.657894736
(must . should) . 0.657657657
(side . along) . 0.656716417
(anything . nothing) . 0.656565656
(feeling . hard) . 0.65625
(held . began) . 0.65625
(days . years) . 0.655737704
(called . taken) . 0.655737704
(room . face) . 0.655462184
(hair . reason) . 0.655172413
(its . every) . 0.654867256
(against . behind) . 0.654761904
(house . place) . 0.65432098
(behind . towards) . 0.652173913
(stood . began) . 0.652173913
(than . over) . 0.652173913
(very . over) . 0.651933701
(old . young) . 0.65168539
(each . its) . 0.651376146
(these . people) . 0.651376146
(next . until) . 0.651162790
(upon . around) . 0.651162790
(rest . feeling) . 0.650793650
(both . well) . 0.650485436
(end . feeling) . 0.65
(behind . under) . 0.65
(whole . along) . 0.65
(course . end) . 0.65
(who . off) . 0.65
(great . place) . 0.65
(night . best) . 0.649350649
(point . smile) . 0.649122807
(whole . black) . 0.649122807
(whole . point) . 0.649122807
(whole . white) . 0.649122807
(others . white) . 0.649122807
(days . whole) . 0.649122807
(off . well) . 0.64885496
(around . against) . 0.648648648
(things . men) . 0.648148148
(things . nothing) . 0.648148148
(alone . whole) . 0.648148148
(half . point) . 0.648148148
(though . while) . 0.648148148
(off . down) . 0.647727272
(night . same) . 0.647727272
(away . well) . 0.647540983
(behind . along) . 0.647058823
(last . first) . 0.647058823
(went . stood) . 0.647058823
(must . might) . 0.647058823
(taken . words) . 0.647058823
(then . now) . 0.646511627
(side . door) . 0.646341463
(feeling . years) . 0.646153846
(course . years) . 0.646153846
(good . very) . 0.645569620
(along . towards) . 0.645161290
(taken . reason) . 0.645161290
(little . over) . 0.645
(must . may) . 0.644444444
(end . smile) . 0.644067796
(really . also) . 0.643678160
(both . men) . 0.643564356
(most . great) . 0.643478260
(most . men) . 0.643478260
(point . kept) . 0.642857142
(hair . black) . 0.642857142
(hair . white) . 0.642857142
(arms . close) . 0.642857142
(case . end) . 0.642857142
(get . make) . 0.642384105
(place . mind) . 0.642105263
(night . people) . 0.642105263
(until . why) . 0.641975308
(side . whole) . 0.641791044
(water . body) . 0.641791044
(large . half) . 0.641509433
(upon . off) . 0.640845070
(day . place) . 0.640776699
(used . along) . 0.640625
(got . felt) . 0.640350877
(never . can) . 0.64
(mother . point) . 0.64
(both . every) . 0.64
(always . also) . 0.639534883
(between . against) . 0.639534883
(off . being) . 0.639455782
(than . some) . 0.63934426
(held . moved) . 0.63934426
(away . upon) . 0.639097744
(another . every) . 0.638888888
(those . these) . 0.638888888
(other . over) . 0.638613861
(behind . door) . 0.638554216
(saw . heard) . 0.638554216
(away . over) . 0.638554216
(once . well) . 0.63809523
(something . even) . 0.638036809
(along . white) . 0.637931034
(called . kept) . 0.637931034
(case . course) . 0.637931034
(around . very) . 0.637583892
(small . young) . 0.6375
(would . could) . 0.637323943
(old . own) . 0.637168141
(off . two) . 0.636942675
(against . without) . 0.636363636
(black . point) . 0.636363636
(body . end) . 0.636363636
(hair . new) . 0.636363636
(days . point) . 0.636363636
(little . very) . 0.636363636
(room . around) . 0.636363636
(will . never) . 0.635802469
(off . never) . 0.635761589
(himself . off) . 0.635761589
(most . off) . 0.635714285
(people . life) . 0.635514018
(men . three) . 0.635416666
(each . both) . 0.635416666
(gave . behind) . 0.635135135
(used . wanted) . 0.635135135
(used . tried) . 0.634920634
(through . over) . 0.634730538
(two . first) . 0.634615384
(each . another) . 0.634615384
(alone . point) . 0.634615384
(large . father) . 0.634615384
(mouth . point) . 0.634615384
(both . almost) . 0.634408602
(once . ever) . 0.634408602
(taken . keep) . 0.633802816
(between . towards) . 0.633802816
(both . without) . 0.633663366
(away . most) . 0.633587786
(away . its) . 0.633587786
(against . under) . 0.633333333
(water . feet) . 0.633333333
(water . ground) . 0.633333333
(held . kept) . 0.633333333
(much . off) . 0.632911392
(other . little) . 0.632850241
(ever . well) . 0.632653061
(large . mother) . 0.632653061
(those . each) . 0.632653061
(upon . every) . 0.632478632
(whole . behind) . 0.632352941
(called . keep) . 0.632352941
----------------
How about scatter-plots?
We've got from before:
(define top-cset-words
(filter (lambda (wrd) (< 1500 (cset-vec-word-observations wrd))) ac))
(define ovprs (all-ovl top-cset-words '()))
(length ovprs) ; = 322003
so:
1) rank the top-cset-words
2) create a pixel array.
(define ranked-csw (sort top-cset-words
(lambda (a b) (> (cset-vec-word-observations a) (cset-vec-word-observations b)))))
(take ranked-csw 20) OK, that worked.
But ovprs is not stored in atomspace ....
what's faster: looking it up, or on-the-fly recompute?
(define short-csw (take ranked-csw 20))
(write-flo "/tmp/scat-overlap.flo" (take ranked-csw 100)
(lambda (wa wb) (poi 'right-overlap wa wb)))
(write-flo "/tmp/scat-cosine.flo" (take ranked-csw 100)
(lambda (wa wb) (poi 'right-cosine wa wb)))
(define (get-cos wa wb)
(define cos-fkey (PredicateNode "*-Cosine Filtered Key-*"))
(define (get-val PR)
(cog-value-ref (cog-value PR cos-fkey) 0))
(define (set-val PR VAL)
(cog-set-value! PR cos-fkey (FloatValue VAL)))
(define wpr (ListLink wa wb))
(define got
(catch #t
(lambda () (get-val wpr))
(lambda (key . args) #f)))
(if got got
(let ((val (poi 'right-cosine wa wb)))
(set-val wpr val)
val)))
(define (get-cos wa wb)
(define cos-fkey (PredicateNode "*-Cosine Filtered Key-*"))
(pair-sym-cache wa wb cos-fkey
(lambda (wx wy) (poi 'right-cosine wx wy))))
(define rankw (Word "the"))
(define cranked-csw (sort top-cset-words
(lambda (a b) (> (get-cos rankw a) (get-cos rankw b)))))
(write-flo "/tmp/scat-pcos.flo" (take cranked-csw 20)
(lambda (wa wb) (get-cos wa wb)))
; Create a sorted list of words, such that the next word in the list has
; the highest cosine of all to the previous word in the list.
(define (drank-helper wlist drl)
(define rest (cdr wlist))
(format #t "duude enter wlsi=~A\n and drl=~A\n" (length wlist) drl)
; If the list is one item long, we are done.
(if (null? rest) (cons (car wlist) drl)
; Otherwise, find the word with the highest cosine to the previous
; word that was picked. rankw is the previous one picked.
; next-rankw will be th next one.
(let ((rankw (car drl))
(next-rankw '()))
(fold
(lambda (item biggest)
(define cosi (get-cos rankw item))
(if (and (> cosi biggest) (not (equal? item rankw)))
(begin (set! next-rankw item) cosi) biggest))
-1e20
wlist)
(drank-helper
(remove (lambda (w) (equal? w next-rankw)) wlist)
(cons next-rankw drl)))))
(define (drank wlist)
(drank-helper (cdr wlist) (cons (car wlist) '())))
(define dranked-csw (drank (take ranked-csw 40)))
(define eranked-csw (drank (take ranked-csw 100)))
(define franked-csw (drank (take ranked-csw 300)))
(write-flo "/tmp/scat-dcos.flo" dranked-csw
(lambda (wa wb) (get-cos wa wb)))
(write-flo "/tmp/scat-fcos.flo" franked-csw (lambda (wa wb) (get-cos wa wb)))
(define (p4 wa wb)
(define co (get-cos wa wb))
(* co co co co))
(write-flo "/tmp/scat-ecos4.flo" eranked-csw (lambda (wa wb) (p4 wa wb)))
(write-flo "/tmp/scat-xxx.flo" (take ranked-csw 100)
(lambda (wa wb) (get-cos wa wb)))
(define n 0)
(for-each (lambda (w) (format #t "~A ~A" n w) (set! n (+ n 1))) franked-csw)
287 (WordNode "up" (ctv 1 0 626139))
288 (WordNode "back" (ctv 1 0 412057))
289 (WordNode "her" (ctv 1 0 1601673))
290 (WordNode "it" (ctv 1 0 1888598))
291 (WordNode "she" (ctv 1 0 929359))
292 (WordNode "he" (ctv 1 0 2349317))
293 (WordNode "there" (ctv 1 0 413356))
294 (WordNode "they" (ctv 1 0 677009))
295 (WordNode "if" (ctv 1 0 428894))
296 (WordNode "when" (ctv 1 0 448962))
297 (WordNode "but" (ctv 1 0 927405))
298 (WordNode "and" (ctv 1 0 5909625))
299 (WordNode "," (ctv 1 0 12471397))
126 (WordNode "No" (ctv 1 0 102421))
127 (WordNode "A" (ctv 1 0 179271))
128 (WordNode "So" (ctv 1 0 100275))
129 (WordNode "If" (ctv 1 0 138542))
130 (WordNode "We" (ctv 1 0 118551))
131 (WordNode "In" (ctv 1 0 151798))
132 (WordNode "As" (ctv 1 0 106652))
133 (WordNode "That" (ctv 1 0 106222))
134 (WordNode "This" (ctv 1 0 136409))
135 (WordNode "They" (ctv 1 0 191230))
136 (WordNode "The" (ctv 1 0 988411))
137 (WordNode "His" (ctv 1 0 144276))
138 (WordNode "She" (ctv 1 0 414833))
139 (WordNode "He" (ctv 1 0 944556))
140 (WordNode "It" (ctv 1 0 443522))
141 (WordNode "There" (ctv 1 0 153165))
142 (WordNode "At" (ctv 1 0 74095))
143 (WordNode "For" (ctv 1 0 74285))
144 (WordNode "Then" (ctv 1 0 100133))
145 (WordNode "When" (ctv 1 0 108786))
146 (WordNode "But" (ctv 1 0 327961))
147 (WordNode "And" (ctv 1 0 289634))
148 (WordNode "You" (ctv 1 0 214860))
149 (WordNode "I" (ctv 1 0 2106188))
150 (WordNode "John" (ctv 1 0 75731))
151 (WordNode "Jim" (ctv 1 0 92333))
152 (WordNode "Ross" (ctv 1 0 202513))
153 (WordNode "Pitch" (ctv 1 0 157986))
154 (WordNode "Jack" (ctv 1 0 373882))
How aobout CMI ?
(make-afunc-cache AFUNC)
(define (make-wild-summer LST)
" Perform wild-card sum get-cos(ITEM, *) with * in LST "
(define (summer item)
(fold (lambda (it acc) (+ acc (get-cos it item))) 0 LST))
(make-afunc-cache summer))
(define summer-100 (make-wild-summer (take ranked-csw 100)))
(define (wild-sum ITEM) (summer-100 ITEM))
(define (wild-wild LST)
(fold (lambda (it acc) (+ acc (wild-sum it))) 0 LST))
(define wild-wild-100 (wild-wild (take ranked-csw 100)))
; compute the cosine-MI as defined in the diary, as CMI
(define (cos-mi wa wb)
(define csim (* (get-cos wa wb) wild-wild-100))
(define sla (wild-sum wa))
(define slb (wild-sum wb))
(- 0 (/ (log (/ csim (* sla slb))) (log 2.0)))
)
(get-cos (Word "any") (Word "little")) = 0.6364163540230326
(wild-sum (Word "any") (take ranked-csw 100))
(wild-sum (Word "little") (take ranked-csw 100))
(cos-mi (Word "any") (Word "little")) = 0.17795006068910382
(write-flo "/tmp/scat-cosmi.flo" (take ranked-csw 100) cos-mi)
(for-each
(lambda (w) (format #t "Its ~A = ~A\n" w (cos-mi w (Word "he"))))
(take ranked-csw 100))
(map (lambda (w) (cons w (cos-mi w (Word "he"))))
(take ranked-csw 100))
(sort
(map (lambda (w) (cons w (cos-mi w (Word "he"))))
(take ranked-csw 100))
(lambda (a b) (> (cdr a) (cdr b))))
-----------------------------------------------------------------
---------
wtf. get-pair-count takes a LL pair, but wild-card return hi-level.
so how do wildcards works?
(define flim '(a b c))
(define (mkit ARG)
(let ((arg ARG))
(lambda () (format #t "duude its ~A and ~A\n" ARG arg))))
(define f (mkit flim))
(set! flim '(d e f))
(set! (cdr flim) '(g h ))
=========================================================================
Multiple MST bugs: OK all of these are fixed, I think. DONE
1) left-wall is not being inserted, so do not get those links.
(Word "###LEFT-WALL###")
Bug:
(observe-mst "17 Book IV., § 23.")
tokenize-text
strip-prefli "§"
string-ref: Value out of range: 0
(observe-mst "PLATE X.—_Bedroom, by De Vries “Cubiculum.”_ ]")
strip-sufli Value out of range: -1
(observe-mst "*****")
pick-best-cost-pair line 255
In procedure car: Wrong type argument in position 1 (expecting pair): ()
ABORT: wrong-type-arg
(mst-parse-text "***** fff")
=========================================================================
update typecodes set typename='ConnectorSeq' where typename='PseudoAnd';
update typecodes set typename='Section' where typename='PseudoWordCset';
update typecodes set typename='ConnectorDir' where typename='LgConnDirNode';
update typecodes set typename='Connector' where typename='PseudoConnector';
=========================================================================
(batch-all-pair-mi psa)
Start computing the basis
Support: found num left=378412 num right=7179408 in 10185 secs
Done with wild-card count N(x,*) and N(*,y) in 27072 secs
fuuuu
opencog/matrix/compute-mi.scm:251:32:
opencog/matrix/compute-mi.scm:251:32: Throw to key `bad-summation' with
args `(count-all-pairs "Error: pair-counts unequal: 31507286.0
31506142.0\n")'.
Hmm. Try again.
(use-modules (opencog) (opencog cogserver))
(use-modules (opencog persist) (opencog persist-sql))
(use-modules (opencog nlp) (opencog nlp learn))
(use-modules (opencog matrix))
(start-cogserver "opencog-mst-en.conf")
(sql-open "postgres:///en_pairs_tthree_mst?user=ubuntu&password=asdf")
(define pca (make-pseudo-cset-api))
(define psa (add-pair-stars pca))
(batch-pairs psa)
=========================================================================
now what.
oncurrent_stack<LLConnection*>::pop(LLConnection*&)
current conn_pool free=30 of 54
current conn_pool free=13 of 54
current conn_pool free=7 of 54
current conn_pool free=7 of 54
current conn_pool free=6 of 54 <<<< 22 seconds!! set (24,8)
current conn_pool free=40 of 54
current conn_pool free=40 of 54
current conn_pool free=42 of 54
current conn_pool free=40 of 54
current conn_pool free=44 of 54 <<< 23 secs!! (set 8,8)
current conn_pool free=18 of 32
current conn_pool free=9 of 32
current conn_pool free=0 of 32
current conn_pool free=2 of 32
current conn_pool free=0 of 32
current conn_pool free=0 of 32 <<< 25 secs (24, 8)
fetch-pairs doesn't seem to multi-thread oh wait it does
omp 8 8 runs with ...
222% on guile or lesss
38% x 8 on postgres = 3.04
(free = 13 of 18 12 of 18, 10 of 18 ...)
total cpu: 66:08
Elapsed time to load ANY-link pairs: 1802 secs
12 12 runs with
240%
28% x 12 = 3.36
total cpu: 73:05
Elapsed time to load ANY-link pairs: 1847 secs
(free = 14 of 18 10 of 18 8 of 18 .... 13 of 18
again: 72:23
load ANY-link pairs: 1819 secs
=========================================================================
system repl common)
repl-default-option-set! prompt "foo"
approx 12:30 PM
oww. clean start, approx 320K atom and 1.6M ob in 2 hours elapsed.
--> approx 800K obs/hour
Again: start exactly: 15:12
15:32: 297K values -> 900K obs/hour.
16:15: 941K values
16:42: 1320K values -> 1M obs/hour ... using the REPL shell
Or ... using the cogserver: start 21:43:47
at 17:04: 975K -> almost 3M obs/hour wow. Big diff...
at 18:20: 4M -> 2.5M/hour
43009543 values in 73425 secs = 586/sec = 2.10M/hour
... and its stable.
91M after 43 hours = 2.086M/hour
112780771 in 193627 = 582/sec = 2.10M/hour
-----
how about MST?
start 18:45 -- ouch 12K in 10 mins -- 72K/hour
17K in 15 mins -- 68K/hour
again: baseline 17500 at 19:08 and then it crashes. Ouch.
NonREPL: fed by four threads
103K MST updates in 20 mins = 300K/hour
245K MST after 1 hour
1.91M MST after 11 hours = 170K/hour
Non-REPL: fed by one thread:
239K MST updates in 30 mins exactly: 480K/hour
1040259 in 14667 = 71/sec = 255K/hour
1385628 in 22727 = 61/sec = 220K/hour
529655 in 5247 = 101/sec = 364K/hour
=========================================================================
Error estimates
(define lpr (Evaluation
(LinkGrammarRelationshipNode "ANY")
(List (Word "big") (Word "deal"))))
count is 1039
(define apr (Evaluation
(LinkGrammarRelationshipNode "ANY")
(ListLink (AnyNode "left-word") (AnyNode "right-word"))))
6.3884586e+08 == 638845863.0
(/ 1039 638845863.0) = 1.626370397267486e-6
ln2 = 19.2299127083157
(define freq-key (PredicateNode "*-FrequencyKey-*"))
(cog-value lpr freq-key) log = 19.229913
estimate:
(1/log2 + ln2) = 19.2299127083157 - 1.4426950408889634 =
= 17.787217667426738
sqrt(p) = 0.0012752922791530912
sqrt (p) * (1+log) = 0.022683901358884775
sqrt (N) = 25275.400352912315
abs err = 8.97469517481691e-7
rel err = abs / p = 0.5518235692124971
so its 19 +/- 0.5 error... not bad...
(use-modules (opencog))
(use-modules (opencog persist) (opencog persist-sql))
(use-modules (opencog nlp) (opencog nlp learn))
(use-modules (opencog matrix))
(sql-open "postgres:///en_pairs_mst?user=linas")
So
(define star-obj (add-pair-stars pair-obj))
(define cnt-obj (add-pair-count-api star-obj))
(define frq-obj (add-pair-freq-api star-obj))
(define wild-wild (cnt-obj 'wild-wild-count))
(define sqw (sqrt wild-wild))
(define (do-one-pair lipr)
(define pr-freq (sqrt (frqobj 'pair-freq lipr)))
(define pr-logli (- (frqobj 'pair-logli lipr) 1.4426950408889634))
(* pr-freq pr-logli))
(define (do-one-pair lipr) (frqobj 'pair-freq lipr))
(define (right-loop left-item)
(fold
(lambda (lipr sum) (+ sum (do-one-pair lipr)))
0
(star-obj 'right-stars left-item)))
(define (get-tot-error)
(define n 0)
(fold
(lambda (lefty sum)
(format #t "sum=~A for ~A of ~A\n" sum n (star-obj 'left-basis-size))
(set! n (+ n 1))
; (+ sum (/ (right-loop lefty) sqw))
(+ sum (right-loop lefty)))
0
(star-obj 'left-basis)))
This gave 0.17825659639240693 for rthree but its wrong
(define (wtf-do-one-pair lipr)
(define pr-freq (sqrt (frqobj 'pair-freq lipr)))
(define pr-logli (- (frqobj 'pair-logli lipr) 1.4426950408889634))
(format #t "Its ~A and ~A \n" lipr pr-freq)
(* pr-freq pr-logli))
(define (wtf-right-loop left-item)
(fold
(lambda (lipr sum) (+ sum (wtf-do-one-pair lipr)))
0
(star-obj 'right-stars left-item)))
WTF ... get a pr-freq of zero for
((EvaluationLink (ctv 1 0 6)
(LinkGrammarRelationshipNode "ANY")
(ListLink
(WordNode "substantively" (ctv 1 0 16))
(WordNode "events" (ctv 1 0 35225))
)
)
)
That can't be right so wtf???
(cog-value wtf freq-key) gives nothing! !???
(wtf-right-loop (WordNode "substantively" ))
(ListLink (Word "substantively") (Word "events")) ; count of 6
(ListLink (Word "substantively") (Word "important")) ; count of 13
(ListLink (Word "substantively") (Any "right-word")) count of 19
so marginal counts OK.
check them all..
(define (get-tot)
(define n 0)
(define lsz (star-obj 'left-basis-size))
(fold
(lambda (lefty sum)
(if (eq? 0 (modulo n 1000))
(format #t "sum=~A for ~A of ~A\n" sum n lsz))
(set! n (+ n 1))
; (+ sum (right-loop lefty))
; (+ sum (cnt-obj 'right-wild-count lefty))
(+ sum (frq-obj 'right-wild-freq lefty))
; (+ sum (right-cnt lefty))
; (+ sum (right-frq lefty))
) 0 (star-obj 'left-basis)))
So the right-marginal sum works out. 638845863.0
double check...
(define (right-cnt left-item)
(fold
(lambda (lipr sum) (+ sum (pair-obj 'pair-count lipr)))
0 (star-obj 'right-stars left-item)))
(define (right-frq left-item)
(fold
(lambda (lipr sum) (+ sum (frq-obj 'pair-freq lipr)))
0 (star-obj 'right-stars left-item)))
This eplicit check reports 638845863.0 so that's perfect.
All atoms got loaded. Just some of the values did not.
why?
The freq is short: 0.9777130000000336 i.e. short by
14237957.7 = 14M out of 639M
Points at failed-to-store or failed-to-load?
(ListLink (Word "substantively") (Word "events")) ; count of 6
(ListLink (Word "substantively") (Any "right-word")) count of 19
(Word "substantively") uuid= 32131120
(Word "events") uuid= 50491
(AnyNode "right-word") = uuid 33712509
select * from atoms where outgoing = '{32131120, 50491}' -> 32131316 = List
LinkGrammarRelationshipNode "ANY" is 7
(Evalu = uuid 32131317
-->
en_pairs_rthree_mall=> select * from valuations where atom=32131317;
key | atom | type | floatvalue | stringvalue | linkvalue
----------+----------+------+-----------------+-------------+-----------
1 | 32131317 | 7 | {1,0,6} | |
33712529 | 32131317 | 1 | {0,26.66593,0} | FrequencyKey
35436504 | 32131317 | 1 | {-0,-12.283042} | MI key
(3 rows)
(* 638845863.0 (expt 2 -26.66593)) so taht's correct.
So correct values are in the DB, but they were not loaded into
atomspace.
And all MST parsing results are bad.... Shit
either....values not loaded, or values not set
batch-word-pair.scm: fetch-incoming-set any-pair-pred
../../guile/PersistSCM.cc: &PersistSCM::fetch_incoming_set
as->fetch_incoming_set
_backing_store->getIncomingSet(_atom_table
SQLBackingStore.cc
AtomStorage::getInc
SQLAtomStorage::getIncomingSet buff
hi = table.add(hi, false);
get_atom_values(hi); at line 1523 hi is in the atom table.
.. is tlbuf addAtom squonky?
esponse::get_all_values_cb lines 304
is it a uuid cockup?
Add prints to get_atom_values.
is it ? print in atom->setValue
So: en_pairs_mst on fanny:
* counts sum,
* no missing values on load.
* right-wild-freqs fail to sum.
-- 0.962623000000125 reproducible
* right-freqs are missing 40%!!! ... 0.6004660000000038
* (WordNode "the") has no count on it !!! wtf???
its missing in the DB too...so that's bad.
(car (star-obj 'left-basis) (WordNode "sumatraus")
right-stars:
to 18
original 2
describe 8
the 4
'right-wild-count = 32 OK
'right-wild-freq = 0.0
(frq-obj 'pair-freq (list-ref (star-obj 'right-stars (WordNode "sumatraus")) 3))
sumatraus = 2371938357
original = 645821
List= 2371941696
ANY = 152
Eval link = 2371941697
select * from valuations where atom=2371941697;
select * from atoms where name='*-FrequencyKey-*'; = 2440463582
again, SQL is in order.
whoa ...
(cog-value (star-obj 'item-pair (list-ref (star-obj 'right-stars
(WordNode "sumatraus")) 1)) (Predicate "*-FrequencyKey-*"))
is perfect!
(frq-obj 'pair-freq (list-ref (star-obj 'right-stars (WordNode "sumatraus")) 1))
is wrong/absent...
(pair-obj 'item-pair (list-ref (star-obj 'right-stars (WordNode "sumatraus")) 1))
is correct.
Ohhh....
(cog-set-value! (Word "the") (Predicate "foobar") (FloatValue 1.0e-12))
Fuuuuuu
Fixed with this:
https://github.com/opencog/atomspace/pull/1295
-------------------------------------
en_pairs_rthree_mall restart
delete disjuncts.
delete from atoms where type = 233 ConnectorSeq
231 | Connector
232 | ConnectorDir
233 | ConnectorSeq
234 | Section
MSTConnector
PseudoConnector
MSTLinkNode 174 175 176
CREATE TEMP TABLE lws AS SELECT uuid FROM atoms WHERE type=234;
select count(*) from valuations, lws where valuations.atom=lws.uuid;
delete from valuations using lws where atom =lws.uuid;
delete from atoms where type=234;
=========================================================================
postgres:///en_pairs_ttwo_sim?user=linas
Has a pile of cosines that should be stored ...
has 802x802 cosines. Which were hard to come by.
(its ranked-csw) (length ranked-csw)
Its just ListLink of the pairs. So to store it:
(sql-open "postgres:///en_pairs_ttwo_sim?user=linas")
(for-each
(lambda (wordl)
(for-each
(lambda (wordr) (store-atom (ListLink wordl wordr)))
ranked-csw))
ranked-csw)
dammit. Kaboom. Start from scratch.
(define foo (add-support-compute poi))
(foo 'right-length (WordNode "###LEFT-WALL###"))
72 can go now 108 is next
108 can go 162 is next
162 done 243 is next
(define left-marg (make-left-summer (take ranked-csw 108) get-cos))
(define cos-wild-wild
(fold (lambda (it acc) (+ acc (left-marg it))) 0 (take ranked-csw 108)))
(define get-cmi (make-get-cmi (take ranked-csw 108)))
(define (mk-cmi-prs wa LST)
(map (lambda (wb) (get-cmi wa wb)) LST)
(define (mk-all-cmi-prs LST)
(map (lambda (wa) (mk-cmi-prs wa LST) LST)))
(define (make-all-cmi WLST RSLT)
(define wlen (length WLST))
(if (< 0 wlen)
(let* ((head (car WLST))
(rest (cdr WLST))
(prs (map (lambda (w) (get-cmi head w)) WLST))
)
(format #t "Pairs remaining: ~A\n" wlen)
(make-all-cmi rest (concatenate prs RSLT)))
RSLT))
(define all-cmi (make-sym-pairs (take ranked-csw 108) get-cmi))
(define all-cmi (make-sym-pairs (take ranked-csw 162) get-cmi))
(define ranked-cmi (sort all-cmi (lambda (a b) (> (cdr a) (cdr b)))))
. But 7.366558296223725 0.001619248635083709
. She 5.439150360752608 0.006026945058317777
. And 5.325353892901934 0.007068524421558074
. He 5.1641299882215455 0.007253773644001228
(define all-cmi (make-sym-pairs (take ranked-csw 162)
(lambda (wa wb) (* (get-cos wa wb) (get-cmi wa wb)))))
(define ranked-cmi (sort all-cmi (lambda (a b) (> (cdr a) (cdr b)))))
product:
(WordNode ";") (WordNode "said")) . 0.296607939087752)
(WordNode "said") (WordNode "upon")) . 0.29633824003351183)
(WordNode "from") (WordNode "Jack")) . 0.2937570566264612)
(WordNode ";") (WordNode "Jack")) . 0.2936624229096209)
(WordNode "with") (WordNode "said")) . 0.2923745344239661)
(WordNode "I") (WordNode ";")) . 0.2919811998332135)
(WordNode "on") (WordNode "said")) . 0.29101313047631)
(WordNode "from") (WordNode "said")) . 0.29024498064059373)
(WordNode "on") (WordNode "Jack")) . 0.28978802747511617)
(WordNode "I") (WordNode "from")) . 0.2890086903794695))
(get-cos (WordNode ";") (WordNode "said")) $92 = 0.2723589177762356
(get-cos (WordNode "said") (WordNode "upon")) $95 = 0.332603453209047
(get-cos (WordNode "from") (WordNode "Jack")) 0.3158230495038972
(get-cos (WordNode ";") (WordNode "Jack")) 0.25023554371175
(get-cos (WordNode "with") (WordNode "said")) 0.3469959252702303
(get-cos (WordNode "I") (WordNode ";")) 0.2607676762450459
(get-cos (WordNode "on") (WordNode "said")) 0.3738730234710499
(get-cos (WordNode "from") (WordNode "said"))
(get-cos (WordNode "on") (WordNode "Jack"))
(get-cos (WordNode "I") (WordNode "from"))
(get-cmi (WordNode ";") (WordNode "said")) 1.0890333296574446
(get-cmi (WordNode "said") (WordNode "upon")) 0.8909656143806125
(get-cmi (WordNode "from") (WordNode "Jack")) 0.930131784516368
(get-cmi (WordNode ";") (WordNode "Jack")) 1.1735440079923058
(get-cmi (WordNode "with") (WordNode "said")) 0.8425878032898321
(get-cmi (WordNode "I") (WordNode ";"))
(get-cmi (WordNode "on") (WordNode "said"))
=========================================================================
GC inf loop. ... 6121046016 observe-mst
=========================================================================
zh
(use-modules (opencog))
(use-modules (opencog persist) (opencog persist-sql))
(use-modules (opencog nlp) (opencog nlp learn))
(use-modules (opencog matrix))
(sql-open "postgres:///zh_pairs?user=ubuntu&password=asdf")
(use-modules (opencog cogserver))
(start-cogserver)
(define ala (make-any-link-api))
(define als (add-pair-stars ala))
(batch-pairs als)
(print-matrix-summary-report als)
Rows: 158038 Columns: 158991
Size: 5922477 non-zero entries of 25126619658 possible
Fraction non-zero: 2.3571E-4 Sparsity (-log_2): 12.051
Total observations: 728947854.0 Avg obs per pair: 123.08
Entropy Total: 18.452 Left: 10.265 Right: 10.207
Total MI: -2.020 < wrong sign
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 6819. 6411.
Count (l_1) 3.7360E+6 3.1244E+6 547.9 487.4
Length (l_2) 2.8408E+5 2.4156E+5 41.66 37.68
RMS Count 2.8231E+5 2.3990E+5 41.40 37.42
mv submitted to gamma-pages wc | 135242
guile -l mst-count-zh.scm
als are teh stars
Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS
LARGE_CONFIG
=========================================================================
Aieee
Expect our sailing in a few hours.
At six this morning, came on a tremendous gale of wind.
The wind altered sufficiently to allow us to escape from the Downs.
She turned to walk to the gate where her carriage was waiting.
The Sabbath day at Burnhead was a long, long day.
He wore a new white sailor suit with pockets.
Maie threw back her veil and smiled
It is the same old tale, I suppose?
The girl gave a little laugh.
#LEFT The -3.54169132577798 << bad
#LEFT girl 0.0477763132765396
#LEFT gave -0.709298163143728 << bad
#LEFT a 1.20476206351401 << maybe ok
#LEFT little 0.748512923494237
#LEFT laugh 0.344735358535928
The girl -3.70796523865541
The gave -0.100239485796575
The a 3.79004006078384 <<< bad
The little -1.63241818246397
The laugh 0.713357934442026
the girl -2.05346491148124
the gave 1.27996427879419
the a 2.78647534800109
the little -0.751899481316491
the laugh 1.38444380659843
girl gave -1.39996148448708
girl a 0.208481595111202
girl little 1.289110990041
girl laugh -0.183992976805794
gave a -2.96100943228456
gave little -1.67864014666756
gave laugh -3.18817482182013
a little -3.7409985984943
a laugh -2.98654866737527
little laugh -3.25530927504408
WTF. Is there a sign reversal????
little laugh N = 496.0 p = 7.764001126512734e-7 OK logli = 20.29669633683304 OK
* laugh N = 50356.0 OK p = 7.88233953077974e-5 OK logli = 13.631016579715935 OK
little * N = 658993.0 OK p = 0.00103153677305726 OK logli = 9.92098903216119 OK
* * N = 638845863.0 OK
MI = 20.29 - 13.63 - 9.92 = -3.25
(define (foo-parse-text plain-text)
(define word-strs (tokenize-text plain-text))
(define word-list (map (lambda (str) (WordNode str)) word-strs))
(define pair-obj (make-any-link-api))
(define mi-source (add-pair-freq-api pair-obj))
(define scorer (make-score-fn mi-source 'pair-fmi))
(define nsco (lambda (left-atom right-atom distance)
(define sco (scorer left-atom right-atom distance))
(if (< sco -1000) sco (- sco))))
; Process the list of words.
(mst-parse-atom-seq word-list nsco)
)
Action plan:
* add minus sign to the MI code. in the matrix directory. DONE.
* provide a conversion script for the databases NEVER
* delete the mst databases. DONE
en_pairs_rthree_mst 18 GB
en_pairs_rthree_mall 29 GB
en_pairs_rtwo_mst 22 GB
en_pairs_tthree_mst 41 GB
en_pairs_mst 7 GB
en_pairs_sim 9 GB
en_pairs_ttwo_mst 15 GB
en_pairs_ttwo_sim 17 GB
en_pairs_ultrasim 23 GB
* uppdate status of all collections DONE
e.g. en_pairs_ttwo_sim on fanny is junk.
* restart MST.
* Redo the report on disjuncts!
=========================================================================
The current Mandarin dataset has enough English in it to parse even
simple English, not well, but not terribly badly, either. Basically,
the wikipedia articles contain scattered English, and this is treated
just like everything else.
The cost is the MI of that pair. The integers are the word-ordinals.
(print-mst "this is a test")
2 . this <--> 3 . is cost= 12.596325047561628
3 . is <--> 4 . a cost= 10.684454066752686
1 . ###LEFT-WALL### <--> 3 . is cost= 0.5236859955283215
1 . ###LEFT-WALL### <--> 5 . test cost= -0.8277888080021967
(print-mst "it is surprising that this works")
5 . that <--> 7 . works cost= 12.824735708193995
5 . that <--> 6 . this cost= 11.23043475606367
3 . is <--> 5 . that cost= 8.50983391898189
2 . it <--> 3 . is cost= 12.720811706832116
1 . ###LEFT-WALL### <--> 7 . works cost= 0.7545942265555112
Some random sentences:
也 常 在 工 業 上 與 實 驗 室 中 ， 用 於 有 機 合 成 中 的 強 鹼 （ 超 強 鹼 ） 。
長 時 間 都 跟 男 人 接 觸 ， 不 擅 長 對 待 女 孩 子 。
把 蕾 當 作 戀 人 看 待 ， 記 憶 力 比 一 般 人 優 異 。
道 德 学 社 创 始 人 。
然 而 ， 此 混 战 亦 可 坚 持 三 十 六 日 。
These are segmented so that its two hanzi per pair. There is NO word
segmentation!! I am hoping that word segmentation will happen
"automatically". See below.
I do not yet have software to draw the graphs. You will have to do this
by hand, for now.
The parses:
(print-mst "也 常 在 工 業 上 與 實 驗 室 中 ， 用 於 有 機 合 成 中 的 強 鹼 （ 超 強 鹼 ） 。")
9 . 實 <--> 10 . 驗 cost= 8.67839880455297
10 . 驗 <--> 11 . 室 cost= 7.808708535957656
5 . 工 <--> 11 . 室 cost= 3.628109329166861
5 . 工 <--> 6 . 業 cost= 5.267964316861255
8 . 與 <--> 9 . 實 cost= 1.5154639602692228
11 . 室 <--> 18 . 合 cost= 1.3112275389754586
18 . 合 <--> 19 . 成 cost= 2.8487390406796376
2 . 也 <--> 19 . 成 cost= 1.3284690162639947
2 . 也 <--> 3 . 常 cost= 2.670074279865112
2 . 也 <--> 4 . 在 cost= 1.5999478270616105
19 . 成 <--> 27 . 鹼 cost= 1.0648751602654372
23 . 鹼 <--> 27 . 鹼 cost= 7.397179300260998
22 . 強 <--> 23 . 鹼 cost= 6.6278644991783935
26 . 強 <--> 27 . 鹼 cost= 6.6278644991783935
25 . 超 <--> 26 . 強 cost= 4.233373422153647
21 . 的 <--> 23 . 鹼 cost= 2.189776253921316
23 . 鹼 <--> 24 . （ cost= 2.0228678213402027
20 . 中 <--> 21 . 的 cost= 0.7882106238577222
11 . 室 <--> 13 . ， cost= 0.7204578405457767
13 . ， <--> 15 . 於 cost= 1.3787498338128241
14 . 用 <--> 15 . 於 cost= 2.3097280192674408
13 . ， <--> 16 . 有 cost= 0.8940307468617856
16 . 有 <--> 17 . 機 cost= 1.5798315464927484
12 . 中 <--> 13 . ， cost= 0.702836797299172
19 . 成 <--> 29 . 。 cost= 0.5752920624938298
28 . ） <--> 29 . 。 cost= 1.1524305408117694
6 . 業 <--> 7 . 上 cost= -0.02469597819435876
1 . ###LEFT-WALL### <--> 2 . 也 cost= -0.5802694975440783
scheme@(guile-user)> (print-mst "長 時 間 都 跟 男 人 接 觸 ， 不 擅 長 對 待 女
孩 子 。")
9 . 接 <--> 10 . 觸 cost= 8.386123586043663
9 . 接 <--> 16 . 待 cost= 5.500641386527644
15 . 對 <--> 16 . 待 cost= 4.527961561831734
4 . 間 <--> 9 . 接 cost= 3.011954338244294
3 . 時 <--> 4 . 間 cost= 5.53719025855275
2 . 長 <--> 4 . 間 cost= 1.4431523444018381
2 . 長 <--> 19 . 子 cost= 1.6192113159084691
18 . 孩 <--> 19 . 子 cost= 6.867211688834326
17 . 女 <--> 18 . 孩 cost= 7.309507215753307
13 . 擅 <--> 16 . 待 cost= 1.4396781889010661
13 . 擅 <--> 14 . 長 cost= 7.846183142809686
12 . 不 <--> 13 . 擅 cost= 4.148490791614584
11 . ， <--> 13 . 擅 cost= 2.360114646965709
4 . 間 <--> 5 . 都 cost= 0.9693867850594948
5 . 都 <--> 6 . 跟 cost= 2.3990108227984237
6 . 跟 <--> 7 . 男 cost= 1.3528584177181244
7 . 男 <--> 8 . 人 cost= 2.705451519003308
2 . 長 <--> 20 . 。 cost= 0.9405658895019222
1 . ###LEFT-WALL### <--> 2 . 長 cost= -0.11680557768830013
scheme@(guile-user)> (print-mst "把 蕾 當 作 戀 人 看 待 ， 記 憶 力 比 一 般 人 優 異 。")
11 . 記 <--> 12 . 憶 cost= 9.984319885371686
6 . 戀 <--> 12 . 憶 cost= 4.479455917846746
6 . 戀 <--> 17 . 人 cost= 3.346336621332618
6 . 戀 <--> 7 . 人 cost= 3.346336621332618
6 . 戀 <--> 19 . 異 cost= 3.1326057184954585
18 . 優 <--> 19 . 異 cost= 7.429036103043035
12 . 憶 <--> 13 . 力 cost= 2.9092434467610424
7 . 人 <--> 8 . 看 cost= 1.5817340934392554
8 . 看 <--> 9 . 待 cost= 6.40562791854984
6 . 戀 <--> 16 . 般 cost= 1.532540145438297
15 . 一 <--> 16 . 般 cost= 6.011468981031394
14 . 比 <--> 16 . 般 cost= 2.133858423252395
19 . 異 <--> 20 . 。 cost= 1.062610265554028
5 . 作 <--> 20 . 。 cost= 0.9473091730162793
4 . 當 <--> 5 . 作 cost= 1.8844566876556712
2 . 把 <--> 4 . 當 cost= 1.601964785223224
2 . 把 <--> 3 . 蕾 cost= 3.301384076256177
1 . ###LEFT-WALL### <--> 4 . 當 cost= 1.4277603296921786
8 . 看 <--> 10 . ， cost= 0.7854131431579994
scheme@(guile-user)> (print-mst "道 德 学 社 创 始 人 。")
6 . 创 <--> 7 . 始 cost= 5.70773284687867
4 . 学 <--> 6 . 创 cost= 3.061665951119931
6 . 创 <--> 8 . 人 cost= 2.1724657173546227
4 . 学 <--> 5 . 社 cost= 1.2513192358165952
8 . 人 <--> 9 . 。 cost= 0.9996485775966537
2 . 道 <--> 9 . 。 cost= 0.566680489393292
2 . 道 <--> 3 . 德 cost= 2.4802112908263467
1 . ###LEFT-WALL### <--> 2 . 道 cost= -0.4271902589965446
scheme@(guile-user)> (print-mst "然 而 ， 此 混 战 亦 可 坚 持 三 十 六 日 。")
10 . 坚 <--> 11 . 持 cost= 8.090136242433331
2 . 然 <--> 10 . 坚 cost= 3.986785521916751
2 . 然 <--> 3 . 而 cost= 5.260819401185174
1 . ###LEFT-WALL### <--> 2 . 然 cost= 1.960750908016161
2 . 然 <--> 5 . 此 cost= 1.6401055467853318
4 . ， <--> 5 . 此 cost= 1.9045829576234112
5 . 此 <--> 8 . 亦 cost= 1.6941756040836786
8 . 亦 <--> 9 . 可 cost= 3.9818559895630496
5 . 此 <--> 7 . 战 cost= 0.7150479036642032
6 . 混 <--> 7 . 战 cost= 3.4641015339868524
11 . 持 <--> 16 . 。 cost= 0.667356443539326
11 . 持 <--> 12 . 三 cost= -0.082652373159668
12 . 三 <--> 13 . 十 cost= 4.910027341075681
13 . 十 <--> 14 . 六 cost= 6.198613466629329
13 . 十 <--> 15 . 日 cost= 1.2911435674834095
scheme@(guile-user)>
FYI, here are comparable parses, for English, using the English pair dataset.
They are mostly good, with occasional mediocre links.
(print-mst "It is the same old tale, I suppose")
9 . I <--> 10 . suppose cost= 5.30352500612982
1 . ###LEFT-WALL### <--> 9 . I cost= 1.81488162256983
1 . ###LEFT-WALL### <--> 2 . It cost= 3.58403195375127
2 . It <--> 3 . is cost= 4.74100503452483
8 . , <--> 9 . I cost= 1.05294445305262
7 . tale <--> 8 . , cost= 1.04077920118952
5 . same <--> 7 . tale cost= 2.71210690237531
4 . the <--> 5 . same cost= 3.45495315601723
6 . old <--> 7 . tale cost= 2.49777533091725
scheme@(guile-user)> (print-mst "The girl gave a little laugh")
5 . a <--> 6 . little cost= 3.7409985984943
6 . little <--> 7 . laugh cost= 3.25530927504408
4 . gave <--> 7 . laugh cost= 3.18817482182013
3 . girl <--> 4 . gave cost= 1.39996148448708
2 . The <--> 3 . girl cost= 3.70796523865541
1 . ###LEFT-WALL### <--> 2 . The cost= 3.54169132577798
scheme@(guile-user)> (print-mst "He wore a new white sailor suit with pockets")
7 . sailor <--> 8 . suit cost= 5.49215876834077
5 . new <--> 8 . suit cost= 5.1696331184917
6 . white <--> 7 . sailor cost= 4.43134225309288
3 . wore <--> 8 . suit cost= 3.5690512406128
2 . He <--> 3 . wore cost= 4.4474126513332
1 . ###LEFT-WALL### <--> 2 . He cost= 3.64809278899352
8 . suit <--> 10 . pockets cost= 3.39943721825401
3 . wore <--> 4 . a cost= 3.26884788962874
8 . suit <--> 9 . with cost= 1.22919683159134
scheme@(guile-user)> (print-mst "Maie threw back her veil and smiled")
2 . Maie <--> 8 . smiled cost= 6.90160018627178
2 . Maie <--> 3 . threw cost= 6.8332928881085
3 . threw <--> 4 . back cost= 4.35035476889946
3 . threw <--> 6 . veil cost= 3.23768129913487
5 . her <--> 6 . veil cost= 3.70652214281638
1 . ###LEFT-WALL### <--> 8 . smiled cost= 1.75135762004661
7 . and <--> 8 . smiled cost= 1.67365955937873
+------->WV------->+--MVs-+---CV->+
+--Wd--+-Sp*i+--I--+Osm+ +Cs+-Sp-+--O-+
| | | | | | | | |
LEFT-WALL I.p will.v do.v it if you say.v so
+--MVs-+---CV->+
| +Cs+-Sp-+--O--+
| | | | |
? if ? say ?
+Cs+
| |
if ?
guile-mst-zh> MST-Processing file >>>止-灣/汤加国徽<<<
(define (print-mst TXT)
(define (prt-mpr MPR)
(format #t "~D . ~A <--> ~D . ~A \t MI = ~6F\n"
(car (caar MPR)) (cog-name (cdr (caar MPR)))
(car (cdar MPR)) (cog-name (cdr (cdar MPR)))
(cdr MPR)
))
(for-each prt-mpr (mst-parse-text TXT))
)
(print-mst "然杜氏 恶 撰 ， 述 一漏 百 ， 且 多 谬误 。")
(print-mst "丙寅年 夏日 ， 王门 胡氏 焚香 敬撰 。")
(print-mst "今书 已成 ， 余喜 史家 案头 ， 又 添 新书 ， 更喜 日后 家事 ， 彼无 遁词 ， 遂 成此序 。")
(print-mst "母猪 的 价格 暴涨 ， 可见 事态 之 严重 。")
(print-mst "干爷 ， 你 不是 这么 教导 我们 的 吗 ？")
Hi Ben, Ruiting,
Below are MST parses of five sentences. Just as before, an MI value
of 4 or below is usually junk, and 10 or above is high confidence.
This dataset probably does not contain MI's above 25; larger datasets
would.
These are 5 sentences I picked from wangxiaobo-7
I will have non-clustered link-grammar parses in about 3 days or so.
I expect that they will be of higher quality, but who knows.
If all goes well, I might have (non-clustered) English LG parses
later today.
--linas
The parses below are completely random junk. About half of the words
have been observed once and only once, and are thus have no meaningful
statistical sample.
(print-mst "然杜氏 恶 撰 ， 述 一漏 百 ， 且 多 谬误 。")
6 . 述 <--> 7 . 一漏 MI = 21.128
4 . 撰 <--> 6 . 述 MI = 19.009
2 . 然杜氏 <--> 4 . 撰 MI = 18.885
7 . 一漏 <--> 8 . 百 MI = 16.029
2 . 然杜氏 <--> 3 . 恶 MI = 15.179
6 . 述 <--> 12 . 谬误 MI = 13.671
7 . 一漏 <--> 10 . 且 MI = 12.479
6 . 述 <--> 11 . 多 MI = 7.7579
1 . ###LEFT-WALL### <--> 2 . 然杜氏 MI = 3.6985
4 . 撰 <--> 13 . 。 MI = 2.6162
5 . ， <--> 6 . 述 MI = 2.3806
9 . ， <--> 10 . 且 MI = 2.3653
(print-mst "丙寅年 夏日 ， 王门 胡氏 焚香 敬撰 。")
5 . 王门 <--> 6 . 胡氏 MI = 21.478
6 . 胡氏 <--> 7 . 焚香 MI = 21.471
7 . 焚香 <--> 8 . 敬撰 MI = 21.301
2 . 丙寅年 <--> 5 . 王门 MI = 18.049
2 . 丙寅年 <--> 3 . 夏日 MI = 19.518
8 . 敬撰 <--> 9 . 。 MI = 4.0183
1 . ###LEFT-WALL### <--> 2 . 丙寅年 MI = 3.6985
4 . ， <--> 5 . 王门 MI = 2.7025
(print-mst "今书 已成 ， 余喜 史家 案头 ， 又 添 新书 ， 更喜 日后 家事 ， 彼无 遁词 ， 遂 成此序 。")
17 . 彼无 <--> 18 . 遁词 MI = 21.236
18 . 遁词 <--> 21 . 成此序 MI = 19.366
15 . 家事 <--> 18 . 遁词 MI = 18.688
13 . 更喜 <--> 15 . 家事 MI = 19.674
20 . 遂 <--> 21 . 成此序 MI = 18.160
14 . 日后 <--> 15 . 家事 MI = 16.529
11 . 新书 <--> 13 . 更喜 MI = 15.155
10 . 添 <--> 15 . 家事 MI = 12.060
21 . 成此序 <--> 22 . 。 MI = 4.0183
9 . 又 <--> 10 . 添 MI = 3.9017
7 . 案头 <--> 9 . 又 MI = 3.9325
5 . 余喜 <--> 7 . 案头 MI = 18.569
5 . 余喜 <--> 6 . 史家 MI = 20.508
3 . 已成 <--> 5 . 余喜 MI = 16.793
2 . 今书 <--> 3 . 已成 MI = 17.234
1 . ###LEFT-WALL### <--> 2 . 今书 MI = 3.6985
16 . ， <--> 17 . 彼无 MI = 3.1355
18 . 遁词 <--> 19 . ， MI = 3.0673
12 . ， <--> 13 . 更喜 MI = 2.9131
4 . ， <--> 5 . 余喜 MI = 2.6880
8 . ， <--> 9 . 又 MI = 1.2527
(print-mst "母猪 的 价格 暴涨 ， 可见 事态 之 严重 。")
4 . 价格 <--> 5 . 暴涨 MI = 16.213
5 . 暴涨 <--> 7 . 可见 MI = 13.902
2 . 母猪 <--> 5 . 暴涨 MI = 12.817
7 . 可见 <--> 8 . 事态 MI = 10.562
8 . 事态 <--> 10 . 严重 MI = 8.6693
8 . 事态 <--> 9 . 之 MI = 6.1207
5 . 暴涨 <--> 6 . ， MI = 3.1828
3 . 的 <--> 4 . 价格 MI = 2.8438
7 . 可见 <--> 11 . 。 MI = 1.7341
1 . ###LEFT-WALL### <--> 7 . 可见 MI = 0.9190
(print-mst "干爷 ， 你 不是 这么 教导 我们 的 吗 ？")
2 . 干爷 <--> 7 . 教导 MI = 9.8548
7 . 教导 <--> 8 . 我们 MI = 6.5231
2 . 干爷 <--> 6 . 这么 MI = 5.1008
2 . 干爷 <--> 4 . 你 MI = 4.5330
2 . 干爷 <--> 5 . 不是 MI = 2.7742
1 . ###LEFT-WALL### <--> 2 . 干爷 MI = 2.7416
2 . 干爷 <--> 3 . ， MI = 2.0854
7 . 教导 <--> 10 . 吗 MI = 1.9647
10 . 吗 <--> 11 . ？ MI = 7.6251
8 . 我们 <--> 9 . 的 MI = 0.7513
run/mst-one.sh run/ss-one.sh
OK what borked?
(print-mst "丙寅年 夏日 ， 王门 胡氏 焚香 敬撰 。")
a 5 . 王门 <--> 6 . 胡氏 MI = 21.478
b 6 . 胡氏 <--> 7 . 焚香 MI = 21.471
c 7 . 焚香 <--> 8 . 敬撰 MI = 21.301
d 2 . 丙寅年 <--> 5 . 王门 MI = 18.049
e 2 . 丙寅年 <--> 3 . 夏日 MI = 19.518 OK its there.
f 8 . 敬撰 <--> 9 . 。 MI = 4.0183
g 1 . LEFT-WALL<--> 2 . 丙寅年 MI = 3.6985
h 4 . ， <--> 5 . 王门 MI = 2.7025
near+ far+
丙寅年.399826 (g)TXYJK- & eTPYYP+ & dTRZZG+
夏日.232814 eTPYYP-
far- near-
王门.270759 dTRZZG- & (h)TRZZH- & aTRZZI+
胡氏.349378 aTRZZI- & bTVWZI+
焚香.506188 bTVWZI- & cTPSCX+
敬撰.227735 cTPSCX- & (f)TPSCY+
-------------------
wtf this one two
(print-mst "然杜氏 恶 撰 ， 述 一漏 百 ， 且 多 谬误 。")
a 6 . 述 <--> 7 . 一漏 MI = 21.128
b 4 . 撰 <--> 6 . 述 MI = 19.009
c 2 . 然杜氏 <--> 4 . 撰 MI = 18.885
d 7 . 一漏 <--> 8 . 百 MI = 16.029
e 2 . 然杜氏 <--> 3 . 恶 MI = 15.179
f 6 . 述 <--> 12 . 谬误 MI = 13.671
g 7 . 一漏 <--> 10 . 且 MI = 12.479
h 6 . 述 <--> 11 . 多 MI = 7.7579
i 1 . ###LEFT-WALL### <--> 2 . 然杜氏 MI = 3.6985
j 4 . 撰 <--> 13 . 。 MI = 2.6162
k 5 . ， <--> 6 . 述 MI = 2.3806
l 9 . ， <--> 10 . 且 MI = 2.3653
然杜氏.274017 (i)TSEOZ- & eTSEPA+ & cTSEPB+
恶.490429 eTSEPA-
撰.425002 cTSEPB- & bTYVJC+ & (j)TVWQA+
near- far-
述.524662 (k)TVWZA- & bTYVJC- & aTWBEZ+ & hTXAHR+ & fTLYRB+
一漏.353033 aTWBEZ- & dTWBFA+ & gTIVQI+
多.375900 hTXAHR-
谬误.163229 fTLYRB-
百.519512 dTWBFA-
且.115701 (l)TBOFC- & dTIVQI-
Above should link, but its not ... why?
wrong commas: expect:
, with TVWZA+
, with TBOFC+
l->r pass removed 117441
LEFT-WALL[15044] 然杜氏[3] 恶[1] 撰[3] ，[10192] 述[5] 一漏[3] 百[2] ，[10192] 且[5] 多[136] 谬误[1] 。[10]
r->l pass removed 35544
LEFT-WALL[3] 然杜氏[3] 恶[1] 撰[3] ，[3] 述[5] 一漏[3] 百[1] ，[1] 且[2] 多[2] 谬误[1] 。[3]
l->r pass removed 0
print_expression_sizes(sent);
prepare/exprune.c: size += size_of_expression(x->exp);
LEFT-WALL[3] 然杜氏[3] 恶[1] 撰[3] ，[3] 述[5] 一漏[3] 百[1] ，[1] 且[2] 多[2] 谬误[1] 。[3]
for (d=sent->word[i].d; d != NULL; d = d->next) c++; is the dj count ..
sent->word[i].alternatives[0]
++++ Finished expression pruning 0.09 seconds
build_disjuncts_for_exp
line 171
After expanding expressions into disjuncts:LEFT-WALL(2) 然杜氏(1) 恶(1) 撰(1) ，(1) 述(1) 一漏(1) 百(1) ，(0) 且(1) 多(2) 谬误(1) 。(3)
So expression prunning removed the comma ... why? Because of cost-max
!cost-max = 50
d->left
void print_connector_list(Connector * e);
void print_disjunct_list(Disjunct * c);
wrong commas: expect:
, with TVWZA+
, with TBOFC+
Hmm first comma did have: TBOFC+ TVWZA+ but that is all....
，
------------------------------------------------------------
Also bad line-wrapping:
母猪 的 价格 暴涨 ， 可见 事态 之 严重 。
glyph_width += mk_wcwidth(ws[i]);
wchar_t wc;
mbstate_t mbss;
memset(&mbss, 0, sizeof(mbss));
mbrtowc(&wc, ssss, MB_CUR_MAX, nbss);
wc = xxx
glyph_width = mk_wcwidth(wc);
ERROR: In procedure gc:
ERROR: In procedure dbi-close: Wrong type argument in position 1: #<finalized smob 556c0402ec00>
This is still an issue....
LG expects:
test: D- & Os- (near- & far-) & (close+) & (far+)
-------------------------------------------------------------
export stuff
en_pairs_tthree_mst 8 gb
Not just en_pairs_rone.sql.bz2 and to the rest by hand just for junk.
(sql-open "postgres:///en_pairs_rone?user=linas")
(observe-mst "He wandered through the fog")
(observe-mst "He thought about home")
!!fog
Found expression for word fog: Debug: TI-
Token "fog" matches nothing in the dictionary.
###LEFT-WALL###: TB+
He: TB- & TG+
thought: TG- & TD+
about: TD- & TE+
home: TE-
LEFT-WALL -> LEFT-WALL
UNKNOWN-WORD
seg fault
Both He's have the wrong disjunct -- classname! ah OK
(define mi-source (add-pair-freq-api pair-obj))
pair-fmi
------------------------
en_pairs_rfive_mi
(sql-open "postgres:///en_pairs_rfive_mtwo?user=linas")
(define pca (make-pseudo-cset-api))
(define psa (add-pair-stars pca))
; (batch-pairs psa)
(psa 'fetch-pairs)
(print-matrix-summary-report psa)
(define psc (add-pair-count-api psa))
(define psf (add-pair-freq-api psa))
Rows: 137078 Columns: 6239997
Size: 8629163 non-zero entries of 855366308766 possible
Fraction non-zero: 1.0088E-5 Sparsity (-log_2): 16.597
Total observations: 18489594.0 Avg obs per pair: 2.1427
Entropy Total: 20.957 Left: 19.143 Right: 9.7111
Total MI: 7.8969
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 4.8875E+4 145.4
Count (l_1) 2.2958E+5 3440. 4.697 23.66
Length (l_2) 1.3729E+4 1773. .2809 12.19
RMS Count 1.3696E+4 1732. .2802 11.91
(length all-csets)
$16 = 8629163
expect approx 8629163 + 3*137078 + 2*6239997 = 21520391 = 21.5M atoms, approx.
Actual: 21651917 which is 131526 more than expected ...
and is 43GB virt = 2KB/atom
and elapsed= 4543 secs = 4.77K atoms/sec
cpu time: 435:31.67 = 26131 secs -? 5.75 parallel
heap: 1421955072
45.036
Uhh oh.
NO MI for DJ (WordNode \"Jud\") ...
(Section (WordNode "Jud")
(ConnectorSeq (Connector (WordNode "Prudie") (ConnectorDir "+"))))
(define psf (add-pair-freq-api psa))
(fetch-incoming-by-type (WordNode "Jud") 'Section)
(psf 'pair-fmi ... throws)
(PredicateNode "*-Mutual Info Key-*")
(cog-keys jud) doesn't list it. ... it was not saved.
how to recomp only one MI?
count=3
left-wild=17.7478927873962
right-wild = 15.6685349957829
wild-wild: 18489594
so freq should be: 1.622534275225297e-7
logli = 22.55524770945383
fmi = 10.861180073725272
mi = 1.7622636939013273e-6
set-pair-freq
set-pair-mi
(Section (WordNode "set")
(ConnectorSeq (Connector (WordNode "watch") (ConnectorDir "+"))))
count=4
right-wild-count 5698.0
right-wild-logli 11.6639703038511
left 15.478432112403
logli=22.140210210174985
fmi = 5.0021922060791155
mi = 2.1633790336337293e-7 * above
Whoa .. store atom didn't work!? Yes it did ....
(Section 66400472
(WordNode "set") 310
(ConnectorSeq 64865853
(Connector 64865826
(WordNode "watch") 69402
(ConnectorDir "+") 64841132
)))
So fetch-atom is not working....
get_atom_values is called...
getLink
doGetLink yes ...
setValue yes... called 1,2,3 times
but cog-keys still shows junk.
oc_to_string
(Section 73785651
(WordNode "Reinaert") 11727194
(ConnectorSeq 73785641
(Connector 73785640
(WordNode "Isengrim") 11727422
(ConnectorDir "+")))) 64841132
Ahhh! Soo .. no MI was stored for this! it was lost!
Also: not fetchable if been open a long time ... !?
count=1
wild-wild = 18489594
right-wild-logli = 18.8182821152876
left- = 18.8922826967314
so fmi = 13.570354601844016
cog-logger
(use-modules (opencog logger))
@0x7ffc6ba59e40:
getLink
doGetLink its in TLB so we are good ...
get_atom_values
@0x7ffc6ba59d90: is what sql works with.
get_all_values_cb
atom->setValue
... fails because atom in TLB has no atomspace!!!!
_atom_space->_value_table.addValuation(key, getHandle(), value);
which is why its not printing...
but fetches after individual access do work corrctly.
Soits the bulk fretch of all apirs that failed to date TLB...
(load-atoms-of-type 'Section)
getIncomingByType
getIncoming
hi = table.add(hi, false);
_tlbuf.addAtom(hi, p->uuid);
wait ... atom->setAtomSpace(_as); does happn so wtf...!?
x/4a 0x7ffc6ba59e40
(gdb) x/4a 0x7ffc6ba59e40
0x7ffc6ba59e40: 0x7f26dfca8440 0x7f26dfca8430
(gdb) x 0x7f26dfca8440
0x7f26dfca8440: 0x7f303c11e1b8 <_ZTVN7opencog4LinkE+16>
(gdb) print *this
$15 = {<> = {<> = {_M_weak_this = {<No data fields>}},
_vptr.Value = 0x7f303c11e1b8 <vtable for opencog::Link+16>,
_type = 127}, _flags = 0 '\000', _content_hash = 11383300620556476978,
_atom_space = 0x0, _truthValue = {<No data fields>},
_mtx = {<No data fields>}, _incoming_set = {<No data fields>}}
So...
atomspace should be 0x55595edc1590
(gdb) x/16 0x7f26dfca8440
0x7f26dfca8440: 0x7f303c11e1b8 <_ZTVN7opencog4LinkE+16> 0x7f26dfca8440
0x7f26dfca8450: 0x7f26dfca8430 0x353635383700007f << type=7f = 127 good
0x7f26dfca8460: 0x9df99b9a63c0a232 0x55595edc1590 << atomspace is good
0x7f26dfca8470: 0x5559659b7c10 0x5559659b7c00
0x7f26dfca8480: 0x0 0x0
0x7f26dfca8490: 0x0 0x0
whereas tlb is holding:
$26 = (struct Handle &) @0x7ffc6ba59d90: {<No data fields>}
(gdb) x 0x7ffc6ba59d90
0x7ffc6ba59d90: 0x7f26dfca8240
(gdb) x/12 0x7f26dfca8240
0x7f26dfca8240: 0x7f303c11e1b8 <_ZTVN7opencog4LinkE+16> 0x7f26dfca8240
0x7f26dfca8250: 0x7f26dfca8230 0x7f26df00007f << type=7f = 127 good
0x7f26dfca8260: 0x9df99b9a63c0a232 0x0 << hash is same, atomspace bad
0x7f26dfca8270: 0x5559659b7c10 0x5559659b7c00
0x7f26dfca8280: 0x0 0x0
0x7f26dfca8290: 0x0 0x0
_tlbuf.addAtom(hi, p->uuid);
(define sec (Section (WordNode "Reinaert") (ConnectorSeq (Connector
(WordNode "Isengrim") (ConnectorDir "+")))))
(define foo (Predicate "foo"))
(cog-value sec foo)
(cog-set-value! sec foo (FloatValue 1 ))
wtf
"Atom not in atomspace" ??? Ahh!
Atom::setValue from
8: SQLAtomStorage.cc:307
opencog::SQLAtomStorage::Response::get_all_values_cb()
10: SQLAtomStorage.cc:1687 :getLink
so doGetLink top-code returned an atom that's not in the AS.
doGetLink does no table insert directly! cause table not available.
But the tlb resolve fixes it up for us! Cause the fetch was already
in the as.
vs. the incoming set ....
getIncomingByType 1566
getIncoming 1502
... get_recursive_if_not_exists 1459
does a _tlbuf.addAtom(link, p->uuid);
but that link is not in AS yet.
add lnk to table
get values of tabled link
_tlbuf.addAtom(hi, p->uuid);
but some other ato already in tlb....
so:
h is in the as.
hr is == h
_handle_map.find(hr); returns ... !?
So the second addAtom fails to resolve or update ad the first failed to
resolve.
ValuationPtr .. s a triple, and thus is too much.
Atom has pairs of key, ValuePtr&
Remove the valuation table, do not even build it. DONE
remove special-case for tv in sql DONE
==================
United:
States+ 36
domain- domain- the- the- States+ States+ 80
the- domain- anyone- in- th- States+ 40
outside- the- States+ 51
owns- the- States+ States+ 108
status- outside- the- States+
99 2.8848641655886156 "Prince"
Prince 773
Andrew+ 626
Vasili+ 149
Andrew's+ 46
Edouard+ 46
Hans+ 33
Bagration+ 32
the- Andrew+ 27
the- Wales+ 27
Hippolyte+ 21
Arthur+ 21
the- 17
Bolkonski+ 16
Natasha- Andrew+ 18
The- Andrew+ 15
Vasili's+ 14
"- Andrew+ 11
LEFT-WALL- Edouard+ 11
112 2.801255090459977 "his" -- 49 dj > 200
140 2.6103991212010254 "think" -- 6 dj > 200
148 2.5431777546500145 "was" -- 35
161 2.4965609797013926 "head" -- 5
171 2.4537177541729895 "middle" -- 1
195 2.382483370288248 "long" -- 5
196 2.3804440649017935 "day" -- 2
208 2.3244970859184058 "going" -- 2
think: 5462
I- 1321
you- 308
do- you- 211
don't- 265
to- 244
I- it+ 202
head: 5961
his- 777
shook- .+ 751
shook- 683
her- 358
his- .+ 238
long 5412
a- 385
as+ 335
how- 236
a- time+ 212
enough+ 201
How many words?
(psf 'left-basis-size)
(psf 'right-basis-size)
(define (escquote STR BEG)
(define pos (string-index STR (lambda (C) (equal? C #\')) BEG))
(if pos
(escquote
(string-replace STR "''" pos pos 1 2)
(+ pos 2))
STR))
(string-replace "fo's" "''" 2 2 1 2)
zen_pairs_mst
(export-all-csets "dict-zen.db" "ZH_cn")
Store 800855 csets
Mandarin word pairs + MI values; Ruitings segemented text
--------------------
1) filter the dictionary write...
e.g.
(define pca (make-pseudo-cset-api)
(define psa (add-pair-stars pca))
(psa 'left-basis-size) 137078
(psa 'right-basis-size) 6239997
(length (psa 'all-pairs)) 8629163
(define fca (add-subtotal-filter psa 50 50 10 #f))
(define fsa (add-pair-stars fca))
(fsa 'left-basis-size) 14666
(fsa 'right-basis-size) 23565
(length (fsa 'all-pairs)) 76834
Need a map over all elts exists, its the looper.
get rid of all-csets DONE
(define cnt 0)
(define (cntr x) (set! cnt (+ cnt 1)))
(define looper (add-loop-api psa))
(looper 'for-each-pair cntr)
fuuuu
(define fsf (add-pair-freq-api fsa))
(psf 'pair-fmi sec) works
(fsf 'pair-fmi sec) fails
(define (foo x) (format #f "bar ~D" x))
(define* (st FOO #:optional (ID (FOO 2)))
(format #t "hey its ~A\n" ID))
(define (foo LLOBJ)
(lambda (message . args)
(case message
((id) "cset")
(else (apply LLOBJ (cons message args)))
)))
2) replace cset-vec-cosine by filtered cosine in gram-sim DONE
(define fsf (add-pair-freq-api fsa))
(define fsm (add-similarity-api fsa #f))
(define fsb (batch-similarity fsa #f))
(fsb 'batch-compute)
(fsb 'paralel-batch 3)
3) use filter-specific predicate. DONE
4) wait .. batch in matrix?
5) make graph of filtering ... why? we can already guess...
filters?
(define key-name
(if (LLOBJ 'filters?)
(string-append "*-Norm Key " (LLOBJ 'id))
"*-Norm Key-*"))
6) create generic API to fetch and store values on matrix by key name
well, maybe not...
7) create generic product matix kind-of-done-ish
8) create generic cosine matrix. kind-of-done-ish
9) fetch all atoms with key DONE
this will make fetching cosines better!?
--------------------------------------------
Fuuuu
He looked away
link-grammar/prepare/exprune.c:318
#0 expression_prune (sent=sent@entry=0x555555928f50)
at ../../link-grammar/prepare/exprune.c:318
#1 0x00007ffff7b6f2e8 in sentence_parse (sent=0x555555928f50,
opts=0x55555576f580) at ../../link-grammar/api.c:700
x->exp == 0
he.2549 has no exp....
sqlite> select * from Morphemes where subscript='he.2549';
he|he.2549|(he.2549)
he.2025041
has no disjunct.
NULL X_node
sqlite3 bad.db
.mode insert
.output dump_all.sql
.dump
.exit
sqlite3 renew.db
.read dump_all.sql
ant ant-optional
-------------------
(for-each
(lambda (item)
(format #t "wtf ~A\n" (/ (log (first item)) (log 2.0))) )
(take sorted-lensq-norm 50))
(define binned-sqlen-norm (bin-count
(take sorted-lensq-norm 50 )
50
(lambda (item)
(define val (/ (log (first item)) (log 2.0)))
(format #t "wtf ~A\n" val)
val)
(lambda (item) 1)
1.0 30.0))
(define cnt 0)
(define many (cog-count-atoms 'SimilarityLink))
(define (sto at)
(store-atom at)
(set! cnt (+ cnt 1))
(if (eqv? 0 (modulo cnt 1000))
(format #t "stored ~A of ~A\n" cnt many))
#f)
(cog-map-type sto 'SimilarityLink)
16:04
almost exactly 1/3rd have sim > 0.5 ...
total = 142270 sim links
198 dupes ... how is that possible??
select count(*) from atoms where type=27; -- 142270 so that's perfect.
where are the dupes coming from?
----------------
617
(define (print-connector CON)
(string-append (cog-name (gar CON)) (cog-name (gdr CON)) " "))
(define (section->dj-str SEC)
(string-concatenate
(map print-connector (cog-outgoing-set (gdr SEC))))
)
(define (print-disjuncts WORD)
(define all-secs (cog-incoming-by-type WORD 'Section))
(define sorted-secs (sort all-secs
(lambda (a b) (> (get-count a) (get-count b)))))
(define (prt-one con)
(format #t "~D ~A\n" (get-count con) (section->dj-str con)))
(for-each prt-one sorted-secs))
porch .. horses
fsm .... filteres similarity
(define fca (add-subtotal-filter psa 50 50 10 #f))
(define fsa (add-pair-stars fca))
(define fsm (add-similarity-api fsa #f))
(define fsb (batch-similarity fsa #f))
(fsb 'compute-similarity (Word "porch") (Word "horses"))
(fsa 'right-stars (Word "porch"))
(fsa 'right-stars (Word "horses"))
Fuu
"cset cut-50-50-10"
"*-SimKey cset cut-50-50-10"
(cog-map-type sto 'SimilarityLink)
(define cos-key (Predicate "*-Cosine Sim Key-*"))
(define cut-key (Predicate "*-SimKey cset cut-50-50-10"))
(define (mov-sim ato)
(define sim (cog-value ato cos-key))
(cog-set-value! ato cut-key sim)
(cog-set-value! ato cos-key (FloatValue 0.0))
#f)
(cog-map-type mov-sim 'SimilarityLink)
(define pss (add-similarity-api psa #f))
(define psb (batch-similarity psa #f))
(define (new-sim ato)
(define fsim (pss 'compute-similarity (gar ato) (gdr ato)))
(cog-set-value! ato cos-key (FloatValue fsim))
#f)
(cog-map-type new-sim 'SimilarityLink)
(define (kill-sim ato)
(define sim (cog-value-ref (cog-value ato cos-key) 0))
(if (< sim 1.0e-25)
(cog-set-value! ato cos-key #f))
#f)
(cog-map-type kill-sim 'SimilarityLink)
(define scnt 0)
(define (showone ato)
(if (not (null? (cog-value ato cos-key)))
(begin
(format #t "yo ~A ~A\n"
(cog-value-ref (cog-value ato cos-key) 0) ato)
(set! scnt (+ scnt 1))))
#f)
(cog-map-type showone 'SimilarityLink)
(define (cntone ato)
(define val (cog-value ato cos-key))
(if
; (not (null? val))
; (and (not (null? val)) (< 1e-10 (cog-value-ref val 0)))
(and (not (null? val)) (<= 0.1 (cog-value-ref val 0)))
(begin (set! scnt (+ scnt 1))
(store-atom ato)
)) #f)
(define scnt 0)
(cog-map-type cntone 'SimilarityLink)
(define (rpt) (set! scnt 0) (cog-map-type cntone 'SimilarityLink) scnt)
2932
3052 + (69*68)/2 - (37*36)/2 = 4732
4700 + (130 * 129) / 2 - (69*68)/2 = 10739
10708 + (427 * 426)/ 2 - (130 * 129) / 2 = 93274.0
17199 at 11:50 -> shoot for about 30K at 512 - 245 = 29890
37044 when done (- (* 0.5 427 426) (* 0.5 245 244))) = 98105
Fuuuu.
(define psu (add-support-api psa))
(psu 'right-length
(define psm (add-similarity-api psa #f))
(define pac (add-pair-cosine-compute psa))
(define (compute-sim A B)
(define val (psm 'pair-similarity (cog-link 'SimilarityLink A B)))
(if
; (and (not (null? val)) (< 1.0e-6 (cog-value-ref val 0)))
(not (null? val))
(format #t "Already got one ~A ~A ~A \n" (cog-value-ref val 0) A B)
(begin (format #t "Gonna do it for ~A=~D ~A=~D\n"
A (length (cog-incoming-by-type A 'Section))
B (length (cog-incoming-by-type B 'Section)))
(psm 'set-pair-similarity
(SimilarityLink A B)
(FloatValue (pac 'right-cosine A B))))))
; (compute-right-cosine A B)))))
(define (batch-simlist ITEM ITEM-LIST)
(for-each (lambda (item) (compute-sim ITEM item)) ITEM-LIST))
(define (make-pairs ITM-LST)
(if (not (null? ITM-LST))
(begin
(batch-simlist (car ITM-LST) (cdr ITM-LST))
(make-pairs (cdr ITM-LST)))))
(define (compute-right-cosine ROW-A ROW-B)
(define prod (compute-right-product ROW-A ROW-B))
(define deno (exact->inexact (*
(get-right-length ROW-A)
(get-right-length ROW-B))))
(if (eqv? 0.0 deno) 0.0 (/ prod deno)))
(define ptm (add-tuple-math psa * 'pair-count))
(define prd (add-support-compute ptm))
(define (compute-right-product ROW-A ROW-B)
(prd 'right-count (list ROW-A ROW-B)))
(define (get-right-length ROW) (psu 'right-length ROW))
Q: does star basis behave corectly on the tuple?
A: duddn't matter only stars are used.
(define ptms (add-pair-stars ptm))
(ptms 'right-stars (list ROW-A ROW-B))
Replace below by hash:
(define (get-right-union TUPLE)
(let* ((start-time (current-time))
(res (delete-duplicates!
(append-map!
(lambda (item) (map! gdr (psa 'right-stars item)))
TUPLE))))
(format #t "tooks ~A secs\n" (- (current-time) start-time))
res))
(define (make-union)
(define cache (make-hash-table))
(define (atom-hash ATOM SZ) (modulo (cog-handle ATOM) SZ))
(define (atom-assoc ATOM ALIST)
(find (lambda (pr) (equal? ATOM (car pr))) ALIST))
(lambda (ITEM)
(if ITEM
(hashx-set! atom-hash atom-assoc cache ITEM #f)
(let ((ats '()))
(hash-for-each-handle
(lambda (PR) (set! ats (cons (car PR) ats)))
cache)
ats))))
(define atom-set (make-union))
(for-each atom-set (map! gdr (psa 'right-stars W)))
wall -- 64125
at -- 28773
sum = 92898
wall-at: 92322
last: 94888
finally: 317206 = (797 * 796) / 2 for len=128
'item-pair return hi-pr only if lopr
'make-pair
(define good-sims
(filter
(lambda (sim) (and
(< 0.5 (sim-cosine sim))
(< 8 (cset-vec-word-len (gar sim)))
(< 8 (cset-vec-word-len (gdr sim)))))
all-sims))
Notes: Chapter- or LEFT-WALL- or End-; 27 total
Summary: Chapter- or LEFT-WALL-; 44 total
Rting: (LEFT-WALL- & :+) or (LEFT-WALL- & :+ & Audiences+) or
(LEFT-WALL- & :+ & Explicit+) or (LEFT-WALL- & :+ & Mature+)
(define (pair-cos A B)
(define cos-key (PredicateNode "*-Cosine Sim Key-*"))
(define SIM (SimilarityLink A B))
(define val (cog-value SIM cos-key))
(if (equal? A B) 1.0
(if (not (null? val)) (cog-value-ref val 0) 0.0)))
sorted-word-obs
vs. (psu 'right-length a)
cset-vec-word-observations
(psu 'right-support 64215.0
(psu 'right-count 972963.0
(psc 'right-wild-count 972963.0
(psu 'right-length 122353.945
(define psu (add-support-api psa))
dranked:
LEFT-WALL -- ... .. .... *** Two Bit - The “ And But So Perhaps Though When As Then While If Have Can Did Do Will Thank Are Where Who What Why How Or Now Here This That It There He She I You We They Some One Most Instead However Besides Well Oh Ah No Yes Yeah Good Ham 2 1 4 3 A Mai Demelza Richard John George nothing something anything everything it there who which that what where when until as because since if before while little small large good great second woman man child boy girl king house city village world town country ship sun latter land others children people men things them us me him her his my your their our the this every another leave take get find see hear speak move stop fight be make give show keep bring meet follow call ask tell help go live stay talk turn run change use read say understand remember know think suppose guess hope promise believe mean am myself fear thought knew saw felt found called happy fine certain new single word moment minute year day night evening morning way book letter party distance light fire forest room air sea earth ground floor table subject story line case question place time thing person family soul heart body own wife brother sister father mother daughter son hands face feet mouth eyes hand mind friends friend thoughts arm arms chest staff words office past water door window truth same whole best present first last least once home back down up at over on in by from upon under through into for with such having like quite only also still born gone already been seen done taken known given brought made making taking after above among between of all near against to will can cannot may must might would should could did does was is seems seemed appeared used began tried wanted needed meant came went turned continued said says spoke had has always never ever usual possible well soon far much late bad strong big young white black dark blue closed opened open hold put stand sleep answer try return come enough ready trying going supposed beginning rest sound corner direction presence name side power nature one most part kind sort number lot bit couple matter group other two three four five ten several many some out length force death life voice hair husband lips head shoulder eye bed business point state view sense feeling thinking saying now indeed therefore however sir yes God dear lady poor old dead wrong right here doing glad afraid sure sorry coming close hard long short real different clear probably certainly simply just almost not hardly easily feel have shall wish need want seem next following Queen captain wall future general London work himself herself smiled sighed nodded asked replied cried got stood sat looked look smile chance reason means longer doubt idea end edge bottom top account instead perhaps and or but though although then thus yet even really heard left lost passed met were are do love told gave took held followed behind within half an raised free better less more rather often true alone again ) too so very pretty a its these those anyone you we they he she North Jack Pitch Ross Jim Dallas Telzey Goth Sir san together along off away around round about how why – ; , : … ! . ? — All From In On By To With After For It’s It's Not Just Even At See paragraph spirit cause middle sight front spite charge form attention throat breath money law human few hundred thousand years hours minutes days times later ago ' " ” 」 ’ _ etc Mr Mrs Dr sama don ul looking especially nor either each any no finally suddenly slowly quietly Mary King Lord Lady Miss Captain I'm I’m My Her His Your ( ‘ Illustration Stats Category Rating Fandom Character Warning Is Really Please Father Mother May course being without let set sitting drawing living high heavy cold public fact particular full relief order hour individual General outside both talking able happened exactly else hell non re self S t six natural than fell deep wide deal agree don’t don't didn’t didn't received paid care drop provide York agreement ape medium author laws terms works forward forth shook enjoyed originally access permission instance chapter comment St Princess Old New Summary Notes Our THE # * Of { including Prince notes states copies copy ‐ YOU THIS OF OR Section CHAPTER Chapter + | Project eBook eBooks Gutenberg } P tax States fee electronic associated Archive Foundation United & copyright Literary donations License trademark distribution refund Van distributing Ooal Additional Posted archive Tags
"Ham" "2" "1" "4" "3" "A" "Mai" "Demelza" "Richard" "John" "George" "nothing" "something" "anything" "everything" "it" "there" "who" "which" "that" "what" "where" "when" "until" "as" "because" "since"
Ham .. 2 0.721
2 .. 1 0.881
1 .. 4 0.791
4 .. 3 0.835
3 .. A 0.375
A .. Mai 0.351
Mai .. Demelza 0.615
Demelza .. Richard 0.515
Richard .. John 0.602
John .. George 0.648
George .. nothing 0.342
nothing .. something 0.483
something .. anything 0.549
anything .. everything 0.369
everything .. it 0.496
it .. there 0.743
there .. who 0.416
who .. which 0.618
which .. that 0.580
that .. what 0.730
what .. where 0.722
where .. when 0.830
when .. until 0.886
until .. as 0.807
as .. because 0.728
because .. since 0.707
-------------------
overlap similarity again. This time, a newer way:
(define acc (add-pair-cosine-compute psa))
(define pso (add-similarity-api psa #f "Overlap"))
(define psob (batch-similarity psa #f "Overlap" 0.0
(lambda (x y) (acc 'right-overlap x y))
and then do the pslon thing....
Nahh skip it.
--------------------
Redo PCA.
First, filter all words of length 128 or more.:w
(define cos-key (PredicateNode "*-Cosine Sim Key-*"))
add-similarity-api foo #f
make-cosine-matrix
-----------------
Bitseat Tadesse
/home/linas/ftp/script-to-download-books.sh
sent splitter quotation marks mishandled.
” after a period is part of that sentence, not the next. also ’
Hmm. was working on zh_pairs_mst !!
find mst-articles |wc 12245
find gamma-pages |wc 123000
find beta-pages |wc 883737
ls -la headings |wc = 23
find alpha-data |wc 1018949
mst+gamma+beta= 1018982 - 23 = 1018959 - crap = alpha-data
zh-wp-tranche-1
todo:
-- maybe save zh_pairs_mst
-- maybe copy tranche
its NOT big-5
CN-GB sort-of works...
iconv: illegal input sequence at position 7594
CN no
GB no
GB2312 no
GB13000 ---- yes
GB18030 -- yes!
GBK -- yes
use 7z -x not unzip
iconv -f GB18030 -t UTF-8
find *.txt | iconv -f GB18030 -t UTF-8 | xargs
find tmp/*.txt | while read f; do cat $f | iconv -f GB18030 -t UTF-8 > `echo $f.utf8 | iconv -f GB18030 -t UTF-8`; rm $f ; done
lg-get-dict-entry WORD
lg-conn-type-match?
lg-conn-linkable?
lg_conn_linkable CON CON
lg_conn_type_match
LGDictReader
/* Recurse, if it's a regex match */
regex_name = match_regex(dict->regex_root, word);
if (regex_name)
dn_head = dictionary_lookup_wild(dict, word);
echo "始知 天理 有 循环 。" | ./submit-one.pl localhost 17008 observe-text
echo "站 在 旁边 的 一名 老者 问 ： “ 不 就是 一个 女子 么 ？" | ./submit-one.pl localhost 17008 observe-text
prior to start:
atoms: 6111927
valuations: 6938625
Suppress printing of "Warning: Combinatorial explosion!" DONE
except its still printing dict ver wtf.
remove the relex check Punt from ss-nosplit-one.sh
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26784 ubuntu 30 10 12.463g 1.495g 34788 S 207.3 0.6 12:24.44 guile
26784 ubuntu 30 10 13.065g 7.990g 34788 S 232.0 3.2 1483:55 guile
foo cnt=951 rate=1.1375598073147588
vs slog of 0.43 previously, after 24 hours
foo cnt=104494 rate=2.8365047910446783
Again with minimal:
(monitor-rate "foo")
foo cnt=571 rate=4.167883182665671
foo cnt=1836 rate=4.636363625198028
foo cnt=8922 rate=3.7725158547155386
foo cnt=71196 rate=4.316740435086596
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3792 ubuntu 30 10 12.392g 1.270g 35144 S 291.3 0.5 7:37.33 guile
3792 ubuntu 30 10 12.886g 4.362g 35252 S 287.3 1.7 108:03.44 guile
3792 ubuntu 30 10 13.053g 6.788g 35252 S 222.7 2.7 694:53.99 guile
cogserver: if it fails the first time, its says runing the second tim
and then stop crashes
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
27600 ubuntu 30 10 12.905g 3.073g 34772 S 283.0 1.2 57:47.82 guile
(monitor-rate "foo")
foo cnt=10805 rate=7.71785713759977
foo cnt=20403 rate=7.336569576772133
foo cnt=25451 rate=7.197680993533868
Still rate-limited by atomspace insert-delete.
Maybe provide a minimal parse! Wow! DONE
Need only: LinkGrammarRelationshipNode WordInstanceNode for pairs
update-word-counts -- just needs WordInstanceLink for parses
update-lg-link-counts -- just needs the Evals for links
; update-disjunct-counts -- word-inst-get-cset needs csets
; update-clique-pair-counts -- needs numbering. WordSequenceLink
No one needs the link instances. DONE
remove use-relex-server DONE
-----------
Bugs: the mst parse algo is O^5 and yuret gives a nice O^3 also
False. Curtis was mistaken when he said this. DONE.
--------
/dev/md127 on /data
/dev/md127 on /home2/linas/lxc-local-containers/learn-en/rootfs/var/lib/postgres
fstab!!!
cd /home2/linas/lxc-local-containers/learn-zh
cp -pr --parents rootfs/var/lib/postgresql /data/lxc-databases/learn-zh/
wtf:
fatal flex scanner internal error--end of buffer missed
en_pairs_rfive_mi
Need lock. DONE.
--------------
also -- dump the all-atoms thing!!!!
line 460 of compute-mi.scm do-one-pair
(right-loop
(WordNode "fields"
(define psf (add-pair-freq-api psa))
(define (dop lipr)
(format #t "yo ~A" lipr )
(format #t "has ~A\n" (psf 'pair-freq lipr))
)
(for-each dop rs)
(Section
(WordNode "fields")
(ConnectorSeq
(Connector (WordNode "the" ) (ConnectorDir "-"))
(Connector (WordNode "broad" ) (ConnectorDir "-"))
(Connector (WordNode "rice" ) (ConnectorDir "+"))
(Connector (WordNode "passes" ) (ConnectorDir "+"))
(Connector (WordNode "." ) (ConnectorDir "+"))
)
)
(psf 'set-pair-freq fa 1.285376022487705e-8)
change make-batch-mi
store-pairs
make-store
store-atom
SQLAtomStorage::flushStoreQueue
deleteValuation(const Handle& key, const Handle& atom)
get_uuid_column_cb
get_incoming_uuid_cb
uvec
Handle getAtom(UUID);
get_atom_values
(use-modules (opencog) (opencog persist) (opencog persist-sql))
(sql-open "postgres:///opencog_test?user=opencog_tester&password=cheese")
(cog-delete (Concept "AAA"))
(define b (Concept "BBB"))
h1 2 3 4 are nodes
hl hl2 hl3
delete hl3 then hl2 then hl
wait -- were the uuids removed? yes they were.
cog-delete-recursive is wierd
how to find handles.... handle_to_scm won't work.
-----------------
(define subtotal-obj (add-subtotal-mi-compute psa))
(subtotal-obj 'cache-all-subtotals)
(for-each cache-left-entropy (star-obj 'right-basis))
(define fail '())
(define (cle rig) (set! fail rig) (psf 'left-wild-freq rig))
(for-each cle (psa 'right-basis))
(define fail '())
(define (wtf rig)
(for-each (lambda (pr) (set! fail pr) (psf 'pair-mi pr))
(psa 'left-stars rig)))
(for-each wtf (psa 'right-basis))
(define cs
(ConnectorSeq
(Connector (WordNode "the") (ConnectorDir "-"))
(Connector (WordNode "broad") (ConnectorDir "-"))
(Connector (WordNode "rice") (ConnectorDir "+"))
(Connector (WordNode "passes") (ConnectorDir "+"))
(Connector (WordNode ".") (ConnectorDir "+"))))
fields, again!
(cog-set-tv! (ListLink (AnyNode "cset-word") cs) (ctv 1 0 1))
(define lw (ListLink (AnyNode "cset-word") cs))
(define sec (Section (WordNode "fields") cs))
(psf 'set-left-wild-freq cs (psf 'pair-freq sec))
(store-atom sec)
(store-atom cs)
(psf 'pair-mi sec)
(define fail '())
(define (cle rig) (set! fail rig) (psf 'right-wild-freq rig))
(for-each cle (psa 'left-basis))
--------------------
(fetch-all-words)
(define aw (get-all-words))
(length aw)
(define abw '())
; Look for foo[!]
(define (is-bad? wstr)
(define wlen (string-length wstr))
(if (< 2 wlen)
(let ((ope (string-ref wstr (- wlen 3)))
(clo (string-ref wstr (- wlen 1))))
(and (equal? #\[ ope) (equal? #\] clo)))
#f))
; Look for trailing blank.
(define (is-bad? wstr)
(define wlen (string-length wstr))
(equal? #\space (string-ref wstr (- wlen 1))))
(define (is-numeric? wstr)
(char-set-contains? char-set:digit (string-ref wstr 0)))
(define (is-alpha? wstr)
(and
(char-set-contains? char-set:letter (string-ref wstr 0))
(char-set-contains? char-set:letter
(string-ref wstr (- (string-length wstr) 1)))))
; Look for embedded punctuation.
; embeded OK short-dash, periiods quote. -’'.
; period colon comma if numeric. :.,·
; dot-if-numeric is OK.
; Embedded bad: two or more dots, two or more dashes
; —
; leading or trailing _
; more than one dash.
(define (is-bad? wstr)
(define wlen (string-length wstr))
(define punc (string-index wstr char-set:punctuation))
(cond
; If no punctuation, then OK.
((not punc) #f)
; If unit-length then OK
((= 1 wlen) #f)
; If numeric, and punct is dot or comma, then its OK
((and (is-numeric? wstr)
(char-set-contains? (->char-set ".,:·") (string-ref wstr punc))) #f)
; Starts with a . ends with a digit.
((and (equal? #\. (string-ref wstr 0))
(char-set-contains? char-set:digit (string-ref wstr (- wlen 1)))) #f)
; If embeded dash, single-quote is OK.
; note the second dash is actually "e2 80 90"
((and (is-alpha? wstr)
(char-set-contains? (->char-set "-‐’'") (string-ref wstr punc))) #f)
; Embedded multi-periods OK, if not consecutive.
((and (is-alpha? wstr)
(equal? #\. (string-ref wstr punc))
(< punc (- wlen 1))
(not (equal? #\. (string-ref wstr (+ punc 1))))) #f)
; else its bad
(else #t))
)
fly‐leaf
hair‐shirt
English: all embeded punct: 342245
Bad embeded punct: 231044
conservative bad punct: 228416
(length (get-all-words)) ; 851964
after: 623548
(define cnt 0)
(define start 0)
(define (get-bad wrdnod)
(define wstr (cog-name wrdnod))
; (define wlen (string-length wstr))
(if (is-bad? wstr)
(begin
(set! cnt (+ cnt 1))
; (set! abw (cons wrdnod abw))
(cog-delete-recursive wrdnod)
(if (equal? 0 (modulo cnt 1000))
(let ((end (current-time)))
(format #t "deleted ~D ~A in ~A secs\n"
cnt wstr (- end start))
(set! start end)))
)))
(define start (current-time))
(for-each get-bad (get-all-words))
----------
en_pairs_rfive_mst
Again: en_pairs_mfive_mst:
total of 522243 words == 522K words appearing in disjuncts
total of 851964 words when all are considered Uhh oh.
After (fetch-all-words) have
zero with []
zero trailing blanks
228416 which embeded punct. Same as before
after deletion, have
(define (first-bad wrdnod)
(define wstr (cog-name wrdnod))
(if (is-bad? wstr)
(begin
(format #t "~A\n" wstr) (foobar))))
_châtelaines 15:07
... delete needs what index??
CREATE INDEX incoming_idx on Atoms USING GIN(outgoing);
report -- fetch only thte stats for reporting.
print-matrix-summary-report
fetch incoming set of (LLOBJ 'wild-wild))
(fetch-incoming-set (AnyNode "cset-word"))
75GB in size
(length (cog-incoming-set (AnyNode "cset-word"))) ; 25162435 = 25M
(cog-report-counts)
((ListLink . 25162435) (Connector . 1032779)
(ConnectorSeq . 25162434) (WordNode . 522243)
(fetch-incoming-set (AnyNode "cset-disjunct"))
above is before;
afterwards have
((ListLink . 24169074) (Connector . 988068)
(ConnectorSeq . 23670834) (WordNode . 623548)
so ... a little shrinkage, not that much,..
Overall, we expect ....
24M conseq + 24M any-con-list-links + 35M sections = 83M atoms
plus 2M slop = 85M!? atoms
82GB for 59M atoms = 1.4K/atom
expect about 115GB when done... actual 113GB
81350152 atoms
((ListLink . 24169074) (SchemaNode . 1) (PredicateNode . 11) (Connector .
971810) (ConnectorDir . 2) (ConnectorSeq . 23670834) (Section . 31914870)
(WordNode . 623548) (AnyNode . 2))
Elapsed time to load csets: 22555 secs -- 6+ hours! ouch
Finished loading sparse matrix pairs
Start computing the basis
(define psa (add-pair-stars pca))
(batch-all-pair-mi psa)
Support: found num left= 444903 num right= 23444293 in 1014 secs
Done with wild-card count N(x,*) and N(*,y) in 29048 secs
Done computing N(*,*) total-count= 69447009.0 in 1101 secs
Going to do individual pair frequencies
Done computing 31914870 pairs in 925 secs
----------
zh only has 5653 bad words...
with txn, running at 7 secs-ish or sometime12 12-14 ish
deleted 1000 阿紫道[!] in 111 secs
deleted 2000 一百二十斤[!] in 97 secs
deleted 3000 可呼守用[!] in 205 secs
deleted 4000 這玉才[!] in 158 secs
deleted 5000 真有假[!] in 68 secs
deleted 6000 吹号[!] in 73 secs
deleted 7000 诺扑乐[!] in 78 secs
deleted 8000 找起[!] in 75 secs
deleted 9000 碎砖[!] in 87 secs
deleted 10000 地向床[!] in 82 secs
deleted 11000 安恬[!] in 98 secs
without txn, run at ...
deleted 1000 肢体冲突[!] in 91 secs
deleted 2000 掮客[!] in 96 secs
deleted 3000 华然[!] in 106 secs
deleted 4000 诚诚[!] in 125 secs
deleted 5000 那陷[!] in 115 secs
write queue is always a store....
(fetch-all-words)
84894
2 .... 18
reg: 377872 RSS fetch in 75 total of
max-load=3 -- 376524 piffle
zen: before
atoms 45733040 = 45M
vals: 26941144 = 27M
468821 words...
383945 bad words ... x 0.1secs/word = 10 hours. Yikes!
zen_three: 351295 words ... 266419 bad ones. Discard thios POS.
again, restart:
zen_pairs_two
zen_pairs_three
why so slow?
---
SELECT key FROM Valuations WHERE atom =
SELECT uuid FROM Atoms WHERE outgoing @> ARRAY[CAST(%lu AS BIGINT)]
SELECT * FROM Valuations WHERE key = %lu AND atom = %lu;
DELETE FROM Valuations WHERE key = %lu AND atom = %lu
\di+
!!Missing:
CREATE INDEX ON Valuations (key); but not needed either...
queue them up...
select waiting, state, query_start, query from pg_stat_activity;
state_change
wtf: SELECT * FROM Atoms WHERE type = -- from loadType wtf??? OK, idle.
CREATE INDEX incoming_idx on Atoms USING GIN(outgoing);
EXPLAIN ANALYZE
SELECT uuid FROM Atoms WHERE outgoing @> ARRAY[CAST(39859733 AS BIGINT)];
17 seconds to run query
wtf zh gets a trailing blank! dammit
1392 of them.
(use-modules (opencog exec))
(cog-execute! (LgParse (PhraseNode "派 解 嚎 嗒") (LgDict "any")))
(cog-get-atoms 'WordNode)
-------------
WTF Now what!????
(cog-report-counts)
95M value updates
1,5M in tlb
1.8M links
90M writes
was: 1554375
(cog-count-atoms 'NumberNode)
(delete-sentence SENT)
1608119
1607958
(cog-count-atoms 'SentenceNode) ;4779
(define cnt 0)
(for-each (lambda (sent)
(set! cnt (+ cnt 1))
(format #t "delete ~D\n" cnt)
(if (cog-atom? sent) (delete-sentence sent)))
(cog-get-atoms 'SentenceNode))
61377
2471125 atoms 209 sentences = approx 547844 atoms = 2621 atoms/sent
134574
359 now in zen
sentence too long, contains more than 254 words
456 now with 2
477 with 0
489
no leak with update-counts
998 with 0
no leak with delete-sentence either. WTF
1028 with 0
Bingo!
Now 1345 with 0
-------------------------------------------
Bugs fixed
Bugs: get rid of [!] in words DONE
Bugs: cleanup zen due to above DONE
what about zen_pairs_three? Ans: its junk! Delete! DONE
Bugs: clean up zh due to above DONE
Bugs: get rid of all-atoms in matrix DONE
Bugs: split sent: quote after period DONE
------------------------------------------
OK, so ... where was I?
-- computed similarity for huge number of English pairs.
I think this is in en_pairs_rfive_mtwo on fanny.
86498236 | 1 | 89 | 0 | *-SimKey cset cut-50-50-10 |
86355966 | 1 | 89 | 0 | *-Cosine Sim Key-* |
select count(*) from valuations where key=86498236; -- 142270 =
select count(*) from valuations where key=86355966; -- 1457522 = 1.4M
-- Need to re-graph cos sim for this much larger set.
(define psm (add-similarity-api psa #f)
(psm 'fetch-pairs) -- size = 1.4GB i.e. 1KB/atom fetched at
rate of 1K atoms/sec
there are 1058120 sims of length > 0.1
(define n 0)
(cog-map-type (lambda (ato) (set! n (+ n 1)) #f) 'WordNode)
have 5843 words that are in similarity-pairs.
By contrast, there are .... fuuu what is in this dataset????
-- cleaned up dashes .. did I save this DB? Yes, done - en_pairs_cfive.
kind-of should redo MI calcs, and MST parsing. See below,
not done yet.
* This dataset is too big to fit in RAM...
* clean up infix punct in rfive_mfive call it rfive_mfx
-- Need to redo the PCA.... DONE
-- If two words are similar, then ...
* issue a uniq group ID
* sum the counts
lxc-ls --active
lxc-stop -n xxx
pg_dump zen_pairs
$(id -nu)
write portno to file?
1) root to kill all lxc
2) each lxc to shutdown cleanly
send (sql-cose) to portno .. which portno?
ERROR: could not create unique index "valuations_key_atom_idx"
DETAIL: Key (key, atom)=(1, 3853322) is duplicated.
CREATE INDEX ON Valuations (key, atom);
/etc/init.d/rc.local
rc-local.service
/lib/systemd/system/rc-local.service.d
rc-local-shutdown.service
/etc/rc.local.shutdown
except /etc/rc.local.shutdown did not actually run. Crap.
Why?
=======
mhatta:
Request dtor calls
ConsoleSocket::OnRequestComplete which does nothing but print...
ConsoleSocket::put() decrements use count.
handle_connection on socket remote close does
SetCloseAndDelete
fractional line is dispatched in the thread of handle_connection
... using OnLine which just calls shell...
but befoer the fractional line is done, the socket is closed.
because ServerSocket::SetCloseAndDelete runs
fuck.
OK, remote end closes,
ServerSocket::handle_connection prints that its leaving...
calls dtor on ConsoleSocket
Generic shell is finishing up....
First one:
ServerSocket::handle_connection() yes too
[ConsoleSocket] OnConnection yes too
[ConsoleSocket] OnLine [scm hush] yes too
...
[ConsoleSocket] OnRequestComplete yes too
[GenericShell] enter eval loop yes too
[GenericShell] line disc: expr yes tooo
[GenericShell] start eval of ............? also missgin 2x
ServerSocket::exiting handle_connection() yes too
[ConsoleSocket] destructor yes too
.................... start eval of for third
ServerSocket::Send( .... miss ... miss
[GenericShell] dtor, wait for eval thread yes too
..................... Send( the second delayed!!
...................... Send third
Below only happens if self-destruct runs
[GenericShell] finishing; eval of ..... not in orig ... miss in third
[GenericShell] exit eval loop yes too
[GenericShell] dtor, joined eval thread yes too
------------------------
wait ..
concurrent_set uses std:less... Handle
a->operator<
... use atoms_less
------------------------
zen = 101M before (+ 55m val)
en = 56M + 28M
how?
1) fetch words
2) fetch incoming for one word
3) do left+right MI for that word
4) do marginals for that word
5) delete the pair.
------------------------
linear
tuple math
fold
----------------------
(define pca (make-pseudo-cset-api))
(define psa (add-dynamic-stars pca))
(define pcos (add-pair-cosine-compute psa))
(define psup (add-support-compute psa))
(define ptu (
right-cosine compute-right-cosine
/ get-right-length "short"
(length (psa 'right-stars (Word "short"))) ; 1840
(length (psa 'right-stars (Word "tall"))) ; 971
(psup 'right-length (Word "short")) ; 208.86119792819346
(psup 'right-length (Word "tall")) ; 64.00781202322104
(pcos 'right-product (Word "tall") (Word "short")) ; 800.0
(length (psa 'right-stars (Word "the"))) ; 215324
(define nou (get-dj-words (Word "the"))) ; length = 37718
(define nou (get-dj-words (Word "tall"))) ; length = 685
(length nou)
(make-gram-class psa nou 0)
(define nou (rank-by-observations nou))
(define nou (drop nou 20))
(assign-to-classes psa 0.3 nou '())
(load-atoms-of-type 'WordClassNode)
(define cls (cog-get-atoms 'WordClassNode))
(define sec (fetch-germ-sections (Word "spoke")))
(define nou (get-germ-endpoints (Word "spoke"))) ; length = 1190
(loop-over-words psa 0.3 nou cls)
start with 30 classes.
---------------------
(define gra (merge-ortho psa (Word "building") (Word "jungle") 0.0))
"island" -- "hill"
(orthogonalize (WordClassNode "island hill") (WordNode "island"))
windows tree trees shadows
(Section (ctv 1 0 15)
(WordClassNode "island hill")
(ConnectorSeq (Connector (WordNode "this") (ConnectorDir "-")))
)
(define this (ConnectorSeq (Connector (WordNode "this") (ConnectorDir "-"))))
(cog-link 'Section (Word "island") this)
island - 0.42243511
hill - 0
windows -
tree - 1.8939653
path - 0
group - 15 before ortho 9.1974925 after
affair - 5.3677896 affter
(cog-get-atoms 'WordClassNode)
(drop nou 39) - restart at 646
(drop nou 12) - restart at 634
(drop nou 34) - restart at 600
by support ot by length....
(assign-to-classes psa 0.3 nou '())
(for-each (lambda (cls)
(if (eq? 1 (length (cog-incoming-set cls)))
(cog-delete-recursive cls)))
(cog-get-atoms 'WordClassNode))
--------------
953620 atoms
total loads = 924632
total stores = 29350
valuation updates = 70232
write items=42976
_num_atom_removes
_num_atom_recursive_removes
_num_valuation_removes
76 flushes in 2000 secs ouch
SELECT key FROM Valuations WHERE atom =
CREATE INDEX incoming_idx on Atoms USING GIN(outgoing);
SELECT uuid FROM Atoms WHERE outgoing @> ARRAY[CAST(%lu AS BIGINT)]
DELETE FROM Atoms WHERE uuid = uuid is primary
664 total initially and 658 finally .. meh
is it idempotent?
664 again ..
(assign-to-classes psa 0.3 nou '())
(define (copy-sect TO FRM)
(for-each (lambda (sect)
(store-atom (Section TO (cog-outgoing-atom sect 1)))
(cog-delete sect))
(cog-incoming-by-type FRM 'Section)))
(for-each (lambda (cls)
(define mem (cog-incoming-by-type cls 'MemberLink))
(if (eq? 1 (length mem)) (begin
(format #t "hey ~A\n" mem)
(copy-sect (Word (cog-name cls)) cls)
(cog-delete-recursive cls)
)))
(cog-get-atoms 'WordClassNode))
(define iset (cog-incoming-by-type (Word "run") 'Section))
(define seq (cog-outgoing-atom (car iset) 1))
(define psup (add-support-compute psa))
(define pss (add-support-api psa))
(fetch-incoming-set seq)
(pss 'left-count seq)
delete-dup-atoms
Oof -- need to fetch sections, need to fetch Connector too
done.
(AnyNode "left-word")
=========================================================================
out-of-control GC
try: (gc-disable)
Tomas writes:
> To just add one more: total memory usage (gc and non-gc) seems to enter
> the gc triggering heuristics [1] (I couldn't follow in detail how, and
> I don't know whether this is still valid in the 2.2 branch).
>
> It just might be that Guile is seeing the outrageous process's memory
> footprint and tries desperately to free some memory (which of course
> can't suceed in your case because most memory usage isn't "guile's").
>
> [1] http://git.savannah.gnu.org/cgit/guile.git/tree/libguile/gc.c?h=v2.0.13#n770
Hans writes:
> Does it have to do with this issue [1], that the Boehm GC initially
> uses a small amount of heap. One can use say
> export GC_INITIAL_HEAP_SIZE=50G
> on the shell command line before running the program.
>
> 1. https://github.com/Macaulay2/M2/issues/500
Mark Weaver writes:
> http://www.hboehm.info/gc/gcdescr.html
and see email about this.
=========================================================================
=========================================================================
=========================================================================
=========================================================================
=========================================================================
2 June 2018
-----------
where we at? Answer: all grammatical class results are gone!???
guile -l x.scm
(sql-open "postgres:///en_pairs_cfive_class?user=ubuntu&password=asdf")
so gram-class.scm ...
(fetch-all-words) ; 82 seconds
(length (get-all-words)) ; 623548
(load-atoms-of-type 'WordClassNode)
(cog-count-atoms 'WordClassNode)
How about members?
(load-atoms-of-type 'MemberLink)
(cog-count-atoms 'MemberLink)
select * from typecodes;
22 | MemberLink
select count(*) from atoms where type=22;
So .. what .. nothing at all??? WTF?? where's the data?
CREATE DATABASE en_pairs_cfive_class WITH TEMPLATE en_pairs_cfive_mst OWNER ubuntu;
(psa 'fetch-pairs)
; 118 GB size
; 490 cpu-mins to load so about 3+ hours wall-clock
(print-matrix-summary-report psa)
31914870 pairs = 32M sections
81350150 atoms = 2.5 atoms/pair
81350150 atoms / 496 mins = 2.7k/cpu-sec load-speed
= 2.5 * 2.7K = 6.8K atoms/wall-clock-sec
118GB / 81M atoms = 1.45KBytes/atom
445K words that have sections
add-tuple-math
There is no count on the ConnectorSeq ? why? How can that be?
Oh, because its mostly meaningless. we want l1 norm, etc.
(define wtf (psa 'right-stars (Word "jump")))
(map (lambda (cs) (cog-keys (gdr cs))) wtf)
(map (lambda (cs) (fetch-atom (gdr cs))) wtf)
select count(*) from atoms; -- same as before
select count(*) from valuations; -- same as before
(print-matrix-summary-report psa) ; same as before
Rows: 444903 Columns: 23444293
Size: 31914870 non-zero entries of 10430436288579 possible
Fraction non-zero: 3.0598E-6 Sparsity (-log_2): 18.318
Total observations: 69447009.0 Avg obs per pair: 2.1760
Entropy Total: 23.092 Left: 21.137 Right: 10.112
Total MI: 8.1569
----
(WordNode "slow-cooked")
add-support-api 'right-count fails
So, some words just haven't been used in the MST oparses.
==========
Non-perpendicularity from the 0.3 thingy --
Cosine=0.6862397194384998 for word "?" -- "!"
------------------------------ Bingo!
Cosine=0.5531525850466233 for class "? !" -- "!"
which looks odd ...
Likewise:
Cosine=0.7398859541229513 for word "would" -- "will"
------------------------------ Bingo!
Cosine=0.6030429912609625 for class "would will" -- "will"
==============================
Performance of par-find code
-- with 2 threads, took 0:30 to do exactly 12 of them
-- with 1 thread, took 1986 = 0:33 to do 12...
-- alt: 1-thread: 01:19:30 - 12:56:12 = 23:18 for first 20
-- again 1 thread:
01:47:45 - 01:28:21 = 19:24 for first 20
01:54:54 - 01:47:45 = 7:09 next 19
02:01:39 - 01:54:54 = 6:45 next 18
02:08:34 - 02:01:39 = 6:55 next 17
02:14:52 - 02:08:34 = 6:18 next 16
02:20:58 - 02:14:52 = 6:06 next 15
02:26:45 - 02:20:58 = 5:47 next 14
02:31:56 - 02:26:45 = 5:11 next 13
alt: 2-threads:
03:01:35 - 02:43:20 = 18:15 first 20
03:07:57 - 03:01:35 = 6:22 next 19
03:13:39 - 03:07:57 = 5:42 next 18
03:19:26 - 03:13:39 = 5:50 next 17
03:24:45 - 03:19:26 = 5:19 next 16
03:29:51 - 03:24:45 = 5:06 next 15
03:34:48 - 03:29:51 = 4:47 next 14
03:39:06 - 03:34:48 = 4:18 next 13
alt-2thr alt-1thr
"of in" -- "-" in 612.1 secs 320.2 secs
", ;" -- "-" in 335.4 secs 237.1 secs
"was is" -- "-" in 187.7 secs 80.51 secs
"her him" -- "-" in 86.21 secs 39.32 secs
"are were" -- "-" in 56.62 secs 34.19 secs
"? !" -- "-" in 54.48 secs 19.55 secs
"they we" -- "-" in 48.27 secs 26.95 secs
"but or" -- "-" in 38.83 secs 30.16 secs
"would will" -- "-" in 28.96 secs 22.97 secs
"city village" -- "-" in 16.09 secs 5.307 secs
Clearly pretty loosing -- execution takes much longer.
but since in parallel... 1.x speedup!?
wtf...
Baseline: 328
355 elapsed for 1 thread. again: 347 -- 93% makes snese due to sched sleep.
232 elapsed for 2 threads. -- 141%
217 elapsed for 3 threads. again: 219 -- 150%
210 elapsed for 4 threads. -- 156%
(for-each cog-delete-recursive (cog-get-atoms 'WordClassNode))
(define (num-s)
(define (nmemb CLS) (length (cog-incoming-set CLS )))
(fold (lambda (CLS cnt) (+ cnt (nmemb CLS))) 0
(cog-get-atoms 'WordClassNode)))
todo- make sure the FAKE list is shortened DONE
sort the class list sort-class-list done
dropdb mrg_tst
createdb mrg_tst -T en_pairs_rfive_mtwo
If cosine is zero, can only do 8 cosines/second apparently.
lxc -- block on 5x5 db. hard code 0.65 0.3
mrg_tst -5x2 with hard-code 0.65, 0.3
WTF:
In opencog/matrix/cosine.scm:
106:24 7 (compute-right-cosine (WordClassNode "of in")
#)
In srfi/srfi-1.scm:
466:18 6 (fold #<procedure 55b602fc1060 at opencog/matrix/support.scm:1…> …)
In opencog/matrix/support.scm:
145:45 5 (_ (# #<Invalid handle>) 39072.43937140673)
In opencog/matrix/fold-api.scm:
247:32 4 (get-func-count _)
In srfi/srfi-1.scm:
592:29 3 (map1 (# #<Invalid handle>))
592:17 2 (map1 (#<Invalid handle>))
In opencog/nlp/learn/pseudo-csets.scm:
93:55 1 (get-pair-count _)
In unknown file:
0 (cog-tv #<Invalid handle>)
=================================================================
(define pca (make-pseudo-cset-api))
(define psa (add-pair-stars pca))
(batch-all-pair-mi psa)
(define wild-obj psa)
(define count-obj (make-compute-count wild-obj))
(define freq-obj (make-compute-freq wild-obj))
(define supp-obj (add-support-compute wild-obj))
(define central-obj (make-central-compute wild-obj))
(define store-obj (make-store wild-obj))
Before redoing the counts:
851964 words 25505 after triming
Existing classes=125 singletons=0 done=1545
--- To-do=24115 ncls=125 sing=0 nredo=1545 2018-06-16 19:56:59 -- "###LEFT-WALL###" ---
--- To-do=24114 ncls=125 sing=1 nredo=1545 2018-06-16 20:24:00 -- "the" ---
after redoing the counts:
After triming, 25505 words left, out of 851964
so no change
Redoing the counts:
(sql-open "postgres:///mrg_two?user=linas&password=asdf")
(define pca (make-pseudo-cset-api))
(pca 'fetch-pairs)
Elapsed time to load csets: 3500 secs ; 38GB
12901243 = 13M sections
25924126 = 26M atoms total = 1.46K/atom
Elapsed time to load csets: 3382 secs
but now its 24634125 atoms .... in mrg_test db
(define psu (add-support-compute pca))
(psu 'cache-all) ; XXX too slow, left not needed.
(psu 'right-marginals)
Finished left support subtotals in 36823 secs
Finished right support subtotals in 520 secs now 457
(define sto (make-store pca))
(sto 'store-wildcards)
(sto 'store-right-marginals)
---------Bingo! Cosine=0.5656 for class "their its" -- "the"
In opencog/nlp/learn/gram-agglo.scm:
417:20 5 (greedy-grow #<procedure 55a4f54a4400 at
opencog/nlp/learn/gr…> …)
In opencog/nlp/learn/gram-class.scm:
814:27 4 (is-small? _)
In srfi/srfi-1.scm:
466:18 3 (fold #<procedure 55a537014fa0 at
opencog/matrix/support.scm:…> …)
In opencog/matrix/support.scm:
166:45 2 (_ #<Invalid handle> 20.850880324934934)
In opencog/nlp/learn/pseudo-csets.scm:
102:55 1 (get-count _)
In unknown file:
0 (cog-tv #<Invalid handle>)
=========================================================================
(define pca (make-pseudo-cset-api))
(define psa (add-dynamic-stars pca))
(define pco (add-pair-cosine-compute psa))
(define pcf (add-pair-freq-api psa))
(define pch (add-pair-cosine-compute pcf 'pair-entropy))
(define pcm (add-pair-cosine-compute pcf 'pair-mi))
(pco 'right-cosine (Word "peace") (Word "you"))
Cosine=0.0282 for word "you" -- "peace" in 2.947 secs
(pch 'right-cosine = 0.03289211797144457
pcm = 0.003475665530244811
(pco 'right-cosine (Word "peace") (Word "love"))
0.1860942684698426
(pch 'right-cosine (Word "peace") (Word "love"))
0.18839731526266035
(pcm 'right-cosine (Word "peace") (Word "love"))
0.03816055080854275
(pco 'right-cosine (Word "taken") (Word "locked"))
0.10177549551488362
(pch 'right-cosine (Word "taken") (Word "locked"))
0.09444483468390132
(pcm 'right-cosine (Word "taken") (Word "locked"))
0.015026716721988248
(define (sim swa swb)
(define wa (Word swa))
(define wb (Word swb))
(format #t "~A -- ~A count=~6F entrop=~6F mi=~6F\n"
swa swb
(pco 'right-cosine wa wb)
(pch 'right-cosine wa wb)
(pcm 'right-cosine wa wb)))
other same
fine nice
him me
men women
up down
found called
came went
eyes hand
cosines computed from various number-sources:
other -- same count=0.5866 entrop=0.5433 mi=0.3159
nice -- fine count=0.5256 entrop=0.4826 mi=0.2364
him -- me count=0.7842 entrop=0.7796 mi=0.6448
men -- women count=0.6077 entrop=0.5841 mi=0.2667
up -- down count=0.5630 entrop=0.5528 mi=0.4108
found -- called count=0.5576 entrop=0.5298 mi=0.3460
came -- went count=0.5900 entrop=0.5656 mi=0.3580
eyes -- hand count=0.7284 entrop=0.6669 mi=0.4827
Actual symmetric MI's: (using code 200 lines below)
other -- same *,*=-12.39 L=-21.52 R=-22.13 MI=4.1232
nice -- fine *,*=-12.39 L=-26.94 R=-25.21 MI=4.5210
him -- me *,*=-12.39 L=-20.81 R=-21.34 LR=-25.50 MI=4.2583
men -- women *,*=-12.39 L=-23.69 R=-25.79 LR=-33.24 MI=3.8489
up -- down *,*=-12.39 L=-22.46 R=-23.31 LR=-27.87 MI=5.5085
found -- called *,*=-12.39 L=-23.62 R=-24.57 LR=-32.09 MI=3.7066
came -- went *,*=-12.39 L=-24.19 R=-24.72 LR=-31.32 MI=5.1930
eyes -- hand *,*=-12.39 L=-23.47 R=-23.19 LR=-28.91 MI=5.3589
men -- nice *,*=-12.39 L=-23.69 R=-26.94 LR=-38.71 MI=-0.477
men -- went *,*=-12.39 L=-23.69 R=-24.72 LR=-36.08 MI=-0.061
nice -- went *,*=-12.39 L=-26.94 R=-24.72 LR=-38.22 MI=1.0490
called -- eyes *,*=-12.39 L=-24.57 R=-23.47 LR=-38.14 MI=-2.499
nice -- eyes *,*=-12.39 L=-26.94 R=-23.47 LR=-40.21 MI=-2.193
nice -- called *,*=-12.39 L=-26.94 R=-24.57 LR=-35.63 MI=3.4810
nice -- called count=0.3794 entrop=0.3334 mi=0.1955
nice -- eyes count=0.0038 entrop=0.0050 mi=-0.000
called -- eyes count=0.0049 entrop=0.0069 mi=0.0007
nice -- went count=0.0353 entrop=0.0329 mi=0.0044
men -- went count=0.0249 entrop=0.0281 mi=0.0021
men -- nice count=0.0233 entrop=0.0262 mi=0.0001
=========================================================================
(psa 'fetch-pairs)
(define cva (make-connector-vec-api))
(define cvs (add-pair-stars cva))
(cvs 'left-basis-size)
(cva 'make-left-stars)
Elapsed time to compute left-stars: 1170 secs
Elapsed time to compute left-stars: 999 secs
(cvs 'right-basis-size)
18265271 == 18M
Argh, but there were only 8629163 Sections so most of those are
singletons! Ouch! What to do ???
(batch-all-pair-mi cva)
(define (how-many)
(fold (lambda (a s) (if (cog-ctv? (cog-tv a)) (+ 1 s) s)) 0 (cvs
'right-basis)))
(define (rate)
(define start-time (current-time))
(define start-count (how-many))
(define sleep-time 100)
(sleep sleep-time)
(let* ((count (how-many))
(done (- count start-count))
(elapse (- (current-time) start-time))
(adjt (+ sleep-time (* 0.5 (- elapse sleep-time))))
(rat (/ done adjt)))
(format #t "Did ~A in ~A secs done=~A rate=~A\n"
done adjt count rat)))
Ouch. It is doing 92/second -> 18M will take
200K secs = 55 hours. Wow. That sucks.
And its slowing down....
done=194977 rate=69.9
done=359402 rate=64.6
done=1527137 rate=47.39
done=2008355 rate=53.03
done=5003278 rate=32.55
8*3600 did 3M == 100 or so ... Hm.
done=6773636 rate=74.01
done=8362448 rate=29.16
done=10414938 rate=33.30
done=11074628 rate=67.14
done=11703321 rate=0.0
done=11811189 rate=69.24
done=13161404 rate=21.38
=========================================================================
(print-matrix-summary-report psa)
Summary Report for Correlation Matrix Word-Disjunct Pairs (Connector
Sets)
Left type: WordNode Right Type: ConnectorSeq Pair Type: Section
Wildcard: (ListLink (ctv 0 0 18489594)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 137078 Columns: 6239997
Size: 8629163 non-zero entries of 855366308766 possible
Fraction non-zero: 1.0088E-5 Sparsity (-log_2): 16.597
Total observations: 18489594.0 Avg obs per pair: 2.1427
Entropy Total: 20.957 Left: 19.143 Right: 9.7111
Total MI: 7.8969
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 4.8875E+4 145.4
Count (l_1) 2.2958E+5 3440. 4.697 23.66
Length (l_2) 1.3729E+4 1773. .2809 12.19
RMS Count 1.3696E+4 1732. .2802 11.91
(psa 'fetch-pairs)
(define cva (make-connector-vec-api))
(cva 'fetch-pairs)
(define cvs (add-pair-stars cva))
(cvs 'left-basis-size)
(cva 'make-left-stars)
Elapsed time to compute left-stars: 1170 secs
Elapsed time to compute left-stars: 999 secs
(cvs 'right-basis-size)
(define predno (Predicate "*-connector left stars-*"))
(length (cog-incoming-set predno)) == 18265271 == 18M
Argh, but there were only 8629163 Sections so most of those are
singletons!
Ouch! What to do ???
(define wild-obj (add-pair-stars cva))
(define count-obj (make-compute-count wild-obj))
(count-obj 'cache-all-right-counts)
(batch-all-pair-mi cva)
Ouch. It is doing 92/second -> 18M will take
200K secs = 55 hours. Wow. That sucks.
(define (how-many)
(fold (lambda (a s) (if (cog-ctv? (cog-tv a)) (+ 1 s) s)) 0 (cvs
'right-basis)))
(define (rate)
(define start-time (current-time))
(define start-count (how-many))
(define sleep-time 100)
(sleep sleep-time)
(let* ((count (how-many))
(done (- count start-count))
(elapse (- (current-time) start-time))
(adjt (+ sleep-time (* 0.5 (- elapse sleep-time))))
(rat (/ done adjt)))
(format #t "Did ~A in ~A secs done=~A rate=~A\n"
done adjt count rat)))
(define (bsize)
(fold (lambda (wld sum) (+ sum 1)) 0 (pcn 'right-basis)))
(define (bcnt)
(fold (lambda (wld sum) (+ sum (pcn 'left-wild-count wld))) 0
(pcn 'right-basis)))
(define (bsqu)
(fold (lambda (wld sum)
(let ((wcnt (pcn 'left-wild-count wld))) (+ sum (* wcnt wcnt)))) 0
(pcn 'right-basis)))
(define (fsqu)
(fold (lambda (wld sum)
(let ((wcnt (pfa 'left-wild-freq wld))) (+ sum (* wcnt wcnt)))) 0
(pfa 'right-basis))) --- 1.8603363179581776e-4
(define (fcnt)
(fold (lambda (wld sum) (+ sum (pfa 'left-wild-freq wld))) 0
(pfa 'right-basis)))
(define pfa (add-pair-freq-api psa #:nothrow #t))
So:
N(*,y) == 'left-count
mmt-count is the wild-card sum_y N(x,y) N(*,y)
(define (wilds WRD)
(fold (lambda (wld sum)
(let ((mrq (pfa 'left-wild-freq wld))
(fre (pfa 'pair-freq (pfa 'get-pair WRD wld))))
(+ sum (* mrq fre)))) 0
(pfa 'right-basis)))
(wilds (WordNode "men"))
7.397675578642773e-8
(define (wilde WRD)
(fold (lambda (str sum)
(let ((mrq (pfa 'left-wild-freq (pfa 'right-element str)))
(fre (pfa 'pair-freq str)))
(+ sum (* mrq fre)))) 0
(pfa 'right-stars WRD)))
; entro is same as cosine-compute 'right-product
(define (entro WA WB)
(fold (lambda (wld sum)
(let ((mrq (pfa 'pair-freq (pfa 'get-pair WA wld)))
(fre (pfa 'pair-freq (pfa 'get-pair WB wld))))
(+ sum (* mrq fre)))) 0
(pfa 'right-basis)))
(define (log2 x) (/ (log x) (log 2)))
(define (ensim swa swb)
(define wa (Word swa))
(define wb (Word swb))
(define sqsu 1.8603363179581776e-4)
(define pa (wilds wa))
(define pb (wilds wb))
(define pab (entro wa wb))
(define ent (log2 (/ (* sqsu pab) (* pa pb))))
(format #t "~A -- ~A *,*=~6F L=~6F R=~6F LR=~6F MI=~6F\n"
swa swb
(log2 sqsu) (log2 pa) (log2 pb) (log2 pab) ent))
(define tcc (add-transpose-compute pfa))
(define tcf (add-transpose-compute pfa #:GET-COUNT 'pair-freq #:LEFT-COUNT 'left-wild-freq))
(tcf 'mmt-count (Word "eyes")) ; 8.6245e-8
(tcf 'total-mmt-count) ;
----------------------
OK, now that trans marginals are available, verify that they give same
answers:
So:
N(*,y) == 'left-count
mmt-count is the wild-card sum_y N(x,y) N(*,y)
x-mtwo.scm
(define pca (make-pseudo-cset-api))
(define psa (add-pair-stars pca))
(define pta (add-transpose-api psa))
(pta 'total-mmt-count) ; 63598403588.0 == 63.6 Gcounts
(pta 'mmt-count (Word "other")) ; 113433851.0
(pta 'mmt-count (Word "same")) ; 74309919.0
(define pco (add-pair-cosine-compute psa))
(pco 'right-product (Word "other") (Word "same")) ; 2309662.0
(pco 'right-cosine (Word "other") (Word "same")) ; 0.58660119
(define (log2 x) (/ (log x) (log 2)))
(define (symmi wa wb)
(define sqsu (pta 'total-mmt-count))
(define pa (pta 'mmt-count wa))
(define pb (pta 'mmt-count wb))
(define pab (pco 'right-product wa wb))
(define ent (log2 (/ (* sqsu pab) (* pa pb))))
(format #t "~A -- ~A *,*=~6F L=~6F R=~6F LR=~6F MI=~6F\n"
(cog-name wa) (cog-name wb)
(log2 sqsu) (log2 pa) (log2 pb) (log2 pab) ent))
other -- same *,*=35.888 L=26.757 R=26.147 LR=21.139 MI=4.1232
Great! same MI as before
(define (symmi wa wb)
(define tot (pta 'total-mmt-count))
(define pa (/ (pta 'mmt-count wa) tot))
(define pb (/ (pta 'mmt-count wb) tot))
(define pab (/ (pco 'right-product wa wb) tot))
(define ent (log2 (/ pab (* pa pb))))
(format #t "~A -- ~A L=~6F R=~6F LR=~6F MI=~6F\n"
(cog-name wa) (cog-name wb) (log2 pa) (log2 pb) (log2 pab) ent))
other -- same L=-9.131 R=-9.741 LR=-14.75 MI=4.1232
nice -- fine L=-14.55 R=-12.81 LR=-22.84 MI=4.5210
him -- me L=-8.418 R=-8.945 LR=-13.11 MI=4.2583
(define pmi (add-symmetric-mi-compute psa))
(pmi 'mmt-fmi (Word "other") (Word "same"))
OK so we are good to go.
================
A summary report for transpose?
mmt-count is the wild-card sum_y N(x,y) N(*,y)
divide by total to get Q(x) = sum_y P(x,y) P(*,y)
1) support == left=dim, since Q(x) is non-zero for all x.
2) sum_x Q(x) = 1 of course
3) length == sqrt sum_x Q(x)Q(x) but why is this interesting?
(define mmt-tot (pta 'total-mmt-count))
(define (que wrd) (/ (pta 'mmt-count wrd) mmt-tot))
(define sumsq (fold
(lambda (wrd sum) (define q (que wrd))
(+ sum (* q q))) 0
(pta 'left-basis)))
for mtwo, this gives: 0.09987939734814384
(sqrt sumsq) ; 0.3160370189521219
================
Does similarity work?
(define pts (add-similarity-api pta #f "pseudo-cset MI"))
Uh no ..
(define bmi
(batch-similarity pta #f "pseudo-cset MI"
0.0 (lambda (wa wb) (pmi 'mmt-fmi wa wb))))
(bmi 'compute-similarity (Word "other") (Word "same"))
(bmi 'batch-compute 12)
================
en_pairs_mtwo=> select count(*) from Atoms;
count
----------
79586531
(1 row)
en_pairs_mtwo=> select count(*) from valuations;
count
-----------
108255875
(1 row)
CREATE INDEX incoming_idx on Atoms USING GIN(outgoing);
SELECT count(*) FROM Atoms WHERE outgoing @> ARRAY[CAST(7 AS BIGINT)];
31798397 originally --
clean: 36 hours
VACUUM FULL;
en_dj_mtwo | 5022 MB | pg_default |
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+-------+------------+-------------
public | atoms | table | linas | 1452 MB |
public | spaces | table | linas | 8192 bytes |
public | typecodes | table | linas | 24 kB |
public | valuations | table | linas | 842 MB |
public | values | table | linas | 8192 bytes |
SELECT count(*) FROM Atoms; 15989736 = 16M
SELECT count(*) FROM Valuations; 9481129 = 9.5M
pca
(pca 'fetch-pairs)
This loads 15274833 atoms. What are the rest ???
There are 851964 WordNodes, but only 137078 are loaded.
851964 - 137078 = 714886
15989736 - 15274833 = 714903 so 7 other atoms not loaded. marginals?
(Connector . 268592) (ConnectorSeq . 6239997)
(Section . 8629163) (WordNode . 137078)
There seem to be no marginals at all, so far... so how is the mrg_test
working???
(batch-all-pair-mi pca)
Support: found num left= 137078 num right= 6239997 in 281 secs
Done with wild-card count N(x,*) and N(*,y) in 43276 secs
== 12 hours!!!
Done computing N(*,*) total-count= 18489594.0 in 123 secs
Going to do individual pair frequencies
Done computing 8629163 pairs in 434 secs
Start computing log P(*,y)
Done computing 6239997 left-wilds in 206 secs
Done with -log P(*,y), start -log P(x,*)
Done computing 137078 right-wilds in 4 secs
Done storing 6239997 left-wilds in 4577 secs
Done computing 8629163 pair MI's in 4368 secs <<=== typical above so OK
Finished left entropy subtotals in 37788 secs
Finished right entropy subtotals in 308 secs
Finished left MI subtotals in 37692 secs <<===
Finished right MI subtotals in 279 secs
Finished left norm marginals in 34690 secs <<===
Finished right norm marginals in 449 secs
Going to compute the left, right and total entropy
Finished left norm totals in 21 secs
Finished right norm totals in 1029 secs
Fuuuuuuuu again.. it was almost done and I screwed it.
start 13:43 expect update after midnight.
(x-batch pca)
Support: found num left= 137078 num right= 6239997 in 276 secs
Going to do column and row subtotals
Finished left entropy subtotals in 35680 secs <====
Finished right entropy subtotals in 374 secs
Finished left MI subtotals in 36424 secs
Finished right MI subtotals in 382 secs
Finished left norm marginals in 36593 secs
Finished right norm marginals in 486 secs
Going to compute the left, right and total entropy
Finished left norm totals in 26 secs
Finished right norm totals in 1227 secs
Done computing totals; start saving wildcards
Done storing 6239997 left-wilds in 4174 secs
Done storing 137078 right-wilds in 83 secs
Finished with MI computations; this took 32.3 hours
en_dj_mtwo
----------
A copy of en_pairs_rfive_mtwo but with all pairs removed, and only
the disjuncts left. Contains disjunct marginals.
Recall that rfive is not "as good as" cfive.
STATUS: Fair; has the disjunct bug #2252
DATE: 27 June 2018
en_dj_mtwo=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+-------+------------+-------------
public | atoms | table | linas | 1976 MB |
public | spaces | table | linas | 8192 bytes |
public | typecodes | table | linas | 24 kB |
public | valuations | table | linas | 4956 MB |
public | values | table | linas | 8192 bytes |
(5 rows)
select count(*) from atoms; <<<< 21651928 == 21.6M atoms
select count(*) from valuations; <<<< 57909950 == 57.9M values
Summary Report for Correlation Matrix Word-Disjunct Pairs (Connector Sets)
Left type: WordNode Right Type: ConnectorSeq Pair Type: Section
Wildcard: (ListLink (ctv 0 0 18489594)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 137078 Columns: 6239997
Size: 8629163 non-zero entries of 855366308766 possible
Fraction non-zero: 1.0088E-5 Sparsity (-log_2): 16.597
Total observations: 18489594.0 Avg obs per pair: 2.1427
Entropy Total: 20.957 Left: 19.143 Right: 9.7111
Total MI: 7.8969
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 4.8875E+4 145.4
Count (l_1) 2.2958E+5 3440. 4.697 23.66
Length (l_2) 1.3729E+4 1773. .2809 12.19
RMS Count 1.3696E+4 1732. .2802 11.91
Above, in plain English:
137K words observed,
6.2M unique disjuncts,
8.6M word-disjunct pairs observed about 2.1 times each
On average, there are 49K distinct disjuncts attached on each word.
This is weighted by the frequency with which the word was observed,
and thus is much, much larger than 6.24M/137K = 45.5 dj/word.
We conclude that frequent words also have a large number of
disjuncts on them - a top-end heavy Zipfian thing.
Likewise, each distinct disjunct appears in 145 different words.
(Again, weighted by the frequency that the disjunct was observed.)
On average, there were 230K observations of disjuncts for each word.
(Again, weighted by the frequency with which the word was observed)
This works out to 4.7 observations per disjunct (per word).
The lengths are anemic. The weighted length of a typical word-vector
is 13.7K disjuncts. Which seems reasonable. This is the quantity
that appears in the denominator of the cosine products. However,
when comparing this to the support: Oofdah. The support is just
sooo much larger. On average, each disjunct contributes to just
0.29 to the length. (Remarkably, its even lower in cfive_mst!)
Technical definitions: (line 99 of report-api.scm)
left-support = sum_x P(x,*) |(x,*)|
left-count = sum_x P(x,*) N(x,*)
left-length = sqrt [ sum_x P(x,*) L(x,*) ]
So x==word, here, and the *'s are disjuncts.
4494883736 Bytes uncompressed = 4.5 GBytes
474253125 Bytes compressed = 474 MBytes
md5sum
8392529389ca2c07666860a64f4d594a en_dj_mtwo.sql
6d03d11dc356665868e7f1d529cc8fe7 en_dj_mtwo.sql.bz2
en_dj_cfive
-----------
Identical to en_pairs_cfive_mst but with word-pairs removed. This
does contain the pseudo-dj marginals, but not the cross-connector
marginals.
STATUS: Fair; has the disjunct bug #2252
DATE: 28 June 2018
en_dj_cfive=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 7453 MB |
public | spaces | table | ubuntu | 8192 bytes |
public | typecodes | table | ubuntu | 24 kB |
public | valuations | table | ubuntu | 18 GB |
public | values | table | ubuntu | 8192 bytes |
(5 rows)
select count(*) from atoms; <<<< 81241106 == 81.2 M Atoms
select count(*) from valuations; <<<< 217088222 == 217 M Valuations
Summary Report for Correlation Matrix Word-Disjunct Pairs (Connector Sets)
Left type: WordNode Right Type: ConnectorSeq Pair Type: Section
Wildcard: (ListLink (ctv 0 0 69447009)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 444903 Columns: 23444293
Size: 31914870 non-zero entries of 10430436288579 possible
Fraction non-zero: 3.0598E-6 Sparsity (-log_2): 18.318
Total observations: 69447009.0 Avg obs per pair: 2.1760
Entropy Total: 23.092 Left: 21.137 Right: 10.112
Total MI: 8.1569
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 2.2742E+5 367.6
Count (l_1) 9.4430E+5 7917. 4.152 21.54
Length (l_2) 2.4549E+4 2397. .1079 6.520
RMS Count 2.4499E+4 2379. .1077 6.472
Ahh -- interesting ... there are words that appear in connectors, that
do not appear in word-disjunct pairs. (that do not have an associated
disjunct).
There are 498239 words, of which 444903 have disjuncts and 53336
are in connectors only. From the cross-connectors, we can find
out the converse: number of words that are in connectors only.
((make-pseudo-cset-api) 'fetch-pairs):
-- Elapsed time to load csets: 6951 secs
-- Atomspace holds 81224849 atoms
-- (cog-report-counts)
((PredicateNode . 14) (ListLink . 24169074) (TypeNode . 2)
(AnchorNode . 1) (Connector . 971810) (ConnectorDir . 2)
(ConnectorSeq . 23670834) (Section . 31914870) (SchemaNode . 1)
(WordNode . 498239) (AnyNode . 2))
-- top says 110 GB RAM
-- (psa 'left-basis-size) -- 444903 -- agrees with above
-- (psa 'right-basis-size) -- 23444293 -- as above, took 8GB more RAM!
17121601272 Bytes uncompressed == 17 GBytes
1873684270 Bytes compressed == 1.87 GBytes
c809c93f690ae0f583848ceb78d0c474 en_dj_cfive.sql
c63df81bd14e5c0d478e654e3be76369 en_dj_cfive.sql.bz2
en_micro_dj_cfive
-----------------
Filtered copy of en_dj_cfive, with the (500 10 10) filter-cut applied:
all words with word-count less than 500 removed; all sections with a
count of less than 10 removed; all dj's with a count of less than 10
removed. DB does NOT contain marginals!
COMMENT ON DATABASE en_micro_dj_cfive IS
'en_cfive Sections (only) with word-count > 500; section-count > 10';
STATUS: Fair; has the disjunct bug #2252
DATE: 9 June 2019
en_micro_dj_cfive=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 73 MB |
public | spaces | table | ubuntu | 8192 bytes |
public | typecodes | table | ubuntu | 48 kB |
public | valuations | table | ubuntu | 101 MB |
public | values | table | ubuntu | 8192 bytes |
(5 rows)
select count(*) from atoms; <<<< 791551 == 792K Atoms
select count(*) from valuations; <<<< 1178145 == 1.2M Valuations
There are 8608 words, and 361894 disjuncts
((make-pseudo-cset-api) 'fetch-pairs):
-- Elapsed time to load csets: 64 secs
-- Atomspace holds 791555 atoms
-- (cog-report-counts)
((PredicateNode . 3) (AnchorNode . 1) (Connector . 125223)
(ConnectorDir . 2) (ConnectorSeq . 187063) (Section . 392715)
(SchemaNode . 1) (WordNode . 86545) (AnyNode . 2))
-- top says 1 GB RAM
-- (psa 'left-basis-size) ;; 10813
-- (psa 'right-basis-size) ;; 187063
-- These dimensions do NOT agree with what the filter reported,
which was 8608 x 361894. This is because some low-count words
remain in connector sets.
96968299 Bytes uncompressed == 97 MBytes
18093513 Bytes compressed == 18 MBytes
3542b295676290071ac2f2e476c3a0c5 en_micro_dj_cfive.sql
eef31e0cd5edd5ad2626279a2886dff5 en_micro_dj_cfive.sql.bz2
en_micro_cfive
--------------
Same as en_micro_dj_cfive, but with marginals and MM^T values.
STATUS: Bad. Filtering failed to discard Sections correctly.
Use `en_micro_marg` instead.
DATE: 9 June 2019
COMMENT ON DATABASE en_micro_cfive IS
'en_cfive Sections + Marginals + MM^T with word-count > 500; section-count > 10';
en_micro_cfive=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 90 MB |
public | spaces | table | ubuntu | 8192 bytes |
public | typecodes | table | ubuntu | 48 kB |
public | valuations | table | ubuntu | 216 MB |
public | values | table | ubuntu | 8192 bytes |
(5 rows)
select count(*) from atoms; <<<< 989440 == 989K Atoms, (200K more)
select count(*) from valuations; <<<< 1980471 == 1.98M Vals (800K more)
((make-pseudo-cset-api) 'fetch-pairs):
-- Elapsed time to load csets: 66 secs
-- Atomspace holds 989442 atoms
-- (cog-report-counts) (omitting bogus AnchorNode, SchemaNode)
((PredicateNode . 13) (ListLink . 197877) (Connector . 125223)
(ConnectorDir . 2) (ConnectorSeq . 187063) (Section . 392715)
(WordNode . 86545) (AnyNode . 2))
-- top says 1.2 GB RAM
(print-matrix-summary-report psa)
Summary Report for Correlation Matrix Word-Disjunct Pairs (Connector Sets)
Left type: WordNode Right Type: ConnectorSeq Pair Type: Section
Wildcard: (ListLink (ctv 0 0 20782007)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 10813 Columns: 187063
Size: 392715 non-zero entries of 2022712219 possible
Fraction non-zero: 1.9415E-4 Sparsity (-log_2): 12.331
Total observations: 20782007.0 Avg obs per pair: 52.919
Entropy Total: 16.564 Left: 13.788 Right: 7.7620
Total MI: 4.9867
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 121.4 7387.
Count (l_1) 2.2239E+4 4.8439E+5 183.1 65.57
Length (l_2) 7511. 3.4794E+4 61.85 4.710
RMS Count 7263. 3.4445E+4 59.80 4.663
MM^T support=392715.0 count=462177929473.0
No M^TM data present
154394374 Bytes uncompressed == 154 MBytes
26066918 Bytes compressed == 26 MBytes
9b326477e688522b098aa5df8a699d59 en_micro_cfive.sql
9077869c247d29252f361adac46f3b2b en_micro_cfive.sql.bz2
en_micro_links
--------------
Same as en_micro_dj_cfive, but with unconnectable connectors/conseqs
removed. Does include marginals and MM^T values.
STATUS: Bad. Contains words below the cutoff. Use `en_micro_marg`
instead.
DATE: 9 June 2019
COMMENT ON DATABASE en_micro_links IS
'en_cfive consistent Sections with word-count > 500; section-count > 10';
en_micro_links=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 51 MB |
public | valuations | table | ubuntu | 168 MB |
select count(*) from atoms; <<<< 572096 == 572K Atoms, (400K less)
select count(*) from valuations; <<<< 1417959 == 1.42M Vals (570K less)
((make-pseudo-cset-api) 'fetch-pairs):
-- Elapsed time to load csets: 37 secs
-- Atomspace holds 572099 atoms
-- (cog-report-counts) (omitting bogus AnchorNode, SchemaNode)
((PredicateNode . 14) (ListLink . 124635) (Connector . 19943)
(ConnectorSeq . 113821) (Section . 302867) (WordNode . 10813))
-- top says 825 MBytes RAM
(print-matrix-summary-report psa)
Summary Report for Correlation Matrix Word-Disjunct Pairs (Connector Sets)
Left type: WordNode Right Type: ConnectorSeq Pair Type: Section
Wildcard: (ListLink (ctv 0 0 18880223)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 10813 Columns: 113821
Size: 302867 non-zero entries of 1230746473 possible
Fraction non-zero: 2.4608E-4 Sparsity (-log_2): 11.989
Total observations: 18880223.0 Avg obs per pair: 62.338
Entropy Total: 16.127 Left: 13.127 Right: 7.9071
Total MI: 4.9067
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 133.4 3462.
Count (l_1) 2.4468E+4 3.6716E+5 183.4 106.1
Length (l_2) 8260. 3.2453E+4 61.91 9.374
RMS Count 7991. +nan.0 59.89 +nan.0
MM^T support=302867.0 count=461960433973.0
No M^TM data present
102270103 Bytes uncompressed == 102 MBytes
17233733 Bytes compressed == 17 MBytes
cb04a74e6a8024bb9e7f1ccafbf71dbe en_micro_links.sql
c7c14d954818babc861d9efcecfc6204 en_micro_links.sql.bz2
en_micro_marg
--------------
Same as en_micro_dj_cfive, but with unconnectable connectors/conseqs
removed. Does include marginals and MM^T values.
Build:
(sql-open "postgres:///en_micro_dj_cfive")
(define pca (make-pseudo-cset-api))
(define psa (add-pair-stars pca))
(psa 'fetch-pairs)
(sql-close)
(define psc (add-support-compute psa))
(psc 'cache-all)
(define fsa (add-subtotal-filter psa 500 10 10 #f))
(define lfa (add-linkage-filter fsa))
(sql-open "postgres:///en_micro_marg")
(define fso (make-store lfa))
(fso 'store-all-elts)
(sql-close)
(sql-open "postgres:///en_micro_marg")
(psa 'fetch-pairs)
(batch-all-pair-mi psa)
(define btr (batch-transpose psa))
(btr 'mmt-marginals)
STATUS: Fair; has the disjunct bug #2252
DATE: 14 June 2019
COMMENT ON DATABASE en_micro_marg IS
'en_cfive consistent Sections with word-count > 500; section-count > 10';
en_micro_marg=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 29 MB |
public | valuations | table | ubuntu | 154 MB |
select count(*) from atoms; <<<< 324343 == 324K Atoms
select count(*) from valuations; <<<< 827435 == 827K Vals
((make-pseudo-cset-api) 'fetch-pairs):
-- Elapsed time to load csets: 37 secs
-- (cog-report-counts) (omitting bogus AnchorNode, SchemaNode)
((PredicateNode . 14) (ListLink . 68722) (Connector . 3226)
(ConnectorSeq . 67111) (Section . 183644) (WordNode . 1623))
(print-matrix-summary-report psa)
Summary Report for Correlation Matrix Word-Disjunct Pairs (Connector Sets)
Left type: WordNode Right Type: ConnectorSeq Pair Type: Section
Wildcard: (ListLink (ctv 0 0 14751096)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 1610 Columns: 67111
Size: 183644 non-zero entries of 108048710 possible
Fraction non-zero: 1.6996E-3 Sparsity (-log_2): 9.2006
Total observations: 14751096.0 Avg obs per pair: 80.324
Entropy Total: 15.167 Left: 12.043 Right: 7.7236
Total MI: 4.6004
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 87.72 1557.
Count (l_1) 2.7723E+4 2.3734E+5 316.0 152.4
Length (l_2) 1.0224E+4 3.0318E+4 116.6 19.47
RMS Count 9851. 2.9865E+4 112.3 19.18
MM^T support=183644.0 count=408941670158.0 entropy=-3.600
No M^TM data present
59977841 Bytes uncompressed == 60MBytes
10305529 Bytes compressed == 10MBytes
60c5a06f518e36c94a0d57027b59e24d en_micro_marg.sql
887a32cfae68484a480c7f34ce44c536 en_micro_marg.sql.bz2
en_micro_fuzz
--------------
Same as en_micro_marg, but after fuzz (cosine) word-classing.
No gram class marginals, and the pseudo-cset marginals are surely
left-over junk from before. Why is the dump size smaller??
(size of Valuations table is bigger; numer of valuations smaller.
Why are there fewer valuations?)
BUILD:
(gram-classify-greedy-fuzz 0.65 0.3 4)
STATUS: Uncertain. Turns out dot-product denominators used for cosine
were not recomputed after merge, leading to questionable
cosines.
DATE: 15 June 2019
COMMENT ON DATABASE en_micro_fuzz IS
'en_micro with (gram-classify-greedy-fuzz 0.65 0.3 4)';
en_micro_fuzz=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 36 MB |
public | valuations | table | ubuntu | 172 MB |
select count(*) from atoms; <<<< 359686 == 360K Atoms
select count(*) from valuations; <<<< 749950 == 750K Vals
56016110 Bytes uncompressed == 56MBytes
9020871 Bytes compressed == 9MBytes
19b27db361caa7d780a1ac62b08c72dd en_micro_fuzz.sql
3324b5cdaf45aedd164327a4f66c2695 en_micro_fuzz.sql.bz2
en_micro_fuzz_exp
-----------------
Same as en_micro_fuzz, but with gram class marginals, so that it's
suitable for export to LG.
BUILD:
(gram-classify-greedy-fuzz 0.65 0.3 4)
(define asc (add-singleton-classes psa))
(asc 'create-hi-count-singles 1)
(define gca (make-gram-class-api))
(define gcs (add-pair-stars gca))
(define gcf (add-wordclass-filter gcs))
(batch-all-pair-mi gcf)
(define sto (make-store gcf))
(sto 'store-all)
(gcs 'store-aux)
(use-modules (opencog nlp lg-export))
(export-csets gcf "dict.db" "EN_us") ; Will store 142358 csets
STATUS: Fair; has the disjunct bug #2252
ISSUE: Uncertain. Turns out dot-product denominators used for cosine
were not recomputed after merge, leading to questionable
cosines.
DATE: 18 June 2019
COMMENT ON DATABASE en_micro_fuzz_exp IS
'en_micro_fuzz + marginals for export (greedy-fuzz 0.65 0.3)';
en_micro_fuzz_exp=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 25 MB |
public | valuations | table | ubuntu | 60 MB |
select count(*) from atoms; <<<< 284291 == 284K Atoms
select count(*) from valuations; <<<< 699105 == 700K Vals
Summary Report for Correlation Matrix WordClass-Disjunct Pairs wordclass-filter
Left type: WordClassNode Right Type: ConnectorSeq Pair Type: Section
Wildcard: (ListLink (ctv 0 0 12003313)
(AnyNode "gram-class-word")
(AnyNode "gram-class-disjunct")
)
Rows: 675 Columns: 67056
Size: 142358 non-zero entries of 45262800 possible
Fraction non-zero: 3.1451E-3 Sparsity (-log_2): 8.3127
Total observations: 12003312.5 Avg obs per pair: 84.318
Entropy Total: 13.746 Left: 11.431 Right: 6.3415
Total MI: 4.0264
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 50.64 2817.
Count (l_1) 3.2160E+4 3.3795E+5 635.0 120.0
Length (l_2) 1.9406E+4 6.0704E+4 383.2 21.55
RMS Count 1.8989E+4 +nan.0 374.9 +nan.0
No MM^T data present
No M^TM data present
53458027 Bytes uncompressed == 53 MBytes
9439051 Bytes compressed == 9 MBytes
8fad77a6ec29bd27cd4d8fc176450beb en_micro_fuzz_exp.sql
5b1c02d1430f3e3b15cbfead3ed0b9c4 en_micro_fuzz_exp.sql.bz2
en_micro_discrim
-----------------
Same as en_micro_marg, but after discrim (cosine) word-classing.
Includes marginals.
BUILD:
(gram-classify-greedy-discrim 0.5 4)
Marginal build just like en_micro_fuzz_exp
Will store 132775 csets
STATUS: Fair; has the disjunct bug #2252
ISSUE: Uncertain. Turns out dot-product denominators used for cosine
were not recomputed after merge, leading to questionable
cosines.
DATE: 18 June 2019
COMMENT ON DATABASE en_micro_discrim IS
'en_micro_marg (greedy-fuzz 0.5 4) plus marginals for export';
en_micro_discrim=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 25 MB |
public | valuations | table | ubuntu | 57 MB |
select count(*) from atoms; <<<< 274228 == 274K Atoms
select count(*) from valuations; <<<< 669606 == 670K Vals
Summary Report for Correlation Matrix WordClass-Disjunct Pairs
Left type: WordClassNode Right Type: ConnectorSeq Pair Type: Section
Wildcard: (ListLink (ctv 0 0 11428005)
(AnyNode "gram-class-word")
(AnyNode "gram-class-disjunct")
)
Rows: 416 Columns: 67056
Size: 132775 non-zero entries of 27895296 possible
Fraction non-zero: 4.7598E-3 Sparsity (-log_2): 7.7149
Total observations: 11428005.3 Avg obs per pair: 86.070
Entropy Total: 13.230 Left: 11.268 Right: 5.5832
Total MI: 3.6218
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 36.30 4172.
Count (l_1) 3.3309E+4 4.8650E+5 917.7 116.6
Length (l_2) 2.2783E+4 8.5731E+4 627.7 20.55
RMS Count 2.2287E+4 +nan.0 614.0 +nan.0
52474191 Bytes uncompressed == 52 MBytes
12924981 Bytes compressed == 13 MBytes
5c986d99da8277fade1ac9d8ec304ba7 en_micro_discrim.sql
6caaf9579d054d458c7fe1b23085eb67 en_micro_discrim.sql.bz2
en_mini_dj_cfive
----------------
Filtered copy of en_dj_cfive, with the (40 5 5) filter-cut applied:
all words with word-count less than 40 removed; all sections with a
count of less than 5 removed; all dj's with a count of less than 5
removed. DB does NOT contain marginals!
Note that there is a HUGE number of WordNodes that appear in
ConnectorSeq's that do NOT appear in Sections!
COMMENT ON DATABASE en_mini_dj_cfive IS
'en_cfive Sections (only) with word-count > 40; section-count > 5';
STATUS: Fair; has the disjunct bug #2252
DATE: 9 June 2019
en_mini_dj_cfive=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 206 MB |
public | typecodes | table | ubuntu | 48 kB |
public | valuations | table | ubuntu | 274 MB |
select count(*) from atoms; <<<< 2139220 == 2.1M Atoms
select count(*) from valuations; <<<< 3191817 == 3.2M Valuations
There are 171522 words, and 647988 disjuncts, and 1063939 Sections
((make-pseudo-cset-api) 'fetch-pairs):
-- Elapsed time to load csets: 157 secs
-- Atomspace holds 2139224 atoms
-- (cog-report-counts)
((PredicateNode . 3) (Connector . 255766) (ConnectorDir . 2)
(ConnectorSeq . 647988) (Section . 1063939) (WordNode . 171522)
(AnyNode . 2))
-- top says 2.5 GB RAM
-- (psa 'left-basis-size) ;; 18533
-- (psa 'right-basis-size) ;; 647988
-- These dimensions do NOT agree with what the filter reported,
which was 42406 x 968594 which is understandable.
271616272 Bytes uncompressed == 272 MBytes
46513972 Bytes compressed == 46 MBytes
5e351f5644984574f0e12bedcf9c94d7 en_mini_dj_cfive.sql
b7e69440de6e6bd6a2d56880f05b2958 en_mini_dj_cfive.sql.bz2
en_mini_cfive
-------------
Same as en_mini_dj_cfive, but with marginals and MM^T values.
Note that this has 10x more words in the connectors, than words in
sections. This needs to be cleaned up.
STATUS: Bad. Filtering failed to discard Sections correctly.
Use `en_mini_marg` instead.
DATE: 9 June 2019
COMMENT ON DATABASE en_mini_cfive IS
'en_cfive Sections + Marginals + MM^T with word-count > 40; section-count > 5';
en_mini_cfive=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 265 MB |
public | typecodes | table | ubuntu | 24 kB |
public | valuations | table | ubuntu | 624 MB |
select count(*) from atoms; <<<< 2805754 == 2.8M Atoms (660K more)
select count(*) from valuations; <<<< 5876443 == 5.9M Vals (2.7M more)
((make-pseudo-cset-api) 'fetch-pairs):
-- Elapsed time to load csets: 212 secs
-- Atomspace holds 2805756 atoms
-- (cog-report-counts) (omitting bogus AnchorNode, SchemaNode)
((PredicateNode . 13) (ListLink . 666522) (Connector . 255766)
(ConnectorSeq . 647988) (Section . 1063939) (WordNode . 171522)
(AnyNode . 2))
-- top says 3.5 GB RAM
(print-matrix-summary-report psa)
Summary Report for Correlation Matrix Word-Disjunct Pairs (Connector Sets)
Left type: WordNode Right Type: ConnectorSeq Pair Type: Section
Wildcard: (ListLink (ctv 0 0 25642000)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 18533 Columns: 647988
Size: 1063939 non-zero entries of 12009161604 possible
Fraction non-zero: 8.8594E-5 Sparsity (-log_2): 13.462
Total observations: 25642000.0 Avg obs per pair: 24.101
Entropy Total: 17.789 Left: 15.123 Right: 7.7718
Total MI: 5.1061
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 165.4 3.0281E+4
Count (l_1) 1.8942E+4 6.4730E+5 114.6 21.38
Length (l_2) 6195. 3.5269E+4 37.46 1.165
RMS Count 6045. 3.5067E+4 36.56 1.158
MM^T support=1063939.0 count=485701926586.0
No M^TM data present
467604993 Bytes uncompressed == 468 MBytes
46513972 Bytes compressed == 47 MBytes
59df6126b629ae4750877d03a7fbf5be en_mini_cfive.sql
2643824974318078d8013b609d81c919 en_mini_cfive.sql.bz2
en_mini_links
-------------
Same as en_mini_dj_cfive, but with unconnectable connectors/conseqs
removed. Does include marginals and MM^T values.
STATUS: Bad. Contains words below the cutoff. Use en_mini_marg instead.
DATE: 9 June 2019
COMMENT ON DATABASE en_mini_links IS
'en_cfive consistent Sections with word-count > 40; section-count > 5';
en_mini_links=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 143 MB |
public | valuations | table | ubuntu | 328 MB |
select count(*) from atoms; <<<< 1545047 == 1.5M Atoms(1.3M less)
select count(*) from valuations; <<<< 3788228 == 3.8M Vals (2.1M less)
((make-pseudo-cset-api) 'fetch-pairs):
-- Elapsed time to load csets: 127 secs
-- Atomspace holds 1545049 atoms
-- (cog-report-counts) (omitting bogus AnchorNode, SchemaNode)
((PredicateNode . 13) (ListLink . 380854) (Connector . 34565)
(ConnectorSeq . 362320) (Section . 748758) (WordNode . 18533))
-- top says 2.0 GB RAM
(print-matrix-summary-report psa)
Summary Report for Correlation Matrix Word-Disjunct Pairs (Connector Sets)
Left type: WordNode Right Type: ConnectorSeq Pair Type: Section
Wildcard: (ListLink (ctv 0 0 22626006)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 18533 Columns: 362320
Size: 748758 non-zero entries of 6714876560 possible
Fraction non-zero: 1.1151E-4 Sparsity (-log_2): 13.131
Total observations: 22626006.0 Avg obs per pair: 30.218
Entropy Total: 17.171 Left: 14.192 Right: 8.0388
Total MI: 5.0603
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 187.0 1.1736E+4
Count (l_1) 2.1460E+4 4.4365E+5 114.7 37.80
Length (l_2) 7016. 3.2343E+4 37.51 2.756
RMS Count 6848. +nan.0 36.61 +nan.0
MM^T support=748758.0 count=485542996090.0
No M^TM data present
278672471 Bytes uncompressed == 278 MBytes
40074275 Bytes compressed == 40 MBytes
312fb63ddc80212bf8848ab710235dbc en_mini_links.sql
647a8825c97223a2f8d93d86cbc86247 en_mini_links.sql.bz2
en_mini_marg
-------------
Same as en_mini_dj_cfive, but with unconnectable connectors/conseqs
removed. Does include marginals and MM^T values.
BUILD: Same as `en_micro_marg`.
STATUS: Fair; has the disjunct bug #2252
DATE: 15 June 2019
COMMENT ON DATABASE en_mini_marg IS
'en_cfive consistent Sections with word-count > 40; section-count > 5';
en_mini_marg=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 108 MB |
public | valuations | table | ubuntu | 551 MB |
select count(*) from atoms; <<<< 1177540 == 1.2M Atoms
select count(*) from valuations; <<<< 2940501 == 2.9M Vals
((make-pseudo-cset-api) 'fetch-pairs):
-- Elapsed time to load csets: 82 secs
-- Atomspace holds 1177547 atoms
-- (cog-report-counts) (omitting bogus AnchorNode, SchemaNode)
((PredicateNode . 15) (ListLink . 277097) (Connector . 14926)
(ConnectorSeq . 269711) (Section . 608241) (WordNode . 7548)
-- top says 1.5 GB RAM
(print-matrix-summary-report psa)
Summary Report for Correlation Matrix Word-Disjunct Pairs (Connector Sets)
Left type: WordNode Right Type: ConnectorSeq Pair Type: Section
Wildcard: (ListLink (ctv 0 0 20817076)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 7385 Columns: 269711
Size: 608241 non-zero entries of 1991815735 possible
Fraction non-zero: 3.0537E-4 Sparsity (-log_2): 11.677
Total observations: 20817076.0 Avg obs per pair: 34.225
Entropy Total: 16.792 Left: 13.687 Right: 8.0653
Total MI: 4.9602
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 173.1 7409.
Count (l_1) 2.2831E+4 3.6516E+5 131.9 49.28
Length (l_2) 7585. 3.1013E+4 43.82 4.186
RMS Count 7399. 3.0766E+4 42.74 4.152
MM^T support=608241.0 count=475267638744.0 entropy=-0.361
No M^TM data present
217959158 Bytes uncompressed == 218 MBytes
33416655 Bytes compressed == 33 MBytes
fcb0152dedb99304bbc0bac0745c9ebe en_mini_marg.sql
fc0f0749efa8caf1416c6aed3509ca17 en_mini_marg.sql.bz2
en_mini_fuzz
------------
Same as en_mini_marg, but after fuzz (cosine) word-classing.
No gram class marginals, and the pseudo-cset marginals are surely
left-over junk from before.
BUILD:
(gram-classify-greedy-fuzz 0.65 0.3 4)
STATUS: Fair; has the disjunct bug #2252
STATUS: Uncertain. Turns out dot-product denominators used for cosine
were not recomputed after merge, leading to questionable
cosines.
DATE: 17 June 2019
COMMENT ON DATABASE en_mini_fuzz IS
'en_mini_marg with (gram-classify-greedy-fuzz 0.65 0.3 4)';
en_mini_fuzz=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 135 MB |
public | valuations | table | ubuntu | 593 MB |
select count(*) from atoms; <<<< 1340138 == 1.34M Atoms
select count(*) from valuations; <<<< 2814992 == 2.81M Vals
215566437 Bytes uncompressed == 216 MBytes
30940665 Bytes compressed == 31 MBytes
3cf78077ef7e79c7a92824f3158b3f6b en_mini_fuzz.sql
7e306eaa526e289f61d06ffa9b0e09fa en_mini_fuzz.sql.bz2
en_mini_fuzz_exp
-----------------
Same as en_mini_fuzz, but with gram class marginals, so that it's
suitable for export to LG. Note that the singleton classes do not
contain the clustering left-overs, and maybe they should ...
BUILD:
(gram-classify-greedy-fuzz 0.65 0.3 4)
(define asc (add-singleton-classes psa))
(asc 'create-hi-count-singles 1)
(define gca (make-gram-class-api))
(define gcs (add-pair-stars gca))
(define gcf (add-wordclass-filter gcs))
(batch-all-pair-mi gcf)
(define sto (make-store gcf))
(sto 'store-all)
(gcs 'store-aux)
(use-modules (opencog nlp lg-export))
(export-csets gcf "dict.db" "EN_us") ; Will store 501957 csets
STATUS: Fair; has the disjunct bug #2252
STATUS: Uncertain. Turns out dot-product denominators used for cosine
were not recomputed after merge, leading to questionable
cosines.
DATE: 18 June 2019
COMMENT ON DATABASE en_mini_fuzz_exp IS
'en_mini_fuzz + marginals for export (greedy-fuzz 0.65 0.3)';
en_mini_fuzz_exp=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 99 MB |
public | valuations | table | ubuntu | 222 MB |
select count(*) from atoms; <<<< 1077157 == 1.08M Atoms
select count(*) from valuations; <<<< 2601283 == 2.60M Vals
Summary Report for Correlation Matrix WordClass-Disjunct Pairs
Left type: WordClassNode Right Type: ConnectorSeq Pair Type: Section
Wildcard: (ListLink (ctv 0 0 16583937)
(AnyNode "gram-class-word")
(AnyNode "gram-class-disjunct")
)
Rows: 3692 Columns: 269115
Size: 501957 non-zero entries of 993572580 possible
Fraction non-zero: 5.0520E-4 Sparsity (-log_2): 10.951
Total observations: 16583936.7 Avg obs per pair: 33.039
Entropy Total: 15.387 Left: 12.969 Right: 6.7695
Total MI: 4.3517
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 97.02 1.1241E+4
Count (l_1) 2.7228E+4 4.3814E+5 280.6 38.98
Length (l_2) 1.5021E+4 5.7309E+4 154.8 5.098
RMS Count 1.4820E+4 +nan.0 152.7 +nan.0
No MM^T data present
No M^TM data present
209846267 Bytes uncompressed == 210 MBytes
31289650 Bytes compressed == 31 MBytes
298adfde5df642dc9c8dc7442eaf3533 en_mini_fuzz_exp.sql
e96831e97c5b7a9c906b0792c49817f7 en_mini_fuzz_exp.sql.bz2
==========
next:
createdb en_dj_etwo -T en_dj_mtwo
guile -l x-mclean.scm
(sql-open "postgres:///en_dj_etwo?user=linas&password=asdf")
(define cra (make-connector-vec-api))
(cra 'fetch-pairs)
Elapsed time to load word sections: 2850 seconds
(cra 'make-left-stars)
Elapsed time to compute left-stars: 958 secs
(EvaluationLink . 18265271)
(Connector . 268594)
(ConnectorSeq . 6239997)
(Section . 8629163)
(WordNode . 137078)
OK, makes sense.
Before batching, we're at 41.346GB VIRT according to top.
during marginal counting, up to 45.990 VIRT
(batch-all-pair-mi cra)
Support: found num left= 137078 num right= 18265271 in 0 secs
Verify that left-stars really got saved, yeah?
(because they are crazy-format...)
----------------
what fraction?
(fold (lambda (evo sum)
(if (cog-ctv? (cog-tv evo)) (+ sum 1) sum)) 0
(crs 'right-basis))
7484838 .... not even half-way, yet
10008397
12988443
15082554
16514195
17648358
18149413
18262285
Done with wild-card count N(x,*) and N(*,y) in 374968 secs
== 374968 = 104 hours = 4.34 days
opencog/matrix/compute-mi.scm:254:32: Throw to key
`bad-summation' with args `(count-all-pairs "Error: pair-counts unequal:
42933043.0 47444396.0\n")'.
wtf...
(define crc (add-pair-count-api cra))
(fold (lambda (wrd sum)
(+ sum (crc 'right-wild-count wrd))) 0 (cra 'left-basis))
(fold (lambda (cro sum)
(+ sum (crc 'left-wild-count cro))) 0 (cra 'right-basis))
Spot check?
(length (crs 'right-stars (Word "Jurisdiction")))
two sections, both count of 1
one section has count f 10032
other has count of 1 ... that's OK, then,
Beat me. en_dj_etwo has these damaged counts, but the cause of damage is
totally unclear.
Restart counting, on the same dataset.
guile -l x-mclean.scm
(cra 'fetch-pairs)
(cra 'make-left-stars)
(batch-all-pair-mi cra)
Its at 40.630g VIRT in top
Support: found num left= 137078 num right= 18265271 in 33 secs
and now, wait 4 days .... till 7th of July
Finished left norm marginals in 368287 secs <<<= 102 hours, 4 days
Finished left totals in 460 secs
Finished right norm marginals in 3233 secs
Finished right totals in 4 secs
Done with wild-card count N(x,*) and N(*,y) in 371984 secs
Total count N(*,*) = 47444396.0 = 42933043.0 <<<< === not symmetric!!!
Finished column (left) norm averages in 2173 secs
Finished row (right) norm averages in 42 secs
Going to do individual pair frequencies
Done computing 23058056 pairs in 5442 secs
Start computing log P(*,y)
Done computing 18265271 left-wilds in 461 secs
Done with -log P(*,y), start -log P(x,*)
Done computing 137078 right-wilds in 10 secs
Stored 40000 of 18265271 left-wilds in 37 secs (1081 pairs/sec)
tored 18240000 of 18265271 left-wilds in 28 secs (1429 pairs/sec)
Done storing 18265271 left-wilds in 12945 secs
Stored 40000 of 137078 right-wilds in 25 secs (1600 pairs/sec)
Stored 80000 of 137078 right-wilds in 26 secs (1538 pairs/sec)
Stored 120000 of 137078 right-wilds in 26 secs (1538 pairs/sec)
Done storing 137078 right-wilds in 88 secs
Done computing and saving -log P(x,*) and P(*,y)
Going to do individual pair MI
ERROR: In procedure scm-error:
Bad method call on cross-section: right-element
scheme@(guile-user) [1]> ,bt
In opencog/matrix/compute-mi.scm:
583:24 7 (batch-all-pair-mi _)
366:24 6 (compute-n-cache-pair-mi _)
In srfi/srfi-1.scm:
640:9 5 (for-each #<procedure right-loop (left-item)> (# # # # # # # # …))
In opencog/matrix/compute-mi.scm:
347:58 4 (right-loop (WordNode "litigant")
)
In srfi/srfi-1.scm:
592:17 3 (map1 (# # #))
In opencog/matrix/compute-mi.scm:
337:75 2 (do-one-pair #)
In opencog/nlp/learn/connector-vec.scm:
381:38 1 (_ _ . _)
In unknown file:
0 (scm-error misc-error #f "~A ~S" ("Bad method call on cross…" …) …)
Fix many things, restart... on 1AM 10 July
$ guile -l x-mclean.scm
scheme@(guile-user)> (cra 'fetch-pairs)
Elapsed time to load word sections: 3032 seconds
Elapsed time to load cross-marginals: 2674 seconds
scheme@(guile-user)> (cra 'make-left-stars)
Elapsed time to compute left-stars: 735 secs
scheme@(guile-user)> (batch-all-pair-mi cra)
Start computing the basis
Support: found num left= 137078 num right= 18265271 in 40 secs
Finished left norm marginals in 393573 secs
Finished left totals in 332 secs
Finished right norm marginals in 6591 secs
Finished right totals in 10 secs
Done with wild-card count N(x,*) and N(*,y) in 400506 secs
Total count N(*,*) = 47444396.0 = 47444396.0
Finished column (left) norm averages in 1538 secs
Finished row (right) norm averages in 31 secs
Going to do individual pair frequencies
Done computing 24213074 pairs in 8776 secs
Start computing log P(*,y)
Done computing 18265271 left-wilds in 308 secs
Done with -log P(*,y), start -log P(x,*)
Done computing 137078 right-wilds in 5 secs
Stored 40000 of 18265271 left-wilds in 12 secs (3333 pairs/sec)
Stored 18240000 of 18265271 left-wilds in 12 secs (3333 pairs/sec)
Done storing 18265271 left-wilds in 5644 secs
Stored 40000 of 137078 right-wilds in 15 secs (2667 pairs/sec)
Stored 80000 of 137078 right-wilds in 13 secs (3077 pairs/sec)
Stored 120000 of 137078 right-wilds in 14 secs (2857 pairs/sec)
Done storing 137078 right-wilds in 48 secs
Done computing and saving -log P(x,*) and P(*,y)
Going to do individual pair MI
Done 10000 of 137078 outer loops in 1074 secs, pairs=1319927 (1229.0 pairs/sec)
Done 130000 of 137078 outer loops in 2877 secs, pairs=22949051 (949.64 pairs/sec)
Done computing 24213074 pair MI's in 23091 secs
Going to do column and row subtotals
Finished left entropy subtotals in 405905 secs
Finished right entropy subtotals in 6812 secs
Finished left MI subtotals in 433689 secs
Finished right MI subtotals in 7360 secs
Going to compute the left, right and total entropy
Done computing totals; start saving wildcards
FATAL: terminating connection due to administrator command
fuckig apt-get fucked me again. Fixed in pull req #1846
(cra 'fetch-pairs)
(load "/tmp/fail.scm")
(rebatch-all-pair-mi cra)
Support: found num left= 137078 num right= 18265271 in 42 secs
Total count N(*,*) = 47444396.0 = 47444396.0
Going to do column and row subtotals
Finished left entropy subtotals in 427085 secs <<< 42.767 per second
Finished right entropy subtotals in 7285 secs
Finished left MI subtotals in 439375 secs <<< 41.571 per second
Finished right MI subtotals in 7270 secs
Done storing 18265271 left-wilds in 10806 secs
Done storing 137078 right-wilds in 85 secs
Finished with MI computations; this took 248. hours
this is for
postgres:///en_dj_etwo?user=linas&password=asdf
x-mclean.scm
scheme@(guile-user)> (print-matrix-summary-report cra)
Summary Report for Correlation Matrix Cross-section Words
Left type: WordNode Right Type: EvaluationLink Pair Type: Section
Wildcard: (ListLink (ctv 0 0 47444396)
(AnyNode "cross-connector word")
(AnyNode "cross-connector section")
)
Rows: 137078 Columns: 18265271
Size: 24213074 non-zero entries of 2503766818138 possible
Fraction non-zero: 9.6707E-6 Sparsity (-log_2): 16.658
Total observations: 47444396.0 Avg obs per pair: 1.9595
Entropy Total: 23.188 Left: 21.192 Right: +nan.0
Total MI: 9.0983
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 1422. 1.2051E+5
Count (l_1) 2.7318E+4 2.3690E+5 19.21 1.966
Length (l_2) 3114. 6763. 2.190 5.6123E-2
RMS Count 3092. +nan.0 2.174 +nan.0
What's with the NaN's ?
((add-report-api cra) 'right-entropy)
(cog-value
(List (Any "cross-connector word") (Any "cross-connector section"))
(Predicate "*-Total Entropy Key-*"))
(FloatValue 21.192079534319085 -nan 23.187774484654337)
(compute-right-entropy)
(define (right-sum FN)
(fold (lambda (left-item sum) (+ sum (FN left-item)))
0 (cra 'left-basis)))
(define (compute-right-entropy)
(define frqobj (add-pair-freq-api cra))
(right-sum
(lambda (x)
; (if (not (finite? (frqobj 'right-wild-freq x))) (format #t "foo ~A\n" x))
(if (equal? 0.0 (frqobj 'right-wild-freq x)) (format #t "foo ~A\n" x))
(*
(frqobj 'right-wild-freq x)
(frqobj 'right-wild-logli x)))))
(define (compute-right-entropy)
(define frqobj (add-pair-freq-api cra))
(right-sum
(lambda (x)
(define lli (frqobj 'right-wild-logli x))
(if (finite? lli)
(* (frqobj 'right-wild-freq x) lli) 0.0))))
11.210368083166648
(- (+ 21.192079534319085 11.210368083166648) 23.187774484654337)
$25 = 9.214673132831397
OK, great, but that is not the total MI ... Why ???
which is
(cog-value (List (Any "cross-connector word") (Any "cross-connector section"))
(PredicateNode "*-Total MI Key-*"))
$34 = (FloatValue 9.0982793869337666)
(right-sum (lambda (x) (frqobj 'right-wild-mi x))) $35 = 9.098279386856897
(left-sum (lambda (x) (frqobj 'left-wild-mi x))) $36 = 9.098279386933767
(right-sum (lambda (x) (frqobj 'right-wild-freq x))) $38 = 1.0000000000003544
(left-sum (lambda (x) (frqobj 'left-wild-freq x))) $39 = 1.000000000122195
(define supobj (add-support-api cra))
(right-sum (lambda (x) (supobj 'right-count x))) $41 = 47444396.0
(left-sum (lambda (x) (supobj 'left-count x))) $42 = 47444396.0
w
p(x,*) ln p(x,*) = 1.0327767416155317e-6
(define (plp w) (* (frqobj 'right-wild-freq w) (frqobj 'right-wild-logli w)))
sum_y p(x,y)ln p(x,y) = 1.074931351403376e-6
(define (rent w) (frqobj 'right-wild-entropy w))
sum_y p(x,y)ln p(x,y) - p(x,*) ln p(x,*) = (- (rent w) (plp w)) = 4.215460978784417e-8
sum_y p(x,y)ln [p(x,y) / p(x,*)] =
sum_y p(x,y)ln [p(x,y) / p(x,*) p(*,y)] + sum_y p(x,y) ln p(*,y)
sum_y p(x,y)ln [p(x,y) / p(x,*) p(*,y)] = 3.17694535835089e-7
(define (rim w) (frqobj 'right-wild-mi w))
sum_y p(x,y) ln p(*,y) = 3.5984914562293424e-7
(define (poo w)
(define (prd dual)
; given a dual, and the item, create the pair ....
(define sect (cra 'get-pair w dual))
(* (frqobj 'pair-freq sect) (frqobj 'left-wild-logli dual)))
(fold (lambda (dual sum) (+ sum (prd dual))) 0.0 (crs 'right-duals w)))
(define (zero w) (- (- (poo w) (rim w)) (- (rent w) (plp w))))
(fold (lambda (w sum) (+ sum (zero w))) 0.0 (crs 'left-basis))
(fold (lambda (w sum) (format #t "so ~A ~A" sum w) (+ sum (zero w))) 0.0 (crs 'left-basis))
(fold (lambda (w sum)
(define z (zero w))
(if (< 1.0e-8 z) (format #t "bad ~A ~A" z w) (format #t "good ~A ~A" z w))
(+ sum z)) 0.0 (take (crs 'left-basis) 20))
(define (audit-mi MANY)
(fold (lambda (w sum)
(define z (zero w))
(+ sum
(if (< 1.0e-8 z) (begin (format #t "bad ~A ~A" z w) 1) 0)))
0 (take (crs 'left-basis) MANY)))
(WordNode "Croker")
(define w (WordNode "Cuddles"))
(define w (WordNode "Dormez"))
-p(x,*) ln p(x,*) = plp = 1.9812442636553735e-6 OK checks
-sum_y p(x,y) ln p(x,y) = rent = 2.149862702806752e-6 OK checks
sum_y p(x,y)ln [p(x,y) / p(x,*) p(*,y)] = rim = 1.3691988899449898e-6 seems bad.
sum_y p(x,y) ln p(*,y) = poo = 1.5742713752273321e-6
(- (poo w) (rim w)) = 2.0507248528234232e-7
(- (rent w) (plp w)) = 1.6861843915137838e-7
so what is wrong?
p(x,*) = 8.43092195756904e-8
(fold (lambda (s sum) (+ sum (frqobj 'pair-freq s))) 0.0 (crs 'right-stars w))
(frqobj 'right-wild-freq w) == same as above. OK
(define (crent w) ; OK< gives same asnwer as rent
(fold (lambda (s sum)
(define p (frqobj 'pair-freq s))
(+ sum (* p (l2 p)))) 0.0 (crs 'right-stars w)))
sum_y p(x,y)ln [p(x,y) / p(x,*) p(*,y)] = rim =
sum_y p(x,y)ln [p(x,y) / p(*,y)] - sum_y p(x,y)ln p(x,*) =
sum_y p(x,y)ln [p(x,y) / p(*,y)] - ln p(x,*) sum_y p(x,y) =
sum_y p(x,y)ln [p(x,y) / p(*,y)] - ln p(x,*) p(x,*) =
sum_y p(x,y)ln [p(x,y)] - sum_y p(x,y) ln [p(*,y)] - p(x,*) ln p(x,*)
sum_y p(x,y)ln [p(x,y)] = - (frqobj 'right-wild-entropy w) = -2.149862702806752e-6
- sum_y p(x,y)ln [p(*,y)] = (poo w) = 1.5742713752273321e-6
- p(x,*) ln p(x,*) = 1.9812442636553735e-6 which checked before
So: (+ -2.149862702806752e-6 (poo w) 1.9812442636553735e-6) = 1.4056529360759538e-6
which is NOT rim. So what's up with rim?
sum_y p(x,y)ln [p(x,y) / p(x,*) p(*,y)] =
(define (rim w) (frqobj 'right-wild-mi w))
(define (crim w)
(define (mi s) (frqobj 'pair-mi s))
(fold (lambda (s sum) (+ sum (mi s)))
0.0 (crs 'right-stars w)))
(crim w) = 1.3691988899449898e-6 agrees with rim.
(define (drim w)
(define (mi d) (frqobj 'pair-mi (cra 'get-pair w d)))
(fold (lambda (d sum) (+ sum (mi d)))
0.0 (crs 'right-duals w)))
(drim w) $103 = 1.3691988899449898e-6
(map (lambda (d) (cra 'get-pair w d)) (crs 'right-duals w))
(define (cdrim w)
(define prs (frqobj 'right-wild-freq w))
(define (mi d)
(define s (cra 'get-pair w d))
(define p (frqobj 'pair-freq s))
(define pls (frqobj 'left-wild-freq d))
(* p (l2 (/ p (* prs pls)))))
(fold (lambda (d sum) (+ sum (mi d)))
0.0 (crs 'right-duals w)))
(cdrim w) $104 = 1.4056529360759508e-6 there we are...
(define (cdrim w)
(define prs (frqobj 'right-wild-freq w))
(define (mi d)
(define s (cra 'get-pair w d))
(define p (frqobj 'pair-freq s))
(define pls (frqobj 'left-wild-freq d))
(define hmi (* p (l2 (/ p (* prs pls)))))
(define cmi (frqobj 'pair-mi s))
(format #t "its ~A == ~A ~A" hmi cmi s)
hmi)
(fold (lambda (d sum) (+ sum (mi d)))
0.0 (crs 'right-duals w)))
Ah ha. Some crazy multiplicity thing, again.
set-pair-mi
here'a a bad one:
(Section
(WordNode "The")
(ConnectorSeq
(Connector (WordNode "###LEFT-WALL###") (ConnectorDir "-"))
(Connector (WordNode "Dormez") (ConnectorDir "+"))))
Here's it's dual:
(EvaluationLink
(PredicateNode "*-connector left stars-*")
(WordNode "The")
(Connector (WordNode "###LEFT-WALL###") (ConnectorDir "-"))
(Connector (VariableNode "$connector-word") (ConnectorDir "+")))
(frqobj 'left-wild-logli d) $116 = 10.0013214122192
(frqobj 'right-wild-logli w) $117 = 23.4997343543985
(frqobj 'pair-logli s) $118 = 25.4997343543985
(- (+ 10.0013214122192 23.4997343543985) 25.4997343543985) = 8.001321412219198
(frqobj 'pair-fmi s) 5.27178121177972 <<<<< bad how ??
(cdrim (WordNode "Frollo/Harry"))
Oh, hang on ... silly goose. Cannot hand MI on the sections. Of course.
Since the pairs are not correctly identified.
There are three fmi's:
word-to-conseq
word-to first-con
word-to-second-con
Crap. Pair needs to be:
(CrossLink
(WordNode foo)
(Dual ..))
The duals are missing counts -- well no, they do have marginals...
Rows: 137078 Columns: 18265271
Size: 24213074 non-zero entries == 24M
268594 Connectors == approx 2x words
18265271 == 18M EvaluationLinks == number of duals
6239997 == 6.2M ConnectorSeq
8629163 = 8.6M Sections
(cog-delete-recursive (Variable "$connector-word"))
(cog-delete-recursive (PredicateNode "*-connector left stars-*"))
(cog-delete-recursive (AnyNode "cross-connector word"))
(cog-delete-recursive (AnyNode "cross-connector section"))
(cog-new-ctv 1 0 CNT)))
dropdb en_dj_two_sim
was guile -l x-mtwo.scm
guile -l x-mmore.scm
guile -l x-test.scm
(define hva (make-shape-vec-api))
(define hvs (add-pair-stars hva))
(define hvp (add-support-api hvs))
(define bth (batch-transpose hva))
(hva 'explode-sections)
(store-referers (Predicate "*-word-shape pair-*"))
(bth 'mmt-marginals)
OK, the new system works
xxxxxxxxxxxxxxxxxxxx <<< here
----------------
OK, we don't need all of pair-MI for the cross-connector calculations.
We only need ....
actually we need transpose partials on pca, as well.
so:
1) we don't need freq's. Nice but not necessary.
2) we do need count marginals, from support-compute.
3) we need transpose marginals, but only for mmt.
need total counts.
precomputed add-support-api
add-transpose-compute 'mmt-marginals
----------------
Redo contents of compute-mi.scm:
*
(define cva (make-connector-vec-api))
(define cvs (add-pair-stars cva))
(define cvp (add-support-api cvs))
(define btc (batch-transpose cva))
(btc 'batch-cross)
(cvp 'wild-wild-count)
(define w (car (cvp 'left-basis)))
(define cvt (add-transpose-compute cvs))
(cvt 'mmt-support w) ------ fails when w is a WordClass...
reports ---
print-matrix-summary-report
add-report-api
make-compute-freq calls 'set-size
(rptobj 'set-size left-size right-size cnt)
(rptobj (add-report-api LLOBJ))
(define rpt-obj (add-report-api btc))
(rpt-obj 'left-support)
(define rtc (make-central-compute btc))
(rtc 'cache-left) because ... left-marginals were cmputed
left margs loop over right-basis which is get-col-fun
which are the cache-lefts.
WTF....
The probability of a column y is
P(*,y) = N(*,y)/N(*,*) == left-count
get-right-fun-avg gives averge over columns. i.e. loop over right-basis
FN must take a column .e.g. FN ... could be a left-count
so get-col-fun-avg gives -left-avg-things are average over columns
===========================================================
Kill the unused words --- OK wayyyy too slow for set intersection.
New set intersection idea:
sort both lists via cog-atom-less? and then single-pass
(define pcs (add-pair-stars pca))
(define bas (pcs 'left-basis))
(length bas) ; 137078
(define all-words (cog-get-atoms 'WordNode))
(length all-words) ; 851964
(length no-usar)
714886
(load-atoms-of-type 'WordNode)
(define no-usar (remove XXXXX no super slow n2 algo!!!
(lambda (wrd) (find (lambda (w) (equal? w wrd)) bas))
all-words))
(length no-usar)
(define start-time (current-time))
(for-each cog-delete diff)
(format #t "Elapsed ~A\n" (- (current-time) start-time))
(define start-time (current-time))
(define diff (atoms-subtract all-words bas))
(format #t "Elapsed ~A\n" (- (current-time) start-time))
again ...
----------------------------------------
Elapsed time to load csets: 2993 secs
21651920 atoms, this time.
851704 words.
714626 to be deleted
deletes are super-slow. Why?
DELETE FROM Atoms WHERE uuid =
DELETE FROM Valuations WHERE key = %lu AND atom = %lu;
SELECT uuid FROM Atoms WHERE outgoing @> ARRAY[CAST(%lu AS BIGINT)] LIMIT 1;
124 8434500
select * from atoms where name='regardoient'; is slow ...
19568878
SELECT uuid FROM Atoms WHERE outgoing @> ARRAY[CAST(47851416 AS BIGINT)] LIMIT 1;
is super-slow .. why?
incoming_index is there ...
drop incoming index, it still runs at same terrible speed.
recreate incoming_index ....
after recreating it, it's now fast. So WTF ...!???
would vacuum analyze have fixed it???
select count(*) from atoms;
21651928
=========================================================================
Failed to load cogserver module opencog/modules/libPersistModule.so
=========================================================================
=========================================================================
And again, now for cfive on LXC
guile -v
guile (GNU Guile) 2.2.2.13-98461
time createdb en_dj_cfive -T en_pairs_cfive_mst
real 14m12.700s
guile -l xclean.scm
(ala 'fetch-pairs)
Elapsed time to load ANY-link pairs: 3188 secs
sql-stats: Atomspace holds 58896045 atoms
top: 75.290g == 78947524 KB psaux
(print-matrix-summary-report ala) -- same as for cfive_mst, as expected.
(cog-report-counts)
((ListLink . 29136245) (EvaluationLink . 29136245) (SchemaNode . 1)
(PredicateNode . 8) (AnchorNode . 1) (WordNode . 623548) (AnyNode . 2)
(LinkGrammarRelationshipNode . 1))
(ala 'delete-pairs)
running at 500/second, need 116544.98 seconds == 32 hours! Yow!
Done.
scheme@(guile-user)> (pca 'fetch-pairs)
Elapsed time to load csets: 8680 secs
81224835 atoms = 81M atoms taking 110GB to load.
(print-matrix-summary-report pca) -- same as for cfive_mst, as expected.
(cog-report-counts)
((ListLink . 24169074) (SchemaNode . 1) (PredicateNode . 8)
(AnchorNode . 1) (Connector . 971810) (ConnectorDir . 2) (ConnectorSeq .
23670834) (Section . 31914870) (WordNode . 498239) (AnyNode . 2))
(define all-words (cog-get-atoms 'WordNode))
(length all-words) ; 498239
(load-atoms-of-type 'WordNode)
(define all-words (cog-get-atoms 'WordNode))
(length all-words) ; 623548
(define bas (pca 'left-basis))
(length bas) ; 444903 --- Hmm Fewer. Wonder why.
(define start-time (current-time))
(define diff (atoms-subtract all-words bas))
(format #t "Elapsed ~A\n" (- (current-time) start-time))
(length diff) ; --- 178645
(define start-time (current-time))
(for-each cog-delete diff)
(format #t "Elapsed ~A\n" (- (current-time) start-time))
Ahh -- interesting ... there are words that appear in connectors,
that do not have corresponding disjuncts for them.
There are 498239 words, of which 444903 have disjuncts and 53336
are in connectors only. How many are in connectors, but not in
disjuncts, total?
pg_dump en_dj_cfive > en_dj_cfive.sql
Also VACUUM FULL; done.
And now
createdb en_dj_efive -T en_dj_cfive
(define cra (make-connector-vec-api))
(cra 'fetch-pairs)
Elapsed time to load word sections: 6054 seconds
Elapsed time to load cross-marginals: 0 seconds
67.9GBytes to load 56786433 == 56.7 M atoms
(cra 'make-left-stars)
Elapsed time to compute left-stars: 6054 secs
requires 139G at that time
sql-stats: Atomspace holds 130597079 atoms -- 73810646 == 74M more
(cog-report-counts)
(Connector . 931184)
(ConnectorSeq . 23444293)
(EvaluationLink . 73810644) <<== 3x larger than num connectors!
(Section . 31914870)
(WordNode . 496078)
guile-dj-en>> (batch-all-pair-mi crs)
Start computing the basis
Support: found num left= 496078 num right= 73810644 in 2828 secs
Got up to 175 GB and halted.
----------------
OK, so ... set up the transpose marginals for pseudo-csets:
createdb en_dj_ptwo -T en_dj_mtwo
guile -l x-mtwo.scm
(define btp (batch-transpose psa))
(btp 'fetch-pairs)
Elapsed time to load csets: 3484 secs
top VIRT 31.255g size
(ListLink . 6377076)
(Connector . 268592)
(ConnectorSeq . 6239997)
(Section . 8629163)
(WordNode . 137078)
(btp 'mmt-marginals)
progress?? I assume its recomputing support....
(define bc (make-central-compute psa))
(define pcs (add-support-api psa))
(define pcr (add-report-api psa))
Finished left norm marginals in 35534 secs <==== same as before
works out to 176 per second. Ouch.
Finished left totals in 227 secs
Finished column (left) norm averages in 1042 secs
Done storing 6239997 left-wilds in 4681 secs << == same as before, expected
Finished mmt norm marginals in 790 secs << === great!
Finished mmt totals in 8 secs
Done storing 137078 right-wilds in 114 secs
----------
Now do crosses.
size= 34.293g VIRT top
(define cra (make-connector-vec-api))
(define bcr (batch-transpose cra))
(cra 'fetch-pairs)
(bcr 'mmt-marginals) ; start 12:10 4 July
Elapsed time to compute left-stars: 1005 secs
Finished left norm marginals in 374539 secs <<<== 104 hours
Finished left totals in 283 secs
Finished column (left) norm averages in 1316 secs
Stored 40000 of 18265271 left-wilds in 28 secs (1429 pairs/sec)
Stored 18240000 of 18265271 left-wilds in 30 secs (1333 pairs/sec)
Done storing 18265271 left-wilds in 13195 secs <<<== 3.7 hours
Finished mmt norm marginals in 6859 secs
Finished mmt totals in 4 secs
Stored 40000 of 137078 right-wilds in 25 secs (1600 pairs/sec)
Stored 80000 of 137078 right-wilds in 24 secs (1667 pairs/sec)
Stored 120000 of 137078 right-wilds in 25 secs (1600 pairs/sec)
Done storing 137078 right-wilds in 84 secs
Done computing and saving sum_y N(x,y) N(*,y)
Restart again 1AM 10 July
$ guile -l x-mtwo.scm
scheme@(guile-user)> (cra 'fetch-pairs)
Elapsed time to load word sections: 3034 seconds
Elapsed time to load cross-marginals: 2818 seconds
scheme@(guile-user)> (bcr 'mmt-marginals)
Finished mmt norm marginals in 8589 secs
Finished mmt totals in 4 secs
Stored 40000 of 137078 right-wilds in 7 secs (5714 pairs/sec)
Stored 80000 of 137078 right-wilds in 8 secs (5000 pairs/sec)
Stored 120000 of 137078 right-wilds in 7 secs (5714 pairs/sec)
Done storing 137078 right-wilds in 25 secs
Done computing and saving sum_y N(x,y) N(*,y)
Argh; we need to force the redo of the supports.
(cog-report-counts)
(EvaluationLink . 18265271) -- so they are there as expected.
(scomp-obj 'left-marginals)
(centr-obj 'cache-left)
(store-obj 'store-left-marginals)
(ListLink . 6377077) ; one more
(EvaluationLink . 18265271)
(Connector . 268594) ; two more
(ConnectorSeq . 6239997) ; as before
(Section . 8629163) ; as before
(WordNode . 137078)
(scomp-obj 'left-marginals)
Finished left norm marginals in 393917 secs <== 109 hours
Finished left totals in 444 secs
(centr-obj 'cache-left)
Finished column (left) norm averages in 2174 secs
(store-obj 'store-left-marginals)
Stored 40000 of 18265271 left-wilds in 10 secs (4000 pairs/sec)
Stored 18240000 of 18265271 left-wilds in 7 secs (5714 pairs/sec)
Done storing 18265271 left-wilds in 3516 secs
(bcr 'mmt-marginals)
Finished mmt norm marginals in 11700 secs
Finished mmt totals in 11 secs
Stored 40000 of 137078 right-wilds in 7 secs (5714 pairs/sec)
Stored 80000 of 137078 right-wilds in 8 secs (5000 pairs/sec)
Stored 120000 of 137078 right-wilds in 7 secs (5714 pairs/sec)
Done storing 137078 right-wilds in 25 secs
Done computing and saving sum_y N(x,y) N(*,y)
RAM usage: 50.831g per top
(cog-report-counts)
(ListLink . 137080) <==== hang on, there used to be 6M of these .. whazzup?
(sql-stats)
Atomspace holds 33677206 atoms <<== 1621 bytes/atom
postgres:///en_dj_ptwo
(pca 'fetch-pairs)
Elapsed time to load csets: 2309 secs
(cog-report-counts)
(ListLink . 6514155) <<== Ahh, OK, they were hiding (+ 137078 6377077)
Crap. We also needed to do right-marginals....
(scomp-obj 'right-marginals) ; Start noon-ish 16 July
Finished right norm marginals in 6478 secs
Finished right totals in 14 secs
(centr-obj 'cache-right)
Finished row (right) norm averages in 32 secs
(store-obj 'store-right-marginals)
Done storing 137078 right-wilds in 81 secs
en_dj_ptwo
-----------
English disjuncts and transpose/cross marginals.
Identical to en_dj_mtwo (which was en_pairs_rfive_mtwo but with all
word pairs removed, and and only the disjuncts left). This one adds the
transpose marginals, viz (batch-transpose (make-connector-vec-api)
'mmt-marginals) that are needed for fast entropy-similarities.
STATUS: BAD.
REASON: Holds incorrect word-shape pairs. Wrong implementation.
Also, missing right marginals on cross-connectors, which are
needed for similarity computations. Fixed in en_dj_qtwo below.
DATE: 15 July 2018
en_dj_ptwo=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+-------+------------+-------------
public | atoms | table | linas | 4228 MB |
public | spaces | table | linas | 40 kB |
public | typecodes | table | linas | 56 kB |
public | valuations | table | linas | 8223 MB |
public | values | table | linas | 8192 bytes |
(5 rows)
select count(*) from atoms; <<<< 40054286 == 40M (19M more than mtwo)
select count(*) from valuations; <<<< 76449384 == 76M (18M more than mtwo)
Summary Report for Correlation Matrix Cross-section Words
Left type: WordNode Right Type: EvaluationLink Pair Type: Section
Wildcard: (ListLink (ctv 0 0 47444396)
(AnyNode "cross-connector word")
(AnyNode "cross-connector section")
)
Rows: 137078 Columns: 18265271
Size: 24213074 non-zero entries of 2503766818138 possible
Fraction non-zero: 9.6707E-6 Sparsity (-log_2): 16.658
Total observations: 47444396.0 Avg obs per pair: 1.9595
No MI statistics are present; run compute-mi to get them.
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 1422. n/a
Count (l_1) 2.7318E+4 n/a 19.21 n/a
Length (l_2) 3114. n/a 2.190 n/a
RMS Count 3092. n/a 2.174 n/a
6562246442 Bytes uncompressed == 6.5 GBytes
709796728 Bytes compressed == 710 MBytes
af614f3c448c2a36024cf1899066f63d en_dj_ptwo.sql
8e730f8cf114e940737579dbfc544770 en_dj_ptwo.sql.bz2
en_dj_qtwo
-----------
English disjuncts+cross from rfive-mtwo plus marginals.
Extended from en_dj_mtwo (which was en_pairs_rfive_mtwo but with all
word pairs removed, and and only the disjuncts left). The extensions
are:
* cross-conector structures from (make-connector-vec-api)
* left and right marginals for the pseudo-csets
* left and right marginals for the cross-conector structures
* 'mmt-marginals for both of the above.
STATUS: BAD.
REASON: Holds incorrect word-shape pairs. Wrong implementation.
DATE: 16 July 2018
en_dj_qtwo=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+-------+------------+-------------
public | atoms | table | linas | 4228 MB |
public | spaces | table | linas | 40 kB |
public | typecodes | table | linas | 56 kB |
public | valuations | table | linas | 8223 MB |
public | values | table | linas | 8192 bytes |
(5 rows)
select count(*) from atoms; <<<< 40054291 == 40M (19M more than mtwo)
select count(*) from valuations; <<<< 76586464 == 76M (18M more than mtwo)
Summary Report for Correlation Matrix Cross-section Words
Left type: WordNode Right Type: EvaluationLink Pair Type: Section
Wildcard: (ListLink (ctv 0 0 47444396)
(AnyNode "cross-connector word")
(AnyNode "cross-connector section")
)
Rows: 137078 Columns: 18265271
Size: 24213074 non-zero entries of 2503766818138 possible
Fraction non-zero: 9.6707E-6 Sparsity (-log_2): 16.658
Total observations: 47444396.0 Avg obs per pair: 1.9595
No MI statistics are present; run compute-mi to get them.
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 1422. 1.2051E+5
Count (l_1) 2.7318E+4 2.3690E+5 19.21 1.966
Length (l_2) 3114. 6763. 2.190 5.6123E-2
RMS Count 3092. +nan.0 2.174 +nan.0
6569432244 Bytes uncompressed == 6.5 GBytes
710826943 Bytes compressed == 711 MBytes
7593d2c6bf14216b42da77c8f6024cad en_dj_qtwo.sql
8cf228f8e7b5a80072d60cf87e77a592 en_dj_qtwo.sql.bz2
=========================================================================
Restart again midnight exactly Sun 5 August 2018
dropdb en_dj_two_sim
createdb en_sh_two
time bzcat en_dj_mtwo.sql.bz2 | psql en_sh_two
real 16m18.544s
user 3m55.964s
sys 0m14.484s
store-wildcards
store-right-marginals
en_sh_two
---------
English disjuncts+shapes from rfive-mtwo plus marginals.
Extended from en_dj_mtwo (which was en_pairs_rfive_mtwo but with all
word pairs removed, and and only the disjuncts left). The extensions
are:
* shape structures from (make-shape-api) are NOT in this dataset
(you must make them manually by (hva 'explode-sections) after loading
this dataset.)
* left and right marginals for the pseudo-csets
* left and right marginals for the shape structures (make-shape-api)
* 'mmt-marginals for both of the above.
STATUS: Fair; has the disjunct bug #2252
DATE: 13 August 2018
en_sh_two=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+-------+------------+-------------
public | atoms | table | linas | 4256 MB |
public | spaces | table | linas | 8192 bytes |
public | typecodes | table | linas | 48 kB |
public | valuations | table | linas | 7647 MB |
public | values | table | linas | 8192 bytes |
select count(*) from atoms; 40352376 <<<< 40.3 M (19M more)
select count(*) from valuations; 76884550 <<<< 76.8 M (19M more)
The above does not contain the exploded sections. After explosion
it looks like below.
(cog-report-counts)
(ListLink . 6514154)
(EvaluationLink . 42478345
(Connector . 268594)
(ConnectorSeq . 6239997)
(Section . 8629163)
(WordNode . 137082)
(print-matrix-summary-report hva)
Summary Report for Correlation Matrix Cross-section Words
Left type: WordNode Right Type: EvaluationLink Pair Type: EvaluationLink
Wildcard: (ListLink (ctv 0 0 18489594)
(AnyNode "shape word")
(AnyNode "shape section")
)
Rows: 137075 Columns: 18265271
Size: 24213074 non-zero entries of 2503712022325 possible
Fraction non-zero: 9.6709E-6 Sparsity (-log_2): 16.658
Total observations: 18489594.0 Avg obs per pair: 0.7636
No MI statistics are present; run compute-mi to get them.
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 3568. 1.1179E+5
Count (l_1) 6.9286E+4 8.0072E+4 19.42 .7163
Length (l_2) 7782. 6245. 2.181 5.5866E-2
RMS Count 7760. 6240. 2.175 5.5824E-2
(print-matrix-summary-report pca)
Summary Report for Correlation Matrix Word-Disjunct Pairs (Connector Sets)
Left type: WordNode Right Type: ConnectorSeq Pair Type: Section
Wildcard: (ListLink (ctv 0 0 18489594)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 137078 Columns: 6239997
Size: 8629163 non-zero entries of 855366308766 possible
Fraction non-zero: 1.0088E-5 Sparsity (-log_2): 16.597
Total observations: 18489594.0 Avg obs per pair: 2.1427
Entropy Total: 20.957 Left: 19.143 Right: 9.7111
Total MI: 7.8969
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 145.4 4.8875E+4
Count (l_1) 3440. 2.2958E+5 23.66 4.697
Length (l_2) 1773. 1.3729E+4 12.19 .2809
RMS Count 1732. 1.3696E+4 11.91 .2802
7016483834 Bytes uncompressed == 7.0GBytes
779019716 Bytes compressed == 780 MBytes
782683944c9dcc2e3ce3d7ec7f89dd84 en_sh_two.sql
54680d4ffe4378350184d748b841523a en_sh_two.sql.bz2
=========================================================================
(sql-open "postgres:///en_sh_two?user=linas&password=asdf")
$ guile -l x-mtwo.scm
scheme@(guile-user)> (hva 'fetch-pairs)
Elapsed time to load word sections: 2610 seconds
scheme@(guile-user)> (hva 'explode-sections)
Elapsed time to create shapes: 1523 secs
scheme@(guile-user)> (hst 'store-pairs (cog-incoming-set (Predicate "*-word-shape pair-*")))
Hang on .. that is not really needed ... is it? What's the point?
Just always remember to explode the sections each time before use ....
scheme@(guile-user)> (bhv 'mmt-marginals)
Finished left norm marginals in 376195 secs <<< 104 hours ....
Finished left totals in 269 secs
Finished column (left) norm averages in 1481 secs
Stored 40000 of 18265271 left-wilds in 27 secs (1481 pairs/sec)
Done storing 18265271 left-wilds in 12672 secs
Finished mmt norm marginals in 1621 secs
Finished mmt totals in 4 secs
Stored 40000 of 137075 right-wilds in 23 secs (1739 pairs/sec)
Stored 80000 of 137075 right-wilds in 24 secs (1667 pairs/sec)
Stored 120000 of 137075 right-wilds in 24 secs (1667 pairs/sec)
Done storing 137075 right-wilds in 81 secs
Done computing and saving sum_y N(x,y) N(*,y)
Crapp. The above needs right marginals, too
(define hvc (add-support-compute hvs))
(hvc 'right-marginals)
(define hst (make-store hvs))
(hst 'store-right-marginals)
$ guile -l x-mmore.scm
scheme@(guile-user)> (pca 'fetch-pairs)
Elapsed time to load csets: 2832 secs
scheme@(guile-user)> (btp 'mmt-marginals)
Finished left norm marginals in 35885 secs <<< same as last time.
Finished left totals in 233 secs
Finished column (left) norm averages in 1187 secs
Done storing 6239997 left-wilds in 4504 secs
Finished mmt norm marginals in 800 secs
Finished mmt totals in 4 secs
Done storing 137078 right-wilds in 113 secs
Done computing and saving sum_y N(x,y) N(*,y)
-- dropdb en_dj_etwo
-- createdb en_sh_mi_two -T en_sh_two
Massive MI computations: started 13 aug 20:00 CDT 2018
guile -l x-mcross.scm
(hvs 'fetch-pairs)
(hvs 'explode-sections)
(batch-all-pair-mi hvs)
Elapsed time to load word sections: 3418 seconds
Elapsed time to load word-shape pairs: 53 seconds
Elapsed time to create shapes: 1558 secs
Start computing the basis
Support: found num left= 137075 num right= 18265271 in 881 secs
Finished left norm marginals in 405063 secs <<<== yikes == 112 hours
Finished left totals in 455 secs
Finished right norm marginals in 1229 secs
Finished right totals in 11 secs
Done with wild-card count N(x,*) and N(*,y) in 406758 secs
Total count N(*,*) = 18489593.9998708 = 18489593.99999996
Finished column (left) norm averages in 2167 secs
Finished row (right) norm averages in 32 secs
Going to do individual pair frequencies
Done computing 24213074 pairs in 3646 secs
Start computing log P(*,y)
Done computing 18265271 left-wilds in 598 secs
Done with -log P(*,y), start -log P(x,*)
Done computing 137075 right-wilds in 11 secs
Stored 40000 of 18265271 left-wilds in 31 secs (1290 pairs/sec)
Stored 80000 of 18265271 left-wilds in 29 secs (1379 pairs/sec)
....
Stored 18240000 of 18265271 left-wilds in 16 secs (2500 pairs/sec)
Done storing 18265271 left-wilds in 7334 secs
Stored 40000 of 137075 right-wilds in 27 secs (1481 pairs/sec)
Stored 80000 of 137075 right-wilds in 18 secs (2222 pairs/sec)
Stored 120000 of 137075 right-wilds in 18 secs (2222 pairs/sec)
Done storing 137075 right-wilds in 70 secs
Done computing and saving -log P(x,*) and P(*,y)
Going to do individual pair MI
Done 10000 of 137075 outer loops in 2116 secs, pairs=2286405 (1080.5 pairs/sec)
...
Done 130000 of 137075 outer loops in 1609 secs, pairs=23270250 (788.97 pairs/sec)
Done computing 24213074 pair MI's in 21653 secs << 6 hours
Going to do column and row subtotals
Finished left entropy subtotals in 419208 secs << 116 hours == 5 days
Finished right entropy subtotals in 1459 secs
Finished left MI subtotals in 396071 secs
Finished right MI subtotals in 1369 secs
Going to compute the left, right and total entropy
Done computing totals; start saving wildcards
Stored 40000 of 18265271 left-wilds in 388 secs (103 pairs/sec)
Stored 80000 of 18265271 left-wilds in 327 secs (122 pairs/sec)
...
Stored 18240000 of 18265271 left-wilds in 22 secs (1818 pairs/sec)
Done storing 18265271 left-wilds in 12769 secs
Stored 40000 of 137075 right-wilds in 27 secs (1481 pairs/sec)
Stored 80000 of 137075 right-wilds in 28 secs (1429 pairs/sec)
Stored 120000 of 137075 right-wilds in 29 secs (1379 pairs/sec)
Done storing 137075 right-wilds in 96 secs
Finished with MI computations; this took 354. hours (15 days)
=========================================================================
en_sh_mi_two
------------
English disjuncts+shapes from rfive-mtwo plus marginals plus entropy.
Extended from en_sh_mtwo (which contains marginals for shapes), and
adds the left, right, total entropy and total MI calculations.
(viz MI(w,dj) for all word-disjunct pairs, and all shape-pairs).
So matrix summary report is same as before, but includes entropy.
This requires an additional 24M atoms, and 127M values!! Hefty!
Built as:
(hvs 'fetch-pairs)
(hvs 'explode-sections)
(batch-all-pair-mi hvs)
STATUS: Fair; has the disjunct bug #2252
DATE: 29 August 2018
en_sh_mi_two=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+-------+------------+-------------
public | atoms | table | linas | 6564 MB |
public | spaces | table | linas | 40 kB |
public | typecodes | table | linas | 56 kB |
public | valuations | table | linas | 20 GB |
public | values | table | linas | 8192 bytes |
select count(*) from atoms; -- 64267359 == 64.2M atoms << 24M more!!
select count(*) from valuations; -- 204432721 == 204M << 127M more !!
(print-matrix-summary-report hvs)
Summary Report for Correlation Matrix Cross-section Words
Left type: WordNode Right Type: EvaluationLink Pair Type:
EvaluationLink
Wildcard: (ListLink (ctv 0 0 18489594)
(AnyNode "shape word")
(AnyNode "shape section")
)
Rows: 137075 Columns: 18265271
Size: 24213074 non-zero entries of 2503712022325 possible
Fraction non-zero: 9.6709E-6 Sparsity (-log_2): 16.658
Total observations: 18489594.0 Avg obs per pair: 0.7636
Entropy Total: 21.956 Left: 18.052 Right: 11.391
Total MI: 7.4869
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 3568. 1.1179E+5
Count (l_1) 6.9286E+4 8.0072E+4 19.42 .7163
Length (l_2) 7782. 6245. 2.181 5.5866E-2
RMS Count 7760. 6240. 2.175 5.5824E-2
17558278594 Bytes uncompressed == 17.5 GBytes
1766113632 Bytes compressed == 1.76 GBytes
134616afe42c9e380d00d6cf3d8cf397 en_sh_mi_two.sql
14d28c0678adf704fc81956458879fd0 en_sh_mi_two.sql.bz2
=========================================================================
OK, so where were we? We want
* cosines between words, using dj's
* cosines between words, using cross
* symmetric mi's, for above two.
ALTER DATABASE en_dj_ptwo RENAME TO en_dj_two_sim;
(sql-open "postgres:///en_dj_two_sim?user=linas&password=asdf")
See "similarity.scm" for the rest.
(bco 'batch-compute 12)
Done 10/12 frac=100.0% Time: 2201 Done: 98.5% rate=-0.00 K prs/sec
wtf?
(cog-map-type (lambda (atm) (display atm) #f) 'SimilarityLink)
bcr and bci went too fast ....
(bcr 'compute-similarity (Word "and") (Word "was"))
(Word "in") (Word "a"))
(WordNode "and") (WordNode "to")
(WordNode "to") (WordNode "of")
(cog-incoming-by-type (Word "in") 'SimilarityLink)
(define s (SimilarityLink (Word "in") (Word "of")))
(define kpco (PredicateNode "*-SimKey pseudo-cset Cosine-*"))
(define kpmi (PredicateNode "*-SimKey pseudo-cset MI-*"))
(define kcco (PredicateNode "*-SimKey Cross Cosine-*"))
(define s (SimilarityLink (Word "and") (Word "was")))
(cog-value s kpco)
(cog-value s kpmi)
batch-simlist
(define basis (crs 'left-basis))
(define supp-obj (add-support-api crs))
(supp-obj 'right-count ITEM) returns 0 why... ?
'right-stars returns stars correctly with counts on them (Sections)
support api is looking for support on... get-right-count
on right-wildcard (supp-obj 'right-wildcard s)
Need to set-right-norms set-right-totals
(add-support-compute 'right-marginals
(WordNode "the") (WordNode "of")
to a and . ,
; bci:
(WordNode "I") -- (WordNode "he") = 0.80011 in 146 secs
(WordNode "_") -- (WordNode "he") = -2.6201 in 118 secs
(WordNode "_") -- (WordNode "I") = 0.17530 in 83 secs
(WordNode "a") -- (WordNode "I") = -2.3699 in 195 secs
(WordNode "a") -- (WordNode "_") = -3.2270 in 167 secs
bcr:
(WordNode "I") -- (WordNode "he") = 0.15193 in 249 secs
(WordNode "_") -- (WordNode "he") = 0.00147 in 200 secs
(WordNode "_") -- (WordNode "I") = 0.06457 in 186 secs
(WordNode "a") -- (WordNode "he") = 0.16861 in 344 secs
(WordNode "a") -- (WordNode "_") = 0.00080 in 268 secs
------------------------
conditional-similarity
(define smi (add-symmetric-mi-compute pca))
(add-transpose-api needs to 'mmt-count so that won't work for experiments...
))
(define pfa (add-pair-freq-api psa)) XXX
(define psu (add-support-api psa))
; p(*,d) = (pfa 'left-wild-freq d)
; N(*,d) = (psu 'left-count d)
(psa 'right-duals w)
; return sum_d N(w,d)/N(*,d) = sum_d p(w|d) = sum_d p(w,d)/p(*,d)
(define (cond-marg w)
(fold (lambda (dule sum)
(+ sum (/ (pca 'get-count (pca 'get-pair w dule))
(psu 'left-count dule))))
0.0 (psa 'right-duals w)))
; return sum_d p(w|d) / N(w,*)
(define (focus w) (/ (cond-marg w) (psu 'right-count w)))
; return sum_d N(wa|d) N(wb|d) =
; sum_d (N(wa,d) / N (*,d)) (N(wb,d) / N (*,d))
; sum_d p(wa|d) p(wb|d)
(define (cond-prod wa wb)
(fold (lambda (dule sum)
(define prb (pca 'get-pair wb dule))
(define cnd (psu 'left-count dule))
(define cnds (* cnd cnd))
(if (null? prb) sum
(+ sum (/
(* (pca 'get-count (pca 'get-pair wa dule))
(pca 'get-count prb))
cnds))))
0.0 (psa 'right-duals wa)))
(* (cond-marg wa) (cond-marg wb))))))
(Word "village")(Word "city")
conclusion: not interesting.
------------------------
Disambiguation
(define ppc (add-support-compute (add-tuple-math psa *)))
(define tpc (add-transpose-api psa))
(define (log2 x) (/ (log x) (log 2)))
(define (marg w)
(define tot (tpc 'total-mmt-count))
(define mw (tpc 'mmt-count w))
(- (log2 (/ mw tot))))
(marg (Word "hear")) ; 12.366607806845698
(marg (Word "see")) 9.54234285794451
(marg (Word "book")) 11.961124708605812
(marg (Word "table")) 12.64933707046002
(smi 'mmt-fmi (Word "hear")(Word "book")) ; -6.275136766789916
; Create a fake ambiguous word from w1, w2
; return fmi between fake ambig word and w
(define (fake-amb w)
(define w1 (Word "hear"))
(define w2 (Word "book"))
(define tot (tpc 'total-mmt-count))
(define mw1 (tpc 'mmt-count w1))
(define mw2 (tpc 'mmt-count w2))
(define mfake (+ mw1 mw2))
(define mww (tpc 'mmt-count w))
(define p1 (ppc 'right-count (list w1 w)))
(define p2 (ppc 'right-count (list w2 w)))
(define pfake (+ p1 p2))
(log2 (/ (* pfake tot) (* mfake mww))))
(fake-amb (Word "hear")) ; 3.6272108847937767
(fake-amb (Word "book")) ; 2.94067874079421
(smi 'mmt-fmi (Word "hear")(Word "see")) ; 4.512966599873214
(fake-amb (Word "see")) ; 3.2969178334833145
(smi 'mmt-fmi (Word "book")(Word "table")) 3.322219311913606
(fake-amb (Word "table")) 2.5436144926146653
; Attempt disambig but thsi is bad.
(define (dab-amb w)
(define w1 (Word "hear"))
(define w2 (Word "book"))
(define wr (Word "see"))
(define tot (tpc 'total-mmt-count))
(define mw1 (tpc 'mmt-count w1))
(define mw2 (tpc 'mmt-count w2))
(define mfake (+ mw1 mw2))
(define mwr (tpc 'mmt-count wr))
(define mww (tpc 'mmt-count w))
(define p1 (ppc 'right-count (list w1 w)))
(define p2 (ppc 'right-count (list w2 w)))
(define pfake (+ p1 p2))
(define pr (ppc 'right-count (list wr w)))
(log2 (/ (* (/ pfake pr) tot) (* (/ mfake mwr) mww))))
OK, well, that's not going anywhere.
----------------------------
; let p(wab,d) == (N(wa,d) + N(wb,d))/ (N(wa,*)+N(wb,*))
; let p(wc,d) == N(wc,d)/N(wc,*)
; let q(d) == max(0, p(wab,d)-p(wc,d))
; so that q(d) is "what's left" after subtracting a-component
; ... or should it be r(d) == max(0, p(wc|d)-p(wab|d)) ?
; ....same diff, since both have common factor p(*,d)
;
; Hang on, it should be max(0, p(d|wab)-p(d|wc))
; where p(d|wc) = p(wc,d) / p(wc,*) and p(d|wab) = p(wab,d) / p(wab,*)
; let M(wc,d) == N(wc,*) q(d)
;
; want:
: sum_d N(w,d) M(wc,d)
; sum_d M(wc,d) N(*,d)
;
; Return M(wc,d) as defined above.
; na == N(wa,*) etc
(define (orth wa wb wc d na nb nc)
(define jad (pca 'get-pair wa d))
(define jbd (pca 'get-pair wb d))
(define jcd (pca 'get-pair wc d))
(define nad (if (null? jad) 0.0 (pca 'get-count jad)))
(define nbd (if (null? jbd) 0.0 (pca 'get-count jbd)))
(define ncd (if (null? jcd) 0.0 (pca 'get-count jcd)))
(define nab (+ nad nbd))
(define pab (/ nab (+ na nb)))
(define pc (/ ncd nc))
(define cab (/ pab (+ na nb)))
(define cc (/ pc nc))
(max 0 (- pab pc))) ; this works well
; (max 0 (- cab cc))) ; this works poorly
; (max 0 (- (* nc pab) ncd))))
; (max 0 (* nab (- (* nc pab) ncd))))
; Return sum_d N(w,d) M(wc,d)
(define (dot-ort wa wb wc w)
(define na (psu 'right-count wa))
(define nb (psu 'right-count wb))
(define nc (psu 'right-count wc))
(fold (lambda (dule sum)
(define j (pca 'get-pair w dule))
(if (null? j) sum
(+ sum (* (pca 'get-count j) (orth wa wb wc dule na nb nc)))))
0.0 (psa 'right-duals w)))
; Return sum_d M(wc,d) N(*,d)
(define (net wa wb wc)
(define na (psu 'right-count wa))
(define nb (psu 'right-count wb))
(define nc (psu 'right-count wc))
(fold (lambda (dule sum)
(define j (pca 'get-pair wc dule))
(if (null? j) sum
(+ sum (* (psu 'left-count dule) (orth wa wb wc dule na nb nc)))))
0.0 (psa 'right-duals wc)))
(define (dab-perp w)
(define w1 (Word "hear"))
(define w2 (Word "book"))
; (define wr (Word "see"))
(define wr (Word "hear"))
(define tot (tpc 'total-mmt-count))
(define prod (dot-ort w1 w2 wr w))
(define mwo (net w1 w2 wr))
(define mww (tpc 'mmt-count w))
(define dabmi (log2 (/ (* prod tot) (* mwo mww))))
(define selfmi (smi 'mmt-fmi wr wr))
(- dabmi selfmi))
with (define wr (Word "hear"))
(dab-perp (Word "paper")) 3.809849277692443
(dab-perp (Word "book")) 4.22468023407624
(dab-perp (Word "table")) 3.7956947813077306
(dab-perp (Word "pen")) 3.958633526718385
(dab-perp (Word "hear")) -6.462882425126517
(dab-perp (Word "see")) -6.420968114731355
(dab-perp (Word "watch")) 1.3797091767554486
(dab-perp (Word "listen")) -4.560170727578292
(dab-perp (Word "she")) -0.9675306992488375
(dab-perp (Word "he")) -0.8299348927624219
(dab-perp (Word "them")) -0.4310170287723736
(dab-perp (Word "near")) 0.3093311573173487
(dab-perp (Word "next")) 3.3311035025102687
(dab-perp (Word "of")) -0.6486492223953286
(dab-perp (Word "by")) -3.7167019275587907
looks goood ...
(define wr (Word "see"))
(dab-perp (Word "paper")) 3.681818807262972
(dab-perp (Word "book")) 4.098796757749786
(dab-perp (Word "table")) 3.6691181735260754
(dab-perp (Word "pen")) 3.8310965428109887
(dab-perp (Word "hear")) 1.998378910173595
(dab-perp (Word "see")) -3.045213408538893
(dab-perp (Word "watch")) 1.302745420003867
(dab-perp (Word "listen")) -2.253805464075623
(dab-perp (Word "she")) -0.9810428066327121
(dab-perp (Word "he")) -0.8404681915026306
(dab-perp (Word "near")) 0.19356312119336394
(dab-perp (Word "next")) 3.2039699189813593
(dab-perp (Word "of")) -0.742138952174102
(dab-perp (Word "by")) -3.629486231491182
looks good
(define (dab-perp-helper w1 w2 wr w)
(define tot (tpc 'total-mmt-count))
(define prod (dot-ort w1 w2 wr w))
(define mwo (net w1 w2 wr))
(define mww (tpc 'mmt-count w))
(define dabmi (log2 (/ (* prod tot) (* mwo mww))))
; (define selfmi (smi 'mmt-fmi wr wr))
dabmi)
; (- dabmi selfmi))
(define (dab-perp w)
(define w1 (Word "hear"))
(define w2 (Word "book"))
(define wr (Word "hear"))
(dab-perp-helper w1 w2 wr w))
Hmm slefmi is needed, and the cond is bad
(define (dab-perp w)
(define w1 (Word "fall"))
(define w2 (Word "car"))
(define wr (Word "car"))
(dab-perp-helper w1 w2 wr w))
=========================================================================
Compositionality
(define pco (add-pair-cosine-compute psa))
(cog-get-atoms 'ConceptNode)
(section-vec-sum (Concept "cap-fra") psa 1.0 0.0
(Word "capital") (Word "France"))
(pco 'right-cosine (Word "capital") (Word "France")) 0.04255500982730526
(pco 'right-cosine (Word "Paris") (Word "France")) 0.5596429825328416
(pco 'right-cosine (Word "capital") (Word "Paris")) 0.05447529708635267
(pco 'right-cosine (Concept "cap-fra") (Word "Paris")) ; 0.40030313552977614
(pco 'right-cosine (Concept "cap-fra") (Word "France")) 0.6748163062971411
(pco 'right-cosine (Concept "cap-fra") (Word "capital")) 0.766034032120569
; OK so above is totally lame
(section-vec-sum (Concept "cpfrm") psa 0.0 0.0
(Word "capital") (Word "France"))
(pco 'right-cosine (Concept "cpfrm") (Word "Paris")) 0.3810198495859706
(pco 'right-cosine (Concept "cpfrm") (Word "France")) 0.35398467437926406
(pco 'right-cosine (Concept "cpfrm") (Word "capital")) 0.16742518929899683
; OK, so that's interesting! similarity to the two components
is much much lower. But similarity to "paris" is about the same,
and is now the highest of the three numbers.
(pco 'right-cosine (Concept "cpfrm") (Word "Madrid")) 0
(length (cog-incoming-by-type (Word "Madrid") 'Section)) 3
(pco 'right-cosine (Concept "cpfrm") (Word "Berlin")) 0
(length (cog-incoming-by-type (Word "Berlin") 'Section)) 16
(pco 'right-cosine (Concept "cpfrm") (Word "London")) 0.5451708663713439
(length (cog-incoming-by-type (Word "London") 'Section)) 911
(length (cog-incoming-by-type (Word "Paris") 'Section)) 248
(length (cog-incoming-by-type (Concept "cpfrm") 'Section)) 6
(length (cog-incoming-by-type (Word "France") 'Section)) 267
(length (cog-incoming-by-type (Word "capital") 'Section)) 300
The 6 sections on "cpfrm" are totally lame --
They are prepositions, punctuation.
whatever compistionality there mayb be, not enough data in this dataset.
Probably need to perform clustering first.
(pmi 'mmt-fmi (Word "France")(Word "capital")) 0.634246152887463
(pmi 'mmt-fmi (Word "France")(Word "Paris")) 5.458600057659794
(pmi 'mmt-fmi (Word "Paris")(Word "capital")) 1.0135814069906335
OK, not too interesting.
=========================================================================
cross-connector similarities.
(define ara (make-shape-vec-api))
(define ars (add-pair-stars ara))
(define art (add-transpose-api ars))
(define aco (add-pair-cosine-compute art))
(define acr
(batch-similarity art #f "Cross Cosine-*" 0.0
(lambda (wa wb) (aco 'right-cosine wa wb))))
(map (lambda (n) (acr 'batch-compute n))
(list-tabulate 100 (lambda (n) (+ n 38))))
(map (lambda (n) (acr 'batch-compute n)) (iota 100 38 1))
(map (lambda (n) (bcr 'batch-compute n)) (iota 100 38 1))
(map (lambda (n) (bcr 'batch-compute n)) (iota 100 140 2))
(define fra (make-shape-vec-api))
(define frs (add-pair-stars fra))
(define frt (add-transpose-api frs))
(define fco (add-pair-cosine-compute frt))
(define fcr
(batch-similarity frt #f "Cross Cosine-*" 0.0
(lambda (wa wb) (fco 'right-cosine wa wb))))
(use-modules (statprof))
(statprof (lambda ()
(fcr 'compute-similarity (Word "this")(Word "that")) #f))
(bci 'batch-compute 22)
(bci 'batch-compute 24)
(bci 'batch-compute 26)
(map (lambda (n) (bci 'batch-compute n)) (iota 30 52 4))
(map (lambda (n) (bci 'batch-compute n)) (iota 222 170 5))
(load "/tmp/similarity-api.scm")
(define bco
(qbatch-similarity pta #f "pseudo-cset Cosine-*" 0.0
(lambda (wa wb) (pco 'right-cosine wa wb))))
(map (lambda (n) (bco 'batch-compute n))
(list-tabulate 10 (lambda (n) (+ (* n 30) 370))))
(map (lambda (n) (bco 'batch-compute n)) (iota 4 550 30))
co:
509.1 657.0 567.5 551.7 528.8 645.8 632.7 501.0 525.4 546.8 697.3
348.4 258.8 312.3 253.7 283.6 279.9 272.6 261.8 276.3 348.6 288.3 249.2 320.3
win 1 co - 20 to 40 now at 31
win 2 co - 144 to 440 now at 440 Done!
win 2 mi - 144 to 440
win 3 mi - 17 to 58 now at 42
crash:
win 1: (bcr 'batch-compute 34) and more
win 2:
(map (lambda (n) (bmi 'batch-compute n))
(list-tabulate 10 (lambda (n) (+ (* n 30) 370))))
win3: died at 44, start 45 next.
FATAL: terminating connection due to administrator command
FATAL: terminating connection due to administrator command
terminate called after throwing an instance of
'opencog::RuntimeException'
what(): Failed to execute!
(/home/linas/src/novamente/src/atomspace-git/openc
og/persist/sql/multi-driver/ll-pg-cxx.cc:113)
Aborted
Fuck postgres stabbed me!
1) rebuild/install
2) guile -l x-mtwo.scm
3) similarity.scm by hand
cog-count-atoms 'SimilarityLink)
94954
(sqrt (* 2 (- (- 94954 (* 33 16)) (* 44 22))))
432.3378308684078
oh nooo major loss
(cra 'fetch-pairs)
Elapsed time to load word sections: 3412 seconds
Elapsed time to load cross-marginals: 2840 seconds
(pca 'fetch-pairs)
Elapsed time to load csets: 2285 secs
(define bmi
(qbatch-similarity pta #f "pseudo-cset MI-*" -999.0
(lambda (wa wb) (pmi 'mmt-fmi wa wb))))
(map (lambda (n) (bmi 'batch-compute n))
(list-tabulate 16 (lambda (n) (+ (* n 30) 190))))
(map (lambda (n) (bmi 'batch-compute n))
(list-tabulate 16 (lambda (n) (+ (* n 40) 640))))
14167
15013
(bci 'batch-compute 144)
binned-sims-all.eps is missing
(define all-sims '())
(cog-map-type
(lambda (ato)
(define val (cog-value ato cos-key))
(if (not (null? val))
(set! all-sims (cons ato all-sims)))
#f)
'SimilarityLink)
7421 Sep 1 2017 en-dj/binned-sims-all.dat
what is this?
terminate called after throwing an instance of 'opencog::RuntimeException'
what(): Failed to execute!
(/home/linas/src/novamente/src/atomspace-git/opencog/persist/sql/multi-driver/ll-pg-cxx.cc:113)
Aborted
[2018-07-25 05:32:07:048] [WARN] PQresult message:
[2018-07-25 05:32:07:048] [WARN] PQ query was: BEGIN;
6: exceptions.cc:142 opencog::RuntimeException::RuntimeException(char const*,
char const*, ...)
7: ll-pg-cxx.cc:47 LLPGConnection::LLPGConnection(char const*)
8: gthr-default.h:747 __gthread_mutex_lock()
9: SQLResponse.h:305
opencog::SQLAtomStorage::Response::get_value_column_cb(char const*, char const*)
10: stl_tree.h:1636 std::_Rb_tree<unsigned long, unsigned long,
std::_Identity<unsigned long>, std::less<unsigned long>, std::allocator<unsigned
long> >::_M_erase(std::_Rb_tree_node<unsigned long>*)
----
[2018-07-25 12:13:03:800] [WARN] PQresult message: ERROR: duplicate key value
violates unique constraint "valuations_key_atom_key"
DETAIL: Key (key, atom)=(104758323, 105104585) already exists.
[2018-07-25 12:13:03:800] [WARN] PQ query was: INSERT INTO Valuations (key, atom,
type, floatvalue) VALUES (104758323, 105104585, 1, '{ 1.11100380697404}');
[2018-07-25 12:13:04:165] [WARN] PQresult message: ERROR: current transaction is
aborted, commands ignored until end of transaction block
[2018-07-25 12:13:04:165] [WARN] PQ query was: BEGIN;
PSQLException
SQLAtomStorage::
try
catch (const SilentException& ex)
(sql-open "postgres:///langtest?user=linas&password=asdf")
throw opencog::RuntimeException(TRACE_INFO,
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE pid <> pg_backend_pid() AND datname = 'langtest';
because its in a diffrent thread ....
connected
kill_data
extract_callback
Handle getNode(Type, const char *);
Handle getLink(Type, const HandleSeq&);
void getIncomingSet(AtomTable&, const Handle&);
void getIncomingByType(AtomTable&, const Handle&, Type t);
void getValuations(AtomTable&, const Handle&, bool get_all);
void storeAtom(const Handle&, bool synchronous = false);
void removeAtom(const Handle&, bool recursive);
void loadType(AtomTable&, Type);
void flushStoreQueue();
// Large-scale loads and saves
void load(AtomTable &); // Load entire contents of DB
void store(const AtomTable &); // Store entire contents of AtomTable
bcr:
Done 80/87 Frac=1.801% Time: 5462 Done: 99.4% Rate=0.021 prs/sec (48.70 sec/pr)
(map (lambda (n) (bcr 'batch-compute n)) (iota 100 88 2)) ; goes to 288
bmi:
Done 10/148 Frac=2.526% Time: 6786 Done: 13.1% Rate=0.005 prs/sec (188.5 sec/pr)
(map (lambda (n) (bci 'batch-compute n)) (iota 70 150 5)) ; goes to 290
(map (lambda (n) (bci 'batch-compute n)) (iota 17 205 5)) ; goes to 290
co:
Done 920/920 Frac=8.153% Time: 15516 Done: 100.% Rate=6.429 prs/sec (0.156 sec/pr)
OK, so got to 920.... stop there.
(bmi 'batch-compute 920) --- done cleanly.
(map (lambda (n) (bco 'batch-compute n)) (iota 7 640 40))
(map (lambda (n) (bco 'batch-compute n)) (iota 6 680 40))
(map (lambda (n) (bco 'batch-compute n)) (iota 5 720 40))
bmi (normal disjuncts) done to 1536
cos on normal disjuncts ...
bmi on shapes
=========================================================================
OK, so now redo the graphs.
good-sims and ranked-sims in disjunct-stats.scm
(define good-sims
(sim-cosine (Similarity (Word "this")(Word "that")))
section 4.1 748
wtf
(SimilarityLink (WordNode "Your") (WordNode "indeed"))
(bmi 'compute-similarity (WordNode "hour") (WordNode "Let"))
ahh its -infity
(FloatValue -inf)
(define (prt-sim sim port)
(format port "~A '~A .. ~A'\n" (sim-cosine sim)
(cog-name (gar sim)) (cog-name (gdr sim))))
(let ((outport (open-file "/tmp/ranked-dj-mi-sims.dat" "w")))
(define cnt 0)
(for-each (lambda (sim)
(set! cnt (+ cnt 1))
(format outport "~A " cnt)
(prt-sim sim outport))
ranked-sims)
(close outport))
=========================================================================
(statprof (lambda ()
(fcr 'compute-similarity (Word "this")(Word "that")) #f))
% cumulative self
time seconds seconds procedure
39.78 30.06 30.06 cog-link
11.86 8.96 8.96 cog-outgoing-atom
10.93 8.26 8.26 cog-outgoing-set
5.63 13.41 4.26 opencog/nlp/learn/connector-vec.scm:135:24:not-var?
4.63 3.50 3.50 cog-handle
3.36 2.54 2.54 opencog/matrix/transpose.scm:132:8
3.28 2.48 2.48 cog-count
2.61 1.97 1.97 opencog/matrix/object-api.scm:513:24
2.44 10.61 1.84 srfi/srfi-1.scm:729:0:take-while
1.93 50.14 1.46 opencog/nlp/learn/connector-vec.scm:131:16:get-pair
1.60 6840244.56 1.21 srfi/srfi-1.scm:590:5:map1
1.51 1.14 1.14 opencog/base/utilities.scm:134:0:gar
1.35 4.51 1.02 opencog/base/atom-cache.scm:84:8:atom-hash
1.18 5.78 0.89 srfi/srfi-1.scm:760:0:drop-while
1.01 0.95 0.76 srfi/srfi-1.scm:586:2:map
1.01 0.76 0.76 opencog/nlp/learn/connector-vec.scm:362:16
0.84 0.64 0.64 equal?
0.50 0.38 0.38 *
0.42 0.32 0.32 opencog/base/atom-cache.scm:86:22
0.34 6.23 0.25 srfi/srfi-1.scm:458:2:fold
0.34 3.11 0.25 opencog/matrix/fold-api.scm:233:16:get-func-count
0.34 0.25 0.25 opencog/matrix/fold-api.scm:226:32
0.34 0.25 0.25 procedure?
0.34 0.25 0.25 opencog/base/utilities.scm:142:0:gdr
0.34 0.25 0.25 opencog/base/atom-cache.scm:85:8:atom-assoc
0.34 0.25 0.25 opencog/matrix/fold-api.scm:186:32
0.25 0.19 0.19 list?
0.25 0.19 0.19 opencog/base/atom-cache.scm:93:40
0.25 0.19 0.19 opencog/matrix/fold-api.scm:251:16
0.17 2.29 0.13 opencog/matrix/support.scm:308:40
0.17 0.13 0.13 cog-atomspace-clear
0.17 0.13 0.13 opencog-extension
0.17 0.13 0.13 opencog/matrix/fold-api.scm:236:40
0.08 5.66 0.06 hashx-set!
0.08 0.25 0.06 opencog/base/atom-cache.scm:88:8
0.08 0.06 0.06 cog-new-link
0.08 0.06 0.06 opencog/nlp/learn/connector-vec.scm:124:16:get-count
0.00 75.57 0.00 ice-9/eval.scm:618:6
0.00 75.57 0.00 opencog/matrix/similarity-api.scm:193:16:compute-sim
0.00 75.50 0.00 opencog/matrix/cosine.scm:128:16:compute-right-cosine
0.00 68.89 0.00 opencog/matrix/support.scm:300:16:sum-right-count
0.00 11.44 0.00 srfi/srfi-1.scm:634:2:for-each
0.00 5.91 0.00 opencog/matrix/fold-api.scm:224:16:right-star-union
0.00 3.69 0.00 opencog/matrix/support.scm:291:32
0.00 2.73 0.00 opencog/matrix/support.scm:319:16:sum-right-length
0.00 0.83 0.00 find
0.00 0.76 0.00 ice-9/boot-9.scm:834:2:with-throw-handler
0.00 0.38 0.00 opencog/matrix/object-api.scm:387:32
0.00 0.38 0.00 opencog/matrix/object-api.scm:404:32
0.00 0.38 0.00 opencog/matrix/object-api.scm:418:16:do-run-query
0.00 0.19 0.00 hash-for-each-handle
---
Sample count: 1189
Total time: 75.565334546 seconds (128.369109622 seconds in GC)
#f
GC Warning: Repeated allocation of very large block (appr. size 536875008):
May lead to memory leak and poor performance
wtf!? o profiling
=========================================================================
Restart the similarities, this time on en_sh_two_sim
iota: nsteps start stepsz
regular disjuncts mi going to 70+14*72 == 1078
now to (+ 1024 (* 8 64)) == 1536
regular disjuncts cos going to 10 + 59*25 = 1485 .. done
(bco 'batch-compute 1536)
cross mi to ... 571 + 50*20 = 1571
(for-each (lambda (n) (mib 'batch-compute n)) (iota 50 571 21))
; below got to 30 + 99*14 = 1416
(map (lambda (n) (mid 'batch-compute n)) (iota 100 30 14))
; this should get to 1536: ... done actually, got to 1591
(for-each (lambda (n) (mid 'batch-compute n)) (iota 7 1416 20))
So it looks like en_sh_two_sim contains both cosine and entropic similarities,
both for disjunct vectors, and for shapes. (well, no cosines for shapes...)
=========================================================================
=========================================================================
=========================================================================
NEW TODO LIST 2018
Authoring TODO:
connector-sets.lyx:
*) stop saying its logli, call it energy instead.
*) on page 13, number-of-words == density of states.
*) explain that density of states is a sum over delta functions
*) Use curly E for energy
*) page 16 change 400+ to >400
*) Compute the smoothed energy P(E) and graph that
*) smoothed energy is from this:
Zipf’s Law Arises Naturally When There Are
Underlying, Unobserved Variables
Laurence Aitchison, Nicola Corradi, Peter E. Latha
*) what is heat capacity?
*) PEEV from above article vs. number of letters in the word.
why is this interesting?
Its not but it can maybe discriminate between utility
of different latent variables in various cases???
*) besides word-legnth what other obvious latent variables are there?
how are these discovered?
*) page 23 show MI vs. energy. Same for page 25 ff
*) page 28: number of connectors vs. MI
*) number of connectors vs energy.
(i.e. some hope that these have bad MI, and thus would not be used).
*) page 33ff: cosine similarity vs. energy of the two words.
--------------------------------------------------
* Finish elimination of 'left-element, 'right-element in filters.
-- Copy en_sh_two to en_sh_two_sim and start running cosines,
entropies, etc. for the top 1K's for both disjuncts and for
cross connectors.
- 13 August 2018 - In process.
-- start old-style MI cross-marginal in en_sh_mi_two
- 13 August 2018 - In process.
- 28 August 2018 - Done, backed-up.
-- Explain in the text, what this is (these are the shapes)
-- there are C+1 MI's:
* word to connector-seq
* word-to-first-connector
* word-to-second-connector
* ...
* word to C'th connector. Dohhh.
And distinct marginals for each.
-- show graph distribution of mi of word-dj pairs in mtwo
What scripts would these be?
-- start cross-marginals in dj_cfive Uh NOPE.
-- in parallel, experiments:
(1a) Show multiple graphs, of cosines, MI, etc.
depending on word-frequency
(1b) scatterplot for cosine vs entropy (Done)
(2a) graphs, for the crosses, too
(3) Scatterplot of pseudo-cset MI vs. cross-MI.
Done, need to add words for this. XXX done where?
(4) show distribution of the sums, above.
While waiting for above, write code for:
* Wait -- is fetch-pairs of pseudo-csets also getting WordClassNodes??
Yes, it is. Why?
* Merging crosses. Do it same way as for pseudo-disjuncts,
just with independent parameter.
`merge-disambig` uses cosines to do fractionals.
Should this use MI in some way?
* The cosines do need the supports, right? (does it actually
use them, or does it ignore them?)
* Also: which cosines: the batch cosines? what do these use?
Merging minor tasks DONE
------------------------
* right-stars (of a word) are sections of any shape with the word in
The 'left-element, 'right-element concept is borken.
left-stars (of a shape) returns sections of fixed shape, but all
words in that shape.
L-ATOMS are words
R-ATOMS are the Eval-predsicle shape
so the 'right-element of a right-star is a shape ... which is unique
only if the word appeared in it just once; otherwise its ambiguous.
So implement 'left-dual and 'right-dual, instead. DONE!
* Finish symmetric MI in atomspace. DONE
* Do duals in the cross-connectors. DONE.
* Computing cosines and entropies for crosses.
How does cosine work of the pseudos??? It works cause cosine
ignores that it's being done on a WordClass... DONE (untested)
* remove words in en_dj_mtwo not in marginals. IN-PROG DONE
* word-pairs.pdf in the readings directory ...
where is the source for this? Is it in git ????? Ughh No.
Mass copy from 2009 fractal directory. scripts are lost.
Create pull req DONE
* save en_dj_mtwo before starting the cross-marginals. DONE
* pair-removal in mfive; this should fit!? Yes, IN PROGRESS DONE
* make-compute-count is dual to add-pair-count-api
Both should be made obsolete.... DONE
* Redo compute-total-count to be one-sided. DONE
* Transpose-marginals for en_dj_ptwo (pseudo-csets) DONE
* Create report for transpose marginals. Why is this interesting? not done.
* Start transpose-marginals for crossed in en_dj_ptwo DONE
* SQL dump en_dj_ptwo -- a dataset with pseudos and with crosses
and with transpose-marginals for both. DONE
* Fix bug re cross-connector MI's -- DONE
* products of conditional probabilities. DONE NO GOOD
* XXX mmt-marginals is using the support obj, but all it needs
are the marginal counts .. wtf I'm confused.
supports are done with add-support-compute
well, it uses the supports, because it does a support variant.
Ohh, both left and right supports were needed. DONE
=========================================================================
Coverage Quantification:
------------------------
Cite:
A Closer Look at Skip-gram Modelling
David Guthrie, Ben Allison, Wei Liu, Louise Guthrie, Yorick Wilks
LREC 2006
Above uses a training set, and then attempts to parse the test set to
obtain coverage. So, here: statistically:
1) given a dict, what fraction of the test set is parseable?
2) earlier question: what portion was parsable by MST Any skipped words?
2a) What fraction of MST clique has zero counts on the training data?
=========================================================================
=========================================================================
OK-ish misses the mark:
arxiv 1703.08314v1 coecke Interacting conceptual spaces
crappier:
Symbolic, Distributed and Distributional Representations for Natural
Language Processing in the Era of Deep Learning: a Survey
arxiv 1702.00764v1
=========================================================================
Bug list Augsut 2018
Crazy idea: iterate; the places we end up at should be the dependents
since everything points at them, and they never point away.
Bugs: relex is returning incorrect disjuncts because LG is reporting the
multi-connectors only ... with the incorrect mutiplicities.
??? Is this still an issue?
Bugs: using ListLink for the partial sums on csets... why???
(ListLink (AnyNode "cset-word") (ConnectorSeq...))
instead of (Section ...)
Bugs: graphing takes too long.
Bugs: follow up on the guile hang bug - its a string parsing thing.
Bugs: while mst parsing, guile goes GC-crazy after a while, and
never recovers. (doesn't happen with shorter dist links.)
Items: create LG feederbacker
Bugs: cleanup embedded dashes in en DONE
-- save en_pairs_r again DONE
-- should probably recompute pair-MI after above,
-- maybe recompute mst?? (since it was so quick!?)
Bugs: sentences with more than 254 words....
Bugs: make postgres run nice. half-done, w/ renice.
What more is there to do?? More automation?
Bugs: lxc containers not shutting down correctly during power loss
Partly done - rc.local.shutdown rc-local-shutdown.service
rc.lxc.shutdown Does the README describe this ??
Bugs/feat: no way to compute marginals incrementally, if the DB does
not fit in RAM.
TODO List
=========
* Count single-word probabilities.
* Redo all the word-pair graphs and charts from wayyy back when.
* Resume work on morphology. (French)
* Do the "Coverage Quantification" task, above.
DONE
----
* Make sure that link-parser fully randomizes linkage choices for
long sentences. Done. See the rand_state and predictable_rand
flags in version 4.8.3.
* Need a multi-language sentence splitter; the maxent splitter only
works for English sentences. Probably something simple will do ...
Done: copied one from moses-smt and put it in the run subdirectory
* Raw psql seems not to be utf8. Were the tables created wrong,
or is this a client bug?
Fixed: turned out to be a guile-related bug.
* Handle database write asynchronously: have a thread collect
up the atoms, and write them. Maybe even multiple threads.
This is OK due to the atomptr design.
Done. Not just one, but multiple writers.
* Can we start a guile thread for each incoming sentence?
Threading will require more subtle sentence cleanup.
No. The main guile bug, opened 5 years ago, is still not fixed.
* Table counts need to be more than 32-bit. Looks like the
any-language goes hog-wild and creates huge counts for just
one sentence ... is this desired? Anyway, they'll overflow...
Done. atoms.sql now uses bigint 64-bit uuids.
* Investigate crash: while writing truth value, stv_confidence
was pure virtual (circa line 920 of AtomStorage.cc) There's some
kind of smart-pointer race. See "Crash" section below.
Done. Ran for a month without issues.
* Fix crash at linkage_set_domain_names+0xe0 Done. Need LG 5.4.0
* Create OO system for generic pair statistics. Done.
* Reduce num samples for 999 to 16. Done.
* Fix (WordNode "…!ANY-PUNCT") Done.
* Rework pipeline to use literary English sources. DONE.
New scripts in the `download` directory.
* Prune low observation counts. Well, no that is a bad idea.
However, the (opencog matrix) code now includes data filters.
* Create an LXC container for public use. DONE.
But not updated.... Well no one is interested in this so punt.
* Implement sql-delete. DONE.
=========================================================================
1 June 2019
-----------
Restart. Where were we?
LXC container learn-en:
\l
en_dj_efive 72 GB failed cross-connector marginal attempt... deleted
en_pairs_cfive_class 90 GB mystery-junk ... deleted .. oh, this was the block-5x5
experiment from the LyX diary .. oops. Should have saved
it?
en_pairs_cfive 16 GB ANY-pairs parse tranche 1-5, infix punct. removed
en_pairs_cfive_mst 72 GB MST disjuncts from tranche 1-5, and marginals.
en_dj_cfive 47 GB Disjuncts + dj-marginals only, no pairs.
LXC container cfives are cleaned-up versions of rfives.
TODO: copy en_dj_cfive, make the entropies, and then cluster.
to make shape entropies:
-- createdb en_sh_cfive -T en_dj_cfive
(i/o bound - 40% CPU but 115MBytes/sec write speed)
-- cross-en.conf
-- guile -l cross.scm
-- cross-shells.sh
-- (hvs 'fetch-pairs)
top: 550% cpu on guile, 4x60% on postgres
more than half of CPU in kernel, not user! kernel-data-transfer-bound
rate = 11.5K atoms/sec dropping over time to 10.3 avg
Elapsed time to load word sections: 5753 seconds
Atoms - 56786428= 57M using ... 68 GBytes RAM
Caution: fetch-pairs also does the explode.
(explode bopped it up to 133 GB RAM and then started hitting swap.)
Seems that postgres was using another 50+ GB ...
I killed it - watching it swap is pointless. This won't fly.
Some kind of incremental design is needed.
-- dropdb en_sh_cfive
Now try MI-sims on LXC
-- createdb en_midj_cfive -T en_dj_cfive -- 5 minutes
-- midj-shells.sh
-- (define psa stars on dj's as usual)
-- (psa 'fetch-pairs) -- Elapsed time to load csets: 6951 secs
-- (define btr (batch-transpose psa))
-- (btr 'mmt-marginals)
Finished mmt norm marginals in 16919 secs
Finished mmt totals in 96 secs
Done storing 444903 right-wilds in 695 secs
Done storing 23444293 left-wilds in 40798 secs (11.3 hours)
Done computing and saving sum_y N(x,y) N(*,y)
-- How many are done?
(define psn (add-support-api psa))
(fold (lambda (ITEM sum)
(if (< 0 (psn 'left-support ITEM)) (+ sum 1) sum))
0 (psa 'right-basis))
en_midj_cfive
-------------
Identical to en_dj_cfive, together with the MM^T marginals (needed for
working with word-vector to word-vector MI). Just like en_dj_cfive,
this does not have any word-pairs in it. Supposedly it contains the
pseudo-dj marginals. It does not contain the cross-connector marginals;
it appears that computing these is prohibitive, at this time.
STATUS: Never created, due to bugs... Keys are getting clobbered...
see below ...
DATE: 3 June 2019
en_midj_cfive=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 7453 MB |
public | spaces | table | ubuntu | 8192 bytes |
public | typecodes | table | ubuntu | 24 kB |
public | valuations | table | ubuntu | 28 GB |
public | values | table | ubuntu | 8192 bytes |
(5 rows)
select count(*) from atoms; <<<< 81241109 == 81.2 M Atoms
(3 more than before)
select count(*) from valuations; <<<< 217533128 == 217.5 M Valuations
(0.5 M more)
==========
(define pcr (add-report-api psa))
(pcr 'left-dim)
(pca 'wild-wild)
(fetch-atom (pca 'wild-wild)) WTF
(ListLink (ctv 0 0 0) (AnyNode "cset-word") (AnyNode "cset-disjunct"))
psql en_midj_cfive
select * from atoms where name="cset-word"; 125258427 125
cset-disjunct 125258428 125
select * from atoms where outgoing='{125258427,125258428}'; 125258429 21
select * from valuations where atom=125258429;
key | atom | type | floatvalue |
-----------+-----------+------+-------------------------+-------------+-----------
1 | 125258429 | 7 | {0,0,0} | *-TruthValueKey-*
61461575 | 125258429 | 1 | {0,0,0} | *-Dimension Key-*
64841127 | 125258429 | 1 | {21.136,10.112,23.092} | *-Total Entropy Key-*
64841129 | 125258429 | 1 | {NaN,NaN,NaN,NaN} | *-Right Norm Key-*
64841130 | 125258429 | 1 | {8.1569044699464} | *-Total MI Key-*
64841131 | 125258429 | 1 | {NaN,NaN,NaN,NaN} | *-Left Norm Key-*
150943106 | 125258429 | 1 | {0,0} | *-Left Total Key-*
150943107 | 125258429 | 1 | {31914870,549827672543} | *-MM^T Product Key-*
150943108 | 125258429 | 1 | {0,0} | *-Right Total Key-*
So the keys are there (the wild-atom got written) but contain invalid values.
Well, en_dj_cfive is healthier:
key | atom | type | floatvalue |
----------+-----------+------+-------------------------------------+-----------
1 | 125258429 | 7 | {0,0,69447009} | *-TruthValueKey-*
61461575 | 125258429 | 1 | {444903,23444293,31914870} | *-Dimension Key-*
64841127 | 125258429 | 1 | {21.136,10.112,23.092} | *-Total Entropy Key-*
64841129 | 125258429 | 1 | {367.5,7917.2,2396.7,2379.0} | *-Right Norm Key-*
64841130 | 125258429 | 1 | {8.1569044699464} | *-Total MI Key-*
64841131 | 125258429 | 1 | {227416.0,944297.6,24549.3,24498.6} | *-Left Norm Key-*
So, -- tv got clobbered
dimension key clobbered
right norm clobbered
left norm clobbered.
Why?
Answer: because very old stale scheme code was being loaded from
a stale old bogus install location :-(
(for-each (lambda (x) (format #t "~A ~A\n" x (cog-value (pca 'wild-wild) x)))
(cog-keys (pca 'wild-wild)))
===================================
Try again. This time, cut first.
-- (sql-open "postgres:///en_dj_cfive")
(psa 'fetch-pairs)
Elapsed time to load csets: 6321 secs
Atomspace holds 81224843 atoms
(sql-close)
(psa 'left-basis-size) ;; 444903
(psa 'right-basis-size) ;; 23444293
(print-matrix-summary-report psa) ;; as expected.
createdb en_micro_dj_cfive
cat atom.sql | psql en_micro_dj_cfive
COMMENT ON DATABASE en_micro_dj_cfive IS 'en_cfive Sections (only) with word-count > 500; section-count > 10';
(sql-open "postgres:///en_micro_dj_cfive")
(define fsa (add-subtotal-filter psa 500 10 10 #f))
(fsa 'left-basis-size) ;; 8608 ; whoa huge cut!
(fsa 'right-basis-size) ;; 361894 ; also a big cut.
(define fso (make-store fsa))
(fso 'store-all-elts) ; Marginals are not to be stored.
Done storing 392715 pairs in 451 secs
(sql-open "postgres:///en_micro_cfive")
(batch-all-pair-mi psa)
Support: found num left= 10813 num right= 187063 in 0 secs
Finished left norm marginals in 139 secs
Done with wild-card count N(x,*) and N(*,y) in 159 secs
Total count N(*,*) = 20782007.0 = 20782007.0
Done computing 392715 pairs in 36 secs
Done storing 187063 left-wilds in 2729 secs
Done computing 392715 pair MI's in 206 secs
Done storing 187063 left-wilds in 103 secs
Finished with MI computations; this took 0.99 hours
(print-matrix-summary-report psa)
Fraction non-zero: 1.9415E-4 Sparsity (-log_2): 12.331
Total observations: 20782007.0 Avg obs per pair: 52.919
Entropy Total: 16.564 Left: 13.788 Right: 7.7620
Total MI: 4.9867
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 121.4 7387.
Count (l_1) 2.2239E+4 4.8439E+5 183.1 65.57
Length (l_2) 7511. 3.4794E+4 61.85 4.710
RMS Count 7263. 3.4445E+4 59.80 4.663
-- (define btr (batch-transpose psa))
-- (btr 'mmt-marginals)
Done storing 187063 left-wilds in 113 secs
Done computing and saving sum_y N(x,y) N(*,y)
Atomspace holds 989445 atoms
sql-stats: 989440 of 989440 reserved uuids used
So what are the five missing atoms?
(cog-report-counts)
((PredicateNode . 14) (ListLink . 197877) (TypeNode . 2) (AnchorNode .
1) (Connector . 125223) (ConnectorDir . 2) (ConnectorSeq . 187063)
(Section . 392715) (SchemaNode . 1) (WordNode . 86545) (AnyNode . 2))
select count(*) from atoms where type=71;
en_micro_cfive=> select * from atoms where type=71;
uuid | space | type | height | name | outgoing
--------+-------+------+--------+-----------------------+----------
791557 | 1 | 71 | 0 | *-Dimension Key-* | y
989438 | 1 | 71 | 0 | *-Entropy Key-* | y
73 | 1 | 71 | 0 | *-FrequencyKey-* | y
791569 | 1 | 71 | 0 | *-Left Norm Key-* | y
791562 | 1 | 71 | 0 | *-Left Total Key-* | y
989440 | 1 | 71 | 0 | *-MM^T Product Key-* | y
72 | 1 | 71 | 0 | *-Mutual Info Key-* | y
791560 | 1 | 71 | 0 | *-Norm Key-* | y
791566 | 1 | 71 | 0 | *-Right Norm Key-* | y
791572 | 1 | 71 | 0 | *-Right Total Key-* | y
989437 | 1 | 71 | 0 | *-Total Entropy Key-* | y
989439 | 1 | 71 | 0 | *-Total MI Key-* | y
1 | 0 | 71 | 0 | *-TruthValueKey-* |
(13 rows)
atomspace is missing: *-TruthValueKey-* !! Huh!
This is missing (PredicateNode "*-Sentence Word Pair-*") Why?
(It is created by common.scm)
also: (PredicateNode "*-MM^T Product Key-*")
that's two. Third is the SchemaNode
three more needed...
AnchorNode ...
two more needed: TypeNode. OK, so it all balances.
==================
Same as above, but a biggger set.
(sql-open "postgres:///en_mini_dj_cfive")
(define fsa (add-subtotal-filter psa 40 5 5 #f))
(fsa 'left-basis-size) ;; 42406
(fsa 'right-basis-size) ;; 968594
Done storing 1063939 pairs in 1218 secs
(sql-open "postgres:///en_mini_cfive")
(psa 'fetch-pairs)
(batch-all-pair-mi psa)
Support: found num left= 18533 num right= 647988 in 31 secs
Done with wild-card count N(x,*) and N(*,y) in 787 secs
Total count N(*,*) = 25642000.0 = 25642000.0
Done computing 1063939 pairs in 137 secs
Done storing 647988 left-wilds
Done computing 1063939 pair MI's in 586 secs
Finished left entropy subtotals in 722 secs
Finished left MI subtotals in 749 secs
Done storing 647988 left-wilds in 349 secs
-- (define btr (batch-transpose psa))
-- (btr 'mmt-marginals)
Done storing 647988 left-wilds in 388 secs
==========
Same as above, but a much bigger set.
(sql-open "postgres:///en_dj_cfive")
(psa 'fetch-pairs)
(psa 'left-basis-size) ;; 444903
(psa 'right-basis-size) ;; 23444293
(print-matrix-summary-report psa)
(sql-close)
createdb en_large_dj_cfive
cat atom.sql | psql en_large_dj_cfive
COMMENT ON DATABASE en_large_dj_cfive IS 'en_cfive Sections (only) with word-count > 8; section-count > 3';
(sql-open "postgres:///en_large_dj_cfive")
(define fsa (add-subtotal-filter psa 8 3 3 #f))
(fsa 'left-basis-size) ;; 95628
(fsa 'right-basis-size) ;; 2167176
(length (fsa 'get-all-elts)) ;; 2358816
(define fso (make-store fsa))
(fso 'store-all-elts) ; Marginals are not to be stored.
(sql-close)
-- Shutdown and restart guile.
(sql-open "postgres:///en_large_dj_cfive")
(psa 'fetch-pairs)
(sql-close)
(cog-report-counts)
(WordNode . 246103)
(ConnectorSeq . 1671691)
(Section . 2358816)
(psa 'left-basis-size) ;; 27655 <<< lots smaller than Word count!
(psa 'right-basis-size) ;; 1671691
createdb en_large_marg
cat atom.sql | psql en_large_marg
COMMENT ON DATABASE en_large_marg IS
'en_cfive consistent Sections with word-count > 8; section-count > 3';
(sql-open "postgres:///en_large_marg")
(define psc (add-support-compute psa))
(psc 'cache-all)
Finished left norm marginals in 2934 secs
(so, I guess 1671691/2934 = 570 per second...)
(define fsa (add-subtotal-filter psc 8 3 3 #f))
(fsa 'left-basis-size) ;; 17985 << continuing to shrink...
(fsa 'right-basis-size) ;; 1671691 << unchanged.
(length (fsa 'get-all-elts)) ;; 2348252
(define lfa (add-linkage-filter fsa))
(lfa 'left-basis-size) ;; 17985
(lfa 'right-basis-size) ;; 949252 << much smaller
(length (lfa 'get-all-elts)) ;; 1563939
(define fso (make-store lfa))
(fso 'store-all-elts)
(sql-close)
-- Stop and restart guile
(sql-open "postgres:///en_large_marg")
(psa 'fetch-pairs)
(batch-all-pair-mi psa)
Finished left norm marginals in 1964 secs
Done with wild-card count N(x,*) and N(*,y) in 2137 secs
Done computing 1563939 pair MI's in 936 secs
Finished left entropy subtotals in 1809 secs
Finished left MI subtotals in 1860 secs
(define btr (batch-transpose psa))
(btr 'mmt-marginals)
en_large_dj_cfive
-----------------
Filtered copy of en_dj_cfive, with the (8 3 3) filter-cut applied:
all words with word-count less than 8 removed; all sections with a
count of less than 3 removed; all dj's with a count of less than 3
removed. DB does NOT contain marginals!
Note that there is a HUGE number of WordNodes that appear in
ConnectorSeq's that do NOT appear in Sections!
BUILD:
(sql-open "postgres:///en_dj_cfive")
(psa 'fetch-pairs)
(psa 'left-basis-size) ;; 444903
(psa 'right-basis-size) ;; 23444293
(sql-close)
createdb en_large_dj_cfive
cat atom.sql | psql en_large_dj_cfive
COMMENT ON DATABASE en_large_dj_cfive IS
'en_cfive Sections (only) with word-count > 8; section-count > 3';
(sql-open "postgres:///en_large_dj_cfive")
(define fsa (add-subtotal-filter psa 8 3 3 #f))
(fsa 'left-basis-size) ;; 95628
(fsa 'right-basis-size) ;; 2167176
(length (fsa 'get-all-elts)) ;; 2358816
(define fso (make-store fsa))
(fso 'store-all-elts) ; Marginals are not to be stored.
(sql-close)
STATUS: Fair; has the disjunct bug #2252
DATE: 28 June 2019
en_large_dj_cfive=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 602 MB |
public | valuations | table | ubuntu | 1158 MB |
select count(*) from atoms; <<<< 4653425 == 4.6M Atoms
select count(*) from valuations; <<<< 7076448 == 7.1M Valuations
-- (psa 'left-basis-size) ;; 27655
-- (psa 'right-basis-size) ;; 1671691
-- after re-filtering, these shrink a bit (of course)
602555918 Bytes uncompressed == 603 MBytes
89639884 Bytes compressed == 90 MBytes
d6ad6a2437b07424088f2f19f54b5ab8 en_large_dj_cfive.sql
fef1307912ac291ce826ad4e8e6861e0 en_large_dj_cfive.sql.bz2
en_large_marg
-------------
Same as en_large_dj_cfive, but count-filtered a second time, and
with unconnectable connectors/conseqs removed. Includes marginals
and MM^T values.
BUILD: Same as `en_micro_marg`.
STATUS: Fair; has the disjunct bug #2252
DATE: 28 June 2019
COMMENT ON DATABASE en_large_marg IS
'en_cfive consistent Sections with word-count > 8; section-count > 3';
en_large_marg=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 332 MB |
public | valuations | table | ubuntu | 1050 MB |
select count(*) from atoms; <<<< 3528943 == 3.5M Atoms
select count(*) from valuations; <<<< 8567215 == 8.6M Vals
((make-pseudo-cset-api) 'fetch-pairs):
-- Elapsed time to load csets: 312 secs
-- Atomspace holds 3528945 atoms
-- (cog-report-counts) (omitting bogus AnchorNode, SchemaNode)
((PredicateNode . 13) (ListLink . 964563) (Connector . 35093)
(ConnectorSeq . 947421) (Section . 1563939) (WordNode . 17910)
-- top says 4.5 GB RAM
Summary Report for Correlation Matrix Word-Disjunct Pairs (Connector Sets)
Left type: WordNode Right Type: ConnectorSeq Pair Type: Section
Wildcard: (ListLink (ctv 0 0 26185082)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 17141 Columns: 947421
Size: 1563939 non-zero entries of 16239743361 possible
Fraction non-zero: 9.6303E-5 Sparsity (-log_2): 13.342
Total observations: 26185082.0 Avg obs per pair: 16.743
Entropy Total: 18.075 Left: 15.220 Right: 8.0117
Total MI: 5.1571
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 219.1 3.5091E+4
Count (l_1) 1.9001E+4 5.4787E+5 86.74 15.61
Length (l_2) 6119. 3.2536E+4 27.93 .9272
RMS Count 6002. 3.2390E+4 27.40 .9230
MM^T support=1563939.0 count=497539636092.0 entropy=2.2975
No M^TM data present
656651442 Bytes uncompressed == 657 MBytes
86400823 Bytes compressed == 86 MBytes
3f0b2d8c2b32c1ecef814e46724e88f8 en_large_marg.sql
9c26bafdf5a7328364709a3d789e516d en_large_marg.sql.bz2
==========
Same as above, but the hugest non-full set.
(sql-open "postgres:///en_dj_cfive")
(psa 'fetch-pairs)
(print-matrix-summary-report psa)
(sql-close)
createdb en_huge_dj_cfive
cat atom.sql | psql en_huge_dj_cfive
COMMENT ON DATABASE en_huge_dj_cfive IS 'en_cfive Sections (only) with word-count > 3; section-count > 1';
(sql-open "postgres:///en_huge_dj_cfive")
(define fsa (add-subtotal-filter psa 3 1 1 #f))
(fsa 'left-basis-size) ;; 151951
(fsa 'right-basis-size) ;; 8233844
(length (fsa 'get-all-elts)) ;; 9315925
(define fso (make-store fsa))
(fso 'store-all-elts) ; Marginals are not to be stored.
(sql-close)
-- Shutdown and restart guile.
(sql-open "postgres:///en_huge_dj_cfive")
(psa 'fetch-pairs)
(sql-close)
(cog-report-counts)
(Connector . 636895) (ConnectorSeq . 7345963)
(Section . 9315925) (WordNode . 387402)
(psa 'left-basis-size) ;; 84774
(psa 'right-basis-size) ;; 7345963
createdb en_huge_marg
cat atom.sql | psql en_huge_marg
COMMENT ON DATABASE en_huge_marg IS
'en_cfive consistent Sections with word-count > 3; section-count > 1';
(sql-open "postgres:///en_huge_marg")
(define psc (add-support-compute psa))
(psc 'cache-all)
Finished left norm marginals in 37844 secs
so 7345963 / 37844 = 194/second ...
(define fsa (add-subtotal-filter psc 3 1 1 #f))
(fsa 'left-basis-size) ;; 57400 << so, a bit smaller
(fsa 'right-basis-size) ;; 7345963 << unchanged
(length (fsa 'get-all-elts)) ;; 9288551
(define lfa (add-linkage-filter fsa))
(lfa 'left-basis-size) ;; 57400 << unchanged
(lfa 'right-basis-size) ;; 6045624 << a bit smaller
(length (lfa 'get-all-elts)) ;; 7907629 << smaller
(define fso (make-store lfa))
(fso 'store-all-elts) ;; Done storing 7907629 pairs in 8034 secs
(sql-close)
-- Stop and restart guile
(sql-open "postgres:///en_huge_marg")
(psa 'fetch-pairs) ; Elapsed time to load csets: 1049 secs
; Support: found num left= 54832 num right= 6031954
(batch-all-pair-mi psa)
Finished left norm marginals in 33135 secs
Finished left entropy subtotals in 32088 secs
Finished left MI subtotals in 33280 secs
xxxxxxxxxxxx
(define btr (batch-transpose psa))
(btr 'mmt-marginals)
RAM usage: 36 GB resident.
en_huge_marg
------------
Same as en_dj_cfive, but with unconnectable connectors/conseqs removed.
Includes marginals, pair-MI's and MM^T values.
Loading and running this takes time and uses significant RAM:
((make-pseudo-cset-api) 'fetch-pairs):
-- Elapsed time to load csets:
-- 36 GBytes RAM
BUILD: Same as `en_micro_marg`.
STATUS: Fair; has the disjunct bug #2252
DATE: 1 July 2019
COMMENT ON DATABASE en_huge_marg IS
'en_cfive consistent Sections with word-count > 3; section-count > 1';
en_huge_marg=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 1878 MB |
public | valuations | table | ubuntu | 6407 MB |
select count(*) from atoms; <<<< 20195313 == 20M Atoms
select count(*) from valuations; <<<< 48124872 == 48M Vals
guile-en> (print-matrix-summary-report psa)
Summary Report for Correlation Matrix Word-Disjunct Pairs (Connector Sets)
Left type: WordNode Right Type: ConnectorSeq Pair Type: Section
Wildcard: (ListLink (ctv 0 0 42284522)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 54832 Columns: 6031954
Size: 7907629 non-zero entries of 330744101728 possible
Fraction non-zero: 2.3909E-5 Sparsity (-log_2): 15.352
Total observations: 42284522.0 Avg obs per pair: 5.3473
Entropy Total: 20.691 Left: 18.450 Right: 8.1271
Total MI: 5.8865
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 268.7 1.8714E+5
Count (l_1) 1.2368E+4 9.9142E+5 46.03 5.298
Length (l_2) 3858. 3.2396E+4 14.36 .1731
RMS Count 3813. 3.2323E+4 14.19 .1727
MM^T support=7907629.0 count=522992873156.0 entropy=6.9016
No M^TM data present
3772947413 Bytes uncompressed == 3.77 GBytes
417542166 Bytes compressed == 418 MBytes
eeed40c6a36fe0319cd8e53fc1d2bc6b en_huge_marg.sql
8c409609d5535473f21d7c469f2dfd53 en_huge_marg.sql.bz2
==========
linkage-filter the full set, now:
(sql-open "postgres:///en_dj_cfive")
(psa 'fetch-pairs) ; 81224843 atoms in 6614 secs
(sql-close)
(sql-open "postgres:///en_midj_links")
(define alf (add-linkage-filter psa))
(define sto (make-store alf))
(sto 'store-all-elts) ; Marginals are not to be stored.
(1000 pairs/sec) ; Done storing 31749357 pairs in 33336 secs
(sql-close)
restart
-- The alf linkage-filter + restart implies that en_midj_links
should contain only consistant connectors & links.
(sql-open "postgres:///en_midj_links")
(batch-all-pair-mi psa)
Support: found num left= 438449 num right= 23308382 in 1095 secs
Finished left norm marginals in 316395 secs (3.6 days)
Done storing 23308382 left-wilds in 16381 secs
Done computing 31749357 pair MI's in 35266 secs
Last time, this took "only" 29048 secs (8 hours) for the similarly
sized en_pairs_rfive_mst dataset. So why is it so much slower now?
Is the pattern matcher in the stars object getting in the way?
Yes, it probably is. This needs to be fixed...
Umm, the en_sh_mi_two which contained the exploded shapes took that long
... probably for the same reason...
OK, system went down. Restart....
(load "/tmp/cm.scm")
(xbatch-all-pair-mi psa)
Restart computing
Going to do column and row subtotals
Finished left entropy subtotals in 339785 secs <<< 94 hours, i.e. 4 days
Finished right entropy subtotals in 1776 secs
Finished left MI subtotals in 336719 secs << 93 hours
Done storing 23308382 left-wilds in 13255 secs
Finished with MI computations; this took 193. hours (8 days!)
------
(define btr (batch-transpose psa))
(btr 'mmt-marginals)
Finished mmt norm marginals in 2822 secs
Finished mmt totals in 21 secs
Done storing 438449 right-wilds in 327 secs
Done storing 23308382 left-wilds in 14668 secs
12587 minutes cpu time for guile = 210 cpu-hours = 9 cpu-days
quick cheat-stats:
31749357 pairs. tot count== 69197410
ALTER DATABASE en_midj_links RENAME TO en_full_marg;
en_full_marg
------------
Same as en_dj_cfive, but with unconnectable connectors/conseqs removed.
Includes marginals, pair-MI's and MM^T values. This is effectively the
"full" cfive dataset, with only impossible linkages removed.
Loading and running this takes time and uses significant RAM:
((make-pseudo-cset-api) 'fetch-pairs):
-- Elapsed time to load csets: 6100 secs
-- 97 GBytes RAM
BUILD:
(sql-open "postgres:///en_dj_cfive")
(psa 'fetch-pairs) ; 81224843 atoms in 6614 secs
(sql-close)
(sql-open "postgres:///en_full_marg")
(define alf (add-linkage-filter psa))
(define sto (make-store alf))
(sto 'store-all-elts) ; Marginals are not to be stored.
(sql-close)
restart
(sql-open "postgres:///en_full_marg")
(batch-all-pair-mi psa)
(define btr (batch-transpose psa))
(btr 'mmt-marginals)
STATUS: Fair; has the disjunct bug #2252
DATE: 27 June 2019
COMMENT ON DATABASE en_full_marg IS
'All consistent en_cfive Sections plus MI plus MM^T';
en_full_marg=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 7356 MB |
public | valuations | table | ubuntu | 27 GB |
select count(*) from atoms; <<<< 80111606 == 80M Atoms
select count(*) from valuations; <<<< 190673853 == 191M Vals
guile-en> (print-matrix-summary-report psa)
Summary Report for Correlation Matrix Word-Disjunct Pairs (Connector Sets)
Left type: WordNode Right Type: ConnectorSeq Pair Type: Section
Wildcard: (ListLink (ctv 0 0 69197410)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 438449 Columns: 23308382
Size: 31749357 non-zero entries of 10219536779518 possible
Fraction non-zero: 3.1067E-6 Sparsity (-log_2): 18.296
Total observations: 69197410.0 Avg obs per pair: 2.1795
Entropy Total: 23.080 Left: 21.119 Right: 10.109
Total MI: 8.1482
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 368.9 2.2490E+5
Count (l_1) 7946. 9.3725E+5 21.54 4.167
Length (l_2) 2405. 2.4518E+4 6.520 .1090
RMS Count 2388. 2.4467E+4 6.473 .1088
MM^T support=31749357.0 count=549825893944.0 entropy=10.840
No M^TM data present
15053083395 Bytes uncompressed == 15 GBytes
1738289510 Bytes compressed == 1.7 GBytes
30812d532e2dfa774fc1caf04ee24722 en_full_marg.sql
b94eee614a869de1517235962e26828b en_full_marg.sql.bz2
==========
fanny:
en_sh_two 25 GB Section and Shape marginals (w/o explodes)
en_sh_mi_two 56 GB Above, w/batch-all-pair-mi
Is this needed? What do we need MI(w,dj) for?
en_sh_two_sim 27 GB Word Similarities
mrg_tst 47 GB Scratch dataset
mrg_two 51 GB
Archives /home2/linas/src/novamente/data/sql-dumps
-- play with mrg_tst
90912007 reserved uuids == 91M atoms
-- Run x-shell.sh
(define pca (make-pseudo-cset-api))
(define psa (add-pair-stars pca))
(psa 'fetch-pairs)
starts out 13.5K/sec, 2/3user, 1/3 kernel 286% guile, 8x42% postgres
suckin it off the disk at 5.5MBytes/sec (24KBytes/transaction in iostat)
.. for words.. then spurts to 200MBytes/sec for csets
.. then low disk-read rates, but 640% on guile, 3x50% on postgres
this is for (load-atoms-of-type 'Section)
running at 80% kernel, 20% user... wtf??
-- Elapsed time to load csets: 3037 secs 24664854 total == 8K/sec avg.
32 GBytes RAM total
(cog-report-counts) (WordNode . 137078) (Connector . 268592)
(ConnectorSeq . 6239997) (ListLink . 6377076) (Section . 11642010)
(WordClassNode . 93)
-- Before: 24664861
(load-atoms-of-type 'SimilarityLink)
approx 600 seconds at 2.5K atoms/sec, more than half of that waiting
on postgres... loaded 1457522 links (SimilarityLink . 1457522)
After: 26122396
-- (cog-get-atoms 'PredicateNode) Hmm no cosines in this dataset.
wtf ... the keys were not loaded ??? That's a bug! Pull req #2199
-- (define (prt-atom h) (display h) #t)
(cog-map-type prt-atom 'SimilarityLink)
(cog-keys x)
-- review learn-lang-diary/similarity.scm
-- (pmi 'mmt-fmi (Word "other") (Word "same"))
(pta 'mmt-count (Word "other"))
-- (define btr (batch-transpose psa))
(btr 'mmt-marginals)
might take ten hours ouch...
Finished left norm marginals in 31830 secs
Finished left totals in 238 secs
Finished column (left) norm averages in 963 secs
Done storing 6239919 left-wilds in 6024 secs
Finished mmt norm marginals in 613 secs
Done storing 6239919 left-wilds in 5701 secs
add-transpose-compute
(define ptc (add-transpose-compute psa))
(ptc 'mmt-marginals)
total loop over 137078 items in a list
Finished mmt norm marginals in 3012 secs -- colliding
Finished mmt norm marginals in 1329 secs - par-for-each -- cpu=100%
Finished mmt norm marginals in 525 secs - for-each --
Finished mmt norm marginals in 359 secs - n-par-for-each 4 -- cpu=250%
Finished mmt norm marginals in 413 secs - n-par-for-each 2 -- cpu=170%
Finished mmt norm marginals in 474 secs - n-par-for-each 1 -- cpu=100%
Finished mmt norm marginals in 370 secs - n-par-for-each 8 -- cpu=250%
Finished mmt norm marginals in 375 secs - n-par-for-each 24 -- cpu=230%
Finished mmt norm marginals in 557 secs - par-for-each --
Finished mmt norm marginals in 530 secs - par-for-each -- cpu=100%
Finished mmt norm marginals in 345 secs - my-par-for-each 6
Finished mmt norm marginals in 349 secs - my-par-for-each 12
Finished mmt norm marginals in 357 secs - my-new-for-each 12
102:
#0 0x00007f343c36c91b in GC_register_finalizer_inner (obj=0x55efee7aba60,
fn=0x7f343d25d490 <finalize_smob>, cd=0x0, ofn=ofn@entry=0x7f320c8f8570,
ocd=ocd@entry=0x7f320c8f8578,
mp=mp@entry=0x7f343c36b7d0 <GC_null_finalize_mark_proc>) at finalize.c:655
#1 0x00007f343c36cd7c in GC_register_finalizer_no_order (obj=<optimized out>,
fn=<optimized out>, cd=<optimized out>, ofn=ofn@entry=0x7f320c8f8570,
ocd=ocd@entry=0x7f320c8f8578) at finalize.c:771
#2 0x00007f343d20e2e1 in scm_i_set_finalizer (obj=<optimized out>,
proc=<optimized out>, data=<optimized out>)
at ../../libguile/finalizers.c:59
#3 0x00007f343d25d8a2 in scm_i_new_smob (tc=tc@entry=5239,
data=data@entry=139822156266736) at ../../libguile/smob.c:440
#4 0x00007f3431c48a4b in scm_new_smob (data=139822156266736, tc=5239)
at /usr/local/include/guile/3.0/libguile/smob.h:140
#5 opencog::SchemeSmob::protom_to_scm (
pa=std::shared_ptr (count 7, weak 3) 0x7f30578d05a0)
at /home/linas/src/novamente/src/atomspace-git/opencog/guile/SchemeSmobNew.cc:160
103:
#0 pthread_cond_wait@@GLIBC_2.3.2 ()
at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f343d27bb95 in scm_pthread_cond_wait (cond=<optimized out>,
mutex=<optimized out>) at ../../libguile/threads.c:1615
#2 0x00007f343d27bd8b in block_self (queue=0x55ef64391cd0,
mutex=mutex@entry=0x55ef63d21fc0, waittime=waittime@entry=0x0)
at ../../libguile/threads.c:316
#3 0x00007f343d27bedf in lock_mutex (current_thread=0x55f0a3d42c60,
waittime=0x0, m=0x55ef63d21fc0, kind=SCM_MUTEX_STANDARD)
at ../../libguile/threads.c:1031
#4 scm_timed_lock_mutex (mutex=0x55ef64391cc0, timeout=<optimized out>)
at ../../libguile/threads.c:1092
#5 0x00007f343d56582a in ?? ()
#6 0x00007f34104bdf90 in ?? ()
#7 0x00007f343d4f00a0 in jump_table_ () from /usr/local/lib/libguile-3.0.so.0
#8 0x000055ef6347b3a8 in ?? ()
#9 0x00007f343d22bf61 in scm_jit_enter_mcode (thread=0x55f0a3d42840,
mcode=0x55f0a3d42840 "\240)ԣ\360U") at ../../libguile/jit.c:4819
#10 0x00007f343d28089c in vm_debug_engine (thread=0x55f0a3d42840)
at ../../libguile/vm-engine.c:370
#11 0x00007f343d28707a in scm_call_n (proc=proc@entry=0x55eff19c9f00,
argv=argv@entry=0x0, nargs=nargs@entry=0) at ../../libguile/vm.c:1605
104, 106, 107: same as above
105:
#0 0x00007f342ff7fb03 in opencog::Variables::is_type (this=0x7f2b146d3c00,
var=..., val=...)
-------------------------------------------
66:
#5 std::lock_guard<std::recursive_mutex>::lock_guard (__m=...,
this=<synthetic pointer>) at /usr/include/c++/6/bits/std_mutex.h:162
#6 opencog::AtomTable::lookupHandle (this=this@entry=0x55ef63c282a0,
a=std::shared_ptr (count 1, weak 1) 0x7f2b24bffdb0)
at /home/linas/src/novamente/src/atomspace-git/opencog/atomspace/AtomTable.cc:247
67: same
69: opencog, but then gc
71: atomspace, but then gc
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f343ca69bb5 in __GI___pthread_mutex_lock (
mutex=mutex@entry=0x7f343d4f0f40 <bytes_until_gc_lock>)
at ../nptl/pthread_mutex_lock.c:80
#2 0x00007f343d213e20 in scm_gc_register_allocation (size=size@entry=16)
at ../../libguile/gc.c:591
---------------------------------------------------------------------
... continuing with the junk-scratch dataset...
-- (prt-all-classes) in word-classes/word-classes.scm
-- (fetch-all-gram-classes)
-- Exporting
(define gca (make-gram-class-api))
(gca 'fetch-pairs) -- before, 24665724 atoms tlbuf holds 24665716 atoms
(define gcs (add-pair-stars gca))
(define gcl (add-loop-api gcs))
(define gcm (add-pair-freq-api gcs))
; (gcl 'for-each-pair (lambda (x) (format #t "Foo ~A\n" x))) too many
Ah haha! FMI frequencies missing on gram class. Hmm.
Add this by... (batch-all-pair-mi gcs)
Start computing the basis
Support: found num left= 93 num right= 2394832 in 101 secs
Finished left norm marginals in 15230 secs
Finished left totals in 1232 secs
Finished right norm marginals in 289 secs
Finished right totals in 0 secs
Done with wild-card count N(x,*) and N(*,y) in 16751 secs
Total count N(*,*) = 4773906.510786109 = 4773906.510891325
Finished column (left) norm averages in 1243 secs
Finished row (right) norm averages in 0 secs
Going to do individual pair frequencies
Done computing 3096306 pairs in 1584 secs
Start computing log P(*,y)
Done computing 2394832 left-wilds in 161 secs
Done with -log P(*,y), start -log P(x,*)
Done computing 93 right-wilds in 0 secs
Done storing 2394832 left-wilds in 5439 secs
Done storing 93 right-wilds in 0 secs
Done computing and saving -log P(x,*) and P(*,y)
Going to do individual pair MI
Done computing 3096306 pair MI's in 24368 secs
Going to do column and row subtotals
Finished left entropy subtotals in 27670 secs
Finished right entropy subtotals in 87 secs
Finished left MI subtotals in 12952 secs
Finished right MI subtotals in 71 secs
Going to compute the left, right and total entropy
Done computing totals; start saving wildcards
Done storing 2394832 left-wilds in 1530 secs
Done storing 93 right-wilds in 0 secs
Finished with MI computations; this took 25.2 hours
(print-matrix-summary-report gcs)
Summary Report for Correlation Matrix WordClass-Disjunct Pairs
Left type: WordClassNode Right Type: ConnectorSeq Pair Type: Section
Wildcard: (ListLink (ctv 0 0 4773906.5)
(AnyNode "gram-class-word")
(AnyNode "gram-class-disjunct")
)
Rows: 93 Columns: 2394832
Size: 3096306 non-zero entries of 222719376 possible
Fraction non-zero: 1.3902E-2 Sparsity (-log_2): 6.1685
Total observations: 4773906.51 Avg obs per pair: 1.5418
Entropy Total: 17.039 Left: 15.822 Right: 4.8679
Total MI: 3.6509
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 14.89 1.3948E+5
Count (l_1) 4086. 2.6703E+5 274.5 1.915
Length (l_2) 3037. 2.4071E+4 204.0 .1726
RMS Count 2953. 2.4060E+4 198.4 .1725
* (export-csets gcs "dict.db" "EN_us")
Store 3096306 csets
fail-insert: make-db-adder: near "t": syntax error
Error: bad section (Section (ctv 1 0 36)
(current-thread)
(cancel-thread (car (all-threads)))
Oooops!
(for-each (lambda (thr) (if (not (eq? thr (current-thread))) (cancel-thread thr))) (all-threads))
-- Do it again. again 24664866 after (pca 'fetch-pairs)
-- (print-matrix-summary-report pcs) looks good.
-- (print-matrix-summary-report gcs) is empty.
-- (gcs 'fetch-pairs) now 27060667 XXX why so many more??
(cog-report-counts)
((PredicateNode . 14) (ListLink . 8772002) (MemberLink . 852)
(AnchorNode . 1) (Connector . 268592) (ConnectorDir . 2) (ConnectorSeq .
6239997) (Section . 11642010) (SchemaNode . 1) (WordNode . 137078)
(AnyNode . 4) (WordClassNode . 93))
-- (use-modules (opencog nlp lg-export))
opencog/nlp/lg-export/export-disjuncts.scm:317:16: warning: possibly
unbound variable `dbi-query'
`dbi-open'
`dbi-get_status'
ERROR: In procedure apply-smob/1:
In procedure dbi-close: Wrong type argument in position 1: #<finalized
smob 55a3ce6361b0>
-- (export-csets gcs "dict.db" "EN_us")
(define pfa (add-pair-freq-api gcs))
(pfa 'set-pair-mi t 0.003 3)
-- (define gfs (add-wordclass-filter gcs))
(gfs 'left-basis-size) ; 96
(gfs 'right-basis-size) ; 127536
(export-csets gfs "dict.db" "EN_us")
Will store 185416 csets
grep VmSwap /proc/14022/status
(define t (current-time)) (gcf 'right-basis-size)
(format #t "Elapsed ~A\n" (- (current-time) t))
22:30...
22:38 ... 100000 of 6239997 connector sequences
22:44 .. 200000 of 6239997 connector sequences
22:53 .. 300000 of 6239997 connector sequences
22:58 .. 400000 of 6239997
23:05 .. 500000
23:11 .. 600000
23:19 .. 700000
23:25 .. 800000
23:31 .. 900000
23:38 .. 1000000
23:45 .. 1100000
23:53 .. 1200000
00:00 .. 1300000
00:04 .. 1400000
00:11 .. 1500000
alternate:
make list of words in word-classes,
make list of connectors containing these
make list of connector seqs containing these. How many? 5837887 of them
Sooo .. almost all of them. No savings.
; Make a list of words in word-classes
(define word-set (make-atom-set))
(for-each
(lambda (wcls)
(for-each word-set
(map gar (cog-incoming-by-type wcls 'MemberLink))))
(cog-get-atoms 'WordClassNode))
(define words (word-set #f))
(length words) ; 2838 OK
(define all-words (cog-get-atoms 'WordNode))
(length all-words) ; 137078
(define unwords (atoms-subtract all-words words))
(length unwords) ; 134240 OK.
(define con-set (make-atom-set))
(for-each
(lambda (word)
(for-each con-set
(cog-incoming-by-type word 'Connector)))
words)
(define ctcrs (con-set #f))
(length ctcrs) ; 5675
(define unctcrs (con-set #f))
(length unctcrs) ; 262917 OK
(define seq-set (make-atom-set))
(for-each
(lambda (ctr)
(for-each seq-set
(cog-incoming-by-type ctr 'ConnectorSeq)))
ctcrs)
(define conseqs (seq-set #f))
(length conseqs) ; 5837887
Yikes!
(define unconseqs (seq-set #f))
(length unconseqs) ; 3795126 OK.
How about the other way? there are 3795126 bad conseqs. Ahh! Much
better. These can be dropped right away. So there should be
(- 6239997 3795126) = 2444871 left...
wtf ... it found only 1968134 in 321 seconds ... !?
oh ... (gcs 'right-basis-size) is 4669759 which is already smaller than
tot.
-- filter-publish top obs N words, as well as classes.
uhh... but this needs recomputed MI's.
-- lg lookup words having connector !?
-- (define wlst (trim-and-rank psa (cog-get-atoms 'WordNode) 5))
51753 of 137078 have count > 5
14845 have count > 50
2670 have count > 500 of these, 2019 are not in classes...
(define swlst (trim-and-rank psa (cog-get-atoms 'WordNode) 500))
; Remove words in word-classes
(define uwlst (filter (lambda (wrd)
(eq? '() (cog-incoming-by-type wrd 'MemberLink)))
swlst))
(for-each (lambda (wrd)
(MemberLink wrd (WordClassNode
(string-append (cog-name wrd) "#uni")))) uwlst)
-- recompute the MI, now (filtered for speed)
(define gca (make-gram-class-api))
(define gcs (add-pair-stars gca))
(define gcf (add-wordclass-filter gcs))
Before starting: Atomspace holds 27064684 atoms
Create the unitary sections: Atomspace holds 30385374 atoms
(gcf 'left-basis-size) 2112
(gcf 'right-basis-size) 1968134 in 216 sec
(batch-all-pair-mi gcf)
Start computing the basis
Support: found num left= 2112 num right= 1968134 in 1 secs (actually 216 sec)
Finished left norm marginals in 24434 secs
Finished left totals in 61 secs
Finished right norm marginals in 210 secs
Finished right totals in 0 secs
Done with wild-card count N(x,*) and N(*,y) in 24705 secs
Total count N(*,*) = 0.0 = 0.0
Finished column (left) norm averages in 322 secs
Finished row (right) norm averages in 0 secs
Going to do individual pair frequencies
Done computing 2829796 pairs in 337 secs
Start computing log P(*,y)
In opencog/matrix/object-api.scm:
707:46 3 (set-left-wild-freq _ +nan.0)
In unknown file:
2 (cog-new-value 2 +nan.0 +nan.0-4.532360141827193i +nan.0+nan.0i)
why is the count zero?
--
(define gsu (add-support-compute gcf))
(gsu 'right-support-set (WordClassNode "it there"))
is empty set...
(gsu 'right-stars (WordClassNode "it there")) is big -- 30272
(gsu 'get-count (car ..) == 0
(gcf 'get-count sec) = 0
(gcs 'get-count sec) !=0 why!?
oohhh filter had a bug. Damn,
(gcf 'get-pair (gar don) (gdr don)) returns,
(gcf 'get-pair-count (gar don) (gdr don)) also works ...
-- createdb mini-test
cat atom.sql | psql mini-test
(sql-close)
(sql-open "postgres:///mini-test?user=linas&password=asdf")
(define gso (make-store gcf))
(gso 'store-wildcards)
(length (psa 'get-all-elts)) -- 8545704
(length (gcs 'get-all-elts)) -- 6416994
(length (gcf 'get-all-elts)) -- 6416994
Hmm that does not seem right... oh bug
(length (gcf 'get-all-elts)) -- 2829796
That's more like it.
(gso 'store-all-elts)
-- Start again, on mini-test
Elapsed time to load grammatical classes: 974 secs
Atomspace holds 6782961 atoms
top -- 7.755g in size
100% of all atoms loaded!
(batch-all-pair-mi gcs)
Start computing the basis
Support: found num left= 2112 num right= 0 in 201 secs
whoops. Failed to store MemberLinks!
Support: found num left= 2112 num right= 1968134 in 0 secs
(define cnt 0)
(for-each (lambda (wc)
(if (eq? '() (cog-incoming-by-type wc 'MemberLink))
(begin (set! cnt (+ cnt 1)) (format #t "Ohhh nooo ~A ~A" cnt wc))))
(cog-get-atoms 'WordClassNode))
(define cnt 0)
(for-each (lambda (wc)
(if (eq? '() (cog-incoming-by-type wc 'MemberLink))
(let* ((wcn (cog-name wc))
(wn (string-copy wcn 0 (string-contains wcn "#uni"))))
(set! cnt (+ cnt 1))
(store-atom (MemberLink (WordNode wn) wc))
(format #t "Fix-hack ~A ~A to ~A\n" cnt wc wn))))
(cog-get-atoms 'WordClassNode))
(length
(filter
(lambda (lst) (< 1 (length lst)))
(map (lambda (wrdcls) (cog-incoming-by-type wrdcls 'MemberLink))
(cog-get-atoms 'WordClassNode))))
(define gca (make-gram-class-api))
(define gcs (add-pair-stars gca))
(batch-all-pair-mi gcs)
Start computing the basis
Support: found num left= 2112 num right= 1968134 in 1 secs
Done with wild-card count N(x,*) and N(*,y) in 112399 secs
That's 31 hours!! WTF! why so long?
Total count N(*,*) = 8364154.64551346 = 8364154.645606807
Done storing 1968134 left-wilds in 1458 secs
Done computing 2829796 pair MI's in 1848 secs
Finished left entropy subtotals in 110243 secs (again, 31 hours!)
Finished left MI subtotals in 111610 secs
Done storing 1968134 left-wilds in 1299 secs
Finished with MI computations; this took 94.3 hours
-- again the is the corrupted test dataset; so kind-of bogus.
(print-matrix-summary-report gcs)
Summary Report for Correlation Matrix WordClass-Disjunct Pairs
Left type: WordClassNode Right Type: ConnectorSeq Pair Type:
Section
Wildcard: (ListLink (ctv 0 0 8364154.6)
(AnyNode "gram-class-word")
(AnyNode "gram-class-disjunct")
)
Rows: 2112 Columns: 1968134
Size: 2829796 non-zero entries of 4156699008 possible
Fraction non-zero: 6.8078E-4 Sparsity (-log_2): 10.521
Total observations: 8364154.65 Avg obs per pair: 2.9557
Entropy Total: 17.211 Left: 15.164 Right: 7.3950
Total MI: 5.3480
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 97.23 3.0108E+4
Count (l_1) 6941. 1.8845E+5 71.39 6.259
Length (l_2) 4549. 2.3722E+4 46.78 .7879
RMS Count 4430. 2.3689E+4 45.56 .7868
-- (use-modules (opencog nlp lg-export))
(export-csets gcs "dict.db" "EN_us") ; start 20:48
Will store 2829796 csets
This took 236 MBytes! Yow! (because disjuncts not merged.
!!he
he.587 [TJEOI- & TJFFA- & TJEOE+]-6.005
LEFT-WALL.803 [TNENU+ & TNEPX+]-3.377
select * from Morphemes where morpheme = "LEFT-WALL";
There is only one ... how can that be? OK.
LEFT-WALL|LEFT-WALL.803|<###LEFT-WALL####uni.803>
select count(*) from Disjuncts where classname = "<###LEFT-WALL####uni.803>";
18883
almost 19K of them, so why only one loaded? (only the last one... oh bug.)
select * from Disjuncts where disjunct like '%TNENU-%';
Uhh ohh .. this is empty. How?
select count(*) from Morphemes where morpheme = "John";
John|John.629|<John#uni.629>
select count(*) from Disjuncts where classname = "<John#uni.629>";
1129
select count(*) from Disjuncts where classname = "<Joe#uni.647>";
284
<Joe#uni.647>|TKIWE- & TKIOY+ & TKIUF+|-14.1628982914518
Joe don't tell!
(Section (ctv 1 0 0.3)
(WordClassNode "don't don’t")
(ConnectorSeq
(Connector
(WordNode "###LEFT-WALL###")
(ConnectorDir "-")
)
(Connector
(WordNode "Joe")
(ConnectorDir "-")
)
(Connector
(WordNode "tell")
(ConnectorDir "+")
)
(Connector
(WordNode "!")
(ConnectorDir "+")
)
)
)
(cog-execute! (Get (VariableList (Glob "g") (Glob "h"))
(Section (WordNode "###LEFT-WALL###")
(ConnectorSeq
(Glob "g")
(Connector (WordNode "don’t") (ConnectorDir "+"))
(Glob "h")))))
(cog-execute! (Get (VariableList
(TypedVariable (Glob "g") (Interval (Number 0) (Number -1)))
(TypedVariable (Glob "h") (Interval (Number 0) (Number -1))))
(Section (WordNode "Joe")
(ConnectorSeq
(Glob "g")
(Connector (WordNode "don't") (ConnectorDir "+"))
(Glob "h")))))
get-cnr-name is broken ....
Now what???
select count(*) from Morphemes;
3
(cog-report-counts)
((PredicateNode . 14) (ListLink . 2632654) (TypeNode . 2) (AnchorNode .
1) (Connector . 5674) (ConnectorDir . 2) (ConnectorSeq . 1968134)
(Section . 2829796) (SchemaNode . 1) (WordNode . 2838) (AnyNode . 4)
(WordClassNode . 2112))
No MemberLinks at all. WTF.
(load-atoms-of-type 'MemberLink)
s.scm
OK, so mini-tst is incomplete. was it scrubbed wrong?
Arghhhh.
Also, lg generator
TB == (WordClassNode "left#uni") (WordNode "don’t")
TE == (WordClassNode "don't don’t") (WordNode "###LEFT-WALL###")
So, for uni's do not create a wc link....
or .. do create one, but also one for each word in the class.
===================
OK, so do cosine for the micro. How?
-- on LXC, micro-en.scm, etc.
createdb en_micro_cosine -T en_micro_links
(sql-open "postgres:///en_micro_cosine")
(psa 'fetch-pairs)
Fresh copy shows: (cog-report-counts)
(ListLink . 124635) (Connector . 19943) (ConnectorSeq . 113821)
(Section . 302867) (WordNode . 10813)
but en_micro_dj_cfive claims that there should be 8608 words, so
wtf...
(define cnt 0)
(for-each (lambda (wrd) (if
(eq? 'SimpleTruthValue (cog-type (cog-tv wrd)))
(set! cnt (+ cnt 1)))) (cog-get-atoms 'WordNode))
(WordNode "Kaladin") in two sections both with count of 11
total count is well under 500 so wtf?? Answer: when psa got
saved, some low-count words got saved, ...
(gram-classify-greedy-fuzz 0.65 0.3 4)
=========
Try again.
-- (sql-open "postgres:///en_micro_dj_cfive")
(psa 'fetch-pairs)
(sql-close)
(psa 'left-basis-size) ;; 10813
(psa 'right-basis-size) ;; 187063
-- No marginals, so cannot filter. So make at least subtotal margs
(define psc (add-support-compute psa))
(psc 'cache-all) ; Finished left norm marginals in 136 secs
(define fsa (add-subtotal-filter psa 500 10 10 #f))
(fsa 'left-basis-size) ;; 1624
(fsa 'right-basis-size) ;; 187063
(define lfa (add-linkage-filter fsa))
(lfa 'left-basis-size) ;; 1624
(lfa 'right-basis-size) ;; 68602
createdb en_micro_marg
cat /home/ubuntu/src/atomspace/opencog/persist/sql/multi-driver/atom.sql | psql en_micro_marg
(sql-open "postgres:///en_micro_marg")
(define fso (make-store lfa))
(fso 'store-all-elts) ; Marginals are not to be stored.
Done storing 207491 pairs in 186 secs
(sql-close)
-- (sql-open "postgres:///en_micro_marg")
(psa 'fetch-pairs)
(psa 'left-basis-size) ;; 8386 -- still too big. why?
(psa 'right-basis-size) ;; 68602
(length (psa 'get-all-elts)) ;; 207491 so that's what we stored.
(define psc (add-support-compute psa))
(psc 'cache-all) ; Finished left norm marginals in 67 secs
(define fsa (add-subtotal-filter psa 500 10 10 #f))
(fsa 'left-basis-size) ;; 1407 -- got smaller still. Why?
(fsa 'right-basis-size) ;; 68602
(length (fsa 'get-all-elts)) ;; after fix: 181322
(define lfa (add-linkage-filter fsa))
(lfa 'left-basis-size) ;; 1407
(lfa 'right-basis-size) ;; 66057
(length (lfa 'get-all-elts)) ;; 201740 - after fix: 176427
(define aset (make-atom-set))
(for-each (lambda (sec) (aset (gar sec))) (lfa 'get-all-elts))
(length (aset #f)) ;; 8170 -- so consistent linkage contains
words that did not pass the filter.
so consistent linkage is buggy...
;; 1404 after fix. Yay!
-- Now filter it a second time ...
(define psd (add-support-compute lfa))
(psd 'cache-all) ; Finished left norm marginals in 71 secs
(define fsb (add-subtotal-filter psd 500 10 10 #f))
(fsb 'left-basis-size) ;; 1381 smaller than 1407
(fsb 'right-basis-size) ;; 64667 smaller than 68602
(length (fsb 'get-all-elts)) ;; 176163 smaller than 181322
(define lfb (add-linkage-filter fsb))
(lfb 'left-basis-size) ;; 1381 smaller than 1407
(lfb 'right-basis-size) ;; 64256 smaler than 66057
(length (lfb 'get-all-elts)) ;; 175446 smaler than 176427
(length (aset #f)) ; 1380 OK, that works for me ...
-- Do it a third time:
(define pse (add-support-compute lfb))
(pse 'cache-all) ; Finished left norm marginals in 21 secs
(define fsc (add-subtotal-filter pse 500 10 10 #f))
(fsc 'left-basis-size) ;; 1378 < 1381 smaller than 1407
(fsc 'right-basis-size) ;; 64211 < 64667 smaller than 68602
(length (fsc 'get-all-elts)) ;; 175420 < 176163 smaller than 181322
(define lfc (add-linkage-filter fsc))
(lfc 'left-basis-size) ;; 1378 < 1381 smaller than 1407
(lfc 'right-basis-size) ;; 64192 < 64256 smaler than 66057
(length (lfc 'get-all-elts)) ;; 175384 < 175446 smaler than 176427
(length (aset #f)) ; 1378 consistent, at last...
Because each iteration knocks out more. I guess that's OK...
(length wrd-lst) 1407
(length clist) 2808
(length unwanted-words) 6992
(length unwanted-cnctrs) 425 -- hmm seems small.
(length unwanted-conseqs) 2545
(length good-conseqs) 66057 so OK....
(define ok-conseq? (make-aset-predicate good-conseqs))
(define (pair-pred SECT) (ok-conseq? (gdr SECT)))
(define cnt 0)
(for-each (lambda (sec) (if (pair-pred sec) (set! cnt (+ cnt 1))))
(fsa 'get-all-elts))
cnt 201740 -- like before...
(define ok-con? (make-aset-predicate clist))
(define (ok-seq? seq) (every ok-con? (cog-outgoing-set seq)))
(for-each (lambda (seq)
(if (not (ok-seq? seq)) (format #t "faail ~A\n" seq)))
good-conseqs)
(for-each (lambda (sec)
(if (not (ok-seq? (gdr sec))) (format #t "faail ~A\n" sec)))
(lfa 'get-all-elts))
; all pass this double-check. WTF. Oh, a bug in filter 'get-all-pairs.
=========
Try again.
-- (sql-open "postgres:///en_micro_dj_cfive")
(psa 'fetch-pairs)
(sql-close)
(psa 'left-basis-size) ;; 10813
(psa 'right-basis-size) ;; 187063
(define psc (add-support-compute psa))
(psc 'cache-all) ; Finished left norm marginals in 185 secs
(define fsa (add-subtotal-filter psa 500 10 10 #f))
(fsa 'left-basis-size) ;; 1624
(fsa 'right-basis-size) ;; 187063
(define lfa (add-linkage-filter fsa))
(lfa 'left-basis-size) ;; 1624
(lfa 'right-basis-size) ;; 68602
(length (lfa 'get-all-elts)) ; 183644
; XXX (define lfb (add-linkage-filter-2pass fsa))
; XXX (lfb 'left-basis-size) ;; 1624
; XXX (lfb 'right-basis-size) ;; 68602 .. wtf ...
createdb en_micro_marg
cat /home/ubuntu/src/atomspace/opencog/persist/sql/multi-driver/atom.sql | psql en_micro_marg
(sql-open "postgres:///en_micro_marg")
(define fso (make-store lfa))
(fso 'store-all-elts) ; Marginals are not to be stored.
Done storing 183644 pairs in 285 secs
(sql-close)
; clean start.
(sql-open "postgres:///en_micro_marg")
(psa 'fetch-pairs) ;; Elapsed time to load csets: 19 secs
(psa 'left-basis-size) ;; 1610 ... well, smaller, it seems ...
(psa 'right-basis-size) ;; 67111 Huh.
(length (psa 'get-all-elts)) ;; 183644 OK
(batch-all-pair-mi psa) ;; MI computations; this took 0.17 hours
(define btr (batch-transpose psa))
(btr 'mmt-marginals) ;;
guile-micro-en> (print-matrix-summary-report psa)
Summary Report for Correlation Matrix Word-Disjunct Pairs (Connector
Sets)
Left type: WordNode Right Type: ConnectorSeq Pair Type: Section
Wildcard: (ListLink (ctv 0 0 14751096)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 1610 Columns: 67111
Size: 183644 non-zero entries of 108048710 possible
Fraction non-zero: 1.6996E-3 Sparsity (-log_2): 9.2006
Total observations: 14751096.0 Avg obs per pair: 80.324
Entropy Total: 15.167 Left: 12.043 Right: 7.7236
Total MI: 4.6004
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 87.72 1557.
Count (l_1) 2.7723E+4 2.3734E+5 316.0 152.4
Length (l_2) 1.0224E+4 3.0318E+4 116.6 19.47
RMS Count 9851. 2.9865E+4 112.3 19.18
MM^T support=183644.0 count=408941670158.0 entropy=-3.600
No M^TM data present
(cog-report-counts)
((PredicateNode . 14) (ListLink . 68722) (AnchorNode . 1) (Connector .
3226) (ConnectorDir . 2) (ConnectorSeq . 67111) (Section . 183644)
(SchemaNode . 1) (WordNode . 1623) (AnyNode . 2))
========
And now the mini:
-- mini_dj_cfive:
(psa 'left-basis-size) ;; 18533
(psa 'right-basis-size) ;; 647988
(length (psa 'get-all-elts)) ;; 1063939
(define fsa (add-subtotal-filter psa 40 5 5 #f))
(fsa 'left-basis-size) ;; 7556
(fsa 'right-basis-size) ;; 647988
(length (fsa 'get-all-elts)) ;; 1045903
(define lfa (add-linkage-filter fsa))
(lfa 'left-basis-size) ;; 7556
(lfa 'right-basis-size) ;; 271344
(length (lfa 'get-all-elts)) ; 608241
createdb en_mini_marg
cat /home/ubuntu/src/atomspace/opencog/persist/sql/multi-driver/atom.sql |
psql en_mini_marg
(sql-open "postgres:///en_mini_marg")
(define fso (make-store lfa))
(fso 'store-all-elts) ; Marginals are not to be stored.
Done storing 608241 pairs in 1160 secs
(sql-close)
(sql-open "postgres:///en_mini_marg")
(psa 'fetch-pairs) ;; Elapsed time to load csets: 19 secs
(psa 'left-basis-size) ;; 7385
(psa 'right-basis-size) ;; 269711
(length (psa 'get-all-elts)) ;; 608241
(batch-all-pair-mi psa) ;; this took 0.73 hours
(define btr (batch-transpose psa))
(btr 'mmt-marginals) ;; 269711 left-wilds in 377 secs
(print-matrix-summary-report psa)
> (print-matrix-summary-report psa)
Summary Report for Correlation Matrix Word-Disjunct Pairs (Connector Sets)
Left type: WordNode Right Type: ConnectorSeq Pair Type: Section
Wildcard: (ListLink (ctv 0 0 20817076)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 7385 Columns: 269711
Size: 608241 non-zero entries of 1991815735 possible
Fraction non-zero: 3.0537E-4 Sparsity (-log_2): 11.677
Total observations: 20817076.0 Avg obs per pair: 34.225
Entropy Total: 16.792 Left: 13.687 Right: 8.0653
Total MI: 4.9602
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 173.1 7409.
Count (l_1) 2.2831E+4 3.6516E+5 131.9 49.28
Length (l_2) 7585. 3.1013E+4 43.82 4.186
RMS Count 7399. 3.0766E+4 42.74 4.152
MM^T support=608241.0 count=475267638744.0 entropy=-0.361
No M^TM data present
==================
Again with cosines....
createdb en_micro_fuzz -T en_micro_marg
(gram-classify-greedy-fuzz 0.65 0.3 4)
Start greedy-agglomeration of 1610 words
Existing classes=0 singletons=0 done=0
--- To-do=1610 ncls=0 sing=0 nredo=0 2019-06-15 05:28:53 -- "." ---
--- To-do=0 ncls=135 sing=540 nredo=1062 2019-06-15 11:12:17 -- "()" ---
So .. not quite 6 hours ... RSS in top is 600 MBytes
((ListLink . 68735) (MemberLink . 1639) (Connector . 3226)
(ConnectorSeq . 67111) (Section . 217212) (WordNode . 1623)
(WordClassNode . 135))
So no similarity links created... Hmm.
OK,
createdb en_micro_fuzz_exp -T en_micro_fuzz
(sql-open "postgres:///en_micro_fuzz_exp")
(define asc (add-singleton-classes psa))
(asc 'create-hi-count-singles 1) 1575 of 1575 -- borken-ish.
(psa 'left-basis-size) ; 1575
are there any unclassed words?
(define aset (make-atom-set))
(for-each (lambda (wcn)
(for-each (lambda (memb) (aset (gar memb)))
(cog-incoming-by-type wcn 'MemberLink)))
(cog-get-atoms 'WordClassNode))
(length (aset #f)) ; 1070 -- so that many words in word-classes.
(cog-get-atoms 'AnchorNode)
OK, bug fixed...
(define gca (make-gram-class-api))
(define gcs (add-pair-stars gca))
(define gcf (add-wordclass-filter gcs))
(batch-all-pair-mi gcf)
Summary Report for Correlation Matrix WordClass-Disjunct Pairs wordclass-filter
Left type: WordClassNode Right Type: ConnectorSeq Pair Type: Section
Wildcard: (ListLink (ctv 0 0 12003313)
(AnyNode "gram-class-word")
(AnyNode "gram-class-disjunct")
)
Rows: 675 Columns: 67056
Size: 142358 non-zero entries of 45262800 possible
Fraction non-zero: 3.1451E-3 Sparsity (-log_2): 8.3127
Total observations: 12003312.5 Avg obs per pair: 84.318
Entropy Total: 13.746 Left: 11.431 Right: 6.3415
Total MI: 4.0264
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 50.64 2817.
Count (l_1) 3.2160E+4 3.3795E+5 635.0 120.0
Length (l_2) 1.9406E+4 6.0704E+4 383.2 21.55
RMS Count 1.8989E+4 +nan.0 374.9 +nan.0
No MM^T data present
No M^TM data present
(export-csets gcf "dict.db" "EN_us")
Will store 142358 csets
==========
createdb en_micro_discrim -T en_micro_marg
(psa 'fetch-pairs)
(gram-classify-greedy-discrim 0.5 4)
Start greedy-agglomeration of 1610 words
Existing classes=0 singletons=0 done=0
--- To-do=1610 ncls=0 sing=0 nredo=0 2019-06-15 20:56:12 -- "." ---
--- To-do=0 ncls=124 sing=292 nredo=1303 2019-06-16 00:29:19 -- "()" ---
So that took 3.5 hours -- faster than fuzz. Didn't seem as goood...
(define asc (add-singleton-classes psa))
(asc 'create-hi-count-singles 1) ; 1562
(define gca (make-gram-class-api))
(define gcs (add-pair-stars gca))
(define gcf (add-wordclass-filter gcs))
(batch-all-pair-mi gcf)
(use-modules (opencog nlp lg-export))
(export-csets gcf "dict.db" "EN_us")
Summary Report for Correlation Matrix WordClass-Disjunct Pairs
Left type: WordClassNode Right Type: ConnectorSeq Pair Type:
Section
Wildcard: (ListLink (ctv 0 0 11428005)
(AnyNode "gram-class-word")
(AnyNode "gram-class-disjunct")
)
Rows: 416 Columns: 67056
Size: 132775 non-zero entries of 27895296 possible
Fraction non-zero: 4.7598E-3 Sparsity (-log_2): 7.7149
Total observations: 11428005.3 Avg obs per pair: 86.070
Entropy Total: 13.230 Left: 11.268 Right: 5.5832
Total MI: 3.6218
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 36.30 4172.
Count (l_1) 3.3309E+4 4.8650E+5 917.7 116.6
Length (l_2) 2.2783E+4 8.5731E+4 627.7 20.55
RMS Count 2.2287E+4 +nan.0 614.0 +nan.0
No MM^T data present
No M^TM data present
==========
createdb en_micro_rediscrim -T en_micro_marg
(psa 'fetch-pairs)
(gram-classify-greedy-discrim 0.5 4)
--- To-do=1610 ncls=0 sing=0 nredo=0 2019-06-24 05:10:25
--- To-do=0 ncls=124 sing=292 nredo=1303 2019-06-24 08:38:22
Three & half hours ...
(sql-open "postgres:///en_micro_rediscrim_exp")
(define asc (add-singleton-classes psa))
(asc 'create-hi-count-singles 1)
(define gca (make-gram-class-api))
(define gcs (add-pair-stars gca))
(define gcf (add-wordclass-filter gcs))
(batch-all-pair-mi gcf)
(define sto (make-store gcf))
(sto 'store-all-elts)
(sto 'store-wildcards)
(gcs 'store-aux)
(sql-close)
==========
Review meaning ...
why long pause after first Bingo! ?
partial answer: it hits postgres hard. Is it updating counts ??
assign-word-to-class calls MERGER 'merge-function
which calls merge-project
... should add timers to line 598, etc, for merge-project to see ...
==========
createdb en_mini_fuzz -T en_mini_marg
(gram-classify-greedy-fuzz 0.65 0.3 4)
To-do=0 ncls=366 sing=3326 nredo=3973 2019-06-17 19:25:00 -- "()" ---
Atomspace holds 1340302 atoms
Time since stats reset=220564 at Sat Jun 15 06:14:32 2019
So this took... 61 hours == 2.5 days to process.
So...
366 classes 4176 classified words
createdb en_mini_fuzz_tmpsav -T en_mini_fuzz
ALTER DATABASE en_mini_fuzz RENAME TO en_mini_fuzz_exp;
ALTER DATABASE en_mini_fuzz_tmpsav RENAME TO en_mini_fuzz;
(define asc (add-singleton-classes psa))
(asc 'create-hi-count-singles 1)
3326 unclassed words so now 3692 WordClasses ...
(define gca (make-gram-class-api))
(define gcs (add-pair-stars gca))
(define gcf (add-wordclass-filter gcs))
(batch-all-pair-mi gcf)
(use-modules (opencog nlp lg-export))
(export-csets gcf "dict.db" "EN_us") ; Will store 501957 csets
==========
createdb en_micro_fuzzier -T en_micro_marg
(psa 'fetch-pairs)
(gram-classify-greedy-fuzz 0.65 0.6 4)
To-do=1610 ncls=0 sing=0 nredo=0 2019-06-19 16:32:41
To-do=0 ncls=136 sing=544 nredo=1062 2019-06-19 20:28:34
So, four hours. Not bad.
(sql-close)
createdb en_micro_fuzzier_exp
cat atom.sql | psql en_micro_fuzzier_exp
(sql-open "postgres:///en_micro_fuzzier_exp")
(define asc (add-singleton-classes psa))
(asc 'create-hi-count-singles 1) ; Created 544 singleton word classes
(define gca (make-gram-class-api))
(define gcs (add-pair-stars gca))
(define gcf (add-wordclass-filter gcs))
(batch-all-pair-mi gcf)
(define sto (make-store gcf))
(sto 'store-all-elts)
(sto 'store-wildcards)
(gcs 'store-aux)
(sql-close)
restart, (gcs 'fetch-pairs)
(use-modules (opencog nlp lg-export))
(export-csets gcs "dict.db" "EN_us") ; Will store 142938 csets
dict.db-micro-fuzzier-unk
==========
createdb en_micro_fizz -T en_micro_marg
(psa 'fetch-pairs)
(gram-classify-greedy-fuzz 0.75 0.3 4)
--- To-do=1610 ncls=0 sing=0 nredo=0 2019-06-19 17:46:11 -- "." ---
--- To-do=0 ncls=115 sing=833 nredo=773 2019-06-19 21:49:41
Four hours. Not bad.
(sql-close)
createdb en_micro_fizz_exp
cat atom.sql | psql en_micro_fizz_exp
(sql-open "postgres:///en_micro_fizz_exp")
(define asc (add-singleton-classes psa))
(asc 'create-hi-count-singles 1) ; Created 833 singleton word classes
(define gca (make-gram-class-api))
(define gcs (add-pair-stars gca))
(define gcf (add-wordclass-filter gcs))
(batch-all-pair-mi gcf)
(define sto (make-store gcf))
(sto 'store-all-elts)
(sto 'store-wildcards)
(gcs 'store-aux)
(sql-close)
restart, (gcs 'fetch-pairs)
(use-modules (opencog nlp lg-export))
(export-csets gcs "dict.db" "EN_us") ; Will store 155391 csets
mv dict.db dict.db-micro-fizz-unk
==========
createdb en_micro_diss -T en_micro_marg
(psa 'fetch-pairs)
(gram-classify-greedy-discrim 0.65 4)
--- To-do=1610 ncls=0 sing=0 nredo=0 2019-06-19 18:06:56 -- "." ---
--- To-do=0 ncls=135 sing=537 nredo=1064 2019-06-19 22:07:32
four hours...
(sql-close)
createdb en_micro_diss_exp
cat atom.sql | psql en_micro_diss_exp
(sql-open "postgres:///en_micro_diss_exp")
(define asc (add-singleton-classes psa))
(asc 'create-hi-count-singles 1) ; Created 537 singleton word classes
(define gca (make-gram-class-api))
(define gcs (add-pair-stars gca))
(define gcf (add-wordclass-filter gcs))
(batch-all-pair-mi gcf)
(define sto (make-store gcf))
(sto 'store-all-elts)
(sto 'store-wildcards)
(gcs 'store-aux)
(sql-close)
restart, (gcs 'fetch-pairs)
(use-modules (opencog nlp lg-export))
(export-csets gcs "dict.db" "EN_us") ; Will store 142238 csets
mv dict.db dict.db-micro-diss-unk
==========
createdb en_micro_dissier -T en_micro_marg
(psa 'fetch-pairs)
(gram-classify-greedy-discrim 0.75 4)
--- To-do=1610 ncls=0 sing=0 nredo=0 2019-06-19 21:05:25
--- To-do=0 ncls=115 sing=833 nredo=772 2019-06-20 01:11:41
four hours...
(sql-close)
createdb en_micro_dissier_exp
cat atom.sql | psql en_micro_dissier_exp
(sql-open "postgres:///en_micro_dissier_exp")
(define asc (add-singleton-classes psa))
(asc 'create-hi-count-singles 1) ; Created 833 singleton word classes
(define gca (make-gram-class-api))
(define gcs (add-pair-stars gca))
(define gcf (add-wordclass-filter gcs))
(batch-all-pair-mi gcf)
(define sto (make-store gcf))
(sto 'store-all-elts)
(sto 'store-wildcards)
(gcs 'store-aux)
(sql-close)
restart, (gcs 'fetch-pairs)
(use-modules (opencog nlp lg-export))
(export-csets gcs "dict.db" "EN_us") ; Will store 155460 csets
mv dict.db dict.db-micro-dissier-unk
==========
createdb en_micro_dissiest -T en_micro_marg
(psa 'fetch-pairs)
(gram-classify-greedy-discrim 0.85 4)
--- To-do=1610 ncls=0 sing=0 nredo=0 2019-06-19 22:20:54
--- To-do=0 ncls=85 sing=1175 nredo=429 2019-06-20 02:45:29
Four plus hours.
(sql-close)
createdb en_micro_dissiest_exp
cat atom.sql | psql en_micro_dissiest_exp
(sql-open "postgres:///en_micro_dissiest_exp")
(define asc (add-singleton-classes psa))
(asc 'create-hi-count-singles 1) ; Created 1175 singleton word classes
(define gca (make-gram-class-api))
(define gcs (add-pair-stars gca))
(define gcf (add-wordclass-filter gcs))
(batch-all-pair-mi gcf)
(define sto (make-store gcf))
(sto 'store-all-elts)
(sto 'store-wildcards)
(gcs 'store-aux)
(sql-close)
restart, (gcs 'fetch-pairs)
(use-modules (opencog nlp lg-export))
(export-csets gcs "dict.db" "EN_us") ; Will store 167842 csets
mv dict.db dict.db-micro-dissiest-unk
==========
createdb en_micro_disinfo -T en_micro_marg
(psa 'fetch-pairs)
(gram-classify-greedy-disinfo 2.0 4)
Ooops -- Bingo! Dist=+inf.0 for class "was is" -- "he"
duuude (WordClassNode "was is") = 0.0 and (WordNode "be") =2820944933.0
duuuude xy=3006719756504542000.0 0.0 lo =+inf.0
---------Bingo! Dist=+inf.0 for class "was is" -- "be"
---- Remaining count = 0.0 for (WordNode "be")
Need to create marginals for the post-merge ... (BTW, cosine???)
i.e. need ...(trans-obj 'mmt-count word-class) to be correct...
(and the other two to be revised)
add-transpose-api -- transobj 'set-mmt-norms SUP CNT
store-atom ...
add-pair-cosine-compute 'right-cosine
(strftime "%c" (localtime (current-time)))
duude got pairs len=4295 Sun 23 Jun 2019 10:21:36 PM UTC
duude done merge Sun 23 Jun 2019 10:37:56 PM UTC
(/ 4295 980.) == 4.38 sections/second ... why so slow?
(mio 'mmt-fmi (Word "this") (Word "this")) << 4.034011726028252
(suppt-obj 'right-length (Word "this")) << 8404.05687748483
(trans-obj 'mmt-count (Word "this")) << 1327823651.0
(trans-obj 'total-mmt-count) << 408941670158.0
(/ (* 8404.05687748483 8404.05687748483 408941670158.0) (* 1327823651.0 1327823651.0))
16.381683554400546
(/ (log 16.381683554400546) (log 2)) << 4.034011726028254 SO OK, that
works.
hang on ...
(define mio (add-symmetric-mi-compute psa))
(mio 'mmt-fmi (WordClass "it he") (Word "but"))
it he but
--- Greedy-checking next 200 items
duuude as for xy= 0.0 and 0.0
duuuude its -inf.0 for -- with
duuude as for xy= 0.0 and 0.0
duuuude its -inf.0 for -- I
duuude as for xy= 0.0 and 0.0
duuuude its -inf.0 for -- ;
Ahhh:
duuude wtf marga= 0.0 for (ListLink
(WordClassNode ", ;")
(AnyNode "cset-disjunct")
)
which is wrong...
assign-word-to-class is returning ... ohh bug ...
Argh... (WordClassNode "the a") (WordNode "from")
(mio 'mmt-fmi (WordClassNode "the a") (WordNode "from"))
+nan.0+4.532360141827193i
(pti 'mmt-count (WordNode "from"))
(cog-incoming-size-by-type (WordNode "from") 'Section)
Ahhh! all gone!
==========
createdb en_micro_disinfo -T en_micro_marg
(psa 'fetch-pairs)
(gram-classify-greedy-disinfo 2.0 4)
--- To-do=1610 ncls=0 sing=0 nredo=0 2019-06-24 05:08:28
--- To-do=0 ncls=34 sing=57 nredo=1499 2019-06-24 09:02:15
Four hours ...
---------Bingo! Dist=2.0813 for word "of" -- "in"
---------Bingo! Dist=2.0510 for class "of in" -- "at"
---------Bingo! Dist=2.1541 for class "of in" -- "on"
---------Bingo! Dist=2.1868 for class "of in" -- "from"
---------Bingo! Dist=2.0750 for class "of in" -- "by"
---------Bingo! Dist=2.0555 for class "of in" -- "into"
---------Bingo! Dist=2.1758 for class "of in" -- "upon"
---------Bingo! Dist=2.1062 for class "of in" -- "through"
I assume the appallingly low MI is due to ... micro??
---------Bingo! Dist=2.7938 for word "the" -- "a"
---------Bingo! Dist=3.5941 for class "the a" -- "his"
---------Bingo! Dist=3.3722 for class "the a" -- "this"
(sql-open "postgres:///en_micro_disinfo_exp")
(define asc (add-singleton-classes psa))
(asc 'create-hi-count-singles 1) ; Created 57 singleton word classes
Wow!
(define gca (make-gram-class-api))
(define gcs (add-pair-stars gca))
(define gcf (add-wordclass-filter gcs))
(batch-all-pair-mi gcf)
(define sto (make-store gcf))
(sto 'store-all-elts)
(sto 'store-wildcards)
(gcs 'store-aux)
(sql-close)
shutdown, restart
(gcs 'fetch-pairs)
(use-modules (opencog nlp lg-export))
(export-csets gcs "dict.db-micro-disinfo" "EN_us")
==========
createdb en_micro_disinfo3 -T en_micro_marg
(psa 'fetch-pairs)
(gram-classify-greedy-disinfo 3.0 4)
--- To-do=0 ncls=70 sing=209 nredo=1381 2019-06-24 17:29:54
Created 209 singleton word classes
(gcs 'fetch-pairs)
(use-modules (opencog nlp lg-export))
(export-csets gcs "dict.db-micro-disinfo3" "EN_us")
==========
createdb en_mini_discrim -T en_mini_marg
(gram-classify-greedy-discrim 0.5 4)
After triming, 7385 words left, out of 7548
Sorting in 2.028 seconds
Start greedy-agglomeration of 7385 words
Existing classes=0 singletons=0 done=0
--- To-do=7385 ncls=0 sing=0 nredo=0 2019-06-18 03:36:40 -- "." ---
--- To-do=0 ncls=371 sing=2097 nredo=5089 2019-06-20 19:57:15
64 hours .... Ouch.
(sql-close)
createdb en_mini_discrim_exp
cat atom.sql | psql en_mini_discrim_exp
(sql-open "postgres:///en_mini_discrim_exp")
(define asc (add-singleton-classes psa))
(asc 'create-hi-count-singles 1) ; Created 2097 singleton word classes
(define gca (make-gram-class-api))
(define gcs (add-pair-stars gca))
(define gcf (add-wordclass-filter gcs))
(batch-all-pair-mi gcf)
(define sto (make-store gcf))
(sto 'store-all-elts)
(sto 'store-wildcards)
(gcs 'store-aux)
(sql-close)
restart, (gcs 'fetch-pairs)
(use-modules (opencog nlp lg-export))
(export-csets gcs "dict.db" "EN_us") ; Will store 498582 csets
mv dict.db dict.db-mini-discrim-unk
======
(load "/tmp/c.scm")
(lg-compare (LgDictNode "en") (LgDictNode "micro-fuzz") "I saw her face")
(define compare (make-lg-comparator (LgDictNode "en") (LgDictNode "micro-fuzz")))
(compare "I saw her face")
(compare "I swooned to the floor")
(compare #f)
LinkGrammarRelationshipNode
Argh.
Segmentation fault. linkage/sane.c:76
76 path[gwordlist_len(path)-1]->sent_wordidx)
(gdb) print path
$1 = (const Gword **) 0x0
======
micro-fuzz with child-directed.txt:
Finished comparing 3037 parses (13567 words, 11183 links)
Dictionary did not contain word for 2031 sentences
Dictionary was missing 109 words
Found 0 length-miscompares
Found 0 word-miscompares
Found 6280 link-count miscompares with 6173 missing and 584 extra links
Missing link-type counts: ((IDBYE . 1) (WR . 2) (BW . 4) (DG . 2) (GN .
1) (IDBIT . 1) (H . 5) (AL . 2) (AF . 3) (IDLE . 2) (BI . 3) (X . 11) (U
. 4) (IDQA . 2) (IDBKD . 7) (PF . 13) (V . 9) (MX . 1) (IDLR . 6) (IDPZ
. 8) (LI . 1) (DD . 6) (RS . 2) (OF . 7) (EE . 3) (C . 17) (QI . 4) (E .
37) (PP . 21) (SJ . 39) (EA . 27) (TO . 24) (SFI . 4) (EL . 24) (VJ .
20) (WV . 72) (W . 36) (K . 106) (G . 8) (CV . 37) (YS . 78) (NS . 1)
(EB . 24) (Q . 136) (AN . 94) (CO . 170) (SF . 82) (B . 213) (R . 97) (A
. 93) (PH . 25) (P . 480) (SI . 483) (M . 116) (MV . 325) (I . 564) (J .
399) (D . 1052) (O . 1717) (S . 1254) (IV . 95))
Missing words: (spoon cat lots eating truck let's build baby rabbit gail
kiss yeah doesn't push duck today tie daddy funny stick lena doll phone
balloons meat egg steak puppy bite cindy alice 're kay nice likes zipper
lift wanna nose can't pull shoes bye 've a@l feed comb 's toys 'll judy
fun mirror knock hang button playing bunny hi dolly puppet dale throw
okay calling smell sing snap knocked balloon turtle telephone color
stack butterfly pat shoe kitty finger mommy scratchy blocks apple pick
dragon doggie clock count brush horsie tower cup sleeping mouse untie
pig zip ear block peekaboo honey toy cow cage gonna ducks pocketbook
monkey paul)
But there are repeated sentences...
Revised:
Finished comparing 1832 parses (9274 words, 7955 links)
Dictionary did not contain word for 1270 sentences
Dictionary was missing 108 words
Found 0 length-miscompares
Found 0 word-miscompares
Found 4381 link-count miscompares with 4429 missing and 380 extra links
Missing link-type counts: ((MX . 1) (WR . 2) (NS . 1) (IDLE . 2) (U . 4)
(BW . 2) (X . 7) (IDLR . 5) (DG . 2) (IDQA . 2) (BI . 3) (EE . 3) (AF .
3) (IDBKD . 7) (LI . 1) (AL . 1) (DD . 6) (H . 4) (IDBIT . 1) (IDBYE .
1) (PP . 14) (CV . 36) (C . 17) (GN . 1) (OF . 6) (IDPZ . 8) (QI . 4) (V
. 8) (RS . 2) (VJ . 20) (YS . 63) (EA . 25) (TO . 24) (EB . 19) (IV .
92) (SFI . 4) (Q . 54) (EL . 23) (I . 464) (AN . 81) (PH . 22) (SF . 67)
(P . 324) (SI . 309) (W . 22) (CO . 126) (M . 111) (WV . 49) (J . 347)
(PF . 9) (E . 29) (B . 142) (R . 61) (MV . 305) (K . 93) (O . 1023) (S .
641) (G . 7) (SJ . 33) (A . 79) (D . 878))
Missing words: (spoon cat lots eating truck let's build baby rabbit gail
kiss yeah doesn't push duck today tie daddy funny stick lena doll phone
balloons meat steak egg puppy bite cindy alice 're kay nice likes zipper
lift wanna nose can't pull shoes bye feed 've comb 's toys 'll judy fun
mirror knock hang button playing bunny hi dolly puppet dale throw okay
calling smell sing snap knocked balloon turtle telephone color stack
butterfly pat shoe kitty finger mommy scratchy blocks apple pick dragon
doggie clock count brush horsie tower cup sleeping mouse untie pig zip
ear block peekaboo honey toy cow cage gonna ducks pocketbook monkey
paul)
Finished verifying dicationary "micro-fuzz" with sentences from
"child-directed-speech.txt"
again, without the missing words:
Examined 1832 sentences; of these, 1270 had words not in the dictionary.
Finished comparing 562 parses (2738 words, 2376 links)
Dictionary was missing 108 words
Found 0 length-miscompares
Found 0 word-miscompares
Found 1050 link-count miscompares with 983 missing and 193 extra links
So, by hand:
expected positives = P = 2376 (the expected links)
of these, got (- 2376 983) == 1393 links = TP
so recall = (/ 1393 2376) = TP/P = 0.5863
including extras, got 1393+193 = 1586 links total
extras = false-positives = FP == 193
so precision = TP/(TP+FP) = (/ 1393 1586) = 0.8783
F1 = harmonic-mean(precision, recall) =
(* 2 0.5863 0.8783) / (+ 0.5863 0.8783) == 0.7032
Automated:
Examined 1832 sentences; 1270 had words not in dictionary (69.323 %).
Finished comparing 562 parses; 482 parsed differently (85.765 %).
Found 2738 word instances, 176 words; expected to find 2376 links
Dictionary was missing 108 words
Found 0 length-miscompares
Found 0 word-miscompares
Found 1050 words w/linkage diffs; 983 missing and 193 extra links
Link precision=0.8783 recall=0.5863 F1=0.7032
Missing link-type counts: ((O . 213) (I . 165) (SI . 126) (MV . 102) (D
. 96) (B . 78) (J . 73) (P . 60) (S . 54) (IV . 42) (R . 34) (CO . 24)
(K . 22) (M . 19) (EL . 15) (E . 15) (WV . 14) (CV . 8) (TO . 8) (X . 7)
(A . 7) (VJ . 7) (Q . 6) (SJ . 6) (IDLR . 5) (C . 5) (W . 5) (IDBKD . 4)
(PF . 4) (U . 3) (DD . 3) (EA . 3) (SFI . 3) (OF . 2) (EE . 2) (SF . 2)
(H . 2) (IDPZ . 2) (QI . 2) (IDQA . 1) (IDLE . 1) (PP . 1) (BW . 1) (AF
. 1) (RS . 1) (EB . 1))
Missing words: (spoon cat lots eating truck let's build baby rabbit gail
kiss yeah doesn't push duck today tie daddy funny stick lena doll phone
balloons meat steak egg puppy bite cindy alice 're kay nice likes zipper
lift wanna nose can't pull shoes bye feed 've comb 's toys 'll judy fun
mirror knock hang button playing bunny hi dolly puppet dale throw okay
calling smell sing snap knocked balloon turtle telephone color stack
butterfly pat shoe kitty finger mommy scratchy blocks apple pick dragon
doggie clock count brush horsie tower cup sleeping mouse untie pig zip
ear block peekaboo honey toy cow cage gonna ducks pocketbook monkey
paul)
Finished verifying dicationary "micro-fuzz" with sentences from
"child-directed-speech.txt"
========
Examined 579 sentences; 497 had words not in dictionary (85.838 %).
Finished comparing 82 parses; 81 parsed differently (98.780 %).
Found 666 word instances, 209 words; expected to find 650 links
Dictionary was missing 696 words
Found 0 length-miscompares
Found 0 word-miscompares
Found 319 words w/linkage diffs; 324 missing and 59 extra links
Link precision=0.8468 recall=0.5015 F1=0.6300
Finished verifying dicationary "micro-fuzz" with sentences from
"corpus-basic.batch"
========
Examined 4236 sentences; 3460 had words not in dictionary (81.681 %).
Finished comparing 776 parses; 745 parsed differently (96.005 %).
Found 4901 word instances, 591 words; expected to find 4359 links
Dictionary was missing 3406 words
Found 2 length-miscompares
Found 7 word-miscompares
Found 2656 words w/linkage diffs; 2237 missing and 700 extra links
Link precision=0.7519 recall=0.4868 F1=0.5910
Finished verifying dicationary "micro-fuzz" with sentences from
"corpus-fixes.batch"
=======
Examined 229 sentences; 205 had words not in dictionary (89.520 %).
Finished comparing 24 parses; 24 parsed differently ( 100.0 %).
Found 169 word instances, 95 words; expected to find 156 links
Dictionary was missing 414 words
Found 0 length-miscompares
Found 0 word-miscompares
Found 83 words w/linkage diffs; 67 missing and 22 extra links
Link precision=0.8018 recall=0.5705 F1=0.6667
Finished verifying dicationary "micro-fuzz" with sentences from
"ull-golden.txt"
=====
Examined 2513 sentences; 2136 had words not in dictionary (84.998 %).
Finished comparing 377 parses; 376 parsed differently (99.735 %).
Found 3323 word instances, 527 words; expected to find 3220 links
Dictionary was missing 2582 words
Found 0 length-miscompares
Found 0 word-miscompares
Found 1911 words w/linkage diffs; 1616 missing and 529 extra links
Link precision=0.7520 recall=0.4981 F1=0.5993
Finished verifying dicationary "micro-fuzz" with sentences from
"ull-silver-short.txt"
====
Examined 2513 sentences; 1570 had words not in dictionary (62.475 %).
Finished comparing 943 parses; 941 parsed differently (99.788 %).
Found 9859 word instances, vocab= 1308 words; expect to find 9716 links
Dictionary was missing 1371 words
Found 0 length-miscompares
Found 0 word-miscompares
Found 5688 words w/linkage diffs; 4845 missing and 1627 extra links
Link precision=0.7496 recall=0.5013 F1=0.6008
Missing link-type counts: ((RW . 922) (MV . 598) (S . 563) (O . 540) (J
. 364) (D . 335) (CV . 267) (I . 234) (P . 201) (WV . 170) (X . 168) (VJ
. 164) (CO . 159) (M . 157) (C . 142) (A . 135) (E . 128) (W . 108) (B .
96) (SJ . 95) (IV . 70) (R . 61) (AJ . 51) (EA . 50) (TO . 38) (SI . 33)
(PP . 32) (K . 31) (G . 31) (PH . 24) (AN . 21) (TH . 17) (EE . 17) (EB
. 15) (QI . 14) (OF . 13) (VC . 12) (RS . 12) (MJ . 9) (L . 9) (Q . 8)
(DD . 8) (PF . 8) (MX . 7) (SF . 7) (EZ . 7) (LI . 6) (RJ . 5) (DT . 5)
(EL . 4) (N . 4) (IDBVS . 3) (EC . 3) (IDOT . 3) (AF . 3) (IDBHA . 3)
(BI . 3) (SX . 3) (Y . 3) (IDBRM . 2) (NI . 2) (IDBHK . 2) (IDBUB . 2)
(EN . 2) (IDCBH . 2) (DG . 2) (EF . 2) (SFI . 2) (Z . 2) (IDPN . 1)
(IDBVL . 1) (IDBHJ . 1) (IDCIN . 1) (DP . 1) (IDBRF . 1) (IDPD . 1) (WN
. 1) (IDPO . 1) (FM . 1) (BW . 1) (IDBJA . 1) (CX . 1) (IDCBT . 1)
(IDBEW . 1) (IDPH . 1) (IDLS . 1) (IDCGX . 1) (IDCGY . 1) (IDLR . 1) (V
. 1) (AL . 1) (IDKM . 1) (IDKN . 1) (JT . 1) (IDPP . 1) (IDPQ . 1) (IDQC
. 1) (OT . 1) (IDOV . 1) (IDCBS . 1) (IDBOO . 1) (IDBIY . 1) (IDBIZ . 1)
(ER . 1) (IDBJN . 1) (IDBCE . 1) (IDCDD . 1) (IDBTF . 1) (IDBSC . 1)
(IDCAU . 1) (NS . 1) (IDOP . 1) (IDBGJ . 1) (XJ . 1) (AA . 1) (HA . 1)
(ND . 1) (IDBJG . 1) (IDBJH . 1))
Finished verifying dictionary "mini-fuzz" with sentences from
"ull-silver-short.txt"
real 68m18.787s
====
(gcs 'fetch-pairs)
(use-modules (opencog nlp lg-export))
(export-csets gcs "dict.db" "EN_us") ; 142420
line 149
(WordNode "an") .. not a member of anything ... how did that happen?
it happened because its not a germ .... ohhh we need to use
the filter during export....
Argh not all sections exported .. wtf ... there are (Section . 142420)
in the atomspace, but "Will store 81700 csets" ... !?
(gcs 'right-basis-size) ; 67104
(gcf 'right-basis-size) ; 33910
fu... not enough, need to pcs 'fetch-pairs too !?
this loads "an" sections, but still no MemberLink ...
(define asc (add-singleton-classes psa))
(asc 'create-hi-count-singles 1)
ahhh ...OK. so lets try it ...
(define nomem
(filter (lambda (cls) (equal? 0
(length (cog-incoming-by-type cls 'MemberLink))))
(cog-get-atoms 'WordClassNode)))
(cog-incoming-by-type (Word "an") 'MemberLink)
(cog-incoming-by-type (WordClass "an#uni") 'MemberLink)
OK, so that's fixed, but now no MI for
(Section (ctv 1 0 4.2)
(WordClassNode "time way")
(ConnectorSeq
(Connector (WordNode "acid") (ConnectorDir "-"))))
(cog-keys sec) only has (PredicateNode "*-TruthValueKey-*")
why? Others have (PredicateNode "*-FrequencyKey-*")
(PredicateNode "*-TruthValueKey-*")
(PredicateNode "*-Mutual Info Key-*")
select * from atoms where name='acid';
WordNode "acid" -- 6035
(ConnectorDir "-") == 16
(WordClassNode "time way") == 380120
select * from atoms where outgoing='{6035,16}';
connector == 6036
seq = 6037
sect = 382615
*-Mutual Info Key-* -- 13
select * from valuations where atom=382615;
OK, so database literally does not have it
7 12 34
Failure to store the MI's for ... some atoms. Crap.
Failure to store singleton memberships ... Crap.
needed to gcs->store-something!?
=============
in the mini-fuzz --
Length miscompare: 9 vs 8
Finish compare of sentence 1275/3059: "Yes, please, I'll have one."
Length miscompare: 8 vs 7
Finish compare of sentence 1290/3079: "That's what I think, yes."
=============
Bug: explosive atomspace
is this the word-test?? No, its just the parses piling up.
Now what?
"It is a tokenizing parser."
Oh.
NumberNode WordInstanceNode PhraseNode LinkGrammarRelationshipNode
... but it is still leaking. How?
valgrind --leak-check=full \
--suppressions=/tmp/valgrind.boost.suppressions \
--suppressions=/tmp/valgrind.guile.suppressions \
--suppressions=/tmp/valgrind.logger.suppressions \
guile -s dict-comp.scm mini-fuzz x.txt
LGHaveDictEntry::evaluate
db_lookup_exp (read-sql.c:212) Ohhh stupid me.
---
==========
leak: 68m18.787s
delete: 75m23.688s
no-delete: 76m48.013s
==========
Wow. Unk word lookups are slow.... why? They should not be triggering
anything ...!
micro-fuzz: 0m19.788s
oh crash ...
Assertion (NULL != bs->exp) failed at
../../link-grammar/dict-sql/read-sql.c:260: Missing disjuncts for word
<UNKNOWN-WORD.759> <didn’t didn't>
select * from Morphemes where morpheme='<UNKNOWN-WORD>';
.schema Disjuncts
#:INCLUDE-MISSING #t
discrim 124
diss 135
fuzz 135
fuzzier 136
fizz -- 115
skirting with madness:
The strength was in the order of gerE > cotD > yfhP
hangups:
People ages 55 to 64 on average are out of work for more than 54 weeks,
while those ages 25 to 34 are unemployed for about 36 weeks, according
to figures from the Bureau of Labor Statistics.
He obtained the lease of the manor of Great Burstead Grange (near East
Horndon) from the Abbey of Stratford Langthorne, and purchased the manor
of Bayhouse in West Thurrock.
this awful threat calmed Bab 's anger at once , and she hastened to
introduce a safer subject .
The delicately lyrical tone of the cello contrasted with the fiercely percussive piano chords
Hundreds of young men, furious about the verdict in the Rodney King
case, looted stores in Los Angeles today
The strength was in the order of gerE > cotD > yfhP P2 > yfhP P1
Based on LG-English parses:
http://langlearn.singularitynet.io/data/aglushchenko_parses/GCB-FULL-ALE-dILEd-2019-04-10/context:2_db-row:1_f1-col:11_pa-col:6_word-space:discrete/dict_31916C_2019-04-08_0007.4.0.dict
Based on "sequential" parses:
http://langlearn.singularitynet.io/data/aglushchenko_parses/GCB-FULL-SEQ-dILEd-2019-05-16-94/GL_context:2_db-row:1_f1-col:11_pa-col:6_word-space:discrete/dict_31548C_2019-05-19_0007.4.0.dict
Based on "DNN-MI-lked MST-Parses":
http://langlearn.singularitynet.io/data/aglushchenko_parses/GCB-GUCH-SUMABS-dILEd-2019-05-21-94/GL_context:2_db-row:1_f1-col:11_pa-col:6_word-space:discrete/dict_34584C_2019-05-25_0007.4.0.dict
./A/Autoethnography -- 128
./H/History of Virginia -- 277
./A/Astor House Hotel (Shanghai) -- 103
28238 / 1154 =
COMMENT ON DATABASE en_micro_diss IS '(discrim 0.65)';
COMMENT ON DATABASE en_micro_diss_exp IS '(discrim 0.65) plus marginals';
COMMENT ON DATABASE en_micro_dissier IS '(discrim 0.75)';
COMMENT ON DATABASE en_micro_dissier_exp IS '(discrim 0.75) plus marginals';
COMMENT ON DATABASE en_micro_dissiest IS '(discrim 0.85)';
COMMENT ON DATABASE en_micro_dissiest_exp IS '(discrim 0.85) plus marginals';
COMMENT ON DATABASE en_micro_fizz IS '(fuzz 0.75 0.3)';
COMMENT ON DATABASE en_micro_fizz_exp IS '(fuzz 0.75 0.3) plus marginals';
COMMENT ON DATABASE en_micro_fuzzier IS '(fuzz 0.65 0.6)';
COMMENT ON DATABASE en_micro_fuzzier_exp IS '(fuzz 0.65 0.6) plus marginals';
=======
Length miscompare:
I'm going drinking. 5 vs 4
Maybe, I'm not sure. 8 vs 7
micro-fuzz:
Prec recall F1
0.705 0.473 0.566
0.627 0.371 0.466
0.534 0.399 0.457
0.577 0.485 0.527
0.545 0.407 0.466
micro-fuzz-unk:
0.660 0.541 0.595
0.555 0.440 0.491
0.502 0.459 0.479
0.484 0.468 0.476
0.492 0.445 0.467
mini-fuzz
0.710 0.480 0.573
0.580 0.376 0.456
0.540 0.399 0.459
0.504 0.362 0.421
0.531 0.392 0.451
micro-discrim:
0.679 0.542 0.603
0.595 0.412 0.487
0.529 0.476 0.501
0.565 0.534 0.549
0.546 0.490 0.522
mini-discrim
0.658 0.538 0.592
0.590 0.452 0.512
0.522 0.469 0.494
0.516 0.475 0.495
0.532 0.475 0.502
micro-discrim-unk:
0.651 0.580 0.614
0.571 0.482 0.523
0.496 0.490 0.493
0.475 0.497 0.485
0.507 0.491 0.499
115 vs 135
Actress Whoopi Goldberg and singer Michael Jackson attended the ceremony
../../link-grammar/parse/count.c:222
#0 0x00007fa689d2c213 in find_table_pointer (ctxt=0x55ea149f7b00, lw=4,
rw=6,
le=0x0, re=0x55ea22126170, null_count=0)
at ../../link-grammar/parse/count.c:222
#1 0x00007fa689d2c319 in table_lookup (ctxt=<optimized out>,
lw=<optimized out>, rw=<optimized out>, le=<optimized out>,
re=<optimized out>, null_count=<optimized out>)
at ../../link-grammar/parse/count.c:255
#2 0x00007fa689d2d512 in mk_parse_set
(mchxt=mchxt@entry=0x55ea144f3910,
ctxt=ctxt@entry=0x55ea149f7b00, lw=lw@entry=4, rw=rw@entry=6,
le=0x0,
re=0x55ea22126170, null_count=0, pex=0x55ea14729f80)
at ../../link-grammar/parse/extract-links.c:356
#3 0x00007fa689d2dbc7 in mk_parse_set
(mchxt=mchxt@entry=0x55ea144f3910,
ctxt=ctxt@entry=0x55ea149f7b00, lw=lw@entry=4, rw=rw@entry=9,
le=0x55ea2202c810, re=re@entry=0x55ea22180d90, null_count=0,
pex=0x55ea14729f80) at ../../link-grammar/parse/extract-links.c:483
#4 0x00007fa689d2dab9 in mk_parse_set
(mchxt=mchxt@entry=0x55ea144f3910,
ctxt=ctxt@entry=0x55ea149f7b00, lw=lw@entry=3, rw=rw@entry=9,
le=0x55ea22019b10, re=re@entry=0x55ea22180d90, null_count=0,
pex=0x55ea14729f80) at ../../link-grammar/parse/extract-links.c:547
In Vienna, Beethoven met someone who would later be greatly influenced by him: Franz Schubert
http://langlearn.singularitynet.io/data/aglushchenko_parses/GCB-FULL-ANY-dILEd-2019-05-09-94/GL_context:2_db-row:1_f1-col:11_pa-col:6_word-space:discrete/dict_36740C_2019-05-11_0007.4.0.dict
Token "this" matches:
this.672 3221 disjuncts
this.331 5037 disjuncts
the old beast was whinnying on his shoulder .
Token "the" matches:
the.54 2879150524 disjuncts
the.24 3397807 disjuncts
Token "old" matches:
old.71 3568091 disjuncts
Token "was" matches:
was.89 4809781 disjuncts
was.19 20761 disjuncts
Token "on" matches:
on.68 3259929866 disjuncts
on.24 3397807 disjuncts
on.17 22768 disjuncts
Token "his" matches:
his.24 3397807 disjuncts
Token "shoulder" matches:
shoulder.44 1274692269 disjuncts
shoulder.39 204008 disjuncts
shoulder.28 269503230 disjuncts
Token "." matches:
..81 509547141 disjuncts
..34 3730768318 disjuncts
----------
(length (links w)) -- 13
but pattern found only 8--
in the m of them -- midst midst o the
in the m of -- midst midst o the
In the m of these -- midst midst o the
the m of this -- "
in the m of these -- "
in the m of all -- ditto
the m of them -- ditto
in the m of crowd -- "
Confusion.
midst- is in 6 conseq's
9 sections contain midst- (sects-w-con)
sec words are: of all them this matches right-words
19 Sections contain midst
13 are linkable (unique)
(r-pre with zero matches not consideered....!?) ohh.
d.scm = blink matches 7 times
in unit test: (right-halves == 5
(right-words = 5
links == 5
--------
Other crazy shit:
en_mini_marg contains this:
$44 = (Section (ctv 1 0 7)
(WordNode "of")
(ConnectorSeq
(Connector (WordNode "one") (ConnectorDir "-"))
(Connector (WordNode "speak") (ConnectorDir "-"))
(Connector (WordNode "December") (ConnectorDir "+"))
(Connector (WordNode "them") (ConnectorDir "+"))
(Connector (WordNode "the") (ConnectorDir "+"))
(Connector (WordNode "gold") (ConnectorDir "+"))
(Connector (WordNode "iron") (ConnectorDir "+"))
(Connector (WordNode "iron") (ConnectorDir "+"))
(Connector (WordNode "iron") (ConnectorDir "+"))
))
is this for real, or a corrupt MST parse?
$44 = (Section (ctv 1 0 7)
(WordNode "of")
(ConnectorSeq
(Connector (WordNode "one") (ConnectorDir "-"))
(Connector (WordNode "speak") (ConnectorDir "-"))
(Connector (WordNode "December") (ConnectorDir "+"))
(Connector (WordNode "them") (ConnectorDir "+"))
(Connector (WordNode "the") (ConnectorDir "+"))
(Connector (WordNode "gold") (ConnectorDir "+"))
(Connector (WordNode "iron") (ConnectorDir "+"))
(Connector (WordNode "iron") (ConnectorDir "+"))
(Connector (WordNode "iron") (ConnectorDir "+"))
))
$47 = (Section (ctv 1 0 6)
(WordNode "of")
(ConnectorSeq
(Connector (WordNode "hatred") (ConnectorDir "-"))
(Connector (WordNode "hatred") (ConnectorDir "-"))
))
$49 = (Section (ctv 1 0 6)
(WordNode "of")
(ConnectorSeq
(Connector (WordNode "Province") (ConnectorDir "-"))
(Connector (WordNode "basis") (ConnectorDir "-"))
(Connector (WordNode "scope") (ConnectorDir "-"))
(Connector (WordNode "name") (ConnectorDir "-"))
(Connector (WordNode "Ontario") (ConnectorDir "+"))
(Connector (WordNode "Ontario") (ConnectorDir "+"))
(Connector (WordNode "Ontario") (ConnectorDir "+"))
(Connector (WordNode "the") (ConnectorDir "+"))
(Connector (WordNode "Ontario") (ConnectorDir "+"))
))
This sure looks like broken MST parses to me ...
$50 = (Section (ctv 1 0 7)
(WordNode "of")
(ConnectorSeq
(Connector (WordNode "warriors") (ConnectorDir "-"))
(Connector (WordNode "assistance") (ConnectorDir "-"))
(Connector (WordNode "cause") (ConnectorDir "-"))
(Connector (WordNode "throne") (ConnectorDir "-"))
(Connector (WordNode "the") (ConnectorDir "+"))
(Connector (WordNode "warriors") (ConnectorDir "+"))
(Connector (WordNode "the") (ConnectorDir "+"))
(Connector (WordNode "the") (ConnectorDir "+"))
(Connector (WordNode "these") (ConnectorDir "+"))
))
There seem to be a lot of these ... WTF.
(observe-mst "This is a long complicated sentence that is meant to test the craziness of the MST stage of the processing")
Ahh .. caught red-handed!
(Section
(WordNode "the")
(ConnectorSeq
(Connector
(WordNode "of")
(ConnectorDir "-")
)
(Connector
(WordNode "craziness")
(ConnectorDir "+")
)
(Connector
(WordNode "stage")
(ConnectorDir "+")
)
(Connector
(WordNode "stage")
(ConnectorDir "+")
)
)
)
(Section
(WordNode "is")
(ConnectorSeq
(Connector
(WordNode "This")
(ConnectorDir "-")
)
(Connector
(WordNode "This")
(ConnectorDir "-")
)
(Connector
(WordNode "that")
(ConnectorDir "-")
)
)
)
(Section
(WordNode "stage")
(ConnectorSeq
(Connector
(WordNode "the")
(ConnectorDir "-")
)
(Connector
(WordNode "the")
(ConnectorDir "-")
)
(Connector
(WordNode "of")
(ConnectorDir "+")
)
)
)
(
(((6 . (WordNode "complicated")) 7 . (WordNode "sentence")) . 4.544568075409504)
(((2 . (WordNode "This")) 7 . (WordNode "sentence")) . 2.5698179211813894)
(((2 . (WordNode "This")) 9 . (WordNode "is")
) . 4.376083014264772) (((2 . (WordNode "This")) 3 . (WordNode "is")
) . 4.376083014264772) (((2 . (WordNode "This")) 10 . (WordNode "meant")
) . 2.85136143940991) (((10 . (WordNode "meant")) 11 . (WordNode "to")
) . 2.995166614025761) (((11 . (WordNode "to")) 12 . (WordNode "test")
) . 2.4676368796347674) (((4 . (WordNode "a")) 7 . (WordNode "sentence")
) . 2.157432042482352) (((4 . (WordNode "a")) 5 . (WordNode "long")
) . 2.8337745270444152) (((12 . (WordNode "test")) 19 . (WordNode "of")
) . 2.076930367265735) (((18 . (WordNode "stage")) 19 . (WordNode "of")
) . 2.098831656855328) (((16 . (WordNode "the")) 18 . (WordNode "stage")
) . 2.3297106242932806) (((13 . (WordNode "the")) 18 . (WordNode "stage")
) . 2.3297106242932806) (((13 . (WordNode "the")) 14 . (WordNode "craziness")
) . 2.4141307658002162) (((14 . (WordNode "craziness")) 15 . (WordNode "of")
) . 1.6669586994997374) (((19 . (WordNode "of")) 20 . (WordNode "the")
) . 1.6553378921463375) (((8 . (WordNode "that")) 9 . (WordNode "is")
) . 1.5597634187783882) (((19 . (WordNode "of")) 21 . (WordNode "processing")
) . -0.24884483707755933))
(print-linkage link-seq)
6-7 complicated <--> sentence MI=4.5446
2-7 This <--> sentence MI=2.5698
2-9 This <--> is MI=4.3761
2-3 This <--> is MI=4.3761
2-10 This <--> meant MI=2.8514
10-11 meant <--> to MI=2.9952
11-12 to <--> test MI=2.4676
4-7 a <--> sentence MI=2.1574
4-5 a <--> long MI=2.8338
12-19 test <--> of MI=2.0769
18-19 stage <--> of MI=2.0988
16-18 the <--> stage MI=2.3297
13-18 the <--> stage MI=2.3297
13-14 the <--> craziness MI=2.4141
14-15 craziness <--> of MI=1.6670
19-20 of <--> the MI=1.6553
8-9 that <--> is MI=1.5598
19-21 of <--> processing MI=-0.249
(observe-mst "This is a long complicated sentence that is meant to test the craziness of the MST stage of the processing")
make-sections
wedge-get-left-overt
Found and fixed https://github.com/opencog/atomspace/pull/2252
Arghhh.
===========
MPG parser -- DONE
(define plain-text "This is a long complicated sentence that is meant to test the craziness of the MST stage of the processing")
(define word-strs (tokenize-text plain-text))
(define word-list (map WordNode word-strs))
(define mst-tree (mst-parse-atom-seq word-list trunc-scorer))
(print-linkage ordered-tree)
(print-linkage (sort-wedgelist (mpg-parse-atom-seq word-list trunc-scorer 20)))
1) cross-any does allow overlaps!! DONE
2) 1e30 reject has to be 1e20 DONE
(1 . (WordNode "###LEFT-WALL###") (5 . (WordNode "long")
===========
dropdb en_pairs_cfive en_pairs_cfive_mst
createdb en_pairs_cfive_mi_1.8
COMMENT ON DATABASE "en_pairs_cfive_mi_1.8" IS
'Same as en_pairs_cfive_mi but all pairs w/ FMI<1.8 removed.';
(define fpm (add-fmi-filter wps 1.8 #f))
(fpm 'left-basis-size)
(fpm 'right-basis-size) << unchanged ...
(length (fpm 'get-all-elts)) <<< 15795739 down from 27892737
(sql-open "postgres:///en_pairs_cfive_mi_1.8")
(define sfp (make-store fpm))
(sfp 'store-all)
Done storing 15795739 pairs in 17337 secs
--------
cleanup words that do not participate any longer ...
(cog-map-type (lambda (word)
(if (= 0 (length (filter
(lambda (wpr) (and
(not (equal? (cog-type (gar wpr)) 'AnyNode))
(not (equal? (cog-type (gdr wpr)) 'AnyNode))
))
(cog-incoming-by-type word 'ListLink))))
(format #t "Empty: ~A\n" (cog-name word))))
'WordNode)
None-such. Interesting ... OK.
--------
createdb en_pairs_mi_1.8tmp -T en_pairs_cfive_mi_1.8
(wps 'fetch-pairs)
(define fpm (add-fmi-filter wps 1.8 #t)) ;; <<< #t so filter-keys!
(batch-all-pair-mi fpm)
Going to compute and store individual pair MI
Single-threaded: << vs. multi-threaded ... no advantage
Start computing the basis
Support: found num left= 619168 num right= 580804 in 373 secs
Finished left norm marginals in 3587 secs << vs. 5266 multi is slower
Finished left totals in 38 secs
Finished right norm marginals in 8654 secs << vs 4658 harrrumpf
Finished right totals in 36 secs
Done with wild-card count N(x,*) and N(*,y) in 12315 secs
Total count N(*,*) = 470207380.0 = 470207380.0
Finished column (left) norm averages in 166 secs << vs 111
Finished row (right) norm averages in 185 secs << vs 131 when multi
Going to do individual pair frequencies
Done computing 15795739 pair frequencies in 595 secs << vs 3919 Yow!
Start computing log P(*,y)
Done computing 580804 left-wild log frequencies in 32 secs << vs 3003 Yow!
Done with -log P(*,y), start -log P(x,*)
Done computing 619168 right-wild log frequencies in 35 secs << vs 3448 Yow!
Stored 40000 of 580804 left-wilds in 34 secs (1176 pairs/sec)
Stored 80000 of 580804 left-wilds in 33 secs (1212 pairs/sec)
...
Done storing 580804 left-wilds in 505 secs
Done storing 619168 right-wilds in 542 secs
Done computing and saving -log P(x,*) and P(*,y)
Going to compute and store individual pair MI
Done 10001 of 619168 outer loops in 481 secs, pairs=238841 (496.55 pairs/sec)
Done 10000 of 619168 outer loops in 578 secs, pairs=238839 (413.22 pairs/sec)
Then from multi--
Finished left entropy subtotals in 3144 secs
Finished right entropy subtotals in 3490 secs
===========
The fpm --
======
Do it again, cutting at MI=5.3
createdb en_pairs_5.3_remi
cat atom.sql | psql en_pairs_5.3_remi
(define fpm (add-fmi-filter wps 5.3 #f))
(length (fpm 'get-all-elts)) << 8953449
(sql-open "postgres:///en_pairs_5.3_remi")
(define fps (make-store fpm))
(fps 'store-all-elts)
(sql-close)
(sql-open "postgres:///en_pairs_5.3_remi")
(wps 'fetch-pairs)
(batch-all-pair-mi wps)
(print-matrix-summary-report wps)
en_pairs_5.3_remi
-----------------
A copy of `en_pairs_cfive_mi`, but with all word-pairs with an MI<5.3
removed. After removal, the MI and marginals are recomputed. The
resulting dataset is reasonably small, taking 24GB and 960 seconds to
load.
BUILD:
(sql-open "postgres:///en_pairs_cfive_mi")
(define wpa (make-any-link-api))
(define wps (add-pair-stars wpa))
(wps 'fetch-pairs)
(sql-close)
(sql-open "postgres:///en_pairs_5.3_remi")
(define fpm (add-fmi-filter wps 5.3 #f))
(define fps (make-store fpm))
(fps 'store-all-elts)
(sql-close)
halt, restart
(batch-all-pair-mi wps)
STATUS: Good. (as good as en_pairs_cfive_mi)
DATE: 12 July 2019
COMMENT ON DATABASE "en_pairs_5.3_remi" IS
'Same as en_pairs_cfive_mi but pairs w/FMI<5.3 cut & MI+marginals recomputed.';
en_pairs_5.3_remi=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 1834 MB |
public | valuations | table | ubuntu | 4689 MB |
select count(*) from atoms; <<<< 20908235 == 21M Atoms
select count(*) from valuations; <<<< 31618063 == 32M values
(print-matrix-summary-report wps)
Summary Report for Correlation Matrix Link Grammar ANY link Word Pairs
Left type: WordNode Right Type: WordNode Pair Type: EvaluationLink
Wildcard: (EvaluationLink (ctv 0 0 1.6649902e+08)
(LinkGrammarRelationshipNode "ANY")
(ListLink
(AnyNode "left-word")
(AnyNode "right-word")
)
)
Rows: 615950 Columns: 573477
Size: 8953449 non-zero entries of 353233158150 possible
Fraction non-zero: 2.5347E-5 Sparsity (-log_2): 15.268
Total observations: 166499020. Avg obs per pair: 18.596
Entropy Total: 18.647 Left: 13.418 Right: 13.647
Total MI: 8.4182
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 2915. 1840.
Count (l_1) 3.1752E+5 2.5220E+5 108.9 137.1
Length (l_2) 1.6175E+5 1.5049E+5 55.48 81.80
RMS Count 1.6155E+5 1.5034E+5 55.41 81.72
No MM^T data present
No M^TM data present
2637685531 Bytes uncompressed == 2.6GB
473114452 Bytes compressed == 473MBytes
83e6b75a2ffd7c248cf153a9108320f6 en_pairs_5.3_remi.sql
ef654a29a94845a166ac47fc470c369e en_pairs_5.3_remi.sql.bz2
========
dropdb en_pairs_cfive_mi_1.8
dropdb en_dj_cfive
dropdb en_full_marg
en_pairs_1.8_remi
-----------------
A copy of `en_pairs_cfive_mi`, but with all word-pairs with an MI<1.8
removed. After removal, the MI and marginals are recomputed. The
resulting dataset is reasonably small, taking 45GB to load.
BUILD:
See en_pairs_5.3_remi
STATUS: Good. (as good as en_pairs_cfive_mi)
DATE: 12 July 2019
COMMENT ON DATABASE "en_pairs_1.8_remi" IS
'Same as en_pairs_cfive_mi but pairs w/FMI<1.8 cut & MI+marginals recomputed.';
en_pairs_1.8_remi=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 3055 MB |
public | valuations | table | ubuntu | 6599 MB |
select count(*) from atoms; <<<< 34614297 == 35M Atoms
select count(*) from valuations; <<<< 52187113 == 52M Valuations
(print-matrix-summary-report fpf)
Summary Report for Correlation Matrix Link Grammar ANY link Word Pairs fmi-cut-1.8
Left type: WordNode Right Type: WordNode Pair Type: EvaluationLink
Wildcard: (EvaluationLink (ctv 0 0 4.7020738e+08)
(LinkGrammarRelationshipNode "ANY")
(ListLink
(AnyNode "left-word")
(AnyNode "right-word")
)
)
Rows: 619168 Columns: 580804
Size: 15795739 non-zero entries of 359615251072 possible
Fraction non-zero: 4.3924E-5 Sparsity (-log_2): 14.475
Total observations: 470207380. Avg obs per pair: 29.768
Entropy Total: 18.590 Left: 12.338 Right: 11.623
Total MI: 5.3712
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 9987. 1.2491E+4
Count (l_1) 1.3805E+6 2.6232E+6 138.2 210.0
Length (l_2) 2.4211E+5 3.2904E+5 24.24 26.34
RMS Count 2.4179E+5 3.2853E+5 24.21 26.30
No MM^T data present
No M^TM data present
4353431720 Bytes uncompressed == 4.3 GBytes
789427829 Bytes compressed == 789 MBytes
dfb1bf8f6e61fba18c93fbb591d34650 en_pairs_1.8_remi.sql
5f51c11907ea6cc624274707ed7eabc5 en_pairs_1.8_remi.sql.bz2
en_pairs_mi_1.8cut
------------------
A copy of `en_pairs_cfive_mi`, but with all word-pairs with an MI<1.8
removed. After removal, a filter is applied and the MI and marginals
are recomputed on that filter. Thus, the dataset contains two sets of
MI data: the original data, at the standard locations, and the
recomputed MI, at the filtered location.
BUILD:
createdb en_pairs_mi_1.8cut -T en_pairs_cfive_mi_1.8
(wps 'fetch-pairs)
(define fpm (add-fmi-filter wps 1.8 #t)) ;; <<< #t so filter-keys!
(batch-all-pair-mi fpm)
STATUS: OK. (as good as en_pairs_cfive_mi)
REASON: It's good, but most ongoing apps do not need both sets of MI,
so its a waste of space ... and mildly confusing.
DATE: 12 July 2019
COMMENT ON DATABASE "en_pairs_mi_1.8cut" IS
'en_pairs_cfive_mi and pairs w/FMI<1.8 cut and both old and new MI/marginals.';
en_pairs_mi_1.8cut=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 3055 MB |
public | valuations | table | ubuntu | 12 GB |
select count(*) from atoms; <<<< 34614308 == 34M Atoms
select count(*) from valuations; <<<< 88578486 == 89M Valuations
(print-matrix-summary-report fpf)
Summary Report for Correlation Matrix Link Grammar ANY link Word Pairs fmi-cut-1.8
(This is same as en_pairs_1.8_remi)
7335895086 Bytes uncompressed == 7.3 GBytes
1289552824 Bytes compressed == 1.3 GBytes
ac2920d95c949d7220c917d108b668cc en_pairs_mi_1.8cut.sql
ff6157f7cdf5dda1ff191ead93ace247 en_pairs_mi_1.8cut.sql.bz2
==========
OK go for MPG parsing.
createdb en_mpg_18 -T en_pairs_1.8_remi
createdb en_mpg_53 -T en_pairs_5.3_remi
OK, 53 hang on split-books/lisa-wit-ah
gc is spinning like crazy .... and allocs are spinning like crazy too.
18 hang on split-books/pascal-br
Crap. 157 threads.
57 GC_allochblk_nth
scheme@(guile-user) [1]> ,bt
In opencog/nlp/learn/mst-parser.scm:
344:31 5 (observe-mpg _)
In opencog/sheaf/mpg-parser.scm:
72:25 4 (mpg-parse-atom-seq _ #<procedure ramp-scorer (LW RW LEN)> 1000000)
In opencog/sheaf/mst-parser.scm:
342:25 3 (mst-parse-atom-seq _ #<procedure ramp-scorer (LW RW LEN)>)
In opencog/sheaf/vo-graph.scm:
206:33 2 (wedge-cross? _ _)
In unknown file:
write items=1853
num_get_nodes=1109
num_get_links=6085
(define tit (tokenize-text "this is a test"))
(define wit (map WordNode tit))
(define pai (make-any-link-api))
(define mis (add-pair-freq-api pai))
(define sco (make-score-fn mis 'pair-fmi))
(define mst (mst-parse-atom-seq wit sco))
(define start-cost-pair (pick-best-cost-pair numa-list))
returns bad node ... it should check and abort...
pick-best-cost-left-pair returns bad node ...
1e-40 wall-this
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12871 101000 30 10 56.172g 0.042t 39880 S 1178 17.1 91:08.50 guile
12871 101000 30 10 59.010g 0.045t 39880 S 1156 18.1 119:18.34 guile
12871 101000 30 10 63.820g 0.049t 39880 S 1318 20.1 135:03.21 guile
%Cpu(s): 3.9 us, 29.5 sy, 20.0 ni, 46.2 id, 0.3 wa, 0.0 hi, 0.1 si, 0.0 st
(gdb) bt
#0 0x00007f85d38ef428 in __GI_raise (sig=sig@entry=6)
at ../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007f85d38f102a in __GI_abort () at abort.c:89
#2 0x00007f85d3f6ecdb in capture_delimited_continuation (
current_registers=<optimized out>, dynstack=<optimized out>,
saved_registers=<optimized out>, saved_mra=<optimized out>,
saved_fp=<optimized out>, vp=<optimized out>) at ../../libguile/vm.c:1327
#3 abort_to_prompt (thread=0x15f692dc0, saved_mra=<optimized out>)
at ../../libguile/vm.c:1454
#8 0x00007f85d3f19581 in scm_jit_enter_mcode (thread=0x15f692dc0,
mcode=0x15f692dc0 "\200\205\267_\001") at ../../libguile/jit.c:4796
#13 0x00007f85d3f6c546 in scm_throw (key=key@entry=0xdc5420, args=0x15af130a0)
at ../../libguile/throw.c:272
#15 0x00007f85d3eedeb5 in scm_error_scm (key=key@entry=0xdc5420,
subr=<optimized out>, message=message@entry=0x1607c9380,
args=args@entry=0x15af130e0, data=data@entry=0x15af130f0)
at ../../libguile/error.c:90
#16 0x00007f85d3eedf4f in scm_error (key=0xdc5420, subr=subr@entry=0x0,
message=message@entry=0x7f85d3fa228c "Wrong type to apply: ~S", args=0x15af130e0,
rest=rest@entry=0x15af130f0) at ../../libguile/error.c:62
#17 0x00007f85d3f6f913 in get_callee_vcode (thread=0x15f692dc0)
at ../../libguile/vm.c:1527
#22 0x00007f85d3f19581 in scm_jit_enter_mcode (thread=0x15f692dc0,
mcode=0x15f692dc0 "\200\205\267_\001") at ../../libguile/jit.c:4796
#30 0x00007f85d3f19581 in scm_jit_enter_mcode (thread=0x15f692dc0,
mcode=0x15f692dc0 "\200\205\267_\001") at ../../libguile/jit.c:4796
#34 0x00007f85d3f6c1aa in catch (tag=tag@entry=0x404, thunk=0x15b341ee0,
handler=0x15b341ec0, pre_unwind_handler=0x15b341ea0) at
../../libguile/throw.c:146
#35 0x00007f85d3f6c505 in scm_catch_with_pre_unwind_handler (key=key@entry=0x404,
thunk=<optimized out>, handler=<optimized out>,
pre_unwind_handler=<optimized out>) at ../../libguile/throw.c:260
#36 0x00007f85d3f6c6bf in scm_c_catch (tag=tag@entry=0x404, body=<optimized out>,
body_data=<optimized out>,
handler=handler@entry=0x7f85c95d0f00
<opencog::SchemeEval::catch_handler_wrapper(void*, scm_unused_struct*,
scm_unused_struct*)>,
handler_data=handler_data@entry=0x7f7e60000980,
pre_unwind_handler=pre_unwind_handler@entry=0x7f85c95d0c40
<opencog::SchemeEval::preunwind_handler_wrapper(void*, scm_unused_struct*,
scm_unused_struct*)>,
pre_unwind_handler_data=0x7f7e60000980) at ../../libguile/throw.c:385
#37 0x00007f85c95d122a in opencog::SchemeEval::do_eval (this=0x7f7e60000980,
expr="(observe-mpg \"The countess, with her loving heart, felt that her
children were being ruined, that it was not the count's fault for he could not
help being what he was -- that (though he tried to hide "...)
at /home/ubuntu/src/atomspace/opencog/guile/SchemeEval.cc:590
#38 0x00007f85c95d12aa in opencog::SchemeEval::c_wrap_eval (p=0x7f7e60000980)
at /home/ubuntu/src/atomspace/opencog/guile/SchemeEval.cc:507
#39 0x00007f85d3eeb47a in c_body (d=0x7f7e85fe2d40)
at ../../libguile/continuations.c:430
#40 0x00007f85d4262b6f in ?? ()
#41 0x00007f85d4262a80 in ?? ()
#42 0x00007f85d41e10c0 in jump_table_ () from /usr/local/lib/libguile-3.0.so.0
#43 0x000000015f692dc0 in ?? ()
#44 0x00007f85d3f19581 in scm_jit_enter_mcode (thread=0x15f692dc0,
mcode=0x15f692dc0 "\200\205\267_\001") at ../../libguile/jit.c:4796
#45 0x00007f85d3f70600 in vm_debug_engine (thread=0x304)
at ../../libguile/vm-engine.c:370
#46 0x00007f85d3f76db2 in scm_call_n (proc=proc@entry=0x15b341fe0,
argv=argv@entry=0x0, nargs=nargs@entry=0) at ../../libguile/vm.c:1605
#47 0x00007f85d3eef8d9 in scm_call_0 (proc=proc@entry=0x15b341fe0)
at ../../libguile/eval.c:490
#48 0x00007f85d3f6c1aa in catch (tag=tag@entry=0x404, thunk=0x15b341fe0,
handler=0x15b341fc0, pre_unwind_handler=0x15b341fa0) at
../../libguile/throw.c:146
#49 0x00007f85d3f6c505 in scm_catch_with_pre_unwind_handler (key=key@entry=0x404,
thunk=<optimized out>, handler=<optimized out>,
pre_unwind_handler=<optimized out>) at ../../libguile/throw.c:260
#50 0x00007f85d3f6c6bf in scm_c_catch (tag=tag@entry=0x404,
body=body@entry=0x7f85d3eeb470 <c_body>,
body_data=body_data@entry=0x7f7e85fe2d40,
handler=handler@entry=0x7f85d3eeb720 <c_handler>,
handler_data=handler_data@entry=0x7f7e85fe2d40,
pre_unwind_handler=pre_unwind_handler@entry=0x7f85d3eeb580
<pre_unwind_handler>,
pre_unwind_handler_data=0xe174a0) at ../../libguile/throw.c:385
#51 0x00007f85d3eeb9e3 in scm_i_with_continuation_barrier (
body=body@entry=0x7f85d3eeb470 <c_body>,
body_data=body_data@entry=0x7f7e85fe2d40,
handler=handler@entry=0x7f85d3eeb720 <c_handler>,
handler_data=handler_data@entry=0x7f7e85fe2d40,
pre_unwind_handler=pre_unwind_handler@entry=0x7f85d3eeb580
<pre_unwind_handler>,
pre_unwind_handler_data=0xe174a0) at ../../libguile/continuations.c:368
#52 0x00007f85d3eebac5 in scm_c_with_continuation_barrier (func=<optimized out>,
data=<optimized out>) at ../../libguile/continuations.c:464
#53 0x00007f85d3575127 in GC_call_with_gc_active (
fn=fn@entry=0x7f85d3f6a070 <with_guile_trampoline>,
client_data=client_data@entry=0x7f7e85fe2e20) at ../pthread_support.c:1343
#54 0x00007f85d3f6ac4f in with_guile (base=base@entry=0x7f7e85fe2df0,
data=data@entry=0x7f7e85fe2e20) at ../../libguile/threads.c:683
#55 0x00007f85d356f132 in GC_call_with_stack_base (
fn=fn@entry=0x7f85d3f6abb0 <with_guile>, arg=arg@entry=0x7f7e85fe2e20)
at ../misc.c:1941
#56 0x00007f85d3f6aff8 in scm_i_with_guile (dynamic_state=<optimized out>,
data=0x7f7e60000980,
func=0x7f85c95d1290 <opencog::SchemeEval::c_wrap_eval(void*)>)
at ../../libguile/threads.c:698
#57 scm_with_guile (
func=func@entry=0x7f85c95d1290 <opencog::SchemeEval::c_wrap_eval(void*)>,
data=data@entry=0x7f7e60000980) at ../../libguile/threads.c:704
#58 0x00007f85c95d126e in opencog::SchemeEval::eval_expr (this=0x7f7e60000980,
expr=...) at /home/ubuntu/src/atomspace/opencog/guile/SchemeEval.cc:479
#59 0x00007f85bc783439 in opencog::GenericShell::eval_loop (this=0x7f7ef0001e90)
at /home/ubuntu/src/opencog/opencog/cogserver/shell/GenericShell.cc:588
#60 0x00007f85c6e5ac80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#61 0x00007f85d3c916ba in start_thread (arg=0x7f7e85fe3700) at
pthread_create.c:333
#62 0x00007f85d39c141d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
up 58
print *this
shows "(observe-mpg \"The countess,
_input_line = "(observe-mpg \"The countess, with her loving heart, felt that her
children were being ruined, that it was not the count's fault for he could not
help being what he was -- that (though he tried to hide "...,
libguile/vm.c 1327:
if (SCM_FRAME_DYNAMIC_LINK (base_fp) != saved_fp)
abort();
#16 0x00007f85d3eedf4f in scm_error (key=0xdc5420, subr=subr@entry=0x0,
message=message@entry=0x7f85d3fa228c "Wrong type to apply: ~S",
args=0x15af130e0,
rest=rest@entry=0x15af130f0) at ../../libguile/error.c:62
#17 0x00007f85d3f6f913 in get_callee_vcode (thread=0x15f692dc0)
at ../../libguile/vm.c:1527
vp->ip = SCM_FRAME_VIRTUAL_RETURN_ADDRESS (vp->fp);
scm_error (scm_arg_type_key, NULL, "Wrong type to apply: ~S",
scm_list_1 (proc), scm_list_1 (proc));
}
149 threads
Half(?) are in
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f85d3c93dbd in __GI___pthread_mutex_lock (mutex=mutex@entry=0xd80f68)
at ../nptl/pthread_mutex_lock.c:80
#2 0x00007f85d3f7845f in scm_c_weak_set_lookup (set=<optimized out>,
raw_hash=raw_hash@entry=2291501516460283252,
pred=pred@entry=0x7f85d3f69270 <string_lookup_predicate_fn>,
closure=closure@entry=0x7f7e177fcf50, dflt=dflt@entry=0x4)
at ../../libguile/weak-set.c:760
#3 0x00007f85d3f68df9 in lookup_interned_symbol (raw_hash=2291501516460283252,
name=0x15abbbd80) at ../../libguile/symbols.c:112
#4 scm_i_str2symbol (str=0x15abbbd80) at ../../libguile/symbols.c:244
One is this:
#0 0x00007f85d3c954ee in __pthread_mutex_unlock_usercnt (decr=1,
mutex=0x7f85d37c4260 <GC_allocate_ml>) at pthread_mutex_unlock.c:55
#1 __GI___pthread_mutex_unlock (mutex=0x7f85d37c4260 <GC_allocate_ml>)
at pthread_mutex_unlock.c:314
#2 0x00007f85d356f0d0 in GC_call_with_alloc_lock (
fn=fn@entry=0x7f85d3f777e0 <do_copy_weak_entry>,
client_data=client_data@entry=0x7f85a47f6ef0) at ../misc.c:1925
#3 0x00007f85d3f7850b in copy_weak_entry (dst=0x7f85a47f6ee0, src=0x100e450)
at ../../libguile/weak-set.c:128
#4 weak_set_lookup (dflt=0x4, closure=0x7f85a47f6f50,
pred=0x7f85d3f69270 <string_lookup_predicate_fn>, hash=671639580896511065,
set=0xd80f60) at ../../libguile/weak-set.c:493
#5 scm_c_weak_set_lookup (set=<optimized out>,
raw_hash=raw_hash@entry=335819790448255532,
pred=pred@entry=0x7f85d3f69270 <string_lookup_predicate_fn>,
closure=closure@entry=0x7f85a47f6f50, dflt=dflt@entry=0x4)
at ../../libguile/weak-set.c:762
#6 0x00007f85d3f68df9 in lookup_interned_symbol (raw_hash=335819790448255532,
name=0x15c4c11e0) at ../../libguile/symbols.c:112
#7 scm_i_str2symbol (str=0x15c4c11e0) at ../../libguile/symbols.c:244
So, here's my next installment on using guile-2.9.2. The first installment said
that I'd piled up CPU-months of guile 2.9.2 experience without any crashes. Well,
now, a different workload crashes in minutes. Below is a highly simplified,
edited gdb session -- it crashes because it unexpectedly aborts, during an
abort(!) because `get_callee_vcode()` failed. Harrumpf.
Background: there are 140 threads, half in guile, the other half waiting for guile
to finish. Yes, that's too many, but anyways ... 70 threads in guile and one
crashed:
#2 0x00007f85d3f6ecdb in capture_delimited_continuation (
current_registers=<optimized out>, dynstack=<optimized out>,
saved_registers=<optimized out>, saved_mra=<optimized out>,
saved_fp=<optimized out>, vp=<optimized out>) at ../../libguile/vm.c:1327
#3 abort_to_prompt (thread=0x15f692dc0, saved_mra=<optimized out>)
at ../../libguile/vm.c:1454
Both frames are interesting, because libguile/vm.c:1327 shows
if (SCM_FRAME_DYNAMIC_LINK (base_fp) != saved_fp)
abort();
hey!? who called this? line 1454 is in the middle of abort_to_prompt () Yow! an
unexpected abort during an abort...
How did we get here?
#15 0x00007f85d3eedeb5 in scm_error_scm (key=key@entry=0xdc5420,
subr=<optimized out>, message=message@entry=0x1607c9380,
args=args@entry=0x15af130e0, data=data@entry=0x15af130f0)
at ../../libguile/error.c:90
#16 0x00007f85d3eedf4f in scm_error (key=0xdc5420, subr=subr@entry=0x0,
message=message@entry=0x7f85d3fa228c "Wrong type to apply: ~S",
args=0x15af130e0,
rest=rest@entry=0x15af130f0) at ../../libguile/error.c:62
#17 0x00007f85d3f6f913 in get_callee_vcode (thread=0x15f692dc0)
at ../../libguile/vm.c:1527
and libguile/vm.c:1527
and libguile/vm.c:1527 tells me that get_callee_vcode () is very unhappy. But why?
I cannot tell .. after that, things peter out in boring stack frames that started
with my call scm_c_catch() ... the same seemingly harmless call that is pending in
70 other threads. (the same call that has survived several CPU month of pounding
with a different collection of scheme code)
My best guess is that the current code, that crashes, is .. compiling into
something new/different/unusual, triggering a bug. Either that, or the 70 threads
are exposing a race condition that has been hithertho hidden. I don't know how to
debug any further. I will try a slightly newer guile shortly.
-- Linas
p.s. here's the whole stack trace. But really, its boring, except for the above
highlights.
above was for
commit 1fb39929f9c1c292b11aedb7b29a166471b1c555
Author: Andy Wingo <wingo@pobox.com>
Date: Mon May 27 19:05:10 2019 +0200
Lets try again.
guard
scm_thread
scm_vm
scm_tc16_thread = scm_make_smob_type ("thread", sizeof (scm_thread));
scm_init_threads
SCM_NEWSMOB (t->handle, scm_tc16_thread, t)
0xf00fbaaa for the thread
0xd00d1111 for the vm
SCM_FRAME_LOCAL
SCM proc = SCM_FRAME_LOCAL (vp->fp, 0);
scm_t_bits scm.h SCM_UNPACK
2e -- 10 1110 -- odd small int -- 11
36 -- 11 0110 -- odd small int -- 13
2a -- 10 1010 -- odd small int -- 10
26 -- 10 0110 -- odd small int -- 9
22 -- 10 0010 -- even small int -- 8 -- just shift >>2 with SCM_I_INUM (exp)
3a --
3e --
42 --
SCM_I_INUM (exp)
if (SCM_I_INUMP(xp)...
#define SCM_PROGRAM_P(x) (SCM_HAS_TYP7 (x, scm_tc7_program)) 0x45 & 0x7f
#define SCM_STRUCTP(X) (!SCM_IMP(X) && (SCM_TYP3(X) == scm_tc3_struct))
who sets SCM_FRAME_LOCAL to short int??
unpack_values does .. why?
because subr-call in vm-engine.c called it
SP_SET (0, ret); ??
RESET_FRAME (1); // sets sp = fp-1
SP_SET (0, ret); ////
NEXT (1);
if (SCM_I_INUMP(SCM_FRAME_LOCAL (thread->vm.fp, 0))) {
printf("duuude fail in foeign call! \n"); assert(0); }
eng halt
Both
VM_DEFINE_OP (7, return_values, "return-values", OP1 (X32))
and
VM_DEFINE_OP (0, halt, "halt", OP1 (X32))
set fp to whack values....
and so does
VM_DEFINE_OP (4, call_label, "call-label", OP3 (X8_F24, X8_C24, L32))
scm_i_vm_prepare_stack
==========
get_callee_vcode
jump_table_
scm_jit_enter_mcode mcode == 0x49cc8949 0x8349c429 0x8f0f38fc 0x00000013
vm_debug_engine
scm_call_n
only four places call scm_jit_enter_mcode
-- two for instrumentation
-- one for return-values
-- one for compose-continuation << probably this
-- one is abort
what does scm_jit_enter_mcode do?
if (!SCM_FRAME_MACHINE_RETURN_ADDRESS (thread->vm.fp))
SCM_FRAME_SET_MACHINE_RETURN_ADDRESS
(thread->vm.fp, scm_jit_return_to_interpreter_trampoline);
so vm.fp is a C pointer to trampoline bytecode
enter_mcode = jit_address_to_function_pointer (enter_mcode_addr);
jit_address_to_function_pointer bless_function_pointer
enter_mcode_addr = emit_code (j, emit_entry_trampoline);
ohh and then there is emit_get_callee_vcode
which is from emit_indirect_tail_call
emit_call_1 (j, scm_vm_intrinsics.get_callee_vcode, thread_operand ());
duuuude wtf return valueo thread=0xbd5540 mcode=49
fp val=7fffd83a0168 yeah=a3aa30
SCM_FRAME_LOCAL (VP->fp, 0)
#define SCM_FRAME_LOCAL(fp,i) (SCM_FRAME_SLOT (fp, i)->as_scm)
#define SCM_FRAME_SLOT(fp,i) ((fp) - (i) - 1)
So what does emit_get_callee_vcode do??
emit_get_callee_vcode (scm_jit_state *j, jit_gpr_t dst)
{
emit_call_1 (j, scm_vm_intrinsics.get_callee_vcode, thread_operand ());
emit_retval (j, dst);
emit_reload_sp (j);
emit_reload_fp (j);
}
emit_reload_fp (scm_jit_state *j)
{
emit_ldxi (j, FP, THREAD, thread_offset_fp);
set_register_state (j, FP_IN_REGISTER);
}
where the heck is thread_offset_fp ??? must be computed durig build...
frame_offset_vra,
thread_offset_sp,
ohh I get it ... its just not needed...
scm_i_vm_prepare_stack
=========
lthread 151 and 143, 142, 141 140, 138, 137, 136, 135, 127, 126, 124
GC_generic_malloc_many mallocx.c:304
scm_inline_gc_alloc libguile/gc-inline.h:79
allocate_words_with_freelist libguile/intrinsics.c:424
thread 150 and 148
GC_malloc_kind_global malloc.c:254
make_bytevector libguile/bytevectors.c:233
thread 125
scm_c_weak_set_lookup libguile/weak-set.c:760
lookup_interned_symbol libguile/symbols.c:112
scm_i_str2symbol libguile/symbols.c:244
========
MPG-Processing file >>>split-books/lisa-sorc-aa<<<
Sentence Splitter v3
Language: en
utf8 "\xA9" does not map to Unicode at ./split-sentences.pl line 72, <STDIN> line 9.
Malformed UTF-8 character (fatal) at ./split-sentences.pl line 115, <STDIN> line 10.
Copyright <A9> 2010 by Eric Flint and Dave Freer
=========
guile 2.2.3
Thread 7 "guile" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff358f700 (LWP 17049)]
scm_i_vm_mark_stack (vp=0x15b827a20, mark_stack_ptr=0x7ffff357ef00,
mark_stack_limit=0x7ffff358eeb0) at ../../libguile/vm.c:995
995 if (SCM_NIMP (sp->as_scm) &&
(gdb) bt
#0 scm_i_vm_mark_stack (vp=0x15b827a20, mark_stack_ptr=0x7ffff357ef00,
mark_stack_limit=0x7ffff358eeb0) at ../../libguile/vm.c:995
#1 0x00007ffff717335e in GC_mark_from (mark_stack_top=0x7ffff357eed0,
mark_stack_top@entry=0x7ffff357eef0,
mark_stack=mark_stack@entry=0x7ffff357eeb0,
mark_stack_limit=mark_stack_limit@entry=0x7ffff358eeb0) at ../mark.c:772
#2 0x00007ffff717397e in GC_do_local_mark (local_mark_stack=0x7ffff357eeb0,
local_top=0x7ffff357eef0) at ../mark.c:1037
#3 0x00007ffff7173b98 in GC_mark_local (
local_mark_stack=local_mark_stack@entry=0x7ffff357eeb0, id=id@entry=5)
at ../mark.c:1170
#4 0x00007ffff7173eaa in GC_help_marker (my_mark_no=my_mark_no@entry=101042)
at ../mark.c:1238
#5 0x00007ffff717de3c in GC_mark_thread (id=<optimized out>)
at ../pthread_support.c:380
#6 0x00007ffff78916ba in start_thread (arg=0x7ffff358f700) at
pthread_create.c:333
#7 0x00007ffff75c741d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb) info thr
(gdb) print sp
$1 = (union scm_vm_stack_element *) 0x7fffe000c038
(gdb) print *sp
Cannot access memory at address 0x7fffe000c038
(gdb) print vp
$2 = (struct scm_vm *) 0x15b827a20
(gdb) print *vp
$3 = {ip = 0x7fffee315f2c, sp = 0x7fffe000c038, fp = 0x7ffff7e03050,
stack_limit = 0x7ffff7e02000, trace_level = 0, sp_min_since_gc = 0x7fffe000c008,
stack_size = 1024, stack_bottom = 0x7ffff7e02000, stack_top = 0x7ffff7e04000,
overflow_handler_stack = 0x304, hooks = {0x4, 0x4, 0x4, 0x4, 0x4},
resumable_prompt_cookie = 0x7fffcade1fe0, engine = 1}
=========
34614581
(WordNode "###LEFT-WALL###") has a minscule incoming set .... why?
one thread at prompt
24 threads in pthread_cond_wait for GC
2 threads in some blocking gc thing
1 immortal
1 in nanosleep in cogserver
1 in boost::asio recv
1 in
0 0x00007f6017f25cd8 in __GI___tls_get_addr (ti=0x7f6017ef42f8) at dl-tls.c:834
#1 0x00007f6017c01e86 in scm_async_tick () at ../../libguile/async.c:148
#2 0x00007f6017c0d6ee in scm_equal_p (x=0x26, y=0x6) at ../../libguile/eq.c:299
#3 0x00007f6017c0d752 in scm_equal_p (x=0x1f33060, y=0x1f330e0)
at ../../libguile/eq.c:308
#4 0x00007f6017c702e0 in scm_srfi1_delete_duplicates_x (lst=0x2606aa0a0, pred=0x9
(observe-mpg \"Contact the Foundation as set forth in Section 3 below.\")
1 in srfi1_find_tail
(observe-mpg \"Section 2.\")
Another with (observe-mpg \"Section 3.\")
Another with (observe-mpg \"Section 4.\")
Another with (observe-mpg \"Section 5.\")
Another with (observe-mpg \"Section 1.\")
ans (observe-mpg \"See paragraph 1.C below.\")
(observe-mpg \"See paragraph 1.E below.\")
1 in CACHE_REGISTER
(observe-mpg \"The Foundation’s EIN or federal tax identification number is
64-6221541.\")
(mst-parse-text "Section 1.")
(graph-add-bridges disco)
((((2 . (WordNode "Section")) 3 . (WordNode "1")) . 7.98934111276043)
(((2 . (WordNode "Section")) 4 . (WordNode ".")) . 4.02007251964681))
((1 . (WordNode "###LEFT-WALL###"))
(2 . (WordNode "Section"))
(3 . (WordNode "1"))
(4 . (WordNode ".")
))
So -- graph-add-linear fails
graph-add-bridges hangs ...
==========
OK go for MPG parsing.
createdb en_mpg_18 -T en_pairs_1.8_remi
createdb en_mpg_53 -T en_pairs_5.3_remi
(observe-mpg "“Can you define anything which you only feel and cannot see—heat for instance, or cold?”")
ERROR: Wrong type to apply: 10
mpg-linear
alpha-guten-tranche-1/split-books/steam-br
39 41 43
(observe-mpg \"INDEMNITY - You agree to indemnify and hold the Foundation, the
trademark owner, any agent or employee of the Foundation, anyone providing copies
of Project Gutenberg-tm electronic works"
45
(observe-mpg \"+ -- -- -- -- -- + -- -- -- -- -- + -- -- -- -- + -- -- -- -- -+ --
-- -- -- -- + -- -- -- -- -- -- -- -+ | | | | | | Relative | | | | | Weight | |
Density, | | | | | of | Volume | Hydro
47
(observe-mpg \"+ -- -- -- -- -- -- -- -- -- -- -- + -- -- -- -+ -- -- -- -- + --
-- -- -+ |Substance |Carbon |Hydrogen|Oxygen | + -- -- -- -- -- -- -- -- -- -- --
+
49
(observe-mpg \"+ -- -- -- -- -- -- -- -- -- -+ -- -- -- -- -- -- -- -- -- -- -- --
-- -- + -- -- -- -- -- -- -- + | Kind of Coal | Per Cent of Combustible | B. t. u.
| | + -- -- -- -- -- -- + -- -- -- "
52
(observe-mpg \"Starting with Tioga and Bradford Counties of northern Pennsylvania,
the bed runs southwest through Lycoming, Clearfield, Centre, Huntingdon, Cambria,
Somerset and Fulton Counties, Pennsy"...)
54
(observe-mpg \"Absolute pressure 117 Absolute zero 80 Accessibility of Babcock &
Wilcox boiler 59 Acidity in boiler feed water 106 Actual evap. corresponding to
boiler horse power 288 Advantages of Bab"
56
(observe-mpg \"Calorimeter, coal, Mahler bomb 184 Mahler bomb, method of
correction 187 Mahler bomb, method of operation of 185 Calorimeter, steam, compact
type of throttling 132 Correction for 131 Loc"
report-avg-gc-cpu-time
10 0.0051
20 0.0264
30 0.0771
40 0.4769
50 2.1231
60 3.5993
70 8.3258
80 29.464
90 51.217
100 298.25
110
OK, this is exponential explosion, not power. Why
graph-add-mpg: length vs time:
10 0.0027
50 0.0742
80 0.1658
120 0.3634
160 0.5767
200 0.7824
300 1.5131
400 3.6476
500 5.7588
600 8.1915
add-linear: trivial
ad-bridges:
10 0.0007
40 0.5250
50 2.0587
60 3.3997
70 8.1694
80 29.494
(use-modules (srfi srfi-1))
(for-each (lambda (x) (x)) (iota 10))
(for-each call-with-new-thread (iota 10))
(catch #t
(for-each call-with-new-thread (iota 10))
(lambda key args ...) #f)
==========
OK go for MPG parsing.
createdb en_mpg_18 -T en_pairs_1.8_remi
createdb en_mpg_53 -T en_pairs_5.3_remi
OK, so
en_mpg_53 blew up to 72 GB RAM but parsed everything in under 20 hours
en_mpg_18 blows up to 90+
mpg_53:
(cog-report-counts)
((PredicateNode . 14) (ListLink . 10142877) (EvaluationLink . 10142877)
(AnchorNode . 1) (Connector . 1114710) (ConnectorDir . 2) (ConnectorSeq .
21886812) (Section . 34540275) (SchemaNode . 1) (WordNode . 689109)
(AnyNode . 2) (LinkGrammarRelationshipNode . 1))
mpg_18:
(cog-report-counts)
((PredicateNode . 15) (ListLink . 16995712) (EvaluationLink . 16995712)
(AnchorNode . 1) (Connector . 1102437) (ConnectorDir . 2) (ConnectorSeq .
29842172) (Section . 39642130) (SchemaNode . 1) (WordNode . 688016)
(AnyNode . 4) (LinkGrammarRelationshipNode . 1))
mpg-53
(psa 'fetch-pairs)
Elapsed time to load csets: 3930 secs 248:41.08 cputime
53.4g RAM total of 58105983 atoms
(batch-all-pair-mi psa)
Support: found num left= 564178 num right= 21886812 in 1193 secs
Finished left norm marginals in 117149 secs << 33 hours!
Finished left totals in 3064 secs
Finished right norm marginals in 1382 secs
Finished right totals in 25 secs
Done with wild-card count N(x,*) and N(*,y) in 121620 secs <<< 34 hours
Total count N(*,*) = 78200160.0 = 78200160.0
Finished column (left) norm averages in 4026 secs
Finished row (right) norm averages in 106 secs
Going to do individual pair frequencies
Done computing 34540275 pair frequencies in 5465 secs
Start computing log P(*,y)
Done computing 21886812 left-wild log frequencies in 813 secs
Done computing 564178 right-wild log frequencies in 18 secs
Done storing 21886812 left-wilds in 15511 secs << 4 hours
Done storing 564178 right-wilds in 487 secs
Done computing 34540275 pair MI's in 28113 secs
Finished left entropy subtotals in 114463 secs << 32 hours
Finished right entropy subtotals in 1373 secs
Finished left MI subtotals in 115187 secs << 32 hours
Finished right MI subtotals in 1379 secs
Done storing 21886812 left-wilds in 11512 secs << 3 hours
Done storing 564178 right-wilds in 308 secs
Finished with MI computations; this took 116. hours << 5 days.
guile -l mpg-53-djmi.scm
==================================================================
en_mpg_53
---------
Brand new recomputation of disjuncts, using the MPG (Maximal Planar
Graph) parser. The starting dataset is one where all word-pairs of
MI < 5.3 are discarded. Then the MPG parse is performed. Any islands
that remain are linked with adjacent words (whether or not such word
pairs can be found in the pairs dataset). This dataset contains both
the word-pairs, and also the disjuncts, and also the disjunct marginals.
The goal of the MPG parse is to be able to apply data cuts much later
in the pipeline.
This is a full copy of `en_pairs_5.3_remi`, with the djs and dj
marginals added.
BUILD:
(psa 'fetch-pairs)
(batch-all-pair-mi psa)
STATUS: Good. The word-pair dataset contains some junk - words with
markup that should have been split off. About 5%(?) of words are
affected.
DATE: 27 July 2019
COMMENT ON DATABASE en_mpg_53 IS
'en_pairs_5.3_remi plus disjuncts plus dj marginals.';
Final RAM usage: 110 GBytes for 80556988 Atoms
en_mpg_53=> \dt+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+--------+------------+-------------
public | atoms | table | ubuntu | 9047 MB |
public | valuations | table | ubuntu | 23 GB |
select count(*) from atoms; <<<< 100967670 == 100M Atoms
select count(*) from valuations; <<<< 225042856 == 225M Values
(print-matrix-summary-report psa)
Summary Report for Correlation Matrix Word-Disjunct Pairs (Connector Sets)
Left type: WordNode Right Type: ConnectorSeq Pair Type: Section
Wildcard: (ListLink (ctv 0 0 78200160)
(AnyNode "cset-word")
(AnyNode "cset-disjunct")
)
Rows: 564178 Columns: 21886812
Size: 34540275 non-zero entries of 12348057820536 possible
Fraction non-zero: 2.7972E-6 Sparsity (-log_2): 18.448
Total observations: 78200160.0 Avg obs per pair: 2.2640
Entropy Total: 22.410 Left: 20.005 Right: 10.120
Total MI: 7.7151
Left Right Avg-left Avg-right
---- ----- -------- ---------
Support (l_0) 447.7 2.7665E+5
Count (l_1) 1.1523E+4 1.0087E+6 25.74 3.646
Length (l_2) 5540. 4.0915E+4 12.37 .1479
RMS Count 4252. 4.0864E+4 9.497 .1477
No MM^T data present
No M^TM data present
(cog-report-counts)
((PredicateNode . 14) (ListLink . 22450991) (TypeNode . 2) (AnchorNode . 1)
(Connector . 1114710) (ConnectorDir . 2) (ConnectorSeq . 21886812) (Section .
34540275) (SchemaNode . 1) (WordNode . 564178) (AnyNode . 2))
Note that 564178 + 21886812 = 22450990 so the ListLinks are
holding the marginals. (plus one for any-any)
18014180469 Bytes uncompressed == 18 GBytes
2235538306 Bytes compressed == 2.2GBytes
36379f212836ab4b07dc71d0a2623854 en_mpg_53.sql
79069ae3d5d6d51d4eea15cf24f2e2bf en_mpg_53.sql.bz2
==================================================================
Try again,
en_mpg_18
(psa 'fetch-pairs)
Elapsed time to load csets: 5249 secs
total of 71223993 Atoms == 71M of them. Takes 66GBytes RAM
(cog-report-counts)
((PredicateNode . 2) (AnchorNode . 1) (Connector . 1103016) (ConnectorDir . 2)
(ConnectorSeq . 29873423) (Section . 39684585) (SchemaNode . 1) (WordNode .
562961) (AnyNode . 2))
So lets go:
Support: found num left= 562961 num right= 29873423 in 1672 secs
Finished left norm marginals in 405577 secs << 113 hours == 4.7 days
Finished left totals in 29755 secs
Finished right norm marginals in 1753 secs
Finished right totals in 24 secs
Done with wild-card count N(x,*) and N(*,y) in 437109 secs << 5 days
Total count N(*,*) = 77906978.0 = 77906978.0
Finished column (left) norm averages in 6499 secs
Finished row (right) norm averages in 123 secs
Going to do individual pair frequencies
Done computing 39684585 pair frequencies in 8344 secs
Done storing 29873423 left-wilds in 21121 secs
Done storing 562961 right-wilds in 475 secs
Done computing 39684585 pair MI's in 35319 secs
<<< atomspace holds 101660390 atoms
<<< RAM usage 129 GBytes
scp ubuntu@10.0.3.208://home/ubuntu/
==================================================================
??? Why are there ListLinks in the above?
?? Why are there 564K words!? Wow!
Issues: there are words that begin/end with equals-signs:
e.g. (WordNode "=Carolina.=")
Also words that begin/end with vertical bar: (WordNode "789│") (WordNode "16.42|")
note the UTF8 bar in above....
a few but annoying: mayeb 1 in 20 of above.
Next: some that begin/end with asterisk.
or backslash: (WordNode "unfitness,\") (WordNode "gone.\")
or (WordNode "@deserted@")
(WordNode "HEALTH"*")
(WordNode "Wiley...............A")
(WordNode "Elizabeth......how")
(WordNode "Mother....we're")
(WordNode "Sorry...sorry...it's") (WordNode "Keith...remember..remember")
(WordNode "classroom.“")
(WordNode "King._~") (WordNode "Confessor.~")
(WordNode "]Klauber")
(WordNode "+SEBASTIEN")
(WordNode "winter/spring")
None of the words have any counts on tham (!) Why?
=========
Start work on info-sim.
use langtest which contains ... 4K words, 26K sections
-- (psa 'fetch-pairs) ; 7 seconds... great!
-- (define pta (add-transpose-api psa))
(pta 'mmt-count (Word "he"))
(pta 'total-mmt-count)
(define pss (add-support-api psa))
make-disc-info
Evaluation TODO:
* Fraction of words with null links.
* Number of linkages found ... (looking for overflow)
TODO:
* Words with less than 1% of orig counts should be canned.
* Why isn't "Remaining count" called on Greedy-checking and done-list?
* MI is wrong, given how we ad-hoc export to gram classes.
* Words that are in word classes, but still have left-over-counts
should get thier own classes ...
* Add 'help method
* Add 'verbose method to enable progress reports.
* user-interrupt: SchemeEval::interrupt: User interrupt from keyboard
will not interrupt running thread. Fix this
* Consistent dj/word-sets
* Counts on MemberLink... (MemberLink should show what's been merged
into it. This is done in some code but not other.)
DONE:
* fix filter-api (DONE)
* Matrix summary report to print present of MMT (DONE-ish)
* cut low-obs words from disjunct sets (DONE - micro and mini)
* Fraction correct S,O links (DONE)
* Why hang? Why not popping timeout? (Bug, DONE)
Dec 2019
========
Upgrade everything to
* Ubuntu Bionic 18.04
* bdwgc version 8.0
* guile-2.9.7
* link-grammar-5.7.0
* current vintage opencog
* Postgres 12.1
* hugepages /etc/sysctl.conf
sudo apt install update-manager-core
sudo do-release-upgrade
The hardest part is the postgres upgrade:
Here they are in ubuntu-16.04 postgres 9.5.19:
actually, now its 9.5.20 after upgrade ...
List of databases
Name | Size | Tablespace | Description
-----------------------+---------+------------+------------------------------------------------------------------------------
en_huge_marg | 18 GB | pg_default | en_cfive consistent Sections with word-count > 3; section-count > 1
en_large_marg | 3089 MB | pg_default | en_cfive consistent Sections with word-count > 8; section-count > 3
en_micro_disinfo | 684 MB | pg_default |
en_micro_disinfo3 | 504 MB | pg_default |
en_micro_disinfo3_exp | 303 MB | pg_default |
en_micro_disinfo_exp | 323 MB | pg_default |
en_micro_marg | 346 MB | pg_default | en_cfive consistent Sections with word-count > 500; section-count > 10
en_mini_marg | 1251 MB | pg_default | en_cfive consistent Sections with word-count > 40; section-count > 5
en_mpg_18 | 78 GB | pg_default | en_pairs_1.8_remi plus disjuncts plus dj marginals.
en_mpg_53 | 69 GB | pg_default | en_pairs_5.3_remi plus disjuncts plus dj marginals.
en_nano | 53 MB | pg_default |
en_pairs_1.8_remi | 22 GB | pg_default | Same as en_pairs_cfive_mi but pairs w/FMI<1.8 cut & MI+marginals recomputed.
en_pairs_5.3_remi | 14 GB | pg_default | Same as en_pairs_cfive_mi but pairs w/FMI<5.3 cut & MI+marginals recomputed.
en_pairs_cfive_mi | 31 GB | pg_default | en_pairs_cfive plus marginals (i.e. clean word-pairs, only)
ubuntu | 7112 kB | pg_default |
(19 rows)
/usr/lib/postgresql/12/bin/pg_upgrade
/usr/lib/postgresql/9.5/bin
/etc/postgresql/12/main/pg_hba.conf
/etc/postgresql/9.5/main/postgresql.conf
systemctl status postgresql.service
service postgresql stop
/usr/lib/postgresql/12/bin/initdb /var/lib/postgresql/12/main
So:
/usr/lib/postgresql/12/bin/pg_upgrade -b /usr/lib/postgresql/9.5/bin -B /usr/lib/postgresql/12/bin -d /etc/postgresql/9.5/main -D /etc/postgresql/12/main -c
Don't forget the -k flag, else the data is copied!
/usr/lib/postgresql/12/bin/pg_upgrade -b /usr/lib/postgresql/9.5/bin -B /usr/lib/postgresql/12/bin -d /etc/postgresql/9.5/main -D /etc/postgresql/12/main -k
service postgresql start
./analyze_new_cluster.sh
Hmm. Looks like it all worked out fine...