| … | … | @@ -43,12 +43,12 @@ Genealogy, as practiced by familysearch.org (controlled by the Church |
| 43 | 43 | of the LDS) is semi-automated. The raw material consists of public data |
| 44 | 44 | about births, deaths, census records, military draft records, |
| 45 | 45 | naturalization records, passenger lists, parish church records, |
| 46 | | obituaries and the like. Each data item provides a bit of information |
| 47 | | that attests a specific event. Some are highly structured: birth |
| 48 | | certificates have a regular form. Others, not so much: obituaries may or |
| 49 | | may not list relatives, and may or may not make the relationships clear. |
| 50 | | The task is to stitch together a genealogical structure out of these |
| 51 | | fragmentary jigsaw pieces. |
| 46 | marriage certificates, obituaries and the like. Each data item provides |
| 47 | a bit of information that attests a specific event. Some are highly |
| 48 | structured: birth certificates have a regular form. Others, not so much: |
| 49 | obituaries may or may not list relatives, and may or may not make the |
| 50 | relationships clear. The task is to stitch together a genealogical |
| 51 | structure out of these fragmentary jigsaw pieces. |
| 52 | 52 | |
| 53 | 53 | The evidentiary issue is that no single attestation of an event or |
| 54 | 54 | relation is fully trustworthy. A short survey is worthwhile. |
| … | … | @@ -73,6 +73,42 @@ the record and make a judgement call: is this an appropriate match? If |
| 73 | 73 | so, it can be added to a record, buttressing support for a particular |
| 74 | 74 | name, date, location, spelling, event. |
| 75 | 75 | |
| 76 | The judgement call is often hierarchical in nature. For example: do the |
| 77 | father's and mother's names match? Well, perhaps they do, but there is |
| 78 | some alteration in spelling: Maria vs. Mary, Elizabeth vs. Betty. One |
| 79 | can assign some probability or confidence in the beleif that these are |
| 80 | the same parents. To butress that beleif, one can look at sibilings: if |
| 81 | two children are born three months apart to the same mother, then |
| 82 | this is a piece of negative evidence that absolutely demands a rejection |
| 83 | of a match. That is, of course, only if the recorded birthday is |
| 84 | correct. Perhaps there are other attestations to the birthday? The |
| 85 | evidentiary chain is now four levels deep: individual -> parents -> |
| 86 | siblings -> birthdays -> evidence of birthdays. Each arrow in this chain |
| 87 | can be assigned a confidence; each act of confidence assignment is some |
| 88 | agglomeration of the supporting evidence. A preponderance of evidence |
| 89 | is desired. This can be taken as a mathematical mean or average, but |
| 90 | it can also be some more complex algorithm extending across the |
| 91 | structure: for example, "voting in new members to the club of facts". |
| 92 | In comp sci, similarity is measured with cosine distance, Jaquard |
| 93 | distance, mutual information, Kullbeck-Liebler divergence or any one |
| 94 | of dozens or hundres of different similarity measures, which can be |
| 95 | aggregated and weighted in a large variety of ways (random forests, |
| 96 | collections of experts, membership clubs, etc.) |
| 97 | |
| 98 | A much simpler, and more easily automated example occurs in operating |
| 99 | system storage integrity. Here, one is presented with mutiple files, |
| 100 | often having the same name, but not always; often having the same |
| 101 | timestamp, but not always; often appearing in the same directory, but |
| 102 | not always; often having the same size, but not always; often having the |
| 103 | same content, but sometimes haveing all-zeros where the data should have |
| 104 | been. Or perhaps MP3's with missng bytes, or images with steganographic |
| 105 | watermarks. The challenge is to find the copy of the file that is least |
| 106 | likely to be corrupted, and is most likely to be "correct". Despite the |
| 107 | fact that all data is directly available from the operating system, |
| 108 | using very pure and well-defineed, structured software interfaces, this |
| 109 | turns out to be a hard problem. |
| 110 | |
| 111 | |
| 76 | 112 | |
| 77 | 113 | Status |
| 78 | 114 | ------ |