R0: PASS — 10/10 seeds reproduce Test A exactly
## Means: candidate types / instance recall / type recall / candidate precision
tier 3000
C-G 10570 1.000 0.979 0.117
M-G-comp 2072 0.306 0.297 0.245
C-L 8634 1.000 0.929 0.117
M-L-comp 1578 0.224 0.210 0.174
M-R-comp 404 0.083 0.080 0.239
C-JT 23350 1.000 0.852 0.131
M-JT-conn 5922 0.507 0.476 0.138
M-JT-comp 2890 0.218 0.196 0.185
M-JTA-conn 10606 0.628 0.549 0.139
M-JTA-comp 4348 0.203 0.162 0.168
tier 10000
C-G 17434 1.000 0.995 0.113
M-G-comp 3614 0.336 0.333 0.258
C-L 16943 1.000 0.976 0.114
M-L-comp 3404 0.291 0.285 0.215
M-R-comp 3790 0.154 0.144 0.255
C-JT 34605 1.000 0.954 0.117
M-JT-conn 14761 0.619 0.603 0.116
M-JT-comp 6104 0.237 0.225 0.186
M-JTA-conn 17526 0.666 0.641 0.120
M-JTA-comp 7273 0.238 0.221 0.177
tier 30000
C-G 23859 1.000 0.999 0.112
M-G-comp 5187 0.343 0.343 0.258
C-L 26920 1.000 0.993 0.112
M-L-comp 5585 0.308 0.305 0.228
M-R-comp 16377 0.154 0.143 0.208
C-JT 49016 1.000 0.982 0.113
M-JT-conn 24847 0.654 0.647 0.109
M-JT-comp 9899 0.272 0.266 0.206
M-JTA-conn 26420 0.683 0.673 0.114
M-JTA-comp 11352 0.262 0.255 0.186
## Predictions (at 30000 unless stated)
B1: FAIL — type recall M-JTA-comp − M-JT-comp -0.011 [-0.019, -0.004]
B2: FAIL — type recall M-JT-comp − M-L-comp -0.039 [-0.067, -0.011]
B3: FAIL — type recall M-JTA-comp − M-G-comp -0.088 [-0.099, -0.076]
B4: PASS — precision M-JTA-comp − C-JT 3000: +0.037 [+0.030, +0.045]; 10000: +0.060 [+0.057, +0.063]; 30000: +0.073 [+0.067, +0.079]
B5: PASS — type recall − M-R-comp 3000 M-JT-comp: +0.117 [+0.095]; 3000 M-JTA-comp: +0.083 [+0.063]; 10000 M-JT-comp: +0.081 [+0.062]; 10000 M-JTA-comp: +0.076 [+0.060]; 30000 M-JT-comp: +0.123 [+0.109]; 30000 M-JTA-comp: +0.112 [+0.098]
B6: PASS — types M-JTA-comp 11352 vs C-JT 49016