| … | … | @@ -0,0 +1,294 @@ |
| 1 | # The library — surface, guarantees, and sync obligations |
| 2 | |
| 3 | **Status:** design. Nothing built. |
| 4 | **Date:** 2026-09-02 |
| 5 | **Depends on:** `spec/design.md` (the decision), `spec/record-vocabulary.md` (the vocabulary), |
| 6 | `spec/markdown-strip.md` (the projection contract + 36 vectors). |
| 7 | **Scope:** this package only. Storage, replication and signing live in `atproto-store`, |
| 8 | `atproto-room` and `atproto-did-peer`; §10 states what those layers require *of us*. |
| 9 | |
| 10 | --- |
| 11 | |
| 12 | ## 1. What this is, and what it is not |
| 13 | |
| 14 | favela-markdown is a **codec and a set of projections**. No storage, no network, no daemon — |
| 15 | the same posture as `atproto-repo`, which is a byte-exact codec with no storage and no network |
| 16 | and is consumed as a sibling by favela-web, ap-bridge and atproto-bridge alike. |
| 17 | |
| 18 | ⛔ **If this package grows a store or a daemon it has become a second peer implementation, and |
| 19 | that is the fork risk** — not the existence of the repo. The substrate is unchanged |
| 20 | (`design.md` §3); only the record file's encoding changes. |
| 21 | |
| 22 | **Three consumers, none of which owns the contract:** |
| 23 | |
| 24 | | consumer | needs | |
| 25 | |---|---| |
| 26 | | favela-web | parse, render, compose, edit | |
| 27 | | `atproto-room/lib/signer.js` `materialize()` | `strip(md) -> {text, facets}` before DAG-CBOR | |
| 28 | | `ap-bridge/lib/as2-project.js` | body to HTML, `source.mediaType` `text/markdown` | |
| 29 | |
| 30 | That is the whole argument for it being a library: each guarantee below is enforced **once** |
| 31 | here instead of being a discipline three packages must separately remember. |
| 32 | |
| 33 | --- |
| 34 | |
| 35 | ## 2. Verified vs assumed |
| 36 | |
| 37 | Read in source on 2026-09-02. Nothing outside this table is verified. |
| 38 | |
| 39 | | claim | where | |
| 40 | |---|---| |
| 41 | | Convergence is the shared StateBase autobase holding `{$oid,size}` pointers; there is no overlay-to-overlay sync | `atproto-store/lib/pointer-substrate.js:1-8` | |
| 42 | | A delete is a `{$deleted:true}` tombstone that "converges symmetrically, like any pointer" | `pointer-substrate.js:60` | |
| 43 | | Bytes are fetched by oid from any connected peer via `ensureLocal(collection, rkey, ptr.$oid)` | `pointer-substrate.js:80-83` | |
| 44 | | `records()` is "the sole input to the MST build", filters `endsWith('.dagcbor')` and calls `dagCbor.decode` unconditionally | `overlay-repo-store.js:24, 299-316` | |
| 45 | | The sign path reads `store.records()` | `repo-manager.js:46,62,81` | |
| 46 | | Signing is byte-identical across lease-holders **because** RFC6979 makes ECDSA deterministic over the same linearized view | `atproto-room/lib/signer.js:1-5` | |
| 47 | | `materialize()` re-encodes records at sign time; it does not use stored bytes | `signer.js:18-33` | |
| 48 | | `migrateInline` depends on canonical encoding — "DAG-CBOR is canonical so the rewrite is byte-identical" — and **asserts** it, skipping on oid mismatch rather than republishing | `pointer-substrate.js:136-145,175-178` | |
| 49 | | `ap-bridge` emits `source: {content, mediaType: 'text/plain'}` | `ap-bridge/lib/as2-project.js:69` | |
| 50 | | `:::spoiler` projects as its **inner text** — concealment is a favela-render feature only | `markdown-strip.md` §4.8 | |
| 51 | |
| 52 | --- |
| 53 | |
| 54 | ## 3. The surface |
| 55 | |
| 56 | ```js |
| 57 | parse(md) -> Record | ParseError // total; fail-closed (§6.2) |
| 58 | serialize(rec) -> md // canonical; serialize(parse(x)) === x |
| 59 | strip(source) -> { text, facets } // markdown-strip.md; byte-deterministic |
| 60 | project(rec, t) -> { output, loss[] } // t: 'bsky' | 'ap' (§4) |
| 61 | maxReach(vis) -> Set<network> // fail-closed (§6) |
| 62 | fromAS2(obj) -> ImportedView // NEVER a Record (§8) |
| 63 | fromBsky(rec) -> ImportedView |
| 64 | STRIP_VERSION -> string // consensus-critical (§10.3) |
| 65 | ``` |
| 66 | |
| 67 | No dependencies beyond a CommonMark parser. Must run under Bare and Node alike (the daemon is |
| 68 | Bare; the tests are Node), so no Node-only builtins. |
| 69 | |
| 70 | --- |
| 71 | |
| 72 | ## 4. Projection returns a loss report |
| 73 | |
| 74 | `project(record, target) -> { output, loss[] }` |
| 75 | |
| 76 | The honesty requirement is stated three separate times in the specs as non-negotiable — |
| 77 | `record-vocabulary.md` §5.3, §10.5.4, and the §4.3 reasoning that *silence on bsky is |
| 78 | indistinguishable from a delivery failure* — and it currently has **no mechanism**. It is a rule |
| 79 | every UI author must remember. |
| 80 | |
| 81 | Every loss is already enumerable from the strip rules. Make it structured output: |
| 82 | |
| 83 | ```js |
| 84 | { rule: 'strip§4.6', kind: 'table-flattened', span: [120, 260] } |
| 85 | ``` |
| 86 | |
| 87 | | kind | source rule | |
| 88 | |---|---| |
| 89 | | `footnote-dropped` | strip §3.9 — reference removed, definition dropped | |
| 90 | | `image-not-in-text` | strip §3.5 — images are embed, not body | |
| 91 | | `table-flattened` | strip §4.6 — "lossy and deliberately unclever" | |
| 92 | | `break-omitted` | strip §4.7 | |
| 93 | | `wikilink-delabelled` | strip §3.6 — no facet, degrades to its label | |
| 94 | | `emphasis-lost` | strip §3.1 — atproto has no bold | |
| 95 | | `spoiler-revealed` | strip §4.8 — **see §5** | |
| 96 | | `truncated` | record-vocabulary §4.3 — 300 graphemes + permalink | |
| 97 | | `visibility-unenforceable` | record-vocabulary §5.2 — atproto has no addressing | |
| 98 | |
| 99 | A composer can then render *"on bsky this becomes…"* from data rather than from a hand-written |
| 100 | warning list that drifts. **No protocol offers this, because no protocol knows what the other one |
| 101 | cannot carry.** |
| 102 | |
| 103 | `loss` MUST be `[]` — not absent — when a post projects losslessly. An empty array is a claim; |
| 104 | a missing field is a silence, and §5.3 exists to forbid silences. |
| 105 | |
| 106 | --- |
| 107 | |
| 108 | ## 5. Directives, pinned — and the spoiler asymmetry |
| 109 | |
| 110 | Closes `markdown-strip.md` §9 item 3, which pins fence *handling* but never the *set*. |
| 111 | |
| 112 | | directive | atproto | ActivityPub | |
| 113 | |---|---|---| |
| 114 | | `:::spoiler` | **revealed** as inner text (strip §4.8) | `summary` + `sensitive: true` — a native CW | |
| 115 | | `:::cw <text>` | revealed; `<text>` dropped | `summary: <text>`, `sensitive: true` | |
| 116 | | `:::details` | revealed | `<details>` in content HTML | |
| 117 | |
| 118 | **The spoiler case is the thesis in one worked example.** The same directive is **lossless to the |
| 119 | fediverse and lossy to bsky** — and today nothing tells the author that a spoilered post arrives |
| 120 | un-spoilered on bsky. §4's loss report is what surfaces it at compose time. |
| 121 | |
| 122 | `record-vocabulary.md` §6 already sketches the AP slot — |
| 123 | `net.favela.ext.ap: { summary: "cw: spoilers", sensitive: true }` — without ever connecting it to |
| 124 | `:::spoiler`. This connects them. The extension carries the AS2 value; the directive is how a human |
| 125 | writes it. |
| 126 | |
| 127 | ⛔ Adding a directive is a **spec change**, per strip §2.1, because it changes the bytes of every |
| 128 | record that uses it. It is not a plugin point. Do not build a registry. |
| 129 | |
| 130 | --- |
| 131 | |
| 132 | ## 6. Visibility and reach, enforced in the codec |
| 133 | |
| 134 | ### 6.1 The lattice |
| 135 | |
| 136 | `maxReach(visibility) -> Set<network>`, total and deny-by-default. `project()` **refuses** to emit |
| 137 | a bsky projection for a record whose `visibility` is not Public, because atproto cannot express |
| 138 | addressing at all (`record-vocabulary.md` §5.2) and a projection it cannot restrict is a leak. |
| 139 | |
| 140 | Written here once, this check would otherwise be written three times — favela-web, hyper-proto-peer, |
| 141 | ap-bridge — and it is the one place where a defect is a **privacy** bug rather than a rendering bug. |
| 142 | |
| 143 | Reach is derivable from visibility; visibility is not derivable from reach (`design.md` §5.1). The |
| 144 | library computes the first and refuses the impossible; the author still chooses within it (§5.4's |
| 145 | independent toggles). |
| 146 | |
| 147 | ### 6.2 Malformed frontmatter is local-only, never public |
| 148 | |
| 149 | `strip` is specified as **total — never throws** (strip §2). Frontmatter parsing has no such rule, |
| 150 | and the gap is dangerous: JSON-in-a-fence was chosen over YAML precisely because `audience: no` |
| 151 | parsing as `false` is a privacy bug. |
| 152 | |
| 153 | **Rule:** a record whose frontmatter does not parse is readable, renderable and **not publishable**. |
| 154 | It degrades to local-only — never to Public. Fail-closed, in the same direction as §6.1. |
| 155 | |
| 156 | --- |
| 157 | |
| 158 | ## 7. Conformance as a shipped artifact |
| 159 | |
| 160 | `design.md` §0 states the payoff as *"this is what makes a second implementation tractable."* Then |
| 161 | make it tractable rather than merely asserted: |
| 162 | |
| 163 | 1. **The 36 vectors, plus a runner** — `favela-markdown conformance <impl>` — so a second |
| 164 | implementation self-certifies instead of guessing. |
| 165 | 2. **Property laws the vectors cannot express:** |
| 166 | - `serialize(parse(x)) === x` for canonical input (see §10.4 — this is load-bearing) |
| 167 | - `strip` idempotent under the §5 normalisation |
| 168 | - `project` pure: same record + same `STRIP_VERSION` → same bytes |
| 169 | - every facet range lies inside `text` and the set is sorted, non-overlapping (strip §6) |
| 170 | 3. **Fuzz UTF-8 specifically.** strip §8 names it: byte-vs-character offsets is *"the single most |
| 171 | likely defect here and produces facets that land on the wrong characters only for non-ASCII |
| 172 | authors."* Thirty-six hand-written vectors may never catch it; a fuzzer will. |
| 173 | |
| 174 | --- |
| 175 | |
| 176 | ## 8. Import — the reverse projection, explicitly not canonical |
| 177 | |
| 178 | `fromAS2(obj)` / `fromBsky(record)` produce an **`ImportedView`**, so the appview holds one model |
| 179 | regardless of which network a post came from. This is what collapses `record-vocabulary.md` §9 — |
| 180 | the read path, *"the largest unestimated part of this design"* — from a per-collection branch in |
| 181 | threading, search, notifications, RSS and bookmarks into one shape. |
| 182 | |
| 183 | ⛔ **An `ImportedView` is a distinct type and can never be serialized as a canonical record.** The |
| 184 | mistake must not be expressible in the API, because `record-vocabulary.md` §8 records what it costs: |
| 185 | *"Never re-encode a record you did not author"* — a second writer re-encoded two of wj62's posts, |
| 186 | dropped `langs`, and the bytes stopped hashing to their converged pointers. They were unrecoverable. |
| 187 | |
| 188 | --- |
| 189 | |
| 190 | ## 9. CLI |
| 191 | |
| 192 | `bin/` already exists and is empty. Four commands: |
| 193 | |
| 194 | ``` |
| 195 | favela-markdown strip <file> # text + facets |
| 196 | favela-markdown project <file> --target bsky # output + loss report |
| 197 | favela-markdown explain <file> # source, both projections, all losses, side by side |
| 198 | favela-markdown lint <file> # frontmatter, visibility/reach, directive set |
| 199 | ``` |
| 200 | |
| 201 | Every projection defect so far has been diagnosed by reading daemon logs. This makes it a one-liner |
| 202 | against a file, offline. |
| 203 | |
| 204 | --- |
| 205 | |
| 206 | ## 10. Sync obligations — what the substrate requires of us |
| 207 | |
| 208 | **favela-markdown synchronises nothing.** Convergence is the autobase; bytes are the overlay; |
| 209 | discovery is `atproto-did-peer`. A pointer is a hash of a file and does not care what encoding the |
| 210 | file is in, so for the **canonical** record `design.md` §9's "does not change replication" holds |
| 211 | exactly. |
| 212 | |
| 213 | What changes is that **the strip moves inside the signing path**, and signing is an elected, |
| 214 | movable role. The four obligations below follow from that, and they are the reason this section |
| 215 | exists rather than being left to the consumer. |
| 216 | |
| 217 | ### 10.1 ⛔ `records()` must learn the encoding BEFORE any `.md` is written |
| 218 | |
| 219 | `overlay-repo-store.js:299-316` — *"the sole input to the MST build"* — filters |
| 220 | `n.endsWith('.dagcbor')` and calls `dagCbor.decode(bytes)` unconditionally. A `.md` file is |
| 221 | therefore **invisible** to it, and so to `repo-manager.js:46`. |
| 222 | |
| 223 | Flip storage to `.md` without teaching `records()`, and the repo signs as **empty** — a |
| 224 | delete-every-record commit. That is the wj62-wipe shape that `pointer-substrate.js:105-155` was |
| 225 | written to prevent, arriving from a direction its guard does not cover: the guard *"aborts only when |
| 226 | record FILES exist"*, and by its own count none do. |
| 227 | |
| 228 | **Requirement:** `EXT` becomes **per-collection**, not a module constant — `.md` with a markdown |
| 229 | parse for `net.favela.*`, `.dagcbor` with `dagCbor.decode` for `app.bsky.*`. Both live in the same |
| 230 | store, which is exactly the double-write of `record-vocabulary.md` §7. Ship this change, with its |
| 231 | own test, **before** a single `.md` record exists. |
| 232 | |
| 233 | ### 10.2 The double-write is what keeps atproto peers in sync |
| 234 | |
| 235 | Restated here because §10.1 makes it structural: a public post is two files sharing one rkey |
| 236 | (§3.1). The canonical `.md` is what favela peers converge on; the derived `.dagcbor` is what the |
| 237 | signed MST commits to and what any atproto reader sees. `record-vocabulary.md` §7's four guarantees |
| 238 | (delete cascades in the same batch, edit re-derives, the projection is never a source of truth, |
| 239 | `sign()` materialises both before signing) are **sync** requirements, not tidiness. |
| 240 | |
| 241 | ### 10.3 The strip is consensus-critical — version it, and guard it in band |
| 242 | |
| 243 | `signer.js:1-5`: *"because ECDSA here is RFC6979-deterministic, any lease-holder that signs the |
| 244 | same linearized view + rev/prev produces byte-identical output."* That byte-identity is what makes |
| 245 | moving the signer between peers safe. |
| 246 | |
| 247 | Under this design the lease-holder must **strip before it encodes**, so the guarantee now rests on |
| 248 | the strip as well as on RFC6979. `markdown-strip.md` §1 argues determinism as cross-*implementation* |
| 249 | interop; the sharper risk is same implementation, **different version, same fleet** — peer A on |
| 250 | strip v1, peer B on strip v2, the signer role moving between them, and one canonical record yielding |
| 251 | two signed chains. That is a fork, not a rendering difference, and it is invisible from every local |
| 252 | surface. |
| 253 | |
| 254 | **Requirement:** export `STRIP_VERSION`, stamp it into the autobase, and have an under-versioned |
| 255 | peer **refuse the signer role** rather than sign with an older strip. This is Keet's interrupt |
| 256 | pattern, already used twice in this codebase — pear-git-v3's in-band schema guard in the replicated |
| 257 | OrgEventLog, and the autobee migration guard. It converts a silent fork into a loud halt. |
| 258 | |
| 259 | ### 10.4 Two open questions become consensus hazards |
| 260 | |
| 261 | - **Grapheme segmentation is unpinned** (`markdown-strip.md` §9 item 1) and the 300-grapheme |
| 262 | truncation (§4.3) happens inside the record that gets **signed**. Two runtimes on different |
| 263 | Unicode versions truncate at different byte counts and produce different CIDs. Pin the |
| 264 | segmentation algorithm and its Unicode version; treat a bump as a `STRIP_VERSION` bump. |
| 265 | - **Round-trip byte-identity is not free** the way DAG-CBOR's is. `migrateInline` leans on |
| 266 | *"DAG-CBOR is canonical so the rewrite is byte-identical"* and **asserts** it, skipping any key |
| 267 | whose re-encoded oid differs rather than republishing changed content. Markdown's equivalent — |
| 268 | `serialize(parse(x)) === x` — must be engineered: JSON frontmatter key order, whitespace, trailing |
| 269 | newline. This is why §7's round-trip law is load-bearing and not a nicety. |
| 270 | |
| 271 | --- |
| 272 | |
| 273 | ## 11. Open |
| 274 | |
| 275 | 1. **Embeds and attachments** across atproto blob refs, AS2 `attachment` and markdown image syntax |
| 276 | (`record-vocabulary.md` §11 #2). The image/video downsampling work belongs here — media policy is |
| 277 | an authoring-time decision of the native record, not a composer concern. |
| 278 | 2. **Threading's markdown-native spelling.** `{root, parent}` is settled as canonical (§3.3); how it |
| 279 | is written in a file is not. |
| 280 | 3. **Community and list URIs** for `visibility` (§11 #4). |
| 281 | 4. **Reply visibility inheritance** (§11 #5) — almost certainly inherit, but the failure mode is |
| 282 | publishing to the wrong people, so it must be explicit rather than emergent. |
| 283 | 5. **Where `STRIP_VERSION` is stamped** — the room's StateBase, the commit, or both. §10.3 fixes the |
| 284 | pattern, not the location. |
| 285 | |
| 286 | --- |
| 287 | |
| 288 | ## 12. What this does not claim |
| 289 | |
| 290 | - It does not synchronise anything. §10 is a list of obligations *on* this library, not features of it. |
| 291 | - It does not make `visibility` enforceable on atproto. Nothing can. |
| 292 | - It does not remove the double-write; §10.2 makes it structural and bounded. |
| 293 | - A loss report does not reduce loss. It makes loss legible, which is what §5.3 has always demanded |
| 294 | and never had a mechanism for. |