OverviewFilesCommitsIssuesDocsReleasesPeers

favela-markdown commits / 5be1311

spec: adopt the record vocabulary and the strip contract
Moved from favela-web/spec/ — record-vocabulary.md (the net.favela.post vocabulary, carrying the 2026-08-31 §0 governing decision), markdown-strip.md (the projection down to text + facets, dialect pinned in §2.1) and the 36 conformance vectors. They describe the record, not the web app. Three consumers project from them: favela-web, atproto-room/lib/signer.js materialize(), and ap-bridge/lib/as2-project.js. The library now owns its own contract, and the vectors have a package to be tested against. README and design.md updated to point at the specs in place; nothing else rewritten. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
5be131142a0453ac4a521b9066b175a511e7c098
by plankatron <fleeky@gmail.com> · 2026-09-03T00:48:40.000Z

Changed files (5) +1125 -6

README.md +9 -4
……@@ -27,7 +27,12 @@ CIDs.** Those are the bridge's problem.
2727
2828Design. Nothing is built.
2929
30- `spec/` — the design, worked out here
31- Prior art: `favela-web/spec/record-vocabulary.md` §0 (the decision),
32 `favela-web/spec/markdown-strip.md` + 36 conformance vectors (the projection down to
33 plain text + facets)
30- `spec/design.md` — the decision and what it changes
31- `spec/record-vocabulary.md` — the `net.favela.post` vocabulary
32- `spec/markdown-strip.md` + `spec/vectors/markdown-strip.json` — the projection down to
33 plain text + facets, with 36 conformance vectors
34
35The vocabulary and strip specs moved here from `favela-web/spec/` on 2026-09-02: they
36describe the record, not the web app, and three separate consumers project from them —
37favela-web, the atproto signer (`atproto-room/lib/signer.js` `materialize()`) and
38`ap-bridge/lib/as2-project.js`.
spec/design.md +4 -2
……@@ -3,8 +3,10 @@
33**Status:** design. Nothing built.
44**Date:** 2026-08-31
55**Supersedes:** `hyper-social/docs/ATPROTO-P2P-BRIDGE.md` §1 as the canonical-path decision.
6**Depends on:** `favela-web/spec/record-vocabulary.md` (vocabulary), `favela-web/spec/markdown-strip.md`
7(the projection contract, dialect pinned in §2.1, 36 vectors).
6**Companion specs (this repo):** `spec/record-vocabulary.md` (the `net.favela.post` vocabulary),
7`spec/markdown-strip.md` (the projection contract, dialect pinned in §2.1) and
8`spec/vectors/markdown-strip.json` (36 conformance vectors).
9Both moved here from `favela-web/spec/` on 2026-09-02 — the library owns its own contract.
810
911---
1012
spec/markdown-strip.md +220 -0
……@@ -0,0 +1,220 @@
1# The strip — `source.content` (markdown) → `text` + `facets`
2
3**Status:** specification. Not implemented.
4**Date:** 2026-08-05
5**Closes:** `record-vocabulary.md` §4.2 and §11 item 1.
6**Vectors:** `spec/vectors/markdown-strip.json` — machine-readable, wire them into a test.
7
8---
9
10## 1. Why this must be a specification
11
12The strip's output is what **bsky sees** and what a **CID commits to**. If two implementations
13produce different `text` from the same `source.content`, they produce different
14`app.bsky.feed.post` records from one canonical record, and `atproto:cid` — the cross-network dedup
15key the whole projection model rests on — stops identifying content.
16
17So "whatever the parser does" is not an answer. Every rule below is normative. Anything a rule does
18not cover is a **defect in this document**, not an implementation choice.
19
20---
21
22## 2. Contract
23
24```
25strip(source: string) -> { text: string, facets: Facet[] }
26```
27
28- **`text`** is UTF-8 plain text. No markup survives.
29- **`facets`** are `app.bsky.richtext.facet` objects with `index: { byteStart, byteEnd }` —
30 **byte offsets into `text`**, UTF-8, half-open `[byteStart, byteEnd)`.
31- **Deterministic**: same input bytes → same output bytes, on every implementation, forever.
32- **Total**: never throws. Malformed markdown degrades to its literal text.
33
34### 2.1 Dialect — pinned
35
36**CommonMark 0.31.2**, plus exactly these extensions and no others:
37
38| extension | syntax |
39|---|---|
40| strikethrough | `~~x~~` |
41| tables | GFM pipe tables |
42| footnotes | `[^1]` and its definition |
43| wiki-links | `[[Page]]`, `[[Page\|label]]` |
44| directives | `:::name` … `:::` (block only) |
45| hashtags | `#tag` (see §3.7) |
46
47An implementation MUST NOT enable further extensions. A new one is a spec change, because it
48changes the bytes of every record that uses it.
49
50---
51
52## 3. Inline rules
53
54Applied to the inline content of each block after §4 produces block text.
55
56### 3.1 Emphasis, strong, strikethrough, code spans
57Markers removed, inner text kept verbatim. `**a**` → `a`. `` `x` `` → `x`. `~~a~~` → `a`.
58Nesting resolves innermost-first; the result is the concatenated text content.
59**No facet is emitted** — atproto has no bold.
60
61### 3.2 Links
62`[label](url)` → `label`, plus one `#link` facet with `uri = url` spanning `label`'s bytes.
63
64- The label is stripped by these same inline rules first, so `[**a**](u)` → `a` with the facet over
65 `a`.
66- An **empty label** (`[](u)`) emits the URL as text and a facet over it.
67- ⛔ **`uri` MUST be `http:` or `https:`** — anything else (including `javascript:`) emits the label
68 as plain text with **no facet**. This mirrors the existing render rule that refuses non-http URIs.
69- The `did:` scheme is the one exception, and only for mentions — §3.4.
70
71### 3.3 Autolinks and bare URLs
72`<https://x>` → `https://x` with a `#link` facet. A bare URL in text likewise.
73
74Bare-URL detection is **exactly** favela-web's shipped `URL_RE`:
75```
76/\bhttps?:\/\/[^\s<]+[^\s<.,:;!?)\]}'"]/g
77```
78so trailing punctuation stays outside the facet ("see https://x.com." keeps the full stop as text).
79This regex is normative; changing it changes records.
80
81### 3.4 Mentions
82`[@alice](did:plc:xyz)` → `@alice`, plus a `#mention` facet with `did = did:plc:xyz`.
83
84⛔ **A mention MUST carry a resolved DID in the link target.** A bare `@alice` with no target is
85**plain text with no facet** — never a guess. This preserves the existing rule (favela-web 0.3.11):
86*a mention resolves to a DID or is omitted.*
87
88### 3.5 Images
89`![alt](url)` produces **no text and no facet**. Images are `embed`, not body text.
90
91The embed pipeline consumes them separately, in source order. An image is therefore invisible to the
92strip — a post whose entire source is one image strips to empty `text`, which is valid.
93
94### 3.6 Wiki-links
95`[[Page]]` → `Page`. `[[Page|label]]` → `label`. **No facet** — a wiki-link is favela-internal and
96has no atproto or AS2 equivalent, so it degrades to its label rather than inventing a URL.
97
98### 3.7 Hashtags
99`#tag` → `#tag` (**the hash is kept in the text**), plus a `#tag` facet whose `tag` value is the
100label **without** the `#`.
101
102Matched as `#` preceded by start-of-text or whitespace, followed by 1–64 characters from
103`[A-Za-z0-9_]` plus any non-ASCII letter. A `#` inside a word (`a#b`) or followed by a digit only
104(`#1`) is literal text. `#` at the start of a line is a heading (§4.2), not a tag.
105
106### 3.8 Hard breaks
107A backslash at end of line, or two-or-more trailing spaces, → a single `\n`. Soft line breaks inside
108a paragraph → a single space.
109
110### 3.9 Footnotes
111A reference `[^1]` is **removed entirely** (no text, no facet). Its definition block is **dropped**.
112Footnotes are a favela-render feature with no projection target.
113
114### 3.10 HTML
115Raw inline and block HTML is emitted as its **literal source text**, unparsed, with no facet.
116We never execute or interpret it; escaping is the renderer's job on the way out.
117
118---
119
120## 4. Block rules
121
122### 4.1 Joining
123Block texts are joined with exactly `\n\n`. A block producing empty text is **omitted entirely**,
124including its separator.
125
126### 4.2 Headings
127ATX (`## T`) and setext both → `T`. The marker, and any closing `##`, are removed. A heading is a
128block like any other — no extra spacing, no case change.
129
130### 4.3 Code blocks
131Fenced and indented → the code contents **verbatim**, including internal newlines, without the
132fence or the info string, and without re-indenting. Trailing newline inside the block is trimmed.
133
134### 4.4 Blockquotes
135`> x` → `x`. The marker is removed; nested quotes flatten. No prefix is added — a `> ` in plain
136text would be a rendering convention we would then have to specify for every consumer.
137
138### 4.5 Lists
139Each item's content becomes one line, joined with `\n` **within** the list block; the list as a
140whole is one block for §4.1 purposes.
141
142- Bullet items are prefixed `• ` (U+2022, space).
143- Ordered items are prefixed with their number and `. ` — using the **rendered sequence**
144 (1., 2., 3.), not the literal source numbers, so `1. 1. 1.` strips as `1. 2. 3.`.
145- Nested lists indent by two spaces per level.
146
147A marker is kept here, unlike blockquotes, because without one a list becomes indistinguishable
148from consecutive paragraphs and loses its meaning entirely.
149
150### 4.6 Tables
151Rows joined with `\n`; cells joined with ` | `. The delimiter row (`---|---`) is dropped. Leading
152and trailing pipes are dropped. Cell content is stripped by the inline rules.
153
154This is lossy and deliberately unclever: a table has no faithful plain-text form, and any attempt to
155align columns would make the output depend on font and width.
156
157### 4.7 Thematic breaks
158`---` → the block is **omitted** (no text, no separator).
159
160### 4.8 Directives
161`:::name` … `:::` → the **contents**, stripped by these rules; the fences and the name are removed.
162A `:::spoiler` therefore projects as its inner text. Concealment is a favela-render feature, and
163silently dropping the content would misrepresent the post everywhere else.
164
165---
166
167## 5. Normalisation
168
169Applied to `text` **after** assembly, **before** facet offsets are computed.
170
1711. **CRLF and CR → LF.** Applied to the *input* before parsing.
1722. **Trailing whitespace removed** from every line.
1733. **Three or more consecutive newlines collapse to two.**
1744. **Leading and trailing whitespace removed** from the whole text.
1755. ⛔ **NO Unicode normalisation.** Do not apply NFC/NFD. Normalising would change bytes the author
176 wrote, and therefore the CID, for reasons invisible to them.
1776. ⛔ **NO case folding, NO smart quotes, NO typographic substitution.**
178
179---
180
181## 6. Facets
182
1831. **Sorted by `byteStart` ascending.** Required for byte-determinism of the encoded record.
1842. **No overlaps.** If two features would cover overlapping ranges, the one starting earlier wins;
185 on an equal start, the longer wins; on both equal, precedence is
186 **mention > link > tag**.
1873. A facet whose range is empty after stripping is **dropped**.
1884. Offsets are recomputed against the **final** `text` from §5 — never against an intermediate.
1895. `features` arrays contain exactly one feature. We never emit a multi-feature facet.
190
191---
192
193## 7. Length
194
195The strip does not truncate. §4.3 of `record-vocabulary.md` owns the 300-grapheme bsky rule, and it
196applies to the **projection**, not to `text`. `text` is the full canonical body.
197
198---
199
200## 8. Test vectors
201
202`spec/vectors/markdown-strip.json`. Each entry is `{ name, why, source, text, facets }`, with
203`facets` as `{ byteStart, byteEnd, type, value }` — a compact form to be expanded into
204`app.bsky.richtext.facet` shape by the test harness.
205
206An implementation is conformant when it reproduces **every** vector byte-for-byte. The multi-byte
207cases are not optional extras: they are the ones that catch an implementation using character
208offsets instead of UTF-8 byte offsets, which is the single most likely defect here and produces
209facets that land on the wrong characters only for non-ASCII authors.
210
211---
212
213## 9. Open
214
2151. **Grapheme counting for the bsky projection** is specified in `record-vocabulary.md` §4.3 but the
216 segmentation algorithm is not pinned. It must be, for the same determinism reason.
2172. **Embed extraction order** (§3.5) is asserted to be source order but not otherwise specified;
218 it belongs in an embed spec, which does not exist yet.
2193. **Directive names** are not enumerated. Only the fence handling is specified; which names favela
220 renders specially is a render concern, but the *set* should be pinned before use.
spec/record-vocabulary.md +558 -0
……@@ -0,0 +1,558 @@
1# The record vocabulary — `net.favela.post`
2
3**Status:** design, not implemented. Nothing here has been built.
4**Revised 2026-08-31** (§0, §4.1) — the canonical record becomes a *markdown file*, not a
5DAG-CBOR record. atproto stops being the canonical P2P path and becomes a projection like any other.
6**Date:** 2026-08-05
7**Scope:** hyper-proto-peer (authoring), atproto-store (storage), atproto-bridge + ap-bridge
8(projection), favela-web (compose + render).
9
10---
11
12## 0. The governing decision (2026-08-31)
13
14**atproto was never meant to be the canonical P2P path. It became one by default.** Building the
15peer layer on atproto's data model imported atproto's constraints — MST derivation, DAG-CBOR
16byte-exactness, lexicon field ceilings, blob CIDs sealed inside signed records — into a network
17that only ever wanted to *interface* with atproto. That machinery is genuinely useful, but it
18belongs at the bridge/relay layer, not underneath every full-mode peer.
19
20**So: the canonical record is a markdown document. Every network — atproto and ActivityPub alike —
21is a lossy projection derived from it on read.**
22
23```
24records/net.favela.post/3msdnle65lk22.md canonical. what the author typed.
25records/app.bsky.feed.post/3msdnle65lk22.dagcbor derived projection (bridge builds the MST from it)
26https://ap.7rnx.net/ap/objects/3msdnle65lk22 derived projection
27```
28
29This is a smaller change than it sounds, because `atproto-store/lib/overlay-repo-store.js` already
30stores one file per record and *derives* MST/commit/CAR on read — "nothing atproto-specific is
31stored beyond the record bytes + a tiny head.json". Only the encoding of the record file changes.
32
33**What it buys.** A full-mode peer needs: parse markdown, one file per record, hyper-overlay
34replication. **No MST, no DAG-CBOR, no byte-exact CIDs.** Those become the bridge's problem, where
35they belong. This is what makes a second implementation tractable.
36
37### 0.1 The two governing properties
38
39Everything else is negotiable; these are not.
40
41- **Mutable.** A post is a file that can be rewritten. `OverlayRepoStore` is already
42 overwrite-in-place on edit and unlink on delete, so on-disk size is O(current records), not
43 O(operations-ever).
44- **Extensible.** New extensions can be added at any time without migrating old records, *because
45 projections are derived on read*. Old posts stay valid; new projections simply carry more. This is
46 the opposite of a lexicon change, which is a migration.
47
48**Consequence:** mutable content means the content hash changes on every edit, so **the rkey is the
49stable identity, never the hash.** §3.1 already depends on this, and it is why the 22 already-
50federated AP object URIs keep resolving.
51
52---
53
54## 1. The decision
55
56**The canonical stored record moves from `app.bsky.feed.post` to `net.favela.post`. bsky and
57ActivityPub become projections of it. The P2P side reads the canonical record directly and is the
58only place the whole post exists.**
59
60### Why — the vocabulary was never chosen
61
62Today the file on disk *is* a bsky post:
63
64```
65records/app.bsky.feed.post/3msdnle65lk22.dagcbor
66 { "$type": "app.bsky.feed.post", "text": "…", "createdAt": "…", "langs": ["en"] }
67```
68
69That is bsky's semantic model. It has the fields bsky decided a post has — and no addressing,
70because on bsky everything is public.
71
72The existing favela lexicon confirms this was inherited rather than designed. `net.favela.local.post`
73is *literally the bsky shape minus `langs`*:
74
75```json
76{ "$type": "net.favela.local.post", "text": "bork", "createdAt": "2026-07-17T18:11:00.877Z" }
77```
78
79So one application's model became the ceiling for every projection. ActivityPub can express
80addressing natively — `to`/`cc` are already being emitted — but the stored record has nowhere to
81say what the addressing should be, so it is hardcoded to Public.
82
83This document does not fix a bug. It makes a decision that was previously a default.
84
85---
86
87## 2. What is verified vs assumed
88
89Everything in §2 was checked during design. Nothing else in this document should be treated as
90verified.
91
92| claim | status |
93|---|---|
94| `net.favela.local.post` is the bsky shape minus `langs` | ✅ decoded from disk |
95| 148 existing `app.bsky.feed.post` records, **zero** with extra fields | ✅ scanned |
96| Lexicon validation is **open** — unknown fields "should be ignored" | ✅ [atproto spec](https://atproto.com/specs/lexicon) |
97| …but preservation is **not guaranteed** — other PDSs "might strip" | ✅ [discussion #2126](https://github.com/bluesky-social/atproto/discussions/2126) |
98| Custom fields are safe only if they cannot collide with future lexicon fields | ✅ [Custom Schemas](https://docs.bsky.app/docs/advanced-guides/custom-schemas) |
99| ap-bridge already emits JSON-LD namespaced extensions | ✅ live: `@context` + `atproto:uri`, `atproto:cid` |
100| ap-bridge already emits `to` / `cc` / `contentMap` | ✅ live, hardcoded to Public |
101| AS2 `source` `{content, mediaType}` already emitted | ✅ live, `text/plain` |
102| `objectIdFor({host, rkey})` → `/ap/objects/<rkey>`, **rkey only** | ✅ read source |
103| `rkeyFromObjectId` matches a **single** path segment | ✅ read source |
104| **22 objects already cached at prsm.space** under that URI shape | ✅ Mastodon API |
105| AP federation works end to end | ✅ 22 public statuses incl. today's |
106
107---
108
109## 3. Core model
110
111### 3.1 The rkey pairing — the mechanism everything rests on
112
113**A canonical record and its atproto projection share one rkey, differing only by collection.**
114
115```
116canonical at://<did>/net.favela.post/3msdnle65lk22
117projection at://<did>/app.bsky.feed.post/3msdnle65lk22
118AP object https://ap.7rnx.net/ap/objects/3msdnle65lk22
119```
120
121This is not a convenience. It resolves four problems at once:
122
123- **No mapping table.** Given the canonical, the projection is derivable, and vice versa.
124- **The AP object URI never changes.** ap-bridge keeps deriving from the rkey, so the 22 already-
125 federated URIs keep resolving. The collection-in-path change discussed earlier is **not needed**.
126- **ap-bridge's pure-function id-map survives** — it never fronts a second collection.
127- **Delete and edit cascade mechanically** by rkey.
128
129### 3.2 The core fields
130
131The intersection of what atproto and AS2 both express, checked field by field against both specs:
132
133```json
134{
135 "$type": "net.favela.post",
136 "createdAt": "2026-08-05T13:38:23.294Z",
137 "langs": ["en"],
138
139 "source": { "content": "hello **world**", "mediaType": "text/markdown" },
140 "text": "hello world",
141 "facets": [ /* byte offsets into `text` */ ],
142
143 "reply": { "root": {…}, "parent": {…} },
144 "embed": { /* images / quote / external */ },
145
146 "audience": ["https://www.w3.org/ns/activitystreams#Public"]
147}
148```
149
150`audience` is the only genuinely new field. Everything else is what is already written today,
151because those parts of bsky's model are sound and reinventing them would be churn.
152
153### 3.3 Two representation decisions, settled on recoverability
154
155**Body: plain text + byte-offset facets is canonical; markdown is the source; HTML is derived.**
156
157Markdown → HTML is deterministic. HTML → byte-offset facets is not reliably recoverable. So the
158canonical machine-readable body is `text` + `facets`, and `source` carries what the human typed.
159This is not "atproto won" — it is the direction that loses nothing.
160
161**Threading: `{root, parent}` is canonical, `inReplyTo` is derived.** atproto's pair carries strictly
162more than AS2's single URI, and the projection down is trivial.
163
164### 3.4 Markdown links *are* facets
165
166`[@alice](did:plc:xyz)` carries display text plus resolved target — exactly what a facet is. A
167CommonMark parser yields exact source ranges, so facet offsets are **derived from the parse**, never
168hand-maintained. This preserves the existing rule (favela-web 0.3.11) that a mention resolves to a
169DID or is omitted, never guessed.
170
171---
172
173## 4. The P2P side is the rich side
174
175**The canonical record is the P2P record, and the P2P side carries everything**: markdown as
176authored, the full extension set, no length ceiling, no facet flattening. bsky and ActivityPub are
177the *edges*, and edges are lossy.
178
179This inverts the usual bridge posture, where the local client degrades to the lowest common
180denominator of its targets — which is how `bridge-envelope`'s `content: string` would have capped
181every projection, and why ap-bridge refused it.
182
183**Richness belongs to the representation, not to one app.** favela-web is the first client of the
184P2P side, not the definition of it. `favela-cli`, `favela-native`, another peer's implementation, or
185anything reading the record files off a shared overlay gets the same complete record. A client that
186renders less is choosing to; nothing in the format holds it back.
187
188This is also what makes the format worth having. If the richest view were a single web app, the
189extra fields would be that app's private state. Because the richest view is the *record*, any P2P
190reader — including one nobody has written yet — sees the whole post.
191
192### 4.1 Dialect — CommonMark, and extensions that carry the rest
193
194**CommonMark is the base. Extensions are markdown syntax carrying whatever plain markdown cannot.**
195
196The extension set is more expansive than any target network because it does not have to ask
197permission: the canonical record keeps everything, and each bridge takes only what it can carry.
198
199**Unknown extensions need no special rule — markdown already degrades on its own.** A reader that
200does not know `:::spoiler` renders the text inside it. Falling through to literal text *is* the
201behaviour, not something this document has to invent. That is a property of the format, and it is
202why the extension set can grow without coordination.
203
204**The dialect is already pinned, in `markdown-strip.md` §2.1: CommonMark 0.31.2 plus exactly
205six extensions** — strikethrough, GFM tables, footnotes, `[[wiki-links]]`, block directives
206(`:::name`), and hashtags. *"An implementation MUST NOT enable further extensions. A new one is a
207spec change, because it changes the bytes of every record that uses it."* Fenced code is CommonMark
208core, not an extension. Markdown links are mentions — `[@alice](did:plc:xyz)` carries display text
209plus resolved target, so facets are *derived from the parse* when projecting (§3.4), never
210hand-maintained.
211
212**The one exception: addressing.** Getting a spoiler wrong makes a post render plainly. Getting
213"who can see this" wrong sends it to the wrong people. So `audience` is a **typed field, never
214parsed out of loose text** — a spelling quirk must not be able to change who receives a post.
215Everything else may be casual.
216
217### 4.2 The strip — must be specified, not merely implemented
218
219`source.content` (markdown) → `text` (plain) must be **byte-deterministic across implementations**,
220or two implementations produce different bsky records from one canonical record and `atproto:cid`
221stops identifying content.
222
223The specification MUST fix, at minimum: emphasis markers removed and inner text kept; link syntax
224reduced to its display text with a facet at that byte range; headings reduced to their text; code
225fences reduced to their contents; block structure joined with `\n\n`; `[[wiki-link]]` reduced to its
226label; directives removed with contents kept. Anything unspecified is a defect in this document.
227
228### 4.3 Length
229
230A favela post has no length limit. `app.bsky.feed.post` is capped at 300 graphemes.
231
232**Rule: if the stripped text exceeds 300 graphemes, the bsky projection is truncated at a grapheme
233boundary and a permalink to the favela post is appended.** This is standard POSSE behaviour and is
234preferred over omitting the post, because silence on bsky is indistinguishable from a delivery
235failure — a class of confusion this project has paid for repeatedly.
236
237The AP projection is **never** truncated; AS2 has no length limit.
238
239---
240
241## 5. Audience
242
243### 5.1 Shape
244
245A list of URIs. `https://www.w3.org/ns/activitystreams#Public` means everyone. A community is its
246room key as a URI; a list is its `at://` URI.
247
248URIs rather than protocol names (`["bsky","fediverse"]`) because a URI names *who*, not *where*.
249Adding a protocol later adds no field — it is another audience that either includes a given
250recipient or does not. Protocol names as flags would need a new field per protocol forever, and
251would put knowledge of projections inside the record, which is the thing this design exists to
252avoid.
253
254### 5.2 What each projection does with it
255
256| projection | enforcement |
257|---|---|
258| **P2P (favela-web and any other client)** | not a projection at all — it is the substrate (§5.4). It renders everything it holds, and `audience` never withholds anything from it. |
259| **ActivityPub** | **native.** `audience` maps onto `to`/`cc`. ap-bridge already emits these — hardcoded to Public. The change is to read the value instead of assuming it. |
260| **atproto / bsky** | **none.** atproto has no addressing. The only lever is whether the `app.bsky.feed.post` projection is written at all. |
261
262### 5.3 The honesty requirement — non-negotiable
263
264The atproto asymmetry is not a limitation of this design; it is a property of atproto, and the UI
265must not conceal it.
266
267**Not writing a bsky projection means bsky does not INDEX the post. It does not mean the fediverse
268or anyone else cannot read it.** The canonical record still rides the firehose and is fetchable by
269anyone who fronts the DID. This is the same caveat DESIGN §10.2 already makes about local-only
270("unindexed by bsky, not unreplicated"), and it MUST be surfaced wherever audience is chosen.
271
272Composer copy must never imply erasure or privacy. Until encryption lands, the honest phrasing
273remains "community, not secret".
274
275### 5.4 ★ Local is the SUBSTRATE, not a scope — reach is N independent toggles
276
277**Every post is local.** The canonical record always exists on the P2P side; that is what authoring
278one means. "Local" is therefore not a destination you opt into, and there is no way to opt out of
279it. What the author actually chooses is **which projections to emit**.
280
281So reach is not a set of mutually exclusive scopes. It is one **independent toggle per projection**,
282with the P2P side always on:
283
284```
285local (canonical) ── always, not a choice
286 ├─ bsky ── toggle
287 └─ fediverse ── toggle
288```
289
290Which yields the same four combinations, described honestly:
291
292| bsky | fediverse | what it is | canonical | `app.bsky.feed.post` | AS2 projection |
293|---|---|---|---|---|---|
294| ✅ | ✅ | public | ✅ | written | emitted |
295| ✅ | ✗ | atproto only | ✅ | written | withheld |
296| ✗ | ✅ | fediverse only | ✅ | **not written** | emitted |
297| ✗ | ✗ | local only | ✅ | not written | withheld |
298
299**Why the framing matters, and why the earlier "four scopes" table was wrong.** Enumerating scopes
300implies a named scope per combination, so adding a third network would mean **eight** scopes and a
301UI listing all of them. Toggles do not: a new network adds **one** toggle and nothing else changes —
302which is the same reason `audience` is a list of URIs rather than a field per protocol (§5.1). The
303combinatorial explosion was hiding in the vocabulary, not in the mechanism.
304
305It also names the thing correctly for the UI. The question a composer should ask is not "what scope
306is this post" but "who is this for", and the answer is always "me and my peers, plus whoever else I
307choose".
308
309No new lexicons beyond `net.favela.post`, and no ap-bridge URI change: each toggle is just whether a
310projection is produced.
311
312**`net.favela.local.post` becomes unnecessary** — local is the substrate, not a lexicon. Its existing
313records are legacy (§8).
314
315⚠ **Neither single-network combination is reachable today**, and for different reasons, which is the
316concrete gap this design closes:
317
318- **atproto only** — ap-bridge reads `app.bsky.feed.post` unconditionally. There is no per-record
319 signal for it to skip one.
320- **fediverse only** — the record must live somewhere bsky will not index, and ap-bridge must read
321 *that*. The only such collection today is `net.favela.local.post`, and teaching ap-bridge to
322 project it would make local-only posts federate — destroying the single guarantee that scope has.
323
324Both therefore require the record to carry the intent. That is `audience`, and it is the reason this
325document exists.
326
327---
328
329## 6. Namespacing
330
331**AS2 side: already solved and shipped.** `@context` declares a prefix; prefixed terms carry the
332extension. Live today:
333
334```json
335"@context": ["https://www.w3.org/ns/activitystreams", { "atproto": "https://atproto.com/ns#" }],
336"atproto:uri": "at://…", "atproto:cid": "bafy…"
337```
338
339**atproto side:** no `@context` equivalent. Protocol-specific extras live under a single dotted
340container key mirroring NSID authority:
341
342```json
343"net.favela.ext": { "ap": { "summary": "cw: spoilers", "sensitive": true },
344 "bsky": { "labels": { … } } }
345```
346
347⛔ **Never `$`-prefixed** — `$type`, `$link`, `$bytes` are atproto's reserved space.
348
349**The constraint is narrower than it first appears.** Namespacing only matters when attaching extras
350to *someone else's* lexicon. On `net.favela.post` we own the namespace outright, so bare `audience`
351is correct and cannot collide.
352
353⛔ **Never hang favela data on `app.bsky.feed.post`.** The projection is bsky's view; anything bsky
354cannot express is simply absent from it. This removes the collision question rather than managing
355it, and it is why §2's "preservation is not guaranteed" finding does not endanger the design.
356
357---
358
359## 7. The double-write — resolved, not deferred
360
361**bsky can only see records that exist in the repo. So the `app.bsky.feed.post` projection must be
362materialised and signed. There will be two records per public post.** This is a real cost and the
363design does not pretend otherwise.
364
365**Why it is acceptable here:**
366
367- **Storage is negligible and measured.** Records average ~132 bytes; all 148 total ~32 KB.
368 Doubling is ~32 KB against a 400 MB substrate. Storage was never the concern.
369- **Coherence is the real cost**, and §3.1's shared rkey makes the pairing mechanical rather than a
370 tracked relationship: one rkey, two collections, cascade on delete and edit.
371- **It is confined to public posts.** Fediverse-only and local-only posts write one record.
372
373**What the implementation MUST guarantee:**
374
3751. Delete of a canonical record deletes its projection in the same batch. A tombstone for one and
376 not the other is a divergence with no recovery path.
3772. Edit re-derives the projection from the canonical. The projection is never edited directly.
3783. The projection is **derived output**. Nothing reads it as a source of truth. If canonical and
379 projection disagree, the canonical wins and the projection is rebuilt.
3804. `sign()` materialises both before signing, so a commit never contains one without the other.
381
382---
383
384## 8. Migration — forward-only, nothing rewritten
385
386There are **148** existing `app.bsky.feed.post` records and an unknown number of
387`net.favela.local.post` records.
388
389**They are not converted.** ⛔ **Never re-encode a record you did not author.** This project has
390already destroyed records exactly this way — a second writer re-encoded two of wj62's posts, dropped
391`langs`, and the bytes stopped hashing to their converged pointers. They were unrecoverable. That
392hazard is *higher* under this design, because every extra field is an unknown field to somebody, and
393the atproto spec warns about precisely this: *"when deserializing and re-serializing data, unknown
394fields risk being stripped away."*
395
396**Legacy rule:** an `app.bsky.feed.post` with **no** canonical counterpart at the same rkey is a
397legacy post. It is read as `audience: [Public]`, `source.mediaType: text/plain`, no extensions. It
398renders, federates and threads exactly as it does today. Nothing rewrites it.
399
400**Two canonical vocabularies must not coexist indefinitely.** New writes go to `net.favela.post`
401from the cutover. Legacy records age out by attrition and are never converted. The end state is one
402vocabulary for everything authored after the cutover, and a bounded, shrinking legacy set —
403distinguishable by a single lookup.
404
405---
406
407## 9. Read path
408
409This section exists because the write path is easy to design and the read path is where the work is.
410It is **the largest unestimated part of this design** and is deliberately called out rather than
411buried.
412
413Affected in favela-web:
414
415- `lib/scopes.js` — `SCOPE_COLLECTION` stops being a scope→collection map; audience replaces it.
416- `lib/appview.js` — ingest folds `net.favela.post` **and** legacy collections, deduplicating by
417 rkey with canonical preferred. `INGEST_VERSION` bumps; a full re-index is required.
418- `lib/render.js` — renders `source.content` as markdown when present, falling back to
419 `text` + `facets` for legacy.
420- Threading, search, notifications, RSS/Atom, bookmarks — every path that assumes a post's
421 collection is `app.bsky.feed.post`.
422- `lib/writer.js` — authors canonical + projection.
423
424Affected elsewhere: hpp's sign/materialize path (§7.4), ap-bridge's `POST_COLLECTION` and its
425projection rule (audience-driven rather than collection-driven), atproto-store's `POST_COLLECTIONS`.
426
427---
428
429## 10. Interop loss
430
431Today any atproto client can read the repo directly and see posts. After the cutover, a client that
432does not know `net.favela.post` sees **only the `app.bsky.feed.post` projections** — so public posts
433remain fully readable by any atproto tool, and non-public posts do not.
434
435That is the correct trade and it is the point: a post addressed away from bsky should not be legible
436as a bsky post. But it must be stated plainly, because it is a genuine reduction in what third-party
437tooling can see.
438
439---
440
441## 10.5 Editing
442
443**Editing is already possible end to end and is blocked only by the absence of a UI.** Researched and
444verified 2026-08-05.
445
446### 10.5.1 The protocol allows it; bsky's application refuses it
447
448`putRecord` with the same rkey replaces the record: **same URI, new CID**. The PDS is
449lexicon-agnostic and accepts it. But bsky's AppView **deliberately ignores post updates** — in their
450own words, *"The Bluesky service prohibits updating posts, so appview intentionally ignores it… the
451PDS is expected to be lexicon agnostic, so putRecord succeeds, but appview, which is in charge of
452application logic, ignores it"* ([discussion #3038](https://github.com/bluesky-social/atproto/discussions/3038)).
453
454The tell is an inconsistency, not an error: **`getRecord` and `getPosts` reflect the edit;
455`getPostThread` does not.** It is described there as *"a specification unique to bsky post"* — an
456application policy, not a protocol limit.
457
458This is the SAME shape as the audience problem (§5), arriving independently: bsky's application
459constraints became ours because its lexicon is our storage format.
460
461| surface | edit visible? | why |
462|---|---|---|
463| **P2P (canonical)** | ✅ | we own the appview |
464| **ActivityPub** | ✅ | `Update{Note}` — see §10.5.3 |
465| **bsky** | ⚠ partially | the record updates; the timeline does not |
466
467### 10.5.2 We do NOT need atproto's edit-history workaround
468
469The community pattern is copy-on-write: each edit copies the old record to a new one carrying
470`$orig`, while the live record accumulates a `$hist` array of strongRefs. rkey stays constant so
471likes and reposts stay attached. Its author names race conditions, absent transactions, and dead
472references in the chain as costs.
473
474**That pattern exists because atproto has no history. This substrate has two:**
475
4761. **The firehose already carries the chain.** `atproto-store/lib/repo-manager.js:99` sets
477 `op.prev = priorCid` on every update, *"so a relay can invert without holding the old tree"*. The
478 previous CID is on the wire, per edit, today.
4792. **The overlay and the autobase hold prior versions natively.**
480
481So `$hist`/`$orig` would reimplement — worse, and inside the record — something the substrate gives
482for free. ⛔ **Do not adopt it.** Edit history is a READ over existing state, not a new field.
483
484### 10.5.3 ⛔ Every edit MUST re-project to ActivityPub with `updated`
485
486An `Update` whose object lacks an `updated` timestamp is **silently discarded** by Mastodon: it
487accepts the POST, returns 2xx, and does nothing — no edit, no attachment re-fetch, `edited_at` stays
488null. Required since Mastodon 3.5.0; unofficial, so reading the AS2 spec will never reveal it.
489Verified live 2026-08-05 (ap-bridge 0.0.3 → 0.0.4).
490
491This costs nothing here: an edit changes the projection, so the fingerprint changes and `Update`
492fires on its own. It is recorded because getting it wrong is invisible from every local surface.
493
494### 10.5.4 The honesty requirement
495
496**The UI must not imply an edit reached bsky.** And bsky's failure mode is *worse than a clean
497refusal*: because `getRecord` reflects the edit while `getPostThread` does not, the same post will
498appear edited in some bsky contexts and stale in others. A user cannot be told "it did not sync"
499because partly it did.
500
501Same rule as §5.3: state what is true per surface, never a single "edited" that means three
502different things.
503
504### 10.5.5 What is actually missing
505
506The write path is **complete**:
507
508- `atproto-store` — `action: 'update'` handled; `prev` recorded (`repo-manager.js:67,99`)
509- `favela-web/lib/writer.js:173` — `putRecord(collection, rkey, record, { action = 'update' })` exists
510- `hyper-proto-peer` — `applyWrite` handles `update`
511
512`lib/render.js` has **zero** mentions of editing a post, and FEATURES.md lists delete and
513edit-profile as done with no post edit. So this is a **UI feature, not a protocol feature**: an edit
514affordance on one's own post, a composer prefilled from the record, and an "edited" marker whose
515history is a read over `prev` rather than a stored field.
516
517Under §5.4's toggle model an edit is unremarkable: the canonical record is replaced, the AP
518projection re-emits as `Update`, and the bsky projection is re-materialised — where it will be
519ignored, honestly and visibly.
520
521---
522
523## 11. Open questions
524
525Honest gaps. This design is not complete until these are closed.
526(Editing was one of these; it is now settled in §10.5.)
527
5281. ~~**The strip specification (§4.2)** is described but not written.~~ **CLOSED** —
529 `spec/markdown-strip.md` (220 lines) and `spec/vectors/markdown-strip.json` (36 vectors) were
530 written afterwards. `test/strip-vectors.test.mjs` validates the vectors themselves; no stripper
531 exists yet.
532
5331b. **What signs a markdown record.** With the canonical record no longer a DAG-CBOR entry in a
534 signed atproto commit chain (§0), authorship and integrity need their own mechanism — a detached
535 per-record signature, or a native append-only log of `{rkey, contentHash, sig}` that peers
536 replicate. Mutability is not an obstacle: the signature covers the current bytes and is
537 re-issued on edit, exactly as atproto re-signs a commit on edit. **This is the next thing to
538 settle; everything in §9 (read path) hangs off it.**
5392. **Embed/attachment mapping** across three representations (atproto blob refs, AS2 `attachment`,
540 markdown image syntax) is not worked through.
5413. ~~**The extension dialect** (§4.1) names candidates but does not pin a set.~~ **CLOSED** —
542 pinned in `markdown-strip.md` §2.1 (CommonMark 0.31.2 + six named extensions).
5434. **Community/list audience URIs** — the URI form for a favela community room is unspecified.
5445. **`audience` semantics for replies** — does a reply inherit its parent's audience? Almost
545 certainly yes, but the failure mode of getting it wrong is publishing to the wrong people.
5466. **Whether the AP projection should carry `source.mediaType: text/markdown`** and let Mastodon
547 render it, or continue sending pre-rendered HTML. Live output currently sends both.
548
549---
550
551## 12. What this design does *not* claim
552
553- It does not make `audience` enforceable on atproto. Nothing can.
554- It does not remove the double-write. It bounds it.
555- It does not improve deliverability, availability or performance. It is a vocabulary decision.
556- Its forcing function is thin today: one fediverse follower, one enforcement point. It is
557 justified by protocol churn and expressiveness, not by present demand. That should be weighed
558 before implementation, not after.
spec/vectors/markdown-strip.json +334 -0
……@@ -0,0 +1,334 @@
1{
2 "$comment": "Vectors for spec/markdown-strip.md. Offsets are UTF-8 BYTE offsets into `text`, half-open [byteStart, byteEnd). An implementation is conformant when it reproduces every entry byte-for-byte.",
3 "spec": "spec/markdown-strip.md",
4 "vectors": [
5 {
6 "name": "emphasis-stripped",
7 "why": "§3.1 markers go, text stays, no facet — atproto has no bold",
8 "source": "hello **world** and *more*",
9 "text": "hello world and more",
10 "facets": []
11 },
12 {
13 "name": "code-span",
14 "why": "§3.1 a code span is text, not markup",
15 "source": "use `npm test` now",
16 "text": "use npm test now",
17 "facets": []
18 },
19 {
20 "name": "link",
21 "why": "§3.2 label becomes text; the uri rides in the facet",
22 "source": "click [here](https://example.com)",
23 "text": "click here",
24 "facets": [
25 {
26 "byteStart": 6,
27 "byteEnd": 10,
28 "type": "link",
29 "value": "https://example.com"
30 }
31 ]
32 },
33 {
34 "name": "link-with-emphasis-label",
35 "why": "§3.2 the label is stripped FIRST, then the facet spans the result",
36 "source": "click [**bold**](https://example.com)",
37 "text": "click bold",
38 "facets": [
39 {
40 "byteStart": 6,
41 "byteEnd": 10,
42 "type": "link",
43 "value": "https://example.com"
44 }
45 ]
46 },
47 {
48 "name": "link-empty-label",
49 "why": "§3.2 an empty label emits the url as text",
50 "source": "[](https://example.com)",
51 "text": "https://example.com",
52 "facets": [
53 {
54 "byteStart": 0,
55 "byteEnd": 19,
56 "type": "link",
57 "value": "https://example.com"
58 }
59 ]
60 },
61 {
62 "name": "link-non-http-refused",
63 "why": "§3.2 ⛔ javascript: emits the label as PLAIN TEXT, no facet",
64 "source": "[click](javascript:alert(1))",
65 "text": "click",
66 "facets": []
67 },
68 {
69 "name": "bare-url-trailing-punctuation",
70 "why": "§3.3 the normative URL_RE stops before trailing punctuation — the full stop stays text",
71 "source": "see https://x.com.",
72 "text": "see https://x.com.",
73 "facets": [
74 {
75 "byteStart": 4,
76 "byteEnd": 17,
77 "type": "link",
78 "value": "https://x.com"
79 }
80 ]
81 },
82 {
83 "name": "mention-resolved",
84 "why": "§3.4 a mention MUST carry a resolved did in the target",
85 "source": "hi [@alice](did:plc:xyz)",
86 "text": "hi @alice",
87 "facets": [
88 {
89 "byteStart": 3,
90 "byteEnd": 9,
91 "type": "mention",
92 "value": "did:plc:xyz"
93 }
94 ]
95 },
96 {
97 "name": "mention-bare-is-text",
98 "why": "§3.4 ⛔ a bare @alice is plain text — never a guessed did",
99 "source": "hi @alice",
100 "text": "hi @alice",
101 "facets": []
102 },
103 {
104 "name": "image-produces-nothing",
105 "why": "§3.5 images are embed, not body text",
106 "source": "![a cat](https://x/c.png)",
107 "text": "",
108 "facets": []
109 },
110 {
111 "name": "image-only-strips-empty",
112 "why": "§3.5 a post that is only an image has empty text — valid",
113 "source": "![](https://x/c.png)",
114 "text": "",
115 "facets": []
116 },
117 {
118 "name": "wikilink",
119 "why": "§3.6 degrades to its label, no facet — no atproto/AS2 equivalent",
120 "source": "see [[Design Note]]",
121 "text": "see Design Note",
122 "facets": []
123 },
124 {
125 "name": "wikilink-labelled",
126 "why": "§3.6 the pipe form uses the label",
127 "source": "see [[Design Note|the note]]",
128 "text": "see the note",
129 "facets": []
130 },
131 {
132 "name": "hashtag",
133 "why": "§3.7 the # stays in TEXT, the facet value drops it",
134 "source": "a #p2p post",
135 "text": "a #p2p post",
136 "facets": [
137 {
138 "byteStart": 2,
139 "byteEnd": 6,
140 "type": "tag",
141 "value": "p2p"
142 }
143 ]
144 },
145 {
146 "name": "hashtag-midword-is-text",
147 "why": "§3.7 a # inside a word is literal",
148 "source": "c#sharp",
149 "text": "c#sharp",
150 "facets": []
151 },
152 {
153 "name": "strikethrough",
154 "why": "§3.1 extension: markers go",
155 "source": "~~gone~~ here",
156 "text": "gone here",
157 "facets": []
158 },
159 {
160 "name": "footnote-dropped",
161 "why": "§3.9 refs and definitions have no projection target",
162 "source": "claim[^1]\n\n[^1]: the source",
163 "text": "claim",
164 "facets": []
165 },
166 {
167 "name": "hard-break",
168 "why": "§3.8 backslash break becomes one newline",
169 "source": "one\\\ntwo",
170 "text": "one\ntwo",
171 "facets": []
172 },
173 {
174 "name": "soft-break-is-space",
175 "why": "§3.8 a soft wrap is a space, not a newline",
176 "source": "one\ntwo",
177 "text": "one two",
178 "facets": []
179 },
180 {
181 "name": "multibyte-before-facet",
182 "why": "★ é is 2 bytes — a character-offset implementation puts this facet one byte early",
183 "source": "café https://x.com",
184 "text": "café https://x.com",
185 "facets": [
186 {
187 "byteStart": 6,
188 "byteEnd": 19,
189 "type": "link",
190 "value": "https://x.com"
191 }
192 ]
193 },
194 {
195 "name": "emoji-before-facet",
196 "why": "★ each emoji is 4 bytes — the classic off-by-six for a char-offset implementation",
197 "source": "🎉🎉 tap [here](https://example.com)",
198 "text": "🎉🎉 tap here",
199 "facets": [
200 {
201 "byteStart": 13,
202 "byteEnd": 17,
203 "type": "link",
204 "value": "https://example.com"
205 }
206 ]
207 },
208 {
209 "name": "multibyte-inside-mention",
210 "why": "★ the facet itself spans multi-byte characters",
211 "source": "héllo [@wörld](did:plc:abc)",
212 "text": "héllo @wörld",
213 "facets": [
214 {
215 "byteStart": 7,
216 "byteEnd": 14,
217 "type": "mention",
218 "value": "did:plc:abc"
219 }
220 ]
221 },
222 {
223 "name": "heading",
224 "why": "§4.2 marker removed, block like any other",
225 "source": "## Title\n\nbody",
226 "text": "Title\n\nbody",
227 "facets": []
228 },
229 {
230 "name": "code-fence-verbatim",
231 "why": "§4.3 contents verbatim, no fence, no re-indent",
232 "source": "```js\nconst a = 1\n```",
233 "text": "const a = 1",
234 "facets": []
235 },
236 {
237 "name": "blockquote-no-prefix",
238 "why": "§4.4 no '> ' is invented — that would be a convention to re-specify",
239 "source": "> quoted\n> more",
240 "text": "quoted more",
241 "facets": []
242 },
243 {
244 "name": "bullet-list",
245 "why": "§4.5 a marker IS kept, or a list becomes indistinguishable from paragraphs",
246 "source": "- one\n- two",
247 "text": "• one\n• two",
248 "facets": []
249 },
250 {
251 "name": "ordered-list-renumbered",
252 "why": "§4.5 the RENDERED sequence, not the literal source numbers",
253 "source": "1. a\n1. b\n1. c",
254 "text": "1. a\n2. b\n3. c",
255 "facets": []
256 },
257 {
258 "name": "table",
259 "why": "§4.6 rows on lines, cells joined with ' | '; lossy and deliberately unclever",
260 "source": "| a | b |\n|---|---|\n| 1 | 2 |",
261 "text": "a | b\n1 | 2",
262 "facets": []
263 },
264 {
265 "name": "thematic-break-omitted",
266 "why": "§4.7 omitted entirely, including its separator",
267 "source": "one\n\n---\n\ntwo",
268 "text": "one\n\ntwo",
269 "facets": []
270 },
271 {
272 "name": "directive-contents-kept",
273 "why": "§4.8 a spoiler projects as its inner text — dropping it would misrepresent the post elsewhere",
274 "source": ":::spoiler\nthe ending\n:::",
275 "text": "the ending",
276 "facets": []
277 },
278 {
279 "name": "crlf-normalised",
280 "why": "§5.1 CRLF becomes LF before parsing",
281 "source": "one\r\n\r\ntwo",
282 "text": "one\n\ntwo",
283 "facets": []
284 },
285 {
286 "name": "blank-lines-collapsed",
287 "why": "§5.3 three or more newlines collapse to two",
288 "source": "one\n\n\n\ntwo",
289 "text": "one\n\ntwo",
290 "facets": []
291 },
292 {
293 "name": "outer-whitespace-trimmed",
294 "why": "§5.4 leading and trailing whitespace is removed from the whole text",
295 "source": "\n\n hello \n\n",
296 "text": "hello",
297 "facets": []
298 },
299 {
300 "name": "empty-source",
301 "why": "§2 total function — empty in, empty out, no throw",
302 "source": "",
303 "text": "",
304 "facets": []
305 },
306 {
307 "name": "whitespace-only-source",
308 "why": "§2 total function — a whitespace-only source degrades to empty, never throws",
309 "source": " \n\n ",
310 "text": "",
311 "facets": []
312 },
313 {
314 "name": "facets-sorted-by-bytestart",
315 "why": "§6.1 sorted ascending — required for byte-determinism",
316 "source": "[a](https://one.example) and [b](https://two.example)",
317 "text": "a and b",
318 "facets": [
319 {
320 "byteStart": 0,
321 "byteEnd": 1,
322 "type": "link",
323 "value": "https://one.example"
324 },
325 {
326 "byteStart": 6,
327 "byteEnd": 7,
328 "type": "link",
329 "value": "https://two.example"
330 }
331 ]
332 }
333 ]
334}