71f2724 · favela-web · pear-git-v3

Changed files (4) +21 -1112

spec/MOVED-record-specs.md +21 -0

…	…	@@ -0,0 +1,21 @@
	1	# The record specs moved to `favela-markdown`
	2
	3	On 2026-09-02 three files left this directory:
	4
	5	\| was \| now \|
	6	\|---\|---\|
	7	\| `spec/record-vocabulary.md` \| `favela-markdown/spec/record-vocabulary.md` \|
	8	\| `spec/markdown-strip.md` \| `favela-markdown/spec/markdown-strip.md` \|
	9	\| `spec/vectors/markdown-strip.json` \| `favela-markdown/spec/vectors/markdown-strip.json` \|
	10
	11	Why. They specify the canonical record, not this web app. favela-web is one of three
	12	consumers that project from them — the others are the atproto signer
	13	(`atproto-room/lib/signer.js` `materialize()`) and `ap-bridge/lib/as2-project.js`. A
	14	contract owned by one of its consumers is a contract the other two cannot depend on, and
	15	these specs stayed invisible here for a month (hyper-social #56 §5.1).
	16
	17	The library that owns them is `hyper-fleeky/favela-markdown`; start at its `spec/design.md`.
	18
	19	Nothing was rewritten in the move — only the two `favela-web/spec/…` path references
	20	inside `favela-markdown`'s own README and design note. The §0 work from 2026-08-31 was
	21	committed here first (`a3d469b`) so this repo's history keeps it.

spec/markdown-strip.md +0 -220

…	…	@@ -1,220 +0,0 @@
1		# The strip — `source.content` (markdown) → `text` + `facets`
2
3		Status: specification. Not implemented.
4		Date: 2026-08-05
5		Closes: `record-vocabulary.md` §4.2 and §11 item 1.
6		Vectors: `spec/vectors/markdown-strip.json` — machine-readable, wire them into a test.
7
8		---
9
10		## 1. Why this must be a specification
11
12		The strip's output is what bsky sees and what a CID commits to. If two implementations
13		produce different `text` from the same `source.content`, they produce different
14		`app.bsky.feed.post` records from one canonical record, and `atproto:cid` — the cross-network dedup
15		key the whole projection model rests on — stops identifying content.
16
17		So "whatever the parser does" is not an answer. Every rule below is normative. Anything a rule does
18		not cover is a defect in this document, not an implementation choice.
19
20		---
21
22		## 2. Contract
23
24		```
25		strip(source: string) -> { text: string, facets: Facet[] }
26		```
27
28		- `text` is UTF-8 plain text. No markup survives.
29		- `facets` are `app.bsky.richtext.facet` objects with `index: { byteStart, byteEnd }` —
30		byte offsets into `text`, UTF-8, half-open `[byteStart, byteEnd)`.
31		- Deterministic: same input bytes → same output bytes, on every implementation, forever.
32		- Total: never throws. Malformed markdown degrades to its literal text.
33
34		### 2.1 Dialect — pinned
35
36		CommonMark 0.31.2, plus exactly these extensions and no others:
37
38		\| extension \| syntax \|
39		\|---\|---\|
40		\| strikethrough \| `~~x~~` \|
41		\| tables \| GFM pipe tables \|
42		\| footnotes \| `[^1]` and its definition \|
43		\| wiki-links \| `[[Page]]`, `[[Page\\|label]]` \|
44		\| directives \| `:::name` … `:::` (block only) \|
45		\| hashtags \| `#tag` (see §3.7) \|
46
47		An implementation MUST NOT enable further extensions. A new one is a spec change, because it
48		changes the bytes of every record that uses it.
49
50		---
51
52		## 3. Inline rules
53
54		Applied to the inline content of each block after §4 produces block text.
55
56		### 3.1 Emphasis, strong, strikethrough, code spans
57		Markers removed, inner text kept verbatim. `a` → `a`. `` `x` `` → `x`. `~~a~~` → `a`.
58		Nesting resolves innermost-first; the result is the concatenated text content.
59		No facet is emitted — atproto has no bold.
60
61		### 3.2 Links
62		`[label](url)` → `label`, plus one `#link` facet with `uri = url` spanning `label`'s bytes.
63
64		- The label is stripped by these same inline rules first, so `[a](u)` → `a` with the facet over
65		`a`.
66		- An empty label (`[](u)`) emits the URL as text and a facet over it.
67		- ⛔ `uri` MUST be `http:` or `https:` — anything else (including `javascript:`) emits the label
68		as plain text with no facet. This mirrors the existing render rule that refuses non-http URIs.
69		- The `did:` scheme is the one exception, and only for mentions — §3.4.
70
71		### 3.3 Autolinks and bare URLs
72		`<https://x>` → `https://x` with a `#link` facet. A bare URL in text likewise.
73
74		Bare-URL detection is exactly favela-web's shipped `URL_RE`:
75		```
76		/\bhttps?:\/\/[^\s<]+[^\s<.,:;!?)\]}'"]/g
77		```
78		so trailing punctuation stays outside the facet ("see https://x.com." keeps the full stop as text).
79		This regex is normative; changing it changes records.
80
81		### 3.4 Mentions
82		`[@alice](did:plc:xyz)` → `@alice`, plus a `#mention` facet with `did = did:plc:xyz`.
83
84		⛔ A mention MUST carry a resolved DID in the link target. A bare `@alice` with no target is
85		plain text with no facet — never a guess. This preserves the existing rule (favela-web 0.3.11):
86		a mention resolves to a DID or is omitted.
87
88		### 3.5 Images
89		`![alt](url)` produces no text and no facet. Images are `embed`, not body text.
90
91		The embed pipeline consumes them separately, in source order. An image is therefore invisible to the
92		strip — a post whose entire source is one image strips to empty `text`, which is valid.
93
94		### 3.6 Wiki-links
95		`[[Page]]` → `Page`. `[[Page\|label]]` → `label`. No facet — a wiki-link is favela-internal and
96		has no atproto or AS2 equivalent, so it degrades to its label rather than inventing a URL.
97
98		### 3.7 Hashtags
99		`#tag` → `#tag` (the hash is kept in the text), plus a `#tag` facet whose `tag` value is the
100		label without the `#`.
101
102		Matched as `#` preceded by start-of-text or whitespace, followed by 1–64 characters from
103		`[A-Za-z0-9_]` plus any non-ASCII letter. A `#` inside a word (`a#b`) or followed by a digit only
104		(`#1`) is literal text. `#` at the start of a line is a heading (§4.2), not a tag.
105
106		### 3.8 Hard breaks
107		A backslash at end of line, or two-or-more trailing spaces, → a single `\n`. Soft line breaks inside
108		a paragraph → a single space.
109
110		### 3.9 Footnotes
111		A reference `[^1]` is removed entirely (no text, no facet). Its definition block is dropped.
112		Footnotes are a favela-render feature with no projection target.
113
114		### 3.10 HTML
115		Raw inline and block HTML is emitted as its literal source text, unparsed, with no facet.
116		We never execute or interpret it; escaping is the renderer's job on the way out.
117
118		---
119
120		## 4. Block rules
121
122		### 4.1 Joining
123		Block texts are joined with exactly `\n\n`. A block producing empty text is omitted entirely,
124		including its separator.
125
126		### 4.2 Headings
127		ATX (`## T`) and setext both → `T`. The marker, and any closing `##`, are removed. A heading is a
128		block like any other — no extra spacing, no case change.
129
130		### 4.3 Code blocks
131		Fenced and indented → the code contents verbatim, including internal newlines, without the
132		fence or the info string, and without re-indenting. Trailing newline inside the block is trimmed.
133
134		### 4.4 Blockquotes
135		`> x` → `x`. The marker is removed; nested quotes flatten. No prefix is added — a `> ` in plain
136		text would be a rendering convention we would then have to specify for every consumer.
137
138		### 4.5 Lists
139		Each item's content becomes one line, joined with `\n` within the list block; the list as a
140		whole is one block for §4.1 purposes.
141
142		- Bullet items are prefixed `• ` (U+2022, space).
143		- Ordered items are prefixed with their number and `. ` — using the rendered sequence
144		(1., 2., 3.), not the literal source numbers, so `1. 1. 1.` strips as `1. 2. 3.`.
145		- Nested lists indent by two spaces per level.
146
147		A marker is kept here, unlike blockquotes, because without one a list becomes indistinguishable
148		from consecutive paragraphs and loses its meaning entirely.
149
150		### 4.6 Tables
151		Rows joined with `\n`; cells joined with ` \| `. The delimiter row (`---\|---`) is dropped. Leading
152		and trailing pipes are dropped. Cell content is stripped by the inline rules.
153
154		This is lossy and deliberately unclever: a table has no faithful plain-text form, and any attempt to
155		align columns would make the output depend on font and width.
156
157		### 4.7 Thematic breaks
158		`---` → the block is omitted (no text, no separator).
159
160		### 4.8 Directives
161		`:::name` … `:::` → the contents, stripped by these rules; the fences and the name are removed.
162		A `:::spoiler` therefore projects as its inner text. Concealment is a favela-render feature, and
163		silently dropping the content would misrepresent the post everywhere else.
164
165		---
166
167		## 5. Normalisation
168
169		Applied to `text` after assembly, before facet offsets are computed.
170
171		1. CRLF and CR → LF. Applied to the input before parsing.
172		2. Trailing whitespace removed from every line.
173		3. Three or more consecutive newlines collapse to two.
174		4. Leading and trailing whitespace removed from the whole text.
175		5. ⛔ NO Unicode normalisation. Do not apply NFC/NFD. Normalising would change bytes the author
176		wrote, and therefore the CID, for reasons invisible to them.
177		6. ⛔ NO case folding, NO smart quotes, NO typographic substitution.
178
179		---
180
181		## 6. Facets
182
183		1. Sorted by `byteStart` ascending. Required for byte-determinism of the encoded record.
184		2. No overlaps. If two features would cover overlapping ranges, the one starting earlier wins;
185		on an equal start, the longer wins; on both equal, precedence is
186		mention > link > tag.
187		3. A facet whose range is empty after stripping is dropped.
188		4. Offsets are recomputed against the final `text` from §5 — never against an intermediate.
189		5. `features` arrays contain exactly one feature. We never emit a multi-feature facet.
190
191		---
192
193		## 7. Length
194
195		The strip does not truncate. §4.3 of `record-vocabulary.md` owns the 300-grapheme bsky rule, and it
196		applies to the projection, not to `text`. `text` is the full canonical body.
197
198		---
199
200		## 8. Test vectors
201
202		`spec/vectors/markdown-strip.json`. Each entry is `{ name, why, source, text, facets }`, with
203		`facets` as `{ byteStart, byteEnd, type, value }` — a compact form to be expanded into
204		`app.bsky.richtext.facet` shape by the test harness.
205
206		An implementation is conformant when it reproduces every vector byte-for-byte. The multi-byte
207		cases are not optional extras: they are the ones that catch an implementation using character
208		offsets instead of UTF-8 byte offsets, which is the single most likely defect here and produces
209		facets that land on the wrong characters only for non-ASCII authors.
210
211		---
212
213		## 9. Open
214
215		1. Grapheme counting for the bsky projection is specified in `record-vocabulary.md` §4.3 but the
216		segmentation algorithm is not pinned. It must be, for the same determinism reason.
217		2. Embed extraction order (§3.5) is asserted to be source order but not otherwise specified;
218		it belongs in an embed spec, which does not exist yet.
219		3. Directive names are not enumerated. Only the fence handling is specified; which names favela
220		renders specially is a render concern, but the set should be pinned before use.

spec/record-vocabulary.md +0 -558

…	…	@@ -1,558 +0,0 @@
1		# The record vocabulary — `net.favela.post`
2
3		Status: design, not implemented. Nothing here has been built.
4		Revised 2026-08-31 (§0, §4.1) — the canonical record becomes a markdown file, not a
5		DAG-CBOR record. atproto stops being the canonical P2P path and becomes a projection like any other.
6		Date: 2026-08-05
7		Scope: hyper-proto-peer (authoring), atproto-store (storage), atproto-bridge + ap-bridge
8		(projection), favela-web (compose + render).
9
10		---
11
12		## 0. The governing decision (2026-08-31)
13
14		atproto was never meant to be the canonical P2P path. It became one by default. Building the
15		peer layer on atproto's data model imported atproto's constraints — MST derivation, DAG-CBOR
16		byte-exactness, lexicon field ceilings, blob CIDs sealed inside signed records — into a network
17		that only ever wanted to interface with atproto. That machinery is genuinely useful, but it
18		belongs at the bridge/relay layer, not underneath every full-mode peer.
19
20		**So: the canonical record is a markdown document. Every network — atproto and ActivityPub alike —
21		is a lossy projection derived from it on read.**
22
23		```
24		records/net.favela.post/3msdnle65lk22.md canonical. what the author typed.
25		records/app.bsky.feed.post/3msdnle65lk22.dagcbor derived projection (bridge builds the MST from it)
26		https://ap.7rnx.net/ap/objects/3msdnle65lk22 derived projection
27		```
28
29		This is a smaller change than it sounds, because `atproto-store/lib/overlay-repo-store.js` already
30		stores one file per record and derives MST/commit/CAR on read — "nothing atproto-specific is
31		stored beyond the record bytes + a tiny head.json". Only the encoding of the record file changes.
32
33		What it buys. A full-mode peer needs: parse markdown, one file per record, hyper-overlay
34		replication. No MST, no DAG-CBOR, no byte-exact CIDs. Those become the bridge's problem, where
35		they belong. This is what makes a second implementation tractable.
36
37		### 0.1 The two governing properties
38
39		Everything else is negotiable; these are not.
40
41		- Mutable. A post is a file that can be rewritten. `OverlayRepoStore` is already
42		overwrite-in-place on edit and unlink on delete, so on-disk size is O(current records), not
43		O(operations-ever).
44		- Extensible. New extensions can be added at any time without migrating old records, *because
45		projections are derived on read*. Old posts stay valid; new projections simply carry more. This is
46		the opposite of a lexicon change, which is a migration.
47
48		Consequence: mutable content means the content hash changes on every edit, so **the rkey is the
49		stable identity, never the hash.** §3.1 already depends on this, and it is why the 22 already-
50		federated AP object URIs keep resolving.
51
52		---
53
54		## 1. The decision
55
56		**The canonical stored record moves from `app.bsky.feed.post` to `net.favela.post`. bsky and
57		ActivityPub become projections of it. The P2P side reads the canonical record directly and is the
58		only place the whole post exists.**
59
60		### Why — the vocabulary was never chosen
61
62		Today the file on disk is a bsky post:
63
64		```
65		records/app.bsky.feed.post/3msdnle65lk22.dagcbor
66		{ "$type": "app.bsky.feed.post", "text": "…", "createdAt": "…", "langs": ["en"] }
67		```
68
69		That is bsky's semantic model. It has the fields bsky decided a post has — and no addressing,
70		because on bsky everything is public.
71
72		The existing favela lexicon confirms this was inherited rather than designed. `net.favela.local.post`
73		is literally the bsky shape minus `langs`:
74
75		```json
76		{ "$type": "net.favela.local.post", "text": "bork", "createdAt": "2026-07-17T18:11:00.877Z" }
77		```
78
79		So one application's model became the ceiling for every projection. ActivityPub can express
80		addressing natively — `to`/`cc` are already being emitted — but the stored record has nowhere to
81		say what the addressing should be, so it is hardcoded to Public.
82
83		This document does not fix a bug. It makes a decision that was previously a default.
84
85		---
86
87		## 2. What is verified vs assumed
88
89		Everything in §2 was checked during design. Nothing else in this document should be treated as
90		verified.
91
92		\| claim \| status \|
93		\|---\|---\|
94		\| `net.favela.local.post` is the bsky shape minus `langs` \| ✅ decoded from disk \|
95		\| 148 existing `app.bsky.feed.post` records, zero with extra fields \| ✅ scanned \|
96		\| Lexicon validation is open — unknown fields "should be ignored" \| ✅ [atproto spec](https://atproto.com/specs/lexicon) \|
97		\| …but preservation is not guaranteed — other PDSs "might strip" \| ✅ [discussion #2126](https://github.com/bluesky-social/atproto/discussions/2126) \|
98		\| Custom fields are safe only if they cannot collide with future lexicon fields \| ✅ [Custom Schemas](https://docs.bsky.app/docs/advanced-guides/custom-schemas) \|
99		\| ap-bridge already emits JSON-LD namespaced extensions \| ✅ live: `@context` + `atproto:uri`, `atproto:cid` \|
100		\| ap-bridge already emits `to` / `cc` / `contentMap` \| ✅ live, hardcoded to Public \|
101		\| AS2 `source` `{content, mediaType}` already emitted \| ✅ live, `text/plain` \|
102		\| `objectIdFor({host, rkey})` → `/ap/objects/<rkey>`, rkey only \| ✅ read source \|
103		\| `rkeyFromObjectId` matches a single path segment \| ✅ read source \|
104		\| 22 objects already cached at prsm.space under that URI shape \| ✅ Mastodon API \|
105		\| AP federation works end to end \| ✅ 22 public statuses incl. today's \|
106
107		---
108
109		## 3. Core model
110
111		### 3.1 The rkey pairing — the mechanism everything rests on
112
113		A canonical record and its atproto projection share one rkey, differing only by collection.
114
115		```
116		canonical at://<did>/net.favela.post/3msdnle65lk22
117		projection at://<did>/app.bsky.feed.post/3msdnle65lk22
118		AP object https://ap.7rnx.net/ap/objects/3msdnle65lk22
119		```
120
121		This is not a convenience. It resolves four problems at once:
122
123		- No mapping table. Given the canonical, the projection is derivable, and vice versa.
124		- The AP object URI never changes. ap-bridge keeps deriving from the rkey, so the 22 already-
125		federated URIs keep resolving. The collection-in-path change discussed earlier is not needed.
126		- ap-bridge's pure-function id-map survives — it never fronts a second collection.
127		- Delete and edit cascade mechanically by rkey.
128
129		### 3.2 The core fields
130
131		The intersection of what atproto and AS2 both express, checked field by field against both specs:
132
133		```json
134		{
135		"$type": "net.favela.post",
136		"createdAt": "2026-08-05T13:38:23.294Z",
137		"langs": ["en"],
138
139		"source": { "content": "hello world", "mediaType": "text/markdown" },
140		"text": "hello world",
141		"facets": [ /* byte offsets into `text` */ ],
142
143		"reply": { "root": {…}, "parent": {…} },
144		"embed": { /* images / quote / external */ },
145
146		"audience": ["https://www.w3.org/ns/activitystreams#Public"]
147		}
148		```
149
150		`audience` is the only genuinely new field. Everything else is what is already written today,
151		because those parts of bsky's model are sound and reinventing them would be churn.
152
153		### 3.3 Two representation decisions, settled on recoverability
154
155		Body: plain text + byte-offset facets is canonical; markdown is the source; HTML is derived.
156
157		Markdown → HTML is deterministic. HTML → byte-offset facets is not reliably recoverable. So the
158		canonical machine-readable body is `text` + `facets`, and `source` carries what the human typed.
159		This is not "atproto won" — it is the direction that loses nothing.
160
161		Threading: `{root, parent}` is canonical, `inReplyTo` is derived. atproto's pair carries strictly
162		more than AS2's single URI, and the projection down is trivial.
163
164		### 3.4 Markdown links are facets
165
166		`[@alice](did:plc:xyz)` carries display text plus resolved target — exactly what a facet is. A
167		CommonMark parser yields exact source ranges, so facet offsets are derived from the parse, never
168		hand-maintained. This preserves the existing rule (favela-web 0.3.11) that a mention resolves to a
169		DID or is omitted, never guessed.
170
171		---
172
173		## 4. The P2P side is the rich side
174
175		The canonical record is the P2P record, and the P2P side carries everything: markdown as
176		authored, the full extension set, no length ceiling, no facet flattening. bsky and ActivityPub are
177		the edges, and edges are lossy.
178
179		This inverts the usual bridge posture, where the local client degrades to the lowest common
180		denominator of its targets — which is how `bridge-envelope`'s `content: string` would have capped
181		every projection, and why ap-bridge refused it.
182
183		Richness belongs to the representation, not to one app. favela-web is the first client of the
184		P2P side, not the definition of it. `favela-cli`, `favela-native`, another peer's implementation, or
185		anything reading the record files off a shared overlay gets the same complete record. A client that
186		renders less is choosing to; nothing in the format holds it back.
187
188		This is also what makes the format worth having. If the richest view were a single web app, the
189		extra fields would be that app's private state. Because the richest view is the record, any P2P
190		reader — including one nobody has written yet — sees the whole post.
191
192		### 4.1 Dialect — CommonMark, and extensions that carry the rest
193
194		CommonMark is the base. Extensions are markdown syntax carrying whatever plain markdown cannot.
195
196		The extension set is more expansive than any target network because it does not have to ask
197		permission: the canonical record keeps everything, and each bridge takes only what it can carry.
198
199		Unknown extensions need no special rule — markdown already degrades on its own. A reader that
200		does not know `:::spoiler` renders the text inside it. Falling through to literal text is the
201		behaviour, not something this document has to invent. That is a property of the format, and it is
202		why the extension set can grow without coordination.
203
204		**The dialect is already pinned, in `markdown-strip.md` §2.1: CommonMark 0.31.2 plus exactly
205		six extensions** — strikethrough, GFM tables, footnotes, `[[wiki-links]]`, block directives
206		(`:::name`), and hashtags. *"An implementation MUST NOT enable further extensions. A new one is a
207		spec change, because it changes the bytes of every record that uses it."* Fenced code is CommonMark
208		core, not an extension. Markdown links are mentions — `[@alice](did:plc:xyz)` carries display text
209		plus resolved target, so facets are derived from the parse when projecting (§3.4), never
210		hand-maintained.
211
212		The one exception: addressing. Getting a spoiler wrong makes a post render plainly. Getting
213		"who can see this" wrong sends it to the wrong people. So `audience` is a **typed field, never
214		parsed out of loose text** — a spelling quirk must not be able to change who receives a post.
215		Everything else may be casual.
216
217		### 4.2 The strip — must be specified, not merely implemented
218
219		`source.content` (markdown) → `text` (plain) must be byte-deterministic across implementations,
220		or two implementations produce different bsky records from one canonical record and `atproto:cid`
221		stops identifying content.
222
223		The specification MUST fix, at minimum: emphasis markers removed and inner text kept; link syntax
224		reduced to its display text with a facet at that byte range; headings reduced to their text; code
225		fences reduced to their contents; block structure joined with `\n\n`; `[[wiki-link]]` reduced to its
226		label; directives removed with contents kept. Anything unspecified is a defect in this document.
227
228		### 4.3 Length
229
230		A favela post has no length limit. `app.bsky.feed.post` is capped at 300 graphemes.
231
232		**Rule: if the stripped text exceeds 300 graphemes, the bsky projection is truncated at a grapheme
233		boundary and a permalink to the favela post is appended.** This is standard POSSE behaviour and is
234		preferred over omitting the post, because silence on bsky is indistinguishable from a delivery
235		failure — a class of confusion this project has paid for repeatedly.
236
237		The AP projection is never truncated; AS2 has no length limit.
238
239		---
240
241		## 5. Audience
242
243		### 5.1 Shape
244
245		A list of URIs. `https://www.w3.org/ns/activitystreams#Public` means everyone. A community is its
246		room key as a URI; a list is its `at://` URI.
247
248		URIs rather than protocol names (`["bsky","fediverse"]`) because a URI names who, not where.
249		Adding a protocol later adds no field — it is another audience that either includes a given
250		recipient or does not. Protocol names as flags would need a new field per protocol forever, and
251		would put knowledge of projections inside the record, which is the thing this design exists to
252		avoid.
253
254		### 5.2 What each projection does with it
255
256		\| projection \| enforcement \|
257		\|---\|---\|
258		\| P2P (favela-web and any other client) \| not a projection at all — it is the substrate (§5.4). It renders everything it holds, and `audience` never withholds anything from it. \|
259		\| ActivityPub \| native. `audience` maps onto `to`/`cc`. ap-bridge already emits these — hardcoded to Public. The change is to read the value instead of assuming it. \|
260		\| atproto / bsky \| none. atproto has no addressing. The only lever is whether the `app.bsky.feed.post` projection is written at all. \|
261
262		### 5.3 The honesty requirement — non-negotiable
263
264		The atproto asymmetry is not a limitation of this design; it is a property of atproto, and the UI
265		must not conceal it.
266
267		**Not writing a bsky projection means bsky does not INDEX the post. It does not mean the fediverse
268		or anyone else cannot read it.** The canonical record still rides the firehose and is fetchable by
269		anyone who fronts the DID. This is the same caveat DESIGN §10.2 already makes about local-only
270		("unindexed by bsky, not unreplicated"), and it MUST be surfaced wherever audience is chosen.
271
272		Composer copy must never imply erasure or privacy. Until encryption lands, the honest phrasing
273		remains "community, not secret".
274
275		### 5.4 ★ Local is the SUBSTRATE, not a scope — reach is N independent toggles
276
277		Every post is local. The canonical record always exists on the P2P side; that is what authoring
278		one means. "Local" is therefore not a destination you opt into, and there is no way to opt out of
279		it. What the author actually chooses is which projections to emit.
280
281		So reach is not a set of mutually exclusive scopes. It is one independent toggle per projection,
282		with the P2P side always on:
283
284		```
285		local (canonical) ── always, not a choice
286		├─ bsky ── toggle
287		└─ fediverse ── toggle
288		```
289
290		Which yields the same four combinations, described honestly:
291
292		\| bsky \| fediverse \| what it is \| canonical \| `app.bsky.feed.post` \| AS2 projection \|
293		\|---\|---\|---\|---\|---\|---\|
294		\| ✅ \| ✅ \| public \| ✅ \| written \| emitted \|
295		\| ✅ \| ✗ \| atproto only \| ✅ \| written \| withheld \|
296		\| ✗ \| ✅ \| fediverse only \| ✅ \| not written \| emitted \|
297		\| ✗ \| ✗ \| local only \| ✅ \| not written \| withheld \|
298
299		Why the framing matters, and why the earlier "four scopes" table was wrong. Enumerating scopes
300		implies a named scope per combination, so adding a third network would mean eight scopes and a
301		UI listing all of them. Toggles do not: a new network adds one toggle and nothing else changes —
302		which is the same reason `audience` is a list of URIs rather than a field per protocol (§5.1). The
303		combinatorial explosion was hiding in the vocabulary, not in the mechanism.
304
305		It also names the thing correctly for the UI. The question a composer should ask is not "what scope
306		is this post" but "who is this for", and the answer is always "me and my peers, plus whoever else I
307		choose".
308
309		No new lexicons beyond `net.favela.post`, and no ap-bridge URI change: each toggle is just whether a
310		projection is produced.
311
312		`net.favela.local.post` becomes unnecessary — local is the substrate, not a lexicon. Its existing
313		records are legacy (§8).
314
315		⚠ Neither single-network combination is reachable today, and for different reasons, which is the
316		concrete gap this design closes:
317
318		- atproto only — ap-bridge reads `app.bsky.feed.post` unconditionally. There is no per-record
319		signal for it to skip one.
320		- fediverse only — the record must live somewhere bsky will not index, and ap-bridge must read
321		that. The only such collection today is `net.favela.local.post`, and teaching ap-bridge to
322		project it would make local-only posts federate — destroying the single guarantee that scope has.
323
324		Both therefore require the record to carry the intent. That is `audience`, and it is the reason this
325		document exists.
326
327		---
328
329		## 6. Namespacing
330
331		AS2 side: already solved and shipped. `@context` declares a prefix; prefixed terms carry the
332		extension. Live today:
333
334		```json
335		"@context": ["https://www.w3.org/ns/activitystreams", { "atproto": "https://atproto.com/ns#" }],
336		"atproto:uri": "at://…", "atproto:cid": "bafy…"
337		```
338
339		atproto side: no `@context` equivalent. Protocol-specific extras live under a single dotted
340		container key mirroring NSID authority:
341
342		```json
343		"net.favela.ext": { "ap": { "summary": "cw: spoilers", "sensitive": true },
344		"bsky": { "labels": { … } } }
345		```
346
347		⛔ Never `$`-prefixed — `$type`, `$link`, `$bytes` are atproto's reserved space.
348
349		The constraint is narrower than it first appears. Namespacing only matters when attaching extras
350		to someone else's lexicon. On `net.favela.post` we own the namespace outright, so bare `audience`
351		is correct and cannot collide.
352
353		⛔ Never hang favela data on `app.bsky.feed.post`. The projection is bsky's view; anything bsky
354		cannot express is simply absent from it. This removes the collision question rather than managing
355		it, and it is why §2's "preservation is not guaranteed" finding does not endanger the design.
356
357		---
358
359		## 7. The double-write — resolved, not deferred
360
361		**bsky can only see records that exist in the repo. So the `app.bsky.feed.post` projection must be
362		materialised and signed. There will be two records per public post.** This is a real cost and the
363		design does not pretend otherwise.
364
365		Why it is acceptable here:
366
367		- Storage is negligible and measured. Records average ~132 bytes; all 148 total ~32 KB.
368		Doubling is ~32 KB against a 400 MB substrate. Storage was never the concern.
369		- Coherence is the real cost, and §3.1's shared rkey makes the pairing mechanical rather than a
370		tracked relationship: one rkey, two collections, cascade on delete and edit.
371		- It is confined to public posts. Fediverse-only and local-only posts write one record.
372
373		What the implementation MUST guarantee:
374
375		1. Delete of a canonical record deletes its projection in the same batch. A tombstone for one and
376		not the other is a divergence with no recovery path.
377		2. Edit re-derives the projection from the canonical. The projection is never edited directly.
378		3. The projection is derived output. Nothing reads it as a source of truth. If canonical and
379		projection disagree, the canonical wins and the projection is rebuilt.
380		4. `sign()` materialises both before signing, so a commit never contains one without the other.
381
382		---
383
384		## 8. Migration — forward-only, nothing rewritten
385
386		There are 148 existing `app.bsky.feed.post` records and an unknown number of
387		`net.favela.local.post` records.
388
389		They are not converted. ⛔ Never re-encode a record you did not author. This project has
390		already destroyed records exactly this way — a second writer re-encoded two of wj62's posts, dropped
391		`langs`, and the bytes stopped hashing to their converged pointers. They were unrecoverable. That
392		hazard is higher under this design, because every extra field is an unknown field to somebody, and
393		the atproto spec warns about precisely this: *"when deserializing and re-serializing data, unknown
394		fields risk being stripped away."*
395
396		Legacy rule: an `app.bsky.feed.post` with no canonical counterpart at the same rkey is a
397		legacy post. It is read as `audience: [Public]`, `source.mediaType: text/plain`, no extensions. It
398		renders, federates and threads exactly as it does today. Nothing rewrites it.
399
400		Two canonical vocabularies must not coexist indefinitely. New writes go to `net.favela.post`
401		from the cutover. Legacy records age out by attrition and are never converted. The end state is one
402		vocabulary for everything authored after the cutover, and a bounded, shrinking legacy set —
403		distinguishable by a single lookup.
404
405		---
406
407		## 9. Read path
408
409		This section exists because the write path is easy to design and the read path is where the work is.
410		It is the largest unestimated part of this design and is deliberately called out rather than
411		buried.
412
413		Affected in favela-web:
414
415		- `lib/scopes.js` — `SCOPE_COLLECTION` stops being a scope→collection map; audience replaces it.
416		- `lib/appview.js` — ingest folds `net.favela.post` and legacy collections, deduplicating by
417		rkey with canonical preferred. `INGEST_VERSION` bumps; a full re-index is required.
418		- `lib/render.js` — renders `source.content` as markdown when present, falling back to
419		`text` + `facets` for legacy.
420		- Threading, search, notifications, RSS/Atom, bookmarks — every path that assumes a post's
421		collection is `app.bsky.feed.post`.
422		- `lib/writer.js` — authors canonical + projection.
423
424		Affected elsewhere: hpp's sign/materialize path (§7.4), ap-bridge's `POST_COLLECTION` and its
425		projection rule (audience-driven rather than collection-driven), atproto-store's `POST_COLLECTIONS`.
426
427		---
428
429		## 10. Interop loss
430
431		Today any atproto client can read the repo directly and see posts. After the cutover, a client that
432		does not know `net.favela.post` sees only the `app.bsky.feed.post` projections — so public posts
433		remain fully readable by any atproto tool, and non-public posts do not.
434
435		That is the correct trade and it is the point: a post addressed away from bsky should not be legible
436		as a bsky post. But it must be stated plainly, because it is a genuine reduction in what third-party
437		tooling can see.
438
439		---
440
441		## 10.5 Editing
442
443		Editing is already possible end to end and is blocked only by the absence of a UI. Researched and
444		verified 2026-08-05.
445
446		### 10.5.1 The protocol allows it; bsky's application refuses it
447
448		`putRecord` with the same rkey replaces the record: same URI, new CID. The PDS is
449		lexicon-agnostic and accepts it. But bsky's AppView deliberately ignores post updates — in their
450		own words, *"The Bluesky service prohibits updating posts, so appview intentionally ignores it… the
451		PDS is expected to be lexicon agnostic, so putRecord succeeds, but appview, which is in charge of
452		application logic, ignores it"* ([discussion #3038](https://github.com/bluesky-social/atproto/discussions/3038)).
453
454		The tell is an inconsistency, not an error: **`getRecord` and `getPosts` reflect the edit;
455		`getPostThread` does not.** It is described there as "a specification unique to bsky post" — an
456		application policy, not a protocol limit.
457
458		This is the SAME shape as the audience problem (§5), arriving independently: bsky's application
459		constraints became ours because its lexicon is our storage format.
460
461		\| surface \| edit visible? \| why \|
462		\|---\|---\|---\|
463		\| P2P (canonical) \| ✅ \| we own the appview \|
464		\| ActivityPub \| ✅ \| `Update{Note}` — see §10.5.3 \|
465		\| bsky \| ⚠ partially \| the record updates; the timeline does not \|
466
467		### 10.5.2 We do NOT need atproto's edit-history workaround
468
469		The community pattern is copy-on-write: each edit copies the old record to a new one carrying
470		`$orig`, while the live record accumulates a `$hist` array of strongRefs. rkey stays constant so
471		likes and reposts stay attached. Its author names race conditions, absent transactions, and dead
472		references in the chain as costs.
473
474		That pattern exists because atproto has no history. This substrate has two:
475
476		1. The firehose already carries the chain. `atproto-store/lib/repo-manager.js:99` sets
477		`op.prev = priorCid` on every update, "so a relay can invert without holding the old tree". The
478		previous CID is on the wire, per edit, today.
479		2. The overlay and the autobase hold prior versions natively.
480
481		So `$hist`/`$orig` would reimplement — worse, and inside the record — something the substrate gives
482		for free. ⛔ Do not adopt it. Edit history is a READ over existing state, not a new field.
483
484		### 10.5.3 ⛔ Every edit MUST re-project to ActivityPub with `updated`
485
486		An `Update` whose object lacks an `updated` timestamp is silently discarded by Mastodon: it
487		accepts the POST, returns 2xx, and does nothing — no edit, no attachment re-fetch, `edited_at` stays
488		null. Required since Mastodon 3.5.0; unofficial, so reading the AS2 spec will never reveal it.
489		Verified live 2026-08-05 (ap-bridge 0.0.3 → 0.0.4).
490
491		This costs nothing here: an edit changes the projection, so the fingerprint changes and `Update`
492		fires on its own. It is recorded because getting it wrong is invisible from every local surface.
493
494		### 10.5.4 The honesty requirement
495
496		The UI must not imply an edit reached bsky. And bsky's failure mode is *worse than a clean
497		refusal*: because `getRecord` reflects the edit while `getPostThread` does not, the same post will
498		appear edited in some bsky contexts and stale in others. A user cannot be told "it did not sync"
499		because partly it did.
500
501		Same rule as §5.3: state what is true per surface, never a single "edited" that means three
502		different things.
503
504		### 10.5.5 What is actually missing
505
506		The write path is complete:
507
508		- `atproto-store` — `action: 'update'` handled; `prev` recorded (`repo-manager.js:67,99`)
509		- `favela-web/lib/writer.js:173` — `putRecord(collection, rkey, record, { action = 'update' })` exists
510		- `hyper-proto-peer` — `applyWrite` handles `update`
511
512		`lib/render.js` has zero mentions of editing a post, and FEATURES.md lists delete and
513		edit-profile as done with no post edit. So this is a UI feature, not a protocol feature: an edit
514		affordance on one's own post, a composer prefilled from the record, and an "edited" marker whose
515		history is a read over `prev` rather than a stored field.
516
517		Under §5.4's toggle model an edit is unremarkable: the canonical record is replaced, the AP
518		projection re-emits as `Update`, and the bsky projection is re-materialised — where it will be
519		ignored, honestly and visibly.
520
521		---
522
523		## 11. Open questions
524
525		Honest gaps. This design is not complete until these are closed.
526		(Editing was one of these; it is now settled in §10.5.)
527
528		1. ~~The strip specification (§4.2) is described but not written.~~ CLOSED —
529		`spec/markdown-strip.md` (220 lines) and `spec/vectors/markdown-strip.json` (36 vectors) were
530		written afterwards. `test/strip-vectors.test.mjs` validates the vectors themselves; no stripper
531		exists yet.
532
533		1b. What signs a markdown record. With the canonical record no longer a DAG-CBOR entry in a
534		signed atproto commit chain (§0), authorship and integrity need their own mechanism — a detached
535		per-record signature, or a native append-only log of `{rkey, contentHash, sig}` that peers
536		replicate. Mutability is not an obstacle: the signature covers the current bytes and is
537		re-issued on edit, exactly as atproto re-signs a commit on edit. **This is the next thing to
538		settle; everything in §9 (read path) hangs off it.**
539		2. Embed/attachment mapping across three representations (atproto blob refs, AS2 `attachment`,
540		markdown image syntax) is not worked through.
541		3. ~~The extension dialect (§4.1) names candidates but does not pin a set.~~ CLOSED —
542		pinned in `markdown-strip.md` §2.1 (CommonMark 0.31.2 + six named extensions).
543		4. Community/list audience URIs — the URI form for a favela community room is unspecified.
544		5. `audience` semantics for replies — does a reply inherit its parent's audience? Almost
545		certainly yes, but the failure mode of getting it wrong is publishing to the wrong people.
546		6. Whether the AP projection should carry `source.mediaType: text/markdown` and let Mastodon
547		render it, or continue sending pre-rendered HTML. Live output currently sends both.
548
549		---
550
551		## 12. What this design does not claim
552
553		- It does not make `audience` enforceable on atproto. Nothing can.
554		- It does not remove the double-write. It bounds it.
555		- It does not improve deliverability, availability or performance. It is a vocabulary decision.
556		- Its forcing function is thin today: one fediverse follower, one enforcement point. It is
557		justified by protocol churn and expressiveness, not by present demand. That should be weighed
558		before implementation, not after.

spec/vectors/markdown-strip.json +0 -334

…	…	@@ -1,334 +0,0 @@
1		{
2		"$comment": "Vectors for spec/markdown-strip.md. Offsets are UTF-8 BYTE offsets into `text`, half-open [byteStart, byteEnd). An implementation is conformant when it reproduces every entry byte-for-byte.",
3		"spec": "spec/markdown-strip.md",
4		"vectors": [
5		{
6		"name": "emphasis-stripped",
7		"why": "§3.1 markers go, text stays, no facet — atproto has no bold",
8		"source": "hello world and more",
9		"text": "hello world and more",
10		"facets": []
11		},
12		{
13		"name": "code-span",
14		"why": "§3.1 a code span is text, not markup",
15		"source": "use `npm test` now",
16		"text": "use npm test now",
17		"facets": []
18		},
19		{
20		"name": "link",
21		"why": "§3.2 label becomes text; the uri rides in the facet",
22		"source": "click [here](https://example.com)",
23		"text": "click here",
24		"facets": [
25		{
26		"byteStart": 6,
27		"byteEnd": 10,
28		"type": "link",
29		"value": "https://example.com"
30		}
31		]
32		},
33		{
34		"name": "link-with-emphasis-label",
35		"why": "§3.2 the label is stripped FIRST, then the facet spans the result",
36		"source": "click [bold](https://example.com)",
37		"text": "click bold",
38		"facets": [
39		{
40		"byteStart": 6,
41		"byteEnd": 10,
42		"type": "link",
43		"value": "https://example.com"
44		}
45		]
46		},
47		{
48		"name": "link-empty-label",
49		"why": "§3.2 an empty label emits the url as text",
50		"source": "[](https://example.com)",
51		"text": "https://example.com",
52		"facets": [
53		{
54		"byteStart": 0,
55		"byteEnd": 19,
56		"type": "link",
57		"value": "https://example.com"
58		}
59		]
60		},
61		{
62		"name": "link-non-http-refused",
63		"why": "§3.2 ⛔ javascript: emits the label as PLAIN TEXT, no facet",
64		"source": "[click](javascript:alert(1))",
65		"text": "click",
66		"facets": []
67		},
68		{
69		"name": "bare-url-trailing-punctuation",
70		"why": "§3.3 the normative URL_RE stops before trailing punctuation — the full stop stays text",
71		"source": "see https://x.com.",
72		"text": "see https://x.com.",
73		"facets": [
74		{
75		"byteStart": 4,
76		"byteEnd": 17,
77		"type": "link",
78		"value": "https://x.com"
79		}
80		]
81		},
82		{
83		"name": "mention-resolved",
84		"why": "§3.4 a mention MUST carry a resolved did in the target",
85		"source": "hi [@alice](did:plc:xyz)",
86		"text": "hi @alice",
87		"facets": [
88		{
89		"byteStart": 3,
90		"byteEnd": 9,
91		"type": "mention",
92		"value": "did:plc:xyz"
93		}
94		]
95		},
96		{
97		"name": "mention-bare-is-text",
98		"why": "§3.4 ⛔ a bare @alice is plain text — never a guessed did",
99		"source": "hi @alice",
100		"text": "hi @alice",
101		"facets": []
102		},
103		{
104		"name": "image-produces-nothing",
105		"why": "§3.5 images are embed, not body text",
106		"source": "![a cat](https://x/c.png)",
107		"text": "",
108		"facets": []
109		},
110		{
111		"name": "image-only-strips-empty",
112		"why": "§3.5 a post that is only an image has empty text — valid",
113		"source": "![](https://x/c.png)",
114		"text": "",
115		"facets": []
116		},
117		{
118		"name": "wikilink",
119		"why": "§3.6 degrades to its label, no facet — no atproto/AS2 equivalent",
120		"source": "see [[Design Note]]",
121		"text": "see Design Note",
122		"facets": []
123		},
124		{
125		"name": "wikilink-labelled",
126		"why": "§3.6 the pipe form uses the label",
127		"source": "see [[Design Note\|the note]]",
128		"text": "see the note",
129		"facets": []
130		},
131		{
132		"name": "hashtag",
133		"why": "§3.7 the # stays in TEXT, the facet value drops it",
134		"source": "a #p2p post",
135		"text": "a #p2p post",
136		"facets": [
137		{
138		"byteStart": 2,
139		"byteEnd": 6,
140		"type": "tag",
141		"value": "p2p"
142		}
143		]
144		},
145		{
146		"name": "hashtag-midword-is-text",
147		"why": "§3.7 a # inside a word is literal",
148		"source": "c#sharp",
149		"text": "c#sharp",
150		"facets": []
151		},
152		{
153		"name": "strikethrough",
154		"why": "§3.1 extension: markers go",
155		"source": "~~gone~~ here",
156		"text": "gone here",
157		"facets": []
158		},
159		{
160		"name": "footnote-dropped",
161		"why": "§3.9 refs and definitions have no projection target",
162		"source": "claim[^1]\n\n[^1]: the source",
163		"text": "claim",
164		"facets": []
165		},
166		{
167		"name": "hard-break",
168		"why": "§3.8 backslash break becomes one newline",
169		"source": "one\\\ntwo",
170		"text": "one\ntwo",
171		"facets": []
172		},
173		{
174		"name": "soft-break-is-space",
175		"why": "§3.8 a soft wrap is a space, not a newline",
176		"source": "one\ntwo",
177		"text": "one two",
178		"facets": []
179		},
180		{
181		"name": "multibyte-before-facet",
182		"why": "★ é is 2 bytes — a character-offset implementation puts this facet one byte early",
183		"source": "café https://x.com",
184		"text": "café https://x.com",
185		"facets": [
186		{
187		"byteStart": 6,
188		"byteEnd": 19,
189		"type": "link",
190		"value": "https://x.com"
191		}
192		]
193		},
194		{
195		"name": "emoji-before-facet",
196		"why": "★ each emoji is 4 bytes — the classic off-by-six for a char-offset implementation",
197		"source": "🎉🎉 tap [here](https://example.com)",
198		"text": "🎉🎉 tap here",
199		"facets": [
200		{
201		"byteStart": 13,
202		"byteEnd": 17,
203		"type": "link",
204		"value": "https://example.com"
205		}
206		]
207		},
208		{
209		"name": "multibyte-inside-mention",
210		"why": "★ the facet itself spans multi-byte characters",
211		"source": "héllo [@wörld](did:plc:abc)",
212		"text": "héllo @wörld",
213		"facets": [
214		{
215		"byteStart": 7,
216		"byteEnd": 14,
217		"type": "mention",
218		"value": "did:plc:abc"
219		}
220		]
221		},
222		{
223		"name": "heading",
224		"why": "§4.2 marker removed, block like any other",
225		"source": "## Title\n\nbody",
226		"text": "Title\n\nbody",
227		"facets": []
228		},
229		{
230		"name": "code-fence-verbatim",
231		"why": "§4.3 contents verbatim, no fence, no re-indent",
232		"source": "```js\nconst a = 1\n```",
233		"text": "const a = 1",
234		"facets": []
235		},
236		{
237		"name": "blockquote-no-prefix",
238		"why": "§4.4 no '> ' is invented — that would be a convention to re-specify",
239		"source": "> quoted\n> more",
240		"text": "quoted more",
241		"facets": []
242		},
243		{
244		"name": "bullet-list",
245		"why": "§4.5 a marker IS kept, or a list becomes indistinguishable from paragraphs",
246		"source": "- one\n- two",
247		"text": "• one\n• two",
248		"facets": []
249		},
250		{
251		"name": "ordered-list-renumbered",
252		"why": "§4.5 the RENDERED sequence, not the literal source numbers",
253		"source": "1. a\n1. b\n1. c",
254		"text": "1. a\n2. b\n3. c",
255		"facets": []
256		},
257		{
258		"name": "table",
259		"why": "§4.6 rows on lines, cells joined with ' \| '; lossy and deliberately unclever",
260		"source": "\| a \| b \|\n\|---\|---\|\n\| 1 \| 2 \|",
261		"text": "a \| b\n1 \| 2",
262		"facets": []
263		},
264		{
265		"name": "thematic-break-omitted",
266		"why": "§4.7 omitted entirely, including its separator",
267		"source": "one\n\n---\n\ntwo",
268		"text": "one\n\ntwo",
269		"facets": []
270		},
271		{
272		"name": "directive-contents-kept",
273		"why": "§4.8 a spoiler projects as its inner text — dropping it would misrepresent the post elsewhere",
274		"source": ":::spoiler\nthe ending\n:::",
275		"text": "the ending",
276		"facets": []
277		},
278		{
279		"name": "crlf-normalised",
280		"why": "§5.1 CRLF becomes LF before parsing",
281		"source": "one\r\n\r\ntwo",
282		"text": "one\n\ntwo",
283		"facets": []
284		},
285		{
286		"name": "blank-lines-collapsed",
287		"why": "§5.3 three or more newlines collapse to two",
288		"source": "one\n\n\n\ntwo",
289		"text": "one\n\ntwo",
290		"facets": []
291		},
292		{
293		"name": "outer-whitespace-trimmed",
294		"why": "§5.4 leading and trailing whitespace is removed from the whole text",
295		"source": "\n\n hello \n\n",
296		"text": "hello",
297		"facets": []
298		},
299		{
300		"name": "empty-source",
301		"why": "§2 total function — empty in, empty out, no throw",
302		"source": "",
303		"text": "",
304		"facets": []
305		},
306		{
307		"name": "whitespace-only-source",
308		"why": "§2 total function — a whitespace-only source degrades to empty, never throws",
309		"source": " \n\n ",
310		"text": "",
311		"facets": []
312		},
313		{
314		"name": "facets-sorted-by-bytestart",
315		"why": "§6.1 sorted ascending — required for byte-determinism",
316		"source": "[a](https://one.example) and [b](https://two.example)",
317		"text": "a and b",
318		"facets": [
319		{
320		"byteStart": 0,
321		"byteEnd": 1,
322		"type": "link",
323		"value": "https://one.example"
324		},
325		{
326		"byteStart": 6,
327		"byteEnd": 7,
328		"type": "link",
329		"value": "https://two.example"
330		}
331		]
332		}
333		]
334		}

favela-web commits / 71f2724

Changed files (4) +21 -1112