1445701 · hyper-social

Changed files (3) +1034 -0

docs/AUTO-DISCOVERY-SPEC.md +726 -0

…	…	@@ -0,0 +1,726 @@
	1	# Auto-Discovery Implementation Spec
	2
	3	Code changes required to implement the layered auto-discovery system described in [AUTO-DISCOVERY.md](AUTO-DISCOVERY.md). Covers keet-daemon (JavaScript), godot-keet (GDScript), and bridge integration.
	4
	5	## Current Codebase
	6
	7	### keet-daemon (JavaScript)
	8
	9	\| File \| Role \| Lines \|
	10	\|------\|------\|-------\|
	11	\| `packages/keet-daemon/index.js` \| Startup, discovery config, registry lifecycle \| ~400 \|
	12	\| `packages/keet-daemon/lib/room-discovery.js` \| Autobase-backed registry (Corestore + Hyperswarm) \| ~780 \|
	13	\| `packages/keet-daemon/lib/room-discovery-apply.js` \| Autobase apply function (Hyperbee mutations) \| ~200 \|
	14	\| `packages/keet-daemon/lib/router.js` \| UDS RPC dispatch (90 methods) \| ~1840 \|
	15	\| `packages/keet-daemon/lib/uds-transport.js` \| Unix Domain Socket framed-stream transport \| ~200 \|
	16
	17	### godot-keet (GDScript)
	18
	19	\| File \| Role \| Lines \|
	20	\|------\|------\|-------\|
	21	\| `addons/keet_social/keet_discovery.gd` \| Discovery API (list, search, register, meta, roles, bans, invites) \| ~420 \|
	22	\| `addons/keet_social/keet_room.gd` \| Room CRUD (list, create, join, leave, members) \| ~200 \|
	23	\| `addons/keet_social/keet_chat.gd` \| Chat (subscribe, send, history, edit, react) \| ~180 \|
	24	\| `scripts/GroupBar.gd` \| Vertical icon bar (home, groups, add) \| ~250 \|
	25	\| `scripts/RoomListPanel.gd` \| Room list with home/group/discover modes \| ~350 \|
	26
	27	---
	28
	29	## Layer 0: Topic-Based DHT Discovery
	30
	31	### keet-daemon Changes
	32
	33	New file: `lib/topic-discovery.js` (~150 LOC)
	34
	35	```javascript
	36	// Topic-based room discovery via DHT announce/lookup
	37	// No Autobase needed — just Hyperswarm topic joins
	38
	39	import Hyperswarm from 'hyperswarm'
	40	import crypto from 'hypercore-crypto'
	41
	42	class TopicDiscovery {
	43	constructor () {
	44	this._swarm = new Hyperswarm()
	45	this._announced = new Map() // topic string → { rooms: Set, topic: Buffer }
	46	this._results = new Map() // topic string → [{ roomId, title, description, memberCount }]
	47	}
	48
	49	// Announce a room on a topic
	50	async announce (topicString, roomMeta) {
	51	const topic = crypto.discoveryKey(Buffer.from('discovery:' + topicString))
	52	// Join topic, respond to lookups with room metadata
	53	// On connection: exchange room metadata via protomux channel
	54	}
	55
	56	// Search for rooms by topic
	57	async search (topicString, timeout = 5000) {
	58	const topic = crypto.discoveryKey(Buffer.from('discovery:' + topicString))
	59	// Join topic temporarily, collect room metadata from peers
	60	// Return after timeout with accumulated results
	61	}
	62
	63	// Unannounce a room from a topic
	64	async unannounce (topicString, roomId) { }
	65
	66	// List all topics we're announcing on
	67	listAnnounced () { }
	68	}
	69	```
	70
	71	Protocol for topic exchange: Use a protomux channel on the swarm connection. Peers exchange room metadata as compact-encoded messages:
	72
	73	```javascript
	74	// Message format on topic swarm connections
	75	{
	76	type: 'room-announce',
	77	roomId: '<hex>',
	78	title: 'Hypercore Dev Chat',
	79	description: 'Discussion about Hypercore protocol',
	80	memberCount: 47,
	81	tags: ['hypercore', 'p2p', 'development'],
	82	registries: ['c5facb91...'], // Layer 1 data
	83	bridges: ['irc:libera:#hypercore'] // Cross-protocol info
	84	}
	85	```
	86
	87	Router additions (`lib/router.js`):
	88
	89	```
	90	topic.search(query) → topicDiscovery.search(query)
	91	topic.announce(topic, roomId) → topicDiscovery.announce(topic, roomMeta)
	92	topic.unannounce(topic, roomId) → topicDiscovery.unannounce(topic, roomId)
	93	topic.announced() → topicDiscovery.listAnnounced()
	94	```
	95
	96	index.js changes:
	97	- Import TopicDiscovery
	98	- Pass to Router constructor alongside existing discovery functions
	99	- No config file needed (topics are ephemeral — rooms re-announce on startup)
	100
	101	### godot-keet Changes
	102
	103	New file: `addons/keet_social/keet_topic_discovery.gd` (~80 LOC)
	104
	105	```gdscript
	106	class_name KeetTopicDiscovery
	107	extends Node
	108	## Topic-based DHT discovery — search by keyword, no registry key needed.
	109
	110	signal search_results(topic: String, rooms: Array)
	111	signal announce_ok(topic: String, room_id: String)
	112	signal topic_error(message: String)
	113
	114	var client: KeetUdsClient
	115
	116	func search(topic: String, callback: Callable = Callable()) -> void:
	117	_require_client()
	118	client.call_rpc("topic.search", {"query": topic}, func(result, err):
	119	if err != null:
	120	topic_error.emit("search: %s" % str(err))
	121	return
	122	var rooms: Array = result if result is Array else []
	123	search_results.emit(topic, rooms)
	124	if callback.is_valid():
	125	callback.call(rooms)
	126	)
	127
	128	func announce(topic: String, room_id: String, callback: Callable = Callable()) -> void:
	129	# ...
	130
	131	func unannounce(topic: String, room_id: String, callback: Callable = Callable()) -> void:
	132	# ...
	133	```
	134
	135	GroupBar.gd changes: Add a search field at the top of the GroupBar (or in the "+" dialog) that calls `topic.search` instead of requiring a hex key. Results show as a list of discoverable rooms with "Join" buttons.
	136
	137	RoomListPanel.gd changes: When creating a room, offer to announce it on topics (tags field already exists in room registration — reuse for topic announcement).
	138
	139	### Bridge Integration
	140
	141	Each bridge announces its bridged rooms on platform-specific topics:
	142
	143	```javascript
	144	// bridge-irc: after connecting to IRC channel
	145	await hub.topicDiscovery.announce('irc:libera:' + channel, roomMeta)
	146	await hub.topicDiscovery.announce(channel.replace('#', ''), roomMeta)
	147
	148	// bridge-discord: after connecting to Discord channel
	149	await hub.topicDiscovery.announce('discord:' + guildName + ':' + channelName, roomMeta)
	150
	151	// bridge-matrix: after connecting to Matrix room
	152	await hub.topicDiscovery.announce('matrix:' + roomAlias, roomMeta)
	153	```
	154
	155	Bridges announce on both platform-specific topics (e.g., `irc:libera:#ai-town`) and generic topics (e.g., `ai-town`, `ai`). Users searching for "ai" find rooms regardless of which platform they originate from.
	156
	157	---
	158
	159	## Layer 1: Room-Carried Registry Metadata
	160
	161	### keet-daemon Changes
	162
	163	router.js changes (~30 LOC):
	164
	165	Extend the `_discoveryRegister` method to also write registry info into the room's config:
	166
	167	```javascript
	168	// After successfully registering a room in a discovery registry,
	169	// attach registry info to the room config so joiners can discover it
	170	async _discoveryRegister (params) {
	171	const result = await this._existingRegisterLogic(params)
	172
	173	// NEW: Write registry reference into room config
	174	const roomConfig = await this.backend.api.core.getRoomConfig(params.roomId)
	175	const registries = roomConfig?.registries \|\| []
	176	const registryKey = params.registryKey \|\| this._defaultRegistryKey
	177	if (!registries.includes(registryKey)) {
	178	registries.push(registryKey)
	179	await this.backend.api.core.setRoomConfig(params.roomId, {
	180	...roomConfig,
	181	registries
	182	})
	183	}
	184
	185	return result
	186	}
	187	```
	188
	189	Add a new route to suggest registries based on room config:
	190
	191	```
	192	discovery.suggestFromRoom(roomId) → reads room config → returns registries the user doesn't follow
	193	```
	194
	195	index.js changes: When a room is joined (detected via subscription events), check its config for registry references. If the user doesn't follow any of them, queue a suggestion.
	196
	197	### godot-keet Changes
	198
	199	keet_discovery.gd additions (~40 LOC):
	200
	201	```gdscript
	202	signal registry_suggested(key: String, label: String, source_room: String)
	203
	204	func check_room_registries(room_id: String) -> void:
	205	_require_client()
	206	client.call_rpc("discovery.suggestFromRoom", {"roomId": room_id}, func(result, err):
	207	if err != null:
	208	return # Silent — suggestions are best-effort
	209	if result is Array:
	210	for suggestion in result:
	211	registry_suggested.emit(
	212	suggestion.get("key", ""),
	213	suggestion.get("label", ""),
	214	room_id
	215	)
	216	)
	217	```
	218
	219	RoomListPanel.gd changes: After joining a room, call `check_room_registries`. If suggestions come back, show a non-intrusive notification: "This room is in 'Hypercore Dev'. Follow it?" with Accept/Dismiss buttons.
	220
	221	GroupBar.gd changes: Suggested groups appear as dimmed icons with a "+" badge. Clicking accepts the suggestion (calls `add_registry`). Right-clicking dismisses.
	222
	223	### Bridge Integration
	224
	225	When a bridge creates or joins a room, it writes the bridge's platform info into the room config:
	226
	227	```javascript
	228	// bridge-irc: after bridging a room
	229	await daemon.handle('room.config.set', {
	230	roomId,
	231	bridges: [{
	232	platform: 'irc',
	233	server: 'libera',
	234	channel: '#ai-town',
	235	bridgeId: 'bridge-irc-libera'
	236	}]
	237	})
	238	```
	239
	240	When godot-keet reads room config, it sees the `bridges` array and can show platform badges (Layer 2 from the Roadmap).
	241
	242	---
	243
	244	## Layer 2: MI-Based Local Clustering
	245
	246	### keet-daemon Changes
	247
	248	New file: `lib/mi-engine.js` (~250 LOC)
	249
	250	The MI computation engine. Runs locally from the user's own room/member data.
	251
	252	```javascript
	253	class MIEngine {
	254	constructor (router) {
	255	this._router = router
	256	this._cooccurrence = new Map() // 'roomA:roomB' → count
	257	this._roomMembers = new Map() // roomId → Set<memberId>
	258	this._clusters = []
	259	this._lastCompute = 0
	260	}
	261
	262	// Gather data from the user's rooms
	263	async collectData () {
	264	const rooms = await this._router.handle('rooms', {})
	265	for (const room of rooms) {
	266	const members = await this._router.handle('members', { roomId: room.id })
	267	this._roomMembers.set(room.id, new Set(members.map(m => m.memberId)))
	268	}
	269	}
	270
	271	// Compute MI between all room pairs
	272	computeMI () {
	273	const allMembers = new Set()
	274	for (const members of this._roomMembers.values()) {
	275	for (const m of members) allMembers.add(m)
	276	}
	277	const N = allMembers.size
	278	if (N === 0) return new Map()
	279
	280	const mi = new Map()
	281	const roomIds = [...this._roomMembers.keys()]
	282
	283	for (let i = 0; i < roomIds.length; i++) {
	284	for (let j = i + 1; j < roomIds.length; j++) {
	285	const a = roomIds[i], b = roomIds[j]
	286	const membersA = this._roomMembers.get(a)
	287	const membersB = this._roomMembers.get(b)
	288	const overlap = new Set([...membersA].filter(m => membersB.has(m)))
	289
	290	if (overlap.size === 0) continue
	291
	292	const pA = membersA.size / N
	293	const pB = membersB.size / N
	294	const pAB = overlap.size / N
	295	const miValue = Math.log2(pAB / (pA * pB))
	296
	297	mi.set(`${a}:${b}`, { roomA: a, roomB: b, mi: miValue, overlap: overlap.size })
	298	}
	299	}
	300
	301	return mi
	302	}
	303
	304	// Cluster rooms by MI (greedy agglomerative)
	305	cluster (mi, threshold = 1.0) {
	306	// Union-find: merge rooms with MI above threshold
	307	// Returns array of clusters: [{ rooms: [roomId, ...], avgMI: number }]
	308	}
	309
	310	// Check if a cluster is already covered by a registry
	311	async findUncoveredClusters (clusters) {
	312	const registries = await this._router.handle('discovery.registries', {})
	313	// For each registry, get its room list
	314	// Compare with clusters — if a cluster's rooms are >70% covered, it's matched
	315	// Return unmatched clusters as suggestions
	316	}
	317
	318	// Full pipeline: collect → MI → cluster → find uncovered → return suggestions
	319	async compute () {
	320	await this.collectData()
	321	const mi = this.computeMI()
	322	const clusters = this.cluster(mi)
	323	const suggestions = await this.findUncoveredClusters(clusters)
	324	this._clusters = clusters
	325	this._lastCompute = Date.now()
	326	return { clusters, suggestions, mi: [...mi.values()] }
	327	}
	328	}
	329	```
	330
	331	Router additions (`lib/router.js`):
	332
	333	```
	334	graph.compute() → miEngine.compute() (triggers full pipeline)
	335	graph.clusters() → miEngine._clusters (cached result)
	336	graph.mi(roomId) → MI values for one room vs all others
	337	graph.neighbors(roomId, limit) → top N rooms by MI to given room
	338	graph.suggestions() → uncovered clusters (suggested groups)
	339	```
	340
	341	index.js changes:
	342	- Import MIEngine, create instance, pass to Router
	343	- Optional: run MI computation periodically (every 5 minutes) and cache results
	344	- Or: run on-demand when UI requests it
	345
	346	### godot-keet Changes
	347
	348	New file: `addons/keet_social/keet_graph.gd` (~120 LOC)
	349
	350	```gdscript
	351	class_name KeetGraph
	352	extends Node
	353	## Social graph computation — MI, clusters, suggestions.
	354	## All computation happens in the daemon (local data only).
	355
	356	signal clusters_computed(clusters: Array)
	357	signal suggestions_ready(suggestions: Array)
	358	signal neighbors_found(room_id: String, neighbors: Array)
	359	signal graph_error(message: String)
	360
	361	var client: KeetUdsClient
	362
	363	func compute(callback: Callable = Callable()) -> void:
	364	_require_client()
	365	client.call_rpc("graph.compute", {}, func(result, err):
	366	if err != null:
	367	graph_error.emit("compute: %s" % str(err))
	368	return
	369	if result is Dictionary:
	370	clusters_computed.emit(result.get("clusters", []))
	371	suggestions_ready.emit(result.get("suggestions", []))
	372	if callback.is_valid():
	373	callback.call(result)
	374	)
	375
	376	func get_neighbors(room_id: String, limit: int = 5, callback: Callable = Callable()) -> void:
	377	_require_client()
	378	client.call_rpc("graph.neighbors", {"roomId": room_id, "limit": limit}, func(result, err):
	379	if err != null:
	380	graph_error.emit("neighbors: %s" % str(err))
	381	return
	382	var rooms: Array = result if result is Array else []
	383	neighbors_found.emit(room_id, rooms)
	384	if callback.is_valid():
	385	callback.call(rooms)
	386	)
	387
	388	func get_suggestions(callback: Callable = Callable()) -> void:
	389	_require_client()
	390	client.call_rpc("graph.suggestions", {}, func(result, err):
	391	if err != null:
	392	graph_error.emit("suggestions: %s" % str(err))
	393	return
	394	var suggs: Array = result if result is Array else []
	395	suggestions_ready.emit(suggs)
	396	if callback.is_valid():
	397	callback.call(suggs)
	398	)
	399	```
	400
	401	GroupBar.gd changes:
	402	- On startup (after rooms load), call `graph.suggestions` to check for uncovered clusters
	403	- If suggestions exist, show a dimmed "sparkle" icon in the GroupBar
	404	- Clicking it shows the suggestion: "These N rooms form a natural group: [room list]. Create group?"
	405	- User can name it → `discovery.createRegistry` → auto-register the clustered rooms
	406
	407	RoomListPanel.gd changes:
	408	- When viewing a room, show "Related rooms" section using `graph.neighbors(roomId)`
	409	- Rooms with highest MI appear as suggestions: "You might also be interested in..."
	410
	411	New file: `scripts/GraphView.gd` (~300 LOC)
	412
	413	Force-directed 2D graph view:
	414
	415	```gdscript
	416	class_name GraphView
	417	extends Control
	418	## 2D force-directed graph visualization of the social graph.
	419	## Rooms as nodes, MI as edge weights, clusters colored.
	420
	421	var nodes: Dictionary = {} # roomId → Node2D
	422	var edges: Array = [] # [{from, to, mi, line: Line2D}]
	423	var clusters: Array = [] # [{rooms, color, avgMI}]
	424
	425	func _ready():
	426	KeetService.graph.clusters_computed.connect(_on_clusters)
	427	KeetService.graph.compute()
	428
	429	func _on_clusters(cluster_data: Array):
	430	_clear()
	431	_create_nodes(cluster_data)
	432	_create_edges()
	433	_color_clusters()
	434
	435	func _physics_process(delta):
	436	_apply_forces(delta) # Repulsion between all nodes, attraction along edges
	437
	438	func _apply_forces(delta):
	439	for id_a in nodes:
	440	var node_a = nodes[id_a]
	441	# Repulsion from all other nodes (Coulomb's law)
	442	for id_b in nodes:
	443	if id_a == id_b: continue
	444	var node_b = nodes[id_b]
	445	var diff = node_a.position - node_b.position
	446	var dist = max(diff.length(), 10.0)
	447	node_a.position += diff.normalized() * (REPULSION / (dist * dist)) * delta
	448	# Attraction along edges (Hooke's law, weighted by MI)
	449	for edge in edges:
	450	var a = nodes[edge.from]
	451	var b = nodes[edge.to]
	452	var diff = b.position - a.position
	453	var force = diff * ATTRACTION * edge.mi * delta
	454	a.position += force
	455	b.position -= force
	456
	457	func _input(event):
	458	if event is InputEventMouseButton and event.pressed:
	459	# Click node → enter room
	460	for id in nodes:
	461	if nodes[id].get_rect().has_point(event.position):
	462	KeetService.room.select_room(id)
	463	# Switch to chat view
	464	```
	465
	466	Main.gd changes: Add a toggle button in the toolbar: "Graph View" ↔ "Room List". Toggling swaps between `GroupBar + RoomListPanel` and `GraphView`.
	467
	468	### Bridge Integration
	469
	470	MI computation naturally includes bridged members. A member who appears via a bridge (e.g., `[irc]dave`) is treated like any other member for co-occurrence counting. Cross-protocol communities cluster automatically.
	471
	472	The graph view can color nodes by protocol — Keet rooms in blue, IRC-bridged rooms in green, Discord-bridged rooms in purple. Bridge rooms (rooms active on multiple protocols) have multi-colored borders.
	473
	474	---
	475
	476	## Layer 3: Gossip-Based Registry Discovery
	477
	478	### keet-daemon Changes
	479
	480	router.js additions (~60 LOC):
	481
	482	```javascript
	483	// When browsing a registry, collect cross-registry links
	484	async _discoveryRelatedRegistries (params) {
	485	const registryKey = params.registryKey \|\| this._defaultRegistryKey
	486	const disc = await this._getDiscovery(registryKey)
	487	if (!disc) return []
	488
	489	// Get all rooms in this registry
	490	const rooms = await disc.list({})
	491	const foreignRegistries = new Map() // key → { label, sharedRooms: count }
	492
	493	for (const room of rooms) {
	494	// Check if this room has registry references (Layer 1 data)
	495	const config = await this.backend.api.core.getRoomConfig(room.roomId).catch(() => null)
	496	const regs = config?.registries \|\| []
	497	for (const regKey of regs) {
	498	if (regKey === registryKey) continue // skip self
	499	if (!foreignRegistries.has(regKey)) {
	500	foreignRegistries.set(regKey, { key: regKey, label: '', sharedRooms: 0 })
	501	}
	502	foreignRegistries.get(regKey).sharedRooms++
	503	}
	504	}
	505
	506	// Sort by shared room count (descending)
	507	const related = [...foreignRegistries.values()]
	508	.filter(r => r.sharedRooms >= 2)
	509	.sort((a, b) => b.sharedRooms - a.sharedRooms)
	510
	511	// Try to get labels from any we already follow
	512	const config = this._loadDiscoveryConfig()
	513	for (const r of related) {
	514	const known = config.registries.find(reg => reg.key === r.key)
	515	if (known) r.label = known.label
	516	r.alreadyFollowing = !!known
	517	}
	518
	519	return related
	520	}
	521	```
	522
	523	New routes:
	524
	525	```
	526	discovery.related(registryKey) → _discoveryRelatedRegistries()
	527	```
	528
	529	### godot-keet Changes
	530
	531	keet_discovery.gd additions (~25 LOC):
	532
	533	```gdscript
	534	signal related_registries(registries: Array)
	535
	536	func get_related(registry_key: String = "", callback: Callable = Callable()) -> void:
	537	_require_client()
	538	var params := {}
	539	if registry_key != "":
	540	params["registryKey"] = registry_key
	541	client.call_rpc("discovery.related", params, func(result, err):
	542	if err != null:
	543	return # Silent
	544	var regs: Array = result if result is Array else []
	545	related_registries.emit(regs)
	546	if callback.is_valid():
	547	callback.call(regs)
	548	)
	549	```
	550
	551	GroupSettings changes (Info tab): Show "Related Groups" section listing registries that share rooms. Each entry shows: registry name (or key prefix), number of shared rooms, and "Follow" button (if not already following).
	552
	553	GroupBar.gd changes: When browsing a group's rooms, periodically check `discovery.related`. If new related registries appear, show a subtle indicator on the GroupBar.
	554
	555	---
	556
	557	## Layer 4: Collaborative Auto-Registry
	558
	559	### keet-daemon Changes
	560
	561	mi-engine.js additions (~80 LOC):
	562
	563	```javascript
	564	// Generate a fingerprint of a local cluster
	565	clusterFingerprint (cluster) {
	566	const sortedRooms = [...cluster.rooms].sort()
	567	return crypto.hash(Buffer.from(sortedRooms.join(':')))
	568	.toString('hex')
	569	.slice(0, 32)
	570	}
	571
	572	// Compare fingerprints using Jaccard similarity
	573	fingerprintSimilarity (fpA, roomsA, fpB, roomsB) {
	574	const setA = new Set(roomsA)
	575	const setB = new Set(roomsB)
	576	const intersection = new Set([...setA].filter(r => setB.has(r)))
	577	const union = new Set([...setA, ...setB])
	578	return intersection.size / union.size
	579	}
	580	```
	581
	582	router.js additions (~40 LOC):
	583
	584	```javascript
	585	// Publish cluster fingerprint (opt-in) — attached to identity profile
	586	async _graphPublishFingerprint (params) {
	587	// Store fingerprint in identity metadata (visible to room members)
	588	// { fingerprint: '<hash>', roomCount: N, timestamp: Date.now() }
	589	}
	590
	591	// Check if any room members have similar fingerprints
	592	async _graphFindSimilarClusters (params) {
	593	// For each member in the room, check their published fingerprint
	594	// Compare with local clusters using Jaccard similarity
	595	// Return members with similarity > threshold
	596	}
	597	```
	598
	599	New routes:
	600
	601	```
	602	graph.publishFingerprint(clusterId) → publish cluster fingerprint to identity metadata
	603	graph.findSimilarClusters(roomId) → find members with similar clusters
	604	graph.mergeCluster(members, label) → create shared registry from similar local clusters
	605	```
	606
	607	### godot-keet Changes
	608
	609	keet_graph.gd additions (~40 LOC):
	610
	611	```gdscript
	612	signal similar_clusters_found(room_id: String, matches: Array)
	613	signal merge_suggested(label: String, members: Array, rooms: Array)
	614
	615	func publish_fingerprint(cluster_id: int) -> void:
	616	# Opt-in: publish current cluster fingerprint
	617
	618	func find_similar(room_id: String, callback: Callable = Callable()) -> void:
	619	client.call_rpc("graph.findSimilarClusters", {"roomId": room_id}, func(result, err):
	620	if err != null: return
	621	var matches: Array = result if result is Array else []
	622	if matches.size() > 0:
	623	similar_clusters_found.emit(room_id, matches)
	624	if callback.is_valid():
	625	callback.call(matches)
	626	)
	627	```
	628
	629	GroupBar.gd changes: When a merge suggestion appears, show a special "merge" icon with tooltip: "3 people have similar room groups. Create shared group?" Clicking opens a dialog showing the overlapping rooms and participants.
	630
	631	---
	632
	633	## Summary: All New Files
	634
	635	### keet-daemon (JavaScript)
	636
	637	\| File \| Layer \| Lines (est) \| Purpose \|
	638	\|------\|-------\|-------------\|---------\|
	639	\| `lib/topic-discovery.js` \| 0 \| ~150 \| DHT topic announce/search \|
	640	\| `lib/mi-engine.js` \| 2, 4 \| ~330 \| MI computation, clustering, fingerprints \|
	641
	642	### keet-daemon Modified Files
	643
	644	\| File \| Layers \| Changes \|
	645	\|------\|--------\|---------\|
	646	\| `index.js` \| 0, 1, 2 \| Import TopicDiscovery + MIEngine, pass to Router \|
	647	\| `lib/router.js` \| 0, 1, 2, 3, 4 \| ~15 new routes (topic., graph., discovery.related, discovery.suggestFromRoom) \|
	648
	649	### godot-keet (GDScript) — New Files
	650
	651	\| File \| Layer \| Lines (est) \| Purpose \|
	652	\|------\|-------\|-------------\|---------\|
	653	\| `addons/keet_social/keet_topic_discovery.gd` \| 0 \| ~80 \| Topic-based search API \|
	654	\| `addons/keet_social/keet_graph.gd` \| 2, 4 \| ~160 \| MI/cluster/suggestion API \|
	655	\| `scripts/GraphView.gd` \| 2 \| ~300 \| Force-directed graph visualization \|
	656
	657	### godot-keet Modified Files
	658
	659	\| File \| Layers \| Changes \|
	660	\|------\|--------\|---------\|
	661	\| `addons/keet_social/keet_discovery.gd` \| 1, 3 \| ~65 LOC added (suggestFromRoom, related, check_room_registries) \|
	662	\| `scripts/GroupBar.gd` \| 0, 1, 2, 4 \| Search field, suggestion icons, merge UI \|
	663	\| `scripts/RoomListPanel.gd` \| 0, 1, 2 \| Topic search in create dialog, related rooms, join suggestions \|
	664	\| `scripts/GroupSettings.gd` \| 3 \| Related groups in Info tab \|
	665	\| `scripts/Main.gd` \| 2 \| Graph view toggle button \|
	666
	667	### Bridge Integration (per bridge)
	668
	669	\| File \| Changes \|
	670	\|------\|---------\|
	671	\| `bridge-*/lib/bridge.js` \| ~10 LOC: announce on topic DHT + write bridge metadata to room config \|
	672
	673	### Estimated Total
	674
	675	\| Component \| New LOC \| Modified LOC \|
	676	\|-----------\|---------\|-------------\|
	677	\| keet-daemon JS \| ~480 \| ~200 \|
	678	\| godot-keet GDScript \| ~540 \| ~150 \|
	679	\| Bridges (5 total) \| ~50 \| — \|
	680	\| Total \| ~1,070 \| ~350 \|
	681
	682	## Implementation Order
	683
	684	```
	685	1. Layer 0 (topic discovery) — standalone, no prerequisites
	686	a. topic-discovery.js + router routes
	687	b. keet_topic_discovery.gd + GroupBar search
	688	c. Bridge topic announcements
	689
	690	2. Layer 1 (room metadata) — needs room config write access
	691	a. Router: write registries to room config on register
	692	b. Router: suggestFromRoom route
	693	c. keet_discovery.gd: check_room_registries + suggestions UI
	694
	695	3. Layer 2 (MI clustering) — needs rooms + members data
	696	a. mi-engine.js (MI computation + clustering)
	697	b. Router: graph.* routes
	698	c. keet_graph.gd + GraphView.gd
	699	d. GroupBar: suggestion icons
	700
	701	4. Layer 3 (registry gossip) — needs Layer 1 data in room configs
	702	a. Router: discovery.related
	703	b. keet_discovery.gd: get_related
	704	c. GroupSettings: related groups section
	705
	706	5. Layer 4 (collaborative clusters) — needs Layer 2 clusters
	707	a. mi-engine.js: fingerprint + similarity
	708	b. Router: graph.publishFingerprint, findSimilarClusters
	709	c. keet_graph.gd: publish + find similar
	710	d. GroupBar: merge suggestion UI
	711	```
	712
	713	Each layer can be deployed independently. Layer 0 alone gives bootstrap-free discovery. Layers 1-4 add progressive intelligence.
	714
	715	## Verification
	716
	717	- [ ] Layer 0: Search "hypercore" → find rooms announced on that topic
	718	- [ ] Layer 0: Bridge-irc announces `#ai-town` → searchable via topic
	719	- [ ] Layer 1: Join room → "This room is in 'Hypercore Dev'. Follow it?" prompt appears
	720	- [ ] Layer 1: Room config shows `registries` and `bridges` arrays
	721	- [ ] Layer 2: `graph.compute` returns clusters with MI values
	722	- [ ] Layer 2: GroupBar shows suggestion icon for uncovered cluster
	723	- [ ] Layer 2: GraphView renders force-directed layout, click node → enter room
	724	- [ ] Layer 3: GroupSettings shows "Related Groups" for registries with shared rooms
	725	- [ ] Layer 4: Publish fingerprint → member in shared room sees "similar cluster" match
	726	- [ ] Cross-protocol: IRC-bridged rooms cluster with Keet rooms by shared members

docs/AUTO-DISCOVERY.md +306 -0

…	…	@@ -0,0 +1,306 @@
	1	# Auto-Discovery: From Hardcoded Registries to Emergent Groups
	2
	3	How discovery groups can create themselves through layered, emergent mechanisms — replacing manual registry management with progressive auto-discovery that works within and across protocols.
	4
	5	## The Problem with the Current Model
	6
	7	Today, discovery is fully manual and hardcoded at three levels:
	8
	9	```
	10	Layer 1: DEFAULT_DISCOVERY_KEY Hardcoded in keet-daemon source code
	11	"Global" registry Everyone starts here. One Autobase. One key.
	12
	13	Layer 2: discovery-keys.json User config file
	14	Additional registries Added manually via addDiscoveryRegistry(key)
	15	Must know the 64-char hex key to join
	16
	17	Layer 3: GroupBar UI User clicks "+" in godot-keet
	18	Create/Join Creates new Autobase or joins by pasted key
	19	```
	20
	21	Every group exists because a human explicitly created it and shared its key. This is the tree model dressed up as a graph — each registry is a silo, you join them one by one like Discord servers. There's no way for the system to:
	22
	23	- Suggest registries you don't know about but should
	24	- Detect that your rooms form a natural group that has no registry
	25	- Propagate discovery through room membership or member overlap
	26	- Bootstrap without a hardcoded key
	27
	28	## Layered Auto-Discovery
	29
	30	Instead of one mechanism, five layers that build on each other. Each layer adds intelligence without removing privacy. Each works independently. Together they replace manual discovery with emergent discovery.
	31
	32	### Layer 0: Topic-Based DHT Discovery (No Registry Needed)
	33
	34	The most radical change — no hardcoded bootstrap key. Rooms advertise on topic-based swarm keys. Anyone can search any topic.
	35
	36	```
	37	Room "Hypercore Dev Chat" announces on:
	38	DHT topic = hash("discovery:hypercore")
	39	DHT topic = hash("discovery:p2p")
	40	DHT topic = hash("discovery:development")
	41
	42	Your client searches for:
	43	DHT topic = hash("discovery:hypercore")
	44	→ discovers all rooms that tagged themselves with "hypercore"
	45	```
	46
	47	How it works:
	48	- Rooms opt into topic-based discovery by announcing on computed DHT topics
	49	- Topics are human-readable strings hashed to DHT keys
	50	- Anyone can search any topic — no account, no key, no bootstrap
	51	- Results are ephemeral (only shows rooms with active seeders)
	52	- Room metadata (title, description, member count) is shared via the topic swarm
	53
	54	What this replaces: The `DEFAULT_DISCOVERY_KEY`. Instead of one global registry that everyone must know, you have infinite topic namespaces that anyone can discover.
	55
	56	What this doesn't provide: Curation, moderation, governance. Anyone can tag any room with any topic. This is pure protocol-level discovery — useful for bootstrapping, not for community governance.
	57
	58	Privacy: Rooms opt in to topic discovery. If you don't announce on any topic, you're invisible. Announcing reveals room existence and metadata, but not membership or content.
	59
	60	Cross-protocol relevance: IRC channels, Discord servers, Matrix rooms can ALL announce on the same DHT topics. A bridge room that exists on both Keet and IRC announces on `hash("discovery:irc:libera:#ai-town")` AND `hash("discovery:ai")`. Discovery becomes protocol-agnostic.
	61
	62	### Layer 1: Room-Carried Registry Metadata
	63
	64	When you join a room, the invite carries which registries the room belongs to. Your client sees this and offers to follow those registries.
	65
	66	```
	67	Current invite flow:
	68	invite token → join room → done
	69
	70	Extended flow:
	71	invite token → join room
	72	→ room config includes: registries: ["c5facb91...", "a3de8f72..."]
	73	→ client: "This room is in 'Hypercore Dev' group. Follow it?"
	74	→ user: yes/no
	75	```
	76
	77	How it works:
	78	- When `discovery.register()` adds a room, the room's config gets a `registries` array
	79	- When you join and call `room.config`, you see which registries it belongs to
	80	- Your client auto-suggests following those registries
	81	- No new Autobases needed — just metadata propagation through existing room configs
	82	- Registries are fetched lazily when you choose to follow
	83
	84	Privacy: You already know the room exists (you joined it). Learning which registries it's in reveals group structure, but only for rooms you've already joined. Same inherent leakage level as seeing member overlap.
	85
	86	Cross-protocol relevance: When a bridge relays a room invite from IRC or Matrix, the bridge can attach registry metadata. "This IRC channel is listed in the 'Libera Network' group." Cross-platform room discovery piggybacks on existing invite flows.
	87
	88	### Layer 2: MI-Based Local Clustering
	89
	90	The social graph navigation pipeline from [SOCIAL-GRAPH-NAVIGATION.md](SOCIAL-GRAPH-NAVIGATION.md), running as a background process. When it detects a high-MI cluster, it offers to create a registry.
	91
	92	```
	93	Background:
	94	1. Compute MI between all your rooms (local data only)
	95	2. Detect clusters (rooms with MI > threshold)
	96	3. Check if cluster matches any existing registry you follow
	97	4. No match → surface as suggested group in GroupBar
	98
	99	UI:
	100	GroupBar shows a dimmed "?" icon with tooltip:
	101	"6 rooms with 80% member overlap — create a group?"
	102	→ user names it → creates local or shared registry
	103	```
	104
	105	How it works:
	106	- MI computation runs periodically from your room/member data
	107	- Clustering algorithm (same as language pipeline — MST + connector profiles)
	108	- Detected clusters compared against existing registries
	109	- Unmatched clusters appear as "suggested groups" in the UI
	110	- User decides: keep local (personal folder), publish (shared Autobase), or dismiss
	111
	112	This is the "emergence needs midwives, not managers" principle. The system detects the supercooled liquid (high-MI cluster). The human nucleates (names the group). The system surfaces the opportunity.
	113
	114	Privacy: Pure local computation. Your room lists and member data never leave your device. The MI values, clusters, and suggestions exist only in your local state.
	115
	116	Cross-protocol relevance: MI computation includes bridged rooms. If you're in an IRC-bridged room and a Keet room and they share members (through the bridge), they cluster together. Cross-protocol communities are detected naturally.
	117
	118	### Layer 3: Gossip-Based Registry Discovery
	119
	120	Registries themselves form a graph. When you follow Registry A, you discover registries that share rooms with A.
	121
	122	```
	123	Registry A has rooms: {1, 2, 3, 4, 5}
	124	Registry B has rooms: {4, 5, 6, 7, 8}
	125	Rooms 4 and 5 are in both → A and B are connected
	126
	127	You follow A → browse rooms → room 4 shows "Also in: Hypercore Dev (B)"
	128	→ "Follow 'Hypercore Dev'?"
	129	```
	130
	131	How it works:
	132	- Each registry's room list is public (that's the point of discovery)
	133	- Room metadata includes which other registries the room belongs to (Layer 1 data)
	134	- When browsing a registry, your client sees cross-registry links
	135	- Registries with high overlap to ones you follow are suggested
	136	- "People who follow 'P2P Apps' also follow 'Bare Runtime'" (similar to Amazon recommendations, but computed locally from public registry data)
	137
	138	This is Level 2 connector theory — MI between communities based on shared rooms. Registries with high MI to your existing registries are natural suggestions.
	139
	140	Privacy: Registry room lists are public by design (that's what discovery means). The cross-registry links are computed from this public data. No membership data is revealed.
	141
	142	Cross-protocol relevance: A bridge-specific registry ("IRC Communities") can share rooms with a topic-specific registry ("Hypercore Dev"). The gossip-based discovery surfaces cross-protocol registries naturally.
	143
	144	### Layer 4: Collaborative Auto-Registry
	145
	146	When multiple users independently detect the same cluster, the system suggests creating a shared registry.
	147
	148	```
	149	Alice detects cluster {A, B, C, D, E} → local group "Dev Rooms"
	150	Bob detects cluster {A, B, C, D, F} → local group "Dev Stuff"
	151	Carol detects cluster {B, C, D, E, G} → local group "Core Dev"
	152
	153	All three overlap significantly.
	154	When Alice, Bob, Carol are in room C together:
	155	"3 people have similar local groups. Create shared group?"
	156	→ collaborative naming → shared Autobase registry
	157	```
	158
	159	How it works:
	160	- Each user's local registry (from Layer 2) can publish a fingerprint — a hash of sorted room keys
	161	- Fingerprints are shared in rooms (attached to member profile, opt-in)
	162	- When fingerprints from different users are similar enough (Jaccard similarity > threshold), the system suggests merging
	163	- Merge = one user creates a shared Autobase, others join as writers
	164	- The resulting registry is a community-created group that no single person authored
	165
	166	This is emergent governance. The community structure (shared cluster detection) creates the conditions for shared governance (merged registry). The registry didn't exist until multiple people independently recognized the same pattern.
	167
	168	Privacy: Fingerprints are opt-in and only reveal which rooms you've clustered (not which rooms you're in — just which rooms are in your local group). Publishing a fingerprint is a conscious choice.
	169
	170	Cross-protocol relevance: Fingerprints can include bridged rooms from any protocol. Cross-protocol community detection happens naturally when bridge users share cluster fingerprints.
	171
	172	## How Layers Interact
	173
	174	```
	175	Layer 0: "I searched for 'hypercore' and found these rooms"
	176	→ You join some rooms
	177	→ Now you have local data for Layer 2
	178
	179	Layer 1: "This room is in 'Hypercore Dev' group"
	180	→ You follow the group
	181	→ Now you have registry data for Layer 3
	182
	183	Layer 2: "Your rooms {A, B, C, D} form a natural cluster"
	184	→ You create a local group
	185	→ Now you have a fingerprint for Layer 4
	186
	187	Layer 3: "'Hypercore Dev' shares rooms with 'Bare Runtime'"
	188	→ You follow 'Bare Runtime'
	189	→ Now you have more rooms for Layer 2 to cluster
	190
	191	Layer 4: "Alice and Bob have the same cluster. Merge into shared group?"
	192	→ Shared registry created
	193	→ Now discoverable via Layer 3
	194	→ Rooms carry its metadata via Layer 1
	195	```
	196
	197	Each layer feeds into the others. The system bootstraps from nothing (Layer 0 needs no prior state) and builds toward community governance (Layer 4 creates shared Autobases with admin/mod/writer roles).
	198
	199	## Cross-Protocol Discovery
	200
	201	The layered approach works across bridge boundaries because each layer is protocol-agnostic at its core:
	202
	203	### Topic DHT as Universal Namespace
	204
	205	```
	206	Keet room announces: hash("discovery:ai-town")
	207	IRC channel announces: hash("discovery:irc:libera:#ai-town")
	208	Both announce: hash("discovery:ai")
	209
	210	Search "ai" → find both Keet rooms AND IRC channels (via bridges)
	211	```
	212
	213	The bridge announces on behalf of the bridged platform. A bridge-irc instance that connects `#ai-town` to a Keet room announces on both the Keet-style topic and the IRC-style topic. Users on either side can discover the bridged room.
	214
	215	### Room Metadata Carries Protocol Info
	216
	217	```
	218	Room config:
	219	registries: ["c5facb91..."]
	220	bridges: [
	221	{ platform: "irc", server: "libera", channel: "#ai-town" },
	222	{ platform: "discord", guild: "12345", channel: "general" }
	223	]
	224	```
	225
	226	When you join a room, you learn not just which registries it's in, but which bridges are active. This lets discovery flow across protocols — join a Keet room, discover it's bridged to IRC, discover the IRC-specific registry.
	227
	228	### Cross-Protocol MI Clustering
	229
	230	Members who appear through bridges still count for MI computation:
	231
	232	```
	233	Room A (Keet): members {alice, bob, carol, [irc]dave, [irc]eve}
	234	Room B (Keet): members {alice, bob, [irc]dave, frank}
	235	Room C (IRC via bridge): members {[irc]dave, [irc]eve, [irc]grace}
	236
	237	MI(A, B) is high (3 shared members including bridged dave)
	238	MI(A, C) is moderate (2 shared bridged members)
	239	```
	240
	241	Bridged members contribute to co-occurrence just like native members. Communities that span protocols cluster naturally.
	242
	243	### Registry Discovery Across Bridges
	244
	245	A bridge can carry registry metadata from the foreign platform:
	246
	247	```
	248	IRC channel #hypercore has topic: "Part of Hypercore Dev network"
	249	Bridge translates: registry metadata for "Hypercore Dev" group
	250
	251	Matrix room !abc:matrix.org is in Matrix space "P2P Development"
	252	Bridge translates: registry metadata for "P2P Development" group
	253	```
	254
	255	Foreign platform grouping structures (IRC networks, Discord servers, Matrix spaces) become discoverable as registries through bridge metadata translation. The bridge is a sheaf morphism — translating structure, not just messages.
	256
	257	## The Bootstrap Problem Solved
	258
	259	Current bootstrap: you must know `DEFAULT_DISCOVERY_KEY` (hardcoded) to find anything.
	260
	261	With layered auto-discovery, bootstrap becomes:
	262
	263	```
	264	1. Search a topic on the DHT (Layer 0) — no prior knowledge needed
	265	2. Find rooms, join one
	266	3. Room carries registry metadata (Layer 1) — discover groups
	267	4. Follow groups, browse rooms
	268	5. MI clustering detects patterns (Layer 2) — discover implicit groups
	269	6. Registry gossip suggests related groups (Layer 3) — discover neighbors
	270	7. Shared clustering merges into community groups (Layer 4) — governance emerges
	271	```
	272
	273	The hardcoded `DEFAULT_DISCOVERY_KEY` becomes unnecessary — it's just one possible entry point among many. A user who searches "p2p chat" on the DHT and joins a room they find there can bootstrap the entire discovery chain without ever knowing a hex key.
	274
	275	## Privacy Summary
	276
	277	\| Layer \| Data Used \| Computed Where \| Shared With \|
	278	\|-------\|-----------\|---------------\|-------------\|
	279	\| 0 \| Room metadata (title, topic) \| DHT (public) \| Anyone searching the topic \|
	280	\| 1 \| Room config (registry list) \| In-room (members only) \| Room members \|
	281	\| 2 \| Your rooms + members \| Local device only \| Nobody \|
	282	\| 3 \| Registry room lists \| Local device \| Nobody (public data, local computation) \|
	283	\| 4 \| Cluster fingerprints \| Local device \| Opt-in per room \|
	284
	285	No layer requires aggregating data across users. No layer exposes membership data to non-members. No layer creates a central registry of registries. The privacy architecture from [COMMUNITY-EMERGENCE.md](COMMUNITY-EMERGENCE.md) is maintained at every level.
	286
	287	## Design Principles
	288
	289	1. Each layer works independently. Deploy Layer 0 alone and you have topic-based discovery. Deploy Layer 2 alone and you have local clustering. No layer depends on all others being active.
	290
	291	2. Layers feed each other. More rooms (from Layer 0/1/3) give Layer 2 better clusters. Better clusters (from Layer 2) create more registries (from Layer 4) for Layer 3 to gossip about.
	292
	293	3. No bootstrap required. Layer 0 requires no prior state — just a search query and a DHT.
	294
	295	4. Human nucleation preserved. Layers 0-3 surface opportunities. Layer 4 suggests merges. But naming a group, publishing a registry, and establishing governance are always human decisions.
	296
	297	5. Protocol-agnostic at every layer. Topic DHT works for any platform. Room metadata carries cross-protocol info. MI computation includes bridged members. Registry gossip crosses protocol boundaries.
	298
	299	6. Privacy by architecture. Local computation, opt-in sharing, no aggregation. The system physically cannot surveil because there's no central point to surveil from.
	300
	301	## Connection to Other Documents
	302
	303	- [SOCIAL-GRAPH-NAVIGATION.md](SOCIAL-GRAPH-NAVIGATION.md) — The MI/MST/clustering math that powers Layer 2
	304	- [COMMUNITY-EMERGENCE.md](COMMUNITY-EMERGENCE.md) — Phase transitions, governance, privacy principles
	305	- [ROADMAP.md](ROADMAP.md) — Implementation phases (Layer 2 = Phase 7, bridges = Phase 0-3)
	306	- [ARCHITECTURE.md](ARCHITECTURE.md) — Bridge architecture, fracture principle

docs/README.md +2 -0

…	…	@@ -14,6 +14,8 @@ P2P social layer built on Hypercore Protocol and Keet. A family of interoperable
14	14	\| [SOCIAL-GRAPH-NAVIGATION.md](SOCIAL-GRAPH-NAVIGATION.md) \| Graph-based navigation, sheaf theory, parsing social structure like language \|
15	15	\| [COMMUNITY-EMERGENCE.md](COMMUNITY-EMERGENCE.md) \| How communities form, govern themselves, privacy architecture, Blacksky case study \|
16	16	\| [ROADMAP.md](ROADMAP.md) \| Evolutionary path from godot-keet to hyper-social — 8 phases, what exists, what to build \|
	17	\| [AUTO-DISCOVERY.md](AUTO-DISCOVERY.md) \| Layered auto-discovery design — 5 layers replacing hardcoded registries, privacy analysis \|
	18	\| [AUTO-DISCOVERY-SPEC.md](AUTO-DISCOVERY-SPEC.md) \| Implementation spec — code changes for keet-daemon, godot-keet, bridges (~1,400 LOC) \|
17	19
18	20	## Quick Summary
19	21

hyper-social commits / 1445701

Changed files (3) +1034 -0