OverviewFilesCommitsIssuesDocsReleasesPeers

hyperswarm-peer commits / 62db3a6

swarmd: the content plane — 100 MB by hash, 996 bytes over the socket
hyper-games #73 NEXT item 4. CONTENT_HASH / SERVE / FETCH / CANCEL were in the schema and answered by nobody. They are real now, on liboverlay, which was already complete and conformance-tested against the JS reference — what was missing was the wiring, and swarmd is the right place for it because swarmd owns the peer lifecycle. ⛔ IT WIRES THE PEERS ITSELF rather than embedding HyperOverlay. UdxTransport::set_mux_callbacks holds ONE pair of callbacks and whoever sets it last owns the peer lifecycle, so a second component that wants per-peer overlay channels cannot simply be dropped in beside the daemon. Content::add_peer is called from the daemon's own mux callback. Threading follows the rule the rest of the daemon follows and the one src/overlay_node.h arrived at independently: the UDX thread owns the Protocol outright — every command marshals in with run_on_loop, every result comes back through a bounded queue, and the main thread never touches liboverlay, not even to read a counter. PROGRESS is dropped under pressure; DONE and FAILED never are, because losing one strands a client waiting for a message that never comes. Measured, on the PUBLIC DHT, two daemons, a swarmd-probe on each: 100 MB asset ... fetched 104857600 bytes in 7.0s; the socket carried 996 bytes. That second number is §6 stated as a measurement rather than an intention, and test_content_e2e.py fails if it ever exceeds 256 KB. Same run at 100 MB on a local testnet: 6.0s, 963 bytes. It keeps the INCONCLUSIVE baseline the channel gate uses — daemons that never met say nothing about content. ⛔ Two bugs in swarmd-probe, both mine, both from this session: * request ids: the send-result branch matched "id >= 4", which swept up the content replies and printed "sent (accepted): (null)" for a fetch. Sends now have their own range from 100 — an id is how a reply finds the call that asked for it, and a range that overlaps is not an id scheme. * it now counts every byte the daemon writes to it, which is where the 996 comes from. A claim about a socket needs a number off that socket. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
62db3a65938d165732576ac515cc7e7a2b20e0f4
by plankatron <fleeky@gmail.com> · 2026-09-10T19:17:59.000Z

Changed files (13) +804 -16

swarmd/README.md +20 -4
……@@ -40,11 +40,13 @@ encodings the same way, with `schema/build.js` and the resulting
4040 schema/ generated wire, committed 3316 lines
4141 uds_server local socket + per-client buffering 7/7 (D2, D3)
4242 control generated RPC dispatch on the socket 10/10
43 send refuses rather than lying about a drop 7/7
4344 daemon UdxTransport + queue + control plane runs
44 channels Protomux channel per peer, SEND/on-message
45 channels Protomux channel per peer, SEND/on-message D1/D4 PASS
46 content liboverlay: hash/serve/fetch/cancel D6 PASS
4547 tools/ swarmd-probe — the smallest real client
4648
47 ./swarmd --socket /tmp/s.sock # public DHT
49 ./swarmd --socket /tmp/s.sock --storage ~/.swarmd # public DHT
4850 [swarmd] listening on /tmp/s.sock
4951 [swarmd] dht=connecting peers=0 local=0
5052 [swarmd] dht=up peers=0 local=0
……@@ -58,5 +60,19 @@ swarmd-probe on each, one frame across a Protomux channel:
5860The gate still exits INCONCLUSIVE rather than FAIL when the daemons never meet,
5961because a message result from a swarm that never meshed says nothing.
6062
61⏳ Content is not built. See `../docs/SWARMD-DESIGN.md` §9 for the order and §8
62for the remaining gates.
63✅ The content plane is built, on liboverlay. Bytes move peer-to-peer; the local
64socket carries a 32-byte hash and a path (§6):
65
66 python3 -B test/test_content_e2e.py --mb 100 --seconds 300 # public DHT
67
68 100 MB asset ... fetched 104857600 bytes in 7.0s;
69 the socket carried 996 bytes.
70
71That number IS §6. If content ever started travelling over the socket it would be
72a hundred million, and the gate fails at 256 KB.
73
74`--storage <dir>` turns the content plane on. Without it the daemon refuses
75content commands rather than accepting a fetch it has nowhere to put.
76
77See `../docs/SWARMD-DESIGN.md` §9 for the order and §8 for the remaining gates
78(D5 and D7 are still unwritten).
swarmd/build.sh +4 -2
……@@ -7,7 +7,7 @@ cd "$(dirname "$0")"
77
88KB=/storage/projects/code/hypercore-kb/holepunchto/repos
99PEER=..
10INC="-Isrc -Ischema -I$PEER/src -I$PEER/thirdparty/libudx/include -I$KB/tier5/libcompact/include -I$KB/tier4/libutf/include -I$KB/tier4/librpc/include"
10INC="-Isrc -Ischema -I$PEER/src -I$PEER/liboverlay/include -I$PEER/thirdparty/libudx/include -I$KB/tier5/libcompact/include -I$KB/tier4/libutf/include -I$KB/tier4/librpc/include"
1111OBJ=${OBJ:-/tmp/swarmd-obj}
1212mkdir -p "$OBJ"
1313
……@@ -40,6 +40,7 @@ gcc -std=c99 -O2 $INC -c -o "$OBJ/swarmd_hrpc.o" schema/swarmd_hrpc.c
4040echo "Building daemon sources..."
4141g++ -std=c++17 -O2 $INC -c -o "$OBJ/uds_server.o" src/uds_server.cpp
4242g++ -std=c++17 -O2 $INC -c -o "$OBJ/control.o" src/control.cpp
43g++ -std=c++17 -O2 $INC -c -o "$OBJ/content.o" src/content.cpp
4344
4445# The transport. Reuses hyperswarm-peer's object cache when test/build_tests.sh
4546# has populated it; otherwise builds what swarmd needs.
……@@ -76,7 +77,7 @@ done
7677echo "Building daemon..."
7778g++ -std=c++17 -O2 $INC -c -o "$OBJ/daemon.o" src/daemon.cpp
7879g++ -std=c++17 -O2 $INC -o swarmd src/main.cpp \
79 "$OBJ/daemon.o" "$OBJ/uds_server.o" "$OBJ/control.o" \
80 "$OBJ/daemon.o" "$OBJ/uds_server.o" "$OBJ/control.o" "$OBJ/content.o" \
8081 "$OBJ/swarmd_schema.o" "$OBJ/swarmd_hrpc.o" $CLIBS $TLIBS \
8182 -lsodium -luv -lpthread
8283
……@@ -91,6 +92,7 @@ g++ -std=c++17 -O2 $INC -o test/test_control \
9192# Drives the REAL Daemon (transport and all), so it links what swarmd links.
9293g++ -std=c++17 -O2 $INC -o test/test_send_truth \
9394 test/test_send_truth.cpp "$OBJ/daemon.o" "$OBJ/uds_server.o" "$OBJ/control.o" \
95 "$OBJ/content.o" \
9496 "$OBJ/swarmd_schema.o" "$OBJ/swarmd_hrpc.o" $CLIBS $TLIBS \
9597 -lsodium -luv -lpthread
9698
swarmd/src/content.cpp +230 -0
……@@ -0,0 +1,230 @@
1#include "content.h"
2
3#include <algorithm>
4#include <chrono>
5#include <cstdio>
6
7namespace swarmd {
8
9// The single place the library's notion of time comes from. liboverlay never
10// calls now() — it is handed the time — so this is also the single place to fake
11// if it ever needs faking.
12static uint64_t now_ms() {
13 using namespace std::chrono;
14 return (uint64_t)duration_cast<milliseconds>(steady_clock::now().time_since_epoch()).count();
15}
16
17// How often a running fetch reports. Fast enough for a progress bar, slow enough
18// that a 100 MB transfer does not spend the socket on telling us about itself.
19static constexpr uint64_t PROGRESS_INTERVAL_MS = 250;
20
21bool Content::open(const std::string &storage_dir, UdxTransport *transport) {
22 if (open_) return true;
23 storage_dir_ = storage_dir;
24 t_ = transport;
25
26 index_.reset(new hyperoverlay::index::FileIndex());
27 if (!index_->open(storage_dir_ + "/overlay-index.log")) {
28 fprintf(stderr, "[swarmd] content: cannot open the index under %s\n",
29 storage_dir_.c_str());
30 index_.reset();
31 return false;
32 }
33 if (index_->truncated_bytes()) {
34 // A daemon gets killed, not shut down. A torn tail is expected; it is
35 // reported rather than hidden, because "entries vanished" is otherwise a
36 // mystery months later.
37 fprintf(stderr, "[swarmd] content: discarded %llu bytes of torn index tail "
38 "from an earlier kill\n",
39 (unsigned long long)index_->truncated_bytes());
40 }
41
42 // What this daemon can serve is whatever a client registered, resolved
43 // through the index rather than a map that dies with the process.
44 proto_.reset(new hyperoverlay::Protocol(index_.get(),
45 [this](const std::string &hash, hyperoverlay::ContentLocation &out) {
46 const hyperoverlay::index::FileMeta *m = index_->get_file(
47 hyperoverlay::index::FileIndex::content_register_key(hash));
48 if (!m || m->linkname.empty()) return false;
49 out.path = m->linkname;
50 out.size = m->size;
51 return true;
52 }));
53
54 proto_->on_complete = [this](const std::string &hash, const std::string &path) {
55 Event e; e.kind = Event::DONE; e.hash = hash; e.detail = path;
56 _push(e);
57 };
58 proto_->on_failed = [this](const std::string &hash, const std::string &why) {
59 Event e; e.kind = Event::FAILED; e.hash = hash; e.detail = why;
60 _push(e);
61 };
62
63 proto_->tick(now_ms());
64 open_ = true;
65 return true;
66}
67
68void Content::close() {
69 if (!open_) return;
70 open_ = false;
71 // The channels belong to the UDX loop and must be torn down there. If the
72 // transport is already stopped there is no loop to marshal to, and nothing
73 // else is touching them either.
74 if (t_) {
75 t_->run_on_loop([this]() {
76 for (auto &kv : peers_) {
77 if (proto_) proto_->remove_peer(kv.second);
78 delete kv.second;
79 }
80 peers_.clear();
81 });
82 }
83 proto_.reset();
84 if (index_) index_->close();
85 index_.reset();
86 t_ = nullptr;
87}
88
89std::string Content::hash_of_file(const std::string &path) {
90 uint64_t size = 0;
91 if (!hyperoverlay::io::file_size(path, size)) return std::string();
92 hyperoverlay::io::File f;
93 if (!f.open_read(path)) return std::string();
94 hyperoverlay::cdc::StreamingHasher hasher;
95 hasher.init(size);
96 std::vector<uint8_t> buf(1u << 20);
97 uint64_t off = 0;
98 while (off < size) {
99 const size_t want = (size_t)std::min<uint64_t>(buf.size(), size - off);
100 if (!f.read_at(off, buf.data(), want)) return std::string();
101 hasher.update(buf.data(), want);
102 off += want;
103 }
104 return hasher.digest();
105}
106
107void Content::serve(const std::string &hash, const std::string &path) {
108 if (!open_ || !t_) return;
109 t_->run_on_loop([this, hash, path]() {
110 uint64_t sz = 0;
111 if (!hyperoverlay::io::file_size(path, sz)) {
112 Event e; e.kind = Event::FAILED; e.hash = hash;
113 e.detail = "serve: no such file: " + path;
114 _push(e);
115 return;
116 }
117 hyperoverlay::index::FileMeta m;
118 m.contentHash = hash;
119 m.linkname = path; // a LOCATOR: the bytes are never read whole
120 m.size = sz;
121 index_->put_file(hyperoverlay::index::FileIndex::content_register_key(hash), m);
122
123 // Precompute the chunk map HERE — when the file is registered — rather
124 // than when a peer first asks. Same work either way, but this way it is
125 // not sitting in front of the first byte of a transfer.
126 if (!index_->has_chunk_map_by_hash(hash)) {
127 const std::vector<hyperoverlay::index::ChunkRef> map =
128 hyperoverlay::Protocol::build_chunk_map(path, sz);
129 if (!map.empty()) index_->put_chunk_map_by_hash(hash, map);
130 }
131 });
132}
133
134void Content::fetch(const std::string &hash, const std::string &dest_path) {
135 if (!open_ || !t_) return;
136 t_->run_on_loop([this, hash, dest_path]() { proto_->request(hash, dest_path); });
137}
138
139void Content::cancel(const std::string &hash, bool discard) {
140 if (!open_ || !t_) return;
141 t_->run_on_loop([this, hash, discard]() {
142 if (proto_->cancel(hash, discard)) return;
143 Event e; e.kind = Event::FAILED; e.hash = hash;
144 e.detail = "cancel: no such fetch";
145 _push(e);
146 });
147}
148
149void Content::add_peer(int32_t peer_id, Protomux *mux) {
150 // UDX THREAD ONLY.
151 if (!open_ || !proto_ || !mux) return;
152 if (peers_.count(peer_id)) return;
153
154 // The pool hook goes in through the CONSTRUCTOR and covers every way a
155 // channel can open — already paired when we attach, paired later by the
156 // remote, or rebuilt after the remote's overlay came up second. Assigning it
157 // on the next line misses the first of those, because the channel can be
158 // fully open before the constructor returns.
159 hyperswarm::OverlayTransport *ot = new hyperswarm::OverlayTransport(
160 mux, proto_.get(), std::to_string(peer_id),
161 [this](hyperswarm::OverlayTransport *t) {
162 if (proto_) proto_->add_peer(t);
163 });
164 if (ot->slot_mismatch()) {
165 // Protomux handed back a slot that is not the registration index, so
166 // every frame on this connection would land in the wrong handler. Refuse
167 // loudly rather than move misrouted bytes.
168 fprintf(stderr, "[swarmd] content: slot-order mismatch on peer %d — "
169 "refusing the overlay channel\n", (int)peer_id);
170 proto_->remove_peer(ot); // it may already have joined from the ctor
171 delete ot;
172 return;
173 }
174 peers_[peer_id] = ot;
175}
176
177void Content::remove_peer(int32_t peer_id) {
178 // UDX THREAD ONLY.
179 auto it = peers_.find(peer_id);
180 if (it == peers_.end()) return;
181 if (proto_) proto_->remove_peer(it->second); // hands its chunks back
182 delete it->second;
183 peers_.erase(it);
184}
185
186void Content::_push(const Event &e) {
187 std::lock_guard<std::mutex> lk(m_);
188 if (events_.size() >= MAX_EVENTS) {
189 if (e.kind == Event::PROGRESS) { dropped_progress_++; return; }
190 for (auto it = events_.begin(); it != events_.end(); ++it) {
191 if (it->kind == Event::PROGRESS) { events_.erase(it); dropped_progress_++; break; }
192 }
193 }
194 events_.push_back(e);
195}
196
197void Content::poll(std::vector<Event> &out) {
198 if (!open_ || !t_) return;
199
200 const uint64_t t = now_ms();
201 const bool want_progress = (t - last_progress_ms_) >= PROGRESS_INTERVAL_MS;
202 if (want_progress) last_progress_ms_ = t;
203
204 // Both of these read Protocol state, so both go to the thread that owns it.
205 t_->run_on_loop([this, t, want_progress]() {
206 if (!proto_) return;
207 proto_->tick(t);
208 if (!want_progress) return;
209 for (const std::string &hash : proto_->active_downloads()) {
210 uint64_t bytes = 0;
211 size_t got = 0, total = 0;
212 if (!proto_->progress(hash, bytes, got, total)) continue;
213 Event e;
214 e.kind = Event::PROGRESS;
215 e.hash = hash;
216 e.received = (uint64_t)got;
217 e.total = (uint64_t)total;
218 _push(e);
219 }
220 });
221
222 std::deque<Event> batch;
223 {
224 std::lock_guard<std::mutex> lk(m_);
225 batch.swap(events_);
226 }
227 out.assign(batch.begin(), batch.end());
228}
229
230} // namespace swarmd
swarmd/src/content.h +101 -0
……@@ -0,0 +1,101 @@
1#ifndef SWARMD_CONTENT_H
2#define SWARMD_CONTENT_H
3
4// The content plane: fetch and serve content-addressed files over liboverlay.
5//
6// ⛔ BULK BYTES NEVER CROSS THE LOCAL SOCKET (docs/SWARMD-DESIGN.md §6). A .res
7// is 108 MB. The socket carries a 32-byte hash and a path; the overlay writes
8// the file and the client opens it. That is also what makes a fetch resumable
9// across a daemon restart, because the partial and its journal are on disk.
10//
11// THREADING, which is the same rule the rest of the daemon follows and the same
12// one src/overlay_node.h arrived at for Godot:
13//
14// UDX thread owns the Protocol OUTRIGHT. Protomux dispatches inbound overlay
15// frames there, and protomux/udx handles belong to that loop, so
16// both halves must live on it or they race. Every command from a
17// client is marshalled in with run_on_loop.
18// main thread drains an event queue and writes to client sockets. It never
19// touches the Protocol — not even to read a counter.
20//
21// The one exception is hash_of_file(), which touches no overlay state at all: it
22// streams a file through the chunker. It runs where it is called.
23
24#include <cstdint>
25#include <deque>
26#include <functional>
27#include <map>
28#include <memory>
29#include <mutex>
30#include <string>
31#include <vector>
32
33#include "../../src/overlay_transport.h"
34#include "../../src/udx_transport.h"
35#include "overlay/file_index.h"
36#include "overlay/protocol.h"
37
38namespace swarmd {
39
40class Content {
41public:
42 struct Event {
43 enum Kind { DONE, FAILED, PROGRESS } kind;
44 std::string hash;
45 std::string detail; // path for DONE, reason for FAILED
46 uint64_t received = 0;
47 uint64_t total = 0;
48 };
49
50 // `storage_dir` holds the index log, the journals and the partials. It must
51 // survive restarts — that is what makes a resumed download cheap.
52 bool open(const std::string &storage_dir, UdxTransport *transport);
53 void close();
54 bool is_open() const { return open_; }
55
56 // The content hash of a file, computed the way the overlay computes it:
57 // blake2b over the hypercore leaf construction, not a plain digest. Empty on
58 // failure. Streams, so a file larger than RAM is not a different path.
59 static std::string hash_of_file(const std::string &path);
60
61 // All of these return immediately; results arrive as events.
62 void serve(const std::string &hash, const std::string &path);
63 void fetch(const std::string &hash, const std::string &dest_path);
64 void cancel(const std::string &hash, bool discard);
65
66 // UDX THREAD ONLY. Called from the daemon's mux callbacks — swarmd owns the
67 // peer lifecycle, so it builds the overlay channels itself rather than
68 // letting a second component take set_mux_callbacks.
69 void add_peer(int32_t peer_id, Protomux *mux);
70 void remove_peer(int32_t peer_id);
71
72 // MAIN THREAD. Advances liboverlay's clock (it never calls now() itself, so
73 // without this no watchdog fires and the library is correct and inert),
74 // samples progress, and hands back everything that has happened.
75 void poll(std::vector<Event> &out);
76
77private:
78 void _push(const Event &e);
79
80 std::unique_ptr<hyperoverlay::index::FileIndex> index_;
81 std::unique_ptr<hyperoverlay::Protocol> proto_;
82 UdxTransport *t_ = nullptr;
83 bool open_ = false;
84 std::string storage_dir_;
85
86 // UDX THREAD ONLY: one overlay channel per connection.
87 std::map<int32_t, hyperswarm::OverlayTransport *> peers_;
88
89 std::mutex m_;
90 std::deque<Event> events_;
91 // Bounded. A fetch that reports faster than the socket drains must drop
92 // PROGRESS — never a terminal event, because losing one strands a client
93 // waiting for a message that never comes.
94 static constexpr size_t MAX_EVENTS = 4096;
95 uint64_t dropped_progress_ = 0;
96 uint64_t last_progress_ms_ = 0;
97};
98
99} // namespace swarmd
100
101#endif
swarmd/src/control.cpp +46 -0
……@@ -94,6 +94,48 @@ static int on_send(void *vctx, const swarmd_send_request_t *req,
9494 return 0;
9595}
9696
97// ---- content -------------------------------------------------------------
98//
99// Nothing here moves bytes. CONTENT_HASH reads a file and answers with 32 bytes;
100// the other three are orders, answered with ok. The file itself never touches
101// this socket — docs/SWARMD-DESIGN.md §6.
102
103static int on_content_hash(void *vctx, const swarmd_path_t *req,
104 swarmd_hash_t *res, hrpc_error_t *) {
105 Ctx *ctx = (Ctx *)vctx;
106 memset(res->hash, 0, 32);
107 if (!ctx->backend->content_hash) return 0;
108 // ⛔ COPY OUT: the view points into the inbound frame buffer.
109 const std::string path((const char *)req->path.data, req->path.len);
110 ctx->backend->content_hash(path, res->hash);
111 return 0;
112}
113
114static int on_content_serve(void *vctx, const swarmd_content_serve_t *req,
115 swarmd_ok_t *res, hrpc_error_t *) {
116 Ctx *ctx = (Ctx *)vctx;
117 const std::string path((const char *)req->path.data, req->path.len);
118 res->ok = ctx->backend->content_serve ? ctx->backend->content_serve(req->hash, path) : false;
119 return 0;
120}
121
122static int on_content_fetch(void *vctx, const swarmd_content_fetch_t *req,
123 swarmd_ok_t *res, hrpc_error_t *) {
124 Ctx *ctx = (Ctx *)vctx;
125 const std::string dest((const char *)req->dest_path.data, req->dest_path.len);
126 res->ok = ctx->backend->content_fetch ? ctx->backend->content_fetch(req->hash, dest) : false;
127 return 0;
128}
129
130static int on_content_cancel(void *vctx, const swarmd_content_cancel_t *req,
131 swarmd_ok_t *res, hrpc_error_t *) {
132 Ctx *ctx = (Ctx *)vctx;
133 res->ok = ctx->backend->content_cancel
134 ? ctx->backend->content_cancel(req->hash, req->discard)
135 : false;
136 return 0;
137}
138
97139void Control::attach(UdsServer *uds, ControlBackend backend) {
98140 _uds = uds;
99141 _backend = std::move(backend);
……@@ -130,6 +172,10 @@ void Control::_on_frame(uint64_t client_id, const uint8_t *data, size_t len) {
130172 h.on_channel_open = on_channel_open;
131173 h.on_channel_close = on_channel_close;
132174 h.on_send = on_send;
175 h.on_content_hash = on_content_hash;
176 h.on_content_serve = on_content_serve;
177 h.on_content_fetch = on_content_fetch;
178 h.on_content_cancel = on_content_cancel;
133179
134180 uint8_t *reply = nullptr;
135181 size_t reply_len = 0;
swarmd/src/control.h +7 -0
……@@ -42,6 +42,13 @@ struct ControlBackend {
4242 std::function<bool(uint64_t handle)> channel_close;
4343 std::function<bool(uint64_t handle, uint32_t slot, uint64_t target,
4444 const uint8_t *data, size_t len, bool *writable)> send;
45
46 // Content. The hash is 32 raw bytes on the wire; liboverlay speaks hex, and
47 // the conversion belongs on the daemon side of this seam, not in the codec.
48 std::function<bool(const std::string &path, uint8_t out_hash[32])> content_hash;
49 std::function<bool(const uint8_t hash[32], const std::string &path)> content_serve;
50 std::function<bool(const uint8_t hash[32], const std::string &dest_path)> content_fetch;
51 std::function<bool(const uint8_t hash[32], bool discard)> content_cancel;
4552};
4653
4754class Control {
swarmd/src/daemon.cpp +91 -1
……@@ -6,7 +6,16 @@
66
77namespace swarmd {
88
9bool Daemon::start(const char *socket_path, const char *bootstrap) {
9// liboverlay names content in hex; the wire carries 32 raw bytes.
10static std::string _hex32(const uint8_t h[32]) {
11 static const char *d = "0123456789abcdef";
12 std::string out(64, '0');
13 for (int i = 0; i < 32; i++) { out[i * 2] = d[h[i] >> 4]; out[i * 2 + 1] = d[h[i] & 15]; }
14 return out;
15}
16
17bool Daemon::start(const char *socket_path, const char *bootstrap,
18 const char *storage_dir) {
1019 _socket_path = socket_path;
1120 if (!_uds.listen(socket_path)) return false;
1221
……@@ -114,6 +123,32 @@ bool Daemon::start(const char *socket_path, const char *bootstrap) {
114123 return true;
115124 };
116125
126 // ---- content ----
127 // hash_of_file touches no overlay state, so it answers on the calling thread.
128 // The other three are orders that cross to the UDX loop inside Content.
129 backend.content_hash = [this](const std::string &path, uint8_t out[32]) {
130 const std::string hex = Content::hash_of_file(path);
131 if (hex.size() != 64) return false;
132 for (int i = 0; i < 32; i++)
133 out[i] = (uint8_t)strtoul(hex.substr(i * 2, 2).c_str(), nullptr, 16);
134 return true;
135 };
136 backend.content_serve = [this](const uint8_t hash[32], const std::string &path) {
137 if (!_content.is_open()) return false;
138 _content.serve(_hex32(hash), path);
139 return true;
140 };
141 backend.content_fetch = [this](const uint8_t hash[32], const std::string &dest) {
142 if (!_content.is_open()) return false;
143 _content.fetch(_hex32(hash), dest);
144 return true;
145 };
146 backend.content_cancel = [this](const uint8_t hash[32], bool discard) {
147 if (!_content.is_open()) return false;
148 _content.cancel(_hex32(hash), discard);
149 return true;
150 };
151
117152 _control.attach(&_uds, backend);
118153
119154 // Peer lifecycle arrives on the UDX thread. Enqueue only.
……@@ -124,6 +159,10 @@ bool Daemon::start(const char *socket_path, const char *bootstrap) {
124159 // meshed must still reach peers that arrive later.
125160 _muxes[id] = mux;
126161 for (auto &kv : _channels) _open_on_peer(kv.second, id, mux);
162 // swarmd owns the peer lifecycle, so it builds the overlay
163 // channel itself: set_mux_callbacks holds ONE pair of callbacks
164 // and whoever sets it last owns them.
165 _content.add_peer(id, mux);
127166
128167 Event e;
129168 e.kind = Event::PEER_UP;
……@@ -136,6 +175,7 @@ bool Daemon::start(const char *socket_path, const char *bootstrap) {
136175 // UDX THREAD. The mux is going away; drop our references to its
137176 // channels rather than let them dangle.
138177 _muxes.erase(id);
178 _content.remove_peer(id);
139179 for (auto &kv : _channels) {
140180 kv.second.per_peer.erase(id);
141181 _publish_open(kv.first);
……@@ -159,6 +199,14 @@ bool Daemon::start(const char *socket_path, const char *bootstrap) {
159199 _t.start_dht(kp, nullptr, 0);
160200 }
161201
202 // AFTER start_dht: Content marshals onto the UDX loop, which has to exist.
203 // A daemon with no storage dir simply has no content plane; its commands are
204 // refused above rather than silently accepted.
205 if (storage_dir && *storage_dir) {
206 if (!_content.open(storage_dir, &_t))
207 fprintf(stderr, "[swarmd] content plane is OFF: %s is not usable\n", storage_dir);
208 }
209
162210 _started = true;
163211 return true;
164212}
……@@ -167,6 +215,7 @@ void Daemon::stop() {
167215 if (!_started) return;
168216 _started = false;
169217 _t.set_mux_callbacks(nullptr, nullptr);
218 _content.close();
170219 _t.stop();
171220 _uds.close_all();
172221}
……@@ -285,8 +334,49 @@ int Daemon::peers() const {
285334 return const_cast<UdxTransport &>(_t).get_swarm_status().peers_connected;
286335}
287336
337void Daemon::_drain_content() {
338 if (!_content.is_open()) return;
339 std::vector<Content::Event> batch;
340 _content.poll(batch);
341 for (const Content::Event &e : batch) {
342 uint8_t raw[32];
343 memset(raw, 0, 32);
344 for (int i = 0; i < 32 && (size_t)(i * 2 + 1) < e.hash.size(); i++)
345 raw[i] = (uint8_t)strtoul(e.hash.substr(i * 2, 2).c_str(), nullptr, 16);
346
347 uint8_t *out = nullptr;
348 size_t out_len = 0;
349 if (e.kind == Content::Event::DONE) {
350 swarmd_content_done_t d;
351 memset(&d, 0, sizeof(d));
352 memcpy(d.hash, raw, 32);
353 d.path.data = (utf8_t *)e.detail.c_str();
354 d.path.len = e.detail.size();
355 if (swarmd_encode_on_content_done(&d, &out, &out_len) < 0) continue;
356 } else if (e.kind == Content::Event::FAILED) {
357 swarmd_content_failed_t f;
358 memset(&f, 0, sizeof(f));
359 memcpy(f.hash, raw, 32);
360 f.reason.data = (utf8_t *)e.detail.c_str();
361 f.reason.len = e.detail.size();
362 if (swarmd_encode_on_content_failed(&f, &out, &out_len) < 0) continue;
363 } else {
364 swarmd_content_progress_t p;
365 memset(&p, 0, sizeof(p));
366 memcpy(p.hash, raw, 32);
367 p.received = e.received;
368 p.total = e.total;
369 if (swarmd_encode_on_content_progress(&p, &out, &out_len) < 0) continue;
370 }
371 if (out == nullptr) continue;
372 _uds.broadcast(out, out_len);
373 free(out); // caller frees
374 }
375}
376
288377void Daemon::poll(int timeout_ms) {
289378 _drain();
379 _drain_content();
290380 _uds.poll(timeout_ms);
291381}
292382
swarmd/src/daemon.h +6 -1
……@@ -24,6 +24,7 @@
2424#include <string>
2525#include <vector>
2626
27#include "content.h"
2728#include "control.h"
2829#include "uds_server.h"
2930#include "../../src/udx_transport.h"
……@@ -32,7 +33,9 @@ namespace swarmd {
3233
3334class Daemon {
3435public:
35 bool start(const char *socket_path, const char *bootstrap);
36 // `storage_dir` may be null, in which case the content plane stays off and
37 // every content command is refused rather than half-answered.
38 bool start(const char *socket_path, const char *bootstrap, const char *storage_dir);
3639 void stop();
3740
3841 // One pass: drain the cross-thread queue, then serve the local socket.
……@@ -82,10 +85,12 @@ private:
8285 void _drain();
8386
8487 void _fill_status(swarmd_status_t *out);
88 void _drain_content();
8589
8690 UdxTransport _t;
8791 UdsServer _uds;
8892 Control _control;
93 Content _content;
8994
9095 // UDX-THREAD ONLY, except where a lock is taken: the peer's mux, so a
9196 // channel opened after the swarm meshed can still reach existing peers.
swarmd/src/main.cpp +9 -4
……@@ -1,8 +1,10 @@
11// swarmd — a generic swarm daemon.
22//
3// swarmd --socket <path> [--bootstrap host:port]
3// swarmd --socket <path> [--bootstrap host:port] [--storage <dir>]
44//
5// Omit --bootstrap to use the public DHT.
5// Omit --bootstrap to use the public DHT. Omit --storage and the content plane
6// stays off — content commands are then refused, which is a better answer than
7// accepting a fetch that has nowhere to put a file.
68
79#include <csignal>
810#include <cstdio>
……@@ -23,12 +25,15 @@ int main(int argc, char **argv) {
2325
2426 const char *sock = nullptr;
2527 const char *bootstrap = nullptr;
28 const char *storage = nullptr;
2629 for (int i = 1; i < argc; i++) {
2730 if (!strcmp(argv[i], "--socket") && i + 1 < argc) sock = argv[++i];
2831 else if (!strcmp(argv[i], "--bootstrap") && i + 1 < argc) bootstrap = argv[++i];
32 else if (!strcmp(argv[i], "--storage") && i + 1 < argc) storage = argv[++i];
2933 }
3034 if (!sock) {
31 fprintf(stderr, "usage: swarmd --socket <path> [--bootstrap host:port]\n");
35 fprintf(stderr, "usage: swarmd --socket <path> [--bootstrap host:port] "
36 "[--storage <dir>]\n");
3237 return 1;
3338 }
3439
……@@ -36,7 +41,7 @@ int main(int argc, char **argv) {
3641 signal(SIGTERM, on_signal);
3742
3843 swarmd::Daemon d;
39 if (!d.start(sock, bootstrap)) {
44 if (!d.start(sock, bootstrap, storage)) {
4045 fprintf(stderr, "failed to start (socket %s)\n", sock);
4146 return 1;
4247 }
swarmd/test/test_content_e2e.py +185 -0
……@@ -0,0 +1,185 @@
1#!/usr/bin/env python3
2"""Gate D6: a large file crosses two daemons BY HASH, and never over the socket.
3
4 python3 -B test/test_content_e2e.py [--bootstrap host:port] [--mb 100]
5 [--seconds N]
6
7One daemon registers a file; the other asks for it by content hash. The bytes
8move peer-to-peer over the overlay channel, and the local socket carries a
932-byte hash and a path — docs/SWARMD-DESIGN.md §6, which is the whole reason
10the content plane exists rather than a SEND loop.
11
12⛔ IT MEASURES §6 RATHER THAN ASSERTING IT. The fetching client counts every
13byte the daemon ever wrote to it. For a 100 MB transfer that number is a few
14hundred; if content ever started travelling over the socket it would be a
15hundred million, and this gate would say so.
16
17⛔ IT ASSERTS ITS OWN BASELINE. If the two daemons never meet, the transfer
18result says nothing about the content plane, so it exits INCONCLUSIVE.
19"""
20
21import hashlib, os, re, subprocess, sys, tempfile, time
22
23HERE = os.path.dirname(os.path.realpath(__file__))
24ROOT = os.path.dirname(HERE)
25SWARMD = os.path.join(ROOT, "swarmd")
26PROBE = os.path.join(ROOT, "tools", "swarmd-probe")
27EXIT_PASS, EXIT_FAIL, EXIT_INCONCLUSIVE = 0, 1, 2
28
29TOPIC = "c0" * 32
30# Generous: the socket carries a handful of control frames plus progress ticks.
31# A megabyte here would still be four orders of magnitude below the payload.
32SOCKET_BUDGET = 256 * 1024
33
34
35def arg(name, default=None):
36 return sys.argv[sys.argv.index(name) + 1] if name in sys.argv else default
37
38
39def read(path):
40 try:
41 with open(path, "rb") as fh:
42 return fh.read().decode("utf-8", "replace")
43 except OSError:
44 return ""
45
46
47def wait_for(path, needle, timeout, proc):
48 end = time.time() + timeout
49 while time.time() < end:
50 if proc is not None and proc.poll() is not None:
51 return False
52 if needle in read(path):
53 return True
54 time.sleep(0.3)
55 return False
56
57
58def peers_seen(text):
59 return max([int(m) for m in re.findall(r"peers=(\d+)", text)] or [0])
60
61
62def digest(path):
63 h = hashlib.sha256()
64 with open(path, "rb") as fh:
65 for block in iter(lambda: fh.read(1 << 20), b""):
66 h.update(block)
67 return h.hexdigest()
68
69
70def main():
71 bootstrap = arg("--bootstrap")
72 megabytes = int(arg("--mb", "100"))
73 window = int(arg("--seconds", "120" if bootstrap else "300"))
74
75 for path in (SWARMD, PROBE):
76 if not os.path.isfile(path):
77 print("missing %s — run ./build.sh" % path)
78 return EXIT_INCONCLUSIVE
79
80 tmp = tempfile.mkdtemp(prefix="swarmd-content-")
81 asset = os.path.join(tmp, "asset.bin")
82 with open(asset, "wb") as fh:
83 # Incompressible, so nothing downstream can cheat by encoding it away.
84 for _ in range(megabytes):
85 fh.write(os.urandom(1 << 20))
86 want = digest(asset)
87 print("%d MB asset, sha256 %s" % (megabytes, want[:16]))
88
89 procs, logs = [], {}
90
91 def spawn(argv, tag):
92 logs[tag] = os.path.join(tmp, tag + ".log")
93 fh = open(logs[tag], "wb")
94 p = subprocess.Popen(argv, stdout=fh, stderr=subprocess.STDOUT)
95 procs.append(p)
96 return p
97
98 try:
99 daemons = {}
100 for tag in ("a", "b"):
101 store = os.path.join(tmp, "store-" + tag)
102 os.makedirs(store)
103 argv = [SWARMD, "--socket", os.path.join(tmp, tag + ".sock"),
104 "--storage", store]
105 if bootstrap:
106 argv += ["--bootstrap", bootstrap]
107 daemons[tag] = spawn(argv, tag)
108
109 for tag in ("a", "b"):
110 if not wait_for(logs[tag], "dht=up", 90, daemons[tag]):
111 print("\nINCONCLUSIVE: daemon %s never reached dht=up." % tag)
112 return EXIT_INCONCLUSIVE
113 print(" both daemons are on the DHT")
114
115 holder = spawn([PROBE, "--socket", os.path.join(tmp, "a.sock"),
116 "--topic-hex", TOPIC, "--serve-file", asset,
117 "--seconds", str(window)], "hold")
118 if not wait_for(logs["hold"], "SERVING ", 60, holder):
119 print("\nINCONCLUSIVE: the holder never registered the file.")
120 return EXIT_INCONCLUSIVE
121 content_hash = read(logs["hold"]).split("SERVING ")[1].split()[0]
122 print(" holder registered it as %s" % content_hash[:16])
123
124 dest = os.path.join(tmp, "got.bin")
125 fetcher = spawn([PROBE, "--socket", os.path.join(tmp, "b.sock"),
126 "--topic-hex", TOPIC, "--fetch-hex", content_hash,
127 "--dest", dest, "--seconds", str(window)], "fetch")
128
129 started = time.time()
130 end = started + window + 30
131 while time.time() < end and fetcher.poll() is None:
132 time.sleep(1)
133 if fetcher.poll() is None:
134 fetcher.terminate()
135 try: fetcher.wait(timeout=10)
136 except subprocess.TimeoutExpired: fetcher.kill()
137
138 out = read(logs["fetch"])
139 met = min(peers_seen(read(logs["a"])), peers_seen(read(logs["b"])))
140 if not met:
141 print("\nINCONCLUSIVE: the two daemons never saw each other, so the "
142 "transfer result says nothing about the content plane.")
143 return EXIT_INCONCLUSIVE
144
145 if "FETCHED " not in out:
146 print("\nFAIL: the daemons met and the fetch did not complete.")
147 print(out[-2000:])
148 return EXIT_FAIL
149
150 elapsed = time.time() - started
151 got = digest(dest)
152 size = os.path.getsize(dest)
153 socket_bytes = int(re.search(r"SOCKET-BYTES (\d+)", out).group(1))
154 print(" fetched %d bytes in %.1fs; the socket carried %d"
155 % (size, elapsed, socket_bytes))
156
157 ok = True
158 if size != megabytes << 20:
159 print("\nFAIL: %d bytes arrived, %d expected" % (size, megabytes << 20))
160 ok = False
161 if got != want:
162 print("\nFAIL: the bytes do not match the original")
163 ok = False
164 if socket_bytes > SOCKET_BUDGET:
165 print("\nFAIL: %d bytes crossed the LOCAL SOCKET for a %d MB file — "
166 "content is not supposed to travel that way (§6)"
167 % (socket_bytes, megabytes))
168 ok = False
169 if not ok:
170 return EXIT_FAIL
171
172 print("\nPASS: %d MB crossed the swarm by hash; the socket carried %d bytes."
173 % (megabytes, socket_bytes))
174 return EXIT_PASS
175 finally:
176 for p in procs:
177 if p.poll() is None:
178 p.terminate()
179 try: p.wait(timeout=5)
180 except subprocess.TimeoutExpired: p.kill()
181 print("(logs and files: %s)" % tmp)
182
183
184if __name__ == "__main__":
185 sys.exit(main())
swarmd/test/test_send_truth +0 -0
(no content changes — binary or rename)
swarmd/test/test_send_truth.cpp +1 -1
……@@ -87,7 +87,7 @@ int main() {
8787
8888 swarmd::Daemon d;
8989 // A bootstrap that answers nothing. The daemon runs; it never gets a peer.
90 if (!d.start(path, "127.0.0.1:1")) { printf("start failed\n"); return 1; }
90 if (!d.start(path, "127.0.0.1:1", nullptr)) { printf("start failed\n"); return 1; }
9191
9292 int fd = dial(path);
9393 check(fd >= 0, "client attached");
swarmd/tools/swarmd_probe.cpp +104 -3
……@@ -1,8 +1,14 @@
11// swarmd-probe — the smallest real client, and the start of the C++ client
22// library SWARMD-DESIGN.md §7 item 7 asks for.
33//
4// swarmd-probe --socket <path> --topic-hex <64 hex> --protocol <name>
4// swarmd-probe --socket <path> --topic-hex <64 hex> [--protocol <name>]
55// [--send <text>] [--seconds N]
6// [--serve-file <path>] [--fetch-hex <64hex> --dest <path>]
7//
8// The content flags exercise the content plane: --serve-file registers a file by
9// its overlay content hash and prints that hash; --fetch-hex asks the swarm for
10// one and waits for on-content-done. Bytes never cross the socket either way —
11// the daemon writes the file and prints where it put it.
612//
713// ⛔ IT USES THE GENERATED ENCODERS. The first version of the channel gate
814// hand-rolled compact-encoding in Python and the daemon answered "undecodable
……@@ -59,12 +65,16 @@ static bool hex32(const char *s, uint8_t *out) {
5965int main(int argc, char **argv) {
6066 const char *sock = nullptr, *topic_hex = nullptr, *proto = "swarmd-probe/v1";
6167 const char *send_text = nullptr;
68 const char *serve_file = nullptr, *fetch_hex = nullptr, *dest = nullptr;
6269 int seconds = 30;
6370 for (int i = 1; i < argc; i++) {
6471 if (!strcmp(argv[i], "--socket") && i + 1 < argc) sock = argv[++i];
6572 else if (!strcmp(argv[i], "--topic-hex") && i + 1 < argc) topic_hex = argv[++i];
6673 else if (!strcmp(argv[i], "--protocol") && i + 1 < argc) proto = argv[++i];
6774 else if (!strcmp(argv[i], "--send") && i + 1 < argc) send_text = argv[++i];
75 else if (!strcmp(argv[i], "--serve-file") && i + 1 < argc) serve_file = argv[++i];
76 else if (!strcmp(argv[i], "--fetch-hex") && i + 1 < argc) fetch_hex = argv[++i];
77 else if (!strcmp(argv[i], "--dest") && i + 1 < argc) dest = argv[++i];
6878 else if (!strcmp(argv[i], "--seconds") && i + 1 < argc) seconds = atoi(argv[++i]);
6979 }
7080 uint8_t topic[32];
……@@ -104,14 +114,44 @@ int main(int argc, char **argv) {
104114 printf("joined topic, opened %s\n", proto);
105115 fflush(stdout);
106116
117 // ---- content ----
118 // A serve is two calls: ask the daemon for the file's overlay hash, then
119 // register it. Asking the daemon rather than computing it here is the point —
120 // a hash from anywhere else is a hash the overlay will reject.
121 if (serve_file) {
122 swarmd_path_t pr;
123 memset(&pr, 0, sizeof(pr));
124 pr.path.data = (utf8_t *)serve_file;
125 pr.path.len = strlen(serve_file);
126 if (swarmd_encode_content_hash(10, &pr, &buf, &len) >= 0) send_frame(fd, buf, len);
127 }
128 if (fetch_hex && dest) {
129 uint8_t h[32];
130 if (!hex32(fetch_hex, h)) { fprintf(stderr, "bad --fetch-hex\n"); return 2; }
131 swarmd_content_fetch_t cf;
132 memset(&cf, 0, sizeof(cf));
133 memcpy(cf.hash, h, 32);
134 cf.dest_path.data = (utf8_t *)dest;
135 cf.dest_path.len = strlen(dest);
136 if (swarmd_encode_content_fetch(12, &cf, &buf, &len) >= 0) send_frame(fd, buf, len);
137 printf("fetching %s -> %s\n", fetch_hex, dest);
138 fflush(stdout);
139 }
140
107141 // ⛔ DO NOT SEND ON A TIMER. The first version fired once, eight seconds in,
108142 // whether or not a peer existed — and a send onto a channel the remote has
109143 // not opened yet is discarded by protomux. It worked on a local testnet,
110144 // where the mesh forms in under a second, and would have been a coin toss on
111145 // the public DHT. The daemon now answers `accepted` truthfully, so the right
112146 // trigger is that answer: keep offering the frame until it is taken.
147 uint64_t socket_bytes_in = 0;
113148 const time_t deadline = time(nullptr) + seconds;
114 const uint64_t FIRST_SEND_ID = 4;
149 // ⛔ SENDS GET THEIR OWN ID RANGE. Request ids are how a reply is matched to
150 // the call that asked for it, and "anything at or above 4" swept up the
151 // content replies too — the fetch's answer was decoded as a send-result and
152 // reported as "sent (accepted): (null)". Ids below 100 are the one-shot
153 // setup calls; sends count from 100.
154 const uint64_t FIRST_SEND_ID = 100;
115155 uint64_t next_send_id = FIRST_SEND_ID;
116156 bool send_accepted = false;
117157 time_t last_attempt = 0;
……@@ -135,6 +175,7 @@ int main(int argc, char **argv) {
135175 if (select(fd + 1, &rf, nullptr, nullptr, &tv) <= 0) continue;
136176 ssize_t r = read(fd, tmp, sizeof(tmp));
137177 if (r <= 0) break;
178 socket_bytes_in += (uint64_t)r;
138179 in.insert(in.end(), tmp, tmp + r);
139180
140181 for (;;) {
……@@ -145,7 +186,28 @@ int main(int argc, char **argv) {
145186 compact_state_t st = { 0, n, in.data() + 4 };
146187 rpc_message_t msg;
147188 if (rpc_decode_message(&st, &msg) >= 0) {
148 if (msg.type == rpc_response && msg.id >= FIRST_SEND_ID) {
189 if (msg.type == rpc_response && msg.id == 10) {
190 // The daemon's answer to CONTENT_HASH. Register it, and print
191 // the hash so a caller can hand it to the other end.
192 swarmd_hash_t hr;
193 memset(&hr, 0, sizeof(hr));
194 compact_state_t body = { 0, msg.len, msg.data };
195 if (swarmd_hash_decode(&body, &hr) >= 0 && serve_file) {
196 char hex[65];
197 for (int i = 0; i < 32; i++) sprintf(hex + i * 2, "%02x", hr.hash[i]);
198 hex[64] = 0;
199 swarmd_content_serve_t cs;
200 memset(&cs, 0, sizeof(cs));
201 memcpy(cs.hash, hr.hash, 32);
202 cs.path.data = (utf8_t *)serve_file;
203 cs.path.len = strlen(serve_file);
204 uint8_t *b2 = nullptr; size_t l2 = 0;
205 if (swarmd_encode_content_serve(11, &cs, &b2, &l2) >= 0)
206 send_frame(fd, b2, l2);
207 printf("SERVING %s %s\n", hex, serve_file);
208 fflush(stdout);
209 }
210 } else if (msg.type == rpc_response && msg.id >= FIRST_SEND_ID) {
149211 // The answer to one of our send attempts. `accepted` is the
150212 // daemon saying it had an open channel to put the frame on.
151213 swarmd_send_result_t res;
……@@ -168,9 +230,36 @@ int main(int argc, char **argv) {
168230 h.on_on_message = [](void *ctx, const swarmd_message_t *rm) {
169231 ((std::string *)ctx)->assign((const char *)rm->bytes, rm->bytes_len);
170232 };
233 static std::string done_path, fail_reason;
234 static bool content_done = false, content_failed = false;
235 h.on_on_content_done = [](void *, const swarmd_content_done_t *d) {
236 done_path.assign((const char *)d->path.data, d->path.len);
237 content_done = true;
238 };
239 h.on_on_content_failed = [](void *, const swarmd_content_failed_t *f) {
240 fail_reason.assign((const char *)f->reason.data, f->reason.len);
241 content_failed = true;
242 };
171243 uint8_t *rep = nullptr; size_t rl = 0;
172244 swarmd_hrpc_dispatch(&h, &msg, &rep, &rl);
173245 if (rep) free(rep);
246 if (content_done) {
247 // §6: bulk bytes never cross this socket. This is the
248 // number that says so — everything the daemon told us,
249 // for a file of any size.
250 printf("SOCKET-BYTES %llu\n",
251 (unsigned long long)socket_bytes_in);
252 printf("FETCHED %s\n", done_path.c_str());
253 fflush(stdout);
254 close(fd);
255 return 0;
256 }
257 if (content_failed) {
258 printf("FETCH FAILED: %s\n", fail_reason.c_str());
259 fflush(stdout);
260 close(fd);
261 return 1;
262 }
174263 if (!got.empty()) {
175264 printf("RECEIVED: %s\n", got.c_str());
176265 fflush(stdout);
……@@ -183,6 +272,18 @@ int main(int argc, char **argv) {
183272 }
184273 }
185274
275 if (fetch_hex) {
276 printf("the fetch never completed inside the window\n");
277 close(fd);
278 return 1;
279 }
280 if (serve_file) {
281 // A holder answers requests; it has nothing to wait for. Reaching the end
282 // of the window is what success looks like.
283 printf("served for %ds\n", seconds);
284 close(fd);
285 return 0;
286 }
186287 // A pure sender has nothing to receive, so say what it actually knows.
187288 if (send_text) {
188289 printf(send_accepted ? "sent, never answered\n"