p2p-db

p2p-db/

ird-p2p-db

ClojureScript library for Node.js and the browser: an encrypted, append-only, peer-to-peer database for the holders of a secret, built on Helia + OrbitDB and designed to pair with ird-p2p-cljs (v0.3.0).

ird-p2p-cljs rooms are ephemeral end-to-end messaging: peers who hold a room's secret talk while they are online together. ird-p2p-db adds the shared history that survives churn: every participant holding a database's secret can append records and replicate them peer to peer through OrbitDB over Helia — no server, no account, no key ever leaves the peers. Records are sealed before OrbitDB sees them, so OrbitDB hashes, signs, stores and syncs ciphertext, giving tamper-evident ordering on top of confidentiality. Write access is possession of the secret.

Designed like its sibling: core.async channels instead of promises, errors as values, injectable node and database constructors, and bytes that never cross the public API as JavaScript objects — they travel as base64 strings.

Runtime dependencies: ClojureScript, core.async and ird-p2p-cljs (from deps.edn), plus Helia, OrbitDB and the libp2p pieces from npm (exact versions below). Node 22+ — @orbitdb/core 4.0.0 declares node >= 22 and the suite runs on it. The same source runs in a browser, reaching Node peers through a circuit relay you run (Browser peers).

Install

A source library, consumed as a git dependency — no registry, no build step of its own. The usual setup is shadow-cljs reading deps.edn:

;; deps.edn
{:paths ["src"]
 :deps {ro.ird/ird-p2p-db {:git/url "https://ird.cod-sursa.ro/p2p-db.git" :git/sha "…"}
        thheller/shadow-cljs {:mvn/version "3.5.0"}}}

ird-p2p-cljs comes along transitively (this repo's deps.edn pins it to v0.3.0 over HTTPS from the same domain). Pin :git/sha to the latest commit (or use :git/tag with its sha prefix):

git ls-remote https://ird.cod-sursa.ro/p2p-db.git HEAD

The landing page at https://ird.cod-sursa.ro/p2p-db/ shows the current release coordinates ready to paste, and https://ird.cod-sursa.ro/releases.json carries them for scripts and agents (head, tag, tag_sha per repo).

Then add the Helia/OrbitDB stack to your package.json. These are the exact versions the suite proves (see Versions); they resolve to a single @libp2p/interface 3.3.0 in the tree:

"dependencies": {
  "helia": "7.1.12",
  "@orbitdb/core": "4.0.0",
  "@libp2p/gossipsub": "17.1.1",
  "@libp2p/tcp": "11.0.28",
  "@libp2p/websockets": "10.1.21",
  "@libp2p/identify": "4.1.14",
  "@chainsafe/libp2p-noise": "17.0.0",
  "@chainsafe/libp2p-yamux": "8.0.1",
  "@multiformats/multiaddr": "13.0.3"
}

Optional, imported only when the matching option is used: "@libp2p/mdns": "12.0.32" for :mdns? true, "blockstore-level": "4.0.1" + "datastore-level": "13.0.1" for :path, and "@libp2p/circuit-relay-v2": "4.2.13" for :relay / :relay-hop? (always, in a browser).

All of them are ESM-only. Under a shadow-cljs :node-script/:node-library/:esm build the library imports them with import() natively; under :node-test define the import helper in a natively-loaded entry file (this repo's shadow-cljs.edn shows the one-line :prepend; it is the same ird_dyn_import global ird-p2p-cljs uses, so an app loading both defines it once).

Quickstart

(require '[cljs.core.async :refer [go <!]]
         '[ird.p2p.async :refer [<?]]        ; <! that rethrows an error value (from ird-p2p-cljs)
         '[ird.p2p.db.core :as db]
         '[ird.p2p.db.errors :as err])

;; a secret shared out of band (an invite link's fragment works: ird.p2p/invite-url),
;; a salt that namespaces your databases, and a name
(def secret (db/random-secret))

(go
  (let [notes (<? (db/open {:secret secret :app-salt "myapp/v1" :name "notes"}))]
    ;; drain the records — history first, then live ones, each a plain map
    (go (loop []
          (when-let [r (<! (db/entries-stream notes))]
            (println (:payload r) (:clock r) (:hash r))
            (recur))))
    ;; tell a peer where to find you, or find a peer
    (println (<? (db/multiaddrs notes)))               ; ["/ip4/127.0.0.1/tcp/54321/p2p/12D3Koo…" …]
    (<? (db/dial notes ["/ip4/192.168.1.20/tcp/4001/p2p/12D3Koo…"]))
    ;; append a record: sealed here, replicated by OrbitDB
    (<? (db/append notes {:text "salut" :at 1726000000}))
    ;; a snapshot in canonical order
    (println (count (<? (db/entries notes))))
    (<! (db/close notes))))

Every asynchronous function returns a core.async promise-chan that delivers exactly one value — the result, or an ex-info error value (see Errors). Nothing is thrown out of a library function; (<? ch) inside a go block rethrows an error value so ordinary try/catch applies, and ird.p2p.async/go-try is a go whose body may throw. entries-stream and peers-stream are ordinary channels to drain; they close when the handle is closed.

open options: :secret, :app-salt and :name (required); :path (a directory — records, heads, keys and the peer id persist across restarts; without it everything is in memory); :listen (multiaddrs, default loopback tcp + websockets on ephemeral ports); :dial (peers to dial as soon as the node is up); :mdns? true (discover LAN peers via mDNS and dial each one found — see Replication); :relay (the multiaddr of a circuit relay you run: the node reserves a slot on it and becomes reachable at <relay>/p2p-circuit/p2p/<its id>); :relay-hop? true (run that relay on this Node peer); :gater (a function of a connecting peer's multiaddr string returning true to let it in — opt-in inbound connection gating; default: everyone may connect); :address (advanced: join an existing database by its OrbitDB address). The handle is a map of public data — :name, :address, :peer-id, :identity, :manifest-name — plus the channels.

A record is {:payload <map> :clock {:id :time} :by <identity hash> :hash <entry hash>}: the payload you appended (any map; ordered JSON on the way through), OrbitDB's Lamport clock, the writer identity's hash, and the entry's content hash (a base58btc CID). entries returns records in canonical order — by clock time, then by hash — which is the same on every peer, however each peer's log was traversed; the stream delivers history in that order and then records as they become known locally.

Replication: native OrbitDB sync on a private swarm

Each handle runs its own Helia node over a libp2p swarm that reaches nothing public: tcp and websockets transports, noise, yamux, identify and gossipsub — no bootstrap list, no DHT, no delegated HTTP routing, no public gateways, no relay discovery, no NAT traversal. Peers meet by explicit dial (:dial at open, db/dial later) or, on a LAN, by mDNS when asked: with :mdns? true a Node peer announces itself and queries under the service tag _ird-p2p-db._udp.local — not libp2p's default _p2p._udp.local, so it neither answers nor dials the other libp2p nodes a LAN may hold (an IPFS daemon, say) — and dials every peer that answers, again whenever one answers while not connected (a restarted peer comes back by itself). mDNS announces the listen addresses, so peers on different machines must :listen on a LAN address; the loopback default only lets peers of one machine meet. A circuit relay enters the picture only as configuration: a peer you run with :relay-hop? true, named by the peers that reserve on it with :relay (Browser peers); it is opt-in, never built in. Nothing rides on the ird relay: the relay carries rooms; this library never sends a record through it.

Replication is OrbitDB's own sync protocol, unchanged:

The address is derived, not exchanged: every holder of the secret computes the same manifest (name, type, access controller) and therefore the same address, so two peers open the same database with nothing but the secret and a way to reach each other.

Key schedule and AAD

Everything derives from the database secret (at least 16 random bytes, 32 recommended, shared as URL-safe base64) and the app salt (a string chosen once per application, e.g. myapp/v1), with the very same HKDF the sibling uses for rooms:

HKDF(info) = HKDF-SHA256(ikm = secret bytes, salt = UTF8(appSalt), info = UTF8(info)), 32 bytes
Name Formula Use
K_db HKDF("db\|<name>") AES-256-GCM key for every record of the database
identity seed HKDF("db\|<name>\|identity") Ed25519 seed of the identity key (its public key's hex is the OrbitDB identity id)
signing seed HKDF("db\|<name>\|identity\|sign") Ed25519 seed of the key that signs OrbitDB entries
manifest name b64url(HKDF("db\|<name>\|manifest")) the name the OrbitDB manifest carries, so the human name never reaches the network

<name> is the database name you pass to open: a non-blank string of at most 128 characters that may not contain | (the label separator). The same secret under two salts, or two names, yields unrelated keys, identities and addresses.

A record is the sibling's at-rest envelope shape:

0x02 ‖ iv(12) ‖ ciphertext ‖ tag(16)        as standard base64

The base64 text of the envelope is what OrbitDB receives as the entry's value. OrbitDB then wraps it in its own entry (id, payload, next, refs, clock, v), signs those bytes with the signing key, and content-addresses the result — the signature and the hash cover the ciphertext, never the plaintext.

Decrypt or drop. Any failure — wrong prefix, wrong key, wrong AAD, authentication failure, a plaintext that is not a JSON object — means the record is dropped silently; it never appears on the stream or in entries, and nothing is logged or answered.

This is this library's own local convention. It does not interoperate with the Swift SDK (ird-p2p-swift), which has no database or mailbox-style envelope for this; the shared parts are the HKDF construction and the envelope shape, not any wire format between the two.

Write access is possession of the secret

OrbitDB decides who may append with an access controller, and this library uses the default ipfs one with a write list of exactly one identity — the identity derived from the secret (the two Ed25519 keys above, fed into OrbitDB's keystore as its raw seed ‖ public key form under the ids its publickey identity provider expects). Every peer holding the secret derives the same identity, so every holder can write; a peer with another secret derives another identity, opens a different database by name (its manifest names a different writer), and if it joins this one by address every append it tries is refused by the access controller — on its own node and on every receiver.

The corollary: anyone with the secret can also read, exactly as in rooms. There is no per-member ACL, no revocation, no forward secrecy: rotating a secret means opening a new database.

Because the identity is shared, :by and :clock :id are the same for every writer: they identify the database's writer identity, not a person or device. Attribute records in the payload if your application needs to (anything JSON).

What peers and the network can see

The swarm is private but not authenticated: any peer that can connect to your node — one you dialed, one that dialed you, one found over mDNS — is served like any other. Such a peer sees:

OrbitDB exposes nothing beyond that: it has no query interface to a peer's data and serves only blocks by hash and the sync protocol. Bitswap will hand any block it holds to any connected peer — this is why the swarm must stay private: bind loopback or a LAN interface, dial only peers you mean to, and treat every listen multiaddr like the secret's transport.

Connection gating is opt-in. By default every inbound connection is accepted. Pass :gater to open — a function of one argument, the remote multiaddr string of a peer connecting to you, returning true to let it in — and libp2p's connection gater refuses everyone else (false, nil or a throw refuses):

(db/open {:secret secret :app-salt "myapp/v1" :name "chat"
          :gater (fn [addr] (or (str/starts-with? addr "/ip4/192.168.1.")
                                (str/ends-with? addr (str "/p2p/" trusted-peer-id))))})

The gater is asked once per inbound connection, right after the noise handshake has established who is calling and before any protocol runs, so the string always ends in /p2p/<the caller's peer id>: /ip4/<ip>/tcp/<port>/p2p/<id> for a direct caller (the port is the caller's ephemeral one), <relay address>/p2p-circuit/p2p/<id> for one arriving through your relay. A refused peer gets no connection — no identify, no gossipsub, no bitswap, no sync, so none of its records reach you and none of yours reach it (its own dial may report success for an instant: libp2p's dialer finishes its side of the upgrade before the reset arrives). Only inbound connections are gated, on Node and in the browser alike: whoever you dial (:dial, db/dial, a relay, a peer found with :mdns?) is never put to the gater, and replicates with you like anyone else. In a browser :gater comes on top of the lifted dial restriction described under Browser peers.

What this library does not protect against: a holder of the secret (they can read and write everything, and cannot be told apart), traffic analysis by anyone on the path (sizes and timing), and a compromised machine. Records are not deniable and never expire on their own.

Browser peers

The same library runs in a browser: ird.p2p.db.core and every public namespace are unchanged, and ird.p2p.db.impl.helia builds the browser variant of the node when no Node process is present. Records, identity and addresses are identical — a browser peer and a Node peer holding the same secret open the same database with zero crypto differences, and the suite proves it with a real headless Chrome peer (Tests).

What a browser peer is. A Helia node over websockets and the circuit relay transport, noise, yamux, identify and gossipsub — no tcp, no mDNS, and still no bootstrap, no DHT, no gateways. A browser can dial but never listen, so the way in is a circuit relay v2 hop you run: a Node peer opened with :relay-hop? true on an address browsers can reach (a websockets listener; behind TLS in production, since a page served over https may only open wss). Every peer that names it in :relay dials it at open and reserves a slot, becoming reachable at <relay>/p2p-circuit/p2p/<its id>; a browser dials such an address (:dial, db/dial) and the relay forwards the noise-encrypted stream between the two — it terminates nothing. The relay is configuration you supply, exactly like :dial: never a built-in address, no discovery, and no per-connection duration or data limits since it is yours. libp2p's browser connection gater, which refuses loopback and LAN dials by default so that a page cannot probe its host's network, is lifted for this node: every address it dials is your own configuration.

;; on the machine browsers can reach: the hop (also a peer of whatever database it opens)
(db/open {:secret s :app-salt "myapp/v1" :name "notes"
          :relay-hop? true :listen ["/ip4/0.0.0.0/tcp/4001/ws"]})
;; => reachable at /ip4/<host>/tcp/4001/ws/p2p/<relay id>  (put it behind wss for https pages)

;; a Node peer anywhere (NAT or not): reserve on the relay
(db/open {:secret s :app-salt "myapp/v1" :name "notes" :relay relay-addr})
;; (db/multiaddrs h) now includes <relay-addr>/p2p-circuit/p2p/<its id>

;; in the browser: reserve on the relay too, and reach the Node peer through it
(db/open {:secret s :app-salt "myapp/v1" :name "notes"
          :relay relay-addr :dial [(str relay-addr "/p2p-circuit/p2p/" node-peer-id)]})

Build. helia, @orbitdb/core and the libp2p packages are ESM-only and the library imports them at runtime through the global ird_dyn_import helper; a browser cannot resolve their bare specifiers, so bundle them once with esbuild from the entry this repo ships, load that chunk before your compiled ClojureScript, and point the helper at it:

npm install esbuild buffer @libp2p/circuit-relay-v2   # plus the stack from Install, minus @libp2p/tcp and @libp2p/mdns
npx esbuild node_modules/ird-p2p-db/browser/vendor-entry.mjs --bundle --format=iife --global-name=irdVendor --platform=browser --outfile=public/vendor.js
<script src="vendor.js"></script>
<script>globalThis.ird_dyn_import = n => Promise.resolve(irdVendor.load(n))</script>
<script src="app.js"></script>   <!-- your shadow-cljs :browser build -->

browser/vendor-entry.mjs re-exports the modules by their bare names and installs Node's Buffer (the buffer package) as a global, which the byte helpers of ird-p2p-cljs use as soon as their namespaces load — hence the order. esbuild's browser resolution picks each package's browser build; level becomes browser-level. The chunk is about 3.5 MB unminified. (Import maps over node_modules were rejected: hundreds of transitive packages with export conditions. shadow-cljs bundling them itself does not apply: the library imports at runtime by design so that Node never bundles.)

Storage. The same rule as on Node: without :path everything is in memory and gone when the tab closes; with :path Helia's blockstore and datastore and OrbitDB's keystore, heads and index live in IndexedDB through Level's browser backend, named level-js-<path>/…, so records and the peer id survive a reload. IndexedDB rather than a memory-only build because it costs nothing (the same level packages, resolved for the browser) and keeps one option meaning one thing on both platforms.

What stays private. Exactly what the swarm section says, with the relay as one more peer on the path: it sees the two peer ids and the bytes of a noise-encrypted connection it cannot open; it is not a peer of the database unless it holds the secret and opens it, and even then it stores and forwards only sealed records. The suite checks that the relay holds no entries and receives nothing on the database's topic.

Limits

Errors

Nothing is thrown out of a library function; failures travel on the channel as ex-info values with a :type:

:type ex-data when
:ird.p2p.db/config-error :detail a precondition failed before anything started (bad secret, salt or name; a payload that is not a map)
:ird.p2p.db/node-error :detail Helia, libp2p or OrbitDB failed to start or answered an error
:ird.p2p.db/dial-error :detail dialing a peer's multiaddrs failed
:ird.p2p.db/append-rejected :detail OrbitDB's access controller refused the entry (a database joined by address with another secret)
:ird.p2p.db/payload-too-large :bytes :limit a serialized payload exceeds 256 KiB
:ird.p2p.db/closed — a call after close

Predicates and the types set live in ird.p2p.db.errors (config-error?, append-rejected?, …, and error? for any error value). Remote tampering is never an error: records that do not decrypt are dropped silently, and entries whose signature or writer does not check out are refused by OrbitDB before they reach this library.

Public API is pure ClojureScript

Every public namespace (src/ird/p2p/db/** except src/ird/p2p/db/impl/**) contains zero JavaScript interop: no ^js, no js/…, no property access, no #js. All interop — Helia, OrbitDB, libp2p, the level stores — lives under src/ird/p2p/db/impl/, and everything crossing a public boundary is plain ClojureScript data (maps, vectors, keywords, strings, numbers) plus core.async channels, with bytes as base64 strings. A gate test (ird.p2p.db.api-purity-test) scans the source and fails on any leak. Cryptography comes from the sibling's ird.p2p.impl.webcrypto through its string-in/string-out surface.

Versions

ird-p2p-cljs v0.3.0 — the release this repo's deps.edn pins — sits on the same interface-v3 js-libp2p line as this library (its v0.2.0 sat on the v2-compatible line, because the last @chainsafe/libp2p-gossipsub targets @libp2p/interface v2). This library needs no gossipsub of its own choosing: Helia brings libp2p v3, and OrbitDB's sync wants a pubsub service — which the libp2p project now publishes for the v3 line as @libp2p/gossipsub. The set in Install — helia 7.1.12, @orbitdb/core 4.0.0, @libp2p/gossipsub 17.1.1 and the libp2p 3.3.x pieces — resolves to one @libp2p/interface (3.3.0) with no duplicate majors, and is the newest of each at the time of writing. It is proven end to end by this repo's always-on two-node test, not by its changelogs. Do not mix it with the libp2p 2.x pins of the sibling's v0.2.0 in one process. Browser peers add @libp2p/circuit-relay-v2 4.2.13 (the same interface 3.3.0 line) and buffer 6.0.3; the browser suite is bundled by esbuild 0.25.10 and run by puppeteer 24.24.0 (dev only). Node 18–21 was investigated and is blocked upstream, so engines stays node >=22: @orbitdb/core@4.0.0 declares engines.node >=22.0.0, and the libp2p 3 line under it needs Node 22 whatever it declares — libp2p@3.3.11 depends on p-retry@8.0.1 (node >=22), @libp2p/circuit-relay-v2@4.2.13 on nanoid@6.0.1 (^22 || ^24 || >=26), and @libp2p/utils, @chainsafe/libp2p-yamux and it-queue call Promise.withResolvers, which Node gained in 22. The older OrbitDB lines are no way around it: @orbitdb/core@2.0.0–3.0.2 declare node >=20.0.0 and are written against helia 5 / libp2p 2 (@chainsafe/libp2p-gossipsub 14) stream and blockstore APIs that helia 7 replaced, so they cannot join this stack and would mean downgrading all of it — the set the v0.2.0 browser peers shipped on — to gain Node 20–21 only; Node 18–19 is declared by nothing newer than @orbitdb/core@1.0.2 (>=16.0.0, 2023), and every Node release below 22 is past its end of life.

Tests

npm install          # helia + orbitdb + the libp2p 3.x pieces + shadow-cljs + esbuild + puppeteer (dev only; ClojureScript comes from deps.edn)
npm test             # 1. test:prepare — puppeteer fetches Chrome for Testing once, esbuild bundles the vendor chunk
                     # 2. shadow-cljs compiles the Node suite and the browser suite
                     # 3. the Node suite runs: key schedule and sealed records (round trips, every way
                     #    opening fails silently), the shared identity (determinism, keystore records),
                     #    the facade with injected node/database failures, the zero-interop API-purity
                     #    gate, and — always on — two real Helia/OrbitDB nodes over loopback tcp:
                     #    history caught up on join, live records over gossipsub both ways, identical
                     #    canonical snapshots, ciphertext-only stored and wire bytes; a wrong-secret
                     #    peer refused by address and separated by name; close/cleanup; :path
                     #    persistence across a restart; two peers that are never told about each
                     #    other meeting through :mdns? alone, replicating both ways, and meeting
                     #    again after one restarts; :gater letting one peer in and refusing another
                     #    (direct and through a relay) while outbound dials stay ungated; a bounded
                     #    soak — two persistent peers taking turns over 600 records, one restarting
                     #    while the other appends, both converging on the exact history in canonical
                     #    order and replicating on (same-process churn; real network churn is out of
                     #    scope) — and the browser-peer test, which starts a
                     #    relay hop and a Node peer reserved on it, launches headless Chrome on the
                     #    compiled browser suite, and checks both views (below)

The browser-peer test (ird.p2p.db.browser-peer-test, always on) is the proof that a browser peer replicates sealed records with a Node peer through a relay: in Chrome, ird.p2p.db.browser.replication-test opens the database with :relay and dials the Node peer's /p2p-circuit address, verifies its only multiaddr is the relayed one and its connection to the Node peer carries /p2p-circuit/ with no limits, catches up with the history, receives a live record and appends one, sees only sealed bytes on gossipsub, and joins the same log with a wrong secret to be refused on append and read nothing; a second test reopens a :path database from IndexedDB with the same peer id and record. The Node side asserts the mirror image — the browser's join and record, ciphertext-only storage and wire, a relay that holds no entries — and requires the in-page cljs.test summary to be green (it is printed, prefixed [browser]). The Node-only real-node tests settle in a few seconds and can be skipped with IRD_P2P_DB_NO_NODE=1 in a sandbox that cannot bind sockets; the browser-peer test has no such switch.

License

MIT.

static mirror of HEAD · about · clone: git clone https://ird.cod-sursa.ro/p2p-db.git