diff --git a/src/_blog/2023-07-rust-libp2p-based-ipfs-bootstrap-node.md b/src/_blog/2023-07-rust-libp2p-based-ipfs-bootstrap-node.md new file mode 100644 index 00000000..b111f690 --- /dev/null +++ b/src/_blog/2023-07-rust-libp2p-based-ipfs-bootstrap-node.md @@ -0,0 +1,170 @@ +--- +title: A Rusty Bootstrapper +description: 'Running rust-libp2p-server on one of our four IPFS bootstrap nodes.' +author: Max Inden (@mxinden) +date: 2023-07-24 +permalink: '/2023-rust-libp2p-based-ipfs-bootstrap-node/' +header_image: '' +tags: + - 'Kademlia' + - 'Rust' +--- + +# Summary + +As of July 13, 2023, one of the four "public good" IPFS bootstrap nodes operated by Protocol Labs has been running [rust-libp2p-server](https://github.com/mxinden/rust-libp2p-server) instead of [Kubo](https://github.com/ipfs/kubo), which uses [go-libp2p](https://github.com/libp2p/go-libp2p/). rust-libp2p-server is a thin wrapper around [rust-libp2p](https://github.com/libp2p/rust-libp2p). We run both Kubo and rust-libp2p-server on IPFS bootstrap nodes to increase resilience. A bug or vulnerability is less likely to be in both Kubo and rust-libp2p-server than Kubo alone. In addition to increasing resilience we gain experience running large rust-libp2p based deployments on the IPFS network. + +![rust-libp2p bootstrap node establishing its first connections](../assets/2023-07-rust-libp2p-based-ipfs-bootstrap-node-connections-established.png) + +# IPFS Public DHT Bootstrap Nodes + +_What is an IPFS bootstrap node?_ + +> A Bootstrap Node is a trusted peer on the IPFS network through which an IPFS node learns about other peers on the network. [...] + +See [IPFS Glossary](https://docs.ipfs.tech/concepts/glossary/#bootstrap-node). + +A new node trying to join the "[public IPFS DHT](https://github.com/ipfs/ipfs/discussions/473)", i.e. trying to bootstrap, will: + +1. Connect to its (pre-) configured bootstrap nodes. +2. Run some variation of the [Kademlia bootstrap process](https://github.com/libp2p/specs/tree/master/kad-dht#bootstrap-process) which boils down to iteratively: + 1. Generating random IDs. + 2. Asking already discovered nodes whether they know anyone closer to those IDs. + +Thus the only thing that an IPFS bootstrap node needs to do is: + +- Allow incoming connections. +- Maintain a healthy Kademlia routing table. +- Reply to Kademlia `FIND_NODE` requests based on nodes in its routing table. + +Let's dive a bit deeper. In the case of Kubo the [DNSAddr](https://github.com/multiformats/multiaddr/blob/master/protocols/DNSADDR.md) addresses of the IPFS bootstrap nodes are shipped within the release binary. + +``` go +// DefaultBootstrapAddresses are the hardcoded bootstrap addresses +// for IPFS. they are nodes run by the IPFS team. docs on these later. +// As with all p2p networks, bootstrap is an important security concern. +var DefaultBootstrapAddresses = []string{ + "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN", + "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa", + "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb", + "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt", + // ... +} +``` + +See [`bootstrap_peers.go` on github.com/ipfs/kubo](https://github.com/ipfs/kubo/blob/v0.21.0/config/bootstrap_peers.go#L11C1-L24C2). + +One can resolve those `/dnsaddr/...` through iterative DNS queries. Below is an example for the node with the peer ID `QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb`. This IPFS bootstrap node is running Kubo. + +``` +dig +short -t txt _dnsaddr.bootstrap.libp2p.io +"dnsaddr=/dnsaddr/am6.bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb" +[...] +``` + +``` +dig +short -t txt _dnsaddr.am6.bootstrap.libp2p.io +"dnsaddr=/ip6/2604:1380:4602:5c00::3/udp/4001/quic-v1/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb" +[...] +``` + +Finally connecting to the bootrap node shows us the protocols it supports. +Below example uses [`libp2p-lookup`](https://github.com/mxinden/libp2p-lookup/) but `ipfs swarm connect` followed by `ipfs id` can be used instead. + + +``` +libp2p-lookup direct --address /ip6/2604:1380:4602:5c00::3/udp/4001/quic-v1/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb + +Lookup for peer with id PeerId("QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb") succeeded. + +Protocol version: "ipfs/0.1.0" +Agent version: "kubo/0.20.0/b8c4725" +Listen addresses: + - "/ip6/2604:1380:4602:5c00::3/udp/4001/quic-v1" + - [...] +Protocols: + - /ipfs/kad/1.0.0 + - [...] +``` + +Note the `Agent version: "kubo/0.20.0/b8c4725"` and the supported protocols `Protocols: - /ipfs/kad/1.0.0`. + +# Motivation + +_Why run both Kubo and rust-libp2p2-server bootstrap nodes?_ + +This choice is influenced by three main areas: the benefit of diverse implementations, the opportunity to test rust-libp2p at large scale, and the presence of Rust in the IPFS network. + +Implementation Diversity: Operating both Kubo and rust-libp2p-server bootstrap nodes contributes to the network's overall resilience and security. It's like having a second line of defense; if one system encounters an issue, the other is there to continue functioning. For instance, a recent bug impacted Kubo IPFS bootstrap nodes, closing incoming connections, right after their successful establishment, due to a QUIC version mismatch. By using both Kubo and rust-libp2p-server, we ensure that nodes can still join the network, even if one set of bootstrap nodes is unavailable. + +Testing Rust-Libp2p at Large Scale: Our use of rust-libp2p-server also provides a valuable opportunity to examine how it behaves at a larger scale. Software performance can vary depending on scale, and these differences are hard to predict without actual real-world deployments. Now we can gain insights similar to those we acquired from other large deployments such as [Polkadot](github.com/paritytech/polkadot/) and [Ethereum](https://blog.libp2p.io/libp2p-and-ethereum/). + +Encouraging Rust in the IPFS Network: Lastly, by operating a rust-libp2p bootstrap node, we hope to motivate other developers to build IPFS-based applications using rust-libp2p. This could lead to an increase in the use of Rust, fostering a more diverse and vibrant ecosystem. + +# rust-libp2p(-server) in Action + +_What is rust-libp2p(-server) and how does it operate as an IPFS bootstrap node?_ + +[rust-libp2p](https://github.com/libp2p/rust-libp2p) is an implementation of the libp2p specification in Rust, a popular systems programming language. The rust-libp2p project was [initiated around 2018](https://www.parity.io/blog/why-libp2p) and since then, it has powered network like Ethereum through its Rust implementation [Lighthouse](https://github.com/sigp/lighthouse) and [Polkadot](github.com/paritytech/polkadot/) along with the [Substrate](https://github.com/paritytech/substrate/) ecosystem. You can find more rust-libp2p users [here](https://github.com/libp2p/rust-libp2p#notable-users). + +[rust-libp2p-server](https://github.com/mxinden/rust-libp2p-server/) is just thin wrapper around rust-libp2p. It combines rust-libp2p's TCP, QUIC and Kademlia-DHT implementation into a single binary. Looking up the new rust-libp2p-server IPFS bootstrap node `ny5` via [`libp2p-lookup`](https://github.com/mxinden/libp2p-lookup/) confirms just that. Note the `Agent version: "rust-libp2p-server/0.12.0"`. and `Protocols: - /ipfs/kad/1.0.0`. + + +``` +libp2p-lookup direct --address /dnsaddr/ny5.bootstrap.libp2p.io + +Lookup for peer with id PeerId("QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa") succeeded. + +Protocol version: "ipfs/0.1.0" +Agent version: "rust-libp2p-server/0.12.0" +Listen addresses: + - [...] +Protocols: + - /ipfs/kad/1.0.0 + - [...] +``` + +## Some Numbers + +On the new bootstrap node we see around 15 new inbound connections per second. The majority of these connections are established via QUIC (see `ip4/udp/quic`). + +![rust-libp2p bootstrap node establishing its first connections](../assets/2023-07-rust-libp2p-based-ipfs-bootstrap-node-new-incoming-connections.png) + +The node is handling > 30k connections concurrently, thus being connected to roughly [15% of nodes of the public IPFS DHT](https://probelab.io/ipfsdht/#client-vs-server-node-estimate). + +![rust-libp2p bootstrap node establishing its first connections](../assets/2023-07-rust-libp2p-based-ipfs-bootstrap-node-connections-established.png) + +Across these connections the node handles around 40 Kademlia requests per second, most of which are Kademlia `FIND_NODE` requests. + +![rust-libp2p bootstrap node establishing its first connections](../assets/2023-07-rust-libp2p-based-ipfs-bootstrap-node-new-incoming-kademlia-requests.png) + +The number of connections does not have a significant impact on CPU usage of the underlying machine. + +![rust-libp2p bootstrap node establishing its first connections](../assets/2023-07-rust-libp2p-based-ipfs-bootstrap-node-cpu.png) + +The node uses `< 300 kbyte` of memory per connection. + +![rust-libp2p bootstrap node establishing its first connections](../assets/2023-07-rust-libp2p-based-ipfs-bootstrap-node-memory.png) + +A small tangent: in case you are interested in more IPFS public DHT metrics, take a look at the [probelab DHT metrics and reports](https://probelab.io/ipfsdht/). + +# Closing + +If you want to learn more: + +- Read up on the [libp2p project](https://libp2p.io/) +- Explore the [rust-libp2p implementation](https://github.com/libp2p/rust-libp2p) +- See the thin rust-libp2p wrapper at [mxinden/rust-libp2p-server](https://github.com/mxinden/rust-libp2p-server/) +- And lastly, the [public IPFS DHT measurements](https://probelab.io/ipfsdht/) are always a good read + +A lot of this work was done by [@mcamou](https://github.com/mcamou) from the [Protocol Labs EngRes Bifrost team](https://pl-strflt.notion.site/Bifrost-2423fee6b15243158e85e35d8e22241d?pvs=4). Mario has handled the deployment and the team is operating the bootstrap nodes as a whole. Thanks, [@mcamou](https://github.com/mcamou) and team! + +FAQ: + +- Do I have to use the default bootstrap nodes? + + No, you don't have to use `/dnsaddr/bootstrap.libp2p.io`. You can remove Protocol Labs' default nodes and add your own or use both for better reliability. + +- Do we plan to run rust-libp2p-server on all IPFS bootstrap nodes? + + No. diff --git a/src/assets/2023-07-rust-libp2p-based-ipfs-bootstrap-node-connections-established.png b/src/assets/2023-07-rust-libp2p-based-ipfs-bootstrap-node-connections-established.png new file mode 100644 index 00000000..3eab6012 Binary files /dev/null and b/src/assets/2023-07-rust-libp2p-based-ipfs-bootstrap-node-connections-established.png differ diff --git a/src/assets/2023-07-rust-libp2p-based-ipfs-bootstrap-node-cpu.png b/src/assets/2023-07-rust-libp2p-based-ipfs-bootstrap-node-cpu.png new file mode 100644 index 00000000..bbb14814 Binary files /dev/null and b/src/assets/2023-07-rust-libp2p-based-ipfs-bootstrap-node-cpu.png differ diff --git a/src/assets/2023-07-rust-libp2p-based-ipfs-bootstrap-node-memory.png b/src/assets/2023-07-rust-libp2p-based-ipfs-bootstrap-node-memory.png new file mode 100644 index 00000000..6a9f2cd8 Binary files /dev/null and b/src/assets/2023-07-rust-libp2p-based-ipfs-bootstrap-node-memory.png differ diff --git a/src/assets/2023-07-rust-libp2p-based-ipfs-bootstrap-node-new-incoming-connections.png b/src/assets/2023-07-rust-libp2p-based-ipfs-bootstrap-node-new-incoming-connections.png new file mode 100644 index 00000000..291b4a22 Binary files /dev/null and b/src/assets/2023-07-rust-libp2p-based-ipfs-bootstrap-node-new-incoming-connections.png differ diff --git a/src/assets/2023-07-rust-libp2p-based-ipfs-bootstrap-node-new-incoming-kademlia-requests.png b/src/assets/2023-07-rust-libp2p-based-ipfs-bootstrap-node-new-incoming-kademlia-requests.png new file mode 100644 index 00000000..4dd788f7 Binary files /dev/null and b/src/assets/2023-07-rust-libp2p-based-ipfs-bootstrap-node-new-incoming-kademlia-requests.png differ