adam/ipfs-blog
1
0
Fork 0
mirror of https://github.com/ipfs/ipfs-blog.git synced 2026-08-09 02:13:04 +02:00
Code Issues Packages Projects Releases Wiki Activity

Update 2023-05-ipfs-unresponsive-nodes-incident.md

Browse Source 
Made some relatively minor changes for syntax, flow, and clarity. Also tweaked the title of the post due to length (it was overflowing in the preview), and adjusted some of the headers.
This commit is contained in:
dame.eth
2023-05-08 10:39:07 -04:00
committed by GitHub
parent f6989a2f77
commit 7756cb2d24
1 changed files with 32 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/_blog/2023-05-ipfs-unresponsive-nodes-incident.md
+32 -31
View File
@@ -1,54 +1,55 @@
---
title: What happens when half of the nodes in your network are unresponsive?
description: "The IPFS DHT experienced a serious incident in the beginning of 2023 and users hardly noticed - that's the power of a decentralized network!"
title: What happens when half of the nodes are unresponsive?
description: "The IPFS DHT experienced a problem in the beginning of 2023, but users hardly noticed thanks to the power of a decentralized network!"
author: Yiannis Psaras
date: 2023-05-05
permalink: '/ipfs-unresponsive-nodes-incident/'
permalink: '/2023-ipfs-unresponsive-nodes/'
header_image: '/2023-05-ipfs-unresponsive-nodes-incident.jpeg'
tags:
- 'ipfs dht'
- 'incident'
- 'dht'
- 'decentralization'
- 'resource manager'
- 'nodes'
---
It depends on what type of system/network you’re running. In 90% of networks, or networked systems, this is a grand-scale disaster. Alerts are popping up everywhere, engineers go far beyond “day-time work” to get things back to normal, customers are panicking and potentially leaving the platform and the customer care lines are on fire. Half of the network is a large fraction, but I would bet that the same would happen even when 10% or 20% of the network experiences outage, or malfunction.
It depends on what type of system/network you’re running. In 90% of networks, or networked systems, this is a grand-scale disaster. Alerts are popping up everywhere, engineers go far beyond “day-time work” to get things back to normal, customers are panicking and potentially leaving the platform and the customer care lines are on fire. Half of the network is a large fraction, but I would bet that the same would happen even when 10% or 20% of the network experiences an outage.
It’s not like that when you run your services on a decentralized, distributed P2P network, such as IPFS! At the beginning of 2023 a critical component of the IPFS network, namely the public IPFS DHT, experienced a large-scale incident. *During this incident, [60% of the IPFS DHT Server nodes became unresponsive](https://github.com/protocol/network-measurements/blob/master/reports/2023/calendar-week-04/ipfs/plots/crawl-unresponsive.png).* Interestingly, no content became unreachable and almost nothing in the network looked like the majority of the network was basically down. We did observe a significant increase in the content routing/resolution latency, in the order of 25% initially, but this definitely didn’t reflect the scale of the event.
It’s not like that when you run your services on a decentralized, distributed P2P network, such as IPFS! At the beginning of 2023, a critical component of the IPFS network, namely the public IPFS DHT, experienced a large-scale malfunction. *During this situation, [60% of the IPFS DHT Server nodes became unresponsive](https://github.com/protocol/network-measurements/blob/master/reports/2023/calendar-week-04/ipfs/plots/crawl-unresponsive.png).* Interestingly, **no content became unreachable and almost nothing in the network looked like the majority of the network was basically down**. We did observe a significant increase in the content routing/resolution latency (in the order of 25% initially), but this in no way reflected the scale of the event.
In this blogpost we’ll go through the timeline of the event from “Incident Detection” to “Root Cause Analysis” and give details of the engineering team’s response, as well as of the items that the team has been monitoring throughout the incident. A summarizing talk on the content of this blogpost was given at [IPFS Thing 2023](https://2023.ipfs-thing.io/) and can be found [here](https://youtu.be/8cGEjdCfm14).
In this blog post, we’ll go through the timeline of the event from “Detection” to “Root Cause Analysis” and give details about the engineering team’s response. A summarizing talk on the content of this blog post was given at [IPFS Thing 2023](https://2023.ipfs-thing.io/) and can be found [on YouTube](https://youtu.be/8cGEjdCfm14).
## ❗️ Incident Detection: we’ve got a problem
## ❗️ Detection: we found a problem!
> At the beginning of 2023 a central component of the IPFS network, namely the public IPFS DHT, experienced a large-scale incident. *During this incident, [60% of the IPFS DHT Server nodes became unresponsive](https://github.com/protocol/network-measurements/blob/master/reports/2023/calendar-week-04/ipfs/plots/crawl-unresponsive.png).*
> At the beginning of 2023, a critical component of the IPFS network, namely the public IPFS DHT, experienced a large-scale malfunction. *During this situation, [60% of the IPFS DHT Server nodes became unresponsive](https://github.com/protocol/network-measurements/blob/master/reports/2023/calendar-week-04/ipfs/plots/crawl-unresponsive.png).*
>
Unresponsive here means that nodes would seem to be online, they would accept connections from other nodes, but they wouldn’t reply to requests. Basically, when a node would try to write to one of the unresponsive nodes, the unresponsive node would terminate the connection immediately.
Given that these nodes seemed to be functional, they occupied several places in other nodes’ routing tables, when in fact they shouldn’t have.
The problem was all down to a misconfiguration of the go-libp2p resource manager - a new feature that shipped with `kubo-v0.17`. The problematic configuration, which was applied manually, i.e., was not based on the default values of `kubo-v0.17`, was set to such values that basically any attempt to interact with the nodes would be flagged as a resource exhaustion event and would trigger the corresponding “defense” mechanism. In practice, this materialized as a connection tear-down. It is worth noting that `kubo` is the most prevalent IPFS implementation using the public IPFS DHT with ~80% of nodes in the DHT being `kubo` nodes (see most recent [stats](https://github.com/protocol/network-measurements/tree/master/reports/2023/calendar-week-17/ipfs#agent-version-analysis)).
The problem came down to a misconfiguration of the go-libp2p resource manager - a new feature that shipped with `kubo-v0.17`. The problematic configuration which was applied manually (i.e. was not based on the default values of `kubo-v0.17`) was set to such values that any attempt to interact with the nodes would be flagged as a resource exhaustion event and would trigger the corresponding “defense” mechanism. In practice, this materialized as a connection tear-down. It is worth noting that `kubo` is the most prevalent IPFS implementation using the public IPFS DHT with ~80% of nodes in the DHT being `kubo` nodes (see most recent [stats](https://github.com/protocol/network-measurements/tree/master/reports/2023/calendar-week-17/ipfs#agent-version-analysis)).
Content was still findable through kubo, so no alarms were raised. However, some of our research teams have been observing unusual error messages:
Content was still findable through kubo, so no alarms were raised. However, some of our research teams observed unusual error messages:
```go
> Application error 0x0 (remote): conn-22188077: system: cannot reserve inbound
connection: resource limit exceeded
```
The error didn’t seem like one that would trigger widespread panic, since PUT and GET operations were completing successfully. We were seeing slower performance than normal and had been investigating whether [recent changes with Hydra boosters](https://discuss.ipfs.tech/t/dht-hydra-peers-dialling-down-non-bridging-functionality-on-2022-12-01/15567) had bigger impact than we were expecting. That was until a physical meeting of our engineering teams, where we allocated some “hack time” to play around with the codebase and one of the items on the agenda was to figure out where this error comes from.
Since PUT and GET operations were completing successfully, the error didn’t seem like one that would trigger widespread panic. We were seeing slower performance than normal and had been investigating whether [recent changes with Hydra boosters](https://discuss.ipfs.tech/t/dht-hydra-peers-dialling-down-non-bridging-functionality-on-2022-12-01/15567) had bigger impacts than we were expecting. It was at this time that we had a physical meeting of our engineering teams and one of the items on the agenda was to figure out where this error was coming from.
## ❓ Incident Diagnosis: what is happening
## ❓ Diagnosis: what was happening?
We quickly realized that [this was a resource manager issue where the remote node is hitting a limit and closing the connection](https://github.com/libp2p/go-libp2p/issues/1928). After some more digging and looking into the details of the resource manager and the error itself (i.e., `cannot reserve **in**bound connection`), we understood the root cause of the issue within the resource manager and that it was related to the remote node. It turns out that the resource manager was manually misconfigured by a very large percentage of nodes to values that were not in the default configuration by the “vanilla” version of the resource manager that shipped with `kubo-v0.17`.
We quickly realized that [there was a resource manager issue where the remote node was hitting a limit and closing the connection](https://github.com/libp2p/go-libp2p/issues/1928). After looking into the details of the resource manager and the error itself (i.e., `cannot reserve **in**bound connection`), we realized that the root cause of the issue was related to the remote node. It turned out that the resource manager was manually misconfigured by a very large percentage of nodes to values that were not in the default configuration by the “vanilla” version of the resource manager that shipped with `kubo-v0.17`.
As mentioned earlier, the GET and PUT operations were completing successfully, so the next step was to identify the scale of the problem. Our main goals were to figure out:
As mentioned earlier, the GET and PUT operations were completing successfully, so our next step was to identify the scale of the problem. Our main goals were to figure out:
- what percentage of nodes in the network were affected,
- if there is a performance penalty in either the PUT or the GET operation, or both.
- what percentage of nodes in the network were affected
- if there was a performance penalty in either the PUT or the GET operation, or both
Through a combination of crawling the network and attempting connections to all ~50k DHT Server nodes (i.e., those that store and serve provider records and content), we found that close to 60% of the network has been affected by this misconfiguration. Clearly this is a very large percentage of the network, which made it urgent to look into the performance impact. We followed the below methodology:
Through a combination of crawling the network and attempting connections to all ~50k DHT Server nodes (i.e., those that store and serve provider records and content), we found that close to 60% of the network had been affected by the misconfiguration. Clearly this was a very large percentage of the network, which made it urgent to look into the performance impact. We followed the below methodology:
1. As a first step, we wanted to figure out which buckets in the nodes’ routing tables do the affected nodes occupy. We found that they occupied the higher buckets of the nodes’ routing tables, which further meant that most likely PUTs would get slower, but GETs should not be affected too much. This is because the DHT lookup from the GET operation terminates when it hits *one* of the 20 closest peers to the target key, while the PUT operation terminates when it has found *all* the 20 closest peers. Since a significant portion of the network is unresponsive, the PUT operation certainly hits at least one unresponsive node, but the GET operation has good chances of finding at least one responsive node within the 20 closest.
1. We wanted to figure out which buckets in the nodes’ routing tables did the affected nodes occupy. We found that they occupied the higher buckets of the nodes’ routing tables, which meant that most likely PUTs would get slower, but GETs should not be affected too much. This is because the DHT lookup from the GET operation terminates when it hits *one* of the 20 closest peers to the target key, while the PUT operation terminates when it has found *all* the 20 closest peers. Since a significant portion of the network was unresponsive, the PUT operation hit at least one unresponsive node, but the GET operation had good chances of finding at least one responsive node within the 20 closest.
![output.png](../assets/ipfs-unresponsive-nodes-incident/output.png)
@@ -77,15 +78,15 @@ Through a combination of crawling the network and attempting connections to all
- The GET operation was also disrupted (in contrast to our initial assumption) and was slowed down by approximately 15%, at times reaching closer to 20%.
![output.png](../assets/ipfs-unresponsive-nodes-incident/output_1.png)
![output.png](../assets/ipfs-unresponsive-nodes-incident/output_1.png)
4. We also experimented with even higher concurrency factors, in particular with `alpha = 20`, as a potential mitigation strategy. We repeated the same experiment with one extra set of runs: the case where we interact with all nodes in the network (i.e., we do not ignore unresponsive peers), but have higher concurrency factor.
We found that the performance indeed increases and goes back to pre-incident levels. However, it was decided *not* to go down this path, as the increased concurrency factor would: i) increase significantly the overhead/traffic in the DHT network, and, ii) stick with nodes that do not upgrade later on (when the incident is resolved) giving a clear advantage advantage to those nodes.
We found that the performance increased and went back to pre-incident levels. However, it was decided *not* to go down this path, as the increased concurrency factor would: i) increase significantly the overhead/traffic in the DHT network, and, ii) stick with nodes that do not upgrade later on (when the incident is resolved) giving a clear advantage advantage to those nodes.
## 🚑 Event Mitigation: stop the bleeding
## 🚑 Mitigation: how we stopped the bleeding.
The team’s immediate focus became:
@@ -106,19 +107,19 @@ We also monitored the situation compared to the pre-incident performance by runn
## 🔧 Addressing the Root Cause
Our immediate actions managed to stop the bleeding and bring the network back to normal pretty quickly. However, it was clear that we had to implement longer term fixes to protect the nodes’ routing tables from unresponsive peers and to avoid inadvertently making nodes unresponsive. Specifically this translated to:
Our immediate actions managed to stop the bleeding and bring the network back to normal quickly. However, it was clear that we had to implement longer term fixes to protect the nodes’ routing tables from unresponsive peers and to avoid inadvertently making nodes unresponsive. Specifically this translated to:
1. Revamped the Kubo resource manager UX to further reduce the likelihood of catastrophic misconfiguration. This was completed in [Kubo 0.19](https://github.com/ipfs/kubo/releases/tag/v0.19.0#improving-the-libp2p-resource-management-integration).
2. Only add peers to the routing table that are responsive requests [during the routing table refresh](https://github.com/libp2p/go-libp2p-kad-dht/pull/810) (done) and [upon adding a node to the routing table](https://github.com/libp2p/go-libp2p-kad-dht/issues/811) (in progress - targeting [Kubo 0.21 in May](https://github.com/ipfs/kubo/issues/9814)).
1. Revamping the Kubo resource manager UX to further reduce the likelihood of catastrophic misconfiguration. This was completed in [Kubo 0.19](https://github.com/ipfs/kubo/releases/tag/v0.19.0#improving-the-libp2p-resource-management-integration).
2. Only adding peers to the routing table that are responsive requests [during the routing table refresh](https://github.com/libp2p/go-libp2p-kad-dht/pull/810) (done) and [upon adding a node to the routing table](https://github.com/libp2p/go-libp2p-kad-dht/issues/811) (in progress - targeting [Kubo 0.21 in May](https://github.com/ipfs/kubo/issues/9814)).
## 📖 Lessons Learned
There are several lessons to be learned from this incident.
In the days since, we have come away from this experience with several important learnings:
🗒️ Significant fundamental changes to the codebase like retroactively adding resource accounting is fraught for disruption. This increases the necessity for documentation, announcements and clear recommendations to node operators.
🗒️ Significant fundamental changes to the codebase (such as retroactively adding resource accounting) is ripe for disruption. This increases the necessity for documentation, announcements, and clear recommendations to node operators.
⏱️ Monitoring software should always be in place to help identify such events from the get go.
⏱️ Monitoring software should always be in place to help identify such events from the start.
📣 It is challenging to monitor and apply changes directly to the software that runs on nodes of a decentralized network. Well-established communication channels go a long way and help the engineering teams make announcements and communicate directly with the community. In IPFS we use a variety of channels, including the IPFS Discord Server [[invite link](https://discord.gg/ipfs)], Filecoin Slack [[invite link](https://filecoin.io/slack)] (mostly in `#engres-ip-stewards` channel), the [discourse discussion forum](https://discuss.ipfs.tech/), and the [IPFS blog](https://blog.ipfs.tech/).
📣 It is challenging to monitor and apply changes directly to the software that runs on nodes of a decentralized network. Well-established communication channels go a long way and help the engineering teams communicate directly with the community. In IPFS, we use a variety of channels including the Discord Server [[invite link](https://discord.gg/ipfs)], Filecoin Slack [[invite link](https://filecoin.io/slack)] (mostly in `#engres-ip-stewards` channel), the [Discourse discussion forum](https://discuss.ipfs.tech/), and the [blog](https://blog.ipfs.tech/).
🚀 Last, but certainly not least, the decentralized, P2P nature of IPFS kept the network running, with all important operations completing successfully, albeit slower than normal. It is exactly because of the structure of the network that there are no single points of failure and performance is not catastrophically disrupted even when more than half of the network nodes are essentially unresponsive.
🚀 Last, but certainly not least, the decentralized, P2P nature of IPFS kept the network running with all important operations completing successfully (albeit slower than normal). It is exactly because of the structure of the network that there are no single points of failure and performance is not catastrophically disrupted even when more than half of the network nodes are essentially unresponsive.