adam/ipfs-blog
1
0
Fork 0
mirror of https://github.com/ipfs/ipfs-blog.git synced 2026-08-08 09:53:11 +02:00
Code Issues Packages Projects Releases Wiki Activity

Fourth try...

Browse Source
This commit is contained in:
dame.eth
2023-05-08 14:31:15 -04:00
committed by GitHub
parent d816906631
commit dc34871f80
1 changed files with 3 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/_blog/2023-05-ipfs-unresponsive-nodes-incident.md
+3 -3
View File
@@ -54,7 +54,7 @@ Through a combination of crawling the network and attempting connections to all
![output.png](../assets/ipfs-unresponsive-nodes-incident/output.png)
<br>
2. After further investigation and given the very large percentage of nodes that were affected by the resource manager misconfiguration, we started looking into the impact of the incident to the GET performance.
<br>
A GET request that hits one of the affected, unresponsive nodes would get the connection shut down by the remote, but would get stuck there until it timed out, at which point it would re-issue the request to another peer. The relatively high concurrency factor of the IPFS DHT (`alpha = 10`) helps in this case, as it means that for any given request up to 10 concurrent requests can be in flight. This helps a lot even with a high percentage of unresponsive nodes as it means that at least one of the 10 peers contacted will respond.
<br>
> This is because the DHT lookup from the GET operation terminates when it hits one of the 20 closest peers to the target key, when the PUT operation terminates when it has found all the 20 closest peers.
@@ -75,8 +75,8 @@ Through a combination of crawling the network and attempting connections to all
![output.png](../assets/ipfs-unresponsive-nodes-incident/output_1.png)
<br>
4. We also experimented with even higher concurrency factors, in particular with `alpha = 20`, as a potential mitigation strategy. We repeated the same experiment with one extra set of runs: the case where we interact with all nodes in the network (i.e., we do not ignore unresponsive peers), but have higher concurrency factor.
<br>
We found that the performance increased and went back to pre-incident levels. However, it was decided *not* to go down this path, as the increased concurrency factor would: i) increase significantly the overhead/traffic in the DHT network, and, ii) stick with nodes that do not upgrade later on (when the incident is resolved) giving a clear advantage advantage to those nodes.
<br>We found that the performance increased and went back to pre-incident levels. However, it was decided *not* to go down this path, as the increased concurrency factor would: i) increase significantly the overhead/traffic in the DHT network, and, ii) stick with nodes that do not upgrade later on (when the incident is resolved) giving a clear advantage advantage to those nodes.
## 🚑 Mitigation: how we stopped the bleeding.