adam/ipfs-blog
1
0
Fork 0
mirror of https://github.com/ipfs/ipfs-blog.git synced 2026-08-08 18:03:04 +02:00
Code Issues Packages Projects Releases Wiki Activity

Update src/_blog/2023-05-ipfs-unresponsive-nodes-incident.md

Browse Source 
Co-authored-by: Steve Loeppky <biglep@protocol.ai>
This commit is contained in:
Yiannis Psaras
2023-05-07 21:13:08 +03:00
committed by GitHub
parent b2874c0b81
commit 128e9bce31
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/_blog/2023-05-ipfs-unresponsive-nodes-incident.md
+1 -1
View File
@@ -13,7 +13,7 @@ tags:
It depends on what type of system/network you’re running. In 90% of networks, or networked systems, this is a grand-scale disaster. Alerts are popping up everywhere, engineers go far beyond “day-time work” to get things back to normal, customers are panicking and potentially leaving the platform and the customer care lines are on fire. Half of the network is a large fraction, but I would bet that the same would happen even when 10% or 20% of the network experiences outage, or malfunction.
It’s not like that when you run your services on a decentralized, distributed P2P network, such as IPFS! At the beginning of 2023 a central component of the IPFS network, namely the public IPFS DHT experienced a large-scale incident. *During this incident, [60% of the IPFS DHT Server nodes became unresponsive](https://github.com/protocol/network-measurements/blob/master/reports/2023/calendar-week-04/ipfs/plots/crawl-unresponsive.png).* Interestingly, no content became unreachable and almost nothing in the network looked like the majority of the network was basically down. We did observe a significant increase in the content routing/resolution latency, in the order of 25% initially, but this definitely didn’t reflect the scale of the event.
It’s not like that when you run your services on a decentralized, distributed P2P network, such as IPFS! At the beginning of 2023 a critical component of the IPFS network, namely the public IPFS DHT, experienced a large-scale incident. *During this incident, [60% of the IPFS DHT Server nodes became unresponsive](https://github.com/protocol/network-measurements/blob/master/reports/2023/calendar-week-04/ipfs/plots/crawl-unresponsive.png).* Interestingly, no content became unreachable and almost nothing in the network looked like the majority of the network was basically down. We did observe a significant increase in the content routing/resolution latency, in the order of 25% initially, but this definitely didn’t reflect the scale of the event.
In this blogpost we’ll go through the timeline of the event from “Incident Detection” to “Root Cause Analysis” and give details of the engineering team’s response, as well as of the items that the team has been monitoring throughout the incident.