From 0817df64f9707da83b899cb09056b536f31ec7b5 Mon Sep 17 00:00:00 2001 From: Ze Bateira Date: Thu, 8 Apr 2021 17:49:23 +0100 Subject: [PATCH 1/3] fix: remove extra newlines in 25-pubsub blog post --- src/_blog/014-pubsub.md | 75 +++++++---------------------------------- 1 file changed, 13 insertions(+), 62 deletions(-) diff --git a/src/_blog/014-pubsub.md b/src/_blog/014-pubsub.md index c529d684..725c3aaa 100644 --- a/src/_blog/014-pubsub.md +++ b/src/_blog/014-pubsub.md @@ -8,45 +8,27 @@ tags: - 'pubsub' --- -We recently merged a simple, experimental pubsub implementation into IPFS. This -implementation is just a beginning. It is far from the performance and security goals -we will achieve in our long-term target. However, even this early implementation opens -the doors to several useful and interesting new applications. +We recently merged a simple, experimental pubsub implementation into IPFS. This implementation is just a beginning. It is far from the performance and security goals we will achieve in our long-term target. However, even this early implementation opens the doors to several useful and interesting new applications. -In this post, I will point out some applications of this technology, show how to -get started started using `ipfs pubsub`, and discuss upcoming improvements. +In this post, I will point out some applications of this technology, show how to get started started using `ipfs pubsub`, and discuss upcoming improvements. ## Why pubsub? -[Publish-Subscribe](https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern), -called 'pubsub' for short, is a pattern often used to handle events in -large-scale networks. 'Publishers' send messages classified by topic or content and -'subscribers' receive only the messages they are interested in, all without direct -connections between publishers and subscribers. This approach offers much greater -network scalability and flexibility. +[Publish-Subscribe](https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern), called 'pubsub' for short, is a pattern often used to handle events in large-scale networks. 'Publishers' send messages classified by topic or content and 'subscribers' receive only the messages they are interested in, all without direct connections between publishers and subscribers. This approach offers much greater network scalability and flexibility. -Some applications include collaborative document editing, "dynamic" website -content, chat applications, multiplayer games, continuously evolving datasets, -and webservice workers passing around messages. It gives us ways to make IPFS fast -for large-scale networks such as datacenters, local area networks, and large p2p -applications. In the near future, IPNS records will be pushed over pubsub, allowing -lightning fast updates of peers' IPNS entries. Peers could use pubsub to track the -head of a [merkle-linked global log](https://en.wikipedia.org/wiki/Blockchain). +Some applications include collaborative document editing, "dynamic" website content, chat applications, multiplayer games, continuously evolving datasets, and webservice workers passing around messages. It gives us ways to make IPFS fast for large-scale networks such as datacenters, local area networks, and large p2p applications. In the near future, IPNS records will be pushed over pubsub, allowing lightning fast updates of peers' IPNS entries. Peers could use pubsub to track the head of a [merkle-linked global log](https://en.wikipedia.org/wiki/Blockchain). ## Getting started with pubsub for go-ipfs _Note: There is also a js-ipfs implementation of pubsub. Documentation will come soon._ -First, you'll need to enable the pubsub code. Make sure you're running go-ipfs 0.4.5 or -above. Once you have that version of ipfs installed, start the daemon with: +First, you'll need to enable the pubsub code. Make sure you're running go-ipfs 0.4.5 or above. Once you have that version of ipfs installed, start the daemon with: ```sh > ipfs daemon --enable-pubsub-experiment ``` -This will tell ipfs to create and enable the pubsub service. It also implies -that you will only be able to use pubsub with other peers who choose to enable -it. +This will tell ipfs to create and enable the pubsub service. It also implies that you will only be able to use pubsub with other peers who choose to enable it. To subscribe to the topic `foo`, run: @@ -62,14 +44,7 @@ To publish a message to the topic `foo`, open up another terminal and run: > ipfs pubsub pub foo "hello world" ``` -You should see "hello world" printed out in the first terminal. You can also -run the `pub` command on any other connected ipfs node and your node will -receive the message. Messages are routed through connected, subscribed peers. -This means that if peers A, B, and C are all subscribed to `foo`, A is connected -to B, and B is connected to C, but A is not directly connected to C, A will -still receive messages that C published to `foo` through B. This can be very -useful for routing messages in networks with poor NAT traversal or otherwise -suboptimal connectivity. +You should see "hello world" printed out in the first terminal. You can also run the `pub` command on any other connected ipfs node and your node will receive the message. Messages are routed through connected, subscribed peers. This means that if peers A, B, and C are all subscribed to `foo`, A is connected to B, and B is connected to C, but A is not directly connected to C, A will still receive messages that C published to `foo` through B. This can be very useful for routing messages in networks with poor NAT traversal or otherwise suboptimal connectivity. To see all peers with pubsub enabled, check the output of: @@ -85,44 +60,20 @@ To see all the topics you are currently subscribed to, run: ## Pubsub in the wild -As an example, we have integrated pubsub into [Orbit](https://github.com/orbitdb/orbit). -This allows Orbit to provide a fully distributed, peer-to-peer chat without _any_ -server anywhere. We are also actively working to put Conflict-Free Replicated Data -Types (CRDTs) on IPFS pubsub using libraries like Y.js and swarm.js. +As an example, we have integrated pubsub into [Orbit](https://github.com/orbitdb/orbit). This allows Orbit to provide a fully distributed, peer-to-peer chat without _any_ server anywhere. We are also actively working to put Conflict-Free Replicated Data Types (CRDTs) on IPFS pubsub using libraries like Y.js and swarm.js. -Together, pubsub and CRDTs open new doors for collaborative editing of distributed -content. We are working with [@edsilv](https://github.com/edsilv) and -[@aeschylus](https://github.com/aeschylus) to prepare a demo of IPFS in two -[IIIF](http://iiif.io/about/) image viewers -[#240](https://github.com/ipfs/notes/issues/240). This will showcase how to -collaboratively annotate images from repositories dispersed around the world. -The demo will take place at the [IIIF 2017 Conference](https://2017iiifconferencethevatican.sched.com/event/AChW/presentation-interoperable-peer-to-peer-research-with-iiif-and-ipfs-room-5.0) in The Vatican. -We will publish a video of the demo, along with all of the code. +Together, pubsub and CRDTs open new doors for collaborative editing of distributed content. We are working with [@edsilv](https://github.com/edsilv) and [@aeschylus](https://github.com/aeschylus) to prepare a demo of IPFS in two [IIIF](http://iiif.io/about/) image viewers [#240](https://github.com/ipfs/notes/issues/240). This will showcase how to collaboratively annotate images from repositories dispersed around the world. The demo will take place at the [IIIF 2017 Conference](https://2017iiifconferencethevatican.sched.com/event/AChW/presentation-interoperable-peer-to-peer-research-with-iiif-and-ipfs-room-5.0) in The Vatican. We will publish a video of the demo, along with all of the code. ## What's next? The next two areas of focus for IPFS pubsub are authentication and message routing. -Currently, any peer can publish to any pubsub topic. We plan to implement an -authenticated mode for pubsub topics, where only authorized peers — those given a -cryptographic key or capability — can publish messages. We are still working out -the sharing and capability granting model. +Currently, any peer can publish to any pubsub topic. We plan to implement an authenticated mode for pubsub topics, where only authorized peers — those given a cryptographic key or capability — can publish messages. We are still working out the sharing and capability granting model. -After that, we plan to improve message routing. The current routing algorithm -floods messages to every subscriber, resulting in some peers receiving the same -message multiple times. We affectionately call this approach "floodsub". We plan -to replace it with a more efficient routing algorithm, which will go a long way -towards reducing overhead and improving scalability. +After that, we plan to improve message routing. The current routing algorithm floods messages to every subscriber, resulting in some peers receiving the same message multiple times. We affectionately call this approach "floodsub". We plan to replace it with a more efficient routing algorithm, which will go a long way towards reducing overhead and improving scalability. -Please note that this is a simple first-blush implementation of the technology. -It has known limitations that we will address in future iterations. As it is -today, the pubsub implementation can be quite bandwidth intensive. It works well -for apps with few peers in the group, but does not scale. We have designed a more -robust underlying algorithm that will scale to much larger use cases but we wanted -to ship this simple implementation so you can begin using it for your applications. +Please note that this is a simple first-blush implementation of the technology. It has known limitations that we will address in future iterations. As it is today, the pubsub implementation can be quite bandwidth intensive. It works well for apps with few peers in the group, but does not scale. We have designed a more robust underlying algorithm that will scale to much larger use cases but we wanted to ship this simple implementation so you can begin using it for your applications. ## Enjoy! -All that said, we hope you give `ipfs pubsub` a try. You can head over to the -[Discussion Forum](https://discuss.ipfs.io/categories) to ask questions, get help, -or simply let us know how it goes. +All that said, we hope you give `ipfs pubsub` a try. You can head over to the [Discussion Forum](https://discuss.ipfs.io/categories) to ask questions, get help, or simply let us know how it goes. From e72aad810c4d31f005ddf4eb0a44c250deee971e Mon Sep 17 00:00:00 2001 From: Ze Bateira Date: Thu, 8 Apr 2021 17:56:34 +0100 Subject: [PATCH 2/3] fix: more posts with extra newlines fixed --- src/_blog/010-go-ipfs-0-4-4-released.md | 53 +++++-------------------- src/_blog/016-go-ipfs-0-4-10.md | 50 +++++------------------ src/_blog/023-go-ipfs-0.4.14.md | 48 ++++------------------ 3 files changed, 27 insertions(+), 124 deletions(-) diff --git a/src/_blog/010-go-ipfs-0-4-4-released.md b/src/_blog/010-go-ipfs-0-4-4-released.md index 8b21c869..1e4b7717 100644 --- a/src/_blog/010-go-ipfs-0-4-4-released.md +++ b/src/_blog/010-go-ipfs-0-4-4-released.md @@ -9,10 +9,7 @@ tags: - 'go-ipfs' --- -[go-ipfs 0.4.4](https://dist.ipfs.io/#go-ipfs) has been released today, -including an important hotfix for a bug we discovered in how _pinning_ works. -If you had a large number of pins, new pins would overwrite existing pins. -Apart from the hotfix, this release is equal to the previous release 0.4.3. +[go-ipfs 0.4.4](https://dist.ipfs.io/#go-ipfs) has been released today, including an important hotfix for a bug we discovered in how _pinning_ works. If you had a large number of pins, new pins would overwrite existing pins. Apart from the hotfix, this release is equal to the previous release 0.4.3. - [How pinning works](#how-pinning-works) - [The bug](#the-bug) @@ -21,9 +18,7 @@ Apart from the hotfix, this release is equal to the previous release 0.4.3. ## How pinning works -Pinning is a means of persisting data in your IPFS repo after adding or fetching it. -It'll prevent objects from getting removed by garbage collection or other methods -of cleaning up the IPFS repo. There are three ways an object can be pinned: +Pinning is a means of persisting data in your IPFS repo after adding or fetching it. It'll prevent objects from getting removed by garbage collection or other methods of cleaning up the IPFS repo. There are three ways an object can be pinned: - **Direct:** Only this object is pinned. Its children aren't pinned. - **Recursive:** This object and all its children are pinned. @@ -31,54 +26,26 @@ of cleaning up the IPFS repo. There are three ways an object can be pinned: - **Indirect:** This object is pinned because one of its parents is pinned. If the pins of all parents are removed, this object isn't pinned any longer. -The `ipfs add` command adds a _recursive pin_ for the added file by default. -With `--pin=false`, it skips pinning. Similarly, the default pin type for -`ipfs pin add` is _recursive_. With `--recursive=false` this changes to _direct_. +The `ipfs add` command adds a _recursive pin_ for the added file by default. With `--pin=false`, it skips pinning. Similarly, the default pin type for `ipfs pin add` is _recursive_. With `--recursive=false` this changes to _direct_. For more information on how pinning works, check out `ipfs pin --help` and `ipfs add --help`. ## The bug -Direct and recursive pins are stored in separate so-called _pinsets_. -Indirect pins aren't stored, since they're derived from recursive pins. +Direct and recursive pins are stored in separate so-called _pinsets_. Indirect pins aren't stored, since they're derived from recursive pins. -Once you had more than 8192 pins, recursive or direct, an -issue with the recursive hash trie implementation caused hash table buckets to -be overwritten, resulting in only 256 pins remaining in the pinset. After that, -the bug wouldn't be triggered again until the number of pins exceeded 8192 again. -The 256 pins that remained would be random. +Once you had more than 8192 pins, recursive or direct, an issue with the recursive hash trie implementation caused hash table buckets to be overwritten, resulting in only 256 pins remaining in the pinset. After that, the bug wouldn't be triggered again until the number of pins exceeded 8192 again. The 256 pins that remained would be random. -We fixed this by instead making sure that each item in a pinset be put into its -own bucket, and by modulo'ing hash output from this process into the final key -space. The details for this can be seen -[in this pull request](https://github.com/ipfs/go-ipfs/pull/3273). We added a -stress test to make sure that this doesn't happen in the future, and will -redouble our efforts to make sure that our test suites are more robust to ensure -that these kinds of problems do not happen in the future. +We fixed this by instead making sure that each item in a pinset be put into its own bucket, and by modulo'ing hash output from this process into the final key space. The details for this can be seen [in this pull request](https://github.com/ipfs/go-ipfs/pull/3273). We added a stress test to make sure that this doesn't happen in the future, and will redouble our efforts to make sure that our test suites are more robust to ensure that these kinds of problems do not happen in the future. -For now, don't run `ipfs repo gc` on sensitive data that is not otherwise backed up, -as IPFS is still not 1.0 and our development may still find problems. +For now, don't run `ipfs repo gc` on sensitive data that is not otherwise backed up, as IPFS is still not 1.0 and our development may still find problems. -[@kyledrake](https://github.com/kyledrake) of [neocities.org](https://neocities.org) -pointed out this bug to us; thank you, Kyle! +[@kyledrake](https://github.com/kyledrake) of [neocities.org](https://neocities.org) pointed out this bug to us; thank you, Kyle! ## Find out if you're affected -If you think you have experienced this issue and have _not_ run a garbage -collection, you can still find the 'lost' pins. We have written a new tool -called 'ipfs-see-all' that allows you to try and recover any old pins that are -still in your local repo. The tool is available on [our distributions -page](https://dist.ipfs.io), or, if you prefer building from source, head over -to [the GitHub repo](https://github.com/whyrusleeping/ipfs-see-all). Once you -have the tool, invoke it as `ipfs-see-all lost-pins` and it will scan for and -print out every pin object that is not actually pinned in your pinset. Note -that this may contain anything you have manually unpinned. +If you think you have experienced this issue and have _not_ run a garbage collection, you can still find the 'lost' pins. We have written a new tool called 'ipfs-see-all' that allows you to try and recover any old pins that are still in your local repo. The tool is available on [our distributions page](https://dist.ipfs.io), or, if you prefer building from source, head over to [the GitHub repo](https://github.com/whyrusleeping/ipfs-see-all). Once you have the tool, invoke it as `ipfs-see-all lost-pins` and it will scan for and print out every pin object that is not actually pinned in your pinset. Note that this may contain anything you have manually unpinned. ## How to upgrade -Depending on how you initially installed IPFS, there are several ways to -upgrade. If you installed IPFS with a pre-built binary, you can head over -to [dist.ipfs.io](https://dist.ipfs.io/#go-ipfs) and grab the latest version -from there. Or alternatively, from the same page you can grab the `ipfs-update` -binary, and use it to perform the upgrade for you. If you installed from -source, you can simply run `git checkout v0.4.4`, then run `make install`. +Depending on how you initially installed IPFS, there are several ways to upgrade. If you installed IPFS with a pre-built binary, you can head over to [dist.ipfs.io](https://dist.ipfs.io/#go-ipfs) and grab the latest version from there. Or alternatively, from the same page you can grab the `ipfs-update` binary, and use it to perform the upgrade for you. If you installed from source, you can simply run `git checkout v0.4.4`, then run `make install`. diff --git a/src/_blog/016-go-ipfs-0-4-10.md b/src/_blog/016-go-ipfs-0-4-10.md index 2498ba84..18daca4d 100644 --- a/src/_blog/016-go-ipfs-0-4-10.md +++ b/src/_blog/016-go-ipfs-0-4-10.md @@ -8,51 +8,21 @@ tags: - 'go-ipfs' --- -Yesterday was the [3rd birthday of go-ipfs](https://github.com/ipfs/go-ipfs/commit/9d0e6a7ffb5deea2c8c8e555d7bf6bcab6fdc6ac). -To celebrate, we would like to share some virtual cake 🎂 with ipfs users and -contributors from all over the world, and also announce the release of -go-ipfs 0.4.10. +Yesterday was the [3rd birthday of go-ipfs](https://github.com/ipfs/go-ipfs/commit/9d0e6a7ffb5deea2c8c8e555d7bf6bcab6fdc6ac). To celebrate, we would like to share some virtual cake 🎂 with ipfs users and contributors from all over the world, and also announce the release of go-ipfs 0.4.10. -go-ipfs 0.4.10 is a patch release that contains several exciting new features, -bugfixes and general improvements. This includes new commands, easier corruption -recovery, and a generally cleaner codebase. +go-ipfs 0.4.10 is a patch release that contains several exciting new features, bugfixes and general improvements. This includes new commands, easier corruption recovery, and a generally cleaner codebase. -The `ipfs pin` command has two new subcommands, `verify` and `update`. `ipfs pin verify` is used to scan the repo for pinned object graphs and check their -integrity. Any issues are reported back with helpful error text to make error -recovery simpler. This subcommand was added to help recover from datastore -corruptions, particularly if using the experimental filestore and accidentally -deleting tracked files. +The `ipfs pin` command has two new subcommands, `verify` and `update`. `ipfs pin verify` is used to scan the repo for pinned object graphs and check their integrity. Any issues are reported back with helpful error text to make error recovery simpler. This subcommand was added to help recover from datastore corruptions, particularly if using the experimental filestore and accidentally deleting tracked files. -`ipfs pin update` was added to make the task of keeping a large, frequently -changing object graph pinned. Previously users had to call `ipfs pin rm` on the -old pin, and `ipfs pin add` on the new one. The 'new' `ipfs pin add` call would -be very expensive as it would need to verify the entirety of the graph again. -The `ipfs pin update` command takes shortcuts, portions of the graph that were -covered under the old pin are assumed to be fine, and the command skips -checking them. +`ipfs pin update` was added to make the task of keeping a large, frequently changing object graph pinned. Previously users had to call `ipfs pin rm` on the old pin, and `ipfs pin add` on the new one. The 'new' `ipfs pin add` call would be very expensive as it would need to verify the entirety of the graph again. The `ipfs pin update` command takes shortcuts, portions of the graph that were covered under the old pin are assumed to be fine, and the command skips checking them. -Next up, we have finally implemented an `ipfs shutdown` command so users can -shut down their ipfs daemons via the API. This is especially useful on -platforms that make it difficult to control processes (Android, for example), -and is also useful when needing to shut down a node remotely and you do not -have access to the machine itself. +Next up, we have finally implemented an `ipfs shutdown` command so users can shut down their ipfs daemons via the API. This is especially useful on platforms that make it difficult to control processes (Android, for example), and is also useful when needing to shut down a node remotely and you do not have access to the machine itself. -`ipfs add` has gained a new flag; the `--hash` flag allows you to select which -hash function to use and we have given it the ability to select `blake2b-256`. -This pushes us one step closer to shifting over to using blake2b as the -default. Blake2b is significantly faster than sha2-256, and also is conjectured -to provide superior security. +`ipfs add` has gained a new flag; the `--hash` flag allows you to select which hash function to use and we have given it the ability to select `blake2b-256`. This pushes us one step closer to shifting over to using blake2b as the default. Blake2b is significantly faster than sha2-256, and also is conjectured to provide superior security. -We have also finally implemented a very early (and experimental) `ipfs p2p`. -This command and its subcommands will allow you to open up arbitrary streams to -other ipfs peers through libp2p. The interfaces are a little bit clunky right -now, but shouldn't get in the way of anyone wanting to try building a fully -peer to peer application on top of go-ipfs and libp2p. For more info on this -command, to ask questions, or to provide feedback, head over to the [feedback -issue](https://github.com/ipfs/go-ipfs/issues/3994) for the command. +We have also finally implemented a very early (and experimental) `ipfs p2p`. This command and its subcommands will allow you to open up arbitrary streams to other ipfs peers through libp2p. The interfaces are a little bit clunky right now, but shouldn't get in the way of anyone wanting to try building a fully peer to peer application on top of go-ipfs and libp2p. For more info on this command, to ask questions, or to provide feedback, head over to the [feedback issue](https://github.com/ipfs/go-ipfs/issues/3994) for the command. -A few other subcommands and flags were added around the API, as well as many -other requested improvements. See below for the full list of changes. +A few other subcommands and flags were added around the API, as well as many other requested improvements. See below for the full list of changes. - Features - Add support for specifying the hash function in `ipfs add` ([ipfs/go-ipfs#3919](https://github.com/ipfs/go-ipfs/pull/3919)) @@ -79,10 +49,10 @@ other requested improvements. See below for the full list of changes. - Clean up bitswap ledgers when disconnecting ([ipfs/go-ipfs#3437](https://github.com/ipfs/go-ipfs/pull/3437)) - Make odds of 'process added after close' panic less likely ([ipfs/go-ipfs#3940](https://github.com/ipfs/go-ipfs/pull/3940)) - General Changes and Refactorings - - Remove 'ipfs diag net' from codebase ([ipfs/go-ipfs#3916](https://github.com/ipfs/go-ipfs/pull/3916)) - Update to dht code with provide announce option ([ipfs/go-ipfs#3928](https://github.com/ipfs/go-ipfs/pull/3928)) - Apply the megacheck code vetting tool ([ipfs/go-ipfs#3949](https://github.com/ipfs/go-ipfs/pull/3949)) - Expose port 8081 in docker container for /ws listener ([ipfs/go-ipfs#3954](https://github.com/ipfs/go-ipfs/pull/3954)) - If you have questions or run into any issues, please post in the IPFS Discussion and Support Forum's [go-ipfs 0.4.10 release thread](https://discuss.ipfs.io/t/ipfs-0-4-10-release/687). For bugs, please open an issue in [ipfs/go-ipfs/issues](https://github.com/ipfs/go-ipfs/issues). + +If you have questions or run into any issues, please post in the IPFS Discussion and Support Forum's [go-ipfs 0.4.10 release thread](https://discuss.ipfs.io/t/ipfs-0-4-10-release/687). For bugs, please open an issue in [ipfs/go-ipfs/issues](https://github.com/ipfs/go-ipfs/issues). diff --git a/src/_blog/023-go-ipfs-0.4.14.md b/src/_blog/023-go-ipfs-0.4.14.md index fbb46992..fc1e2c40 100644 --- a/src/_blog/023-go-ipfs-0.4.14.md +++ b/src/_blog/023-go-ipfs-0.4.14.md @@ -8,9 +8,7 @@ tags: - 'go-ipfs' --- -[go-ipfs 0.4.14](https://dist.ipfs.io/#go-ipfs) has been released today. Not -only have we improved memory and CPU usage but we also managed to fix a lot of -bugs, ship a major improvement to IPNS performance and lots of refactoring! \o/ +[go-ipfs 0.4.14](https://dist.ipfs.io/#go-ipfs) has been released today. Not only have we improved memory and CPU usage but we also managed to fix a lot of bugs, ship a major improvement to IPNS performance and lots of refactoring! \o/ - [Refactoring](#refactoring) - [IPNS Improvements](#ipns-improvements) @@ -20,56 +18,24 @@ bugs, ship a major improvement to IPNS performance and lots of refactoring! \o/ # Refactoring -The release took longer than expected due to our refactoring and extracting of -our [commands library](https://github.com/ipfs/go-ipfs-cmds). This refactor had -two stages. The first round of the refactor disentangled the commands code from -core go-ipfs code, allowing us to move it out into a separate repository. The -code was previously very entangled with the go-ipfs codebase and not usable for -other projects. The second round of the refactor had the goal of fixing several -major issues around streaming outputs, progress bars, and error handling. It -also paved the way for us to more easily provide an API over other transports, -such as websockets and unix domain sockets. It took a while to flush out all -the kinks on such a massive change. We're pretty sure we've got most of them, -but if you notice anything weird, -[please let us know](https://github.com/ipfs/go-ipfs/issues/new). +The release took longer than expected due to our refactoring and extracting of our [commands library](https://github.com/ipfs/go-ipfs-cmds). This refactor had two stages. The first round of the refactor disentangled the commands code from core go-ipfs code, allowing us to move it out into a separate repository. The code was previously very entangled with the go-ipfs codebase and not usable for other projects. The second round of the refactor had the goal of fixing several major issues around streaming outputs, progress bars, and error handling. It also paved the way for us to more easily provide an API over other transports, such as websockets and unix domain sockets. It took a while to flush out all the kinks on such a massive change. We're pretty sure we've got most of them, but if you notice anything weird, [please let us know](https://github.com/ipfs/go-ipfs/issues/new). # IPNS Improvements -Beyond that, we've added a new experimental way to use IPNS. With the new -pubsub IPNS resolver and publisher, you can subscribe to updates of an IPNS -entry, and the owner can publish out changes in real time. With this, IPNS can -become nearly instantaneous. To make use of this, simply start your go-ipfs -daemon with the `--enable-namesys-pubsub` option, and all IPNS resolution and -publishing will use pubsub. Note that resolving an IPNS name via pubsub without -someone publishing it via pubsub will result in a fallback to using the DHT. -Please give this a try and let us know how it goes! +Beyond that, we've added a new experimental way to use IPNS. With the new pubsub IPNS resolver and publisher, you can subscribe to updates of an IPNS entry, and the owner can publish out changes in real time. With this, IPNS can become nearly instantaneous. To make use of this, simply start your go-ipfs daemon with the `--enable-namesys-pubsub` option, and all IPNS resolution and publishing will use pubsub. Note that resolving an IPNS name via pubsub without someone publishing it via pubsub will result in a fallback to using the DHT. Please give this a try and let us know how it goes! # Resource Usage Improvements -Memory and CPU usage should see a noticeable improvement in this release. We -have spent considerable time fixing excess memory usage throughout the codebase -and down into go-libp2p. Fixes in peer tracking, bitswap allocation, pinning, -and many other places have brought down both peak and average memory usage. An -upgraded hashing library, base58 encoding library, and improved allocation -patterns all contribute to overall lower CPU usage across the board. +Memory and CPU usage should see a noticeable improvement in this release. We have spent considerable time fixing excess memory usage throughout the codebase and down into go-libp2p. Fixes in peer tracking, bitswap allocation, pinning, and many other places have brought down both peak and average memory usage. An upgraded hashing library, base58 encoding library, and improved allocation patterns all contribute to overall lower CPU usage across the board. # IPFS Core API -This release also brings the beginning of the go-ipfs 'Core API'. Once -finished, the Core API will be the primary way to interact with go-ipfs using -go. Both embedded nodes and nodes accessed over the HTTP API will have the same -interface. Stay tuned for future updates and documentation. +This release also brings the beginning of the go-ipfs 'Core API'. Once finished, the Core API will be the primary way to interact with go-ipfs using go. Both embedded nodes and nodes accessed over the HTTP API will have the same interface. Stay tuned for future updates and documentation. # Note Regarding Insecure Hash Functions -This release of go-ipfs disallows the usage of insecure hash functions and -lengths. go-ipfs does not create these insecure objects for any purpose, but -it did allow manually creating them and fetching them from other peers. If you -currently have objects using insecure hashes in your local go-ipfs repo, please -remove them before updating. +This release of go-ipfs disallows the usage of insecure hash functions and lengths. go-ipfs does not create these insecure objects for any purpose, but it did allow manually creating them and fetching them from other peers. If you currently have objects using insecure hashes in your local go-ipfs repo, please remove them before updating. # Full Changelog -As always, you can find the full changelog over at ipfs/go-ipfs's Github -repository: -https://github.com/ipfs/go-ipfs/blob/master/CHANGELOG.md#0414-2018-03-22 +As always, you can find the full changelog over at ipfs/go-ipfs's Github repository: https://github.com/ipfs/go-ipfs/blob/master/CHANGELOG.md#0414-2018-03-22 From d2fd44435202d2d1caf818e9136c4bb581f1f8aa Mon Sep 17 00:00:00 2001 From: Ze Bateira Date: Thu, 8 Apr 2021 18:20:47 +0100 Subject: [PATCH 3/3] fix: last one --- src/_blog/005-ipfs-0-4-0-released.md | 156 +++++---------------------- 1 file changed, 24 insertions(+), 132 deletions(-) diff --git a/src/_blog/005-ipfs-0-4-0-released.md b/src/_blog/005-ipfs-0-4-0-released.md index ce1c621c..2ba66063 100644 --- a/src/_blog/005-ipfs-0-4-0-released.md +++ b/src/_blog/005-ipfs-0-4-0-released.md @@ -11,23 +11,15 @@ tags: - go-ipfs --- -[go-ipfs 0.4.0](http://dist.ipfs.io/#go-ipfs) has been -released! Among the many changes are a revamped implementation of the IPFS -communication protocols, increased performance, improvements to IPNS (the -Interplanetary Naming System), many bugfixes, and lots of new features to make -IPFS even more powerful. +[go-ipfs 0.4.0](http://dist.ipfs.io/#go-ipfs) has been released! Among the many changes are a revamped implementation of the IPFS communication protocols, increased performance, improvements to IPNS (the Interplanetary Naming System), many bugfixes, and lots of new features to make IPFS even more powerful. ![0.4.0](../assets/005-ipfs-0-4-0-released-fireworks.jpg) ## The IPFS **Files API** -In 0.4.0, we've added a new feature, the Files API, available through the -subcommand: `ipfs files`. This subcommand allows a program to interact with IPFS -using familiar filesystem operations, namely: creating directories, reading, writing -and deleting files, listing out different directories, and so on. +In 0.4.0, we've added a new feature, the Files API, available through the subcommand: `ipfs files`. This subcommand allows a program to interact with IPFS using familiar filesystem operations, namely: creating directories, reading, writing and deleting files, listing out different directories, and so on. -This feature enables any application that uses a filesystem-like backend -to use IPFS for storage without changing the application logic at all. +This feature enables any application that uses a filesystem-like backend to use IPFS for storage without changing the application logic at all. It is used like so: @@ -58,159 +50,63 @@ QmNU8HmaeRa8VtfqAoQRJhSE5Zx54vyYf2nT1bDGcYXaNv # Subsequent edits will produce a new /ipfs address for the root directory. ``` -One great example of this is [ipfs-blob-store](https://github.com/ipfs/ipfs-blob-store), -an IPFS-backed storage driver that implements the -[blob-store-interface](https://github.com/maxogden/abstract-blob-store), so that any app -that uses any other blob-store storage driver (S3, IndexedDB, LevelDB, etc) can now use -IPFS. [registry-mirror](https://github.com/diasdavid/registry-mirror) uses this -approach to mirror the npm registry onto IPFS. +One great example of this is [ipfs-blob-store](https://github.com/ipfs/ipfs-blob-store), an IPFS-backed storage driver that implements the [blob-store-interface](https://github.com/maxogden/abstract-blob-store), so that any app that uses any other blob-store storage driver (S3, IndexedDB, LevelDB, etc) can now use IPFS. [registry-mirror](https://github.com/diasdavid/registry-mirror) uses this approach to mirror the npm registry onto IPFS. We are looking forward to seeing more use cases of this new convenient and powerful API. ## Why we're changing the protocol -The most important change allows IPFS implementations to use pluggable stream -multiplexers, such as [yamux](https://github.com/hashicorp/yamux), -[spdystream](https://github.com/docker/spdystream), or -[muxado](https://github.com/inconshreveable/muxado). Instead of locking IPFS -permanently into a single multiplexer that won't work for every language or -situation, this change allows each IPFS implementation to implement the -multiplexers that they choose to, and then negotiate which to use during the -initial connection handshake. +The most important change allows IPFS implementations to use pluggable stream multiplexers, such as [yamux](https://github.com/hashicorp/yamux), [spdystream](https://github.com/docker/spdystream), or [muxado](https://github.com/inconshreveable/muxado). Instead of locking IPFS permanently into a single multiplexer that won't work for every language or situation, this change allows each IPFS implementation to implement the multiplexers that they choose to, and then negotiate which to use during the initial connection handshake. -This modularity with stream muxing makes it easier for certain languages to -improve performance. For example, the Go programming language may have muxado -and yamux implementations that are really good, but many languages lack good (or -any) implementations of them. Or take Node.js, which works well with -spdy-transport, and it would be nice to take advantage of that. And -then there are options like -[multiplex](https://github.com/maxogden/multiplex), which may not have the -same performance, but are much easier to implement. +This modularity with stream muxing makes it easier for certain languages to improve performance. For example, the Go programming language may have muxado and yamux implementations that are really good, but many languages lack good (or any) implementations of them. Or take Node.js, which works well with spdy-transport, and it would be nice to take advantage of that. And then there are options like [multiplex](https://github.com/maxogden/multiplex), which may not have the same performance, but are much easier to implement. -So, by supporting as many muxers as we can, we get to choose the best -multiplexers for the job. It also makes it much easier to implement the IPFS -protocols in a new language. And of course, if a better multiplexers standard -comes along, it will be easier to upgrade IPFS to support it in the future. +So, by supporting as many muxers as we can, we get to choose the best multiplexers for the job. It also makes it much easier to implement the IPFS protocols in a new language. And of course, if a better multiplexers standard comes along, it will be easier to upgrade IPFS to support it in the future. -The same code that allows us to easily select a stream muxer is also being -used to select which IPFS sub-protocol to use for any given stream between -peers. Now, if we need to make a breaking protocol change to any of those (like -the DHT or bitswap) we can do so seamlessly, and provide easy backwards -compatibility. We won't need to "break" anything because we will be able to -have nodes run multiple protocols at the same time for compatibility. +The same code that allows us to easily select a stream muxer is also being used to select which IPFS sub-protocol to use for any given stream between peers. Now, if we need to make a breaking protocol change to any of those (like the DHT or bitswap) we can do so seamlessly, and provide easy backwards compatibility. We won't need to "break" anything because we will be able to have nodes run multiple protocols at the same time for compatibility. -In addition to the multiplexers changes, the protocol revamp has also improved -efficiency and performance in a few important ways, including the elimination of -a double wrapping of the length prefixer, and the removal of some unnecessary -round trips between nodes. +In addition to the multiplexers changes, the protocol revamp has also improved efficiency and performance in a few important ways, including the elimination of a double wrapping of the length prefixer, and the removal of some unnecessary round trips between nodes. ### Breaking changes -This release contains a **breaking change** to the network wire protocol in the -form of a major refactor and upgrade to the libp2p handshake protocol. Because -of the refactor, **all IPFS daemons earlier than 0.4.0 will not be able to -communicate with the newest version**. It is strongly recommended that everyone -running an IPFS node upgrades to the latest version as soon as possible, as -these nodes will, after a certain time, no longer be able to communicate with -the majority of the network until they are upgraded. There are instructions on -how to update below. +This release contains a **breaking change** to the network wire protocol in the form of a major refactor and upgrade to the libp2p handshake protocol. Because of the refactor, **all IPFS daemons earlier than 0.4.0 will not be able to communicate with the newest version**. It is strongly recommended that everyone running an IPFS node upgrades to the latest version as soon as possible, as these nodes will, after a certain time, no longer be able to communicate with the majority of the network until they are upgraded. There are instructions on how to update below. -Refactoring the protocol is not something to be done lightly. But at this early -stage, this is necessary to ensure we have the right design for IPFS in place -for the future. It's better to improve the protocol now during this alpha stage -of the project than when there are a lot more people running nodes across a -lot of different implementations. +Refactoring the protocol is not something to be done lightly. But at this early stage, this is necessary to ensure we have the right design for IPFS in place for the future. It's better to improve the protocol now during this alpha stage of the project than when there are a lot more people running nodes across a lot of different implementations. -One of the important changes that's been made to the protocol means that _there -should never be a breaking change like this again_. This is due to a change to -allow nodes to announce the version of the protocol they are using when -connecting to other IPFS nodes. The goal is to roll any future protocol changes -into the implementations gradually, so that we can still support legacy -protocols for a period of time, making it easier to deprecate old versions over -time. +One of the important changes that's been made to the protocol means that _there should never be a breaking change like this again_. This is due to a change to allow nodes to announce the version of the protocol they are using when connecting to other IPFS nodes. The goal is to roll any future protocol changes into the implementations gradually, so that we can still support legacy protocols for a period of time, making it easier to deprecate old versions over time. ### The public gateway and bootstrappers -We provide two essential public services to the IPFS community: the public -gateway at https://ipfs.io and the default bootstrap nodes. We're making sure -that despite the breaking changes, both will continue to work with 0.4.x and -0.3.x for a while. You can read more about the details of this in an earlier -blog post: [Migrating ipfs.io from go-ipfs 0.3.x to 0.4.0](/9-v04x-migration). -We expect to keep this grace period open until the end of April 2016. +We provide two essential public services to the IPFS community: the public gateway at https://ipfs.io and the default bootstrap nodes. We're making sure that despite the breaking changes, both will continue to work with 0.4.x and 0.3.x for a while. You can read more about the details of this in an earlier blog post: [Migrating ipfs.io from go-ipfs 0.3.x to 0.4.0](/9-v04x-migration). We expect to keep this grace period open until the end of April 2016. ## Changes to the repo -We made a few changes to our on disk storage format (called the 'repo'). -The way object pinning (`ipfs pin add`) works has also been upgraded to be much -more efficient, which will improve the overall speed of adding and downloading -IPFS data. Previously, when you pinned a file recursively, it would add a -recursive type pin to the root, and then add an indirect type pin to each child -node. This made enumeration for GC convenient, but was very slow and made a lot -of things needlessly complicated. We have switched to just adding the recursive -pin to the root object, and then doing the enumeration of child nodes when we -actually run a garbage collection. +We made a few changes to our on disk storage format (called the 'repo'). The way object pinning (`ipfs pin add`) works has also been upgraded to be much more efficient, which will improve the overall speed of adding and downloading IPFS data. Previously, when you pinned a file recursively, it would add a recursive type pin to the root, and then add an indirect type pin to each child node. This made enumeration for GC convenient, but was very slow and made a lot of things needlessly complicated. We have switched to just adding the recursive pin to the root object, and then doing the enumeration of child nodes when we actually run a garbage collection. -Because of this change, you will need to run a migration (from repo version 2 -to version 3). If you update with the `ipfs-update` tool, this will be done for -you automatically. If you updated manually, and did not run the migration, ipfs -will fail to run, and print a message saying that there is a mismatch in the repo -versions. +Because of this change, you will need to run a migration (from repo version 2 to version 3). If you update with the `ipfs-update` tool, this will be done for you automatically. If you updated manually, and did not run the migration, ipfs will fail to run, and print a message saying that there is a mismatch in the repo versions. ## Other improvements and fixes -In addition to a major protocol improvement and upgrade, this release adds a lot -of new functionality, performance speedups, and stability fixes that make this -the best version of IPFS to date. +In addition to a major protocol improvement and upgrade, this release adds a lot of new functionality, performance speedups, and stability fixes that make this the best version of IPFS to date. -This release also includes performance and usability improvements to IPNS, -which is IPFS's mutability layer. IPNS creates a link from an IPFS node's -public key to the hash of an arbitrary objects hash, in a way that is -cryptographically verifiable. We call this "publishing" a hash, and you can try -it out using the `ipfs publish` command. By allowing users to change what the -pubkey hash points to, we provide users with a single hash they can give to their -users to get the latest version of their data. This creates a seamless way to -use IPFS to verify content, and to distribute content via trustless nodes in a -smart, safe way. This brings IPFS closer to the goal of being a global -filesystem of data, that can allow everyone in the world to help serve the -world's data in a way that enriches and empowers everybody. +This release also includes performance and usability improvements to IPNS, which is IPFS's mutability layer. IPNS creates a link from an IPFS node's public key to the hash of an arbitrary objects hash, in a way that is cryptographically verifiable. We call this "publishing" a hash, and you can try it out using the `ipfs publish` command. By allowing users to change what the pubkey hash points to, we provide users with a single hash they can give to their users to get the latest version of their data. This creates a seamless way to use IPFS to verify content, and to distribute content via trustless nodes in a smart, safe way. This brings IPFS closer to the goal of being a global filesystem of data, that can allow everyone in the world to help serve the world's data in a way that enriches and empowers everybody. ## How to upgrade -Depending on how you initially installed IPFS, there are several ways to -upgrade. If you installed IPFS with a pre-built binary, you can either head over -to [dist.ipfs.io](http://dist.ipfs.io/#go-ipfs) and grab the latest version -from there. Or alternatively, from the same page you can grab the `ipfs-update` -binary, and use it to perform the upgrade for you. If you installed from -source, you can simply update your git repo to the `v0.4.0` tag, run `make toolkit_upgrade && make install`. +Depending on how you initially installed IPFS, there are several ways to upgrade. If you installed IPFS with a pre-built binary, you can either head over to [dist.ipfs.io](http://dist.ipfs.io/#go-ipfs) and grab the latest version from there. Or alternatively, from the same page you can grab the `ipfs-update` binary, and use it to perform the upgrade for you. If you installed from source, you can simply update your git repo to the `v0.4.0` tag, run `make toolkit_upgrade && make install`. -Please upgrade your IPFS nodes as soon as you can, so you can take advantage -of the improvements! +Please upgrade your IPFS nodes as soon as you can, so you can take advantage of the improvements! ## Changelog -This is a major release with plenty of new features and bugfixes. -It also includes breaking changes which make it incompatible with v0.3.x -on the networking layer. +This is a major release with plenty of new features and bugfixes. It also includes breaking changes which make it incompatible with v0.3.x on the networking layer. - Major Changes - Multistream - - The addition of multistream is a breaking change on the networking layer, - but gives IPFS implementations the ability to mix and match different - stream multiplexers, e.g. yamux, spdystream, or muxado. - This adds a ton of flexibility on one of the lower layers of the protocol, - and will help us avoid further breaking protocol changes in the future. + - The addition of multistream is a breaking change on the networking layer, but gives IPFS implementations the ability to mix and match different stream multiplexers, e.g. yamux, spdystream, or muxado. This adds a ton of flexibility on one of the lower layers of the protocol, and will help us avoid further breaking protocol changes in the future. - Files API - - The new `files` command and API allow a program to interact with IPFS - using familiar filesystem operations, namely: creating directories, - reading, writing, and deleting files, listing out different directories, - and so on. This feature enables any other application that uses a - filesystem-like backend for storage, to use IPFS as its storage driver - without having change the application logic at all. + - The new `files` command and API allow a program to interact with IPFS using familiar filesystem operations, namely: creating directories, reading, writing, and deleting files, listing out different directories, and so on. This feature enables any other application that uses a filesystem-like backend for storage, to use IPFS as its storage driver without having change the application logic at all. - Gx - - go-ipfs now uses [gx](https://github.com/whyrusleeping/gx) to manage its - dependencies. This means that under the hood, go-ipfs's dependencies are - backed by IPFS itself! It also means that go-ipfs is no longer installed - using `go get`. Use `make install` instead. + - go-ipfs now uses [gx](https://github.com/whyrusleeping/gx) to manage its dependencies. This means that under the hood, go-ipfs's dependencies are backed by IPFS itself! It also means that go-ipfs is no longer installed using `go get`. Use `make install` instead. - New Features - Web UI - Update to new version which is compatible with 0.4.0. (@dignifiedquire) @@ -233,11 +129,7 @@ on the networking layer. - Add an `IPFS_LOW_MEM` environment veriable which relaxes Bitswap's memory usage. (@whyrusleeping) - The Docker image now lives at `ipfs/go-ipfs` and has been completely reworked. (@lgierth) - Security fixes - - The gateway path prefix added in v0.3.10 was vulnerable to cross-site - scripting attacks. This release introduces a configurable list of allowed - path prefixes. It's called `Gateway.PathPrefixes` and takes a list of - strings, e.g. `["/blog", "/foo/bar"]`. The v0.3.x line will not receive any - further updates, so please update to v0.4.0 as soon as possible. (@lgierth) + - The gateway path prefix added in v0.3.10 was vulnerable to cross-site scripting attacks. This release introduces a configurable list of allowed path prefixes. It's called `Gateway.PathPrefixes` and takes a list of strings, e.g. `["/blog", "/foo/bar"]`. The v0.3.x line will not receive any further updates, so please update to v0.4.0 as soon as possible. (@lgierth) - Incompatible Changes - Install using `make install` instead of `go get` (@whyrusleeping) - Rewrite pinning to store pins in IPFS objects. (@tv42)