mirror of
https://github.com/ipfs/ipfs-blog.git
synced 2026-07-03 00:11:18 +02:00
fix: more posts with extra newlines fixed
This commit is contained in:
Ze Bateira
@@ -9,10 +9,7 @@ tags:
|
||||
- 'go-ipfs'
|
||||
---
|
||||
|
||||
[go-ipfs 0.4.4](https://dist.ipfs.io/#go-ipfs) has been released today,
|
||||
including an important hotfix for a bug we discovered in how _pinning_ works.
|
||||
If you had a large number of pins, new pins would overwrite existing pins.
|
||||
Apart from the hotfix, this release is equal to the previous release 0.4.3.
|
||||
[go-ipfs 0.4.4](https://dist.ipfs.io/#go-ipfs) has been released today, including an important hotfix for a bug we discovered in how _pinning_ works. If you had a large number of pins, new pins would overwrite existing pins. Apart from the hotfix, this release is equal to the previous release 0.4.3.
|
||||
|
||||
- [How pinning works](#how-pinning-works)
|
||||
- [The bug](#the-bug)
|
||||
@@ -21,9 +18,7 @@ Apart from the hotfix, this release is equal to the previous release 0.4.3.
|
||||
|
||||
## How pinning works
|
||||
|
||||
Pinning is a means of persisting data in your IPFS repo after adding or fetching it.
|
||||
It'll prevent objects from getting removed by garbage collection or other methods
|
||||
of cleaning up the IPFS repo. There are three ways an object can be pinned:
|
||||
Pinning is a means of persisting data in your IPFS repo after adding or fetching it. It'll prevent objects from getting removed by garbage collection or other methods of cleaning up the IPFS repo. There are three ways an object can be pinned:
|
||||
|
||||
- **Direct:** Only this object is pinned. Its children aren't pinned.
|
||||
- **Recursive:** This object and all its children are pinned.
|
||||
@@ -31,54 +26,26 @@ of cleaning up the IPFS repo. There are three ways an object can be pinned:
|
||||
- **Indirect:** This object is pinned because one of its parents is pinned.
|
||||
If the pins of all parents are removed, this object isn't pinned any longer.
|
||||
|
||||
The `ipfs add` command adds a _recursive pin_ for the added file by default.
|
||||
With `--pin=false`, it skips pinning. Similarly, the default pin type for
|
||||
`ipfs pin add` is _recursive_. With `--recursive=false` this changes to _direct_.
|
||||
The `ipfs add` command adds a _recursive pin_ for the added file by default. With `--pin=false`, it skips pinning. Similarly, the default pin type for `ipfs pin add` is _recursive_. With `--recursive=false` this changes to _direct_.
|
||||
|
||||
For more information on how pinning works, check out `ipfs pin --help` and `ipfs add --help`.
|
||||
|
||||
## The bug
|
||||
|
||||
Direct and recursive pins are stored in separate so-called _pinsets_.
|
||||
Indirect pins aren't stored, since they're derived from recursive pins.
|
||||
Direct and recursive pins are stored in separate so-called _pinsets_. Indirect pins aren't stored, since they're derived from recursive pins.
|
||||
|
||||
Once you had more than 8192 pins, recursive or direct, an
|
||||
issue with the recursive hash trie implementation caused hash table buckets to
|
||||
be overwritten, resulting in only 256 pins remaining in the pinset. After that,
|
||||
the bug wouldn't be triggered again until the number of pins exceeded 8192 again.
|
||||
The 256 pins that remained would be random.
|
||||
Once you had more than 8192 pins, recursive or direct, an issue with the recursive hash trie implementation caused hash table buckets to be overwritten, resulting in only 256 pins remaining in the pinset. After that, the bug wouldn't be triggered again until the number of pins exceeded 8192 again. The 256 pins that remained would be random.
|
||||
|
||||
We fixed this by instead making sure that each item in a pinset be put into its
|
||||
own bucket, and by modulo'ing hash output from this process into the final key
|
||||
space. The details for this can be seen
|
||||
[in this pull request](https://github.com/ipfs/go-ipfs/pull/3273). We added a
|
||||
stress test to make sure that this doesn't happen in the future, and will
|
||||
redouble our efforts to make sure that our test suites are more robust to ensure
|
||||
that these kinds of problems do not happen in the future.
|
||||
We fixed this by instead making sure that each item in a pinset be put into its own bucket, and by modulo'ing hash output from this process into the final key space. The details for this can be seen [in this pull request](https://github.com/ipfs/go-ipfs/pull/3273). We added a stress test to make sure that this doesn't happen in the future, and will redouble our efforts to make sure that our test suites are more robust to ensure that these kinds of problems do not happen in the future.
|
||||
|
||||
For now, don't run `ipfs repo gc` on sensitive data that is not otherwise backed up,
|
||||
as IPFS is still not 1.0 and our development may still find problems.
|
||||
For now, don't run `ipfs repo gc` on sensitive data that is not otherwise backed up, as IPFS is still not 1.0 and our development may still find problems.
|
||||
|
||||
[@kyledrake](https://github.com/kyledrake) of [neocities.org](https://neocities.org)
|
||||
pointed out this bug to us; thank you, Kyle!
|
||||
[@kyledrake](https://github.com/kyledrake) of [neocities.org](https://neocities.org) pointed out this bug to us; thank you, Kyle!
|
||||
|
||||
## Find out if you're affected
|
||||
|
||||
If you think you have experienced this issue and have _not_ run a garbage
|
||||
collection, you can still find the 'lost' pins. We have written a new tool
|
||||
called 'ipfs-see-all' that allows you to try and recover any old pins that are
|
||||
still in your local repo. The tool is available on [our distributions
|
||||
page](https://dist.ipfs.io), or, if you prefer building from source, head over
|
||||
to [the GitHub repo](https://github.com/whyrusleeping/ipfs-see-all). Once you
|
||||
have the tool, invoke it as `ipfs-see-all lost-pins` and it will scan for and
|
||||
print out every pin object that is not actually pinned in your pinset. Note
|
||||
that this may contain anything you have manually unpinned.
|
||||
If you think you have experienced this issue and have _not_ run a garbage collection, you can still find the 'lost' pins. We have written a new tool called 'ipfs-see-all' that allows you to try and recover any old pins that are still in your local repo. The tool is available on [our distributions page](https://dist.ipfs.io), or, if you prefer building from source, head over to [the GitHub repo](https://github.com/whyrusleeping/ipfs-see-all). Once you have the tool, invoke it as `ipfs-see-all lost-pins` and it will scan for and print out every pin object that is not actually pinned in your pinset. Note that this may contain anything you have manually unpinned.
|
||||
|
||||
## How to upgrade
|
||||
|
||||
Depending on how you initially installed IPFS, there are several ways to
|
||||
upgrade. If you installed IPFS with a pre-built binary, you can head over
|
||||
to [dist.ipfs.io](https://dist.ipfs.io/#go-ipfs) and grab the latest version
|
||||
from there. Or alternatively, from the same page you can grab the `ipfs-update`
|
||||
binary, and use it to perform the upgrade for you. If you installed from
|
||||
source, you can simply run `git checkout v0.4.4`, then run `make install`.
|
||||
Depending on how you initially installed IPFS, there are several ways to upgrade. If you installed IPFS with a pre-built binary, you can head over to [dist.ipfs.io](https://dist.ipfs.io/#go-ipfs) and grab the latest version from there. Or alternatively, from the same page you can grab the `ipfs-update` binary, and use it to perform the upgrade for you. If you installed from source, you can simply run `git checkout v0.4.4`, then run `make install`.
|
||||
|
||||
@@ -8,51 +8,21 @@ tags:
|
||||
- 'go-ipfs'
|
||||
---
|
||||
|
||||
Yesterday was the [3rd birthday of go-ipfs](https://github.com/ipfs/go-ipfs/commit/9d0e6a7ffb5deea2c8c8e555d7bf6bcab6fdc6ac).
|
||||
To celebrate, we would like to share some virtual cake 🎂 with ipfs users and
|
||||
contributors from all over the world, and also announce the release of
|
||||
go-ipfs 0.4.10.
|
||||
Yesterday was the [3rd birthday of go-ipfs](https://github.com/ipfs/go-ipfs/commit/9d0e6a7ffb5deea2c8c8e555d7bf6bcab6fdc6ac). To celebrate, we would like to share some virtual cake 🎂 with ipfs users and contributors from all over the world, and also announce the release of go-ipfs 0.4.10.
|
||||
|
||||
go-ipfs 0.4.10 is a patch release that contains several exciting new features,
|
||||
bugfixes and general improvements. This includes new commands, easier corruption
|
||||
recovery, and a generally cleaner codebase.
|
||||
go-ipfs 0.4.10 is a patch release that contains several exciting new features, bugfixes and general improvements. This includes new commands, easier corruption recovery, and a generally cleaner codebase.
|
||||
|
||||
The `ipfs pin` command has two new subcommands, `verify` and `update`. `ipfs pin verify` is used to scan the repo for pinned object graphs and check their
|
||||
integrity. Any issues are reported back with helpful error text to make error
|
||||
recovery simpler. This subcommand was added to help recover from datastore
|
||||
corruptions, particularly if using the experimental filestore and accidentally
|
||||
deleting tracked files.
|
||||
The `ipfs pin` command has two new subcommands, `verify` and `update`. `ipfs pin verify` is used to scan the repo for pinned object graphs and check their integrity. Any issues are reported back with helpful error text to make error recovery simpler. This subcommand was added to help recover from datastore corruptions, particularly if using the experimental filestore and accidentally deleting tracked files.
|
||||
|
||||
`ipfs pin update` was added to make the task of keeping a large, frequently
|
||||
changing object graph pinned. Previously users had to call `ipfs pin rm` on the
|
||||
old pin, and `ipfs pin add` on the new one. The 'new' `ipfs pin add` call would
|
||||
be very expensive as it would need to verify the entirety of the graph again.
|
||||
The `ipfs pin update` command takes shortcuts, portions of the graph that were
|
||||
covered under the old pin are assumed to be fine, and the command skips
|
||||
checking them.
|
||||
`ipfs pin update` was added to make the task of keeping a large, frequently changing object graph pinned. Previously users had to call `ipfs pin rm` on the old pin, and `ipfs pin add` on the new one. The 'new' `ipfs pin add` call would be very expensive as it would need to verify the entirety of the graph again. The `ipfs pin update` command takes shortcuts, portions of the graph that were covered under the old pin are assumed to be fine, and the command skips checking them.
|
||||
|
||||
Next up, we have finally implemented an `ipfs shutdown` command so users can
|
||||
shut down their ipfs daemons via the API. This is especially useful on
|
||||
platforms that make it difficult to control processes (Android, for example),
|
||||
and is also useful when needing to shut down a node remotely and you do not
|
||||
have access to the machine itself.
|
||||
Next up, we have finally implemented an `ipfs shutdown` command so users can shut down their ipfs daemons via the API. This is especially useful on platforms that make it difficult to control processes (Android, for example), and is also useful when needing to shut down a node remotely and you do not have access to the machine itself.
|
||||
|
||||
`ipfs add` has gained a new flag; the `--hash` flag allows you to select which
|
||||
hash function to use and we have given it the ability to select `blake2b-256`.
|
||||
This pushes us one step closer to shifting over to using blake2b as the
|
||||
default. Blake2b is significantly faster than sha2-256, and also is conjectured
|
||||
to provide superior security.
|
||||
`ipfs add` has gained a new flag; the `--hash` flag allows you to select which hash function to use and we have given it the ability to select `blake2b-256`. This pushes us one step closer to shifting over to using blake2b as the default. Blake2b is significantly faster than sha2-256, and also is conjectured to provide superior security.
|
||||
|
||||
We have also finally implemented a very early (and experimental) `ipfs p2p`.
|
||||
This command and its subcommands will allow you to open up arbitrary streams to
|
||||
other ipfs peers through libp2p. The interfaces are a little bit clunky right
|
||||
now, but shouldn't get in the way of anyone wanting to try building a fully
|
||||
peer to peer application on top of go-ipfs and libp2p. For more info on this
|
||||
command, to ask questions, or to provide feedback, head over to the [feedback
|
||||
issue](https://github.com/ipfs/go-ipfs/issues/3994) for the command.
|
||||
We have also finally implemented a very early (and experimental) `ipfs p2p`. This command and its subcommands will allow you to open up arbitrary streams to other ipfs peers through libp2p. The interfaces are a little bit clunky right now, but shouldn't get in the way of anyone wanting to try building a fully peer to peer application on top of go-ipfs and libp2p. For more info on this command, to ask questions, or to provide feedback, head over to the [feedback issue](https://github.com/ipfs/go-ipfs/issues/3994) for the command.
|
||||
|
||||
A few other subcommands and flags were added around the API, as well as many
|
||||
other requested improvements. See below for the full list of changes.
|
||||
A few other subcommands and flags were added around the API, as well as many other requested improvements. See below for the full list of changes.
|
||||
|
||||
- Features
|
||||
- Add support for specifying the hash function in `ipfs add` ([ipfs/go-ipfs#3919](https://github.com/ipfs/go-ipfs/pull/3919))
|
||||
@@ -79,10 +49,10 @@ other requested improvements. See below for the full list of changes.
|
||||
- Clean up bitswap ledgers when disconnecting ([ipfs/go-ipfs#3437](https://github.com/ipfs/go-ipfs/pull/3437))
|
||||
- Make odds of 'process added after close' panic less likely ([ipfs/go-ipfs#3940](https://github.com/ipfs/go-ipfs/pull/3940))
|
||||
- General Changes and Refactorings
|
||||
|
||||
- Remove 'ipfs diag net' from codebase ([ipfs/go-ipfs#3916](https://github.com/ipfs/go-ipfs/pull/3916))
|
||||
- Update to dht code with provide announce option ([ipfs/go-ipfs#3928](https://github.com/ipfs/go-ipfs/pull/3928))
|
||||
- Apply the megacheck code vetting tool ([ipfs/go-ipfs#3949](https://github.com/ipfs/go-ipfs/pull/3949))
|
||||
- Expose port 8081 in docker container for /ws listener ([ipfs/go-ipfs#3954](https://github.com/ipfs/go-ipfs/pull/3954))
|
||||
|
||||
If you have questions or run into any issues, please post in the IPFS Discussion and Support Forum's [go-ipfs 0.4.10 release thread](https://discuss.ipfs.io/t/ipfs-0-4-10-release/687). For bugs, please open an issue in [ipfs/go-ipfs/issues](https://github.com/ipfs/go-ipfs/issues).
|
||||
|
||||
If you have questions or run into any issues, please post in the IPFS Discussion and Support Forum's [go-ipfs 0.4.10 release thread](https://discuss.ipfs.io/t/ipfs-0-4-10-release/687). For bugs, please open an issue in [ipfs/go-ipfs/issues](https://github.com/ipfs/go-ipfs/issues).
|
||||
|
||||
@@ -8,9 +8,7 @@ tags:
|
||||
- 'go-ipfs'
|
||||
---
|
||||
|
||||
[go-ipfs 0.4.14](https://dist.ipfs.io/#go-ipfs) has been released today. Not
|
||||
only have we improved memory and CPU usage but we also managed to fix a lot of
|
||||
bugs, ship a major improvement to IPNS performance and lots of refactoring! \o/
|
||||
[go-ipfs 0.4.14](https://dist.ipfs.io/#go-ipfs) has been released today. Not only have we improved memory and CPU usage but we also managed to fix a lot of bugs, ship a major improvement to IPNS performance and lots of refactoring! \o/
|
||||
|
||||
- [Refactoring](#refactoring)
|
||||
- [IPNS Improvements](#ipns-improvements)
|
||||
@@ -20,56 +18,24 @@ bugs, ship a major improvement to IPNS performance and lots of refactoring! \o/
|
||||
|
||||
# Refactoring
|
||||
|
||||
The release took longer than expected due to our refactoring and extracting of
|
||||
our [commands library](https://github.com/ipfs/go-ipfs-cmds). This refactor had
|
||||
two stages. The first round of the refactor disentangled the commands code from
|
||||
core go-ipfs code, allowing us to move it out into a separate repository. The
|
||||
code was previously very entangled with the go-ipfs codebase and not usable for
|
||||
other projects. The second round of the refactor had the goal of fixing several
|
||||
major issues around streaming outputs, progress bars, and error handling. It
|
||||
also paved the way for us to more easily provide an API over other transports,
|
||||
such as websockets and unix domain sockets. It took a while to flush out all
|
||||
the kinks on such a massive change. We're pretty sure we've got most of them,
|
||||
but if you notice anything weird,
|
||||
[please let us know](https://github.com/ipfs/go-ipfs/issues/new).
|
||||
The release took longer than expected due to our refactoring and extracting of our [commands library](https://github.com/ipfs/go-ipfs-cmds). This refactor had two stages. The first round of the refactor disentangled the commands code from core go-ipfs code, allowing us to move it out into a separate repository. The code was previously very entangled with the go-ipfs codebase and not usable for other projects. The second round of the refactor had the goal of fixing several major issues around streaming outputs, progress bars, and error handling. It also paved the way for us to more easily provide an API over other transports, such as websockets and unix domain sockets. It took a while to flush out all the kinks on such a massive change. We're pretty sure we've got most of them, but if you notice anything weird, [please let us know](https://github.com/ipfs/go-ipfs/issues/new).
|
||||
|
||||
# IPNS Improvements
|
||||
|
||||
Beyond that, we've added a new experimental way to use IPNS. With the new
|
||||
pubsub IPNS resolver and publisher, you can subscribe to updates of an IPNS
|
||||
entry, and the owner can publish out changes in real time. With this, IPNS can
|
||||
become nearly instantaneous. To make use of this, simply start your go-ipfs
|
||||
daemon with the `--enable-namesys-pubsub` option, and all IPNS resolution and
|
||||
publishing will use pubsub. Note that resolving an IPNS name via pubsub without
|
||||
someone publishing it via pubsub will result in a fallback to using the DHT.
|
||||
Please give this a try and let us know how it goes!
|
||||
Beyond that, we've added a new experimental way to use IPNS. With the new pubsub IPNS resolver and publisher, you can subscribe to updates of an IPNS entry, and the owner can publish out changes in real time. With this, IPNS can become nearly instantaneous. To make use of this, simply start your go-ipfs daemon with the `--enable-namesys-pubsub` option, and all IPNS resolution and publishing will use pubsub. Note that resolving an IPNS name via pubsub without someone publishing it via pubsub will result in a fallback to using the DHT. Please give this a try and let us know how it goes!
|
||||
|
||||
# Resource Usage Improvements
|
||||
|
||||
Memory and CPU usage should see a noticeable improvement in this release. We
|
||||
have spent considerable time fixing excess memory usage throughout the codebase
|
||||
and down into go-libp2p. Fixes in peer tracking, bitswap allocation, pinning,
|
||||
and many other places have brought down both peak and average memory usage. An
|
||||
upgraded hashing library, base58 encoding library, and improved allocation
|
||||
patterns all contribute to overall lower CPU usage across the board.
|
||||
Memory and CPU usage should see a noticeable improvement in this release. We have spent considerable time fixing excess memory usage throughout the codebase and down into go-libp2p. Fixes in peer tracking, bitswap allocation, pinning, and many other places have brought down both peak and average memory usage. An upgraded hashing library, base58 encoding library, and improved allocation patterns all contribute to overall lower CPU usage across the board.
|
||||
|
||||
# IPFS Core API
|
||||
|
||||
This release also brings the beginning of the go-ipfs 'Core API'. Once
|
||||
finished, the Core API will be the primary way to interact with go-ipfs using
|
||||
go. Both embedded nodes and nodes accessed over the HTTP API will have the same
|
||||
interface. Stay tuned for future updates and documentation.
|
||||
This release also brings the beginning of the go-ipfs 'Core API'. Once finished, the Core API will be the primary way to interact with go-ipfs using go. Both embedded nodes and nodes accessed over the HTTP API will have the same interface. Stay tuned for future updates and documentation.
|
||||
|
||||
# Note Regarding Insecure Hash Functions
|
||||
|
||||
This release of go-ipfs disallows the usage of insecure hash functions and
|
||||
lengths. go-ipfs does not create these insecure objects for any purpose, but
|
||||
it did allow manually creating them and fetching them from other peers. If you
|
||||
currently have objects using insecure hashes in your local go-ipfs repo, please
|
||||
remove them before updating.
|
||||
This release of go-ipfs disallows the usage of insecure hash functions and lengths. go-ipfs does not create these insecure objects for any purpose, but it did allow manually creating them and fetching them from other peers. If you currently have objects using insecure hashes in your local go-ipfs repo, please remove them before updating.
|
||||
|
||||
# Full Changelog
|
||||
|
||||
As always, you can find the full changelog over at ipfs/go-ipfs's Github
|
||||
repository:
|
||||
https://github.com/ipfs/go-ipfs/blob/master/CHANGELOG.md#0414-2018-03-22
|
||||
As always, you can find the full changelog over at ipfs/go-ipfs's Github repository: https://github.com/ipfs/go-ipfs/blob/master/CHANGELOG.md#0414-2018-03-22
|
||||
|
||||
Reference in New Issue
Block a user