From 45213ce8a02e4f9e1f8ce86b0c6bee99a8f85ece Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Fri, 21 May 2021 22:27:07 +0000 Subject: [PATCH] Update from Forestry.io Marcin Rataj created src/_blog/distributed-wikipedia-mirror-project-update-2021-q2.md --- .../front_matter/templates/blog-post.yml | 1 + ...wikipedia-mirror-project-update-2021-q2.md | 122 ++++++++++++++++++ 2 files changed, 123 insertions(+) create mode 100644 src/_blog/distributed-wikipedia-mirror-project-update-2021-q2.md diff --git a/.forestry/front_matter/templates/blog-post.yml b/.forestry/front_matter/templates/blog-post.yml index bd1e2d73..186aa1d9 100644 --- a/.forestry/front_matter/templates/blog-post.yml +++ b/.forestry/front_matter/templates/blog-post.yml @@ -153,6 +153,7 @@ pages: - src/_blog/audius-uses-ipfs-for-content-streaming-storage-to-empower-artists-creators-worldwide.md - src/_blog/congrats-gitcoin-grants-round-9-winners.md - src/_blog/decentralized-games-at-gamedev.js-jam-2021.md +- src/_blog/distributed-wikipedia-mirror-project-update-2021-q2.md - src/_blog/go-ipfs-0.8.0-and-remote-pinning-is-here.md - src/_blog/hey-ethdenver-hack-on-ipfs-with-these-bounties.md - src/_blog/how-we-put-ipfs-in-brave.md diff --git a/src/_blog/distributed-wikipedia-mirror-project-update-2021-q2.md b/src/_blog/distributed-wikipedia-mirror-project-update-2021-q2.md new file mode 100644 index 00000000..05c5f9a0 --- /dev/null +++ b/src/_blog/distributed-wikipedia-mirror-project-update-2021-q2.md @@ -0,0 +1,122 @@ +--- +title: Distributed Wikipedia Mirror Project Update (2021 Q2) +description: Status update for 2021 Q2, usage instructions, current build process, + and open problems +author: Marcin Rataj +date: 2021-05-31 +permalink: "/2021-05-31-distributed-wikipedia-mirror-project-update" +translationKey: '' +header_image: '' +tags: +- censorship + +--- +[Distributed Wikipedia Mirror Project](https://github.com/ipfs/distributed-wikipedia-mirror#readme) and [Kiwix](https://kiwix.org/) are happy to the announce general availability of updated [English](https://en.wikipedia-on-ipfs.org/) and [Turkish](https://tr.wikipedia-on-ipfs.org) mirrors, along with new languages: [Myanmar](https://my.wikipedia-on-ipfs.org/), [Arabic](https://ar.wikipedia-on-ipfs.org), [Chinese](https://zh.wikipedia-on-ipfs.org) and [Russian](https://ru.wikipedia-on-ipfs.org). + +A handy, up-to-date list can be found at https://ipfs.kiwix.org. + +The idea of a distributed Wikipedia mirror goes back to 2017, when the IPFS Project created a snapshot of English and Turkish languages and put it on IPFS. To learn why we did it, please read the original [Uncensorable Wikipedia on IPFS](https://blog.ipfs.io/24-uncensorable-wikipedia/) post. + +Below is a short status update with improved usage instructions, current build process, open problems, and future work that could be contributed to the project. + +## Improved access to Wikipedia mirrors + +### User-friendly `ipns://{dnslink}` and public gateways + +Browsers with built-in support for IPFS addresses ([Brave](https://brave.com/brave-integrates-ipfs/), [Opera](https://blog.ipfs.io/2020-03-30-ipfs-in-opera-for-android/), or [Firefox](https://www.mozilla.org/en-US/firefox/new/)/[Chromium](https://en.wikipedia.org/wiki/Chromium_(web_browser)) with [IPFS Companion](https://github.com/ipfs/ipfs-companion#readme)) can load the latest snapshot using [DNSLink](https://docs.ipfs.io/concepts/dnslink/): + +- `ipns://{dnslink}` +- `ipns://en.wikipedia-on-ipfs.org` + +To ensure true P2P transport, offline storage and content integrity, you can run your own IPFS node with [IPFS Desktop](https://github.com/ipfs/ipfs-desktop#readme) combined with the [IPFS Companion](https://github.com/ipfs/ipfs-companion#readme) browser extension. You can also use the [Brave browser, which has built-in support for IPFS](https://brave.com/brave-integrates-ipfs/). + + +When it is not possible to run your own IPFS node, one of the [many public gateways](https://ipfs.github.io/public-gateway-checker/) can be used as a proxy for accessing the mirror. For example: + +- [https://dweb.link/ipns/my.wikipedia-on-ipfs.org](https://dweb.link/ipns/my.wikipedia-on-ipfs.org) +- [https://cf-ipfs.com/ipns/my.wikipedia-on-ipfs.org](https://cf-ipfs.com/ipns/my.wikipedia-on-ipfs.org) + +### Robust and immutable `ipfs://{cid}` + +If DNS resolution is blocked, or a public gateway can't be trusted, accessing the immutable snapshot using underlying cryptographic content identifier ([CID](https://docs.ipfs.io/concepts/content-addressing/)) is advised: + +- `ipfs://{cid}` + +The `{cid}` of a specific mirror can be found in [`snapshot-hashes.yml`](https://github.com/ipfs/distributed-wikipedia-mirror/blob/main/snapshot-hashes.yml), or read from its DNSLink record with`ipfs resolve -r /ipns/en.wikipedia-on-ipfs.org`. At the time of writing this post, the English mirror points at `ipfs://bafybeiaysi4s6lnjev27ln5icwm6tueaw2vdykrtjkwiphwekaywqhcjze` + +Sharing CIDs via [sneakernet](https://en.wikipedia.org/wiki/Sneakernet) is a popular way of routing around DNS issues and censorship. Turkish citizens resorted to that in 2017 when [Turkey blocked Wikipedia](https://en.wikipedia.org/wiki/Block_of_Wikipedia_in_Turkey). History does not repeat itself, but it rhymes: Myanmar started experiencing internet blackouts earlier this year: + +

Confirmed: #Myanmar has blocked all language editions of the Wikipedia online encyclopedia, part of a widening post-coup internet censorship regime imposed by the military junta 📚

Network data show restriction in effect on major providers.

📰 Report: https://t.co/Jgc20OBk27 pic.twitter.com/qstGEefO4E

— NetBlocks (@netblocks) February 19, 2021
+ +To address this critical need, we [created a mirror of Myanmar Wikipedia](https://github.com/ipfs/distributed-wikipedia-mirror/pull/83) and [shared both DNSLink and CID](https://github.com/ipfs/distributed-wikipedia-mirror/pull/83#issuecomment-786021082). + +

In response to ongoing internet restrictions / censorship in Myanmar, @Wikipedia in MY is now on @IPFS:https://t.co/trt0AbEMuW

Huge props to @playingwithsid who proposed it, & coordinated w/ native speakers.

Epic implementation effort by @lidelOrg & Kelson of @KiwixOffline!

— dietrich (@dietrich) February 25, 2021
+ +## How to help co-hosting this? + +You can run your own IPFS node and co-host a subset of wikipedia, store a full copy, or even follow collaborative cluster to pull in future updates automatically. + +It is also possible to donate co-hosting costs by pinning specific CID to a remote service. + +### Lazy co-hosting with your own IPFS node + +It is possible to keep a lazy-loaded copy. which does not fetch the entire Wikipedia, but keeps the browsed subset of pages around. + +``` +$ ipfs files cp /ipfs/{cid} /my-wikipedia-snapshot +``` + +One can convert a lazy copy to full one by recursively pinning the [DAG](https://docs.ipfs.io/concepts/glossary/#dag) behind a CID: + +``` +$ ipfs pin add --progress {cid} +``` + +A recursive pin will preload the entire mirror to the local datastore. +Be wary that the English one is far bigger than other ones, and pinning it requires hundreds of gigabytes and may take a very long time. + +The size of a specific mirror can be read with `ipfs files stat /ipfs/{cid}`. + +### Collaborative cluster + +This is an advanced option aimed at server administrators and power users. The `wikipedia` cluster includes all language versions and its size only grows over time. + +``` +$ ipfs-cluster-follow wikipedia run --init wikipedia.collab.ipfscluster.io +``` + +See _Instructions_ at [collab.ipfscluster.io](collab.ipfscluster.io). + +### Donate remote pins + +When co-hosting with your own IPFS node is not possible, one can still help by pinning snapshot CIDs to a remote pinning service. (TODO: insert sentence that summarizes what pinning services are) [Learn how to _work with remote pinning services_](https://docs.ipfs.io/how-to/work-with-pinning-services/). + +## How is a mirror built? + +The current setup relies on [Wikipedia snapshots in the ZIM format](https://download.kiwix.org/zim/wikipedia/) produced by the [Kiwix](https://kiwix.org/) project. + +We don't have a web-based reader of ZIM archives (yet – more in the next section), and the way we produce a mirror requires an expensive (TODO: expensive or complex? expensive implies money) build process: +1. Unpacking ZIM archive with [openzim/zim-tools](https://github.com/openzim/zim-tools) +2. Adjusting JS to fixup unpacked form +3. Import snapshot to IPFS +4. Include original ZIM inside of unpacked IPFS snapshot + +While this works, the need for unpacking and customizing the snapshot makes it difficult to reliably produce updates. And including the original ZIM for use with [Kiwix offline reader](https://www.kiwix.org/en/kiwix-reader), partially duplicates the data. + +We would love to mirror more languages, and increase the update cadence, but for that to happen we need to remove the need for unpacking ZIM archives. We will be looking into putting [all ZIMs from Kiwix](https://download.kiwix.org/zim/wikipedia/) on IPFS and archiving them for long term storage on [Filecoin](https://filecoin.io/) as part of https://farm.openzim.org pipeline. + +## Help Wanted and Open Problems + +If you are still reading this, there is a high chance you are interested in improving the way the distributed Wikipedia mirror works. + +Below are areas that could use a helping hand, and ideas looking for someone to explore them. + +- **Search.** There's no search function currently. Leveraging the index present in ZIM, or building a DAG-based search index optimized for use in web browsers would make existing mirrors more useful. See [distributed-wikipedia-mirror/issues/76](https://github.com/ipfs/distributed-wikipedia-mirror/issues/76). + +- **Web-based ZIM reader.** The biggest impact for the project would be to create a web-based reader capable of browsing original ZIM archives without the need for unpacking them, nor installing any dedicated software. Want to help make it a reality? See [kiwix-js/issues/659](https://github.com/kiwix/kiwix-js/issues/659) + +- **Improving the way ZIM is represented on IPFS.** When we store an original ZIM on IPFS, the DAG is produced by `ipfs add --cid-version 1`. This works fine, but with additional research on customizing DAG creation, we may improve deduplication and speed when doing range requests for specific bytes. There are different stages to explore here: + - Stage 1: Invest some time to benchmark parameter space to see if low hanging fruits exists + - Stage 2: Create a DAG builder that understands ZIM format and maximizes deduplication of image assets by representing them as sub-DAGs with dag-pb files. + - Stage 3: Research augmenting or replacing ZIM with [IPLD](https://ipld.io/). How can we maximize block deduplication across all snapshots and languages? How would an IPLD-based search index work? + If any of this sound interesting, please comment in [distributed-wikipedia-mirror/issues/42](https://github.com/ipfs/distributed-wikipedia-mirror/issues/42) \ No newline at end of file