Ethereum Swarm decentralized data warehouse

The Ethereum blockchain is interesting for its smart contracts, as well as the ability to create decentralized DApp applications (Decentralized Application). However, such an application needs a decentralized data warehouse.

Storing large amounts of data on the blockchain can cost a lot of money. Decentralized storages like Ethereum Swarm come to the rescue (“swarm” means “swarm”, “heap”). In short, Ethereum Swarm is a program code that runs on an Ethereum peer-to-peer network. It provides decentralized storage of data on the disks of nodes, the owners of which give their resources to common use.

In this article, we will talk about how to set up a local Ethereum Swarm host for an Ethereum private network in order to test technology and develop decentralized applications that store data in Ethereum Swarm.

Why DApp needs a decentralized data warehouse

The Ethereum blockchain (like other blockchains) is a distributed journal that is very well protected from fraud and attack by malicious users.

This protection is achieved, on the one hand, using hashing and cryptographic algorithms, algorithms for adding new blocks to the blockchain, and on the other hand, with a huge number of Ethereum nodes that provide data storage and transaction processing. These sites are located all over the world. No one has enough resources to “break” the blockchain, and this provides confidence in the information recorded in it.

“SkyNet everywhere and nowhere is a single center. There's nothing to turn off. ”(Terminator 3: Rise of the Machines)

But as I noted above, the blockchain is not suitable for storing large data, such as documents, images, product descriptions, orders, etc. Theoretically, of course, you can write them to the blockchain, but the cost of recording will be too high.

Where to store DApp data of large volume?

For example, you are going to use the blockchain to fix the copyrights to the images, save forever the details of the order placed in the online store, or save scans of paper documents. Some other data warehouse will be required here.

Of course, to store DApp data, you can rent a server, a place in the cloud storage, or use some other traditional storage. But servers, clouds, and storage are controlled by companies or individuals. They may be blocked for any reason or attacked by intruders.

When using centralized storage, the main advantage of DApp is lost - resistance to attacks and malicious actions. Moreover, your application ceases to be decentralized. This means that trust in DApp is also lost. After all, if you block or disrupt the operation of the centralized storage, access to some of the information recorded by DApp with the blockchain will disappear and the whole point of using DApp will disappear.

How Ethereum Swarm Works

Imagine that tens of thousands of nodes of the Ethereum Swarm network are scattered all over the world, which provide their resources for storing data downloaded by users. It is assumed that site owners will receive a reward for providing resources, while the cost of posting data will be lower than in traditional cloud storage.

When a user uploads a file to the Ethereum Swarm network, this file first goes to one of the nodes. Next, the file is replicated to other network nodes during synchronization. It uses the bzz protocol, which runs on top of the Ethereum network.

As long as at least one Ethereum Swarm host is running, the downloaded file remains available. This ensures the reliability of data storage, as it’s almost impossible to disable or block a huge number of Ethereum Swarm nodes.

You can read more about all this on the Ethereum Swarm website .

The documentation noted that to date, Ethereum Swarm is implemented in version 0.2 as proof of concept (POC, proof of concept). In this version, the safety of the downloaded data is not guaranteed. A stable release is expected in the second quarter of 2018, so the wait is not long. At the same time, it is planned to create a reward system for providing resources for the Ethereum Swarm network.

However, we will not wait for anything, but we will start testing Ethereum Swarm right now. We will create our own Ethereum Swarm node in our private Ethereum network.

Installing Ethereum Swarm and Geth on Debian and Ubuntu

I will talk about installing Ethereum Swarm on Debian and Ubuntu servers. In general, you need to follow this instruction , but there are some nuances.

Go installation

Before starting work it is very important to install Go of the new version, not lower than 1.9.2. The Debian and Ubuntu repositories may have older versions of Go, so we install from the source.

Download and unpack the sources under an unprivileged user:

$ curl -O https://storage.googleapis.com/golang/go1.9.2.linux-amd64.tar.gz
$ sudo tar -C /usr/local -xzf go1.9.2.linux-amd64.tar.gz

We create the user directory go and set the environment variables:

$ mkdir -p ~/go; echo "export GOPATH=$HOME/go" >> ~/.bashrc
$ echo "export PATH=$PATH:$HOME/go/bin:/usr/local/go/bin" >> ~/.bashrc
$ source ~/.bashrc

Check that the environment variables are set:

$ printenv | grep go
PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/frolov/go/bin:/usr/local/go/bin
GOPATH=/home/frolov/go

On a Debian operating system, use the .profile file instead of the .bashrc file. You can simply copy:

cp .bashrc .profile

Check the go version:

$ go version
go version go1.9.2 linux/amd64

The go version must be at least 1.9.2.

If you previously installed the old version of go from the OS repository, delete it like this:

$ sudo apt-get remove golang-go
$ sudo apt-get remove --auto-remove golang-go

Install Geth and Ethereum Swarm

Download the source code from the repository:

$ mkdir -p $GOPATH/src/github.com/ethereum
$ cd $GOPATH/src/github.com/ethereum
$ git clone https://github.com/ethereum/go-ethereum
$ cd go-ethereum
$ git checkout master
$ go get github.com/ethereum/go-ethereum

We start compiling the geth client and the swarm daemon:

$ go install -v ./cmd/geth
$ go install -v ./cmd/swarm

Check the version of geth and swarm installed:

$ geth version
Geth
Version: 1.8.0-unstable
Architecture: amd64
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.9.2
Operating System: linux
GOPATH=/home/frolov/go
GOROOT=/usr/local/go
$ swarm version
Swarm
Version: 1.8.0-unstable
Network Id: 0
Go Version: go1.9.2
OS: linux
GOPATH=/home/frolov/go
GOROOT=/usr/local/go

Preparing a private blockchain to launch Ethereum Swarm

First of all, create the genesis.json file in the user's home directory:

{
  "config": {
     "chainId": 1907,
     "homesteadBlock": 0,
     "eip155Block": 0,
     "eip158Block": 0
  },
  "difficulty": "40",
  "gasLimit": "5100000",
  "alloc": {}
}

Next, create a subdirectory node1 in the home directory:

$ mkdir node1

Node initialization can be done using the init_node.sh batch file:

geth --datadir node1 account new
geth --datadir node1 init genesis.json

When you run this file, an account will be created and a password will be requested, which you need to save in a safe place.

Create a start_node.sh file to start the node:

geth --datadir node1 --nodiscover --mine --minerthreads 1 --maxpeers 0 --verbosity 3 --networkid 98765 --rpc --rpcapi="db,eth,net,web3,personal,web3" console

Run this file and wait until the DAG generation is complete.

Using the attach_node.sh file, you can open the geth console and connect to the private node:

geth --datadir node1 --networkid 98765 attach ipc://home/frolov/node1/geth.ipc

Running the swarm daemon

Here you will need the address of the account created during the initialization phase of the site. If you haven’t saved it, it’s okay. Just go to the geth console opened with the attach_node.sh script and issue the following command there:

> web3.eth.accounts
["0xcd9fcb450c858d1a7678a2bccf36ea5decd2b09b"]

The command will show the address of the created accounts. Immediately after initialization, there will be only one address.

To start the Ethereum Swarm daemon in single-node mode (Singleton), prepare the swarm_start.sh command file:

swarm --bzzaccount cd9fcb450c858d1a7678a2bccf36ea5decd2b09b --datadir "/home/ethertest/data/node1" --maxpeers 0 -nodiscover --verbosity 4 --ens-api /home/ethertest/data/node1/geth.ipc

Indicate in it the address of the account created on your site, without "0x".

When starting the daemon, you will need to enter the password for the previously created account:

$ sh swarm_start.sh
Unlocking swarm account 0xCD9Fcb450C858D1A7678a2bCCf36EA5decd2B09B [1/3]
Passphrase:

Upload file to Ethereum Swarm

The easiest way to upload a file is with the swarm command with the up parameter. Additionally, you need to specify the path to the downloaded file:

$ swarm up start_node.sh
f2073b8f0cf0cfe1e165060882da71a37bb6fd97bdec6be71b4f66ebcf0aba9f

This command will return the hash of the downloaded file. The hash can be used to read the file. You can do this with the wget or curl commands:

$ wget http://localhost:8500/bzz:/f2073b8f0cf0cfe1e165060882da71a37bb6fd97bdec6be71b4f66ebcf0aba9f/
$ curl http://localhost:8500/bzz:/f2073b8f0cf0cfe1e165060882da71a37bb6fd97bdec6be71b4f66ebcf0aba9f/

The wget command allows you to save the contents of a file to a local disk. Use the -O option to specify a file name. The curl command will output the contents of the file to the console, so in this form it should not be used to view the contents of binary files. A slash is required at the end of the URL, otherwise a redirect will occur.

When a file is uploaded to Ethereum Swarm as described above, a manifest is created and saved for it. This is a header describing the content available in the repository by the given identifier.

Below, we downloaded the Net-Ethereum-0.28.tar.gz file with the simple swarm up command:

$ swarm up Net-Ethereum-0.28.tar.gz
8da3713d49c62740f5ab594b06173975ac97cb3dd3848ae996484ec264a10e2f

Now, by specifying the bzz-list protocol, we can view the manifest:

$ curl http://localhost:8500/bzz-list:/8da3713d49c62740f5ab594b06173975ac97cb3dd3848ae996484ec264a10e2f/
{"entries":[{"hash":"543ee6e744f93de76ac132b8ab71982e32beaf90d1005e771dde003b2a4a54c3","path":"/","contentType":"application/gzip","mode":420,"size":12403,"mod_time":"2018-01-13T14:57:54+03:00"}]}

The manifest will be shown in JSON format.

In the manifest let it be stored to the file (file name), its size, type (Content Type), date and time of modification, and also the hash of the file.

To extract the contents of a file by its identifier and save it under the name t.tar.gz, do this:

$ wget -O t.tar.gz http://localhost:8500/bzz:/8da3713d49c62740f5ab594b06173975ac97cb3dd3848ae996484ec264a10e2f/ 

Download directories with subdirectories

To recursively load a directory with its contents in the Ethereum Swarm repository, specify the --recursive parameter:

$ swarm --recursive up Net-Ethereum/
4fb1f2270381c022461037151f70ce081082f0ae1a2a23d8c7ea602da69b4115

The manifest will display information about all the files in the loaded subdirectory:

Other features of Ethereum Swarm

In this article, I did not talk about all the features of Ethereum Swarm, limiting myself to only the most necessary.

In addition to loading individual files and directory contents, you can add data to the manifest. This creates a new data item with an augmented manifest and with added files.

It is possible to download files without a manifest, as well as read them in the so-called “raw” form.

When reading a file, you can specify its name (in whole or in part). In this case, the ENS (Ethereum Name Service) name service will be involved. Such an opportunity will allow you to get only some files from a block united by a common manifest.

I also did not describe the SWarm Accounting Protocol (SWAP) and some other features. The Ethereum Swarm project is actively developing, it will definitely bring something else interesting for developers of decentralized applications.

Perl Net :: Ethereum :: Swarm Module

In order to work with the Ethereum Swarm decentralized data warehouse on Perl systems, I developed and posted the Net :: Ethereum :: Swarm module on CPAN .

With this module, you can upload text and binary files to Ethereum Swarm, receive a manifest for the downloaded data, and also upload files from Ethereum Swarm by their identifier.

The Net :: Ethereum :: Swarm module works with the Ethereum Swarm host using HTTP GET and POST requests. Using requests is described in The HTTP API section of the documentation.

Interaction via HTTP requests can be easily implemented in almost any programming language.

My other books and articles on the Ethereum blockchain and smart contracts

• Book Creating Smart Solidity Contracts for the Ethereum Blockchain. Practical guide
• Solidity Contract Experiments on the Ethereum Blockchain Rinkeby Test Network
• Net :: Ethereum Solidity Contract Module