March 15, 2018 at 23:25

Blockchain on Go. Part 3: read-only memory and command line interface

Original author: Ivan Kuznetsov
  • Transfer
  • Tutorial
Content

  1. Blockchain on Go. Part 1: Prototype
  2. Blockchain on Go. Part 2: Proof-of-Work
  3. Blockchain on Go. Part 3: read-only memory and command line interface
  4. Blockchain on Go. Part 4: Transactions, Part 1
  5. Blockchain on Go. Part 5: Addresses
  6. Blockchain on Go. Part 6: Transactions, Part 2
  7. Blockchain on Go. Part 7: Network

Introduction


In the previous part, we built a blockchain with a PoW system and the possibility of mining. Our implementation is getting closer to a fully functional blockchain, but it still lacks some important functions. Today we will begin to store the blockchain in the database, after that we will make a command line interface for operations with the blockchain. In essence, blockchain is a distributed database. For now, we omit the “distributed” and focus on the “database”.

Database selection


So far, we do not have a database in the implementation, we just create blocks when the program starts and store them in memory. We cannot reuse or share our blockchain with others, so we need to save it to disk.

What database do we need? In fact, any will do. The Bitcoin Paper says nothing about a specific database, so the choice is up to the developer. Bitcoin Core , which was originally published by Satoshi Nakamoto and which is currently the reference implementation of Bitcoin, uses LevelDB (although it was introduced to the client only in 2012). And we will use ...

Boltdb


Because:

  1. She is simple and minimal.
  2. It is implemented on Go
  3. She does not need to start the server
  4. It allows us to build the data structures we need.

From BoltDB README :
Bolt is just a key-value repository, inspired by Howard Chu’s LMDB project . The aim of the project is to provide a simple, fast and reliable database for projects that do not require a full-fledged database server such as Postgres or MySQL.

Since Bolt is intended to be used as such a low-level element of functionality, simplicity is key. The API will be small and focus only on getting values ​​and setting values. It's all!
Sounds perfect for our needs! Take a moment to review the base.

BoltDB is a key-value storage, which means that there are no tables, as in relational DBMSs (MySQL, PostgreSQL, etc.), there are no rows and columns. Instead, data is stored in key-value pairs (as in the Golang map). Pairs are stored in “baskets” that are designed to group similar pairs (like tables in relational DBMSs). Thus, to get the value, you need to know the basket and the key.

The important thing about BoltDB is that there are no data types: keys and values ​​are byte arrays. Since we store Go structures (in particular Block), we must serialize them, that is, implement a mechanism for translating the structure into a byte array and restoring it back from the array. We will use encoding / gobfor this, although JSON, XML, Protocol Buffersalso suitable. We use encoding/gobbecause it is simple and it is part of the Go standard library.

Database structure


Before we begin to implement persistent logic, we must decide how we will store our data in the database. And for this we will use the method that we use Bitcoin Core.

If in a simple way, then Bitcoin Core uses two “baskets” for storing data.

  1. blocks stores metadata describing all blocks in a chain
  2. chainstate stores the state of the chain, which is all unspent transaction outputs and some metadata

Blocks are also stored as separate files on disk. This is done to improve performance: reading one block does not require loading all (or some) into memory. This we will not implement.

In blockspairs key->valueit is: In pairs it is: (A detailed explanation can be found here ) Since we do not have transactions yet, we will only make a basket . In addition, as mentioned above, we will store the entire database in one file, without storing blocks in separate files. Therefore, we do not need anything related to file numbers. Therefore, the pairs that we will use are:
  1. 'b' + 32-байтовый хэш блока -> запись индекса блока
  2. 'f' + 4-байтовый номер файла -> запись информации о файле
  3. 'l' -> 4-байтовый номер файла: номер использованного файла для последнего блока
  4. 'R' -> 1-байтовый boolean : находимся ли мы в процессе переиндексации
  5. 'F' + 1-байтовая длина имени флага + строка имени флага -> 1 байт boolean: различные флаги, которые могут быть включены или выключены
  6. 't' + 32-байтовый хеш транзакции -> запись индекса транзакции

chainstatekey->value
  1. 'c' + 32-байтовый хеш транзакции -> запись о непотраченном выходе транзакции для этой транзакции
  2. 'B' -> 32-байтовый хеш блока: хеш блока, до которого база данных представляет собой неизрасходованные выходы транзакции



blockskey->value

  1. 32-byte block hash -> block structure (serialized)
  2. 'l' -> hash of the last block in the chain

This is all we need to know in order to implement the mechanism of constancy (persistence).

Serialization


As mentioned earlier, in BoltDB values ​​can only be of []bytetype, and we want to store the structure Blockin the database. We will use encoding/gobto serialize structures.

Let's implement a method Serializefor Block(error handling omitted for brevity)

func (b *Block) Serialize() []byte {
	var result bytes.Buffer
	encoder := gob.NewEncoder(&result)
	err := encoder.Encode(b)
	return result.Bytes()
}

Everything is simple here: at the beginning, we declare a buffer where the serialized data will be stored, then we initialize the gobencoder and encode the block, the result is returned as an array of bytes.

Now we need a deserialization function that receives an array of bytes and returns Block. This will not be a method, but an independent function:

func DeserializeBlock(d []byte) *Block {
	var block Block
	decoder := gob.NewDecoder(bytes.NewReader(d))
	err := decoder.Decode(&block)
	return &block
}

That's all we need for serialization.

Persistence


Let's start with the function NewBlockchain. Now she creates a new instance Blockchainand adds a genesis block to it. We want to do the following:

  1. Open db file
  2. Check if the blockchain is saved there
  3. If he is there:
    1. Create new instance Blockchain
    2. Set the tip of the instance Blockchainto the hash of the last block stored in the database

  4. If there is no existing blockchain

    1. Create Genesis Block
    2. Save to DB
    3. Save Genesis Hash as Hash of Last Last Block
    4. Create a new instance Blockchainwith a tip pointing to the genesis block

In code, it looks like this:

func NewBlockchain() *Blockchain {
	var tip []byte
	db, err := bolt.Open(dbFile, 0600, nil)
	err = db.Update(func(tx *bolt.Tx) error {
		b := tx.Bucket([]byte(blocksBucket))
		if b == nil {
			genesis := NewGenesisBlock()
			b, err := tx.CreateBucket([]byte(blocksBucket))
			err = b.Put(genesis.Hash, genesis.Serialize())
			err = b.Put([]byte("l"), genesis.Hash)
			tip = genesis.Hash
		} else {
			tip = b.Get([]byte("l"))
		}
		return nil
	})
	bc := Blockchain{tip, db}
	return &bc
}

Let's analyze the code in parts.

db, err := bolt.Open(dbFile, 0600, nil)

This is the standard way to open a BoltDB file. Please note that it will not return an error if there is no file.

err = db.Update(func(tx *bolt.Tx) error {
...
})

In BoltDB, database operations are performed as part of a transaction. There are two types of transactions: read-only and read-write. Here we open a read-write transaction (db.Update(...)), because we plan to put the genesis block in the database.

b := tx.Bucket([]byte(blocksBucket))
if b == nil {
	genesis := NewGenesisBlock()
	b, err := tx.CreateBucket([]byte(blocksBucket))
	err = b.Put(genesis.Hash, genesis.Serialize())
	err = b.Put([]byte("l"), genesis.Hash)
	tip = genesis.Hash
} else {
	tip = b.Get([]byte("l"))
}

This is the core of the function. Here we get a basket that stores our blocks: if it exists, then we read the key lfrom it, if it does not exist, then we generate the genesis of the block, create a basket, save the block in it and update the key lthat stores the hash of the last block in the chain.

Also notice the new way to create Blockchain:

bc := Blockchain{tip, db}

We do not store all the blocks; instead, we only store the tip of the chain. We also store the connection to the database, because we want to open it once and keep it open while the program is running. This is how the structure Blockchainlooks now:

type Blockchain struct {
	tip []byte
	db  *bolt.DB
}

The next thing we want to change is the method AddBlock: adding blocks to the chain is now not as simple as adding an element to an array. From now on we will store blocks in the database:

func (bc *Blockchain) AddBlock(data string) {
	var lastHash []byte
	err := bc.db.View(func(tx *bolt.Tx) error {
		b := tx.Bucket([]byte(blocksBucket))
		lastHash = b.Get([]byte("l"))
		return nil
	})
	newBlock := NewBlock(data, lastHash)
	err = bc.db.Update(func(tx *bolt.Tx) error {
		b := tx.Bucket([]byte(blocksBucket))
		err := b.Put(newBlock.Hash, newBlock.Serialize())
		err = b.Put([]byte("l"), newBlock.Hash)
		bc.tip = newBlock.Hash
		return nil
	})
}

Consider the code piece by piece:

err := bc.db.View(func(tx *bolt.Tx) error {
	b := tx.Bucket([]byte(blocksBucket))
	lastHash = b.Get([]byte("l"))
	return nil
})

This is a different (read-only) type of BoltDB transaction. Here we get the hash of the last block from the database to use it to mine the hash of the new block.

newBlock := NewBlock(data, lastHash)
b := tx.Bucket([]byte(blocksBucket))
err := b.Put(newBlock.Hash, newBlock.Serialize())
err = b.Put([]byte("l"), newBlock.Hash)
bc.tip = newBlock.Hash

After mining a new block, we save the serialized view in the database and update the key l, which now saves the hash of the new block.

Done! It was not difficult, right?

Checking Blockchain


All new blocks are now stored in the database, so we can reopen the blockchain and add a new block to it. But after implementing this, we lose one useful feature: we cannot print blocks because we no longer store them in an array. Let's fix it.

BoltDB allows you to go through all the keys in the basket, but all the keys are stored in sorted by bytes, and we want the blocks to be printed in the order in which they are placed on the blockchain. Also, since we do not want to load all the blocks into memory (our blockchain can be very huge), we will read them one by one. For this purpose, we need a blockchain iterator:

type BlockchainIterator struct {
	currentHash []byte
	db          *bolt.DB
}

An iterator will be created every time we want to iterate over the blocks in the blockchain and it will store the block hash of the current iteration and the connection to the database. Because of the latter, the iterator is logically bound to the blockchain (this is an instance Blockchainthat stores the connection to the database) and, thus, is created in the method Blockchain:

func (bc *Blockchain) Iterator() *BlockchainIterator {
	bci := &BlockchainIterator{bc.tip, bc.db}
	return bci
}

Note that the iterator first points to the tip of the blockchain, so the blocks will be received from top to bottom, from the newest to the oldest. In fact, choosing a tip means “voting” for the blockchain . The blockchain can have several branches and the longest of them is considered the main one. After receiving the tip (it can be any block in the blockchain), we can recreate the entire blockchain and find its length, and the work necessary to build it. This fact also means that the tip is a kind of blockchain identifier.

BlockchainIteratordoes only one thing: returns the next block from the blockchain.

func (i *BlockchainIterator) Next() *Block {
	var block *Block
	err := i.db.View(func(tx *bolt.Tx) error {
		b := tx.Bucket([]byte(blocksBucket))
		encodedBlock := b.Get(i.currentHash)
		block = DeserializeBlock(encodedBlock)
		return nil
	})
	i.currentHash = block.PrevBlockHash
	return block
}

That's all about the database!

Command Line Interface (CLI)


So far, our implementation does not provide us with any interface for interacting with the program: we just ran NewBlockchain, bc.AddBlockin main. It's time to improve it! We want to have the following commands:

blockchain_go addblock "Pay 0.031337 for a coffee"
blockchain_go printchain

All command line related operations will be handled by the structure CLI

type CLI struct {
	bc *Blockchain
}

The "entry point" of the structure is a function Run

func (cli *CLI) Run() {
	cli.validateArgs()
	addBlockCmd := flag.NewFlagSet("addblock", flag.ExitOnError)
	printChainCmd := flag.NewFlagSet("printchain", flag.ExitOnError)
	addBlockData := addBlockCmd.String("data", "", "Block data")
	switch os.Args[1] {
	case "addblock":
		err := addBlockCmd.Parse(os.Args[2:])
	case "printchain":
		err := printChainCmd.Parse(os.Args[2:])
	default:
		cli.printUsage()
		os.Exit(1)
	}
	if addBlockCmd.Parsed() {
		if *addBlockData == "" {
			addBlockCmd.Usage()
			os.Exit(1)
		}
		cli.addBlock(*addBlockData)
	}
	if printChainCmd.Parsed() {
		cli.printChain()
	}
}

We use the standard flag package to parse command line arguments.

addBlockCmd := flag.NewFlagSet("addblock", flag.ExitOnError)
printChainCmd := flag.NewFlagSet("printchain", flag.ExitOnError)
addBlockData := addBlockCmd.String("data", "", "Block data")

To get started, we create two subcommands addblockand printchainthen add a flag -datato the first. printchaindoes not require any flags.

switch os.Args[1] {
case "addblock":
	err := addBlockCmd.Parse(os.Args[2:])
case "printchain":
	err := printChainCmd.Parse(os.Args[2:])
default:
	cli.printUsage()
	os.Exit(1)
}

Then we check the command specified by the user and parse the associated subcommand.

if addBlockCmd.Parsed() {
	if *addBlockData == "" {
		addBlockCmd.Usage()
		os.Exit(1)
	}
	cli.addBlock(*addBlockData)
}
if printChainCmd.Parsed() {
	cli.printChain()
}

Next, we check which subcommand we parsed and run the associated function. 

func (cli *CLI) addBlock(data string) {
	cli.bc.AddBlock(data)
	fmt.Println("Success!")
}
func (cli *CLI) printChain() {
	bci := cli.bc.Iterator()
	for {
		block := bci.Next()
		fmt.Printf("Prev. hash: %x\n", block.PrevBlockHash)
		fmt.Printf("Data: %s\n", block.Data)
		fmt.Printf("Hash: %x\n", block.Hash)
		pow := NewProofOfWork(block)
		fmt.Printf("PoW: %s\n", strconv.FormatBool(pow.Validate()))
		fmt.Println()
		if len(block.PrevBlockHash) == 0 {
			break
		}
	}
}

This code is similar to the one that was before. The only difference is what we use now BlockchainIteratorto iterate over the blocks in the blockchain.

Also, do not forget to change the function mainaccordingly:

func main() {
	bc := NewBlockchain()
	defer bc.db.Close()
	cli := CLI{bc}
	cli.Run()
}

Note that a new one Blockchainis created regardless of which command line arguments were passed.

That's all! Check that everything works as we expect:

$ blockchain_go printchain
No existing blockchain found. Creating a new one...
Mining the block containing "Genesis Block"
000000edc4a82659cebf087adee1ea353bd57fcd59927662cd5ff1c4f618109b
Prev. hash:
Data: Genesis Block
Hash: 000000edc4a82659cebf087adee1ea353bd57fcd59927662cd5ff1c4f618109b
PoW: true
$ blockchain_go addblock -data "Send 1 BTC to Ivan"
Mining the block containing "Send 1 BTC to Ivan"
000000d7b0c76e1001cdc1fc866b95a481d23f3027d86901eaeb77ae6d002b13
Success!
$ blockchain_go addblock -data "Pay 0.31337 BTC for a coffee"
Mining the block containing "Pay 0.31337 BTC for a coffee"
000000aa0748da7367dec6b9de5027f4fae0963df89ff39d8f20fd7299307148
Success!
$ blockchain_go printchain
Prev. hash: 000000d7b0c76e1001cdc1fc866b95a481d23f3027d86901eaeb77ae6d002b13
Data: Pay 0.31337 BTC for a coffee
Hash: 000000aa0748da7367dec6b9de5027f4fae0963df89ff39d8f20fd7299307148
PoW: true
Prev. hash: 000000edc4a82659cebf087adee1ea353bd57fcd59927662cd5ff1c4f618109b
Data: Send 1 BTC to Ivan
Hash: 000000d7b0c76e1001cdc1fc866b95a481d23f3027d86901eaeb77ae6d002b13
PoW: true
Prev. hash:
Data: Genesis Block
Hash: 000000edc4a82659cebf087adee1ea353bd57fcd59927662cd5ff1c4f618109b
PoW: true

(the sound of opening a beer can )

References


Original article
The first part of a series of articles
Sources
Bitcoin Core Data Storage
BoltDB
encoding / gob
flag
Tags:

Also popular now: