Dive into Ethereum development. Part 5: Oraclize

    Access to large files and various external dynamic data is often a very important part of a decentralized application. At the same time, Ethereum does not provide for an outward circulation mechanism in itself - smart contracts can only be read and written within the blockchain itself. In this article we will consider Oraclize, which just gives the opportunity to interact with the outside world by querying virtually any Internet resources. A related topic is IPFS, briefly mentioning it.



    IPFS


    IPFS is a distributed file system with content addressing. This means that the content of any file added there is considered a unique hash. The same hash is then used to search for and retrieve this content from the network.
    The main information is already described in this article and in several others, so there is no point in repeating.

    Why use IPFS in conjunction with Ethereum?


    Any volumetric content stored on the blockchain is too expensive and harmful to the network. Therefore, the best option is to save any link to the file that is located in the off-line repository, not necessarily IPFS. But IPFS has several advantages:

    • A link to a file is a hash unique to the specific contents of the file, so if we put this hash on the blockchain, then we can be sure that the file received from it is the one that was originally added, the file cannot be changed.
    • Distributed system insures against the inaccessibility of a specific server (due to blocking or other reasons)
    • The link to the file and the hash confirmation are combined into one line, which means you can write less to the blockchain and save gas

    Among the shortcomings we can mention that since there is no central server, then for the availability of the files it is necessary that at least one of these files be “distributed”. But if you have a specific file, then it is easy to connect to the distributors - launch the ipfs-daemon and add the file via ipfs add.

    The technology is very suitable for the ideology of decentralization, therefore, considering Oraclize now, we will more than once encounter the use of IPFS in different oracle mechanisms.

    Oraclize


    To perform almost any useful work, a smart contract needs to receive new data. At the same time, there is no built-in ability to execute a request from the blockchain to the outside world. You can of course add everything that is required by transactions manually, but it is impossible to verify where this data came from and their accuracy. Plus, you may need to organize additional infrastructure for the rapid updating of dynamic data, such as exchange rates. And updates with a fixed interval will lead to gas overspending.

    Therefore the service provided by Oraclize, it is necessary by the way: in a smart contract you can send a request to almost any API or resource on the Internet, be sure that the data came from the specified resource unchanged, and use the result in the same smart contract.

    Oraclize is not only an Ethereum service, similar functionality is provided to other blockchains, but we will describe only a bundle with Ethereum.

    Beginning of work


    All you need to get started is to add yourself to the project one of the oraclizeAPI files from the repository . You only need to choose the appropriate compiler for your version (solc): oraclizeAPI_0.5.sol for versions starting at 0.4.18, oraclizeAPI_0.4.sol for versions from 0.4.1, oraclizeAPI_pre0.4.sol for everything older, support This version is already discontinued. If you use truffle, do not forget to rename the file to usingOraclize - it requires that the name of the file and the contract match.

    By including the appropriate file in your project, inherit the contract from usingOraclize. And you can start using Oracle, which boils down to two main things: sending a request using a helper oraclize_query, and then processing the result in a function__callback. The simplest smart contract (to get the current price of air in dollars) may look like this:

    pragma solidity 0.4.23;
    import"./usingOraclize.sol";
    contract ExampleContract is usingOraclize {
        string public ETHUSD;
        event updatedPrice(string price);
        event newOraclizeQuery(string description);
        functionExampleContract() payable{
            updatePrice();
        }
        function__callback(bytes32 myid, string result) {
            require (msg.sender == oraclize_cbAddress());
            ETHUSD = result;
            updatedPrice(result);
        }
        functionupdatePrice() payable{
            if (oraclize_getPrice("URL") > this.balance) {
                newOraclizeQuery("Oraclize query was NOT sent, please add some ETH to cover for the query fee");
            } else {
                newOraclizeQuery("Oraclize query was sent, standing by for the answer..");
                oraclize_query("URL", "json(https://api.coinmarketcap.com/v1/ticker/ethereum/?convert=USD).0.price_usd");
            }
        }
    }

    Function sending request - updatePrice. You can see that first there is a check that is oraclize_getPrice(“URL”)greater than the current balance of the contract. This is done because the call oraclize_querymust be paid, the price is calculated as the sum of the fixed commission and the payment of gas for the callback call. “URL”- this is the designation of one of the types of data sources, in this case it is a simple request for https, then we will consider other options. Answers on request can be parsed in advance as json (as in the example) and in several other ways (we will consider further). The __callbackstring with the answer is returned. At the very beginning, it is checked that the call passed from the trusted address oraclize

    All options for using oraclize are based on the same scheme, only data sources differ and the ability to add authentication to __callback. Therefore, in future examples, we will cite only significant differences.

    Price of use


    As already mentioned, additional air is paid for oraclize requests, and it is removed from the balance of the contract, and not the calling address. The only exception is the first request from each new contract, it is provided free of charge. It is also interesting that in the test networks the same mechanics is preserved, but the payment goes on the air of the corresponding network, that is, in testnets, the requests are actually free.

    It has already been mentioned that the request price is made up of two quantities: a fixed commission and a callback call charge. The fixed commission is determined in dollars, and the amount of air is calculated from the current rate. The commission depends on the source of the data and the additional supporting mechanisms that we will focus on. The current price table looks like this:


    As you can see, the price per URL request is equal to a few cents. Is it a lot or a little? To do this, let's look at how much the second part costs - gas for a call to a callback.
    This works according to the following scheme: the amount of air needed to pay for a fixed amount of gas at a fixed price is transferred in advance from the contract together with the request. This amount should be enough to fulfill the callback, and the price should be adequate to the market, otherwise the transaction will not work or will hang for a very long time. At the same time, it is clear that it is not always possible to know the amount of gas in advance, and therefore the fee must be with a margin (the margin is not returned). The default values ​​are 200 thousand gas at a price of 20 gwei. This is enough for an average callback with several entries and some kind of logic. And although the price of 20 gwei may seem too high at the moment (at the time of writing, the average is 4 gwei), but at the moments of the influx of transactions, the market price may suddenly jump and be even higher therefore, in general, these values ​​are close to those actually used. So, with such values ​​and the price of air in the region of $ 500, payment for gas will approach $ 2, so we can say that the fixed commission takes an insignificant part.

    If you know what you are doing, then there is an option to change the limit and price of gas, thus significantly saving on requests.

    The price of gas can be set as a separate function - oraclize_setCustomGasPrice(<цена в wei>). After the call, the price is saved and used in all subsequent requests.
    The limit can be set in the request itself oraclize_query, specifying it with the last argument, for example:

    oraclize_query("URL", "<запрос>", 50000);

    If you have complex logic in __callbackand gas is spent more than 200k, then you will definitely need to set a limit that covers the worst case of gas consumption. Otherwise, if you exceed the limit __callbackjust roll back.

    By the way, oraclize recently received information that requests can be paid outside the blockchain, which will allow you not to spend the entire limit or return the balance (and the payment is not from the contract). We have not had to use it yet, but oraclize offers to contact them at info@oraclize.it, if this option is interesting. So keep in mind.

    How does it work


    Why, inheriting from the usual smart contract, we get functionality that was not originally supported by the blockchain mechanisms? Actually service OrakLayz consists not only of contracts with functions-helpers. The main job of getting data is external service. Smart contracts form applications for access to external data and put them in the blockchain. External service - monitors new blocks of the blockchain and if it detects an application - executes it. Schematically, this can be represented as:


    Data sources


    In addition to considering URL, oraclize provides another 4 options (which you have seen in the section on price): WolframAlpha, IPFS, randomand computation. Consider each one of them.

    1. URL


    The example already considered uses this data source. This is the source for HTTP requests to various APIs. In the example was the following:

    oraclize_query("URL", "json(https://api.coinmarketcap.com/v1/ticker/ethereum/?convert=USD).0.price_usd");

    This is the receipt of the price of the broadcast, and since api provides the json string with the data set, the request is wrapped in a json parser and returns only the field we need. In this case, this is GET, but the source URL also supports POST requests. The type of request is automatically determined by the optional argument. If there is a valid json there as in this example:

    oraclize_query("URL", "json(https://shapeshift.io/sendamount).success.deposit",
      '{"pair":"eth_btc","amount":"1","withdrawal":"1AAcCo21EUc1jbocjssSQDzLna9Vem2UN5"}')

    then the request is processed as POST (the api used is described here , if interested)

    2. WolframAlpha


    This data source allows you to access the WolframAlpha service , which can provide answers to various requests for facts or calculations, for example

    oraclize_query(“WolframAlpha”, “president of Russia”)

    will return Vladimir Putin, and the request

    oraclize_query(“WolframAlpha”, “solve x^2-4”)

    will return x = 2.
    As you can see, the result was incomplete, because the symbol ± was lost. Therefore, before using this source, you need to check that the value of a specific request can be used in a smart contract. In addition, authentication is not supported for answers, therefore oraclize themselves recommend using this source only for testing.

    3. IPFS


    As you can guess, allows you to get the contents of the file in IPFS on the multihash. The timeout for receiving content is 20 seconds.

    oraclize_query(“IPFS”, “QmTL5xNq9PPmwvM1RhxuhiYqoTJcmnaztMz6PQpGxmALkP”)

    will return Hello, Habr!(if the file with such contents is still available)

    4. random


    Random number generation works in the same way as other sources, but if used oraclize_query, it takes time-consuming preparation of arguments. To avoid this, you can use the helper function oraclize_newRandomDSQuery(delay, nbytes, customGasLimit), specifying only the execution delay (in seconds), the number of bytes generated and the gas limit for the call __callback.
    Uses randomhave a couple of features to keep in mind:

    • To confirm that the number is actually random, a special type of verification, Ledger, is used, which can be performed on the blockchain (unlike all the others, but more on that later). This means that in the constructor of a smart contract you need to set this method of verification by function:

      oraclize_setProof(proofType_Ledger);

      And at the beginning of the callback there should be the check itself:

      function__callback(bytes32 _queryId, string _result, bytes _proof)
      	{
      	    require (oraclize_randomDS_proofVerify__returnCode(_queryId, _result, _proof) == 0) );
                  <...>

      This check requires a real network and will not work on ganache, so for local testing you can temporarily remove this line. By the way, the third argument in __callbackthis is an additional parameter _proof. It is required whenever one of the confirmation types is used.
    • If you use a random number for critical moments, for example to determine the winner in the lottery, fix the user input before sending newRandomDSQuery. Otherwise, this situation may occur: oraclize calls _callback and the transaction is visible to everyone in the pending list. Along with this you can see the random number itself. If users can continue, roughly speaking, to make bets, then they will be able to indicate the price of gas more, and push through their bet before the _callback is executed, knowing in advance that it will be advantageous.


    5. computation


    It is the most flexible of the sources. It allows you to write your own scripts and use them as a data source. Calculations take place on AWS. For execution, you need to describe the Dockerfile and put it together with arbitrary additional files in a zip-archive, and upload the archive to IPFS. Execution must meet the following conditions:

    • Write the answer that you need to return, the last line in stdout
    • The answer must be no more than 2500 characters.
    • Initialization and execution should not go longer than 5 minutes in total

    For an example of how this is done, consider how to perform the simplest join of the passed strings and return the result.

    Dockerfile:

    FROM ubuntu:16.04
    MAINTAINER "info@rubyruby.ru"
    CMD echo"$ARG0$ARG1$ARG2$ARG3"

    Environment variables ARG0, ARG1etc. - These are the parameters passed with the request.
    Add dockerfile to the archive, run the ipfs server and add this archive there

    $ zip concatenation.zip Dockerfile
    $ ipfs daemon &
    $ ipfs add concatenation.zip
    QmWbnw4BBFDsh7yTXhZaTGQnPVCNY9ZDuPBoSwB9A4JNJD

    The resulting hash is used to send a request through oraclize_querya smart contract:

    oraclize_query("computation", ["QmVAS9TNKGqV49WTEWv55aMCTNyfd4qcGFFfgyz7BYHLdD", "s1", "s2", "s3", "s4"]);

    The argument is an array, in which the first element is the multi-cache archive, and all the rest are parameters that fall into the environment variables.

    If you wait for the query, then the __callbackresult will come s1 s2 s3 s4.

    Helper parsers and subqueries


    From the answer returned by any source, you can pre-select only the required information using a number of helpers, such as:

    1. JSON parser


    You saw this method in the very first example, where from the result that coinmarketcap returns, only the price was returned:

    json(https://api.coinmarketcap.com/v1/ticker/ethereum/?convert=USD).0.price_usd

    The use case is pretty obvious, going back to the example:

    [
        {
            "id": "ethereum", 
            "name": "Ethereum", 
            "symbol": "ETH", 
            "rank": "2", 
            "price_usd": "462.857", 
            "price_btc": "0.0621573", 
            "24h_volume_usd": "1993200000.0", 
            "market_cap_usd": "46656433775.0", 
            "available_supply": "100800968.0", 
            "total_supply": "100800968.0", 
            "max_supply": null, 
            "percent_change_1h": "-0.5", 
            "percent_change_24h": "-3.02", 
            "percent_change_7d": "5.93", 
            "last_updated": "1532064934"
        }
    ]

    Since this is an array, we take an element 0, and from it - a fieldprice_usd

    2. XML


    Use similar to JSON, for example:

    xml(https://informer.kovalut.ru/webmaster/getxml.php?kod=7701).Exchange_Rates.Central_Bank_RF.USD.New.Exch_Rate

    3. HTML


    You can parse XHTML with XPath. For example, get a market cap with etherscan:

    html(https://etherscan.io/).xpath(string(//*[contains(@href, '/stat/supply')]/font))

    Get MARKET CAP OF $46.148 BillionB

    4. Binary helper


    Allows you to cut pieces from raw data using the slice (offset, length) function. That is, for example, we have a file with the contents of “abc”:

    echo"abc" > example.bin

    Put it on IPFS:

    $ ipfs add example.bin
    added Qme4u9HfFqYUhH4i34ZFBKi1ZsW7z4MYHtLxScQGndhgKE

    And now we will cut out 1 character from the middle:

    binary(Qme4u9HfFqYUhH4i34ZFBKi1ZsW7z4MYHtLxScQGndhgKE).slice(1, 1)

    In the answer we get. b

    As you may have noticed, in the case of the binary helper, not the URL source was used, but IPFS. In fact, parsers can be applied to any sources, let's say it is not necessary to apply JSON to what the URL returns, you can add such content to the file:

    {
      "one":"1",
      "two":"2"
    }

    Add it to IPFS:

    $ ipfs add test.json
    added QmZinLwAq5fy4imz8ZNgupWeNFTneUqHjPiTPX9tuR7Vxp

    And then disassemble like this:

    json(QmZinLwAq5fy4imz8ZNgupWeNFTneUqHjPiTPX9tuR7Vxp).one

    We get 1

    And a particularly interesting use case is to combine any data sources and any parsers in one query. This is possible with a separate data source nested. We use the newly created file in a more complex query (the addition of values ​​in two fields):

    [WolframAlpha] add ${[IPFS] json(QmZinLwAq5fy4imz8ZNgupWeNFTneUqHjPiTPX9tuR7Vxp).one} to ${[IPFS] json(QmZinLwAq5fy4imz8ZNgupWeNFTneUqHjPiTPX9tuR7Vxp).two}

    The resulting 3
    Request is formed as follows: specify the data source nested, then for each request add the source name in front of it in square brackets, and further enclose all subqueries in ${..}.

    Testing


    Oraclize provides a useful request validation service without the need for smart contracts. Just go, select the data source, the verification method and see what will return in __callback if you send the corresponding requests.

    For local verification, in conjunction with the smart contract, you can use a special version of the Remix IDE that supports oraclize requests.

    And to check locally with ganache, you will need an ethereum bridge , which will deploy smart oraclize contracts into your testnet. To test, first add the following line to your contract constructor:

    OAR = OraclizeAddrResolverI(0x6f485C8BF6fc43eA212E93BBF8ce046C7f1cb475);

    run

    ganache-cli

    Then

    node bridge --dev

    Wait for the contracts to go through and be tested. In the output, node bridgeyou will see the sent requests and the received responses.

    Another help not only for testing, but also for real use - the ability to monitor requests here . If you request a public network, you can use the hash of the transaction in which the request is executed. If you use authentication, then keep in mind that they are guaranteed to be sent only to mainnet, for other networks it can arrive 0. If the request was on a local network, then you can use the request id, which returns oraclize_query. By the way, this id is recommended to be always saved, for example, in a similar mapping:

    mapping(bytes32=>bool) validIds;

    During the request, mark the id sent as true:

    bytes32 queryId = oraclize_query(<...>);
    validIds[queryId] = true;
    

    And then in __callbackchecking that the request with such id was not processed yet:

    function__callback(bytes32 myid, string result) {
            require(validIds[myid] != bytes32(0));
            require(msg.sender == oraclize_cbAddress());       
            validIds[myid] = bytes32(0);
            <...>

    This is necessary because __callbackone request can be called more than once due to the peculiarities of the operation of the Oraclize mechanisms.

    Authentication


    In the table with sources you could see that different sources can support different types of confirmations, and different commissions may be charged. This is a very important part of oraclize, but a detailed description of these mechanisms is a separate topic.

    The most commonly used mechanism, at least by us, is TLSNotary with storage in IPFS. Storage in IPFS is more efficient, because it is __callbacknot the proof that is returned (maybe around 4-5 kilobytes), but a much smaller multi-cache in size. To set this type, add a line in the constructor:

    oraclize_setProof(proofType_TLSNotary | proofStorage_IPFS);

    We can only say that this type, roughly speaking, protects us from the unreliability of data obtained from Oraclize. But Oraclize uses Amazon servers, which act as an auditor, so they only have to trust.

    Read more here .

    Conclusion


    Oraclize provides tools that significantly increase the number of use cases for smart contracts, as well as IPFS, which can be seen in several variations of Oracle requests. The main problem is that we again use external data that is subject to the threats from which the blockchain was supposed to protect: centralization, blocking capabilities, code changes, result substitution. But while this is all inevitable, and the option of obtaining data is very useful and viable, you just need to be aware of why the use of the blockchain was introduced into the project and whether the appeal reduces external unreliable sources to zero.

    If you are interested in some topics on the development of Ethereum not yet disclosed in these articles - write in the comments, perhaps we will reveal in the following.

    Diving into Ethereum development:
    Part 1: introduction
    Part 2: Web3.js and gas
    Part 3: user application
    Part 4: deploy and debugging in truffle, ganache, infura

    Also popular now: