June 15, 2017 at 20:15

Interplanetary File System - No more copying to the network

    Everyone has a good idea for IPFS, but there was only one drawback with it. Data uploaded to the network was copied to the block storage, doubling the space they occupied. Moreover, the file was cut into blocks that are not very suitable for reuse.


    An experimental option has appeared --nocopythat eliminates this drawback. In order to use it, it is necessary to fulfill several conditions.


    A new type of identifier has also appeared. We will also analyze it.


    Let me remind you: InterPlanetary File System is a new decentralized file sharing network (HTTP server, Content Delivery Network ). I began the story about it in the article "IPFS Interplanetary File System" .

    image


    --nocopy


    This option forces IPFS to use source files as the source of blocks. Thus, files are not copied to the block storage and do not take up 2 times more space.


    To use this option, perform the following steps:


    1. need to enable filestore


      ipfs config --json Experimental.FilestoreEnabled true

    2. in the directory where the ".ipfs" directory lies (it is usually in the user directory) you need to make a link to the directory or file that you need to upload to the network


      The file can be linked with hardlink:


      fsutil hardlink "[каталог где .ipfs]\[имя файла]" "[путь к файлу]\[имя файла]"

      or


      mklink /h "[каталог где .ipfs]\[имя файла]" "[путь к файлу]\[имя файла]"

      The directory can be linked with a symbolic link:


      linkd "[каталог где .ipfs]\[имя каталога]" "[путь к каталогу]\[имя каталога]"

      or


      mklink /j "[каталог где .ipfs]\[имя каталога]" "[путь к каталогу]\[имя каталога]"

    3. And now we add


      ipfs add -r -w --nocopy "[каталог где .ipfs]\[имя каталога или файла]"

      The key -wwraps the target in the directory thereby preserving its name.


      The result will be the identifiers CIDv1 and CIDv0 (multicash)



    Webseed


    Now you can add WebSeed to the magnet or torrent .


    For file:


    http://127.0.0.1:8080/ipfs/[идентификатор]/[имя файла]
http://gateway.ipfs.io/ipfs/[идентификатор]/[имя файла]

    For the catalog:


    http://127.0.0.1:8080/ipfs/[идентификатор]/
http://gateway.ipfs.io/ipfs/[идентификатор]/

    WebSeed Magnet Example 


    magnet:?xt=urn:btih:953edbe75de612bc966194d2ee60099b3bc1a1aa&dn=Magnet-ссылка.txt&ws=http://gateway.ipfs.io/ipfs/QmfQCxNW9r2974xR5dXopXfQqsEvgexhza6aQgqTGL7Yh3/Magnet-ссылка.txt

    Content Identifier (CID)


    Due to these changes in IPFS, RAW blocks appeared. The key --nocopyautomatically enables the use of RAW blocks. But you can enable this mode with the key --raw-leaves. In this regard, a new CID (Content IDentifier) ​​or in Russian "content identifier" has appeared.


    Old id


    It is called CIDv0 and usually has the constant prefix "Qm".


    CIDv0 is just a multi-hash in Base58


    [varint ID хеша][varint длинна хеша][хеш]

    Example CIDv0: QmPbs8syAxac39bcNuMLpHXnqjKUguqakCM8LN8sZVPD9R


    New id


    RAW blocks have their own CID. It can be distinguished by the constant prefix "zb2rh".


    CIDv1 contains more information. 


    [префикс основания][varint версия CID][varint тип контента][varint ID хеша][varint длинна хеша][хеш]

    Example CIDv1: zb2rhe143L6sgu2Nba4TZgFMdPidGMA6hmWhK9wLUoVGWYsR7


    Let's take it apart


    1. z- base58 Bitcoin [ base prefix ]


      base58: b2rhe143L6sgu2Nba4TZgFMdPidGMA6hmWhK9wLUoVGWYsR7
      translate to
      HEX:01 55 12 20 6D542257CBD1BE7FD0AE8914F42066BCBF1E79487EF67B959A86DBEE4670B386


    2. 01- v1 [ varint version of CID ]


    3. 55- raw binary [varint content type ]


    4. 12- sha2-256 [varint hash ID ]


    5. 20 - 32 bytes [varint hash length]


    6. 6D542257CBD1BE7FD0AE8914F42066BCBF1E79487EF67B959A86DBEE4670B386 - sha2-256 digest [hash]



    Conclusion


    The option --nocopyhelps a lot when you want to share with the world, for example , a Wikipedia dump . Or other useful but very large volumes of information.


    References


    Windows links, symbolic and not only
    ipfs command reference
    List of experimental IPFS features


    Other parts


    1. IPFS Interplanetary File System
    2. We publish the site in the IPFS interplanetary file system
    3. We host a site in the IPFS interplanetary file system for Windows
    4. No more copying to the network
    5. Switch your site to localhost (local IPFS gateway)
    6. We localize the global gateway or sites in IPFS
    Tags:

    Also popular now: