Console version of Cyberduck: working with cloud storage

In previous publications, we have repeatedly talked about utilities that can be used to work with our cloud storage ( 1  and  2 ). In one article, we already mentioned the Cyberduck utility - a convenient file manager for MacOS, Linux, and Windows that works with the FTP, SFTP, WebDAV, OpenStack Swift, and AmazonS3 protocols. You can also download a profile from our official website with which Cyberduck is automatically configured to work with our cloud storage.

Most users are familiar with the graphical version of Cyberduck. More recently, the  console version appeared. In this article, we will talk about its capabilities and show how it can be used to work with storage.

general information

The console version of Cyberduck works with all major operating systems - MacOS, Windows and Linux. The program can be used as an FTP and SFTP client, as well as for working with various cloud storage services.
The documentation for the console version of Cyberduck is published on the official website, but, unfortunately, most of the important functions are not described in as much detail as we would like. Below we will talk about what operations with our repository can be performed with its help, as well as give detailed explanations of the syntax of the main commands.

Installation

Mac OS

The console version of Cyberduck for MacOS is installed using the Homebrew package manager:

$ brew install duck

Linux

In this article, we describe the installation procedure for Ubuntu 14.04; users of other distributions are referred to the  official documentation .

To install the console version of Cyberduck, first add the appropriate repository:

$ echo 'deb https://s3.amazonaws.com/repo.deb.cyberduck.io nightly main'> / etc / apt / sources.list
$ echo 'deb https://s3.amazonaws.com/repo.deb.cyberduck.io stable main'> / etc / apt / sources.list

Then add the key:

 $ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys FE7097963FEFBE72

and execute the commands

$ sudo apt-get update
$ sudo apt-get install duck

Basic Cloud Storage Operations

All commands for working with cloud storage are as follows:

$ duck - <argument> swift: // <username> @ auth.selcdn.ru / <object path> -p <password>

After entering the command, the program will ask for the name of the account owner (Tenant). In response, you need to specify the user whose account is used to access the repository. The dialog mode can be turned off altogether - the -q option is used for this.

Retrieving a list of files in a container

To get a list of files stored in a container, use the -l (or −−list) option:

$ duck -l swift: //username@auth.selcdn.ru/ <container path> -p <password>
Listing directory images ...
1.jpg
2.jpg
3.png

File download

To download a file from storage, use a command of the form:

$ duck -d swift: //username@auth.selcdn.ru/ <file path> <file name> -p <password>

Opening a file for editing on a local machine

Using the console version of CyberDuck, you can open files for editing on the local machine; upon completion of editing, the updated (with all changes) version of the file will be loaded into the repository. To do this, use the −−edit option:

$ duck --edit swift: // <username> @ auth.selcdn.ru / <file path> -p <password>

The file will be opened in the application used in the system for this type of file; Downloading a modified version starts automatically.
This feature of Cyberduck will be especially interesting for users who host static sites in our storage. To, for example, quickly edit the text on the site, it is enough to execute the above command, make changes to the necessary files and save them.

Upload object to storage

The download operation is performed using a command of the form:

$ duck --upload swift: //username@auth.selcdn.ru <full path to the object in the repository> <path to the object on the local machine> -p <password>

Pay special attention to the fact that when loading an object, you must specify the full path to the storage location of this object itself. For example, if we want to save the file myimage.png in the images container, then the path to it should be specified like this: / images / myimage.png.

Large (larger than 2GB) objects Cyberduck loads into storage in parts.

Object Versions and Backup

The console version of Cyberduck is a convenient tool for backing up and archiving data. We will consider these functions in more detail with specific practical examples.

Imagine that we have a directory on the local machine, the contents of which must be periodically copied to the cloud storage. To do this, a special script was written and the Cron task was added, which sends a backup copy to the storage every day at the specified time.
The script looks like this:

#! / bin / bash
SWIFT_USERNAME = username
SWIFT PASSWORD = password to enter the repository
SWIFT_AUTH_URL = auth.selcdn.ru
BACKUP_PATH = path to the backup storage location
LOCAL_PATH = path to the folder on the local machine
duck --upload swift: // $ SWIFT_USERNAME @ $ SWIFT_AUTHURL / $ BACKUP_PATH / $ LOCAL_PATH --existing rename --password $ SWIFT_PASSWORD -q

Pay attention to the syntax of the duck command. The above example uses the --existing switch, which indicates what to do with files already in the repository. The rename option renames an existing backup, adding time and date to its name.

With cyberduck, you can also perform differential backups. To do this, use the compare option:

$ duck --upload swift: //username@auth.selcdn.ru <full path to the object in the repository> <path to the file on the local machine> --existing compare -p <password>

When executing the above command, the program compares the downloaded backup copy with the size, date of change, and checksum already available. If the parameters differ, the old version will be deleted, and the new one will be uploaded to the repository.

When using the skip option, only new files will be downloaded to the repository (those that appeared in the folder on the local machine after the previous download). Existing files will not be downloaded, even if they were changed on the local machine.

Finally, the overwrite option simply removes the existing backup from the repository and loads a new one.

Synchronize local files with files in storage

File synchronization is a process whereby two directories, one of which is located on the local machine and the other in the repository, will contain the same set of files in the same version with the most recent change date. If on a local machine any files have been changed, added or deleted, these same files will be changed, added or deleted in the repository, and vice versa.

Synchronization is performed using the command:

$ duck --synchronize swift: /// <path to the folder in the repository> <path to the folder on the local machine>

Using the synchronization function, you can keep the backup copies of data from the local machine in the current state.
Here is an example of a simple script:

# bin / bash
SWIFT_USERNAME = username
SWIFT PASSWORD = password to enter the repository
SWIFT_AUTH_URL = auth.selcdn.ru
BACKUP_PATH = path to the backup storage location
LOCAL_PATH = path to the folder on the local machine
duck --synchronize swift: // $ SWIFT_USERNAME @ SWIFT_AUTHURL / $ BACKUP_PATH $ LOCAL_PATH --password $ SWIFT_PASSWORD -q

It is enough to add the appropriate task to cron - and the data will be automatically synchronized with the specified frequency.

The described function will be useful for those who place static sites in the repository. To update the site, just make the appropriate changes to the files on the local machine, and then run the synchronization command.

Copy files

To copy a file from one container to another, use a command of the form:

$ duck --copy swift: // / <full path to file> / <path to new storage location> -p <password>

-V option

In order to display information on the console about all HTTP requests made during operations with the storage, as well as about responses to them, the -v (or -verbose) option is used. This helps to understand how third-party applications interact with storage.

Conclusion

The console version of Cyberduck is a convenient tool for working with cloud storage, which has many features. The appearance of such a tool should please Windows users. The fact is that for Windows but recently there were no console programs for working with cloud storage based on OpenStack Swift at all, and for this purpose you had to use FTP clients, which is not always convenient.
We hope that you will appreciate the console version of Cyberduck and will use it in everyday practice.

Readers who for one reason or another cannot post comments here are welcome to join our blog .