Theoretical minimum of * nix-based systems for WebDev-padawan

Remember: the power of a Jedi Knight is the power of the universe.
But remember: anger, fear - all this leads to the dark side of the Force.
As soon as you take the first step along the dark path,
you will no longer be able to turn off it ...

Good afternoon, dear galactic senate! Denis Melsky is in touch again, and today on the agenda is the determination of the theoretical minimum of knowledge of * nix systems for the young Padawan of web-mastery.

I would like to start with the fact that we all know perfectly well: at 67.4%, our favorite Internet is spinning on * nix-based servers, and in the life of an average web developer in a vacuum, that's 90%.



For fans of proofs - welcome .

That is, in our craft without knowledge of * nix-systems in any way. Let's take an excursion into the * nix world and understand what knowledge a young Padawan should have.

I propose to consider three junior degrees of knowledge of Zen control of the shaitan machine aka * nix-server on the example of the beloved ubuntu.

1st junior

Let's start with the basics - forget about the GUI, only the console, only hardcore ^ _ ^!


Several beautiful consoles in xmonad to increase motivation.

We start our adventure by getting to the console (in the case of an SSH connection, we will be there right away). By the way, if you are a windows user, the putty magic program will help you .

If you are already in Linux and you installed it, I believe that you can find the console there and how to get into it. If not, here is a manual using ubuntu as an example with the most popular DWMs. There you will also find a description of the basic console commands. Consider this list in more detail and group a little.

Let's look at the structure of the file system.

Yes, do not be alarmed, the usual C: and D: there isn’t, everything comes from the root (/).

/	The root directory containing the entire file hierarchy.
/ bin /	Basic system utilities required both in single-user mode and during normal work for all users (cat, ls, cp).
/ boot /	Boot files (including bootloader files, kernel, etc.). Often taken out in a separate section.
/ dev /	Main files of system devices (for example, physical devices: sata-hard drives / dev / sda, video cameras or TV tuners / dev / video or pseudo-devices, for example, black holes / dev / null, / dev / zero).
/ etc /	System-wide configuration files and configuration files of installed programs (the name comes from et cetera).
/ home /	Contains user home directories, which, in turn, contain personal settings and user data. Often posted on a separate section.
/ lib /	The main libraries required for running programs from / bin / and / sbin /.
/ media /	Mount points for removable media (CD-ROM, DVD-ROM, flash-drives).
/ opt /	Additional software.
/ proc /	A virtual file system that represents the state of the kernel of the operating system and running processes in the form of file directories.
/ root /	The root user home directory.
/ sbin /	The main system programs for administering and configuring the system, for example, init, iptables, ifconfig.
/ tmp /	Temporary files (see also / var / tmp).
/ usr /	Secondary hierarchy for user data; contains most user applications and utilities used in multi-user mode. It can be mounted on a read-only network and shared by several machines.
/ var /	Modifiable files: log files (log files), temporary mail files, spooler files.
/ var / cache /	Application Cache Data Packages are downloaded here before installation on the system, but here they are stored for some time.
/ var / lib /	Status Information. Permanent data changed by programs in the process of work (databases, metadata of a package manager, etc.).
/ var / log /	Various log files (log files).
/ var / www /	Apache web server directory, everything inside is broadcast by it to the Internet (default configuration)

Now we have learned what the * nix file system looks like.

Then I recommend learning how to navigate in space ( cd - www.linfo.org/cd.html ), create files ( touch - www.linfo.org/touch.html ), edit and delete too (understand the joke about sudo rm -rf / - I strongly recommend google for those who are not in the know), get to know how the console text editor works (for example, Nano) - yes, yes, do not scare newcomers to Vim and Emacs. There is also a nice version of mcedit.


Nano


MCEdit

Chip and Dale to the rescue! In any confusing situation, enter man% commandName%, and the delightful utility man in * nix systems will tell you how this or that command (program) works in bash.
If you get lost in the file system, the pwd command will help.

Now let's highlight some more features of this OS family.

* nix-systems are case-sensitive, i.e. file.txt and file.txt are different files. Both the / uploads and / uploads directories are also different directories.

Some more important differences:

• Miscellaneous Win slashes - Backslash “\”; * nix - Slash “/”

• Win line break characters - “\ r \ n - CRLF”; * nix - “\ n - CR”; mac - “\ r - LF” Related article: en.wikipedia.org/wiki/Newline

In PHP development, to eliminate these cross-platform problems, it is recommended to use PHP_EOL for a new line in the console and DIRECTORY_SEPARATOR for the correct slashes.

In the context of discussing the file system and linux features, let's look at an interesting feature - symlinks. If you explain in a simple way - these are shortcuts, as in the well-known windows, only here the shortcut can be on another server, and on the directory, and on the file. The difference from shortcuts in windows is that here shortcuts are used not only on the desktop, but in the entire file system. There is a good wiki article about en.wikipedia.org/wiki/Symbolic_link and some syntax from the debian wiki to follow : wiki.debian.org/SymLink .

Why do many developers love * nix systems? Yes, because they are standardized by the POSIX standard system, that they all have in common and helps to quietly migrate from one standardized OS to another (both for the developer and the user. The topic is disclosed here: en.wikipedia.org/wiki/POSIX . We

continue our acquaintance.

The main difference * nix-systems - their multi-user approach. The logical conclusion follows from this: if there are many users, it is necessary to distinguish between their spheres of influence. One of the main tools for this is the rights to files and directories.

Rights designations are in alphanumeric or digital format. We

see rights we can through the commands ls - l or ls - la, and change it through chmod .


I found an amazing picture that explains the essence of what is happening.

I will add that in the life of a web developer, you always need to remember about rights in linux, because there is a mundane situation: they developed under windows, deployed and suddenly (!) Nothing works. In general, there is nothing wrong with them, but keep in mind.

PS I advise you to understand this point well, since putting 777 on the entire project is also not very secure.

For users of the system, there is a standard rules mode for the files they create - umask. It depends on what rights this user’s files will be created by default.
Read here: ru.wikipedia.org/wiki/Umask .

I briefly mentioned the presence of users and groups in * nix systems, but there is still an administrative user - root.
The root-user helps you do a lot: install software, mount (https://help.ubuntu.com/community/Mount) sections, grant permissions to files and folders where your usual user is not enough, etc.

For this there is The magical sudo team. I propose to read more here: help.ubuntu.ru/wiki .

Under the root you have to be very careful. Especially on live servers. Especially deleting something through the console.

Since we remembered the live servers, they have such a property - memory runs out.

First we check that we have RAM, top / htop is suitable for this.



Let's also remember the wonderful tool - ps. She reports on working processes. It is also convenient with several tricks:

• It is convenient to use the conveyor and the less utility to scroll through the displayed information using the up and down buttons, for example, ps aux | less.
• Using the grep utility, it is convenient to search and display only the necessary processes, for example, ps aux | grep node.

To check how much free space we have on hard, there are commands: df -h and df –k.

If the problem is in RAM, we look at what we consume more than necessary, and do kill, or if these are necessary processes, we think further :).

If the hard resources run out and there is nothing to delete, archivers come to the rescue. The main archiver in the linux world is tar. Here is a small guide on the subject, which in everyday life should be enough for you: help.ubuntu.ru/wiki/tar .

It is worth adding that the console has options for working with several programs simultaneously - the GNU Screen utility: help.ubuntu.ru/wiki/screen .

In general, the first degree of initiation can be completed on this, but I will delight some who are very afraid of the console: there is a console 2-panel classic file manager - Midnight commander.



We continue with the heading “What not to do” :).

1. Do not know why it is and how it works - do not delete!
2. I saw a file or a folder that begins with a dot - especially not delete ^ _ ^!

These are dotfiles - hidden files, just by ls they are not visible, can be seen through ls –la. Often these are system files or program files (sometimes settings). And here, too, the wiki reveals the topic well: en.wikipedia.org/wiki/Hidden_file_and_hidden_directory .

2nd junior

The first junior will help us do something, but this is not enough for the daily tasks of the web developer, so let's go further to master the level that is enough for us to resolve everyday tasks.

The first thing that needs to be mentioned at this level is the aptitude package manager (we take it as an example using ubuntu, and generally debian-based systems).
With it, we can install and remove programs in the th system, I recommend that you read in more detail, as they say, on the manufacturer’s website: help.ubuntu.ru/wiki/apt .

The next daily task is to install a lamp (linux apache php mysql) server.



You won’t believe it, but after installing the server on Windows in Ubuntu, doing this is simple and pleasant, literally in a few commands: help.ubuntu.com/community/ApacheMySQLPHP

Of course, Virtual Hosts will come in handy. The hosts file is located at / etc / hosts, and the hosts must be edited under the root.

It's time to mention the basic Apache commands.

We include modules in apache, including the PHP module (if we put it by hand) - a2enmod% moduleName%.
Server restart - sudo service apache2 restart.

Back to the hosts. In apache, and in nginx, the host system is not very complicated, but, as practice shows, it is better to tell, so as not to see the huge and terrible httpd.conf / nginx.conf later.

Hosts that are configured and exist (but not the fact that they are included!) Are separate files in the / etc / apache2 / sites-available folder. And the hosts that are used and active at the moment are symlinked in the / etc / apache2 / sites-enabled folder.



In real life, it looks like this: we create a config file for the new host in sites-available, then the a2ensite% hostName% apache command creates a symlink in the sites-enabled folder, thereby activating the host. The reverse procedure is a2dissite .

When you do this with your hands or simply write to the main config file, somewhere one cat cries, well, or a dog - who is more sorry for anyone :).

Another common task is to raise https. A good manual is here:
help.ubuntu.ru/wiki/apache_%D0%B8_https .


A picture explaining the essence of why we need https.

If you are still tormented by the question why is this all security, I advise you to read a good manual on the topic of security for developers:www.owasp.org/index.php/PHP_Security_Cheat_Sheet - here is an example of PHP, but much is relevant for all Web developers.

Also, when working with lamp, try to close the use of exec (executing commands in the OS console via php) www.php.net/manual/ru/function.exec.php .

At the php level, this is a potential gap in your defense.

I will add a frequently encountered task - closing the server for unauthorized users using the web server methods, as in this screenshot.



This is very simple to do through htpasswd, here is an example: doc.norang.ca/apache-basic-auth.html .

It's time to mention the databases. In our junior race we will consider MySql. In general, a lot of books have been written on Database Administration issues and a lot of them appear with experience, but some basic things are simply necessary.

The first is that the config lives at /etc/mysql/my.cnf, to visit, as usual, under the root.

You can restart the pug with the sudo service mysql restart command .

If you did something wrong with the rights of your root or simply lost the root password from mysql, you can reset it and set a new one using the sudo dpkg-reconfigure mysql-server-5.5 (or 5.6) command , in general, substitute the version you need :).

Let's move on to the following burning issue in the life of a web developer:

The hobbit of SQL dumps back and forth.
The database mysqldump command is used to backup the database to the sql file with the following syntax:

mysqldump —opt -u [uname] -p [pass] [dbname]> [backupfile.sql]
[uname] Username
[pass] Password (Be careful between there is no space with the p parameter and password)
[dbname] Name of our database
[backupfile.sql] As we call the dump file (you can also specify the path to it if you are in the wrong folder where you want to create it)
[--opt] Additional options
Example : mysqldump -u root -p Tutorials> tut_backup.sql

• add-drop-table - adds a DROP TABLE to the dump before CREATE TABLE.
• no-data - dump only tables without content.
• add-locks - Adds LOCK TABLES and UNLOCK TABLES to the dump.

And if the database is large and the VPN connection is not the fastest, you can immediately compress our dump into the archive with the following command:

mysqldump -u [uname] -p [pass] [dbname] | gzip -9> [backupfile.sql.gz]

Now let's analyze the knurling of the base (condition: the base does not exist, knurling from scratch).
The basic syntax would be:

mysql -u [uname] -p [pass] [db_to_restore] <[backupfile.sql]

Following our example, we get something like:
mysql -u root -p Tutorials <tut_backup.sql

And if we packaged into the archive, it will be like this:

gunzip <[backupfile.sql.gz] | mysql -u [uname] -p [pass] [dbname]

If we roll the database not from scratch, but it has already been created, there is another command:

mysqlimport -u [uname] -p [pass] [dbname] [backupfile.sql]

With the archive by analogy.

I took this beautiful dump crib from here: webcheatsheet.com/sql/mysql_backup_restore.php .

The next important point in MySQL is Grants rights. "Pug" we have multi-user, if there are many users, then they will have their rights - life is cruel :). I advise you to read about them. The most common task is to open the user an input not from the localhost. It is solved as follows:

Delete the line bind-address 127.0.0.1 from the main config.
Then we execute the following commands:

~ # mysql -u root mysql -p
mysql> GRANT ALL PRIVILEGES ON database. * TO username @ »%» IDENTIFIED BY 'password' WITH GRANT OPTION;
mysql> exit;
~ # mysqladmin -u root -p flush-privileges

Here database is the database to which we assign the user username with password password, and% indicates that the user can come not only from the localhost, but from anywhere.

These commands in respect collected from here: saradmin.ru/?p=792 .

Node JS we can also install in two commands “sudo apt-get install nodejs” “sudo apt-get install npm”.
Node projects are usually easy to start, something in the spirit of node server.js
I want to share an interesting nodemon tool - it gives us much more opportunities in the field of development on nodeJS, since it monitors changes in the project files and restarts the server automatically:
nodemon .io



Further, I recommend that you familiarize yourself with the work in the console of the most popular web development VCS in the world - git and svn. There are a lot of different and good manuals for them, I think you will pick it to your taste;).



On the topic of VCS, you may need a manual for generating SSH keys for Git: help.github.com/articles/generating-ssh-keys .
I will share an interesting trick: if you need live output of changing files to the console (often logs), use the tail –f command , for example, ( tail -f /var/log/apache2/error.log ).

3rd junior

So we got to the 3rd junior! A pretty good level, after which there is already hardcore, but there’s nothing terrible either, everything is quite interesting and fun.

The experience of real lifting servers with a full stack (lamp + ftp (s) + ssh) begins according to the situation, with the fastening of CI systems, the experience of lifting hosting systems such as Virtualmin / WebMin is also interesting.
In real operation, it is not recommended to leave a clean ftp server, it is better to use SFTP (ftp over ssh) for security.


help.ubuntu.ru/wiki/webmin

An interesting experience with nginx instead of apache - an excellent manual can be found here: help.ubuntu.ru/wiki/nginx-phpfpm .
I will also supplement it with a good online tool that transforms rewrite rules from apache into nginx format: winginx.com/en/htaccess.

Even at this level, you do not need to be afraid of BASH scripting and know what sed and grep are. I recommend reading the basics here:
help.ubuntu.com/community/Beginners/BashScripting

Good level - knowledge of vim or emacs. It’s a very holistic topic, but it’s impossible not to mention.

If at times you really miss some programs from windows, or you have a specific software that you still need and can’t find an analogue (what is so terrible you need ?!), there is wine - w ine i s n ot an e mulator.


IE In Ubuntu ("works" even more fun than in the native environment).

This is really not a windows emulator, but a set of libraries to get windows programs under nixes. There is a database of which programs and even games wine supports - appdb.winehq.org .

Let's touch on the network topic, the first guest in our studio will be netstat ( net work stat istics), meet me! Tulsa will help us look at network activity statistics, open ports, our network interfaces, etc.

Basic information:
en.wikipedia.org/wiki/Netstat

Examples of use: putty.org.ru/articles/netstat-linux-examples.html .
Thanks netstat.
Our next guest is Iptables, welcome!
IPtables is a standard firewall management interface.
Basic Information: en.wikipedia.org/wiki/Iptables
Thank you, Iptables!

And in conclusion of the network theme, let's call our headliner - nmap. Welcome guest nmap !

A very famous utility in the field of network security, we could see it in dozens of films;).


nmap.org/movies

Basic Information: en.wikipedia.org/wiki/Nmap#Bibliography .
Examples of use: habrahabr.ru/post/88064 .

Thanks to nmap for such a fascinating story and a happy childhood.

I suggest switching to a slightly advanced level of MySQL tuning - PIMP MY DB.



In live projects, it is very important to keep MySQL in a battle state tuned for maximum stability and performance, otherwise we get a very unpleasant bottleneck.

DB Tuning can be divided into two parts:

Optimization of the database structure (normalization / denormalization, foreign keys, indexes, etc.).
Optimization of DB server settings.
A lot of guides and manuals have been written about optimizing the database structure, and there is no silver bullet here. We always look at a specific project and individual problems. Explain to help :).

I advise you to read:

ruhighload.com/post habrahabr.ru/post/108418
In the matter of tuning and optimizing DB settings, Percona - MySQL fork was very successful. I recommend getting to know them better.
From the basic set for tuning, they have a toolkit and wizard to configure your server.

www.percona.com/software/percona-toolkit
tools.percona.com/wizard

Also known is mysqltuner (http://mysqltuner.com/).
For testing the load on MySQL, there is an interesting sysbench tool. You can read about it here: ruhighload.com/index.php/2010/03/05/sysbench-testiruem-proizvoditelnost-mysql .

Add to our adventure some stylish, fashionable and youthful technologies - CI.



Wiki: Continuous Integration (Eng. Continuous Integration) - the practice of software development, which consists in the implementation of frequent automated assemblies of the project to quickly identify and solve integration problems.

In practice, this is a very convenient software that allows you to collect builds, run all kinds of tests, do js / css minification, monitor the quality of the code, deploy, etc.

The most popular are Jenkins, Travis, TeamCity.

en.wikipedia.org/wiki/Jenkins_ (software)
en.wikipedia.org/wiki/Travis_CI
en.wikipedia.org/wiki/TeamCity

P. S. Cool tool Guake is a quake-style exit console.