March 19, 2018 at 12:17SSH in humans is not safe enough. How i fight paranoia
Under my supervision, about 1000 iron servers, VPS do not even start to count. A couple of dozen of them have very critical data. And banal ssh with keys in a standard situation is not safe enough . Not all “leather people” protect keys, let’s talk how to protect yourself from the possibility of losing a key by the user.
"Standard" hosting company. The largest number of staff in customer care and support. Only 3 lines support about 12 people have honest SSH access to servers. The set is completed by the technical vanguard of the company - a “service department” of 8 people.
An SSH authorization server is configured for the support department and partially the service department. This is a server that has a key for authorization almost anywhere. The Almighty Key is one of the most valuable information that cannot be missed. Directly read it can 4 people in the office. The backup on paper is in the safe. Also, developers who need to see firsthand where and how it broke have temporary access to this server.
An employee can obtain ssh access through an authorization server using an almighty key. Technically, these employees can call only one command - ssh, ssh, in turn, uses the private key to connect to the remote server. Typically, employees use a local script to quickly use the authorization server. Historically, it's called go. Here are its contents:
Another plus of the general authorization server is the ability to log and read statistics for all employees. We know which employee spent how much time on which server. In case of emergency, we know what the employee entered there and what answered him, since all input and output of the ssh session is copied.
The second major vulnerability is the omnipotent keys of several employees. Such keys are allowed to register on the server only with the parameter from = "". In authorized_keys, it looks like this:
The indicated almighty 4 people realize their importance and encrypt the laptop drive without forgetting about the password for the private part of the key.
Work outside the office is only possible using a VPN to the office. If there is no electricity in the office, we have a backup VPN server that can also announce our office network.
The last serious point is the large service server. The server monitors almost everything and knows about all the pieces of iron and why they are needed, except that all ansible tasks are launched on it. The server has its own private key with a password. After entering the server, magic with ssh-agent in bashrc prompts you to enter the password for the key. Then you can work at full strength. There is no direct ssh to this server, “two tributaries, three slams” and you are on the server.
These rules do not cancel the normal firewall setting on the servers. But the firewall is usually configured a little wider and starts ssh from the office network, then through sshd_config we press the list of ip which can become root:
The last bastion of verification on each server is a .bashrc file, when shell is initialized via ssh, bash starts and it checks the connection source:
This number of restriction options is used to compose the desired level of protection without blocking the employee’s work. But at the same time maintain confidence that the server is closed. As the company grows, I am less and less confident in the programs launched in the office. I trust all the employees, but I have difficulty sleeping.
Extend the functionality of the authorization server so that you can specify a list or mask of servers for each user. It is possible to add bashrc so that at the time of connection it drops information about the user and ip. Next, check the boxes if the user has a new ip or something like that. I’m not sure that this is a working scheme, but you can try it at your leisure.
Ps: Message from bfuvx how to bypass bashrc
ssh -vvv -i everebody root@149.154.64.101 We
catch something like "debug2: shell request accepted on channel 0" and send "Ctrl + c" at this moment:
Last failed login: Tue Mar 20 02:41:12 EDT 2018 from 58.242.83.24 on ssh: notty
There were 23 failed login attempts since the last successful login.
Last login: Tue Mar 20 02:40:14 2018 from 95.154.75.23
^ C-bash-4.2 #
Who are we protecting
"Standard" hosting company. The largest number of staff in customer care and support. Only 3 lines support about 12 people have honest SSH access to servers. The set is completed by the technical vanguard of the company - a “service department” of 8 people.
Bulk user
An SSH authorization server is configured for the support department and partially the service department. This is a server that has a key for authorization almost anywhere. The Almighty Key is one of the most valuable information that cannot be missed. Directly read it can 4 people in the office. The backup on paper is in the safe. Also, developers who need to see firsthand where and how it broke have temporary access to this server.
An employee can obtain ssh access through an authorization server using an almighty key. Technically, these employees can call only one command - ssh, ssh, in turn, uses the private key to connect to the remote server. Typically, employees use a local script to quickly use the authorization server. Historically, it's called go. Here are its contents:
Another plus of the general authorization server is the ability to log and read statistics for all employees. We know which employee spent how much time on which server. In case of emergency, we know what the employee entered there and what answered him, since all input and output of the ssh session is copied.
Demigods
The second major vulnerability is the omnipotent keys of several employees. Such keys are allowed to register on the server only with the parameter from = "". In authorized_keys, it looks like this:
The indicated almighty 4 people realize their importance and encrypt the laptop drive without forgetting about the password for the private part of the key.
Work outside the office is only possible using a VPN to the office. If there is no electricity in the office, we have a backup VPN server that can also announce our office network.
Almighty server
The last serious point is the large service server. The server monitors almost everything and knows about all the pieces of iron and why they are needed, except that all ansible tasks are launched on it. The server has its own private key with a password. After entering the server, magic with ssh-agent in bashrc prompts you to enter the password for the key. Then you can work at full strength. There is no direct ssh to this server, “two tributaries, three slams” and you are on the server.
These rules do not cancel the normal firewall setting on the servers. But the firewall is usually configured a little wider and starts ssh from the office network, then through sshd_config we press the list of ip which can become root:
And if all this does not work
The last bastion of verification on each server is a .bashrc file, when shell is initialized via ssh, bash starts and it checks the connection source:
This number of restriction options is used to compose the desired level of protection without blocking the employee’s work. But at the same time maintain confidence that the server is closed. As the company grows, I am less and less confident in the programs launched in the office. I trust all the employees, but I have difficulty sleeping.
Finishing plans
Extend the functionality of the authorization server so that you can specify a list or mask of servers for each user. It is possible to add bashrc so that at the time of connection it drops information about the user and ip. Next, check the boxes if the user has a new ip or something like that. I’m not sure that this is a working scheme, but you can try it at your leisure.
Ps: Message from bfuvx how to bypass bashrc
ssh -vvv -i everebody root@149.154.64.101 We
catch something like "debug2: shell request accepted on channel 0" and send "Ctrl + c" at this moment:
Last failed login: Tue Mar 20 02:41:12 EDT 2018 from 58.242.83.24 on ssh: notty
There were 23 failed login attempts since the last successful login.
Last login: Tue Mar 20 02:40:14 2018 from 95.154.75.23
^ C-bash-4.2 #