April 24, 2012 at 13:54

User authorization management system on thousands of servers

    When managing a large fleet of servers (100 or more) at some point, the question arises of simplifying the performance of routine tasks.

    One of the main requirements in such conditions is to have a full understanding of what and when happens on servers located in the zone of personal responsibility, but at least several dozen developers have access to them.

    Today we’ll talk about user authorization on Linux servers using the MySQL database and the Puppet application.

    System Requirements and Possible Solutions


    So, we were faced with the task of introducing a centralized system for controlling user access to servers.

    Of course, we already had such a system, and it worked quite acceptable, carried out the tasks assigned to it and, on the whole, met our requirements. Management was also centralized.

    But over time, new problems began to appear, which in the current conditions became problematic. And more and more often we understood: it is time to change something in this scheme. In the end, it was decided to finalize the current system to match our needs.

    As a result of the thoughts of the IT department employees, several ideas appeared:

    1. Using the LDAP directory in various variations.
    2. Creating a list of users in one place, putting this list on all servers (possible access control at the access.conf level).
    3. LDAP paired with Kerberos.
    4. Formation of packages (for example, rpm) based on some kind of storage with further installation (for example, as a package update).

    Each of the ideas had its pros and cons. For example, in the case of LDAP (or LDAP plus Kerberos), we would have to study additional services, solve the problems of optimizing data storage in the LDAP directory, install additional LDAP proxies to reduce the load on the main server (because we have really large loads). When choosing options 2 and 4, we would get the same tool, which, alas, ceased to satisfy our needs.

    Once on Monday (after a thorough rest and due to a good mood), one interesting thought came up: in our work, we use the Puppet application, including for issuing rights and authorities to users. So why not use it as a primary tool?

    Here the fun began: there was no ready-made solution, but there was a list of mandatory requirements for our system:
    1. When choosing a user, get a list of hosts available to him.
    2. Grant user sudo privileges, which may vary depending on the host.
    3. Immediate revocation of privileges and access on demand.
    4. Easy to manage (add, delete, edit).
    5. No need to create new services and services, which subsequently have to be studied, maintained and maintained.
    6. Do not bind the server to external authorization sources, as using local Unix authorization is more reliable and already tested by time.

    And as bonuses I would like to receive the following:
    • support for rsa, dsa keys in unlimited quantities;
    • change tracking;
    • templates support for the same tasks (later in the article describes why this is necessary);
    • support for "regular expressions" in the "Servers to which you have access" field;
    • group support;
    • quota support.

    As a result, it was decided to store data structures in MySQL (it seemed convenient for us to store in the database. The structure that is in it can be stored in other ways.), Generate puppet-like manifests with a “self-written” handler, and also “attach” to everything this simple web ui. Looking ahead, I want to say that the result exceeded our expectations, because important and interesting points were added at the writing stage.

    Implementation of the idea


    Immediately, we note that we have several geographically remote sites where the equipment is located; further we will designate them as “Platform 1”, “Platform 2”, etc. Please note that the data storage structure below is shown as an example only.

    Servers (nodes) in our example will have the following notation:
    • www is a web cluster that includes all nodes, for example www1, www2 ... wwwN;
    • www134 is one specific node from the web cluster group;
    • www [15-17] - these are the www15, www16, www17 nodes from the web cluster group (used to not write them in turn).

    Database structure


    1. The Users table is the main table containing the following information:
    • user login;
    • Full name of the user;
    • groups to which he belongs;
    • command interpreter (we do not mind that users use what is convenient for them);
    • the server to which the user must have access;
    • public keys
    • file system quota (there is a need to use it on some hosts);
    • identifiers of sudo templates associated with the user.

    2. The UsersWithBigUid table is similar to the previous one, but is used to establish “users for official needs” (for example, there is a geographically remote user who does not have to appear anywhere, but needs access to a specific server, node or nodes).

    3. VpnOnlyUsers table :
    • user login;
    • hosts that a user must have access to through a VPN connection.

    4. Table TmplSudoersRules with a description of each sudo rule included in a particular template.

    5. TmplSudoers table containing names and comments for sudo-templates.

    6. The SystemUsers table is almost the same as the table with ordinary users, but is used to establish users with unlimited access rights. The table contains an additional field with the platform identifier in case the user is needed only on one of the bottom.

    7. The Sudoers table contains personalized sudo rules for users.

    8. Hostaliases tableconsists of aliases of server names, which in certain cases are more convenient than names, for example, when several roles are assigned to one server. Using this functionality, we get another level of abstraction and we don’t care about the name of the node.

    Creating a new user and his web interface


    1. Fill in the general information about the user.


    2. We give out access to servers.
    Note . Fields * servers - a comma separated list of server names and (or) groups. An example of filling these fields:
    • % chars is a group, for example, “www”;
    • % chars% int is a specific node, for example, “www89”;
    • % chars [% int-% int] is the range of nodes in a particular group, for example, “www [17-29]”.


    3. Add one or more sshpub keys.


    4. Add sudo-rights to the user.


    After adding a user and specifying sudo-rights, accesses will be updated on all servers in a few minutes. The user’s establishment on servers can be accelerated if necessary.

    Let's see what we need to do to get the ACL for the VPN. Everything is more than simple:
    1. Select a user.


    2. Open the VPN tab, select "Generate VPN". We get the rule that you only need to add to RADIUS.


    Note. Of course, the data themselves do not appear. In our case, fields with a list of servers to which the user has access are processed. Based on this data, a VPN access-list is formed. A certain flexibility is also provided here, i.e. we can give access to both 1 node in the subnet and the entire subnet. No duplication, i.e. if you have access to the subnet 10.11.12.0/24, then it makes no sense to add something like “ ip: inacl # N = permit ip any host 11.11.12.18 ”. In our example, the received data must be entered into the VPN authorization configuration manually (we suggest that you independently think about how this can be done - let it be a kind of “homework”).

    A few examples (Task - Solution)


    Task 1: revoke all user access.
    Solution: clear the fields with the user’s servers, after which he will automatically be deleted on all servers (the home directory is not deleted).

    Task 2: revoke access to a specific server.
    Solution: remove this host from the general host list of the user.

    Task 3: provide access to a specific server (s).
    Solution: add the required host (s) to those allowed by the user.

    Task 4: to enable the restart of nginx on machines in a www-cluster to a group of persons.
    Solution:
    a. We start a new sudoers-template, give it an adequate name and comment.
    INSERT INTO `tmplsudoers` (`tmplname`, `comments`)
VALUES
	('suwwwrun', 'su to wwwrun user and restart nginx if needed');
    Note. This example is provided as an alternative to our web interface.

    b. Add the rules specific to this template.
    INSERT INTO `tmplsudoersrules` (`tmplid`, `hostname`, `runas`, `commands`, `nopasswd`)
VALUES
	(26, 'www[0-9]*', 'ALL', '/bin/su - wwwrun', 1);
INSERT INTO `tmplsudoersrules` (`tmplid`, `hostname`, `runas`, `commands`, `nopasswd`)
VALUES
	(26, 'www[0-9]*', 'ALL', '/etc/init.d/nginx *', 1);

    c. Now we just need to add this template to the user in the interface, after which the whole set of rules will work for him.
    Note . In addition to template rules, we can set unique rules for individual users.

    Task 5: to ensure the safe addition of sudo, since in some cases, seemingly harmless at first glance, sudo can be detrimental to security (for example, less, mount or rsync). Allow HelpDesk to give these rights to employees.
    Solution: templates are made by experienced system administrators, and HelpDesk employees only put the appropriate marks.
    Allow HelpDesk employees to assign templates, but prohibit their adjustment, as well as prohibit the issuance of unique rules to a specific user.

    Task 6: configure sudoers in such a way as not to disrupt the server (s), because the presence of incorrect syntax and / or duplication of HOSTALIASES directives can have bad consequences.
    Solution:
    a. Before adding or changing the sudoers-file for the user, the syntax is checked (in case of an error, a notification is generated, the sudoers file for the user is not changed).
    Python function example:
    VISUDO ='/usr/sbin/visudo'
def visudoCheck(filename,user):
    visudo_cmd = 'echo -ne "%s "; echo "%s" | %s -c -f -' % (user,filename,VISUDO)
    (visudo_status, visudo_output) = commands.getstatusoutput(visudo_cmd)
    if not visudo_status == 0:
       print "\n"+filename
       sys.stderr.write(visudo_output[:1024])
    return visudo_status


    b. Templates are made by experienced system administrators, and HelpDesk employees only put appropriate marks.

    c. When forming the strings “Host_Alias” the following rules are used to avoid duplication:
    • if this is not a template, but a “specific-rule”, then this mask is taken: “STRING% USERNAME %% USERID%”
    • if it is a template, then the following mask is taken: “STRING% USERNAME %% TMPLRULEID%”, where% TMPLRULEID% is the id of the rule entry.

    Other tasks are solved just as easily, we can discuss your questions in the comments.

    The principle of operation of the circuit


    1. Data input and / or change occurs through the web user interface.
    2. Cron launches a task that forms the Puppet manifest for the platform we need.
    3. If at the second stage there is a change in the rule, then a commit is made to the Git repository, which stores the data and reports all changes.
    4. On the server with Puppet, the git repository is checked, and as soon as it becomes clear that there are updates and (or) changes, fetch is performed with a new config.

    The script is launched with only one parameter - the identifier of the platform for which the rules are formed. After execution we get a similar result:
    class virtualusers {
@group { "wwwaccess": gid => 1001, ensure => present }
@user { "petek":
        	 	ensure => $hostname ? {
                                	simplehostname1 => present,
                                	/hostgroup1(\d.*)$/ => present,
                                	/ hostgroup2(\d.*)$/ => present,
                                	/ hostgroup3(\d.*)$/ => present,
                                	…
                                	…
                                	default => absent,
                    	},
        	 	comment => Petek Petkovich',
        	 	gid     => '1001',
        	 	home    => '/home/petek',
        	 	uid     => '1018',
        	 	password 	=> '*',
        	 	shell   => '/bin/bash',
 }  	
	@virtualuser_key { "petek":
        	  	group => '1001',
        	  	name => 'petek',
        	  	key =>"ssh-rsa dqwedqwedqwedqedqwedqwedqwedqwedqewdqwed=",
        	  	require => User["petek"];
	}
@exec { "petek_quota":
        	  	command => "/usr/sbin/setquota -u petek 5242880 5242880 0 0 -a /filesystem/",
        	  	path    	=> "/usr/sbin",
        	  	onlyif  	=> "/usr/bin/test `/usr/sbin/repquota -ua | /usr/bin/egrep '^petek\s*' | /usr/bin/awk {'print \$4'}` -ne \"5242880\"",
        	  	}
@file { "/etc/sudoers.d/1018_petek" :
        	ensure => present,
        	owner => 'root',
        	mode => 0440,
        	content => '';
}
if $hostname =~ /^hostgroup1\d*$/ {
        	realize (Virtualuser_key['petek',’petek2’], File['/etc/sudoers.d/1018_petek', '/etc/sudoers.d/1020_petek2',])
realize (Exec['petek_quota'])
}
}

    For example, and for readability, only a piece of the manifest is left. In truth, the readability of the manifestos is not of much interest to us, since the syntax is checked immediately after their formation. If it is in perfect order, then puppet master is able to read it, in which case we need not intervene in the process.

    Here is an example syntax check:
    PUPPET='/usr/bin/puppet'
def puppetparsercheck(filename):
    puppet_cmd = '%s parser validate %s' % (PUPPET, filename)
    (puppetparser_status, puppetparser_output) = commands.getstatusoutput(puppet_cmd)
    print str(puppet_cmd)
    if not puppetparser_status == 0:
       sys.stderr.write(puppetparser_output[:1024])
    return puppetparser_status


    Additional implementation advantages


    1. On each individual node, only those sudo rules that correspond to it apply; you can see them by typing sudo -l.
    2. To delete users, you do not need to perform separate actions, because if the user does not have access, then by default it is “ensure => absent” (in the Puppet language).

    Thus, we received a means of controlling user access to servers that meets all our requirements. It uses the tools that existed before its writing (which is wonderful in itself), and also saved us from the introduction and support of additional services.

    And the most important thing, perhaps, is that any employee of the "first line of support" (those who help with simple user requests) can handle such a system.
    Tags:

    Also popular now: