Unmanaged management and critical monitoring

    They don’t go to a strange monastery with their charter. At my next job, my tasks include the creation and subsequent support of workflow on the Alfresco platform, along with the maintenance of other systems. An institution with prescribed rules, established customs and procedures. There are no many familiar IT things in the infrastructure, but everything works reliably and suits everyone. As a person with intellect and the beginnings of education, I will not try to violate other people's traditions and globally change anything. Considering over 500 workplaces, some are not always nearby and the criticality of the created service for the organization, I’ll do some things in my own way. These include monitoring and orchestration.

    Most jobs are located in the building on 6 floors. There are remote jobs. The park is administered by different employees, some places belong to other organizations and may not have an IT employee at all. Global network and security in the area of ​​responsibility of specially trained people. DHCP, DNS, AD, if any, is not everywhere. ITIL / ITSM are just imported letters. Operating Systems WindowsXP, WindowsVista, Windows7. Workplaces can migrate to different parts of the building or beyond with the obligatory change of IP address. Users working on computers usually have administrator rights, therefore, they can change everything, including% COMPUTERNAME%. It is advisable to prevent the change of the latter since used for shared printers.

    Baseline and Goals

    The goals of the project in monitoring are checking for the availability of web applications, removing parameters with graphing java & postgresql & OS, notification via rss, sms and voice call in critical cases.

    The main goal of the orchestration is to exclude communication with over 500 users. Have the ability to quickly mass change parameters, install software packages, certificates and “soft policies - like GPO”.

    Taking into account the strategic priorities of the state, focus on full import substitution, tomorrow workstations may have MSVSfera or ROSA at the workflow users, respectively, configuration management and monitoring tools would be better to have open-source cross-platform, web-based management interface and zero cost of implementation & ownership.

    Dynamic DNS

    IP addresses are maintained according to the rule: 10. building. Cabin. When changing the location of the workplace, the IP address also changes. Changes manually because There is no DHCP, therefore option 82 is not applicable. Duplicates may occur in inventory numbers of system units. The only thing that is relatively constant is the MAC address. We will bind to it. Define the template for the name of the workstation as PC- {maksezabraziteli}. If there are several network connections on the computer, we will assume that Ethernet will be the first immediately after installing the operating system. We will use its address as a constant for% COMPUTERNAME%.

    To work with certificates, you must have the FQDN (Fully Qualified Domain Name), and from the conditions it follows that there is no DNS. A convenient solution in this situation is dynamic DNS, it is convenient for matching% COMPUTERNAME% with an IP address and, accordingly, determining the current location of the workplace if it becomes necessary to personally communicate with the user.

    The first of the solutions for dynamically updating DNS zones by the client was the Nsupdate utility from the Bind package for Windows, but in this case the clients had a common key and therefore could change not only their records in the zone. The solution was safer when the client sends an HTTP request, and the server makes changes according to its rules. The way to store zones in this option is obvious - this is the SQL database. By default, PowerDNS server is configured to serve requests from the database. It is used in many large projects and seems very stable. We will use it.

    How to install on Ubuntu
    Install PowerDNS:
    $ sudo apt-get install -y pdns-server pdns-backend-mysql

    Test PowerDNS:
    Verify that PowerDNS is running:
    $ sudo netstat -tap | grep pdns
    Should return :
    roto @ salt: ~ # netstat -tap | grep pdns

    tcp        0      0 *:domain                *:*                     LISTEN      891/pdns_server-ins

    Check that PowerDNS answers:
    $ sudo dig 127 .0.0.1
    Should return :
    roto @ salt: ~ # dig 127 .0.0.1
    ; <<>> DiG 9.9.5-3ubuntu0.1-Ubuntu <<>> @
    ; (1 server found)
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21529
    ;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
    ;; WARNING: recursion requested but not available
    ; EDNS: version: 0, flags:; udp: 2800
    ;.              IN  NS
    ;; Query time: 1 msec
    ;; SERVER:
    ;; WHEN: Fri Feb 13 11:29:05 KRAT 2015
    ;; MSG SIZE  rcvd: 239

    For comfortable zone management, many different interfaces to PowerDNS have been written. We approved the powerdns-webinterface
    What does it look like

    How to install on Ubuntu
    Install LAMP and the necessary module:
    $ sudo apt-get install gettext

    Download panel scripts:
    $ cd / tmp
    $ wget powerdns-webinterface.googlecode.com/files/powerdns-webinterface-1.5.3.tar.gz
    $ tar xvfz webinterface-1.5 .3.tar.gz
    $ sudo mv webinterface-1.5.3 / var / www / powerdns Allow

    writing to the working folder:
    $ sudo chown -R www-data: www-data / var / www / powerdns / tmp / templates_c Perform the

    import bases:
    $ sudo mysql -u user-name -p pdns db </ path-to / install.sql

    We will correct configs / db.php to the correct details for connecting to the pdns db database

    We go to the browser at the link http: /// powerdns with username / password admin: admin and change the password.

    Configuration Management Tools for Windows PC Automation

    Initially, inventory systems were considered for centralized software distribution. Existing proven solutions ocsinventory-ng and Mandriva Pulse have the ability to install packages in their functionality. Both include the ability to integrate with an ITIL / ITSM element such as Service Desk (in particular GLPI ) conveniently, but not intrinsically in the current situation. Changing the parameters of the Windows registry and deploy certificates is presented only as the manufacture of installation packages. A user with administrator privileges can delete these packages from the machine. Based on this, I began to consider configuration management systems.

    Puppet is considered the most common. It has many modules and user interfaces. Changes in the configuration files (manifests) for clients, the latter receive them the next time they access the host server, or through a request from the server for immediate execution using the push function.
    There are modules for working with Windows systems. Support for WindowsXP systems was discontinued after puppet-3.1.1.msi, and the current puppet-3.7.3.msi could not be started because tied to calling system functions. In older versions there are no `facts` such as` system32` and in the manifest you need to describe different scenarios depending on the operating system. This all complicates the use of the system and it was decided to abandon in favor of other tools.
    SaltStack is similar to Puppet in that it uses the push method to communicate with clients (minion). Minion does not open any ports, but itself connects to the server and waits for commands from it, thus increasing the security of the service. If Puppet has manifests for describing configurations, then SaltStack introduced the concept of `state`. Changes made on the master will be performed on all minions. It is worth noting that the execution of states is initiated from the wizard, and not by the clients themselves as it is implemented in Puppet. To maintain the current state in the event of a prolonged shutdown of minion, a parameter has been added to its configuration file:
    startup_states: 'highstate' (C: \ salt \ conf \ minion)
    States are configuration files in the YAML format with the sls extension.
    What does it look like
    sls tree

    Reading state begins with the file top.sls , unless otherwise specified. It contains a list of .sls files for specific groups.
    roto@salt:/etc/salt/files# cat top.sls
    # '*':
    # Для всех ПК с Windows
      - match: nodegroup
      - gr_inventory
      - gr_7z
      - gr_chrome
      - gr_libreoffice
      - gr_klitecodec
      - gr_essentials
    # Гарант
      - match: nodegroup
      - gr_garant
      # Консультант+
      - match: nodegroup
      - gr_consultant

    Groups are defined in the master configuration file (/ etc / salt / master) and allow you to specify them quite flexibly. Read more in the documentation.
    In the examples from the documentation, groups are defined by variables. For dynamic types of database passwords, tokens, hashes, these are pillar and static grain. In addition, variables can be used in the description of states and the salt command line.
    roto @ salt: / etc / salt / files # salt –G 'cpuarch: AMD64' test.ping

    As an example, listing the Microsoft antivirus ` status` :
    roto@salt:/etc/salt/files# cat gr_essentials.sls
    {% if grains['cpuarch'] == 'AMD64' %}
         - installed
         {% elif grains['osrelease'] == '7' %}
              - installed
           {% elif grains['osrelease'] == 'XP' %}
               - installed
    {% endif %}

    * There are no other x64 systems in the park except Windows7.

    Software packages are installed from the repository specified in the master configuration file (/ etc / salt / master). Description of the package in the init.sls file, its listing:
    roto@salt:/etc/salt/files/win/repo/w32xp_essentials# cat init.sls
        installer: 'salt://win/repo/w32xp_essentials/mseinstall.exe'
        full_name: Microsoft Security Essentials
        locale: ru_RU
        reboot: False
        install_flags: '/q /s /runwgacheck '
        uninstaller: '%ProgramFiles(x86)%\Microsoft Security Client\Setup.exe'
        uninstall_flags: ' /U /S'

    Salt master has a built-in file server and can give any files to minions. TCP Ports:
    • 4505 # for management
    • 4506 # for file transfers

    SaltStack has the SaltPad and Halite web interfaces. The first one could not be started, and with the help of the second one it is possible to view the system message log, the status of minions, and also send commands to them.
    What does it look like

    During the tests, he showed himself to be an extremely unreliable, incomplete solution, noticeably inferior to the user interfaces of other systems, in particular Puppet-Dashboard.
    What does it look like

    In order to avoid problems, I installed SHELLINABOX.
    What does it look like

    How to install on Ubuntu
    Install Shell In A Box for access via ssh via any html5 browser:
    $ sudo apt-get install openssl shellinabox Edit
    default configs:
    $ sudo vi / etc / default / shellinabox
    # TCP порт shellinboxd'а на котором отвечать
    # IP адрес или имя SSH сервера
    SHELLINABOX_ARGS="--o-beep -s /:SSH:localhost --localhost-only" 

    Restarting the service:
    $ sudo service shellinabox restart
    Checking the port:
    $ sudo netstat -nap | grep shellinabox
    Open in html 5 compatible browser https: //. We use it.

    The final installation script is as follows:
    @Echo Off
    rem /*************************************
    rem  *      Install for SaltStack        *
    rem  *-----------------------------------*
    rem  * (c) 2015 by Aleksey Ovchinnikov   *
    rem  * License: GPL                      *
    rem  * Feel free to customize on your    *
    rem  * needs as long this copyright      *
    rem  * remains intact                    *
    rem  *************************************/
    SetLocal EnableExtensions
    rem /// Проверяем, что скрипт запущен от Администратора
    AT > NUL
    	echo Please use "Run as Administrator"!
    	exit /b 
    rem /// Получаем mac первого адаптера
    %SYSTEMROOT%\System32\getmac.exe /NH /FO csv > %TEMP%\all_mac.tmp
    type %TEMP%\all_mac.tmp | findstr /r /c:"-" > %TEMP%\all_mac.txt
    del %TEMP%\all_mac.tmp
    	for /f "usebackq  delims=" %%i in (`find /n /v "" %TEMP%\all_mac.txt ^| find "[1]"`) do (
    	  set mac=%%i
    	For /f "delims=, tokens=1" %%i in ("%mac%") do ( set mac=%%i  )
    set mac=%mac:~4,17%
    	For /f "delims=^- tokens=*" %%i in ("%mac%") do ( set mac=%%i  )
    set mac=%mac:-=%
    set mac=%mac:~0,12%
    rem /// Меняем имя компьютера
    If not "%ComputerName%"=="PC-%mac%" (
    rem	wmic.exe /interactive:off ComputerSystem Where "name = '%computername%'" call rename Name='PC-%mac%'
    rem	wmic os where Primary='TRUE' reboot
    	REG ADD HKLM\SYSTEM\ControlSet001\Control\ComputerName\ComputerName /v ComputerName /t REG_SZ /d "PC-%mac%" /f
    	REG ADD HKLM\SYSTEM\CurrentControlSet\Control\ComputerName\ComputerName /v ComputerName /t REG_SZ /d "PC-%mac%" /f
    	REG ADD HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters /v "NV Hostname" /t REG_SZ /d "PC-%mac%" /f
    	REG ADD HKLM\SYSTEM\ControlSet001\Services\Tcpip\Parameters /v "NV Hostname" /t REG_SZ /d "PC-%mac%" /f
    	shutdown -t 0 -r -f	
    exit /b
    echo ...get NAME  ^> %ComputerName%
    rem /// Получаем IP компьютера
    	FOR /F "usebackq tokens=2 delims=[]" %%i IN (`ping %Computername% -n 1 -4`) DO if not "%%i"=="" Set ip=%%i
    echo ...get IP    ^> %ip%
    rem /// Получаем версию системы
    set key=HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion
    	For /F "delims=" %%a in ('reg query "%key%" /v "ProductName" ^| find /i "ProductName"') do ( set OSName=%%a )
    set OSName=%OSName:ProductName=%
    set OSName=%OSName:REG_SZ=%
    	For /F "tokens=* delims= "  %%a in ("%OSName%")  do ( set OSName=%%a )
    echo ...get OS    ^> %OSName%
    rem /// Получаем разрядность системы
    Set xOS=x64
    If "%PROCESSOR_ARCHITECTURE%"=="x86" If Not Defined PROCESSOR_ARCHITEW6432 Set xOS=x86
    echo ...get TYPE  ^> %xOS%
    rem /// Передаем для записи в DNS
    set data=%ip%:%COMPUTERNAME%:%COMPUTERNAME%_%OSName%_%xOS%
    set data=%data: =_%
    echo ...add DNS   ^> 
    	c:\saltstack\curl.exe -s http://salt.домен.ру/dns/update.php?data=%data%"
    rem /// Устанавливаем SaltStack учитывая разрядность Windows
     If %xOS%==x86 (c:\saltstack\Salt-Minion-3.1.5-win32-Setup.exe /S /master=salt.домен.ру /minion-name=%COMPUTERNAME%) Else (c:\saltstack\Salt-Minion-3.1.5-AMD64-Setup.exe /S /master=salt.домен.ру /minion-name=%COMPUTERNAME%)
    rem /// Добавляем записи для автообновления стэйтов призапуске системы
    echo.startup_states:  'highstate' >>"C:\salt\conf\minion"
    echo ...auto state    ^> Done!
    attrib +H +S c:\salt
    echo ...hide folder    ^> Done!
    cd c:\
    rmdir /s /q c:\saltstack

    Zabbix Monitoring System

    There are many monitoring systems. I’ve known Zabbix for a long time, I chose it.

    What does it look like

    How to install on Ubuntu
    Install the necessary packages:
    $ sudo apt-get install zabbix-server-mysql zabbix-frontend-php ca-certificates-java libslf4j-java jarwrapper libandroid-json-org-java liblogback-java

    in the server configuration file / etc / zabbix / zabbix_server. conf configure the database connection:
    $ sudo mcedit /etc/zabbix/zabbix_server.conf
    ### Option: DBHost
    #       Database host name.
    #       If set to localhost, socket is used for MySQL.
    #       If set to empty string, socket is used for PostgreSQL.
    # Mandatory: no
    # Default:
    # DBHost=localhost
    ### Option: DBName
    #       Database name.
    #       For SQLite3 path to database file must be provided. DBUser and DBPassword are ignored.
    #       Sample SQLite3 DBName:
    #                               DBName=/var/lib/zabbix/zabbix.sqlite3
    # Mandatory: yes
    # Default:
    # DBName=
    ### Option: DBUser
    #       Database user. Ignored for SQLite.
    # Mandatory: no
    # Default:
    # DBUser=
    ### Option: DBPassword
    #       Database password. Ignored for SQLite.
    #       Comment this line if no password is used.
    # Mandatory: no
    # Default:
    # DBPassword=

    Preparing the database:
    mysql –uuser-name –p
    mysql> create database zabbix character set utf8 collate utf8_unicode_ci;
    mysql> grant all privileges on zabbix. * to zabbix @ localhost identified by 'zabbix';
    mysql> exit;

    We load the structure and values ​​into the database:
    $ sudo cd / usr / share / zabbix-server-mysql
    $ sudo gunzip * .gz
    $ sudo mysql -uuser-name zabbix -p <schema.sql
    $ sudo mysql -uuser-name zabbix - p <images.sql
    $ sudo mysql -uuser-name zabbix -p <data.sql

    Restart zabbix:
    $ sudo service zabbix-server star t

    And check that everything is correct:
    $ tail -n 100 / var / log / zabbix- server / zabbix_server.log Copy the

    necessary file:
    $ sudo cp /usr/share/doc/zabbix-frontend-php/examples/apache.conf /etc/apache2/conf-available/zabbix.conf Configure

    php parameters for the web interface:
    $ sudo mcedit / etc / apache2 / conf- available / zabbix.conf
    php_value max_execution_time 300
    php_value memory_limit 128M
    php_value post_max_size 16M
    php_value upload_max_filesize 2M
    php_value max_input_time 300
    php_value date.timezone Asia/Krasnoyarsk

    Add the zabbix web interface configuration file to apache:
    $ sudo a2enconf zabbix.conf

    And restart the web server:
    $ sudo service apache2 reload

    Open the URL of your web server / zabbix and follow all the steps:

    At the last step, you will be asked to enter:

    Default login / password: Admin / zabbix Change password after login.

    JMX Monitoring

    Starting with version 2.0, zabbix has added native support for monitoring JMX. The interaction takes place through the JMX application management API through a daemon called the Zabbix Java gateway. It is written in Java.
    How to install on Ubuntu
    Install the necessary packages:
    $ sudo apt-get install --no-install-recommends zabbix-java-gateway

    In the server configuration file /etc/zabbix/zabbix_java_gateway.conf, configure the connections and startup options for Java-gateway:
    $ sudo mcedit / etc / zabbix /zabbix_java_gateway.conf

    # адрес нашего Java-gateway
    # порт для Java Gateway
    # количество Java Gateway процессов, которые будут обрабатывать информацию StartJavaPollers=5

    In the server configuration file /etc/zabbix/zabbix_server.conf, configure the connection to the Java-gateway:
    $ sudo mcedit /etc/zabbix/zabbix_server.conf
    ### Option: JavaGateway
    #<----->IP address (or hostname) of Zabbix Java gateway.
    #<----->Only required if Java pollers are started.
    # Mandatory: no
    # Default:
    # JavaGateway=
    ### Option: JavaGatewayPort
    #<----->Port that Zabbix Java gateway listens on.
    # Mandatory: no
    # Range: 1024-32767
    # Default:
    # JavaGatewayPort=10052
    ### Option: StartJavaPollers
    #<----->Number of pre-forked instances of Java pollers.
    # Mandatory: no
    # Range: 0-1000
    # Default:
    # StartJavaPollers=0

    Allow startup:
    $ sudo mcedit / etc / default / zabbix-server

    We start the service:
    $ sudo service zabbix-java-gateway restart

    The next configuration is done on the machine where the monitoring will be performed. Add to the application server startup script or servlet container:
    -Djava.rmi.server.hostname= \
    -Dcom.sun.management.jmxremote \
    -Dcom.sun.management.jmxremote.port=12345 \
    -Dcom.sun.management.jmxremote.authenticate=false \
    -Dcom.sun.management.jmxremote.ssl=false \

    Close access to this port to everyone except the IP addresses of the zabbix server and the administrator machine using firewall.

    From the admin machine, launch JConsole from jdk and connect to the machine where monitoring will be performed.
    What does it look like

    If the connection is successful, you need to go to the zabbix web interface and configure the parameters for monitoring. Go to Settings> Hosts, click Create host and enter the necessary values.
    What does it look like

    After a while, data will appear and you can build a graph.
    What does it look like

    PostgreSQL Monitoring

    According to the PostgreSQL wiki , a lot of monitoring solutions have been written. I settled on libzbxpgsql (Lib-Zabbix-PostgreSQL). This is an initially compiled module written in C for the zabbix agent and xml template for the server. The advantage over scripts is obviously one process and no external dependencies.
    How to install on Ubuntu
    It is proposed to compile the module yourself, the binary package is only in rpm. Take it.

    We install the necessary packages on the machine with the database that we will monitor:
    $ sudo apt-get install alien zabbix-agent

    Download the module:
    $ sudo wget downloads.sourceforge.net/project/libzbxpgsl/rpms/libzbxpgsql-0.1.1-1.el7. centos.x86_64.rpm

    Convert the package to deb :
    $ sudo alien libzbxpgsql-0.1.1-1.el7.centos.x86_64.rpm

    Install the package:
    $ sudo dpkg -i libzbxpgsql_0.1.1-2_amd64.deb

    In the agent configuration file / etc / zabbix / zabbix_agentd.conf configure server connections and options:
    $ sudo mcedit /etc/zabbix/zabbix_agentd.conf

    And restart the agent:
    $ sudo service zabbix-agent restart

    Save template_postgresql_server.xml
    Open the zabbix web interface and go to Settings> Templates, click Import configuration. Import our xml file.

    Configure the settings for this host. Add the necessary templates and Template PostgreSQL Server. In the Macros tab, enter the parameters for connecting to the database that we will monitor.
    What does it look like

    After a while, data will appear and you can build a graph.
    What does it look like

    This article is written as documentation for my current employer. Comments, comments and inaccuracies are welcome.

    Also popular now: