Sphinx indexing from a remote server using PHP

Good day, dear readers!

I want to tell you about an interesting problem that has become before me in the framework of the project and, naturally, about its solution.

Initial data:
Standard LAMP set (hereinafter referred to as CC),
Yii framework (version is not important here),
remote server (hereinafter referred to as CSS), on which the Sphinx daemon is installed, searchd.
A user with root rights has been created on the user interface (but not the root itself).
The ssh2_mod module for PHP is installed on the SS.

Immediately make a reservation, in this article I will not describe the features of Sphinx, who are interested, can read the official manual sphinxsearch.com/docs/current.html .
I will limit myself only to general information.

So, Sphinx is a search daemon, in my case it works with MySQL. The main feature is that it indexes the database for certain queries (described in the sphinx config), and saves the result of the selection to its files. To keep the information up-to-date (in MySQL, adding and editing records is also possible), you need to run the sphinx indexing. Then, he will reselect and save it to himself.

Task:
Run the sphinx indexing on the CSS.
The reason for the remote start is that it is necessary to run the commands on the crown with specific parameters defined in the code. Crones are launched from SS.
Those. the server runs the crown, whose method performs indexing on the CSS.

The only solution that I found was to use ssh2_mod for apache2 (anyone interested, the installation manual for CentOS can be viewed here www.stableit.ru/2010/12/ssh2-php-centos-55-pecl.html ).

I looked at the ssh2 manual (http://www.php.net/manual/en/book.ssh2.php), found the wonderful ssh2_exec function, which accepts the current session and command as input, but as it turned out, it has a number of limitations.
For example, when I tried to execute the indexer --all --rotate command for the delta index, I received an error:

WARNING: failed to open pid_file '/var/run/sphinx/searchd.pid'.
WARNING: indices NOT rotated.

This error means that my user does not have enough rights to execute rotate (and I have a user with root privileges, sudo -s), although I directly executed this command directly from the console without any errors.
Next, I decided to search for more, and found that you can emulate command input through the terminal (ssh2_shell function). Using the standard stream and the fwrite function, you can write commands to the “terminal” and get the same standard output stream, i.e. result issued by the terminal. Occurs by line-by-line reading from the output stream using fgets.

Everything is fine, the delta index execution check was successful, I was delighted, but ...
“BUT” happened when I tried to index the main index (about 400k records, it takes several minutes). It turned out that the output stream breaks off at the slightest delay in the execution of a command in the terminal. In simple language, when you enter a command, and the terminal "thinks". As a result, I still had "unindexed" files.

I decided to google how people solve problems, came across a piece of code, right in ssh2 mana on php.net. The author of the solution suggested putting the start and end markers of the command (echo '[start]'; $ command; echo '[end]') and set max_execution_time for the script.
The code is below.

$ip = 'ip_address'; 
$user = 'username'; 
$pass = 'password'; 
$connection = ssh2_connect($ip); 
ssh2_auth_password($connection,$user,$pass); 
$shell = ssh2_shell($connection,"bash"); 
//Trick is in the start and end echos which can be executed in both *nix and windows systems. 
//Do add 'cmd /C' to the start of $cmd if on a windows system. 
$cmd = "echo '[start]';your commands here;echo '[end]'"; 
$output = user_exec($shell,$cmd); 
fclose($shell); 
function user_exec($shell,$cmd) { 
  fwrite($shell,$cmd . "\n"); 
  $output = ""; 
  $start = false; 
  $start_time = time(); 
  $max_time = 2; //time in seconds 
  while(((time()-$start_time) < $max_time)) { 
    $line = fgets($shell); 
    if(!strstr($line,$cmd)) { 
      if(preg_match('/\[start\]/',$line)) { 
        $start = true; 
      }elseif(preg_match('/\[end\]/',$line)) { 
        return $output; 
      }elseif($start){ 
        $output[] = $line; 
      } 
    } 
  } 
} 

It seemed to me a good solution, but ...
BUT here was the preg_match condition. When outputting information, everything that the terminal gives to the output is written to $ output. The above-described problem with the “thoughtful terminal” has again become relevant, because when paused, the terminal issued a command to output the completion marker echo '[end]' (the command itself, and not the result of the execution). Everything was solved by adding a limit to the beginning and end of the line in preg_match: and checking for is_string for $ line. It only remained to file a file, and, voila, a component was created in the Yii project, which is a kind of layer for ssh2 functions.
preg_match('/^\[start\]\s*$/',$line)


params['ssh'];
        $params = array('user', 'password', 'host', 'port');
        foreach($params as $param) {
            if(isset(${$param}) && !is_null(${$param})) {
                $this->{$param} = ${$param};
            } else {
                $this->{$param} = @$config[$param];
            }
        }
        return true;
    }
    /**
     * Connect to Ssh
     *
     * @return resource
     * @throws SshException
     */
    public function connect()
    {
        $this->ssh = @ssh2_connect($this->host, $this->port);
        if(empty($this->ssh)) {
            throw new SshException('Cant connect to ssh');
        }
        if(empty($this->execType)) {
            $this->execType = self::EXEC_TYPE_SHELL;
        }
        return $this->ssh;
    }
    /**
     * Login to ssh
     *
     * @throws SshException
     * @return bool
     */
    public function login()
    {
        if(!@ssh2_auth_password($this->ssh, $this->user, $this->password)) {
            throw new SshException('Cant login by ssh');
        }
        return true;
    }
    /**
     * Exec command by ssh
     *
     * @param $cmd
     * @param $type
     *
     * @return string
     * @throws SshException
     */
    public function exec($cmd, $type = self::EXEC_TYPE_SHELL)
    {
        if(is_null($this->ssh)) {
            $this->connect();
            $this->login();
        }
        $this->execType = $type;
        switch($this->execType) {
            case self::EXEC_TYPE_EXEC: $result = $this->execCommand($cmd); break;
            case self::EXEC_TYPE_SHELL: $result = $this->execByShell($cmd); break;
            default: throw new SshException('Incorrect exec type'); break;
        }
        return $result;
    }
    /**
     * Executes command by the direct ssh2_exec
     *
     * @param $command
     *
     * @return string
     * @throws SshException
     */
    private function execCommand($command)
    {
        if (!($stream = ssh2_exec($this->ssh, $command))) {
            throw new SshException('Ssh command failed');
        }
        stream_set_blocking($stream, true);
        $data = "";
        while ($buf = fread($stream, 4096)) {
            $data .= $buf;
        }
        fclose($stream);
        return $data;
    }
    /**
     * Executes command within the shell opening
     *
     * @param $command
     *
     * @return string
     */
    private function execByShell($command)
    {
        $this->openShell();
        return $this->writeShell($command);
    }
    /**
     * opens shell
     *
     * @throws SshException
     */
    private function openShell()
    {
        if(is_null($this->shell)) {
            // here is hardcoded width and height, you can change them.
            $this->shell = @ssh2_shell($this->ssh,  $this->shellType, null, 80, 40, SSH2_TERM_UNIT_CHARS);
        }
        if( !$this->shell ) {
            throw new SshException('SSH shell command failed');
        }
    }
    /**
     *
     * Write the command to the open shell
     *
     * @param $cmd
     * @param int $maxExecTime in sec
     *
     * @return string
     */
    private function writeShell($cmd, $maxExecTime = self::MAX_EXECUTION_TIME)
    {
        // write start marker
        fwrite($this->shell, $this->getMarker(self::START_MARK));
        // write command
        fwrite($this->shell, $cmd . PHP_EOL);
        // write end marker
        fwrite($this->shell, $this->getMarker(self::FINISH_MARK));
        stream_set_blocking($this->shell, true);
        sleep(1);
        $output = "";
        $start = false;
        // define the time until the script can be executed
        $timeUntil = time() + $maxExecTime;
        while(true) {
            if(time() > $timeUntil) {
                break;
            }
            $line = fgets($this->shell, 4096);
            // if any delay is happened while command is processing
            if(!is_string($line)) {
                sleep(1);
                continue;
            }
            // define the start executed command
            if(preg_match('/^' . self::START_MARK . '\s*$/', $line)) {
                $start = true;
            } elseif(preg_match('/^' . self::FINISH_MARK . '\s*$/', $line)) {  // define the last executed command
                break;
            } elseif($start) {
                // add console output to the script output data
                $output .= $line;
            }
        }
        return $output;
    }
    /**
     * Disconnect from ssh
     */
    public function disconnect() {
        $this->exec('exit');
        $this->ssh = null;
        if(!is_null($this->shell)) {
            fclose($this->shell);
        }
    }
    /**
     * Disconnect in destruct
     */
    public function __destruct() {
        $this->disconnect();
    }
    /**
     * Returns marker command
     *
     * @param string $type
     *
     * @return string
     */
    private function getMarker($type = self::START_MARK)
    {
        return 'echo "' . $type . '"' . PHP_EOL;
    }
}

P.S. This class can be expanded, because ssh2 is not limited to only two functions for executing commands, there are also functions for working with files, and other types of authorization, etc. etc.

Thank you for your attention, I hope the article will be useful.
I will be glad to hear any feedback and constructive criticism!

Posted by Vladislav Ivanenko, PHP Developer Zfort Group