We control the browser using PHP and Selenium

Intro


Hello! Today I will tell you how PHP can work with Selenium .

Most often this is necessary when you are faced with the task of writing autotests for the web interface or your parser / crawler.

From Wikipedia
“Selenium is a web browser automation tool.
In most cases, it is used to test Web applications, but is not
limited to this. In particular, the Selenium WebDriver implementation for the phantomjs browser is
often used as a web grabber. ”


We will consider the following nuances:

  • Using Behat / Mink to Connect with Selenium
  • Running Selenium in docker, and remote access via VNC
  • Extend Behat functionality with Extension Feature

So let's go!

1. Cooking Behat and Mink


Behat is a php framework that was originally created for Behavioral Testing (BDD). It is the official PHP implementation of the more well-known Cucumber product , which is often used in other programming languages.

By itself, he does not know how to work with Selenium and is intended more for writing functional tests without using a browser.

Installs as a regular package:

$ composer require "behat/behat"

To teach behat to work with a browser, we need its Mink extension, as well as a bridge for working with a specific vendor (in our case, Selenium). You can find the full list of vendors on the Mink page . Given the versions, your composer.json should look something like this:

    "require":       {
        "behat/behat" : "^3.4",
        "behat/mink-extension" : "2.2",
        "behat/mink-selenium2-driver" : "^1.3"
    }

After installation, you will see the vendor / bin / behat file responsible for running the tests. If vendor / bin / behat --version showed you the installed version, then with a high degree of probability the installation was successful :)

The final phase is the configuration

Create the main behat.yml configuration file in the project root
default:
# Указываем путь автолоадеру к «контекст» классам
  autoload:
    '': '%paths.base%/src/Context'
  suites:
# объявляем тест сьюты    
    facebook_suite:
# путь(и) к файлам сценариев, написанных на Gherkin language 
      paths:
        - '%paths.base%/scenario/facebook'
      contexts:
# Закрепляем определенный «контекст» класс за сьютом. 
# API класса доступно в сценарии
        - Dossier\Context\FacebookContext:
# опционально передаем параметры в конструктор класса FacebookContext
            base_url: 'https://www.facebook.com/'
            user: 'email@gmail.com'
            pass: 'password'
    vk_suite:
      paths:
        - '%paths.base%/scenario/vk'
      contexts:
        - Dossier\Context\VkContext:
# Здесь передаем инстанс класса как зависимость
            - "@lookup"
      services:
# маппим алиас к классу сервису
          lookup: 'Dossier\Context\AccessLimit\Lookup'
  extensions:
# Объявляем список расширений, используемых behat
    Behat\MinkExtension:
      browser_name: 'chrome'
      default_session: 'selenium2'
      selenium2:
# адрес Selenium сервера. В данном случае стандартный IP докера(в вашем случае может быть localhost или удаленный сервер)
        wd_host: 'http://172.17.0.1:4444/wd/hub'
# браузер используемый по умолчанию
        browser: chrome


Script files or (* .feature files) - yml files written in the pseudo-language Gherkin , contain, in fact, a set of step-by-step instructions that your browser will execute during the execution of a particular suite. You can find out more about the syntax by clicking on the link above.

Each such “instruction” in turn matches the methods of the “context” class with the help of regular expressions specified in class annotations. Behat \ MinkExtension \ Context \ MinkContext
The names of the methods themselves do not matter , although it’s good practice to stick to a naming convention similar to CamelCase.

If you are missing the default Gherkin constructs, you can extend the functionality in the MinkContext descendant classes by correctly specifying annotations. This role is performed by “contextual” classes.

2. Installation and configuration of the environment


Those of you who have already worked with Selenium know that after starting the test, the browser will start on the machine and go through the steps specified in the .feature file.

Running Selenium in Docker is a bit trickier. Firstly, you will need Xs in the container, and secondly, you will want to see what happens inside the container.

The guys from Selenium have already taken care of everything and you don’t have to collect your container. A container with a Standalone server on board will be immediately available on port 5900, where you can knock from any VNC client (for example, from this ). Inside the container you will be greeted by the friendly Fluxbox interface with Chrome preinstalled. In my case, it looks like this:



In order to succeed, you can start the docker container, according to the instructions on the site:

$ docker run -d -p 4444:4444 -p 5900:5900 -v /dev/shm:/dev/shm selenium/standalone-chrome-debug:3.11.0-californium

An important point, without the shared / dev / shm volume, chrome will not have enough memory and it will not be able to start, so do not forget to specify it.

In my case, docker-compose is used , and the YAML file will look like this:

version: '2'
services:
  selenium:
    image: selenium/standalone-chrome-debug:3.11.0
    ports:
     - "4444:4444"
     - "5900:5900"
    volumes:
     - /dev/shm:/dev/shm
    network_mode: "host"

I want my tests to access Facebook via a VPN enabled on the host machine, so it is important to specify network_mode .

To start the container using compose, execute the following command:

$ docker-compose up

Now we try to connect via VNC to localhost: 5900 and open a browser inside the container. If you succeed and you see something similar to the screenshot above, you have passed this level.

3. From theory to practice. Automate


In the example below, I will get all Facebook users by the given last name and first name. The script will look like this:

src / scenario / facebook / facebook.feature
Feature: Facebook Parse
  In order parse fb
  @first-level
  Scenario: Find person in facebook
    Given I am on "https://facebook.com/"
    When I fill in "email" with "some@gmail.com"
    And I fill in "pass" with "somepass"
# Кастомная инструкция	
    Then I press tricky facebook login button
    Then I should see "Поиск"
# Кастомные инстукции
    Then I am searching by input params
    Then I dump users


And accordingly the Context class (constructor and namespace are omitted)

src / Context / FacebookContext.php
class FacebookContext extends MainContext
{
    /**
     * @Then /^I press tricky facebook login button$/
     */
    public function pressFacebookButton()
    {
        $this->getSession()->getPage()->find(
             'css', 
             'input[data-testid="royal_login_button"]'
        )->click();
    }
    /**
     * Собираем интересующую меня информацию. Аватар, ссылки, доп. информацию
     * @Then /^I dump users$/
     */
    public function dumpUsers()
    {
        $session = $this->getSession();
        $users = $this->getSession()->getPage()->findAll(
            'xpath', 
            $session->getSelectorsHandler()
               ->selectorToXpath('css', 'div._4p2o')
        );
        if (!$users) {
            throw new \InvalidArgumentException("The user with this name was not found");
        }
        $collection = new UserCollection('facebook_suite');
        foreach ($users as $user) {
            $img = $user->find('xpath', $session->getSelectorsHandler()
              ->selectorToXpath(
                 'xpath', 
                 $session->getSelectorsHandler()->selectorToXpath('css', 'img')
            ));
            $link = $user->find('xpath', $session->getSelectorsHandler()
              ->selectorToXpath(
                'xpath',              
                $session->getSelectorsHandler()->selectorToXpath('css','a._32mo')
            ));
            $outputInfo = new OutputUserInfo('facebook_suite');
            $outputInfo->setName($link ? $link->getText(): '')
                ->addPublicLinks($link ? $link->getAttribute('href') : '')
                ->setPhoto($img ? $img->getAttribute('src') : '');        
           $collection->append($outputInfo);
        }
        $this->saveDump($collection);
    }
    /**
     *  Получаем поисковый запрос и подставляем его в URL
     * @Then /^I am searching by input params$/
     */
    public function search()
    {
        if (!Registry::has('query')) {
            throw new \BadMethodCallException('No search query received');
        }
        $criteria = Registry::get('query');  
        $this->getSession()->visit("https://www.facebook.com/search/people/?q=" . urldecode($criteria->getQuery()));
    }
}


Often there is a need for custom methods such as FacebookContext :: pressFacebookButton, because by default all selectors in mink can only search by name | value | id | alt | title.

If you need to select according to another attribute, you will have to write your own method. The Facebook Login button has an id attribute, but changes its value periodically according to some logic of its own. Therefore, I had to re-bind to data-testid, which, for now, remains static.

Now, for all this to start, you need to make sure that Selenium is running and listening on the specified port.

After which we execute:

$ vendor/bin/behat 

Inside the container, the browser instance should start and go to the specified instructions.

4. Customization behat. Extensions


The Behat framework has an excellent extension mechanism built in through behat.yml . Note that many of the framework classes are declared final to moderate the temptation to simply inherit them.

The extension allows you to supplement the behat functionality, declare new console arguments and options, modify the behavior of other extensions, etc. It consists of a class implementing the
Behat \ Testwork \ ServiceContainer \ Extension interface (it is also specified in behat.yml) and auxiliary classes, if necessary.

I want to teach behat to take the name of the person you are looking for through the new input argument --search-by-fullname , so that later I can use this data inside the suite.

Below is the code that performs the necessary operations:

SearchExtension
use Behat\Behat\Gherkin\ServiceContainer\GherkinExtension;
use Behat\Testwork\Cli\ServiceContainer\CliExtension;
use Behat\Testwork\ServiceContainer\Extension;
use Behat\Testwork\ServiceContainer\ExtensionManager;
use Symfony\Component\Config\Definition\Builder\ArrayNodeDefinition;
use Symfony\Component\DependencyInjection\ContainerBuilder;
use Symfony\Component\DependencyInjection\Definition;
use Symfony\Component\DependencyInjection\Reference;
class SearchExtension implements Extension
{
    /**
     * Здесь можно модифицировать контейнер со всем Extensions перед выводом
     */
    public function process(ContainerBuilder $container)  { }
    /**
     * Уникальный префикс для конфигурации раширения в behat.yml
     * @return string
     */
    public function getConfigKey()
    {
        return 'search';
    }
    /**
     * Этот метод вызывается сразу после активации всех расширений, но
     * перед вызовом метода configure(). Это позволяет расширениям
     * вклиниться в конфигурацию других расширений
     * @param ExtensionManager $extensionManager
     */
    public function initialize(ExtensionManager $extensionManager){}
    /**
     * Установка дополнительной конфигурации расширения
     * @param ArrayNodeDefinition $builder
     */
    public function configure(ArrayNodeDefinition $builder){ }
    /**
     * Загружает сервисы расширения в контейнер
     * @param ContainerBuilder $container
     * @param array $config
     */
    public function load(ContainerBuilder $container, array $config)
    {
      $definition = new Definition('Dossier\BehatSearch\SearchController', array(
            new Reference(GherkinExtension::MANAGER_ID)
     ));  
     $definition->addTag(CliExtension::CONTROLLER_TAG, array('priority' => 1));    
     $container->setDefinition(
          CliExtension::CONTROLLER_TAG .' . search', 
          $definition
     );
    }
}


In the SearchExntesion :: load method, the SearchController service is thrown , which is responsible directly for declaring the parameters and accepting / processing them.

SearchController
use Behat\Testwork\Cli\Controller;
use Dossier\Registry;
use Dossier\User\Criteria\FullnameCriteria;
use Symfony\Component\Console\Command\Command as SymfonyCommand;
use Symfony\Component\Console\Exception\InvalidOptionException;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Output\OutputInterface;
class SearchController implements Controller
{
    const SEARCH_BY_FULLNAME = 'search-by-fullname';
    /**
     * Configures command to be executable by the controller.
     * @param SymfonyCommand $command
     */
    public function configure(SymfonyCommand $command)
    {
        $command->addOption( '--' . self::SEARCH_BY_FULLNAME,
            null,
            InputOption::VALUE_OPTIONAL,
            "Specify the search query based on fullname of the user. 
            Must be started from surname"
        );
    }
    /**
     * Executes controller.
     *
     * @param InputInterface $input
     * @param OutputInterface $output
     *
     * @return null|integer
     */
    public function execute(InputInterface $input, OutputInterface $output)
    {
        $reflect = new \ReflectionClass(__CLASS__);
        foreach ($reflect->getConstants() as $constName => $option) {
            if ($input->hasOption($option) && ($optValue = $input->getOption($option))) {
                $queryArgs = explode(',', $optValue);
                Registry::set('query', new FullnameCriteria(
                    $queryArgs[0], 
                    $queryArgs[1] ?? null, 
                    $queryArgs[2] ?? null)
                );
                return null;
            }
        }
        throw new \InvalidOptionException("You must specify one of the following
            options to proceed: " . implode(', ', $reflect->getConstants()));
    }
}


If everything is declared correctly, the list of available behat commands will be supplemented with a new argument --search-by-fullname :

$ vendor/bin/behat --help

[mkardakov@mkardakov-local dossier.io]$ vendor/bin/behat --help | grep search-by-fullname
      --search-by-fullname[=SEARCH-BY-FULLNAME]  Specify the search query based on fullname of the user. Must be started from surname
[mkardakov@mkardakov-local dossier.io]$ 

Having received the input data inside the SearchController, they can be passed directly to Context classes, or stored in a database, etc. In the example above, I use the Registry pattern for this. The approach is quite working, but if you know how to do it differently, please tell us in the comments.

That's all. Thanks for attention!

Also popular now: