September 29, 2017 at 15:28Sonata import bundle
Until now, one of the best admin panels for Symfony is the SonataAdminBundle, and for good reason. Easy installation, configuration, many features out of the box and a large community.
The only thing missing from it is the import of files. Agree, an important function.
The network contains many import implementations for Sonata, but there are minor flaws everywhere - the ability to import only text fields, not entities, does not work with collections, it is problematic to load huge databases that can be processed for more than one hour ...
Today I want to introduce you my implementation, which I have been using successfully for quite some time, but only now my hands reached to comb it all and place it in a separate bundle.
I will not describe the entire installation and configuration process here. Moreover, unlike many implementations, it is extremely simple. You can read all of this on README.md and the github wiki:
In this article I want to describe only the interesting moments that I encountered during its creation.
For the first time, the idea of implementing this bundle came to me at a time when it was necessary to transfer a fairly large table of regions, cities, towns, squares, and just about everything to the customer’s base (~ 3 million lines). Complicated all that we did not have access to its server.
I tried several ready-made solutions, but I realized that they are designed for small volumes that can be downloaded while waiting for a response from the server.
It was necessary to implement this not through a web server, but through php-cli. Fortunately, Symfony has very good tools for working with console commands.
There is a great Application class to call :
But this method also does not work, because it works through a web server. There is only one thing left: Symfony \ Component \ Process \ Process , since it works directly with the console. We create a simple command (thanks to oxidmod for a more beautiful and correct solution):
The last line for asynchronous operation. And run it all in the background.
Agree that waiting for more than a minute, not understanding what exactly is happening, is difficult. And if this process lasts for an hour? Two?
That is why we need some kind of console command log. I usually use text files for logs, but this time, due to the amount of information, I decided to use a database.
An entity is responsible for each row: Doctrs \ SonataImportBundle \ Entity \ ImportLog .
Each entry corresponds to a line from the file and it has everything you need:
It is from these data that we will continue to monitor the download process and display the final detailed report.
Since an iterator is used to parse the file, the percentage of completion cannot be derived. We simply display the total processed number of records.
Unfortunately, I never learned how to catch FatalError. Therefore, in the case, for example,
the team will fall with FatalError.
Another exception that I encountered is an ORMException.
What is so interesting about him? A common exception when trying to process a request with invalid data.
Actually, this is exactly what it is intended for, though after throwing such an exception, EntityManager closes the connection, and responds to any attempt to query the database:
EntityManager is closed
In my bundle, such an exception is thrown in 2 cases. The first one is if entity validation is incorrectly configured (entities are validated before being added to the database)
And the second is related to the work of the bundle with fields of type choice and entity . If we essentially have a child entity (for example, a book has an author. The author is selected from the database), then when importing a book, we can specify the author either with an ID or with a name. If the field is not numeric, then the system tries to find the entity by the name field. If the entity does not have such a field (for example, the author’s name is not stored in name, but in login or in username), then we get an ORMException.
In principle, they were quite frequent, so I had to make a small hack to restart the EntityManager, so that after throwing an exception, the system could set the STATUS_ERROR file and successfully display all this in the interface:
By default, Sonata exports only simple fields (text, date, numbers). In order for it to export nested entities, they must be explicitly set in the getExportFields method. In addition, nested entities must configure the __toString () method; The representation of the entity as a string will be exported.
ImportBundle also uses this method so that the newly imported file can be loaded into the database without changes. If you recreate the file, then the table with the column-field correspondence is on the import page.
I never liked the fact that for the sake of changing a couple of lines in a bundle, you need to do (not so complicated, but not too convenient) an add-in using easy-extends .
Therefore, everything that I can, I put in configs. Even the class with which the file is parsed. So, in which case, you can always implement loading both XML and JSON and XLS.
Read more about all configuration options on the wiki.
If you have non-standard fields in the database (for example, in my case, center_point is the coordinates in the database), then you need to declare a class that will process the data from the file and bring them to the form in which they will be flood into mysql.
For example: type center_point is the coordinate (MySql type is point ). When added to the database and retrieved from the database, it is an object of the Point class . The Point object has a __toString method.
Using it, the import occurs, and in the import file we get beautiful coordinates. If we try to fill the same x, y in the database, then an ORMException awaits us. This is exactly what the mappings array is for . In this case, it just takes a service with id doctrs.form_format.point , which implements the Doctrs \ SonataImportBundle \ Service \ ImportAbstract interface , and based on the value received returns the desired type, which we can fill in the database.
Here is the code of the service itself
Service Code doctrs.form_format.city_pa
As you can see, in the mappings parameter we do not specify class names, but id services, which gives us freedom of action. For example, to convert the type city_autocomplete, I needed a container.
I used this bundle for six months (at that time it was not yet issued and I just pulled it with bitbucket). Of course, there were some non-critical errors, but after registering with packagist.org I try to fix everything so that there are no questions or slurred error messages.
There are small plans to improve this bundle, but let's see if their hands reach them.
I will be glad to any comments and remarks.
The only thing missing from it is the import of files. Agree, an important function.
The network contains many import implementations for Sonata, but there are minor flaws everywhere - the ability to import only text fields, not entities, does not work with collections, it is problematic to load huge databases that can be processed for more than one hour ...
Today I want to introduce you my implementation, which I have been using successfully for quite some time, but only now my hands reached to comb it all and place it in a separate bundle.
I will not describe the entire installation and configuration process here. Moreover, unlike many implementations, it is extremely simple. You can read all of this on README.md and the github wiki:
In this article I want to describe only the interesting moments that I encountered during its creation.
Large amounts of data
For the first time, the idea of implementing this bundle came to me at a time when it was necessary to transfer a fairly large table of regions, cities, towns, squares, and just about everything to the customer’s base (~ 3 million lines). Complicated all that we did not have access to its server.
I tried several ready-made solutions, but I realized that they are designed for small volumes that can be downloaded while waiting for a response from the server.
Decision
It was necessary to implement this not through a web server, but through php-cli. Fortunately, Symfony has very good tools for working with console commands.
There is a great Application class to call :
$application = new Application();
// ... register commands
$application->run();But this method also does not work, because it works through a web server. There is only one thing left: Symfony \ Component \ Process \ Process , since it works directly with the console. We create a simple command (thanks to oxidmod for a more beautiful and correct solution):
$command = sprintf(
'/usr/bin/php %s/console promoatlas:sonata:import %d "%s" "%s" > /dev/null 2>&1 &',
$this->get('kernel')->getRootDir(),
$fileEntity->getId(),
$this->admin->getCode(),
$fileEntity->getEncode() ? $fileEntity->getEncode() : 'utf8'
);The last line for asynchronous operation. And run it all in the background.
Reporting
Agree that waiting for more than a minute, not understanding what exactly is happening, is difficult. And if this process lasts for an hour? Two?
That is why we need some kind of console command log. I usually use text files for logs, but this time, due to the amount of information, I decided to use a database.
An entity is responsible for each row: Doctrs \ SonataImportBundle \ Entity \ ImportLog .
Each entry corresponds to a line from the file and it has everything you need:
- ts - time stamp
- status - what happened to the string,
- message - error messages
- line - line number from the file,
- foreignId - entity ID
It is from these data that we will continue to monitor the download process and display the final detailed report.
Since an iterator is used to parse the file, the percentage of completion cannot be derived. We simply display the total processed number of records.
Mistakes
Unfortunately, I never learned how to catch FatalError. Therefore, in the case, for example,
function setOwner(Owner $owner);
$owner = $em->findOwner(); // не найдено, вернет null
$entity->setOwner($owner);the team will fall with FatalError.
Another exception that I encountered is an ORMException.
What is so interesting about him? A common exception when trying to process a request with invalid data.
Actually, this is exactly what it is intended for, though after throwing such an exception, EntityManager closes the connection, and responds to any attempt to query the database:
EntityManager is closed
In my bundle, such an exception is thrown in 2 cases. The first one is if entity validation is incorrectly configured (entities are validated before being added to the database)
$validator = $this->getContainer()->get('validator');
$errors = $validator->validate($entity);And the second is related to the work of the bundle with fields of type choice and entity . If we essentially have a child entity (for example, a book has an author. The author is selected from the database), then when importing a book, we can specify the author either with an ID or with a name. If the field is not numeric, then the system tries to find the entity by the name field. If the entity does not have such a field (for example, the author’s name is not stored in name, but in login or in username), then we get an ORMException.
In principle, they were quite frequent, so I had to make a small hack to restart the EntityManager, so that after throwing an exception, the system could set the STATUS_ERROR file and successfully display all this in the interface:
if (!$this->em->isOpen()) {
$this->em = $this->em->create(
$this->em->getConnection(),
$this->em->getConfiguration()
);
}Configure Import / Export
By default, Sonata exports only simple fields (text, date, numbers). In order for it to export nested entities, they must be explicitly set in the getExportFields method. In addition, nested entities must configure the __toString () method; The representation of the entity as a string will be exported.
ImportBundle also uses this method so that the newly imported file can be loaded into the database without changes. If you recreate the file, then the table with the column-field correspondence is on the import page.
Extensibility
I never liked the fact that for the sake of changing a couple of lines in a bundle, you need to do (not so complicated, but not too convenient) an add-in using easy-extends .
Therefore, everything that I can, I put in configs. Even the class with which the file is parsed. So, in which case, you can always implement loading both XML and JSON and XLS.
doctrs_sonata_import:
mappings:
- { name: center_point, class: promaotlas.form_format.point}
- { name: city_autocomplete, class: promoatlas.form_format.city_pa}
upload_dir: %kernel.root_dir%/../web/uploads
class_loader: Doctrs\SonataImportBundle\Loaders\CsvFileLoader
encode:
default: utf8
list:
- cp1251
- utf8
- koir8Read more about all configuration options on the wiki.
Custom field types
If you have non-standard fields in the database (for example, in my case, center_point is the coordinates in the database), then you need to declare a class that will process the data from the file and bring them to the form in which they will be flood into mysql.
For example: type center_point is the coordinate (MySql type is point ). When added to the database and retrieved from the database, it is an object of the Point class . The Point object has a __toString method.
public function __toString(){
retrun $this->x . ', ' . $this->y;
}Using it, the import occurs, and in the import file we get beautiful coordinates. If we try to fill the same x, y in the database, then an ORMException awaits us. This is exactly what the mappings array is for . In this case, it just takes a service with id doctrs.form_format.point , which implements the Doctrs \ SonataImportBundle \ Service \ ImportAbstract interface , and based on the value received returns the desired type, which we can fill in the database.
Here is the code of the service itself
class Point implements ImportAbstract {
public function getFormatValue($value){
$value = explode(',', $value);
$point = new \PHPOpenGIS\MainBundle\Geometry\Point($value[0], $value[1] ?? 0);
return $point;
}
}Service Code doctrs.form_format.city_pa
class CityPa implements ImportAbstract, ContainerAwareInterface {
private $container;
public function setContainer(ContainerInterface $container = null) {
$this->container = $container;
}
public function getFormatValue($value){
/** @var ContainerInterface $container */
$container = $this->container;
$city = $container->get('promoatlas.city_autocomplete')->byName($value);
return $city;
}
}As you can see, in the mappings parameter we do not specify class names, but id services, which gives us freedom of action. For example, to convert the type city_autocomplete, I needed a container.
Conclusion
I used this bundle for six months (at that time it was not yet issued and I just pulled it with bitbucket). Of course, there were some non-critical errors, but after registering with packagist.org I try to fix everything so that there are no questions or slurred error messages.
There are small plans to improve this bundle, but let's see if their hands reach them.
I will be glad to any comments and remarks.