A set of Ruby libraries for CMS and media publishing site
The set of libraries for developing CMS media publications is practically no different from any other application. Using an example application for Tape and Vedomosti, we decided to comment on the choice of each library. Description is in discussion format for each gem.
Previous review article: Restarting Media Editions: An Overview
Written about the tape was relevant until April 2014.
foxweb : There should be an explanation why not Puma.
7even : It does not exist.
zaur : This is my conservative approach that does not give guys freedom to unleash the potential of the Puma web server. I prefer to use the classic web server model when several processes are running. And I don’t like the model where the handlers competitively process requests in different threads in the classical interpreter. The lack of confidence in the thread safety of the MRI interpreter and the reluctance of everyone to switch to jRuby or Rubinius predetermined the choice.
7even : Ready to switch to ruby.
foxweb : Well now yes.
zaur : 7even , most recently there was a different opinion.
7even: Rubinius is developing rapidly. A few months ago, he fell on the constructions of the species
gem 'newrelic_rpm' gem 'newrelic-redis'
foxweb: Our Rails application and message queue at Redis worked under the constant monitoring of the NewRelic service. If finances allow, you should definitely use this tool on all critical projects. NewRelic saves hours and workdays, a quiet sleep of developers, and rescues some from dismissal and even from physical injuries. Before someone from the editorial staff came running shouting “Nothing works!”, We already knew what it was, on which server, for what reason it wasn’t working, who had broken it, who would repair it and which part of the body would be amputated. We put some charts on an HTML page, which spun 24 hours a day on a large monitor on the wall. Everyone always knew “what’s with the site.” The jambs of one developer are seen by the whole team. NewRelic minimizes the reaction time “dropped - raised”. For online media, this is especially critical.
zaur : When setting up the module, you must remember to exclude the test and dev environments. It is done in configuration files.
gem 'rails', '~> 3.2'
foxweb : This is Rails.
zaur : With that, the old rails. They did not switch to new ones, everyone was afraid that “everything will break”.
foxweb : To be more precise, most gems at that time were not adapted for Rails 4.
foxweb: Faye is a simple implementation of WebSockets, pub / sub, focused on exchanging messages between different users sitting on the same site. Initially, this is such a convenient chat on web sockets. In our project, Faye sent / received messages between editors in the admin area. Someone saved the new text, someone uploaded 10 pictures - received messages. After some time, the server reports that the pictures were generated and published - everyone who is involved in this particular article received messages. All this was very convenient for the editors, increased the level of interaction and accelerated the work of the editors as a whole. There was no need to write on Skype “guys, nobody touch this news, I’m saving it now!” Or “go to the gallery, I created it!” It was always clear which editors are currently editing this particular article. In combination with the journaling and versioning system, as well as the diff comparison service, the editorial team work on the same materials has become really convenient. The only thing we did not have time to implement is the separation by roles and access levels in the notification system. It would be interesting to make sure, for example, the editor-in-chief could see everything on a special page, without having to “shine” anywhere, and the trainees would receive messages of a certain kind. Although, a lot of work was done so that faye could not be used from the side. Anything can happen. For example, authorization via cookies and tokens stored in Redis was implemented to some extent. The token was tied to the account of a specific editor. Thanks to this, the editors could see the messages as if on behalf of the author - “Vasily Petrov saved the document”.
ksavelyev : Well, actually not very simple, Faye is good for everyone, but it does not scale.
foxweb : A plugin for faye that allows you to use Redis as an intermediate storage for pub / sub messages, as the name implies. We used it only for authorization and storage of tokens (it seems).
zaur : ... and for storing a queue of background processes, such as Resque or Cloudy (a self-made craft for working with video).
foxweb : Thin is a server for Faye.
zaur : Not your favorite Puma.
7even : And here zaur does not confuse the asynchronous server :)
ksavelyev : Zaur had no choice, otherwise Faye does not work;)
foxweb : Nice move from MySQL to Postgres. They made a separate branch in git, everything was fine-tuned there, tested and stained. First, you dumped the structure into an SQL file. Naturally, it turned out to be unsuitable for import directly into Postgres. I had to replace, for example, quotation marks throughout the file, some things were corrected manually. I think 7even will write a great story about why MySQL in such projects is bad.
7even : There is no hstore in MySQL :) Actually, seriously, Postgres in the Ruby community is much more popular and, as a result, it is easier to google a solution to any problem. Plus, custom data types (arrays, json and the same hstore) make life easier. Psql (console client) is much more convenient (if at least a little to master it).
zaur : And here is ksavelyev considers that it was necessary to pass completely to MongoDB.
gem 'postgres_ext' gem 'activerecord-postgres-json', github: 'michaelbn/activerecord-postgres-json' gem 'activerecord-postgres-hstore'
zaur : When switching from MySQL to PostgreSQL, the rails were the third version. Used these libraries to support PostgreSQL extensions.
ksavelyev : Adds the Marionette library to Asset Pipelite. Marionette adds another level of abstraction to Bacbon, introduces the concept of a module, an application, and makes it convenient to work with visual representations of collections. Makes from Bacon almost that Angular.
zaur : from Angular disowned to the last.
gem 'haml_assets' gem 'dust_assets' gem 'haml_coffee_assets'
ksavelyev : Adds support for haml / dust templates, allows you to write templates with the specified syntax and use them inside the Asset Pipeline, as a result, inside the JS application, we get a variable containing compiled and ready-to-use templates that were originally written in haml / dust.
ksavelyev : Accelerates the execution of rake assets: precompile by compiling and merging only changed assets, contains additional functions for deleting old assets.
foxweb : Gem interface to the system pop-up notification utility. During testing on Linux and OS X, a pop-up status message is displayed - successful, unsuccessful. Convenient: tests run by themselves, at this time you can switch to another window, and a pop-up notification will remind you that testing is complete.
gem 'guard' gem 'guard-rspec' gem 'guard-zeus'
foxweb : Spring was introduced in Rails 4.1 for background code preloading. In Rails 3, Zeus had to be used for the same purpose. If a developer changes a file, Zeus immediately reboots it and runs the RSpec tests related to it. Guard is known to many as a tool for tracking events related to changing project files.
guard-rspecjust runs tests when saving files.
zaur : That one, that the second brought more inconvenience. It’s easier to run the required test directly with your hands. And so solid excuses for ruby's poor performance.
7even : It's not about “low ruby performance”, but about the slow start of the rail application. Naturally, if it started in a couple of seconds, these crutches would not be needed.
foxweb: 470 specs in 14 seconds - it's fast enough.
gem 'rspec-rails', '~> 2.14'
foxweb : It seems that we used TestUnit before, but then from the 7even feed pretty quickly switched to RSpec. This has taken the testing process to a new level. We had a full TDD. Then we stopped writing tests, because almost no new code appeared, and some code was very difficult or inexpedient (in time) to cover with tests.
zaur : Controversial decision, won the majority. And this new level is called “nafig tests”.
foxweb : When creating new entities, it was necessary to test and fill with random data (name, surname, email, position, etc.). In the process of debugging, an ordinary programmer can make 1-3 entries, then he gets bored, and the code remains underdeveloped. Actually, using the classes of this gem, you can generate very diverse and, most importantly, plausible information. 7even contributed to it by adding a generator of Russian names and surnames. Thus, it was possible to debug performance on different data sets before deploying to production. The use of this gem is not associated with tests. When writing tests, for example, standard rail fixtures were used.
foxweb : Silent Assets?
ksavelyev : Exactly, silent. When the rail application is launched locally on the developer's machine in dev mode, the web server gives the static itself. In production, Nginx does this, which maintains separate logs for assets. In development, information that the web server gave the bundle of assets is usually not needed by anyone, this gem blocks garbage messages that static files were given to the browser. If necessary, if there is no such gem, you can always filter the logs using grep, for example, the command
tail -f development.log | grep SELECTwill monitor the log for the appearance of SQL queries in them.
foxweb : When outputting data to the console (maybe import logs or some sort of rake task), colored lines could be displayed in the console using colored methods, which increased the general readability of the logs.
foxweb : At some point, the number of entities and relationships between them increased so much that it became difficult to keep them in mind and transfer knowledge to new employees. Using this gem, it was possible to generate diagrams of models with links and a bunch of different options in a couple of actions. The diagram was then printed on a large sheet of A3 and thought a lot.
zaur : Then they threw it and painted on the board.
foxweb : Someone was experimenting with parsing Ruby. Perhaps for a proprietary file annotation engine.
7even : Not me.
foxweb : With Capistrano, everything was fine until Mina appeared. Capistrano worked slowly, but was able to do a multi-server deployment. Mina is fast and her output is much more readable. We did not manage to transfer CMS and the frontend to Mina, but special projects and, for example, Dom.lenta.ru were deployed initially with the help of Mina. The multiserver deployment was quite simply “emulated” using a simple cycle through the list of hosts (it seems that there were 4 of them at most). That is, just 4 deployments to 4 hosts in one run.
zaur : Now neither one nor the other is used for deployment. We switched to a control circuit using ansible. Deployment without interruption in the service provides HAProxy.
7even : It became 5 times slower than mines, but zero-downtime.
foxweb: I'll tell you a secret, it is not even zero-downtime in the current setting;)
gem 'better_errors' gem 'binding_of_caller'
ksavelyev : better_errors significantly improves the standard information screen of the 500th error. A call stack browser and an interactive ruby console appear, allowing you to view the status of objects.
foxweb : I don’t understand how we used to debug rail applications without a console in the browser.
7even : Tests must be written, and not in the browser, manually poked links.
ksavelyev : If frontendders wrote tests in the backend, and backends would make shadows and integrate fonts into the page, the world would definitely be better.
foxweb : I meant just those cases when it is impossible to catch layout errors in a normal console, for example.
7even : A set of pry, awesome_print, and several pry buns.
foxweb : Adds beautiful coloring, syntax highlighting and output formatting to the console in a standard rail console. Left in production, because often I had to climb into the console on working servers.
7even : To annotate the models, they initially used annotate_models, but at some point it started to break down on new data types in PostgreSQL - and we decided to write our own implementation.
foxweb : you decided, you wrote.
7even : Github .
gem 'rabl' gem 'jbuilder' gem 'gon'
zaur : For serialization, we started using Rabl. Awfully slow decision. Gradually began to switch to JBuilder. And for transferring JSON to the page at the stage of HTML generation, they used Gon.
foxweb : From the same category as Ffaker, but can use data from CSV files as a source.
7even : The fastest JSON parser.
foxweb : Peter is better than Moscow. What's better? Than Moscow.
foxweb : At first it was Haml, but then it turned out that Slim is not only easier to read and write, but also faster to render. To replace Haml with Slim throughout the project, you can use, for example, haml2slim .
foxweb : Well-known gem for Russification of dates and standard rail messages.
gem 'whenever', require: false
foxweb : Cronifix - a library for managing crown crowns in Ruby. Something happened there once a day with the news.
foxweb : The gem for working with Markdown, but no one remembers where we used it.
foxweb : Paginator - breaks large lists into pages (1,2,3 ... 100).
ksavelyev : Mother sloth makes adding gems from one function to the gemfile. In this case, it was lazy for us to correctly bite off a piece of HTML with a regular expression.
foxweb : I didn't know about that.
gem 'devise' gem 'devise-encryptable'
zaur : Classics of the genre for creating authorization on the site.
foxweb : Gem for ImageMagick.
foxweb : Gem for determining the type of content being downloaded.
foxweb : A simple process manager for the developer. When you are working on a large project, you need to run many small services. Foreman launches everything you need with one command. In our example, these are Rails, Faye, tests, publishing and image processing queue services.
7even : Running tests through Foreman made their output so unreadable that I soon sawed them out of Procfile.
foxweb : I remembered, it seems that everything came to the general output along with the tests, including Faye messages and application logs.
7even : Faraday is an indispensable library for accessing external network resources. Thanks to the middleware system (similar to Rack), you can set all transformations of both the request and response once — for example, add the OAuth authorization header to the request before sending it, or parse the response JSON and wrap it in Hashie :: Mash.
foxweb : “External network resources” - banal HTTP requests to the old version of the site.
7even : Okay, to any network resources. In this case, the old version of the site, but we used Faraday in a variety of situations - there were both API and HTML-ki.
7even : There are many useful middleware for faraday in this gem.
foxweb : Is this, for example, JSON?
7even : And, for example, mashify .
7even : Hashie - A collection of hash extensions. We used Hashie :: Mash - a hash variation in which the access to the elements is not like accessing the key, but as a call to the method of the same name (for example, not
user.name). Unfortunately, in the future this gem was abused and used as a replacement for models.
foxweb : Layout designers liked the new format and called it “dot notation”.
7even : So I’m talking about this.
foxweb : Well-known XML / HTML parsing library. Actively used in parsing old archives. Using it, during the import process, we replaced outdated links, image tags, cleaned out irrelevant code, garbage, and much more. It seems that they even closed unclosed tags.
gem 'eventmachine' gem 'em-synchrony' gem 'em-hiredis'
7even : For processing the video, a third-party service E **** ******* m was used, the interaction with which was through their REST API. A daemon worked on our side, which, through the queue in the radish, got the path to the video file loaded by the editor, sent it to the E **** server, transferred it to another Redis queue, polled the server every 5 minutes - and when the processing was on the E *** side * was completed, the daemon pumped out the finished files. Since all these processes had to go in parallel, the daemon was written in EventMachine.
zaur : Not a mat is painted over with asterisks.
7even : This wrapper around ffmpeg allowed you to get the width, height, bitrate, duration and other metadata of the video file.
foxweb : This data was then used to display video content on the pages of the site.
foxweb : Simple implementation of the progress bar in the console. Used when importing data from the old database into a new format. When you are waiting for processing more than 500,000 materials, the progress bar helps you plan your time and meditate in the process. The class is pretty simple. At the input, parameters such as the final number of the cycle, all kinds of view settings, etc. are set. When withdrawing, he himself calculates the interest, time, twists the counter, moves the bar, in general, does everything that a normal progress bar in any OS does. To inform the editor-in-chief that it takes three days to import old data is priceless.
foxweb : A gimmick gadget that increments the version number in the VERSION file on request and puts the corresponding tag on the commits in Git. Played and forgot.
zaur : One of the bicycles for distributing settings among different environments, of which there were more than three. Design error, reckoning.
7even : Has sunk into oblivion.
7even : Both configatron and rails_config were used to store application settings at different times. But somewhere the gem did not work outside the Rails context - which deprived us of the ability to access the settings from Faye and the daemon on the event machine - but somewhere else there were other problems; therefore, as a result, its own implementation was written. It is currently available on Github .
7even : Everything is simple: monitoring services.
foxweb : Often it was necessary to restart the crashed or hung services directly on the production. Then we got tired and we put Bluepill. Like Foreman, only for the server.
zaur : Awful monitoring of services. But others were no better. It was necessary to be able to follow the process that could change the PID. At the same time, not many could do this. Changed the web server its process at the time zero-downtime restart. For example, during the deployment, for the smooth operation of the application, they used such wonderful Unicorn functionality.
foxweb : Awesomely useful editorial tool. Allows you to watch the history of text editing, as in Wikipedia. Naturally, in order to have something to compare, the history of versions should be preserved. She already worked for us, and to fasten such an instrument was a matter of a couple of hours. The useful effect for the editors was huge.
zaur : We had a case when the editor made a typo. The site was corrected on time, but in the RSS to the Rambler main page. The news, of course, went wrong.
ksegoh : And then the guys found out the difference between Syria and Libya.
zaur: One of the scenarios for using versioning. You can trace how the trainee writes a note. For example, he can write news paragraphs sequentially. Or he can paint the plan in several sentences, and then expand the text. That is, you can see the train of thought of a person, or identify copy-pasteur.
foxweb : Gem for extracting data from Excel spreadsheets. It came in handy during the Olympics, when editors and content providers provided data on competitions and participants in a tabular form. There was neither time nor desire to make a fuss of the web interface for working with tables.
foxweb : Also an “Olympic” gem, it provided the opportunity to pick up these very tables via SSH from file servers.
gem 'resque' gem 'resque-web' gem 'resque-pool'
zaur : Images of different sizes were generated in the background. One of the reasons was unknown problems with ImageMagick. At the same time, for some time (and I still have) partially blamed Sidekiq (manager of competitive background processes). Changed the library version, tried others, reinstalled the OS, changed the server. In general, this ghost still flies in that house. Background processes are handled by Resque, converts ImageMagick. Archives with images were also processed in the background.
foxweb : Mysticism mysticism, but there were rational explanations. Often I ran to editors who, under the guise of JPEG, uploaded, for example, PNG or used invalid characters in the file name, which, of course, does not remove the blame for ImageMagick.
zaur: And from you. "Wrong" characters and in the current admin area were a pleasure. Cyrillic is your curse. By the way, they came across HTTP headers ...
zaur : I don't know why this is here.
ksavelyev : How why? You and Seva solved the problem of code execution after the end of a successful transaction.
Vedomosti editorial admin API service
7even : Since the service was conceived exclusively as an API that renders JSON, it was decided to make the application easier and use only the necessary set of functionality that rails-api provides.
7even : After picking with rabl and jbuilder, the optimal choice for serializing objects in JSON seemed to be development from the same rails-api, allowing you to use the object approach.
7even : At some point, jazz_hands lost compatibility with new versions of Ruby (and maybe some gems), and besides, we still didn’t use all its functionality - that's why there was such a mini-gem that simply combines pry and awesome_print .
gem 'ice_nine', require: %w(ice_nine ice_nine/core_ext/object)
7even : Deep freeze objects. Ruby allows you to change the values of constants, so for real immunity you need to use
#freeze; for hashes and arrays, the usual one
#freezeonly prohibits adding and removing elements - and to prohibit changing the elements themselves, you need to recursively freeze all objects. Or connect this gem and call
7even : For application self-sufficiency and the ability to check each route without using the front-end, a small console API client was written in which highline came in handy.
7even : Support for CORS requests.
7even : Since tests use a database, database_cleaner was used to clean all tables before each test.
zaur : It also came in handy when they wrote their rake task to reset the database in dev mode. This need arises because pow keeps the application running. And, therefore, the database had active connections that did not allow you to simply recreate the database.
7even : A plugin for guard that starts generating YARD documentation from changed files.
7even : We decided to use apiary.io for documenting the API, and this gem makes it possible to generate documentation in HTML and view it locally.
group :development do gem 'puma' end
foxweb : Here is Puma, but in production it's still Unicorn.
group :production do gem 'unicorn' end
foxweb : Here is Unicorn, but in development it's still Puma.
Frontend CMS Gazette
source 'https://rubygems.org' source 'https://rails-assets.org'
ksavelyev : A line with the source rails-assets.org began to appear more and more often in projects with a rich front-end. The point of adding another source is that rails-assets can make gems from almost any github repository. In order for the repository to transform into heme, it must be a valid bower repository. By specifying an additional source, we can add type calls to the gem file
gem 'rails-assets-BOWER_PACKAGE_NAME'. All JS / CSS resources of this gem will be available in Asset Pipeline and further they can be used in your project as if you added the necessary files with your hands.
gem 'puma' gem 'unicorn', group: :production
zaur : Well, you understand what I think about it.
ksavelyev : This is my favorite gem, how much time he saved for me, I can’t even imagine. This is such a Swiss knife for a technologist. The main task of Compass is to simplify and systematize work with various vendor prefixes in SASS / SCSS. The compass adds many useful functions that help the technologist insert custom fonts on the page, turn catalogs with icons into sprites, set complex gradients in CSS, operate on backgrounds, shadows, and tranches without thinking about how they are implemented in different browsers and make many more interesting and useful things. In general, it frees up time for creativity.
gem 'rails-assets-angular' gem 'rails-assets-angular-cookies' gem 'rails-assets-angular-animate' gem 'rails-assets-angular-sanitize' gem 'rails-assets-angular-ui-router' gem 'rails-assets-angular-promise-tracker' gem 'rails-assets-angular-loading-bar' gem 'rails-assets-restangular' gem 'rails-assets-angular-contenteditable' gem 'rails-assets-angular-ui-ace' gem 'rails-assets-angular-bootstrap-colorpicker' gem 'rails-assets-ng-sortable'
ksavelyev : Actually, these are not gems, but bower repositories that were transformed into gems thanks to rails-assets, which I wrote about above.
For you tried:
- foxweb (@foxweb) - Alexey Kurepin (note notes!)
- 7even (@ 7even) - Vsevolod Romashov
- ksavelyev (@ksavelyev) - Konstantin Savelyev
- ksegoh - Ksenia Hoggut
- zaur (@kavkaz) - Zaur Abasmirzoev