RSS feed aggregation: how I chose an aggregator
For several years I have been collecting Russian-language testing blogs. And at some point, looking at the English-language aggregators, I decided to make the same mix from my collection.
At the moment, I managed to collect 27 blogs, either entirely dedicated to software testing, or those where there is a corresponding highlighted category. An updated list can be found here: software-testing.ru/about/blogs , so if you want to add only individual blogs to your RSS reader, you can go and choose a couple of them.
And under the cut - a description of how I did it and what kind of rake I met on my way (with pictures).
I set the task for myself as follows: To make one of several RSS / Atom feeds into which all entries from different feeds will be collected and sorted by time. In addition, it should be possible to trim the length of the aggregated tape to the desired size and trim too long records, also to some specified size.
It looks easier than simple, but it wasn’t there ...
Of the several aggregators I tried, only Yahoo! could successfully cope with the task. Pipes, and even that one could not do everything that I wanted (a slightly truncated version of my Wishlist is formulated above). Perhaps some problems have already been eliminated, if so - I apologize in advance to the authors of the respective aggregators.
At first, I had a crazy idea - to share all the blogs I collected in Google Reader, I saw such a button there. However, upon closer examination, it turned out that you can only share individual records, you can’t automatically share the entire tape, so the Google chloridist left the race before he could start. Here is an excerpt from the documentation:
The feedinformer has a limitation - you can mix up to 100 feeds, but it suited me perfectly. The interface seemed convenient, so I successfully used it for several days, but a surprise awaited me - the aggregated tape was not updated. New records appeared in the source tapes - but not in the aggregated ones.
It turned out that there is a “main” method of publication, using a special widget, and if you use it, then everything is updated quite adequately. But if you publish only the RSS feed, the update does not occur. I did not need a widget, so the feedformer also flew to the basket.
This aggregator refused to work with the Cyrillic alphabet, for which it was immediately banned, without specifying its functionality.
This is generally something! Not only does he also not support the Cyrillic alphabet, he SO mixed up blog entries that I could not even guess by what principle he tried to sort them out. One word is a mixer. In the furnace without question!
This is an amazingly functional combine harvester that can do various cool things with RSS, even read aloud. But I just had to aggregate several tapes. And of course he could do it, but I couldn’t get him to make the records have both description and content. No configuration options! With great regret I had to abandon exfruits ...
This is the winner of the aggregator race. For several years it has been officially considered a beta, but this does not affect stability. Of course, I didn’t use all the functionality here either, because this is also a monster, and I had to solve a simple task.
In general, the result was such a thing:
The result of this design is an aggregated tape, which includes all the records from all the tapes. There were a lot of them there, so another one was made that grabbed the result of the first tape and cut it:
Why was this done? The fact is that Yahoo! Pipes has its troubles. Firstly, it caches bad tapes, so it works relatively slowly. No cache settings were found. Secondly, the result of his work is crookedly displayed in some RSS readers (people complained).
Therefore, it was decided to additionally pass the feed through FeedBurner. He, in turn, was squeamish and refused to accept a tape in which there were more than 500 entries. Well, that had to be cut.
The result is:
- A complete aggregated feed unloaded from Yahoo! Pipes, a little slow (due to the large size and poor caching) and slightly incompatible with some RSS readers;
- Aggregated tape trimmed to 50 entriespassed through FeedBurner to increase speed and compatibility;
- A web page containing the latest entries from the aggregated feed for those who for one reason or another do not use RSS readers (for example, for search engines), but stuffing RSS feeds in Joomla is a separate matter for discussion.
Summary: at the moment I have not been able to find a single solution that would cope even with such a simple task of information aggregation, I had to combine two solutions. However, I do not consider this a serious problem, specialization and the ability to combine services is more of a plus than a minus. Much more depressing is the fact that the vast majority of services were simply untenable, in particular when processing Russian-language information.
At the moment, I managed to collect 27 blogs, either entirely dedicated to software testing, or those where there is a corresponding highlighted category. An updated list can be found here: software-testing.ru/about/blogs , so if you want to add only individual blogs to your RSS reader, you can go and choose a couple of them.
And under the cut - a description of how I did it and what kind of rake I met on my way (with pictures).
I set the task for myself as follows: To make one of several RSS / Atom feeds into which all entries from different feeds will be collected and sorted by time. In addition, it should be possible to trim the length of the aggregated tape to the desired size and trim too long records, also to some specified size.
It looks easier than simple, but it wasn’t there ...
Of the several aggregators I tried, only Yahoo! could successfully cope with the task. Pipes, and even that one could not do everything that I wanted (a slightly truncated version of my Wishlist is formulated above). Perhaps some problems have already been eliminated, if so - I apologize in advance to the authors of the respective aggregators.
Google Reader, shared records
At first, I had a crazy idea - to share all the blogs I collected in Google Reader, I saw such a button there. However, upon closer examination, it turned out that you can only share individual records, you can’t automatically share the entire tape, so the Google chloridist left the race before he could start. Here is an excerpt from the documentation:
6. Can I customize my open page?
Unfortunately, at present you cannot set up your open page, you can only choose which posts to share.
feed.informer - feed.informer.com
The feedinformer has a limitation - you can mix up to 100 feeds, but it suited me perfectly. The interface seemed convenient, so I successfully used it for several days, but a surprise awaited me - the aggregated tape was not updated. New records appeared in the source tapes - but not in the aggregated ones. It turned out that there is a “main” method of publication, using a special widget, and if you use it, then everything is updated quite adequately. But if you publish only the RSS feed, the update does not occur. I did not need a widget, so the feedformer also flew to the basket.
Feed Rinse - feedrinse.com
This aggregator refused to work with the Cyrillic alphabet, for which it was immediately banned, without specifying its functionality.
RSS Mix - www.rssmix.com
This is generally something! Not only does he also not support the Cyrillic alphabet, he SO mixed up blog entries that I could not even guess by what principle he tried to sort them out. One word is a mixer. In the furnace without question!
XFruits - xfruits.com
This is an amazingly functional combine harvester that can do various cool things with RSS, even read aloud. But I just had to aggregate several tapes. And of course he could do it, but I couldn’t get him to make the records have both description and content. No configuration options! With great regret I had to abandon exfruits ...Yahoo! Pipes - pipes.yahoo.com
This is the winner of the aggregator race. For several years it has been officially considered a beta, but this does not affect stability. Of course, I didn’t use all the functionality here either, because this is also a monster, and I had to solve a simple task. In general, the result was such a thing:
The result of this design is an aggregated tape, which includes all the records from all the tapes. There were a lot of them there, so another one was made that grabbed the result of the first tape and cut it:
Why was this done? The fact is that Yahoo! Pipes has its troubles. Firstly, it caches bad tapes, so it works relatively slowly. No cache settings were found. Secondly, the result of his work is crookedly displayed in some RSS readers (people complained).
Therefore, it was decided to additionally pass the feed through FeedBurner. He, in turn, was squeamish and refused to accept a tape in which there were more than 500 entries. Well, that had to be cut.
The result is:
- A complete aggregated feed unloaded from Yahoo! Pipes, a little slow (due to the large size and poor caching) and slightly incompatible with some RSS readers;
- Aggregated tape trimmed to 50 entriespassed through FeedBurner to increase speed and compatibility;
- A web page containing the latest entries from the aggregated feed for those who for one reason or another do not use RSS readers (for example, for search engines), but stuffing RSS feeds in Joomla is a separate matter for discussion.
Summary: at the moment I have not been able to find a single solution that would cope even with such a simple task of information aggregation, I had to combine two solutions. However, I do not consider this a serious problem, specialization and the ability to combine services is more of a plus than a minus. Much more depressing is the fact that the vast majority of services were simply untenable, in particular when processing Russian-language information.