Ratings Service / Online-service + REST API for searching movie ratings

Ratings Service is an online service that allows you to find out the rating of a movie by its name.



Distinctive features:

1. search on several Internet databases at the same time (at the moment this is KinoPoisk.Ru and KinoKopilka.ru)
2. easy interface; can be used from a mobile phone, for example, via Opera Mini
3. the ability to receive sample results in XML format in REST style
4. hosting on Google App Engine / Java
5. Open source

I would like to talk about some features of the implementation and share my impressions of working with Google App Engine / Java. Below you will also find the project address on Google Code, with the laid out source codes.

It took me three days to write the service and it was really fun. It’s fun, mainly because a lot of things do not work in GAE / Java, and I had to implement some of the things from scratch myself, for example, write a simple engine for XPath queries :)

But first things first.

Development tools

According to the Getting Started Guide of the IDE, only Eclipse is officially officially supported at the moment, for this there is a Google Eclipse Plugin. And three days ago only the version for Ganymede / 3.4 was available (I used it), today the version for Galileo / 3.5 is already available.

Creating a GAE Web Application project is no more complicated than creating a regular Java project in Eclipse. Running and debugging the application works fine on the local machine. In addition to this, deploying with one button makes development just enjoyable.

I also really liked the nice little thing that the log4j.properties file was already in the project and configured as it should :) All that remains for me is to add jar log4j to the classpath and enable debugging for my code:

log4j.category.dmitrygusev.ratings = DEBUG, A1

Of the negative aspects (I don’t know where to relate it - rather to Eclipse), deploy ( GAE Issue # 1235 ) stopped working on the third day , but it was quickly fixed.

Another negative point is that the directory structure of the project in GAE Web Application is different from the directory structure of Maven2. It is clear that you can configure the Maven2 project for Google App Engine, but I would like to see Maven2 support directly in the Google Eclipse Plugin.

Frameworks & libraries

As for the frameworks and third-party libraries, most of them that would really be useful do not work in GAE / Java, mainly because Google partially blocked access to the Reflection API, as well as other APIs that might entail security risks.

Most of this project lacked support for XML frameworks (at least JAXB) and XPath. That standard Java XML API (DOM, SAX, StAX?) Which is supported and which I used to easily easily use, now seems not very user friendly.

As a result, I decided to stop at the NanoXML Lite library, which showed itself well in the mobile client project for the www.4konverta.com online service .

All NanoXML Lite APIs Consist of One ClassXMLElement. It is very simple to work with it, for example:

XMLElement xml = new XMLElement ();
xml.parseString ("hello world");
xml.getChildren (); // Returns the children
xml.getAttribute ("attr"); // Returns the attribute value, etc.

But there are two drawbacks that had to be fixed:

1. NanoXML Lite initially does not support Mixed Content, which is often found in xhtml;
2. NanoXML Lite, like any other, does not support the essence  , which made it difficult to build a system of xhtml-templates (see below).

I had to write XPath support myself (112 lines of formatted code), in my opinion it turned out pretty well:

@Test
public void testXPathSlashInAttributeValue2 () {
	XMLElement xml = XMLUtils.createElementFromString ("");
	List results = xml.find ("// [@ a = '.. / .. / .. / file.txt'] / value");
	assertEquals (1, results.size ());
	assertEquals ("0", results.get (0) .getAttribute ("b"));
}

As for the Web Framework, at first I wanted to use Tapestry5 because I knew it well, but since the latest version (5.1) is not supported, just because of the limitations of the XML parser, I decided to abandon it. Those who are interested, the previous version - 5.0.18 - judging by the reviews, it works.

As a result, I decided to abandon the Web framework altogether and do everything on the Servlet API. To do this, I had to implement a couple of helper classes that work with xml page templates, which were based on xhtml and modified NanoXML Lite.

So, for example, the code for processing the index page (see the figure) looks like this:

private void responseWithQueryForm (HttpServletResponse resp) throws Error,
		IOException {
	XMLElement indexXml = TemplateLoader.loadTemplate ("index-template.htm", "UTF-8");
	XMLElement queryFormXml = TemplateLoader.loadTemplate ("query-form-template.htm", "UTF-8");
	ResultsXhtmlRender.fixCssLinks (indexXml);
	XMLElement queryDiv = indexXml.findById ("query-form");
	queryDiv.setContent (null);
	queryDiv.addChild (queryFormXml);
	resp.setContentType ("text / html");
	resp.setCharacterEncoding ("UTF-8");
	resp.getWriter (). println ("");
	resp.getWriter (). println (indexXml.toString ());
}

How does it work

The scheme of the service is very simple.

The query string entered by the user, the service passes to the search engines KinoKopilka and KinoPoisk in turn (this is another GAE / Java limitation - multithreading cannot be used). This is what explains that the query takes 5-10 seconds to complete. In addition, rating data and some other details that appear in the search results are on other pages and not displayed on the search pages of movie services, which also slows down the time.

GAE / Java provides URLFetchService for parsing HTML pages - a fairly convenient API (I advise you to look at the Low-Level API section) built on top of the Apache HTTP Client . Apache HTTP Client itself on GAE / Java refused to work.

Initially, I wanted to parse the results with my XML parser, but due to the fact that KinoKopilka and especially KinoPoisk produce non well formed xml, most parsing is done using regular expressions. Why is the majority - because in KinoKopilka xml is almost correct, there is one extra closing tag (I hope it will be corrected :), now I put the missing opening tag with pens and use NanoXML Lite + SimpleXPathService. (I didn’t just cite validators at the end of the link to xhtml / css, I hope that popular resources will use them sooner or later. It is time for beginners to think about it now.)

The result is a model of Movie and Rating objects (in the subject area two entity classes) containing the parsing results.

Further, this model, depending on the required output format, is rendered either in the form of xhtml using a template system, or using an xml render based on NanoXML Lite.

Examples of performing search queries are presented in the figures below:

Using

The service is available for public use, including the REST API, which you can freely use in your projects, as we do on planet33 , in order to receive additional information about films.

I also had plans to write a mobile client for this API, but I found another solution that completely suits me, namely the use of the minimalist block layout xhtml + handheld CSS media type , which allows me to use the service in Opera Mini on my Sony Ericsson.

I also posted the source codes of the project for review on Google Code under the GPL, I hope they will be useful to you and you will learn something new for yourself.

The structure of the project is shown in the figure:



The project has several warnings (Warnings) - this thing Google Eclipse Plugin hand, he says that I have removed unused jar'y of daddy WEB-INF/lib...

to have earned search "kinopoisk", you need to add the file Config.propertiesto src/dmitrygusev/ratings/utils/the following content:

kinopoisk-username = your-kinopoisk-username
kinopoisk-password = your-kinopoisk-password

Resources used

1. Ratings Service on Google Code - code.google.com/p/ratings-service
2. Google App Engine / Gettings Started: Java - code.google.com/appengine/docs/java/gettingstarted
3. NanoXML Lite Sources - prdownloads.sourceforge.net/nanoxml/NanoXML-2.2.1.tar.gz
4. CSS / XHTML online validation by w3c, respectively jigsaw.w3.org/css-validator and validator.w3.org
5. CSS block layout - webmolot.com/css
6. Mobile style - CSS Mobile Profile 2.0 - dev.opera.com/articles/view/mobile-style-css-mobile-profile-2-0
7. Service feedback used Idea.Informer.com - reformal.ru
8. FAMFAMFAM Silk Icons (a set of icons in PNG format, see the asterisk in the favicon service) - www.famfamfam.com/lab/icons/silk