Clojure Cup - experience participating in a hackathon on writing a web application in Clojure

This weekend, the Clojure Cup was held - a 48-hour hackathon at Clojure. This will be a relatively long post about how he went on in our team, with technical details and more. Well, at the end of the post, where without it, we will tell quite a bit about our project. And for those who are too lazy to read, I can immediately go to CodeNotes and even vote for us on the Clojure Cup page.

Work environment

There are two different “modes” in participating in a hackathon: participation for pleasure and victory. In the first case, we lazily code something cool for two days; in the second, everything is a little different, with hardcore and an early start of preparation. We chose the second option.
There were four of us in the team - two, including me and si14 , are in St. Petersburg, one in Tyumen ( zloy_alu ), and one in Munich. But we did not experience any communication problems - more or less successful task isolation, excellent Trello service and simply divine FlowDock helped. The latter, in my opinion, is an ideal service for intra-team interaction. Judge for yourself - there is chat, and file sharing, and separate discussion threads, and notifications about various events (deploy, push to the repository, new tasks in trello, mention on Twitter). It all looks something like this:


Since we clearly planned to win, we started preparations in advance. In particular:

• checked the selected libraries and the structure of the application as a whole (for more details see below about Clony);
• tested integration with third-party services like NewRelic;
• started a blog and Twitter;
• Learned how to deploy quickly and conveniently (Makefiles and Ansible);
• prepared documentation for the JSON API between the server and the web app, which allowed us to write these pieces more or less independently;
• prepared sketches of the interface on paper;
• bought a domain and issued an SSL certificate;
• We learned how to configure nginx for normal work with web sockets and static resources;
• prepared the contacts of the beta testers pool (here came some file - we lagged behind the schedule and did the mailing only in 11 Moscow, so we managed to get only one review before the end of the hackathon).

Technical details

Now for details on what might be useful for Clojure web developers.

The experience of creating a small Clony application turned out to be very useful . This is an extremely simple thing on Clojure using the http-kit (http-server and client for Clojure) and Compojure (router) communicating with its front-end web sockets. It uses Stuart Sierra's workflow , which makes development much more enjoyable: we completely restarted all of our application (or its pieces) from nrepl in a split second. During the hackathon, we made several improvements that are waiting for their backport in Clony.

The first day

We knew in advance that we would be deploying to a virtual machine, so we learned how to use Ansible . Thanks to him, we were online 4 hours after the start of the competition - with a fully developed production configuration and a test application. This would have happened earlier if not for the problems with DigitalOcean , which provided hosting for participants (they banned our account due to suspected fraud). We also had local virtual machines ( Vagrant- super!) With a completely identical configuration, but in the end everyone scored on them and deployed immediately to production. This happened also because some things had to be changed directly on the fly on the server by hand - starting from the MariaDB configuration (damn these ANSI_QUOTES! We had to take Postgres right away) and ending with the fact that we forgot to write logs to disk. Plus we made a fairly complete integration with NewRelic . I can’t say that he critically helped us out, but the ability to easily see such a large number of parameters - server response time, DOM rendering time by clients, network latency, the amount of free memory on the server, GC JVM statistics - greatly calmed the nerves. The library for integration turned out to be very small and we will definitely post it in open-source.

On the first day, we spent a lot of time trying to get clang - the integration of Angular.js with ClojureScript. The abundance of minor troubles eventually forced to abandon ClojureScript entirely and switch to bare JavaScript. ClojureScript is much better than a language, but in a hackathon, there was simply no time to fight your own misunderstanding of its subtleties.

Trying to do everything as correctly as possible, we decided to add a library for SQL migrations to the project. It was interesting to find that the very popular Lobos migration library does not work at all . As a result, we lost about an hour searching and proceeding with a new one; stopped at Ragtime .

As a result, at the end of the first day we had a beautiful main page and a semi-working list of repositories (a short description of our project is slightly lower). In the interval 22: 00–03: 00 the team went to bed (no one had been sleeping since 4 in the morning) and we gathered again around 8 in the morning on Sunday.

Second day

On Sunday, things went better, but there were quite a few problems:

• I had to redo a small piece of OAuth authorization;
• at some point, GitHub banned us on the limit of API requests, which was extremely unpleasant, and we had to reduce the number of requests sent;
• To generate pictures, we first tried to use something in pure Java, but in the end we took binders to ImageMagick. There are two of them: one works through sh and pipes, the other through JNI. The first one slows down (30 + ms per picture), it was not possible to start the second one;
• both with slow image generation and with the number of requests to GitHub, core.cache helped us out - the excellent “core” Clojure library;
• we made a risky decision to take core.async to work with events on the server. As a result, this turned out to be a good solution: the first working version of the code using it was born in an hour and a half and there were practically no problems with it, despite the fact that no one in the team had used it before. Using core.async, we made it possible to write a particular user to the web socket from anywhere in the code and the complex logic of restarting the repository scan in case of server overload: we process the scan request queue, discarding everything that overflows the queue, and in the absence of tasks in the main queues we select from the base what was thrown out during overload. The code is very compact and seems to work;
• bugs on the client, some of which, alas, we did not have time to fix until the end of the hackathon.

In general, the second day was more than productive. We tried to focus on the first impression, so we paid attention to the little things: redirect to our own domain, beautiful “How to Use”, clear text on the main one, nice design, dies with “sorry, you don’t have TODO in the repository, here’s the bug report form, if this is not true". Of course, to speed up the development, we used Bootstrap , but we tried to do everything as beautifully and “non-defaultly” as possible. How much this will help us get the best marks from the judges, we will find out on Friday.

The most interesting - what is it all about?

It will be strange not to say anything about our project.

It seems to us that humanity has not yet come up with the perfect bugtracker. However ... Why do we need a bug tracker at all? The essence of our project is to remove a separate bugtracker from your life. And we tried and will try further so that he can look like this:

def somefun ():
    # TODO: @ si14, please fix this before release
    return true

And we will do the rest - we will send the user a notification that you created a task for him, display it in our interface, and we will also monitor your commits and scan the code for new TODO, FIXME and NOTE. In addition, you can generate a nice picture for your GitHub repository with an indication of how many TODOs are in your code and a link to information about it. Our CodeNotes project does not yet have all the desired functionality (plus there are still quite a few annoying bugs, what can I hide), but when the vote ends, we will immediately continue to improve it and make it the best bug tracker for small projects.

In the meantime, by the way, you can vote for us and comment on what you think of the idea, hackathons, and Clojure web programming in general.