Build Go web app in 2017

A couple of weeks ago, I started developing another web application, only already on Go . Being mainly a backend developer, I did not often write entire web applications, so each such case was more like a challenge. At such moments, I would like someone to write a guide on web development for people who do not have the opportunity to delve into the intricacies of great design all day, but who just need to create a working, functional website without too much fuss.

I decided to take this opportunity and make such a guide in which to start from scratch a web application in the way it should be done in 2017 (in my understanding). I spent many hours digging through the things that I had previously avoided, just so that at least once in many years I could confidently say that I have my own opinion and experience in this matter and get a working recipe for myself , which may be useful not only to me.

The article begins a short series covering what I learned in the process. This first post is a general introduction describing the current state of affairs, problems, and why I consider Go a good choice. Subsequent articles will be more detailed and contain more code. I am curious how much my experience correlates with yours; maybe I'm wrong about something, so feel free to comment.

If you are only interested in the code, it is here .

Introduction

Previously, my basic knowledge of HTML, CSS, and JavaScript was enough for my modest website building needs. Most of the applications that I have ever created were made using mod_python , directly using the mechanism for publishing handlers (note: you can see an example here ). It's funny that being an early Python follower, I also worked a lot with Rails.. Over the past few years, I have focused on the (big) data infrastructure, which is not web development at all, although the need for web interfaces here is not uncommon. In fact, the application that I am currently working on is a data application, but it is not open source and what it does does not matter for this article. In general, this should clarify which side I look at all this.

Python and Ruby

A year ago, I would recommend Python or Ruby as a web application environment. There may be other similar languages, but from my point of view, Python and Ruby dominate the world.

Most of the time, the main task of a web application was to build web pages using server-side HTML layout. Both Python and Ruby are very well suited for extracting data from a database and turning it into a bunch of HTML code using templates. There are many frameworks / tools to choose from, e.g. Rails, Django, Sinatra, Flask, etc. etc.

Although these languages ​​have certain significant limitations, such as GIL , the ease with which they solve the problem of generating HTML is much more valuable than the compromises that you have to make.

Gil

GIL (Global Interpreter Lock) tears a separate mention. Of course, this is the biggest limitation of any solution in Python or Ruby, but this is a very slippery topic, people often prefer to pretend that there is no problem. And if we are talking about this, emotions usually hit over the edge, in the Ruby and Python communities there are endless discussions on the topic of GIL.

For those unfamiliar with this problem - GIL allows you to do just one thing at a time. When you create threads and they “look” as parallel executing, in fact the interpreter still executes the instructions sequentially. This means that one process can use only one CPU.

Alternative implementations exist, such as those based on the JVM, but they are not often used. I don’t know exactly why, maybe they are not fully compatible or probably do not support C extensions correctly, and they still may have GIL. I'm not sure, but as far as I can tell, the implementation in C is usually used all the same. To make an interpreter without GIL, you have to completely rewrite it, and this can already change the behavior of the language (in my naive understanding), and therefore it seems to me that GIL will stay.

Web applications of any significant scale necessarily require the ability to service requests in parallel, using the capabilities of each available CPU on the machine. So far, the only possible solution is to run several instances of the application as separate processes.

This is usually done using additional software such as Unicorn / Gunicorn, with each process listening on its own port and starting behind some kind of connection balancer, such as Nginx and / or Haproxy. Alternatively, this can be done through Apache and its modules (such as mod_python or mod_wsgi), which is difficult anyway. Such applications typically rely on the database server as an arbiter for any competitive-sensitive tasks. When implementing caching, in order not to store multiple copies of the same on the same server, a shared-memory store is required, such as Memcached or Redis, and usually both. Also, such applications can not do background processing, for this there is a separate set of tools, such as Resque. And then all these components require monitoring to make sure that it all works. Logs must be consolidated, and they have their own additional tools. Given the inevitable complexity of this setup, a configuration manager such as Chef or Puppet is also required. And yet, these kits are generally not able to support a large number of long-term connections - a problem known asC10K .

As a result, a simple web-based database application requires a whole bunch of components before it can serve the Hello World! Page. And almost all of this is due to the GIL.

The advent of single-page applications

Farther and farther into the past is the generation of HTML on the server. The final (and correct) trend is to build the user interface and render completely on the client side using JavaScript. Applications, whose user interface is fully controlled by JS, are sometimes called a single-page application and, in my opinion, they are the future, whether we like it or not. In such applications, the server only serves the data, usually in the form of JSON, without generating HTML code. In this case, that enormous complexity, introduced primarily for the possibility of using the popular scripting language [to create a web application], is unnecessary. Especially considering that Python or Ruby is of little benefit when all the output is JSON.

A look at Golang

Go is gradually eroding the established world of web applications. It natively supports parallel execution, eliminating the need for almost all of the components commonly used to work with GIL constraints.

Programs on Go are binaries that are natively launched, so you do not need to install anything language-specific on the server. The problem of ensuring the correct version of the runtime required by the application disappears; There is no separate runtime environment - it is built into the binary. Go programs can easily and elegantly run tasks in the background, so there is no need for tools like Resque. These programs run as a single process, so caching becomes trivial, which means Memcached or Redis are not needed. Go can manage an unlimited number of concurrent connections, eliminating the need for front-end protection such as Nginx.

With Go, a tall multi-layered tower of Python, Ruby, Bundler, Virtualenv, Unicorn, WSGI, Resque, Memcached, Redis, etc., etc. reduced to just one binary. The only third-party component that is usually still needed is the database (I would recommend PostgreSQL). It is important to note that all these tools can still be used, but you can do without Go without them.

The launch time of such a Go-program will most likely be an order of magnitude greater than any Python / Ruby application, it will require less memory and lines of code.

Ok, but is there a popular framework?

The short answer is this: the framework is optional and not recommended. There are many projects that claim to be a great framework, but I think it's better to do without them. This is not only my personal opinion, I find this opinion quite common in the Go community.

You need to understand why frameworks were created at all. In the Python / Ruby world, this happened because these languages ​​were not originally designed to serve web pages, and many external components were needed to solve this problem. The same can be said about Java, which, like Python and Ruby, is as old as the web, as we know it, or even a little older.

As far as I remember, early versions of Python out of the box did not provide anything for working with a database, there were no templates, HTTP support was confusing, working with a network was non-trivial, even encryption was illegal then, and in general, a lot was missing. The framework provided all these necessary pieces and set language-specific development rules for all common web application variations.

Go, on the other hand, was created by people who already had experience and were versed in web development. It includes almost everything you need. One or two external packages may be needed to solve some specific tasks, such as OAuth, but in no case this pair of packages is not a "framework".

If all of the above regarding frameworks does not sound convincing enough, it is useful to consider the learning curve for the frameworks and the risks. It took me about two years to build a relationship with Rails. Frameworks can become abandoned and outdated, and porting an application to a new framework is difficult, and sometimes impossible. Given how fast things are changing in information technology, the framework certainly should not be chosen lightly.

I would like to highlight the tools and frameworks that try to simulate idioms common to Python, Ruby, or JavaScript environments. Everything that looks, feels, or claims to be the “Rails for Go” role, including techniques such as injections, dynamic publication of methods, etc. that are highly dependent on reflection, does not fit into Go’s ideology, therefore it’s better to stay away.

Undoubtedly, frameworks make some things easier, especially in the typical world of CRUD applications for business, where applications have many pages with many fields, manipulate data in complex and constantly changing database schemes. Not sure if Go is a good choice in such an environment, especially if performance and scalability are not a priority.

Another problem common to frameworks is that they abstract the low-level mechanisms from the developer so that over time they become so mysterious that it is literally impossible to understand what is really happening there. What starts with a lexical alias for a single line of JavaScript becomes a layer in the layers of transpilers, minimizers, on top of helpers hidden somewhere in the sub-dependencies. Once something breaks, and it’s impossible to figure out where to look for the problem. It's nice when you know exactly what is going on, and Go is very good at that.

What about database and ORM?

Similar to frameworks, ORMs in Go are not very common. For starters, Go does not support objects - what is indicated by O in the abbreviation ORM.

I know that instead of using the convenient User.find(:all).filter..., which is provided by something like ActiveRecord, writing SQL manually is something unheard of in some communities, but I still think that this attitude should change. SQL is a great language. Dealing with SQL directly is not so difficult, and in return we get more freedom and opportunities. Perhaps the most tedious part of such direct work is copying data from the database cursor into structures, but the sqlx project is very useful here .

Conclusion

In my opinion, the article describes in sufficient detail the current situation on the server side. I think it’s better to separate the client part into a separate post, therefore for today - that’s all. To summarize, we build the application with approximately the following requirements:

• Minimal dependency on third-party packages.
• Without a web framework.
• PostgreSQL as a database.
• Single page application.

Continuation