The Flask Mega-Tutorial, Part 18: Deployment on Heroku Cloud

This is the eighteenth article in a series where I describe my experience writing a Python web application using the Flask microframework .

The purpose of this guide is to develop a fairly functional microblogging application, which I decided to call microblog for a complete lack of originality.



In the previous article, we considered the option of traditional hosting. We saw two relevant examples of hosting on Linux servers, first on a regular server running CentOS, and then on the Raspberry Pi minicomputer. Those readers who did not administer Linux systems earlier may have decided that this requires too much effort and could be implemented somehow easier.

Today we will see if deploying “to the cloud” is a solution to the problem of excessive process complexity.

But what does it mean to deploy to the cloud?

Cloud hosting providers offer a platform on which our application can run. All that is required of the developer is to provide the application, and everything else, including server hardware, operating system, language interpreter and database, is taken over by the service.

Sounds too good to be true, right?

We’ll look at deploying the application on the Heroku platform , one of the most popular cloud hosting platforms. I chose Heroku not only because of its popularity, but also because it provides a free level of service, so we will deploy our application without spending a cent. If you want to know more about this type of service and what other providers offer, you can check out the Wikipedia page onPaaS .

Hosting on Heroku

Heroku was one of the first PaaS service platforms. In the beginning, she offered hosting services only for Ruby applications, but later support for many other languages ​​such as Java, Node.js and our favorite, Python, was included.

In fact, to deploy the application to Heroku you only need to download the application using git (you will see how it works very soon). Heroku looks for the Procfile file in the root folder of the application for instructions on how the application should be executed. For Python projects, Heroku also expects to see a requirements.txt file containing a list of required third-party packages.

After downloading the application, we can assume that the job is done. Heroku will apply its magic and the application will be available online in seconds. The amount of the bill at the end of the period directly depends on the computing power consumed by your application, therefore the more users your application has, the more you will have to pay.

Ready to experience Heroku? Let's start!

Creating a Heroku Account

Before you post the application on Heroku, you need to register there. Therefore, follow the link and create an account.

After authorization, you will be taken to the control panel, from where you can manage all your applications. We will not use the control panel intensively, but it provides a good overview of your account.

Install Heroku Client

Despite the fact that some tasks can be performed directly from the web interface, there are tasks that can be solved only from the terminal, so we will do everything in the console.

Heroku offers the Heroku Client utility, which we will use to create and manage our application. This utility can be run under Windows, Mac OS X and Linux. If the Heroku toolkit is available for your platform , then this is the easiest way to install the Heroku client.

The first thing we will do with the help of the client is to log into our account:

$ heroku login

Heroku will ask you for an email and password for your account. At the first authorization, the client will send your ssh key to the Heroku server.

Subsequent commands can be executed without authorization.

Git setup

git is the foundation for deploying applications on Heroku, so it must also be installed. If you installed the Heroku toolkit, then git is already installed.

To deploy the application to Heroku, it must be present in the local repository, so run the following commands in the console:

$ git clone git://github.com/miguelgrinberg/microblog.git
$ cd microblog

Creating a Heroku Application

To create a new Heroku application, just call the create command from the root folder of the application:

$ heroku create flask-microblog
Creating flask-microblog... done, stack is cedar
http://flask-microblog.herokuapp.com/ | git@heroku.com:flask-microblog.git

In addition to setting the URL, this command adds to our repository a remote repository ( git remote ), which we will soon use to upload application code to the cloud.

Naturally, the name flask-microblog is now taken by me, so come up with some other name for your application.

Exclude local file storage

Some functions of our application save information in the form of files on disk.

And here we are faced with a difficult task. Applications running on Heroku do not have the ability to permanently store files on disk, because Heroku uses a virtualization platform that does not store data as files, the file system is cleared of all files except the application files itself, each time the instance is launched. Strictly speaking, an application can store temporary files on disk, but should be able to recover these files if they disappear. In addition, if two instances are running, each of them uses its own virtual file system and there is no way to split files between them.

This is really bad news for us. For starters, this means that we cannot use sqlite as a database.

Our Whoosh full-text search database will also stop working, because it stores its data in the form of files.

The third problem point is our logging system. We saved our log in the / tmp folder and now, when working on Heroku, this will also stop working.

So, we have identified 3 main problems for which we need to look for solutions.

We will solve the first problem by migration to the database offered by Heroku, which is based on PostgreSQL .

For a full-text search to function, we do not have a ready-made alternative available. We will have to implement full-text search using PostgreSQL functionality, but this will require changes to our application. Of course, it’s a pity, but the solution to this problem would now lead us far away from the topic of the article, so for posting on Heroku, we simply turn off full-text search.

And finally, since we cannot write our logs, we will add our logs to the logging system used by Heroku, which, by the way, is very easy to use, because it sends everything that is output to stdout to the log.

Creating a Heroku Database

To create the database, we use the Heroku client:

$ heroku addons:add heroku-postgresql:dev
Adding heroku-postgresql:dev on flask-microblog... done, v3 (free)
Attached as HEROKU_POSTGRESQL_ORANGE_URL
Database has been created and is available
 ! This database is empty. If upgrading, you can transfer
 ! data from another database with pgbackups:restore.
Use `heroku addons:docs heroku-postgresql:dev` to view documentation.

Note that we use the development database, because this is the only free option. For the battle server, you will need to select a different database option.

And how does our application know the parameters for connecting to the database? Heroku puts the database URI in the environment variable $ DATABASE_URL. If you remember, we made changes to our configuration file in a previous article, incl. the value of this variable will be used to connect to the database, as required.

Disabling full-text search

To disable full-text search, our application must be able to determine whether it is running on Heroku or not. To do this, we will create a custom environment variable, again using the Heroku client:

heroku config:set HEROKU=1

Now the HEROKU environment variable will be set to 1 when our application is launched on the Heroku virtual site.

Turning off full-text search is now quite simple. To get started, add a variable to the configuration file (config.py file):

# Whoosh does not work on Heroku
WHOOSH_ENABLED = os.environ.get('HEROKU') is None

Then, cancel the creation of the full-text search database (app / models.py file):

from config import WHOOSH_ENABLED
if WHOOSH_ENABLED:
    import flask.ext.whooshalchemy as whooshalchemy
    whooshalchemy.whoosh_index(app, Post)

We also add information about full-text search in g in our before_request handler so that our templates can see it (file app / views.py):

from config import WHOOSH_ENABLED
@app.before_request
def before_request():
    g.user = current_user
    if g.user.is_authenticated():
        g.user.last_seen = datetime.utcnow()
        db.session.add(g.user)
        db.session.commit()
        g.search_form = SearchForm()
    g.locale = get_locale()
    g.search_enabled = WHOOSH_ENABLED

And finally, remove the search field in the base template (file app / templates / base.html):

        {% if g.user.is_authenticated() and g.search_enabled %}
        
{{g.search_form.hidden_tag()}}{{g.search_form.search(size=20,placeholder=_('Search'),class="search-query")}}
        {% endif %}

We fix logging

Under the control of Heroku, everything that is output to the stdout stream immediately enters the Heroku application log. But the logs that are written to files on the disk will not be available. So on this platform, we must disable logging to files and instead use a logger that writes errors directly to stdout (file app / __ init__.py):

if not app.debug and os.environ.get('HEROKU') is None:
    import logging
    from logging.handlers import RotatingFileHandler
    file_handler = RotatingFileHandler('tmp/microblog.log', 'a', 1 * 1024 * 1024, 10)
    file_handler.setLevel(logging.INFO)
    file_handler.setFormatter(logging.Formatter('%(asctime)s %(levelname)s: %(message)s [in %(pathname)s:%(lineno)d]'))
    app.logger.addHandler(file_handler)
    app.logger.setLevel(logging.INFO)
    app.logger.info('microblog startup')
if os.environ.get('HEROKU') is not None:
    import logging
    stream_handler = logging.StreamHandler()
    app.logger.addHandler(stream_handler)
    app.logger.setLevel(logging.INFO)
    app.logger.info('microblog startup')

Web server

Heroku does not provide its web server. Instead, it expects the application to launch its own server on the port, the number of which will be obtained from the environment variable $ PORT.

We know that the Flask development server is not suitable for work in production, because it is single-processor and single-threaded, so we need a better solution. The Heroku Guide for Python Applications recommends gunicorn , which we will apply.

In our local environment, gunicorn is installed as a regular python module:

$ flask/bin/pip install gunicorn

To run it, we need to pass a single argument with the name of the Python module that defines the application and the application object itself, separated by a colon.

Let's create a separate Python module for Heroku (runp-heroku.py file):

#!flask/bin/python
from app import app

Now, for example, if we want to start the gunicorn server locally, using this module, we must run the following command:

$ flask/bin/gunicorn runp-heroku:app
2013-04-24 08:42:34 [31296] [INFO] Starting gunicorn 0.17.2
2013-04-24 08:42:34 [31296] [INFO] Listening at: http://127.0.0.1:8000 (31296)
2013-04-24 08:42:34 [31296] [INFO] Using worker: sync
2013-04-24 08:42:34 [31301] [INFO] Booting worker with pid: 31301

Requirements.txt file

Very soon, we will upload our application to Heroku, but first we must tell the server which modules our application needs to run. On our local PC, we managed the dependencies using a virtual environment, installing modules into it using pip.

Heroku does the same. If the requirements.txt file is found in the root folder of the application, then Heroku installs all the modules listed in it using pip.

To create the requirements.txt file, we must use the freeze option when pip is called:

$ flask/bin/pip freeze > requirements.txt

You need to add the gunicorn server to the list, as well as the psycopg2 driver, which SQLAlchemy needs to connect to the PostgreSQL database. The final form of the requirements.txt file will be like this:

Babel==0.9.6
Flask==0.9
Flask-Babel==0.8
Flask-Login==0.1.3
Flask-Mail==0.8.2
Flask-OpenID==1.1.1
Flask-SQLAlchemy==0.16
Flask-WTF==0.8.3
git+git://github.com/miguelgrinberg/Flask-WhooshAlchemy
Jinja2==2.6
MySQL-python==1.2.4
psycopg2==2.5
SQLAlchemy==0.7.9
Tempita==0.5.1
WTForms==1.0.3
Werkzeug==0.8.3
Whoosh==2.4.1
blinker==1.2
coverage==3.6
decorator==3.4.0
flup==1.0.3.dev-20110405
guess-language==0.2
gunicorn==0.17.2
python-openid==2.2.5
pytz==2013b
speaklater==1.3
sqlalchemy-migrate==0.7.2

Some of these packages will be unclaimed by the version of our application designed for Heroku, but there is nothing to worry about having unused packages in the system. And it seems to me that it's better to have a complete list of required packages.

Procfile

The final requirement is to tell Heroku how to launch the application. Heroku requires a Procfile file in the root folder of the application.

This file is quite simple, it just defines the names of the processes and commands associated with them (Procfile file):

web: gunicorn runp-heroku:app
init: python db_create.py && pybabel compile -d app/translations
upgrade: python db_upgrade.py && pybabel compile -d app/translations

The web label is associated with the web server. Heroku needs this task to run our application.

The other two tasks, called init and upgrade, are user tasks that we will use to work with our application. The init task initializes our application by creating a database and compiling language files. The upgrade job is like init, but instead of creating a database, it updates the database to the last migration.

Application Deployment

Now we are starting the most interesting part, in which we will place the application in our Heroku account. It's pretty simple, we just use git to submit the application:

$ git push heroku master
Counting objects: 307, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (168/168), done.
Writing objects: 100% (307/307), 165.57 KiB, done.
Total 307 (delta 142), reused 272 (delta 122)
-----> Python app detected
-----> No runtime.txt provided; assuming python-2.7.4.
-----> Preparing Python runtime (python-2.7.4)
-----> Installing Distribute (0.6.36)
-----> Installing Pip (1.3.1)
-----> Installing dependencies using Pip (1.3.1)
...
-----> Discovering process types
       Procfile declares types -> init, upgrade, web
-----> Compiled slug size: 29.6MB
-----> Launching... done, v6
       http://flask-microblog.herokuapp.com deployed to Heroku
To git@heroku.com:flask-microblog.git
 * [new branch]      master -> master

The heroku label that we use in our git push team was automatically registered in our git repository when we created our application using heroku create. To see how this remote repository is configured, you can run git remote -v in the application folder.

When you first download the application to Heroku, we need to initialize the database and translation files, and for this we need to complete the init task, which we included in our Procfile:

$ heroku run init
Running `init` attached to terminal... up, run.7671
/app/.heroku/python/lib/python2.7/site-packages/sqlalchemy/engine/url.py:105: SADeprecationWarning: The SQLAlchemy PostgreSQL dialect has been renamed from 'postgres' to 'postgresql'. The new URL format is postgresql[+driver]://:@/
  module = __import__('sqlalchemy.dialects.%s' % (dialect, )).dialects
compiling catalog 'app/translations/es/LC_MESSAGES/messages.po' to 'app/translations/es/LC_MESSAGES/messages.mo'

The warning belongs to SQLAlchemy, because it doesn’t like URIs starting with postgres: // instead of postgresql: //. This URI forms Heroku through the value of the environment variable $ DATABASE_URL, so it's not up to us to change this. It is hoped that this URI format will work for a long time.

Believe it or not, our application is already available online. In my case, the application is available at flask-microblog.herokuapp.com . You may well become my follower from my profile page . I don’t know exactly how long the application will be available at this address, but nothing prevents you from checking if it is available or not!

Application update

Sooner or later, the time will come to update our application. This will happen like an initial deployment. First, the application will be uploaded to the server using git:

$ git push heroku master

Then, the update script is executed:

$ heroku run upgrade

Logging

If something abnormal happens with the application, it can be useful to study the logs. Remember that for Heroku version of the application, we write all the logs in stdout, and Heroku collects them in its own log.

To view the logs, the Heroku client is used:

$ heroku logs

The above command will display all the logs, including Heroku logs. To view the logs of only your application, run the command:

$ heroku logs --source app

Things like the call stack and other application errors, everyone will be in this log.

Is it worth it?

Now we have an idea of ​​deploying the application on a cloud platform, and therefore we can compare this type of hosting with a traditional hosting option.

On the issue of simplicity, victory is beyond the clouds. At least for Heroku, the application deployment process was very simple. When deploying to a dedicated server or VPS, you had to do a lot of preparatory work. Heroku takes care of this and allows us to focus on our application.

The issue of cost is a moot point. The cost of cloud hosting services is usually more expensive than dedicated servers, because you pay not only for the server, but also for administration services. A typical Heroku tariff plan, which includes two instances and the cheapest production database, will cost $ 85 (this is at the time of writing these lines.About a year ago - approx. per. ) On the other hand, if you look well, then you can very well pick up a pretty decent VPS for ~ $ 40 a year.

In the end, it seems to me that the question of choice comes down to choosing what is more important for you: time or money.

The end?

The updated application is available, as always, on github . Or you can download it as a zip archive from the link:

Download microblog 0.18 .

With the deployment of our application in all possible ways, it seems that our tour is coming to an end.

I hope these articles were a useful introduction to the development of a real web application, and that the knowledge that I dumped on you for these 18 articles motivates you to create your own project.

However, I do not put an end to, and do not deny the likelihood of, articles about microblogs. If, and when, an interesting topic comes to mind, I will write more, but I expect that the frequency of updates will now slightly decrease. From time to time, I can make some minor corrections in the application that do not deserve a separate blog article, so you can track these changes on GitHub .

In my blog I will continue to write articles related to web development and software in general, incl. I invite you to follow me on Twitter or Facebook if you have not already done so, and in this way you will be notified of my subsequent articles.

Thank you again for being a loyal reader.

Miguel