Simple file hosting on Google App Engine

Surely each of you in your life has found a file hosting convenient for you, and after some time you have discovered that it starts to hurt your eyes on the amount of advertising, the conditions are already far from so loyal, and in general it is time to find something new. There are two options for further actions - either find a new, not yet untwisted file hosting service and use it until it goes bad, or organize your own solution. For the second option, in turn, you can purchase a hosting service (you really have to fill up cones until you find a bona fide hoster with quality services) or use a cloud service.

Google’s PaaS hosting turned out to be a rather interesting find - Google App Engine (hereinafter GAE), which makes it possible to store up to 5 GB of files with 1 GB of incoming and 1 GB of outgoing traffic per day, and, among other things, it uses the High Replication model. that is, your data will be stored on several servers all over the world at once!
A feature of GAE is a somewhat non-standard interface for working with files, which is why I made my own service, which I will discuss in this article.

GAE makes it possible to create applications in Java, Go and Python. Since the first two languages ​​are almost unfamiliar to me, I wrote my service in Python. Previously, you could only use version 2.5, which created some difficulties, but recently added support for 2.7, so now you do not need to recall outdated approaches.

To start using GAE, download and install Python version 2.7, as well as the appengine SDK for your operating system. The narration will be conducted on the example of the SDK version for Linux (Windows and Mac have a convenient graphical interface, so it will be easy to understand, although you can do everything from the console, as described below).

To make things easier with the settings in the future, it is recommended to register the application in advance at http://appengine.google.com by clicking on the “Create application” button. You will be prompted to enter a unique identifier for the application, according to which it will be available as a subdomain of appspot.com, as well as a name that can be changed in the future, unlike the identifier. All other settings can not be touched, in our case they do not have much significance. To get started, create a folder with a name that matches your identifier, in which all files related to the application will be stored.

Any application consists, at a minimum, of the app.yaml file, which contains the name of the application, the version of the runtime, and a description of the URL handlers and various errors. The URL handler consists of a regular expression, with which links are checked, as well as descriptions of the necessary files. I note that in the names of the scripts for the handlers, now we need to put the extension (which, in general, is not an extension, but an object in the application), not '.py', but '.app'

For our application, the app.yaml file looks like this:

application: fileshare # здесь вместо fileshare нужно подставить свой идентификатор
version: 1
runtime: python27
api_version: 1
threadsafe: true
handlers:  
- url: /favicon\.ico
  static_files: favicon.ico
  upload: favicon\.ico
- url: /delete/.* # Ссылка для удаления файла
  script: main.app
  login: admin # Размещать и удалять файлы сможет только администратор приложения.
- url: /get/.* # ссылка для прямого доступа к файлу, открывать может кто угодно, хотя если добавить login: admin или login: required, скачать его сможет только администратор или любой пользователь Google соответственно.
  script: main.app
- url: .* # Страница, на которой размещена информация обо всех загруженных файлах и форма для загрузки.
  script: main.app
  login: admin

For storage of files, App Engine has a special Blobstore storage, and for data - Datastore (I note that a non-relational data storage model is used; for access to data, there is its own framework similar to ORM from Django). In our application, each file is assigned a FileRecord object in the Datastore with a built-in unique integer identifier (meaning retrieving the object key by id), which contains the BlobReferenceProperty object:

class FileRecord(db.Model):
  blob = blobstore.BlobReferenceProperty()

For a more specific analysis of the URL, and in general, to initialize the application, an app object is created at the end of the file, which is a representative of WSGIApplication:

app = webapp2.WSGIApplication(
          [('/', MainHandler),
           ('/upload', UploadHandler),
           ('/delete/([^/]+)?', DeleteHandler),
           ('/get/([^/]+)?', GetHandler),
          ], debug=False)

Depending on the URL received, the application starts the corresponding handler. In this case, the handlers from app.yaml are partially duplicated, since it contains settings for authorization.

The application works as follows:

• When accessing the '/', the user is authorized, then all entries are downloaded and the form for downloading the file is displayed.
• When accessing '/ upload', which occurs automatically when a file is uploaded, an object is created in Blobstore, which is associated with an object in Datastore, then, if successful, a transition to '/' is made.
• When accessing '/ delete /', the object number is retrieved from the URL, after which it is deleted.
• When accessing '/ get /', the object number is also retrieved from the URL, after which the file associated with it is sent for download.

To test the application, run the debug server in the console:

python google_appengine/dev_appserver.py <папка с вашим приложением>

If you want the server to be accessible over the network, add the --address 0.0.0.0 parameter after dev_appserver.py.

If there are no problems, you can start downloading the application:

python google_appengine/appcfg.py update <папка с вашим приложением>

Next, you will be asked for the login and password for your Google account, after which the application will download and be available at appid.appspot.com , where appid is your unique identifier.
The source code of the application on code.google.com (for use, simply edit the file app.yaml): http://code.google.com/p/fileshare-appengine

Advantages over existing file hosting:

• No need to watch ads, wait a minute, etc.
• The file can be downloaded via a direct link.
• Very high reliability.
• Only you control what is uploaded to your file hosting service. It will not be closed due to another user
• Within the specified limits on the use of resources, everything is free.
• If even the allocated resources are few, you can make another application (these are personal speculations, there is nothing specific in the rules for this case, so it's best to find out. UPDATE: this is still forbidden by the rules).

Misunderstandings:

• It is not clear about the maximum file size - 2 GB is written in the Russian documentation, and 32 MB is written in the English.
• It turned out that Russian file names are not supported. I do not quite understand why.

UPDATE: If an error of the UnicodeDecodeError type is thrown at the stage of loading the script on Windows, it may help to delete all the keys that have Cyrillic from the registry branch HKEY_CLASSES_ROOT / Mime / Database / ContentType
UPDATE2: Now you can view the sources at the link http://code.google .com / p / fileshare-appengine / source / browse /
UPDATE3: as well as on GitHub https://github.com/SergeyBurma/fileshare-appengine