Serverless approach for rapid development of a working video service

I work in outsourcing, where the main principle can be described with the phrase "sell a lot, do it fast." The faster we do, the more we earn. And, it is desirable that everything does not work on crutches and snot, but with an acceptable level of quality. I will tell about my experience when in a short time it was necessary to develop a promotional service.

Given: root account on AWS, no restrictions on the choice of technology stack, one back-end, and one month for development.

Task:implement a promotional service where users upload from one to four videos lasting from one to four seconds, which are then embedded in the original video sequence. That is, we replace fragments of the original video (series screensaver) with custom ones. Killer feature - the ability to send your name, which in the form of beautiful text overlays the corresponding video fragment. In addition, the user can upload a video from 4 to 30 seconds long, and on the service side it will be trimmed to 4 seconds.

Decision

Writing your bike service in such a tight time frame is an idea. In addition, for the service to cope with the loads and everyone who wants to receive the coveted video, the infrastructure will be required. And preferably not with a price tag from the plane. Therefore, we immediately focus on ready-made solutions with minimal customization.

The standard solution for working with video is FFmpeg, a cross-platform console utility that, through arguments, allows you to slice and overlay sound. The small thing is to write a wrapper and put it into practice. We are writing a prototype that sticks together two videos, and ... the fun begins. The library on .NET Core 2 should spin on any virtual machine, so we take the AWS EC2 instance and everything will work


FFmpeg, although it simplifies the task, but for a really working solution you need to create an EC2 instance, design a network infrastructure for it, including the Load Balancer. The simple task of deploying from scratch is “a bit” complicated, and the infrastructure starts demanding money immediately - the amount for the runtime is withdrawn from the client’s account every hour.

Our service does not involve Long-Running processes, does not require a large and bold relational database, and fits perfectly into the event architecture with a chain of microservice calls. The solution suggests itself - we can abandon EC2 and implement a true-serverless application, like the standard Image Resizer based on AWS Lambda.

By the way, despite the obvious dislike of AWS developers for .NET, they support .NET Core 2.1 as a runtime, which gives a full range of development opportunities.

And the cherry on the cake - AWS provides a separate service for working with video files - AWS Elemental MediaConvert.

The essence of the work is incredibly simple: we take the S3 link to the outgoing video, through the AWS Console, .NET SDK or just JSON write what we want to do with the video and call the service. He himself implements the queues for processing incoming requests, he unloads the result in S3 and, most importantly, generates a CloudWatch Event for each status change. This allows us to implement lambda triggers to complete video processing.


The final version of the architecture looks something like this

The entire backend is placed in two lambdas. Another is for rotating vertical videos, since such work cannot be done in one go.

Text overlay was done by rendering a transparent image on the fly in accordance with the video resolution. The SixLabors.ImageSharp library is perfect for this purpose, and memory leak problems elegantly bypass the lambda life cycle.

The front in the form of a SPA application written in JS and compiled via pug will be placed in the public S3 basket. To download the videos themselves, we don’t need any server code - just open the REST endpoints that S3 offers us. The only thing - do not forget to configure policies and CORS.

Underwater rocks

• AWS MediaConvert for some unknown reason imposes sound only on each video clip individually, and we need a fervent song from the whole screensaver.
• vertical videos need to be processed separately. AWS does not like black stripes and puts the rollers at 90 °.

Easy rink

Despite the beauty of Stateless, you need to keep track of what needs to be done with the video: glue or overlay sound on an already finished video sequence. Fortunately, MediaConvert supports the transfer of metadata through its Job, and we can always apply a simple flag like “isMasterSoundJob”, parsing this metadata at any stage.

Serverless makes NoOps work fine - an approach that assumes the need for a separate team responsible for the project infrastructure. Therefore, it was a small matter - we deploy the solution on AWS without the participation of system administrators, who always have something to do.
And in order to speed it all up, we automate the deployment script to AWS CloudFormation as much as possible, allowing you to deploy one button directly from VS. As a result, a file with 200 lines of code allows you to roll out a ready-made solution, although the CloudFormation syntax can be shocking out of habit.

Total

Serverless is not a panacea. But it will greatly facilitate life in situations with three limits: "limited resources — short time — little money."

Characteristics of applications suitable for Serverless

• without long-running processes. Hard limit API Gateway - 29 seconds, hard limit lambda - 15 minutes;
• described by Event-Driven architecture;
• breaks down into loosely coupled components by type of SOA;
• does not require a lot of work with his condition;
• written in .NET Core. To work with the .NET Framework, you will still need at least a Docker with the appropriate runtime.

Benefits of a Serverless Approach

• reduces infrastructure costs;
• reduces the cost of delivering a solution;
• Auto scalability
• development at the forefront of technological progress.

Disadvantages, on a concrete example

• Distributed tracing and logging - partially solved through AWS X-Ray and AWS CloudWatch;
• inconvenient debugging;
• Cold Start in the absence of load;
• AWS user-hostile interface is a universal problem :)