May 31, 2011 at 15:03Large Files and Sinatra
Recently I ran into an interesting problem, when an attempt to send a large file through
The study showed that Sinatra itself reads and gives the file in pieces of 512 bytes, but the thin web server (as well as WEBrick) buffers the output in RAM at its own level, which leads to such sad consequences.
To solve the problem, it was enough to switch to the Rainbows web server (a web server based on unicorn code, but designed to work without proxying, for slow clients and / or services). But when uploading large files, the process ate about 30% of the CPU on one core.
Rainbows optimize file uploadsusing, for example, the sendfile gem, which provides the appropriate APIs for the operating system. But for this it is necessary that the file upload go through the Rack :: File API.
In the current Sinatra master branch, the send_file method was rewritten using the Rack :: File API, so we can just backport the corresponding functionality into existing versions of the Sinatra gem:
In this case, the rainbows configuration file will look something like this:
Now we use an effective file upload technique if the gem rack version 1.3 or higher is installed on the system and the sendfile gem is installed. By the way, when using ruby 1.9, the sendfile gem is most likely not needed.
PS: If your service is located behind a proxy server, then it is more optimal to use the capabilities provided by proxy servers, for example, the X-Accel-Redirect (nginx) or X-Sendfile (Lighttpd, Apache) APIs .
Sinatra::Helpers.send_fileled to the eats of all the RAM (the typical file size is 14Gb). The study showed that Sinatra itself reads and gives the file in pieces of 512 bytes, but the thin web server (as well as WEBrick) buffers the output in RAM at its own level, which leads to such sad consequences.
To solve the problem, it was enough to switch to the Rainbows web server (a web server based on unicorn code, but designed to work without proxying, for slow clients and / or services). But when uploading large files, the process ate about 30% of the CPU on one core.
Rainbows optimize file uploadsusing, for example, the sendfile gem, which provides the appropriate APIs for the operating system. But for this it is necessary that the file upload go through the Rack :: File API.
In the current Sinatra master branch, the send_file method was rewritten using the Rack :: File API, so we can just backport the corresponding functionality into existing versions of the Sinatra gem:
if Sinatra::VERSION < '1.3.0' && Rack.release >= '1.3'
# Monkey patch old Sinatra to use Rack::File to serve files.
Sinatra::Helpers.class_eval do
# Got from Sinatra 1.3.0 sources
def send_file(path, opts={})
if opts[:type] or not response['Content-Type']
content_type opts[:type] || File.extname(path), :default => 'application/octet-stream'
end
if opts[:disposition] == 'attachment' || opts[:filename]
attachment opts[:filename] || path
elsif opts[:disposition] == 'inline'
response['Content-Disposition'] = 'inline'
end
last_modified opts[:last_modified] if opts[:last_modified]
file = Rack::File.new nil
file.path = path
result = file.serving env
result[1].each { |k,v| headers[k] ||= v }
halt result[0], result[2]
rescue Errno::ENOENT
not_found
end
end
end
In this case, the rainbows configuration file will look something like this:
# try to use sendfile when available
begin
require 'sendfile'
rescue LoadError
end
Rainbows! do
use :ThreadSpawn
end
Now we use an effective file upload technique if the gem rack version 1.3 or higher is installed on the system and the sendfile gem is installed. By the way, when using ruby 1.9, the sendfile gem is most likely not needed.
PS: If your service is located behind a proxy server, then it is more optimal to use the capabilities provided by proxy servers, for example, the X-Accel-Redirect (nginx) or X-Sendfile (Lighttpd, Apache) APIs .
