Network Render in Blender

picture for those in the tank

A couple of years ago I was asked to write an educational program on a network render in a blender.
I got a little carried away and it turned out what I present to your attention.
The review was never published to the end on blender3d.org.ua , although pdf can be found there. Please save, maybe someone will be interested.

Network Rendering in a Blender

This guide was written to demonstrate the capabilities of a blender in this
relatively new role for him.
For many years now, programs like 3Dstudio have been used online for collaboration or rendering, while in Blender this feature has appeared relatively recently. With the release of the short film Sintel, the authors demonstrated the render studio they worked at when creating the film, and also complained that they had to use third-party development for the network render instead of the network render built into Blender 2.5, since by that time it was unable to provide a studio with a stable and fast result.
Let's see how things are now, and at the same time compare the internal network render of Blender
with the LuxBlend plugin, which recently received OpenCL support.

Internal netrendering

With the advent of Blender 2.5, the need to have at least some kind of network render inside the program caused the netrender to be instantly included in the code written, like most plug-ins, in Python Martin Poirier.
The idea turned out to be not bad: use the internal Blender render, which was to guarantee the identity of the image processed locally or remotely; centralization of management on the master server; simple server management and fast, not complicated installation.
However, the first tests showed that much remains to be done, and at the moment, although the plugin is delivered inside the distribution kit, it must be activated manually.

The

Netrender installation in the blender is divided into three components:



Client - Sending work to the master server.
Master - Receiving work from clients.
Slave - Server dependent on the master, processing render frames.

Client.

It can be any machine with Blender installed; when the scene is ready, the ip is configured and the port of the master accepting the requests is set, the number of frames required for rendering is set, then the task is sent to the master, which will distribute it between the other servers.

The following options are present in the client interface:

Animation on network - the entire work is sent as soon as the first frames are ready, they are transmitted to the client in .exr format for display in the editor.

Send job- sends all selected frames to the master server, after clicking this button, “Job send” should appear in the information window, this may take some time if your .blend and textures take up enough space. This will take even more time if the master is located outside the local network and the channel to it is limited.
It may even seem that the Blender is frozen, check the network traffic and make sure that the client has started transmitting data over the network. This option immediately frees Blender for the next task, as it does not wait for the result of the wizard.

Send current frame job- It sends only the selected frame, but keep in mind that, as in other cases, it is necessary to send all the information (.blend and textures), so that it will take as much time as sending all the work. The result does not wait, instantly freeing the client.

After the wizard accepts the work, the transfer of the entire file and textures to a free slave begins, it follows that it is better to place them on the same local network, otherwise we can get a lot of traffic and another potential timeout point.
What we are trying to achieve is, by clicking on the button, continue to work without thinking about which server will render, as well as about problems that may arise.

From time to time, you can glance at the web-interface built into the wizard (you can access it by clicking on “Open Master Monitor” or entering in the browser (http: // host: ip) and picking up the finished frames. There is no mysticism in the work of the slave, this is Blender’s distribution, to which the file and frame numbers are transmitted to the render through parameters; as soon as the task is ready, it will transfer the result in the .exr format to the master, .log file and accept the next portion of frames.
The wizard, on the other hand, is a much more complex code, the task of which, in addition to interacting with the client, is the distribution of works and their frames between available slaves. The load balance has not yet been polished, and it is difficult to predict who will receive the next frame, nevertheless, in general, it copes with its task, at least I did not have to see idle slaves with available work.

We take the result.

When several frames are ready, we can pick them up from the master, at the moment there are at least two details about which, in my opinion, it is worth telling.
Firstly, the blender works in the .exr format, in fact, this means that every finished frame will be saved in the open HDR photo format, which gives good opportunities for post-processing, but is far from ideal in terms of size. Thus, our photo taken on one remote server can very easily take more time to download, process and transfer than if we were rendering locally.
Let's say our .blend file takes 1mb + another 512kb for textures, it’s 1.5mb just for “raising” work to the master, the master gives these 1.5mb to the slave (anyway, with a single frame you won’t be able to use more), the slave renders in 10 minutes and sends the .exr file to the master, if FullHD-like permission was set, then with the exception of the case of a 90s machine or a very good client-master connection, we will get the frame much later than if we used a local render.

On this note, let me make a small digression: the fact is that, firstly, network rendering is most often used by studios on a local network, with a channel of at least 1 GB / s, which greatly changes the matter. Secondly, in many well-established programs for editing three-dimensional images, it is possible to use several machines when processing one frame, the picture is divided into parts equal to the number of servers, or other methods are used that reduce the time spent on rendering in proportion to the number of servers involved .
Since this function is not yet available in the blender, obviously, advantageous results can be achieved only by animating scenes with a sufficient number of frames to compensate for delays in downloading and receiving files. In addition, do not forget about the ability to completely unload the client after the material is submitted for rendering, which can be very useful if you need to continue working on another project.

Tools

For a minimal installation, you need one client computer and one master / slave, since both functions can be implemented by one computer with the master and slave servers running simultaneously. Render speed will directly depend on the speed of the slave machine.

pros

• Easy to install - only activation of the plugin and settings is required, as it is done through the blender interface
• Full unloading of the client immediately after the return of work
• Scalability up to the number of servers equal to the number of frames
• Identical to the local render - there is no need to configure something with an eye to work on the network
• Theoretical compatibility between all systems. - You can easily transfer from windows to linux and send the work, while one of the slaves from Linux turned into a mac (by the way, this is the merit of the python on which plug-ins for the blender are written)
• Promising collective opportunities - theoretically, an unlimited number of clients can send work to one master, the master will distribute them among available slaves
• Priority tasks - upon receiving a small work, the master will give it to one of the slaves as soon as he finishes the portion of frames (chunk-size), even if his queue is not empty
• We take only the frames we need as they are ready from any browser or blender
• The ability to pick up only .jpg frames without loading the channel, leaving heavy .exr files on the server

Minuses

• Some difficulties when working on the Internet - if the connection with the master is interrupted, you have to start the transfer more times, while removing unsuccessful attempts
• Arbitrary timeouts - (in recent versions they are much less)
• There is no possibility of several slave machines working on one frame
• Nuances with external files - (for example, textures) when working together on different operating systems. There have been annoying errors when connecting Linux to Win and the Mac.
• Some functions do not work yet - (from the examples you can mark the video instead of the texture)
• Before the start of the render, the slave will be idle for some time, doing preprocessing (identical to the usual render, this is a minus only when comparing with the lux renderer)
• When the wizard stops, all the slave servers instantly fall off and you have to lift them by hand, this is all the more strange if you consider that when you raise the slave, he will dutifully wait for the connection, regardless of whether he finds the wizard immediately or not
• Any change in the settings of the master or slave servers entails pre-setting these in the blender and then copying the finished .blend files to each server
• If the master server process crashes, we will lose the last job; it is necessary to correctly complete the wizard so that you can continue rendering from the same place or pick up the result without directly interfering with the file structure (all finished frames in the .exr format are likely to be in place)

Lux-renderer

Luxrender is of interest in this review only for comparing its capabilities with the capabilities of the internal network renderer in the blender, its advantages and disadvantages are where to read.
Starting with version 0.8, the suite has the opportunity to use the power of modern video cards to calculate light intersections through OpenCL.
The network render in the suite is moving under the slogan “CPU vs GPU? CPU + GPU + network rendering is better, "in the future, developers promise to expand the use of video cards in rendering, for the time being one should not hope for a miracle, the video card is unlikely to load more than 30%, although a certain performance increase can already be observed now.
Unlike the internal network render, Lux can combine several servers to work on a single frame, a master server is not provided, its functions are partially performed by the client. The whole system is reduced to one client and several slave servers, as soon as after using the LuxBlend plugin the work gets to the slave, it calculates several samples. The result will be transferred back to the client after a certain period of time, there it intersects with the result of the local render, and at the output we will get a picture with more samples than if the slave did not participate in the process.
Tools: One client and one server, while they work in pairs and give a theoretically linear increase in performance.



pros

• Combining any number of slave machines to work on one frame
• Visual representation of the degree of completion of the overall work on the frame
• Ability to alternate rendering of multiple frames for animation
• The ability to use the power of not only the CPU, but also the video card of each slave
• If necessary, you can save the result and continue rendering later

Minuses

• The client cannot be unloaded, as soon as it is stopped, the lux server will clear the queue and go into waiting for a new client
• Selfish teamwork system “who first got up and slippers”, two clients cannot connect to one slave
• With a large number of servers, adding them to the list can be compared with the operation of the hosts file, which is at least inconvenient
• If, after waiting for the end of work, after pressing the quit button to close the client, a specially loaded slave may fall off, it will be necessary to restart it manually, otherwise it will continue to wait for a nonexistent client

Installation

In the expectation that the network render of the luxury will be used in some cases when it is necessary to get a picture with ray tracing, and in the rest, the internal network render, we can put both.

Installing the Luxrendrader

First we go to the site and download the tip.zip file, this is the version under development, if you have a blender version newer than 2.58, it is better to use it, rather than a stable assembly. Next, unpack the archive into the plugins folder and start the blender, activate the suite in the plugins menu. We will need the suite itself, which can be taken here , it seems to me that for our purposes it’s better not Installer, but Archive, but it’s to your taste.
Those who have already dealt with this render just customize everything for themselves, not forgetting that OpenCL support is available only when choosing: “Render: Hybrid (CPU + GPU)”, “Surface integrator: Path” and in “Advanced > Light strategy: one ”, otherwise select options as you wish and leave“ Sampler: Traditional CPU ”. Next, we set the path to our render in “Path to LuXrender installation”, and we can assume that half the battle is done.
Server installation is described in detail on the wiki pages , the main thing is to open the corresponding port on the server for external access, in principle, the following command is enough: luxconsole -s -p 10001In this case, the suite will listen on TCP port 10001 (-p) in server mode (-s), and we and more is not necessary.
Returning to the client, we configure LuxRender networking, adding our server to the list, for example: hostname: 10001 or IP: port.
When you press F12, a luxury window should appear and in the Network tab we will see our server. In the Log tab, there should be no errors related to the connection.
We check the server load, if we see the processors at 100%, it means, as the saying goes, “water has gone to the house” and the wizard will begin to give samples after a period of time specified in the “Upload interval” option of the blender. If the connection has not been established, you can find out about this from the logs, most likely the port on the wizard did not open, we can check by running telnet on it and write something in the console, you should see the typed text on the server.

In short, the log contains entries similar to:

[192.168.2.3:10001] 00:00:36 - 6T: 3.67 S / p 78.79 kS / s 99% Eff 77.86 kC / s - Using CPU
Server: port, time (during which the server does work), number of cores / trides, the number of samples per pixel, the number of samples per second (in thousands), the percentage of frame efficiency (the average value between the number of samples per second and efficiency), the latter means whether it is used on the OpenCL server for rendering.

Here is another example:

[192.168.2.4:10001] 00:06:38 - 4T: 25.92 S / p 51.20 kS / s 99% Eff 50.51 kC / s - Using CPU We
translate:
After 6:38 minutes of rendering, the server 192.168.2.4 on port 10001 managed to get 22.92 samples per pixel at a speed of 51.20 thousand samples per second and 99% efficiency, its work is estimated at 50.51 thousand contributions per second using only the central processor.

Installing an internal network render.

Everything necessary for work is included in the blender, activate the netrender plugin in the settings, select netrender in the appropriate section and proceed with the configuration.
First, it’s easier to configure the wizard, select the port, leave servername empty so that you can also accept requests on new interfaces without having to configure everything again.
It is better to give a temporary folder in Linux a separate partition on the hard drive, the main thing is not to forget to create it, otherwise the server will not start. We leave Broadcast enabled if the master is on the local network, then the client will be able to find it himself, however, for this you will need to open another UDP port (8000). I do not recommend Clear on exit, it is better to be able to resume work, and saving on free space is not our task.
Further, graphics users can safely click "Start service" and continue to configure the slave server.
The rest saves a .blend file called master.blend, this is the configuration for our environment, it will be useful to us if the server does not have a graphical interface and there is no way to configure it. We’ll give the finished file to the server, we also need the startup.blend file, the user profile is saved in it, without it the blender just won’t understand that it needs to activate the network render plug-in, copy it to the user directory: (.blender / blender version / config / startup.blend), then we can start the server itself from the blender folder: (./ blender -b master.blend -a)
This means just “start the blender (./blender), read the file (-b) and render all frames (-a). The nettrader himself is responsible for the last parameter, (we write only for the start, and this does not mean that all frames will always be rendered.
After this, a legend should appear that the server is waiting for work, check by going to our ip: our port is the web interface of the wizard, through it we can manage the work and monitor their progress, it will also allow us to quickly pick up the finished .jpg files , but more on that later.
So, we start the blender, configure it to work as Client in the network render settings, set the IP and port, or click on the button with two arrows if we specified Broadcast on the wizard. As soon as we connect, it will be possible to send work, in the browser window we will see that it has been assigned a number and that it is in the Queued state - standby mode, in which rendering does not occur, since the wizard is not suitable for this purpose, and we still not configured. Well, let's fix this omission.

Slave Setup

In the graphical interface, everything is simple, select Slave in the options of the netrender and find the master. The number of threads should be equal to the number of processor cores (intel owners can experiment). We leave Auto-detect with the exception of cases when a slave needs to do something other than rendering, since in this case we get a full load of all the cores. Thumbnails is a separate topic, the fact is that if we have several slave machines, in case of a crash it is more convenient to collect information from one place rather than going to each server, it is better to leave this option turned off, Clear on exit in this case we can activate , since filling the hard drive on each slave with duplicates is a dubious occupation.
If you need to run a slave on a server without a graphical interface, just save the .blend file and upload it to the server, the launch method is identical to the wizard, it’s easier not to get confused by using the name say slave.blend.

Trial run

After connecting to the master, our slave should accept the job and start rendering, through the number of frames specified by the “chunk-size” parameter in the client, it will give the finished part of the work to the master and you can look at the result via the web, or load it into the blender with the button “Get frame”. There are many programs that allow you to download a list of files from the server, to the flashget under windows or wget under linux, for the latter command loads the entire list:
wget сервер:порт/thumb_номер работы_{первый кадр..последний кадр}.jpg
for example:
wget 192.168.2.3:10001/thumb_1_{1..20}.jpg
Via flashget: Right button on the icon> Add new> Add batch download> http://192.168.2.3:10001/thumb_1_ (*) ​​.jpg, then set the condition for loading frames below.

Additional settings

In case of insufficient memory on the slave server, by increasing the number of “pieces” of the frame in “Render> Performance> Tiles”, it is possible to free up some RAM by sacrificing increased downtime when loading the processor thread.
If the geometry on the stage or the speed of the server does not allow rendering in less than five minutes, most likely the wizard and server slaves will be disconnected.
To solve this problem, inside the file "blender-version-linux-glibc27-x86_64 / version / scripts / addons / netrender / master.py" we find the variable "self.slave_timeout" and assign a value equal to the number of minutes that we expect the render will take " chink-size “frames for each slave (we focus naturally on the weak link).
If you need to look at the general statistics of work, you can use this script (only under * NIX):
blenderartists.org/forum/showthread.php?233628-Netrender-AWK-statistics-script

That's all, beautiful renderings to all.

Useful links.

Official documentation of network rendering.
Note on support for network rendering in Cycles.