August 3, 2013 at 00:36Implementing a Dynamic Download Content System (DLC) for a mobile game in Unity 3D
Recently, for one game on Unity 3D that we were developing, it became necessary to add a DLC system . Although this turned out to be far from as simple as it seemed at the beginning, we successfully coped with the problems that arose and the game went into gold. In this article I want to present our version of the DLC implementation, talk about the problems that have arisen and how we solved them.
The game has a store where a player buys things for game or real currency. The store has over 200 items. When a player enters the game, he has 20 items available in the store. If there is Internet, the game without the knowledge of the user polls the server for DLC and, if any, downloads it in the background. When the player re-enters the store, he will see all the new things from the DLC.
There is still a set of locations. Each location has a set of textures and .asset files. New locations should also be added through the DLC.
Resource loading from the DLC must be synchronous.
Platform: iOS (iPhone 3GS and above.) And Android (Samsung Galaxy S and above).
In the game, things are completely determined by the itemdata.txt file, which contains information about things and their textures. This means that in each DLC there will be an itemdata.txt file with a set of those things that are in the DLC + test for these things. And when the store requests a database of things, we glue all the text files from all the DLCs and give it this file.
Similarly for locations there is a locationdata.txt file with a list and characteristics of locations + textures and asset files for them.
The corresponding C # code for loading resources in the game logic will look like this:
Similarly, when requesting a texture, we check for its presence in the DLC. If it is there, load, otherwise load from game resources. If not there, then load something defaulted.
Similarly for .asset files there will be a GetAsset (string assetName) function. Its implementation will be similar, so skip it.
We decided what should be in the DLC. It remains to decide in what form it is all stored.
The first option is to store the DLC as a zip archive. In each archive - a text file + N textures. Textures must be in PVRTC format to save video memory. But here we have the first problem - Unity only supports downloading textures from the file system in PNG or JPG format [ link ]. Then the texture can be written to the PVRTC texture [ link ]. This is a slow process because requires conversion to PVR in real time. In addition, because In the DLC, it is planned to store files of the type .asset, and possibly game levels (.scene), such a method is completely unsuitable.
The second option is to use AssetBundle. This solution is ideal for DLC in games.
Judging by the documentation, it has a lot of advantages:
Of the minuses, only that AssetBundle requires a Pro version of Unity and does not support encryption . We decided to dwell on this decision, because it is obviously more attractive and allows us to solve all our problems.
To begin with, a test version of the DLC system with the most basic functionality was made.
First, all 200-odd textures of store items and location files were packed into one AssetBundle and uploaded to the server. The file turned out to be about 200 mb. Packaging in AssetBundle was performed by a script in the editor. How to make resource packaging in AssetBundle is well described in the documentation. You can also use my script to create an AssetBundle .
Next, after starting the game, we take the following steps:
The above code is so far the only possible way to load a resource synchronously from AssetBundle. And as it turned out, there are a lot of nuances. Let's sort them in order.
The function
The reader may wonder why not load the AssetBundle asynchronously, for example, with the LoadFromCacheOrDownload function, and then simply take the necessary resources from it synchronously. After all, it is logical that AssetBundle, when loading from the file system, should load only the file header, and therefore it should be occupied in memory a little.
However, this was not the case. The loaded AssetBundle is stored in memory completely with all its contents unpacked. Thus, in order to load one texture from 200, Unity will load all 200 textures into memory, take one, and then free up memory for the remaining 199 textures. We found this out experimentally by measuring memory on the device.
Obviously, this is not acceptable for mobile devices.
The given option is the only way we found to implement synchronous loading of DLC and resources from it.
An uncompressed AsssetBundle is required, resulting in large losses of time and traffic when loading the DLC.
The option is suitable for relatively small AssetBundles, as consumes a lot of RAM.
Let's try to take into account all the previous problems and find solutions for them.
The problem with loading large assetBundles can be solved in two ways.
The first is to use the WebClient class. However, we had problems with it on iOS. WebClient could not download anything, but it worked perfectly on the desktop.
The second option is to use native OS functions. For example, NSURLConnection for iOS and URLConnection for Android respectively, which support buffered loading directly to a file on disk.
But this is not such a big problem, because in any case, we need to reduce the size of the AssetBundle for synchronous loading. Therefore, for now, we have left the current way to download bundles from the server.
A much more serious problem is the synchronous loading of the AssetBundle. Because it should not only be uncompressed, but also take up little memory space, we must somehow split our one large DLC file into many small files. However, if we split into too small files, there will be many of them and this will greatly increase the download time, because You will have to reconnect for each file. So, we still have to keep them compressed to better save load time and traffic.
To solve this problem, it was decided to use its own archiver. An open archiver library for C # was chosen, which, with little effort, turned out to be built under Mono in Unity.
Further, the algorithm of actions was as follows:
Here we had another problem. Because we now use the bundle compressed by the archiver, we can no longer deflate it with the LoadFromCacheOrDownload function. So, now we have to define our own version control system for the DLC.
The following solution was chosen for the DLC version control system. On the server, in the folder where the DLC files were located, a dlcversion text file was created. It contained a list of DLCs in the folder and md5 hashes for them. These hashes were considered at the stage of the DLC applet to the server. The client had the exact same file, and when the application started, the client compared its file with the file on the server. If some DLC file had excellent hashes or there was no hash at all, it was believed that the file on the client was out of date and the client pulled a new DLC file from the server.
After the new DLC file was downloaded and unpacked, its hash was checked again with the server one, and only after that the obsolete file was replaced with a new one and the corresponding entry was made in the client dlcversion file.
The described system was successfully implemented and works fine. The only minus that we had was a small drawdown in fps (lags) when downloading and unpacking the DLC in the background. And also the peak values of application memory consumption increased slightly.
Thanks for attention. I will be glad to answer your questions.
Formulation of the problem
The game has a store where a player buys things for game or real currency. The store has over 200 items. When a player enters the game, he has 20 items available in the store. If there is Internet, the game without the knowledge of the user polls the server for DLC and, if any, downloads it in the background. When the player re-enters the store, he will see all the new things from the DLC.
There is still a set of locations. Each location has a set of textures and .asset files. New locations should also be added through the DLC.
Resource loading from the DLC must be synchronous.
Platform: iOS (iPhone 3GS and above.) And Android (Samsung Galaxy S and above).
DLC content and work with it in the game
In the game, things are completely determined by the itemdata.txt file, which contains information about things and their textures. This means that in each DLC there will be an itemdata.txt file with a set of those things that are in the DLC + test for these things. And when the store requests a database of things, we glue all the text files from all the DLCs and give it this file.
Similarly for locations there is a locationdata.txt file with a list and characteristics of locations + textures and asset files for them.
The corresponding C # code for loading resources in the game logic will look like this:
public String GetItemDataBase() {
if(DLCManager.isDLCLoaded() == true) {
//склеить все файлы itemdata.txt во всех загруженных DLC и вернуть как один string
String itemListStr = DLCManager.GetTextFileFromAllDLCs(“itemdata”);
return itemListStr;
}
else {
//загружаем файл по умолчанию
TextAsset itemTextFile = Resources.Load(“itemdata”) as TextAsset;
return itemTextFile.text;
}
return String.Empty;
}
Similarly, when requesting a texture, we check for its presence in the DLC. If it is there, load, otherwise load from game resources. If not there, then load something defaulted.
public Texture GetTexture(string txname) {
Texture tx = null;
if(DLCManager.isDLCLoaded() == true) {
tx = DLCManager.GetTextureFromDLC(txname);
}
if(tx == null) {
tx = Resources.Load(txname) as Texture;
}
if(tx == null) {
Assert(tx, “Texture not find: ” + txname);
tx = Resources.Load(kDefaultItemTexturePath) as Texture;
}
return tx;
}
Similarly for .asset files there will be a GetAsset (string assetName) function. Its implementation will be similar, so skip it.
DLC file
We decided what should be in the DLC. It remains to decide in what form it is all stored.
The first option is to store the DLC as a zip archive. In each archive - a text file + N textures. Textures must be in PVRTC format to save video memory. But here we have the first problem - Unity only supports downloading textures from the file system in PNG or JPG format [ link ]. Then the texture can be written to the PVRTC texture [ link ]. This is a slow process because requires conversion to PVR in real time. In addition, because In the DLC, it is planned to store files of the type .asset, and possibly game levels (.scene), such a method is completely unsuitable.
The second option is to use AssetBundle. This solution is ideal for DLC in games.
Judging by the documentation, it has a lot of advantages:
- It can store any Unity resources, including those compressed into the desired texture format (what we need).
- This is an archive with good compression.
- Simple and convenient to use.
- It supports the version parameter and the hash amount (when loaded by the LoadFromCacheOrDownload function), which is convenient for version control of the DLC
Of the minuses, only that AssetBundle requires a Pro version of Unity and does not support encryption . We decided to dwell on this decision, because it is obviously more attractive and allows us to solve all our problems.
Implementation (Option 1)
To begin with, a test version of the DLC system with the most basic functionality was made.
First, all 200-odd textures of store items and location files were packed into one AssetBundle and uploaded to the server. The file turned out to be about 200 mb. Packaging in AssetBundle was performed by a script in the editor. How to make resource packaging in AssetBundle is well described in the documentation. You can also use my script to create an AssetBundle .
Next, after starting the game, we take the following steps:
- First you need to download the DLC from the server. We do this according to the code from the Unity manual. Next, we write the downloaded data to a file on disk for future use.
// Start a download of the given URL using assetBundle version and CRC-32 Checksum WWW www = WWW.LoadFromCacheOrDownload (urlToAssetBundle, version, crc32Checksum); // Wait for download to complete yield return www; // Get the byte data byte[] byteData = www.bytes; // Тут можно вставить свой метод дешифровки бандла, если необходимо byteData = MyDescriptionMethod(byteData); //сохраняем byteData в файл с расширением .unity3d ... // Frees the memory from the web stream www.Dispose(); //DLC успешно загружено и его можно использовать в игре DLCManager.SetDLCLoaded(true);
With this code, we are very likely to get memory crashes on low devices like iPhone 3GS, because The WWW class does not support buffered loading and stores all loaded information in memory. We will talk about this problem a bit later. While we remember this moment and move on. - Loading resources from the DLC.
Now we need to define the GetTextureFromDLC (), GetAssetFromDLC () and GetTextFileFromAllDLCs () functions. We will omit the definition of the latter, because it will hardly differ from the first one except for the type of the loaded resource.
The main task of the GetTextureFromDLC function is to synchronously load a texture by name from the DLC.
Let's try to define it as follows.public Texture GetTextureFromDLC(String textureName) { //загружаем DLC с диска. Можем использовать только синхронный метод. AssetBundle asset = AssetBundle.CreateFromFile(pathToAssetBundle); //синхронная загрузка текстуры из DLC Texture texture = asset.Load(textureName) as Texture; //выгрузка бандла из памяти без удаления объекта texture asset.Unload(false); return texture; }
The above code is so far the only possible way to load a resource synchronously from AssetBundle. And as it turned out, there are a lot of nuances. Let's sort them in order.
The function
AssetBundle.CreateFromFileaccording to the documentation synchronously downloads assets from disk. But there is one caveat - “Only uncompressed asset bundles are supported by this function.” Thus, only an uncompressed AssetBundle can be loaded synchronously. Which will significantly increase traffic and DLC boot time from the server. In addition, Unity does not support converting AssetBundle from compressed to uncompressed, so it will not work to download a compressed bundle, and then unpack it on the client.The reader may wonder why not load the AssetBundle asynchronously, for example, with the LoadFromCacheOrDownload function, and then simply take the necessary resources from it synchronously. After all, it is logical that AssetBundle, when loading from the file system, should load only the file header, and therefore it should be occupied in memory a little.
However, this was not the case. The loaded AssetBundle is stored in memory completely with all its contents unpacked. Thus, in order to load one texture from 200, Unity will load all 200 textures into memory, take one, and then free up memory for the remaining 199 textures. We found this out experimentally by measuring memory on the device.
Obviously, this is not acceptable for mobile devices.
Summary
The given option is the only way we found to implement synchronous loading of DLC and resources from it.
An uncompressed AsssetBundle is required, resulting in large losses of time and traffic when loading the DLC.
The option is suitable for relatively small AssetBundles, as consumes a lot of RAM.
Work on bugs (Option 2)
Let's try to take into account all the previous problems and find solutions for them.
The problem with loading large assetBundles can be solved in two ways.
The first is to use the WebClient class. However, we had problems with it on iOS. WebClient could not download anything, but it worked perfectly on the desktop.
The second option is to use native OS functions. For example, NSURLConnection for iOS and URLConnection for Android respectively, which support buffered loading directly to a file on disk.
But this is not such a big problem, because in any case, we need to reduce the size of the AssetBundle for synchronous loading. Therefore, for now, we have left the current way to download bundles from the server.
A much more serious problem is the synchronous loading of the AssetBundle. Because it should not only be uncompressed, but also take up little memory space, we must somehow split our one large DLC file into many small files. However, if we split into too small files, there will be many of them and this will greatly increase the download time, because You will have to reconnect for each file. So, we still have to keep them compressed to better save load time and traffic.
To solve this problem, it was decided to use its own archiver. An open archiver library for C # was chosen, which, with little effort, turned out to be built under Mono in Unity.
Further, the algorithm of actions was as follows:
- When creating a bundle, the BuildOptions.UncompressedAssetBundle option was specified to get an uncompressed bundle.
- Then the bundle was archived and encrypted by the archiver and uploaded to the server.
- While the application was running, a separate stream was created, which in the background pumped out bundles, unpacked them and put them in a special folder.
Here we had another problem. Because we now use the bundle compressed by the archiver, we can no longer deflate it with the LoadFromCacheOrDownload function. So, now we have to define our own version control system for the DLC.
The following solution was chosen for the DLC version control system. On the server, in the folder where the DLC files were located, a dlcversion text file was created. It contained a list of DLCs in the folder and md5 hashes for them. These hashes were considered at the stage of the DLC applet to the server. The client had the exact same file, and when the application started, the client compared its file with the file on the server. If some DLC file had excellent hashes or there was no hash at all, it was believed that the file on the client was out of date and the client pulled a new DLC file from the server.
After the new DLC file was downloaded and unpacked, its hash was checked again with the server one, and only after that the obsolete file was replaced with a new one and the corresponding entry was made in the client dlcversion file.
The described system was successfully implemented and works fine. The only minus that we had was a small drawdown in fps (lags) when downloading and unpacking the DLC in the background. And also the peak values of application memory consumption increased slightly.
Thanks for attention. I will be glad to answer your questions.
