The problem with reading files larger than 1MB in Android
Good afternoon.
When writing a single Android application, I encountered the problem of downloading files from the assets directory whose size exceeds 1 MB.
Suppose, in the project folder, in assets lies the file.dat file , which must be loaded into memory and read data from it.
We open the file and read the first 8 bytes in buffer. As a result of executing the code on Android 2.2 and lower, we get a message in LogCat 06-14 17: 10: 48.554: DEBUG / asset (382): Data exceeds UNCOMPRESS_DATA_MAX (1048852 vs 1048576) and IOException 06-14 17: 10: 48.654: WARN / System.err (698): java.io.IOException We look at the source code of the android. In file
frameworks / base / include / utils / Asset.h is the same constant UNCOMPRESS_DATA_MAX And in the frameworks / base / include / utils / Asset.cpp file is checked for file size As you can see from the comment, data exceeding the size of UNCOMPRESS_DATA_MAX in bytes is not unpacked to the buffer from the .apk file. There are several ways to solve the problem: 1. The simplest is to change the file.dat file extension to such that this file will not be compressed when packed into an .apk file. The list of these extensions is defined in Package.cpp Or, you can manually compile the project using aapt
with the option "-0", specifying file extensions that do not need to be compressed in .apk . If you use an empty line, instead of the extension, not a single file will be compressed.
This method is bad in that the size of the installation file downloaded by the user from the market grows significantly (by the difference between compressed and uncompressed file.dat ), and the file size is quite critical for mobile devices. In my case, the text information used in the application is stored in file.dat , in unpacked form it takes a little more than 1MB, and in the .apk file it is compressed to 40Kb.
If your assets files do not compress well in zip (media), this method is great.
2.The second way is to split file.dat into 1MB pieces, for example from the console:
The resulting files must be placed in assets , the first time you start the application, “glue” it into pieces and put in the directory with the application. This method is well described on stackoverflow, it was proposed by a person named Seva Alekseyev, so I will only give the link stackoverflow.com/questions/2860157/load-files-bigger-than-1m-from-assets-folder/3093966#3093966
In this case, your application will store two copies of the data, some in the .apk file, others uploaded to the application directory. It also adds a little extra code and the need to monitor the existence and integrity of the "decompressed" data.
There is no described problem for Android 2.3.1 and higher.
Of course, you can think of something else, for example, downloading files over the network (the so-called cache), which is not always reliable if the user has turned off the Internet, or generating a file from scratch, but the two methods described above were enough for me.
When writing a single Android application, I encountered the problem of downloading files from the assets directory whose size exceeds 1 MB.
Suppose, in the project folder, in assets lies the file.dat file , which must be loaded into memory and read data from it.
We open the file and read the first 8 bytes in buffer. As a result of executing the code on Android 2.2 and lower, we get a message in LogCat 06-14 17: 10: 48.554: DEBUG / asset (382): Data exceeds UNCOMPRESS_DATA_MAX (1048852 vs 1048576) and IOException 06-14 17: 10: 48.654: WARN / System.err (698): java.io.IOException We look at the source code of the android. In file
AssetManager am = context.getAssets();
InputStream is = am.open("file.dat", AssetManager.ACCESS_BUFFER);
// Read file
is.read(buffer, 0, 8);frameworks / base / include / utils / Asset.h is the same constant UNCOMPRESS_DATA_MAX And in the frameworks / base / include / utils / Asset.cpp file is checked for file size As you can see from the comment, data exceeding the size of UNCOMPRESS_DATA_MAX in bytes is not unpacked to the buffer from the .apk file. There are several ways to solve the problem: 1. The simplest is to change the file.dat file extension to such that this file will not be compressed when packed into an .apk file. The list of these extensions is defined in Package.cpp Or, you can manually compile the project using aapt
enum {
/* data larger than this does not get uncompressed into a buffer */
#ifdef HAVE_ANDROID_OS
UNCOMPRESS_DATA_MAX = 1 * 1024 * 1024
#else
UNCOMPRESS_DATA_MAX = 2 * 1024 * 1024
#endif
}; if (mUncompressedLen > UNCOMPRESS_DATA_MAX) {
LOGD("Data exceeds UNCOMPRESS_DATA_MAX (%ld vs %d)\n",
(long) mUncompressedLen, UNCOMPRESS_DATA_MAX);
goto bail;
}/* these formats are already compressed, or don't compress well */
static const char* kNoCompressExt[] = {
".jpg", ".jpeg", ".png", ".gif",
".wav", ".mp2", ".mp3", ".ogg", ".aac",
".mpg", ".mpeg", ".mid", ".midi", ".smf", ".jet",
".rtttl", ".imy", ".xmf", ".mp4", ".m4a",
".m4v", ".3gp", ".3gpp", ".3g2", ".3gpp2",
".amr", ".awb", ".wma", ".wmv"
};with the option "-0", specifying file extensions that do not need to be compressed in .apk . If you use an empty line, instead of the extension, not a single file will be compressed.
This method is bad in that the size of the installation file downloaded by the user from the market grows significantly (by the difference between compressed and uncompressed file.dat ), and the file size is quite critical for mobile devices. In my case, the text information used in the application is stored in file.dat , in unpacked form it takes a little more than 1MB, and in the .apk file it is compressed to 40Kb.
If your assets files do not compress well in zip (media), this method is great.
2.The second way is to split file.dat into 1MB pieces, for example from the console:
split file.dat --bytes=1024KThe resulting files must be placed in assets , the first time you start the application, “glue” it into pieces and put in the directory with the application. This method is well described on stackoverflow, it was proposed by a person named Seva Alekseyev, so I will only give the link stackoverflow.com/questions/2860157/load-files-bigger-than-1m-from-assets-folder/3093966#3093966
In this case, your application will store two copies of the data, some in the .apk file, others uploaded to the application directory. It also adds a little extra code and the need to monitor the existence and integrity of the "decompressed" data.
There is no described problem for Android 2.3.1 and higher.
Of course, you can think of something else, for example, downloading files over the network (the so-called cache), which is not always reliable if the user has turned off the Internet, or generating a file from scratch, but the two methods described above were enough for me.