Fast deserialization of really big JSON responses

Under the cutter is a small but useful description of how to quickly and easily turn a received JSON response into a set of objects. No manual parsing. And if you encountered OutOfMemory problem on old smartphones - and for this there is a solution that supports Android 2.X versions.

In addition, under the cut will be a link to the github repository with sample code. But there will be no pictures, but there is a place for a small tablet.

So, on the current project, I had to parse the response of the service, consisting of a stack of objects nested inside each other, inside which there could be objects inside which ... The data was in JSON format, in addition, gzip-compression by the server was used, nevertheless the difference in the size of the transmitted data was significant (4 megabytes versus 300 kilobytes in compressed form - this is not a joke for mobile communications).

As a lazy person, I couldn’t hand each field and object with my hands ... Thus, the Gson library was involved, judging by the test - the fastest deserializer from the JSON format. Well, now, let's get started, and start right away with the code. For simplicity, we’ll conduct all the output to the console, so as not to think about the views and more.

This is what the objects that fly to us from the network look like:

public class HumorItem {
    public String text;
    public String url;
}
public class HumorItems {
    List Items; //тут может быть больше списков, и не только списки, для примера упростим.
}

And so - the code that downloads and deserializes it.
First version of the code
public class LoadData extends AsyncTask {
	String _url="";
	public LoadData(String url){
		_url=url;
	}
	@Override
	protected Void doInBackground(Void... voids) {
		try {
			//скачивание данных
			HttpClient httpclient = new DefaultHttpClient();
			HttpPost httppost = new HttpPost(_url);
			HttpResponse response = httpclient.execute(httppost);
			HttpEntity httpEntity=response.getEntity();
			InputStream stream = AndroidHttpClient.getUngzippedContent(httpEntity); //для скачивания gzip-нутых данных
			BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(stream));
			StringBuilder responseBuilder= new StringBuilder();
			char[] buff = new char[1024*512];
			int read;
			while((read = bufferedReader.read(buff)) != -1) {
				responseBuilder.append(buff, 0, read) ;
				Log.d("скачано " + PrepareSize(responseBuilder.length()));
			}
			//парсинг полученных данных
			HumorItems list= Gson.fromJson(responseBuilder.toString(),HumorItems.class);
			//тестовый вывод
			for (HumorItem message:list.Items){
				Log.d("Текст: "+message.text);
				Log.d("Ссылка: "+message.url);
				Log.d("-------------------");
			}
			Log.d("ВСЕГО СКАЧАНО "+list.Items.size());
		} catch (IOException e) {
			e.printStackTrace();
			Log.e("ошибка "+e.getMessage());
		}
		return null;
	}
}

Log Wrapper and File Size Method
public class Log {
    public static final String TAG="hhh";
    public static void d(String text){
        android.util.Log.d(TAG,text);
    }
    public static void e(String text){
        android.util.Log.e(TAG,text);
    }
}
    public String PrepareSize(long size){
        if (size<1024){
            return size+" б.";
        }else
        {
            return size/1024+" кб.";
        }
    }

And this solution worked great! For the time being. The answer for one of the combination of parameters weighed about 8 megabytes. When testing on parts of phones - the program crashed, where on the fifth megabyte downloaded, where on the third.

Google first suggested a simple solution - put largeHeap in the AndroidManifest file.


This option allows the application to allocate more RAM for itself. The option is certainly lazy and simple, but Android phones below the 3rd version are not supported. And in general, some defeatist approach - “why optimize if you can buy more iron?”

Next, after several attempts, this simple option was chosen:
  • We do not fill the variable with the file, no - download the data directly to the USB flash drive (well, or the internal memory, which turns up by the hand).
  • We set Gson on this file. The problem in parsing and the memory occupied by the file does not occur.

No sooner said than done:
The second version of the code is with a temporary file
public class LoadBigDataTmpFile extends AsyncTask {
        String _url="";
        File cache_dir;
        public LoadBigDataTmpFile(String url){
            _url=url;
            cache_dir = getExternalCacheDir();
        }
        @Override
        protected Void doInBackground(Void... voids) {
            try {
                //скачивание данных
                HttpClient httpclient = new DefaultHttpClient();
                HttpPost httppost = new HttpPost(_url);
                HttpResponse response = httpclient.execute(httppost);	
                HttpEntity httpEntity=response.getEntity();
                InputStream stream = AndroidHttpClient.getUngzippedContent(httpEntity);
                //нечто новое - открываем временный файл для записи
                BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(stream));
                File file = new File(cache_dir, "temp_json_new.json");
                if (file.exists()){ //если таковой уже есть - удаляем и создаём новый
                    file.delete();
                }
                file.createNewFile();
                FileOutputStream fileOutputStream=new FileOutputStream(file,true);
                BufferedWriter bufferedWriter=new BufferedWriter(new OutputStreamWriter(fileOutputStream));
                char[] buff = new char[1024*1024];
                int read;
                long FullSize=0;
                while((read = bufferedReader.read(buff)) != -1) {
                    bufferedWriter.write(buff,0,read);	//запись в файл
                    FullSize+=read;
                    Log.d("скачано " + PrepareSize(FullSize));
                }
                bufferedWriter.flush();
                fileOutputStream.close();
                //парсинг из файла
                Log.d("начали парсинг...");
                FileInputStream fileInputStream=new FileInputStream(file);
                InputStreamReader reader = new InputStreamReader(fileInputStream);
                HumorItems list= Gson.fromJson(reader,HumorItems.class);
                Log.d("закончили парсинг.");
                /тестовый вывод
                for (HumorItem message:list.Items){
                                Log.d("Текст: "+message.text);
                                Log.d("Ссылка: "+message.url);
                                Log.d("-------------------");
                }
                Log.d("ВСЕГО СКАЧАНО "+list.Items.size());
            } catch (IOException e) {
                e.printStackTrace();
                Log.e("ошибка "+e.getMessage());
            }
            return null;
        }
    }

That’s all. The code is tested in combat conditions, it works stably with a bang. However, you can make it even easier and do without a temporary file.
Third code option - no temporary file
public class LoadBigData extends AsyncTask {
	String _url="";
	public LoadBigData(String url){
		_url=url;
	}
	@Override
	protected Void doInBackground(Void... voids) {
		try {
			//скачивание данных
			HttpClient httpclient = new DefaultHttpClient();
			HttpPost httppost = new HttpPost(_url);
			HttpResponse response = httpclient.execute(httppost);
			HttpEntity httpEntity=response.getEntity();
			InputStream stream = AndroidHttpClient.getUngzippedContent(httpEntity);
			//открывам потом на чтение данных
			BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(stream));
			//и сразу направляем его в десериализатор
			InputStreamReader reader = new InputStreamReader(stream);
			HumorItems list= Gson.fromJson(reader,HumorItems.class);
			//тестовый вывод
			for (HumorItem message:list.Items){
				Log.d("Текст: "+message.text);
				Log.d("Ссылка: "+message.url);
				Log.d("-------------------");
			}
			Log.d("ВСЕГО СКАЧАНО "+list.Items.size());
		} catch (IOException e) {
			e.printStackTrace();
			Log.e("ошибка "+e.getMessage());
		}
		return null;
	}
}

Minus - it will not be possible to control the download process (interrupt it in an adequate way), and also it is not known how much data has already been downloaded. You can’t draw a beautiful progress bar.

There is one more option, given in the documentation , which allows you to sequentially pull out objects and immediately process them, but it is problematic to work with it if you have an object of different arrays of objects, and not just an array of the same type. However, if you have a beautiful solution, I will gladly see it in the comments, and I will definitely include it in the article in update!

As a bonus, some statistics.
file sizeThe number of objects insideEmulator deserialization timeHighscreen Boost Deserialization Time
5.79 MB400035 seconds2 seconds
13.3 MB90001 minute 11 seconds5 seconds

An example of use is on a github , test files are in the same place.
Link to the Gson library .

If anyone will be interested in the development topic for android, then at least posts on push notifications (server and client side) - there were articles on this topic on the hub, but they are all outdated), about working with the database and others on the topic of development for Android .

Update The github showed a solution to the problem: “Minus - it will not be possible to control the download process (interrupt it in an adequate way), and it is also not known how much data has already been downloaded. You cannot draw a beautiful progress bar. ” Details are in the last commit in the repository.

Also popular now: