August 25, 2016 at 15:16What Google is silent about and why you should use Apache HttpComponents in Android
This article had to be published much earlier (almost six years), thereby saving Android developers a huge number of months of meaningless development - but alas, there is not always time for this.
If you are developing for Android, then you probably faced the fact that you open your application, which worked fine for several years, and then it suddenly turns out that Apache httpComponents became deprecated, and they are not recommended for use. First, let's look at what happened, and then draw conclusions what to do.
I did not dig too far, but many interesting things can be obtained from the org.apache.hc.dev mailing list and Apache jira. For example, the fact that:
Moreover, throughout the history the version of the library has not changed, and if you turn on apache legacy, then this is still the same pre-beta.
Confirmation can be read here .
Apache developers were actively working on the library, but the Google managers they appointed did not answer for months, changed, and in the end said that this was not a priority, users had the old version, and in general, they had to throw this library out. In confirmation - a bit from the mailing list mentioned by me (work began in 2008, and in 2010 a new manager suddenly appeared):
Then the first alarm bell rang:
They cannot make changes to the code of an existing library due to incompatibilities? Oh rly? And if they take it out in their own separate package based on Apache code, then suddenly they can? In general, breaking changes in Android is a separate topic that is beyond the scope of this article, but what’s worth only limiting application permissions in the six ... At the same time, the guys from Apache actively tried to provide maximum compatibility, and were ready to do anything for that. But alas.
And then the library was simply cut out without any adequate explanation with the official recommendation to use HttpUrlConnection. Here is at least something remotely similar to the explanation of the situation from Jesse Wilson, who was at that time in the Dalvik team (by the way, he was Google’s second manager for relations with Apache. Remember this name).
In his article, you can see that among the advantages he sees:
Extremely dubious explanations. If you are sitting in a foil hat now, the rational argument is that Google decided to quietly force users to drive all their traffic through their proxy ...
UPD . As I was rightly pointed out, compression means not using GCP, but using standard gzip ... which Apache has for many, many years. So all the more strange.
Unfortunately, most developers blindly believe Google and immediately believe that the Apache library is “bad”, and you need to run to throw it out of your code.
Apache developers commented briefly on this:
If you are developing for Android, you probably saw recommendations for replacing Apache's HttpComponents with Square’s now popular OkHttp library.
Do you still remember the sweet guy Jesse Wilson from the Dalvik team? Do you know that he now works in Square ? And is he the creator of OkHttp? Moreover, do you know that OkHttp started as a fork of the AOSP (Android Open Source Project) piece, which in turn took its code from Apache Harmony?
So this is essentially the creation of an Apache fork with the subsequent discarding of the original from circulation (the second option from Jesse voiced earlier in communication with Apache). Sounds pretty vile, doesn't it? The only thing that is not clear is whether it was an initiative of Google or Jesse himself. But he did very ugly, throwing competitors out with the help of Google and coming all in white with his decision.
Let's figure out what are the options for how to live on.
1. Use the HttpUrlConnection
Google Recommended Approach. It really makes sense if you have a simple application. But God forbid you try to do something not entirely trivial. In my practice, there were two such cases - when I tried to use SSL proxies and when I wanted to contact a certain IP address with my host name. Both tasks cannot be implemented using HttpUrlConnection.
2. Use other advanced alternatives
For example, the same OkHttp. I have not tried it myself, but they say that the library is good. However, you will spend a lot of time rewriting good, ready-made code (if you already have an application). Well, as for OkHttp, I would not use such an unpleasant smelling fork.
3. Use the legacy library
Yes, you can continue to use the same ancient beta as before. But why? As a quick plugging of a hole, this solution is not bad, but if not ... Of course not. It's a shame that just such an answer is the most popular solution on the same stack overflow - people just do not understand that they are using the version of the library from 2008.
4. Use the latest versions of Apache HttpComponents
The pluses are that we continue to use the same code without wasting time rewriting and studying the new library and its bugs. Moreover, the code written in HttpComponents will work for you anywhere. The library is monstrously powerful and allows you to do really anything.
The question is how to connect it?
If you just take and substitute in gradle, then there will be a conflict of classes with this very ancient version.
In release 4.3, apache developers suggested using special “HC4” postfixes in classes to avoid conflicts, but it worked somehow very badly.
But for release 4.5, they already recommend another, the only logical way out - to use the library repackaged under a different class space, compiled by a friend on the github (link below). There, however, it is actually version 4.4, not 4.5 - but this is not so fundamental. And if you are worried that you are using a library that was not clear by anyone (although it is quite popular), then you can always compile it yourself from the source. At the moment, I think this is the most correct way out of this bad situation (and I’m doing it that way).
There are no notes on using the fifth version of the library on Android yet - perhaps this is due to the fact that it is still in the alpha version. Or simply Apache decided to no longer deal with Google and Android. However, even if so - there will always be enthusiasts who can correctly repack the latest version. And working with her is a pleasure.
In the process of writing the article, I talked about this topic with a developer from Apache, who confirmed my assumptions, but I will not mention it out of harm's way.
Introduction
If you are developing for Android, then you probably faced the fact that you open your application, which worked fine for several years, and then it suddenly turns out that Apache httpComponents became deprecated, and they are not recommended for use. First, let's look at what happened, and then draw conclusions what to do.
What happened
I did not dig too far, but many interesting things can be obtained from the org.apache.hc.dev mailing list and Apache jira. For example, the fact that:
Android included an old pre-beta library release
Moreover, throughout the history the version of the library has not changed, and if you turn on apache legacy, then this is still the same pre-beta.
Confirmation can be read here .
English confirmation
Google Android 1.0 was released with a pre-BETA snapshot of Apache HttpClient. To coincide with the first Android release Apache HttpClient 4.0 APIs had to be frozen prematurely, while many of interfaces and internal structures were still not fully worked out. As Apache HttpClient 4.0 was maturing the project was expecting Google to incorporate the latest code improvements into their code tree. Unfortunately it did not happen. Version of Apache HttpClient shipped with Android has effectively become a fork. Eventually Google decided to discontinue further development of their fork while refusing to upgrade to the stock version of Apache HttpClient citing compatibility concerns as a reason for such decision. As a result of those Android developers who would like to continue using Apache HttpClient APIs on Android cannot take advantage of newer features,
Google just scored a problem
Apache developers were actively working on the library, but the Google managers they appointed did not answer for months, changed, and in the end said that this was not a priority, users had the old version, and in general, they had to throw this library out. In confirmation - a bit from the mailing list mentioned by me (work began in 2008, and in 2010 a new manager suddenly appeared):
The new manager admits that they do not pay attention to the task ... Well, okay.
Jesse Wilson commented on HTTPCLIENT-898:
I'm Bob's successor on the Android team. If you've got questions about our use of the HTTP client code, I'll do my best to answer 'em. I regret that we haven't given this code much attention lately; that said I'm happy that it hasn't really needed it.
I'm Bob's successor on the Android team. If you've got questions about our use of the HTTP client code, I'll do my best to answer 'em. I regret that we haven't given this code much attention lately; that said I'm happy that it hasn't really needed it.
Then the first alarm bell rang:
You know, two years later, we decided that nothing should be done.
I'm quite sorry that Android included an unreleased rev of Apache HTTP client, and I suspect we'll be paying for that mistake for quite a while.
Because of the strict compatibility requirements of our platform, we will be unable to make forwards-incompatible API changes to the HTTP code. Unlike your desktop and serverside users, including the API as a part of the core platform significantly reduces our ability to make API changes.
These days we aren't spending much time on the HTTP client code. Our users seem to be mostly satisfied with the ancient version in the SDK, and have been directing their complaints elsewhere (crypto, locales, XML ...).
That said, we do want to figure out a long term strategy for an HTTP client API that will work for both us and the Apache HTTP client team. We're considering a variety of options ...
- Discouraging our users from using the built-in Apache HTTP client, preferring the JDK's own URLConnection classes. Whether this is feasible depends mostly on whether the new APIs in Java 6 (which we're working on) will satisfy the use cases that Apache HTTP client has covered for years.
- Replacing the Apache HTTP client API with a 3rd API in a "com.google" or "android." Package. Such an API would likely be based on parts of your own code, but with a more limited API.
- Tidying up the version of Apache HTTP that we're already stuck with. This includes deprecating APIs that shouldn't have ever been exposed as public, and possibly filling in any gaps using newer code from your project.
But none of this work is particularly urgent since we're actively fighting fires in other areas of the core libraries.
Because of the strict compatibility requirements of our platform, we will be unable to make forwards-incompatible API changes to the HTTP code. Unlike your desktop and serverside users, including the API as a part of the core platform significantly reduces our ability to make API changes.
These days we aren't spending much time on the HTTP client code. Our users seem to be mostly satisfied with the ancient version in the SDK, and have been directing their complaints elsewhere (crypto, locales, XML ...).
That said, we do want to figure out a long term strategy for an HTTP client API that will work for both us and the Apache HTTP client team. We're considering a variety of options ...
- Discouraging our users from using the built-in Apache HTTP client, preferring the JDK's own URLConnection classes. Whether this is feasible depends mostly on whether the new APIs in Java 6 (which we're working on) will satisfy the use cases that Apache HTTP client has covered for years.
- Replacing the Apache HTTP client API with a 3rd API in a "com.google" or "android." Package. Such an API would likely be based on parts of your own code, but with a more limited API.
- Tidying up the version of Apache HTTP that we're already stuck with. This includes deprecating APIs that shouldn't have ever been exposed as public, and possibly filling in any gaps using newer code from your project.
But none of this work is particularly urgent since we're actively fighting fires in other areas of the core libraries.
They cannot make changes to the code of an existing library due to incompatibilities? Oh rly? And if they take it out in their own separate package based on Apache code, then suddenly they can? In general, breaking changes in Android is a separate topic that is beyond the scope of this article, but what’s worth only limiting application permissions in the six ... At the same time, the guys from Apache actively tried to provide maximum compatibility, and were ready to do anything for that. But alas.
Guys, we have little time, we took your development and scored on it, do not bother us for nothing.
I explained on several earlier occasions that Android doesn't allow binary incompatibilities of any kind (not my rule). I understand that the HttpClient team is more tolerant of binary incompatibilities. While I'm not saying it would be impossible to make these changes in Android, I am saying that it would take a lot of convincing (and time), it would annoy other people who are time-constrained and who have higher priorities, and it could likely fail anyway.
The final
And then the library was simply cut out without any adequate explanation with the official recommendation to use HttpUrlConnection. Here is at least something remotely similar to the explanation of the situation from Jesse Wilson, who was at that time in the Dalvik team (by the way, he was Google’s second manager for relations with Apache. Remember this name).
In his article, you can see that among the advantages he sees:
- Caching (which, if necessary, is implemented anywhere)
- Low library weight (yes, but a dubious argument)
- Compression, which saves the battery and speeds up the download (yes, but if desired, we can use the Google Compression Proxy through the Apache HttpComponents as a regular proxy)
Extremely dubious explanations. If you are sitting in a foil hat now, the rational argument is that Google decided to quietly force users to drive all their traffic through their proxy ...
UPD . As I was rightly pointed out, compression means not using GCP, but using standard gzip ... which Apache has for many, many years. So all the more strange.
Unfortunately, most developers blindly believe Google and immediately believe that the Apache library is “bad”, and you need to run to throw it out of your code.
Apache developers commented briefly on this:
Google has used the project when it suited their goals and screwed us afterwards. There is nothing we can do about it.
Epilogue with OkHttp
If you are developing for Android, you probably saw recommendations for replacing Apache's HttpComponents with Square’s now popular OkHttp library.
Do you still remember the sweet guy Jesse Wilson from the Dalvik team? Do you know that he now works in Square ? And is he the creator of OkHttp? Moreover, do you know that OkHttp started as a fork of the AOSP (Android Open Source Project) piece, which in turn took its code from Apache Harmony?
So this is essentially the creation of an Apache fork with the subsequent discarding of the original from circulation (the second option from Jesse voiced earlier in communication with Apache). Sounds pretty vile, doesn't it? The only thing that is not clear is whether it was an initiative of Google or Jesse himself. But he did very ugly, throwing competitors out with the help of Google and coming all in white with his decision.
And what?
Let's figure out what are the options for how to live on.
1. Use the HttpUrlConnection
Google Recommended Approach. It really makes sense if you have a simple application. But God forbid you try to do something not entirely trivial. In my practice, there were two such cases - when I tried to use SSL proxies and when I wanted to contact a certain IP address with my host name. Both tasks cannot be implemented using HttpUrlConnection.
2. Use other advanced alternatives
For example, the same OkHttp. I have not tried it myself, but they say that the library is good. However, you will spend a lot of time rewriting good, ready-made code (if you already have an application). Well, as for OkHttp, I would not use such an unpleasant smelling fork.
3. Use the legacy library
Yes, you can continue to use the same ancient beta as before. But why? As a quick plugging of a hole, this solution is not bad, but if not ... Of course not. It's a shame that just such an answer is the most popular solution on the same stack overflow - people just do not understand that they are using the version of the library from 2008.
4. Use the latest versions of Apache HttpComponents
The pluses are that we continue to use the same code without wasting time rewriting and studying the new library and its bugs. Moreover, the code written in HttpComponents will work for you anywhere. The library is monstrously powerful and allows you to do really anything.
The question is how to connect it?
If you just take and substitute in gradle, then there will be a conflict of classes with this very ancient version.
In release 4.3, apache developers suggested using special “HC4” postfixes in classes to avoid conflicts, but it worked somehow very badly.
But for release 4.5, they already recommend another, the only logical way out - to use the library repackaged under a different class space, compiled by a friend on the github (link below). There, however, it is actually version 4.4, not 4.5 - but this is not so fundamental. And if you are worried that you are using a library that was not clear by anyone (although it is quite popular), then you can always compile it yourself from the source. At the moment, I think this is the most correct way out of this bad situation (and I’m doing it that way).
What's next?
There are no notes on using the fifth version of the library on Android yet - perhaps this is due to the fact that it is still in the alpha version. Or simply Apache decided to no longer deal with Google and Android. However, even if so - there will always be enthusiasts who can correctly repack the latest version. And working with her is a pleasure.
References
- Repository with a repackaged version of httpComponents;
- Release Notes 4.5, where they recommend using this repository;
- Interesting fragments of the correspondence of guys from Apache and Google;
- Curious problems with comments in Apache jig - one and two .
In the process of writing the article, I talked about this topic with a developer from Apache, who confirmed my assumptions, but I will not mention it out of harm's way.