Pokemon Go Hacking Chronicle

Introduction from the translator

He translated this story, since it was considered interesting in my previous article about the relationship between Niantic and the developers of any additional software.
This sweet story does not claim to be the greatest and most professional hacking of all time, but allows you to plunge into the cozy atmosphere of an open source community engaged in a common cause. Perhaps someone will remember their past days ...
The author of the original admits that he is not well versed in the development, but this is also possible for the better - the general atmosphere is well transmitted without unnecessary technical details. If you are interested in technical details, then you can follow the links in the text or just parse the resulting API. Yes, you may be confused that the text sometimes says either about creating an API, or about hacking it. This refers to two different APIs - the API of the game itself and the API that developers make to control the game from the outside. We'll have to distinguish them by context. Yes, it should also be borne in mind that updates were written in a hurry and sometimes by a tired and sleepy person - some formulations will sound strange. I also apologize for the abundance of links, some of which are already inoperative. The format of the reddit is not very consistent with the format of the article, but it seemed to me

Background

The API has stopped accepting requests from all sources except the updated official client. Now he needs "unknown 6", which was in earlier versions of the API, but was not tested. At the moment, only the official client can create the correct "unknown 6". We are not completely sure that the case is in “unknown 6”, but it is very likely that this is actually the only large piece of data that we cannot recreate for now.

We cannot just take and compare versions of the application before and after the update, because this variable was already in previous versions of the API, but the server did not check it.

The final value of the variable itself does not matter, and a simple selection does not help - we need not a value, but a way to generate “unknown 6”.

We are trying to find exactly where it is generated in order to recreate our version and restore the operation of our APIs ...
This is quite difficult, because some fragments of the code are difficult to analyze.

History

08/04/2016 - 00:49: Apparently, the logic and protocol have changed.
08/04/2016 - 01:37: The protocol has not changed, no new hashes have appeared. Perhaps now requests are signed by some cryptographic methods, but we are not sure yet.
08/04/2016 - 02:07: It becomes obvious that the change is not trivial, and reverse engineering can take a lot of time.
08/04/2016 - 08:08: Everyone is working on the definition of “unknown 6”. All known data is collected here.
08/04/2016 - 15:06: We confirm that “unknown 6” is responsible for the changes.
5 August 2016, 14:00 - Breakthrough? The developers seem to have found where “unknown 6” is generated. Now we need to recreate the process and hope that everything works. And the fact that it was “unknown 6” that broke everything for us.
14:30 - The channel in discord went into private mode, because other people are taking over our breakthrough. I was also thrown out of discord, so there probably won't be any more updates.
(16:20) I was allowed back. We are now sure that it is precisely the “unknown 6”. This is glorious, otherwise all the work to recreate it would be unnecessary. We are on the right track.
16:30 - A lot of difficult work and incomprehensible words. As I understand it, now we know where most of the “unknown 6” is created. Now we need to understand the input data for the generation and methods for further encryption of this unknown. There is still a lot of work to do ...
18:00 - We disassembled the encryption part, and now we can partially understand the source data. As soon as we analyze all the encryption, we can repeat its process by running through the decompiled code.
19:30 - We are one step closer ...
20:30 - Breakthrough # 2: We tied together two pieces of code responsible for generating “unknown 6”. We found where the encryption function is called. As I said earlier, we now have a decompiled encryption function.
21:15 - We have two things left to do:
Bring the decompiled version of encryption into a human form. This is custom encryption, and the decompiled code would take about 200 pages of text ... People work on it - slowly but surely. Not the hardest part of the job, but time consuming.
Determine the remaining input. It can be quite difficult ... There are 3-4 fields left, and we rejoice at every little breakthrough.
22:30 - No news, except that they work, but I thought it would be nice to write something about the last day.
It is insanely cool to watch the work of programmers with reddit and their joint efforts to crack the “unknown 6”. By the way, this is the same unknown on which hackers of the Congress “broke”. But the Pokemon Go developer community is much larger. Some have been hacking for 20 hours out of the 24 that have occurred since the API change. / u / keyphact hasn’t slept for 40 hours (dude, seriously - go to sleep). People work purposefully and tirelessly. I feel like we can do it.

We found the core of the creation of “unknown 6” in a few hours. The encryption functions were decompiled and the locations of their calls were found. Only 10% of the input data and the normal implementation of the encryption functions are left. But so close, but so far away. Can we handle it?

23:30 - We understood most of the encryption. However, they still do not understand how the input data (protobuffer) is stored, it is insanely complicated. This is required to get ahead of the remaining input.
01:30 - We have fully working encryption! (although we did not fully recognize him!) This can be considered a breakthrough # 3. It remains to cope with the protobuffer.

This is insanely interesting and informative for me too. I hardly understand what the developers are doing. For example, what is the “protobuffer format”? I was told that it is somewhere between the input and encryption. It takes the input data, forms a packet from them and sends it for encryption.

Now we have encryption, but we cannot determine the input, because we do not know how the packet is formed from them. So we are making our own packet format (protobuffer). Gradually, step by step ... I hope that when we deal with the protobuffer, the input will become clearer.
03:30 - No special news. We are working and moving forward.

It should be mentioned that ensuring community health was an insanely challenging task. It was a nightmare without any logical decision, and the moderators did not have any “good” answer to it.
It all started easy and simple - an open channel on discord, where everyone could work together to fix the API. But a little later it became clear that everything is not so easy and simple ...

The number of people on the channel grew and grew. Which in turn led to a huge amount of spam, like “When will the API be ready?” or “What happened to the API?”. It became impossible to work, and we had to limit the rights to post messages to the channel.

To make matters worse, some presented our work as their own and made false promises to the community about when the new API would be ready. And the moderators were spammed with requests for the right to post to the channel. We decided to completely hide the channel.

We tried to create a second channel in which to check suitability for general work. But there, in the end, the same problems started as in the first channel. And people were angry with the moderators because they no longer had access to the materials of the first channel, and they had to repeat the work that we had already completed a long time ago.
Now we are moving towards transparency again. We made the first channel again accessible to all for reading. Hope no one abuses this again. We also made an open github and wiki for everyone .

What other conclusions can we draw from this lesson? No, probably. I just want to thank the moderators for the insane amount of work they have done to solve this unsolvable situation.

And now I'm going to sleep. We opened the channel for API updates, since the messages in the main channel are somewhat complicated for understanding by all sorts of technical terms. I'll be back to you tomorrow.

5 August 2016, 13:00 - Here is a detailed technical description of our current problem.
13:30 - No special news, we work work. There is some progress with the protobuffer, we have identified several fields, but there are still quite a lot of obscure data. We also analyze encryption functions to fully understand how it happens. Reverse engineering is a difficult and tedious job
. I also want to answer a few comments. Little FAQ:
Q: I think I can help, how can I do this?
A: Unfortunately, the main channel is closed to new developers. But you can help in our public repository and do pull requests.
Q: Developers should try x.
A: I have no idea what you're talking about, but they probably already tried it. If you are sure that you really have a breakthrough idea, write it down somewhere on discord.
15:30 - No special news, still working the work.

We have a new thread on reddit. There are more technical details that you are unlikely to understand if you are not a programmer. If you do not see updates, then this means that the developers are actively working. We also raised twitter , in which everything is somewhat more understandable - I will write there with every update, as soon as they give me access there.

We also made discord invites permanent, their lifespan should no longer expire, * crossed fingers *.
We want to keep you informed and to avoid the appearance of fake twitter accounts.
I reached the limit on the length of the text. Further updates will be here .

August 5, 2016, 16:00 - we have defined another input field! It's cool to finally see some kind of progress. Do not rely on the solution right now, we have one more field left, but we are already working on it.
17:00 - We checked the field that I mentioned earlier - yes, everything is fine! Everyone is in a great mood, we are moving forward.
18:00 - We think that the last field is somehow connected with the field that we just hacked. Hope this helps somehow.
18:30 - We want to reiterate that the API hacking community does not support bots. We just break the API. So we want to confirm that Niantic can trace any MITM applications, for example, those that modify pokeball throws so that they are always accurate. If you use such an application, Niantic most likely already knows about it.

We don’t know whether you will be banned for this, we just made sure that Niantic can (theoretically) determine it. But this does not bother us, we are engaged in hacking the API.

20:00 - No special news at the front of the development. Still working on the remaining input.
We adapt to the many channels through which we communicate with you. We have Discord, Twitter, Reddit, this post and github repositories. They are simply bursting with messages. However, our internal communication is much better. Setting up and supporting interaction is sometimes debt and time consuming, but it is exciting. It's nice to know that developers can safely go about their business by cracking the API.
23:30 - I am back and waiting for updates to translate them from programmer to human. We have already gone far, but to the final destination even further.
00:45 - Progress made in the last hours can be called a breakthrough # 4.
We opened 3 more input fields. One of the fields is an encrypted (or rather, hashed) version of the authorization ticket, in combination with the GPS position, a second field is obtained. The third is also somehow related to the authorization ticket.
“Combination” is a powerful simplification of what is actually there. The real complexity of the work of programmers is far beyond my level of understanding.
We are currently working on the remaining field. Or in the fields.
GMT 03:30 - We didn’t write much, as there is little progress.
We’ve been trying to hack one of the fields for 12 hours. We know more than when we started, but there was no breakthrough yet.
We know that the field is not combined with an authorization token, however, it depends on the session (maybe not directly). We also know its length (16 bytes). We are working on updating the information.
Right now, most developers have gone to bed. I must say, this is a well-deserved rest. I’ll go and I’ll sleep.

6 august 2016, 13:00 - So, now it will be purely my comments. This is the only real source of information, the rest are fakes. Now I will say I am to myself and THEY about the developers. This decision was made to remove unnecessary pressure on the developers.

While I was sleeping, nothing really happened, perhaps because the developers were also sleeping. The field, which brought us so much hemorrhoids, deserves its own name, now it will be “unknown 22”. An additional complication is that it is attached to the session, and collecting data is rather difficult. We are analyzing this field and one more.
14:30 - No news, just wanted to answer the question “how so far nothing has happened? You said that there were 3-4 fields left, but much more were hacked! ”

In fact, these 3-4 fields turned out to be a combination of many other fields that also need to be hacked. It is impossible to answer the question how many fields are left. We do not want to create false expectations. Yes, and we don’t know.
17:00 - Breakthrough # 5: The developers found that the “unknown 22” is not needed. One of them answered this with the expected “Yes, your mother!” Now developers are trying to build a demo application to test this hypothesis. They are very excited and pray.
It should be noted that even if the API call is successful, it is just a prototype, not a fix for the API. The developers scored on quite a few fields - for example, on a field that is transmitted only from Android devices. To get around it, developers emulate the use of iOS. Much more needs to be done so that this does not work so dumb.
17:30 - The first versions of the prototype do not work.

18:00 - No news, I just wanted to tell you why the “unknown 22” turned out to be such a hemorrhoids - we got a working theory about this. “Unknown 22” is a random variable that is generated during application initialization and then bound to the session.
The developers carefully searched for everything that could affect the “unknown 22”, until they gradually realized that it had no input parameters. This is just a random variable. I will try to explain why it was so difficult to determine.

Real-world example: Imagine that we want to determine the temperature in New York. There are a huge number of values ​​that correlate with it. For example, selling ice cream. When sales rise, the temperature rises. However, calculating the temperature based on ice cream sales is futile. Correlation does not mean causation. Consider this when reading about “unknown 22”.
At first, the developers tried to change the authorization token, and each time they did, “Unknown 22” also changed! So they decided to verify that authorization affects “Unknown 22”. To do this, they needed data.

It took a long time to collect this data because they had to log in and go back. Also, “unknown 22” could be influenced by many other fields, including, for example, SessionID, Auth_token, Auth_ticket. They tried to understand the dependencies on each of them, until someone realized: the “unknown 22” had no input parameters.

“Unknown 22” is randomly generated every time the application starts.
So Niantic cannot in any way check what meaning “should be” there - it is by accident! So you can just shove anything there. This is still a theory, but all developers stick to it.
18:30 - Breakthrough # 6. We seem to have the first successful API call! Wait for updates, confirmation will be within an hour.
18:35 - It works! Believe me, it causes me as much excitement as you do.
19:00 - The public chat in discord is still empty. Still waiting for updates from the developers. Are you updating the chat too to see nothing?
20:00 - For quite some time there was no information. However, they say that they are working on an implementation, so apparently there’s nothing more to break. The next update should be great, stay with us ...
20:30 - They removed the public github. Probably moving somewhere. Apparently, work is in progress. Update: the repository is closed for any copyright reasons there.
22:00 - I am already starting to doubt, but I still believe that they were able to make a successful API call then. Although apparently, they think when, how and what they publish. Hard work, given that they closed the github.
From the very beginning, everyone knew that this work has 2 stages. First reverse engineering, then the creation of a new API. A successful API test call is important because it means working somewhere in the middle.

However, this does not mean that reverse engineering is complete. Some bypassed fields were not necessary, but they may still want to hack them.

I will look like an idiot if it turns out that they did not get an API call, but I'm ready to put up with this opportunity.
I reached the limit of the number of characters, I will continue here .

Here I will also write my own soaps, as in the previous comment. I am not an advanced programmer, but I was with the group “unknown 6” from the very beginning.
6 august 2016, 23:00 - A small update. They are looking for ways to get around copyright so that their materials are not torn down again. They also write that they are working on an implementation, which means that they did make a successful API request and are already working on a working version of the API.
00:00 - They say that the “last breakthrough” remains, let's hope that this means something good.
Their work is hampered by wild spam requesting rights and updates. Please just let them code! They won’t work faster, and believe me, you can live another day without an API.

Some also blame developers for doing it for their own benefit. It is not true. I know many of these guys, and they do it just for fun, for challenge. They are not going to sell the API to anyone.
They also say, “The fact that a certain paid service says that it has an API fix does not mean that we sold it to them.”
00:30 - I just wanted to say - I hate bots.
00:45 - They just confirmed that the API is working (NOT FULLY). But they didn’t write about it at all:

For all those who spread rumors that we first launched our bot.
Excited by our success, a team member used our not yet fully prepared API in his bot and posted a screenshot. And some other team members also added APIs to projects not related to bots (for example, pgoapi and RocketAPI).

In the end, as soon as we finish, everyone will have access to the completed version at the same time.

1:15 - Done, API released!
Victory. The developers broke the API in 3 days and 5 hours. A worthy achievement.
1:30 - This API is easily tracked, and Niaintic can determine that you are using an unofficial client. This is quite suitable for the developers, and you should not roll a barrel on them.
For example, altitude is not taken into account. All requests are allegedly sent from an iOS device, which is extremely suspicious, since it is Android with obviousness ... There is still a lot of work, but we got a working API and with this our participation in the project was completed.

1:45 - I'll go to sleep. In the last nights I slept very little. I want to say many thanks to all the developers, moderators and everyone else who participated. And those who patiently waited for a solution to the problem.
Your support was amazing. A week ago, I could not even imagine that I would be a full-fledged manager of the Pokemon Go hacker community.
Thank you all,
/ u / DutchDefender

From translator

As you know, the story is not over, the story is just beginning. Does Niantic listen to requests for application improvements? Or will he try to defend himself again? Does the community have the strength, desire and skills to crack new methods of protection? We will find out soon.