“I was very negative towards coroutines”: Artyom Zinnatullin on Android development
Among Android developers, Artyom Zinnatullin is so respected that you can compose an analogue of “facts about Chuck Norris” about him - something like this:
- Artyom is so harsh that when he sees him, the github itself turns green (which of us can boast of such a schedule of contributions?)
- Artyom is so harsh that for him git is a messenger .
- Artyom is so harsh that in his applications context is a podcast .
When we interviewed him at our Mobius conference, it was intended for an online broadcast. But having seen how they are referring to it in the Android chat, we decided that it could also be of interest to many on Habré, and made a text version for you (we also attach the video recording).
How to live with a project on a million lines of code? What is the disadvantage of corotin Kotlin? What’s wrong with Google? How is development in San Francisco different from Russian? What was the Mobius talk about? Under the cut - about all this.
Evgeny Trifonov : At this Mobius, I missed your talk “Android builds at Lyft”, but after it I saw a crowd of people wishing to ask a question in the discussion area. And I wanted to clarify: but after all, most viewers are not working in a giant project like Lyft, was this experience relevant for them anyway?
Artyom : This is an interesting thing. The original outline of the report in my head, and how I eventually implemented it, are very different thanks to your classy program committee.
Initially, I was going to tell you how it all started at Lyft, why we came to certain technical solutions. He talked to Sergey Boishtyan from the program committee for two hours, he listened and said: “Cool, of course, but you did keynote.” And in the end, I realized that such a report, of course, is interesting to listen to, but it is really not relevant for anyone.
And then I redid it, shifting the emphasis to fundamental engineering approaches to the choice of assembly systems, other systems. I didn’t have a goal to tell which tools we use specifically. I don’t want someone to take and blindly start using them, and then write me formidable letters that not everything works as I told you. I wanted to convey exactly engineering practices about how to make a choice, and what is important in my (naturally, subjective) opinion. Therefore, I hope that in the end the experience is relevant to more people, and not just "the dude from Lyft came out and said something."
Oleg olegchir Chirukhin : Are there any unusual Lyft choices that are difficult for others to make?
Artyom: Oh sure. We have two build systems in the project at the same time, I do not recommend absolutely anyone (laughs) .
It is very painful to maintain: you constantly chase after two birds with one stone, in both, something does not work to the end. But this is our current state, so historically, because one build system began to shut up on part of the tasks, we had to start a second one. I talked about how to avoid this and correctly migrate to one of them.
Oleg : And what kind of build systems?
Artyom : We use Gradle and Buck, and I talked about how to come to Bazel from Google.
Oleg : This is some kind of movement towards evil: from the pretty Gradle to Bazel, in which there are not even normal dependencies.
Artyom: Now there are more or less. Well, yes, of course, there are trade offs, and, of course, Gradle has its merits. It all depends on the type of project. Some Gradle will be more suitable than Buck and Bazel, because they have some fundamental points that they will not collect incrementally within the same module, but Gradle will be, and for many this is very important. And it's cool that Gradle can do that.
Another thing is that when you add modules - more, more modules, eight hundred, a thousand - Gradle is so redesigned that it will linearly slow down the assembly in some places. But it seems to me that Gradle can fix all this if the community puts pressure on them - which maybe I am doing. We will see. (note: a few days after this interview, Artyom wrote a long post about Gradle problems)
Oleg:That is, Bazel just because you want to support a large number of modules?
Artyom : Let's just say that in our case we don’t “feel like it,” but dividing the project into modules allows our business to move faster. Basically, as I understand it, this is isolation so that it does not work out spaghetti, which is then difficult to maintain. Modules give more control over which parts of the code interact with which. We have almost a million lines of code. If it were in one module, I would have to spaghetti. Because on top of the language - Java, Kotlin - it will be necessary to wind something up to prohibit calls between packages, between which no one expected them. Plus, the question will arise that Gradle will not take out so much code in one module. It will not collect it in parallel, incrementally assemble it inside the module.
Each solution has trade offs. In our case, it seems to me that this is the right solution, but there is a problem - in that we currently support two build systems.
Oleg : And what is better for hundreds of modules: monorepo or many repositories?
Artyom: This is a very sore point. Probably one repository is better from the point of view that you don’t have to think about versioning and there isn’t this dependency hell when you go and clone a dozen repositories to make one change, and then open one pull request and another ten after it. “Friction” is removed from the system, and people are not afraid to change the code. For them, an atomicity of changes arises: everything is commiled into one project, and changes from one module are automatically transferred to others without their explicit consent. At the same time, all the checks that you automatically wrote in CI will be executed and verify that the code is compiled, tested and all this.
Oleg : And if you do not come to the fact that you, like in some Chrome, the branches will change for two minutes, while you drink tea?
Artyom: Yes, of course, there is a possibility. But here, probably, the question is already in the size of the product: does Chrome need to contain so much code? Maybe it’s worth highlighting some parts in separate instruments, which they will periodically tighten when major changes occur in them? This is probably a question for the organization of the project. Cool example, by the way. I have a similar one: correspondence with dudes from Yandex.Browser, where they also have big plugs.
Chrome can be divided into several components, and if you take some V8 - I'm not a big specialist, but as far as I understand, it could be a separate project in general, right? And why, then, should the GUI know about the engine, reassemble it each time and think about the source code lying somewhere nearby? Bazel, by the way, also supports this.
In general, now all the big build systems - that Gradle, that Buck, that Bazel - support such a thing as composite builds when you refer, for example, to another Bazel assembly. This is a tricky situation, but, nevertheless, this works, it allows you to remove part of the files from the repository, reduce the size. The IDE, for example, will go crazy to index all these files, so I want to somehow separate them from the general component of the project.
But we are far from that. It seems to me that we can calmly figure another five years. We are unlikely to run into a two-minute checkout yet. We do not have many people.
Eugene : Does Lyft still have its own specifics, besides two build systems?
Artyom: Yes, there are a couple of atypical stories. It so happened that people who came to the company (from Google, Facebook, everywhere) hate mono-repositories. As a result, we at Lyft have three mono-repositories: Android, iOS and L5 (these are our autonomous cars ).
And everything else is more than 1,500 git repositories: for all microservices, for all libraries separately. This is historically the case. This has its own huge price that we pay: pushing changes through them is really difficult. On the other hand, when working with each of them, you have instant git clone, instant git push, everything is very fast, the IDE indexes per second. I can say that this is really an interesting part. From the dudes from San Francisco, I would expect a single repository.
Oleg: And when one of these separate repositories is updated - the API changes, for example - how does this change apply to the rest of the company?
Artyom: It hurts. (laughs) Well, I’m not a backend developer in that I don’t write feature backends, I write infrastructure backends - they are usually quite autonomous in this regard.
As a rule, this is just a bunch of rallies, cross-interaction, and then planning.
Oleg: So rallies are part of the assembly system? (laugh)
Artyom : Yes, first we need to put together a rally, then put together a repository. Plus, unfortunately, historically, we have many of these microservices - this is Python, which also has its own jokes.
Oleg: Some dislike for Python slipped.
Artyom : Rather dislike for dynamic typing. Python, not Python - it makes no difference, but dynamic typing is a sore thing.
Eugene : And it slipped “for a company from San Francisco”, and it’s curious to ask this: but in terms of development, companies from San Francisco differ from companies from Russia, is there a noticeable difference?
Artyom : A very big difference. I’m not a big fan of classifying it like that, but it seems to me that here is a more correct engineering school.
Oleg : Here - where is it?
Artyom: In Russia, in the countries of the former USSR. People pay more attention to the technical aspects of how their system components work. And in the States it often happens that some library solves a problem, and people don’t even look at how it is implemented. They, as a rule, absolutely do not care that it slows down or that they use it incorrectly.
I will interview a lot of people there, because this is part of the work, and the general level of knowledge, perhaps, is still lower. There is something to change. Every time a person comes from Eastern Europe, it becomes more interesting during interviews, because people are not afraid to resist, to argue somewhere. While US candidates very often may not answer questions at all or answer “I don’t remember when I last used it.” For questions like “How does an HTTP request work?” or “What data format will you choose?” they cannot give normal engineering answers, but say: "Well, I've used this for the last five years." Cool, of course, but the lord does not pull.
On the other hand, there are projects that took years to compare with what we are doing here. People make more mass products, and there simply is more scale. For example, Chrome or Uber - they already have more than a thousand modules there. This is just the scale of the problems. Let's say in Uber under three hundred Android developers. The question arises: why? (laughs) But, nevertheless, they managed to make this colossus work, constantly release. I would say such issues are less frequently resolved here.
Here Yandex is a good example. I have a friend on Yandex.Maps: ten people make an Android application. In Google, most likely a hundred are sitting. And at the same time, Yandex.Mart has more functionality. That's the difference, in my opinion.
Evgeny: In addition, the Valley is also associated with start-ups, and they have a “move fast and break things” approach, and it seems that this should also affect the development: live on the bleeding edge, use the latest. It's true?
Artyom : I did not work in startups, Lyft is difficult to call that: there are already three thousand three people, somewhere more than a thousand of them are engineers. That is, it is an already formed company.
It is cutting edge technology that is rarely used. If the technology is hyped, then yes. If the technology is niche, but cool, very often not. Until all the conferences talk about it, very few people will use it.
But at the same time, what I really love (in San Francisco and partly in the Valley) - a lot of issues are resolved due to the fact that the companies are physically close. Very often you write to someone in a chat room: “Let's have lunch together at our place or in your office and decide, we will advance some question”, and then once - an open source project or pull request appears in another project, something fixed.
What is interesting: people often discuss things that actually should not be discussed on the NDA. But this is how the whole Valley moves, in the end everyone understands where the rest is moving, and the whole industry goes together. Say, Lyft and Uber mobilists constantly talk about technical things, because we use open-source from Uber. And, of course, there are directly hardcore experts in some technologies. This is also cool: you can just cross with them.
I love this, and this was not enough for me in some cities where I lived. Here in St. Petersburg there was a very cool Java User Group (I don’t already know how it is now): you come after work, and Shipilev takes out your brain, and something is good!
And there it again appears: for example, it also has its own Java User Group, and there often come dudes, say, from Oracle, who filmed some new Reactive JDBC. And you sit, argue, because some Project Reactor leader or Reactive leader in Spring is sitting in the same place, a hot discussion is going on, and this is cool.
Oleg : I’ll ask about something else: I looked at the Mainframer repository , and Rust is used there. Why is all this written not on the blessed Javka, but on some kind of Rust?
Artyom: Recently, I have moved to the side that the program should have a minimal amount of resources. That is, I want to be very close to how iron digests bytes. And in Java, a lot of things are happening around (I’m not even talking about garbage collection), that is, JIT and all this. I really like that Java is now moving towards the fact that there will also be ahead-of-time compilation. It seems to me that it will be very cool, for example, to start the launch of a microservice by downloading from the cache its ahead-of-time compilation, which originally happened on some other machines, and starting it very quickly, without warming up. This is cool, but Java has a price. I can’t just ask people who are building an iOS project to have Java on their system.
Mainframer was originally written in the Bash dialect. But I wanted to rewrite it in a system language to get normal multithreading, the ability to write normal unit tests, and not just integration tests on top of the utility ...
Oleg : And one could take, for example, Python.
Artyom : Yes. But then the question would arise with the fact that, firstly, this is dynamic typing, and, secondly ...
Oleg : So in Bash, it’s also dynamic typing.
Artyom: So I wanted to rewrite it. And besides this, there is a problem with the fact that Python is now two: on macOS, by default the second, and almost all on Linux is now the third. There are all sorts of such jokes. If I need some kind of addiction to bind that I will ask people to run pip? Or will I have to bang her?
I wanted to take a system language that requires zero dependencies, so that I can put a binary that will weigh, conditionally, less than a megabyte, and work with minimal overhead.
Oleg : You could take Golang, at least there is a garbage collector.
Artyom : That's exactly why I wanted to try Rust. And it worked. Plus, Golang is kinda sad with generics.
Evgeny: Since they started discussing languages ... In the context of Android development, the question “Kotlin or Java” is already tired, but still I’ll ask it in order to continue to the next question.
Artyom : Well, Kotlin, of course.
Eugene : Now the question that really interests. Recently, at Kotlin, coroutines became stable, and the voices “Hooray, let's get away from RxJava” are heard. Therefore, when I see a person in front of me who is very close to RxJava, I immediately want to ask his opinion about coroutines.
Artyom : I was very negative towards the coroutines. In principle, it is still mostly negative, but this has partly changed the very long conversation with Roma Elizarov , who is working on them.
As a user of programs, I want them to be as non-blocking as possible, to use resources as correctly as possible. By this I mean both parallelism and the fact that they use the correct operating system APIs for non-blocking access to the network or files - there are a lot of problems with this in operating systems, but, nevertheless, there are such APIs. What exactly does this solve? As a user, it doesn’t matter to me - if only the developers would solve this problem so that they would be comfortable. I have no big problems with this. And this is the vision of Roma Elizarov. After this conversation, I somehow let it go.
Prior to this, to me, like my friend Arthur Dremov, after several years of using Java in production, this seemed like a step backward: the code again becomes imperative, unclean, it loses understanding of the pipeline, it again becomes a mess, which the compiler turns into an asynchronous mess for you.
I don’t use coroutines, but all the examples that I’m now observing have switched to a structured approach, when you don’t even see which piece of code from this is coroutine. To be honest, I’m very scared to look at it. Because I open the pull request on GitHub, some methods are called to upload the picture and profile, one of them goes to the network, and the other goes to the local SQLite, and here the local SQLite can easily be blocking. I don’t see this in the code, because the coroutines are made so that you do not see it. Maybe it’s good, but for me this is still a minus of the design, because in Rx-approaches it is very obvious: do you understand whether this is part of the synchronous pipeline or not.
Perhaps this is my only complaint to coroutines: I want to see when my asynchrony occurs, and when not. Ideally, I want people to write more functional code when there are small reusable, or at least testable pieces that combine with others. And we come back to the fact that inline it all into the logic, and then the compiler then just flips it.
Oleg : Let me give you a little opposition. Legacy code is much more than new. And if we take some things like working with the network, working with files, and so on, then no one will quickly rewrite all this, for example, using RxJava. And if we have autocoroutines, then we can, for example, track all syscalls, automatically wrap them and send them to the lock for parking.
Artyom: True, in any case, you will have to call functions from the context of corutin. But this is an interesting thought, yes.
Oleg : Maybe they should be combined somehow? The top-level API will be on RxJava, and the low-level API on coroutines.
Artyom: Yes, there are such shifts now. But then the question arises, because at the moment, RxJava can do everything that the coroutines do, and coroutines cannot do everything that RxJava does. That is, the first technology can absorb the second, and the second the first - not. And therefore, most likely, there will be progress towards the fact that there will be some kind of cross-platform Rx on Kotlin on coroutines. But this is a different way of thinking. This is how you can take forEach and drove to make some kind of map, but you can take streams and write on them. And it seems that even the enterprise Java community has already approved and started to stream, because it is more expressive. And with coroutines, we are now going in the opposite direction.
And it also seems that for Kotlin, coroutines are an addition. As far as I understand, for languages like Go, this is the basis, initially part of the type system, and all the APIs that they provide work just as well: the standard library uses coroutines very much there. And in your situation, Oleg, it turns out that you write a code that may be legacy for you, but it is asynchronous, and that's cool. And what we get in Java and Kotlin right now is legacy, and you don’t understand whether it is asynchronous or not asynchronous. One thing is better than the other. Better to have some kind of understanding of what is going on.
But, as I said at the beginning, as a user of programs I am glad. The more tools that are more suitable for more people are given to them in order to write more correct programs - the more I am happy. Therefore, here I have absolutely no complaints.
Eugene : And the last question, quite general. After criticizing coroutine, which many are very happy with, I would like to ask: what are the main problems of modern Android development? I wonder if this will turn out to be contrary to the opinions of others.
Artyom: An interesting question ... It is difficult to answer. I would say that, in principle, there are no special problems in Android development. There are a very large number of tools that you can use and get great program quality. There is a problem that it is difficult to scale it to a large team: people need to correctly understand how this tool is used. And in this, RxJava loses a lot to coroutines, because it is very easy to use it absolutely incorrectly: use it for some small asynchronous things and not express logic everywhere in streams. In this regard, the coroutines are likely to go better.
It seems to me interesting to compare here with iOS. I am ashamed to say this, but here at Lyft iOS developers are only introducing dependency injection and RxSwift this year. And now 2019 is coming. I definitely know iOS teams in which this is not the case, have been using modern approaches for a long time, clean, that's all. But it seems to me that Android in this regard is far from the worst platform.
Perhaps the only thing I don't like is what Google is doing now. For a long time, their position was "we are not opinionated, use whatever you want: here you have the framework, and how you use it is not particularly important to us." Part of the community kicked them for a long time - and, in my opinion, in vain.
It was a golden time when you could say “RxJava is a solution because ...” And now people come and say: “No, we will use LiveData”. You start asking why - it loses in all respects to RxJava, and coroutines, and everything else. But, nevertheless, Google found that the community is important.
Many of the community will now say, “Cool that Google is promoting MVVM.” And for them, the third question is that this MVVM is absolutely crooked and wrong and, in my opinion, violates all the principles of what MVVM should be. And many projects have already switched to what Google now recommends.
It seems to me that they don’t have the right feeling where the scope of the projects ends. Very often, the wrong architecture ends up scattering over several projects.
But at the same time there are very cool works: for example, Room is very well made and in general a very cool library. And some Architecture Components are a very controversial set of things that were already implemented in the community five years ago. Why did Google come so late, and even with a crooked decision? These are the moments that bother me.
Eugene: I suspect that many people had a burning desire to object to this place. Well, you can then do it in the comments. Thanks for answers!
Artyom: Very interesting questions.
The next Mobius will be held in St. Petersburg on May 22-23 . Now his program has not yet been announced, but this is the most profitable moment for buying a ticket: on February 1, ticket prices will rise. And also, since the program has not yet been completed, the moment is also suitable for getting into it ourselves: we are in full swing accepting applications for reports. All information about the conference and tickets are on the site .