“I was very negative towards Korutin”: Artyom Zinnatullin about Android development

Among Android developers, Artyom Zinnatullin enjoys such respect that you can write an analogue of the “facts about Chuck Norris” about him - something like this:

• Artyom is so severe that when he sees it, the githab itself turns green (which of us can boast of such a schedule of contributions?)
• Artyom is so severe that for him git is an instant messenger .
• Artyom is so severe that in his applications context is a podcast .

When we interviewed him at our Mobius conference, it was meant for online broadcasting. But after seeing how it is referenced in the Android chat, we decided that at Habré it might also interest many, and made a text version for you (also we are making a video).

How to live with a project on a million lines of code? What is the disadvantage of Kotlin Korutin? What's wrong with Google? How does the development in San Francisco differ from the Russian? What was the report on Mobius? Under the cut - about all this.


Evgeni Trifonov : On this Mobius, I missed your report on Android builds at Lyft, but after it, in the discussion area I saw a crowd wanting to ask a question. And I wanted to clarify: but after all, the majority of viewers do not work in a giant project like Lyft, did this experience still be relevant for them?

Artyom : This is an interesting thing. The initial outline of the report in my head, and how I ended up implementing it, are very different due to your great program committee.

Initially, I was going to tell you how it all began in Lyft, why we came up with certain technical solutions. He told two hours to Sergey Boishtyan from the program committee, he listened and said: “It's great, of course, but you did keynout.” And in the end, I realized that such a report, of course, is interesting to hear, but it is really not very relevant for anyone.

And then I redid it, shifting the emphasis on fundamental engineering approaches to the choice of the assembly system, other systems. I didn’t have a goal to tell which tools we specifically use. I do not want anyone to take and blindly use them, and then wrote me formidable letters that not everything works the way I told you. I wanted to bring engineering practices on how to make a choice, and what is important in my (naturally, subjective) opinion. Therefore, I hope that in the end, the experience turned out to be relevant to more people, and not just "the dude from Lyft came out and said something."

Oleg olegchir Chirukhin : Are there any unusual choices of Lyft that are difficult for others to make?

Artyom : Yes, of course. We have two build systems in the project at the same time, I absolutely do not recommend it to anyone (laughs) .

It is very painful to support: constantly chasing two hares, in both something does not work until the end. But this is our current state, historically, because one build system began to shut up on the part of the tasks, we had to start the second one. I talked about how to avoid this and correctly migrate to one of them.

Oleg : And what kind of build system?

Artyom : We use Gradle and Buck, and I talked about how to get to Bazel from Google.

Oleg : This is some kind of movement towards evil: from the neat Gradle to Bazel, in which even dependencies are not normal.

Artyom : Now there is more or less. Well, yes, of course, there are trade-offs, and, of course, Gradle has its advantages. It all depends on the type of project. Some gradle will suit more than Buck and Bazel, because they have fundamental points, according to which they will not collect incrementally within one module, but there will be a Gradle, and for many it is very important. And it's cool that Gradle can do that.

Another thing is that when you add modules - more, more modules, eight hundred, thousand, - Gradle is so designed that it will linearly slow down the assembly in some places. But it seems to me that Gradle can fix it all, if the community puts pressure on them - which is maybe what I do. We'll see. (note: a few days after this interview, Artem wrote a big post about the problems of Gradle)

Oleg: That is, Bazel just because you want to support a large number of modules?

Artyom : Let's just say, in our case, we don’t want, but the division of the project into modules allows our business to move faster. Basically, as far as I understand, this is isolation, so that spaghetti cannot be obtained, which is then difficult to maintain. Modules give more control over which parts of the code they interact with. We have almost a million lines of code. If it were in one module, it would have to be spaghettied. Because on top of the language - Java, Kotlin - it will be necessary to cheat something to bar calls between packages, between which no one expected them. Plus, there will be a question that Gradle will not export such amount of code in one module. It will not be parallel to it, incrementally assemble inside the module.

Every solution has trade-offs. In our case, it seems to me, this is the right solution, but there is also a problem - that at the moment we support two assembly systems.

Oleg : And what is better for hundreds of modules: monorepo or many repositories?

Artyom : This is a very sensitive issue. Probably one repository is better from the point of view that you don’t need to think about versioning and there is no dependency hell when you go and clone a dozen repositories to make one change, and then open one pull request and another ten after it. “Friction” is removed from the system, and people are not afraid to change the code. For them, atomicity of changes arises: everything is committed to one project, and changes to one module are automatically transferred to the others without their explicit consent. In this case, all the checks that you automatically wrote in CI will be executed and verify that the code is compiled, tested and all this.

Oleg : And if you do not come to the fact that you, like in some Chrome, the branches will change for two minutes while you drink tea?

Artyom : Yes, of course, there is a possibility. But here, probably, the question is already in the size of the product: does Chrome need to keep in itself so much code? Maybe it is worth allocating some parts into separate instruments, which they will periodically pull up when major changes occur in them? This is probably a question for the organization of the project. Cool example, by the way. I have a similar one: correspondence with dudes from Yandex. Browser, where they also have big gags.

Chrome can be broken down into several components, and if you take a V8, I'm not a big specialist, but as far as I understand it, it could be a separate project, right? And why then to the graphic interface to know about the engine, each time to rebuild it and think about the fact that the source code should lie somewhere near? Bazel, by the way, supports this too.

In general, now all the big build systems - that Gradle, Buck, Bazel - support such a thing as composite builds when you refer, for example, to another Bazel build. This is a tricky situation, but, nevertheless, this works, it allows you to remove some of the files from the repository, to reduce the size. IDE, for example, will go crazy with indexing all these files, so I want to somehow separate them from the overall component of the project.

But we are still far from that. It seems to me that we can still calmly figure five more years. In the two-minute checkout, we are unlikely to resist. We do not have many people.

Evgeny : Does Lyft have its own specifics, besides the two assembly systems?

Artyom : Yes, there are a couple of atypical stories. It so happened that people who came to the company (from Google, Facebook, from everywhere), hate monorepositories. As a result, we have three mono-repositories in Lyft: Android, iOS and L5 (these are our autonomous cars ).

And the rest is more than 1500 git-repositories: for all microservices, for all libraries separately. So historically. It has its own huge price that we pay: it is really difficult to push changes through them. On the other hand, when working with each of them, you have instant git clone, instant git push, everything is very fast, IDE indexes in a second. I can say that this is the really interesting part. From the dudes from San Francisco, I would expect a mono-repository.

Oleg : And when one of these individual repositories is updated - the API changes, for example - how does this change apply to the rest of the company?

Artyom: It hurts. (laughs) Well, I am not a backend developer in the sense that I do not write feature backends, I write infrastructure backends — they are usually quite autonomous in this regard.

As a rule, this is just a bunch of rallies, cross-interaction and then planning.

Oleg: So rallies are part of the assembly system? (laugh)

Artyom : Yes, you first need to gather a rally, then build a repository. Plus, unfortunately, historically, we have many of these microservices - this is Python, which is also with its own jokes.

Oleg : I slipped some kind of dislike for Python.

Artyom : Rather, a dislike for dynamic typing. Python, not Python - it makes no difference, but dynamic typing is a sore thing.



Eugene : And it slipped "for a company from San Francisco", and it is curious to ask this: what is the difference from the point of view of developing companies from San Francisco from companies from Russia?

Artyom : Very big difference. I am not a big fan of this, but it seems to me that there is a more correct engineering school here.

Oleg : Where is this here?

Artyom : In Russia, in the countries of the former USSR. People pay more attention to the technical aspects of the components of their system. And in the States it often happens that a library solves a problem, and people don’t even look at how it is implemented. They, as a rule, absolutely no matter what it slows down or that they use it incorrectly.

I am there a lot of people talking, because this is part of the work, and the general level of knowledge, perhaps, is lower for now. There is something to change. Every time a person comes from Eastern Europe, it becomes more interesting at interviews, because people are not afraid to resist something, to argue somewhere. While candidates from the United States very often may not answer questions at all or answer "I do not remember when I last used it." For questions like “How does the HTTP request work?” Or “What data format will you choose?” They cannot give normal engineering answers, but they say: “Well, I've used this for the last five years.” Cool, of course, but the senor does not pull.

On the other hand, there are projects that have gone for years compared to what we are doing here. People make more mass products, and there is simply more scale. For example, Chrome or Uber - they already have more than a thousand modules there. This is just a scale of problems. Say, in Uber under three hundred Android developers. The question arises: why? (laughs) But, nevertheless, they managed to make this colossus work, constantly being released. I would say that such issues are solved less frequently.

Here is Yandex - a good example. I have a friend in Yandex.Maps: An Android application is made by ten people. In Google, most likely, a hundred sits. And at the same time, Yandex.Maps has more functionality. That's the difference in my opinion.

Evgeniy : In addition, Dolyna is also associated with startups, and they have the “move fast and break things” approach, and it seems that this should also have an effect on the development: live on the bleeding edge, use all the newest things. It's true?

Artyom : I did not work in startups, Lyft is hard to call it: there are already three thousand people, somewhere more than a thousand of them are engineers. That is, it is already an established company.

It is cutting edge technologies that are used quite rarely. If the technology is promoted, then yes. If the technology is niche, but cool - very often not. Until they talk about it at all conferences, very few people will use it.

But at the same time, I love very much (in San Francisco and partly in the Valley) - very many issues are resolved due to the fact that companies are physically close. Very often you write to someone in the little chaplet: "Let's have lunch together with us or in your office and decide, advance some question," and then once - and an open source project or pull request appears in another project, something is fixed.

What is interesting: people very often discuss things that generally should not be discussed by the NDA. But this is how the whole Valley moves, as a result, everyone understands where the rest are moving, and the whole industry goes together. Let's say Lyft and Uber mobiles constantly communicate about technical stuff, because we use open sources from Uber. And, of course, there are directly hardcore experts on some technologies. This is also cool: you can simply cross with them.

I love this, and I lacked it in some cities where I lived. Here in St. Petersburg there was a very cool Java User Group (I don’t know how it is now): you come after work, and Shipilev is carrying out your brain, and something is good!

And there it appears again: for example, there also has its own Java User Group, and there often come guys, say, from Oracle, who have written down some new Reactive JDBC. And you are sitting, arguing, because there is some kind of Project Reactor lead or Reactive lead in Spring in the same place, there is a really heated discussion, and this is great.

Oleg : I will ask about something else: I looked at the Mainframer repository , and Rust is used there. Why is all this written not on blessed Dzhavka, but on some Rust?

Artyom : I have recently resisted in the direction that the program should have a minimum amount of resources. That is, I want to be very close to how the iron digests the bytes. And in Java, a lot of things are happening around (I’m not even talking about garbage collection), that is, JIT and all this. I really like the fact that Java is now moving towards the fact that there will be more ahead-of-time compilation. It seems to me that it will be very cool, for example, to start microservice launch from what you download from the cache its ahead-of-time compilation, which initially occurred on some other machines, and start it very quickly, without warming up. This is great, but Java has a price. I can't just ask people who build an iOS project to have Java on the system.

Initially Mainframer was written in the Bash dialect. But I wanted to rewrite it in the system language to get normal multithreading, the ability to write normal unit tests, and not just integration tests on top of the utility ...

Oleg : And one could take, for example, Python.

Artyom : Yes. But then the question would arise from the fact that, firstly, it is dynamic typification, and, secondly ...

Oleg : So in Bash, too, dynamic typing.

Artyom : So I wanted to rewrite. And besides this, there is a problem with the fact that Python is now two: in macOS, the default is the second, and almost all in Linux is now the third. There are all sorts of such jokes. If I need to link some kind of dependency, that I will ask people to run pip? Or do I have to bang it?

I wanted to take a system language that requires zero dependencies so that I can put a binary that weighs, conditionally, less than a megabyte, and works with minimal overhead.

Oleg : It was possible to take Golang, at least there is a garbage collector.

Artem : That's exactly the reason why I wanted to try Rust. And earned. Plus, Golang is somehow sad with generics.

Eugene : Once started to discuss languages ​​... In the context of Android-development, the question "Kotlin or Java" was already tired, but still ask it in order to continue to the next question.

Artyom : Well, Kotlin, of course.

Eugene : Now the question that really interests. Recently Kortlin became stable in Kotlin, and voices "hurray, let's leave from RxJava" are heard. Therefore, when I see a person in front of me who is very close to RxJava, I immediately want to ask his opinion about the korutinas.

Artyom : I was very negative towards Korutin. In principle, it is still mostly negative, but this has partly changed a very long conversation with Roma Elizarov , who is working on them.

As a user of programs, I want them to be as non-blocking as possible, to use resources as properly as possible. By this I mean parallelism, and the fact that they use the correct operating system APIs for non-blocking calls to the network or files — there are many problems with operating systems, but, nevertheless, there are such APIs. What exactly is it solved? As a user, it doesn’t matter to me, as long as the developers solve this problem so that they feel comfortable. With this, I have no big problems. And this is the vision of Roma Elizarov. After this conversation, I somehow let go.

Before that, like my friend Arthur Dremov , after several years of using Java in production, this seemed like a step back: the code again becomes imperative, unclean, it understands the understanding of the pipeline, again becomes a mess that the compiler turns into an asynchronous mashio for you.

I do not use corutines, but all the examples that I am seeing now have moved to a structured approach, when you do not see at all which piece of code from this is coroutine. I, frankly, very scary to look at it. Because I open the pull request on GitHub, there are some methods called to load the image and profile, one of them goes online, and the other goes to the local SQLite, and now the local SQLite can easily turn out to be blocking. I don’t see this in the code, because the corutines are made so that you don’t see it. Maybe this is good, but for me this is so far a minus of design, because in the Rx-approaches it is very obvious: you understand, this is part of the synchronous pipeline or not.

Perhaps this is my only claim to the corintines: I want to see when I have asynchrony, and when not. Ideally, I want people to write more functional code when there are small reusable or at least test pieces that are combined with others. And we come back to the fact that it's all inline in logic, and then the compiler just pecks it.

Oleg : Give me a little bit of poppling. Legacy code is much more than new. And if we take some things like working with the network, working with files and so on, then no one will quickly rewrite all this, for example, using RxJava. And if we have autocircuits, then we can, for example, track all syscalls, automatically wrap them and send them to a lock there, to the parking lot.

Artyom : True, in any case, you will have to call functions from the context of corutin. But this is an interesting thought, yes.

Oleg : Maybe they somehow combine? The top-level API will be on RxJava, and the low-level API will be on corutinas.

Artyom : Yes, there are now such shifts. But then the question arises, because at the moment RxJava can do everything that the Korutinas do, and the Korutins can't do everything that RxJava does.That is, the first technology can absorb the second, and the second one - not. And therefore, most likely, there will be progress towards the fact that there will be some cross-platform Rx on Kotlin on Korutin. But this is a different mindset. This is how you can take forEach and drive to do some map, or you can take streams and write on them. And, it seems, even an enterprise Java-community has already approved and began to write streams, because it is more expressive. And with korutinami we are now going in the opposite direction.

And it seems that for Kotlin Korutiny is an addition. As far as I understand, for languages ​​like Go, this is the basis, initially a part of the type system, and all the APIs that they provide work the same way: the standard library uses a lot of cortinas there. And in your situation, Oleg, it turns out that you are writing a code that can be legacy for you, but it is asynchronous, and that's cool. And what we now get in Java and in Kotlin is legacy, and you no longer understand whether it is asynchronous or not asynchronous. One is better than another. It is better to have some kind of understanding of what is happening.

But, as I said at the beginning, as a user of the programs, I am pleased. The more tools that are more suitable for more people, are given to them in order to write more correct programs - the more I am pleased. Therefore, I have absolutely no complaints here.



Eugene : And the last question is quite general. After criticizing Korutin, whom many are very satisfied with, I would like to ask: what are the main problems of modern Android development? I wonder if this will also go against other people's opinions.

Artyom: An interesting question ... It is difficult to answer. I would say that, in principle, there are no special problems in Android development. There are a very large number of tools that you can use and get great quality programs. There is a problem in that it is difficult to scale to a large team: people need to correctly understand how this tool is used. And in this, RxJava loses very much to the Korutinas, because it is very easy to use it absolutely wrong: to use for some small asynchronous things and not to express logic everywhere in the streams. In this regard, korutiny, most likely, will go better.

I think it's interesting to compare with iOS. I am ashamed to say this, but in our Lyft iOS developers only this year introduce dependency injection and RxSwift. And now comes the 2019th. I know for sure the iOS commands, in which this is not the case, have been using modern approaches for a long time, clean, that's all. But it seems to me, Android in this regard is far from the worst platform.

Perhaps the only thing I don’t like is what Google is doing now. For a long time, their position was “we are not opinionated, use whatever you want: here is the framework for you, and how you use it is not particularly important to us”. Part of the community for a long time they kicked - and, in my opinion, in vain.

It was a golden time when you could say “RxJava is a solution, because ...” And now people come and say: “No, we will use LiveData”. You begin to ask why - she loses in all respects both RxJava, and Korutin, and anything. But, nevertheless, Google considered that this was important for the community.

Many in the community will now say, “It's cool that Google is promoting MVVM.” And for them, the third question is that this MVVM is absolutely crooked and wrong and, in my opinion, violates all the principles of how MVVM should be. And many projects have already switched to what Google recommends now.

It seems to me that they do not have the right feeling, where the scope of projects ends. Very often, the wrong architecture eventually scatters across several projects.

But at the same time there are some very cool work: for example, Room is very well made and generally very cool library. Any Architecture Components is a very controversial set of things that were already implemented in the community five years ago. Why did Google come so late, and with a crooked solution? These are the moments that strain me.

Yevgeny: I suspect that many have a burning desire to object to this place. Well, you can then do this in the comments. Thanks for answers!

Artyom: Very interesting questions.

Следующий Mobius состоится в Петербурге 22-23 мая . Сейчас его программа ещё не оглашена, зато это самый выгодный момент для покупки билета: уже 1 февраля цены на билеты вырастут. А также, раз программа пока не завершена, момент подходит ещё и для того, чтобы самому в неё попасть: мы вовсю принимаем заявки на доклады. Вся информация о конференции и билеты — на сайте .