January 30, 2017 at 10:43How not to waste time on continuous integration
Any software company sooner or later comes to the need for continuous integration, be it the development of a mobile application, the creation of a regular website, or even more so if it is a complex micro-service structure.
Earlier, I wrote that our team is developing turnkey iOS and Android applications, including a backend and a frontend. From this, there is an urgent need for a scalable distribution process and established standards for project management.
But it’s one thing when you have the only product where you can do it somehow, and another thing if every month a new, with special requirements, large team appears, for which you need to quickly get CI / CD.
You probably thought that now there will be instructions for setting up this very integration, where I will share best practices, scripts, infrastructure recommendations and other life hacks.
I thought so too.
But he suddenly realized that all this is just copy-paste on an industrial scale, and the right decision lies not with the instructions, but with the tool. A tool that analyzes, configures and builds projects on its own. In the end, continuous integration was designed to simplify life, and not add new problems with the need to create huge tutorials for writing the same exorbitant in volume scripts. And then also support.
For this reason, services like Travis and Circle make me happy a lot.for the capacity of their language, the rigor and accessibility of the infrastructure, but nevertheless, this is not enough. All this imposes responsibilities for the study of their guides, requires a deep understanding of the features of the platform on which the product is developed. It’s not enough just to be able to make iOS or Android applications, you need to know the features of the SDK and the device under their hood.
You can say that it’s normal that every developer should know their platform and be able to configure the assembly from the command line, but I do not agree. The developer must be able to develop, write a product, know the SDK methods, appropriate libraries and be able to apply them. But there’s no way to understand distribution systems, scripting languages and infrastructure. Just like before, to create a simple site, you also had to be a little system administrator. Exaggerating a little, but I think you understand.
In order not to reinvent the wheel, I had to dive deep into the Internet, where I expected to find existing services and test their strength. As everyone knows, an idea never comes to just one head.
I decided to take some test project, which I will give to the experiments for all the services found.
The success criterion will be the ability to conduct a full cycle of continuous integration - from obtaining source code to deploying in Crashlytics.
To begin with, we will focus on iOS, as I personally, as an author, have more expertise in this area, and secondly, the creation of apple has always been famous for its puzzling setting due to certificates and other obstacles.
As a sample for testing, let's take the good old projectwhich I am abused in each of my articles. And so, let's go.
After long diving right up to the 20th page of Google, I found three products that fit the description. Two of them did not start in principle, so we won’t even consider them, and the third one worked, but somehow: I couldn’t determine the appropriate version of Xcode, inexplicably long collected Carthage and Cocoapods (for the uninitiated - dependencies), and the impression was made I’ve never heard of a cache of any kind. And in the end, it flipped the assembly, as expected:
Oh, these tales. Without false modesty, I realized that I can do better now. Because in addition to one ambitious wording, there was also a clear understanding that competitors do not work where, how to fix it and make it generally better and faster.
Inspired by the idea, along with colleagues who supported the idea, we sat down to develop a prototype. I will omit the pompous narrative of how we worked day and night under 'Eye of the tiger', and instead of juicy marketing, I’d better tell you what we achieved, what chips we implemented and what really works now.
The main goal was to fully automate the configuration of continuous integration for everyone, so that even the manager could build a normal business process in his company, having only a link to the repository and with a complete lack of technical knowledge. We even conducted an experiment asking a person who is not developing for iOS to set up continuous integration using our creation. We were very glad that he succeeded. It was a signal that we were moving in the right direction.
A small demo video on how everything happens.
If compared with Travis, then with equal resources, projects are assembled 4-5 times faster due to optimization of dependencies and the lack of the need to virtualize each individual worker and deliver him the missing software.
I forgot to say. The project decided to call Buildben. At first they just wanted Ben, in honor of our wonderful office cat, and then a pun was born between Big Ben and Build Ben.
But we were distracted. You probably all understood that the class does everything by itself, the cat is cool, we are great and all that. As they say - what will surprise us?
But with what. We have specially prepared some special features that you will like:
At one point, we realized that we were cooking in our own boiler for too long, and almost no one had heard about the service except for a couple of acquaintances and friends. Yes, and the funds are not rubber, the cloud at Amazon will not pay for itself, you know.
It was decided to go look for an investor in the best traditions of startups. We didn’t have to look for a long time, which is surprising, we became interested and made an appointment.
They asked a lot of questions, asked about the experience as teams, what projects we developed,who we see ourselves in 5 years , what our target audience was, asked to count many different complex numbers, and a dozen other strange indicators, while reading lectures on how to correctly live. As a result, we were told the following:
"- Gentlemen, everything is great, the project is interesting. But there is one thing. You need users. Without them, there is no certainty that the project is really in demand and should be developed. Find at least a couple of dozen people and companies who want to use your services, and we will continue the conversation. “
And where can the techie find the techie who loves new products and is ready to go in the forefront? Of course, here at the Habr. My little acquaintance with the audience on several written articles has shown that the people here, although they like to criticize, are reasonable and inquisitive.
I not the seller and marketer and do not know how to speak beautifully about love, but if you read to this point, it means that you found it interesting topic, and we will be happy if you use it.
we have a smalllanding site where you can leave a request for beta testing, you are welcome. Each application triggers my alarm clock, and I run to respond immediately. But, unfortunately, our resources are now very limited, and we can not invite everyone, but we will try to the maximum.
At the end of the post I’ll add a FAQ, which will be updated as we communicate in the comments.
Security issues can be found there now. This is a particularly important aspect that everyone understands.
I also want to say and even duplicate that the development of a boxed version that you can deploy in yourself and forget about paranoia is among our top priorities and we know about its necessity.
Perhaps this is all I wanted to tell. I will be glad to answer questions, listen to advice and sanctify any aspects in more detail.
Oh yes, by the way, this is the same cat I was talking about:
Miracle, isn’t it?
And by tradition I want to share useful links:
UPD1: Oops! Unexpectedly for us, we received dozens of applications and a confident list of bug reports.
Forced to suspend the admission of new participants until next Monday.
We will pay attention to all new users, thank you for being with us.
Question: Why can I be sure that my code will not leak anywhere?
Answer: First of all, this is our main primary responsibility. We do not read your code and no one outside can access it, everything is hidden in an individual sandbox for each project. And even more than that, we immediately delete it after assembly, leaving only the logs. This is written in our confidentiality agreement, which you sign automatically by registering in our service. Those. We in the confidentiality agreement write that we delete the source code after assembly.
Q: Well, what about accessing iTunes Connect? How can I be sure that you will not break our application?
Answer:We recommend using a separate account for external access, regardless of whether you use our service or your own script. You can determine the border where you want our product to go. For our part, we are more than anyone interested in the safety of information.
Question: Do you plan to release a boxed version that you could deploy to yourself and not worry about security?
Answer: We plan very soon. This is one of the priority areas.
Earlier, I wrote that our team is developing turnkey iOS and Android applications, including a backend and a frontend. From this, there is an urgent need for a scalable distribution process and established standards for project management.
But it’s one thing when you have the only product where you can do it somehow, and another thing if every month a new, with special requirements, large team appears, for which you need to quickly get CI / CD.
You probably thought that now there will be instructions for setting up this very integration, where I will share best practices, scripts, infrastructure recommendations and other life hacks.
I thought so too.
Start
But he suddenly realized that all this is just copy-paste on an industrial scale, and the right decision lies not with the instructions, but with the tool. A tool that analyzes, configures and builds projects on its own. In the end, continuous integration was designed to simplify life, and not add new problems with the need to create huge tutorials for writing the same exorbitant in volume scripts. And then also support.
For this reason, services like Travis and Circle make me happy a lot.for the capacity of their language, the rigor and accessibility of the infrastructure, but nevertheless, this is not enough. All this imposes responsibilities for the study of their guides, requires a deep understanding of the features of the platform on which the product is developed. It’s not enough just to be able to make iOS or Android applications, you need to know the features of the SDK and the device under their hood.
You can say that it’s normal that every developer should know their platform and be able to configure the assembly from the command line, but I do not agree. The developer must be able to develop, write a product, know the SDK methods, appropriate libraries and be able to apply them. But there’s no way to understand distribution systems, scripting languages and infrastructure. Just like before, to create a simple site, you also had to be a little system administrator. Exaggerating a little, but I think you understand.
Study
In order not to reinvent the wheel, I had to dive deep into the Internet, where I expected to find existing services and test their strength. As everyone knows, an idea never comes to just one head.
I decided to take some test project, which I will give to the experiments for all the services found.
The success criterion will be the ability to conduct a full cycle of continuous integration - from obtaining source code to deploying in Crashlytics.
To begin with, we will focus on iOS, as I personally, as an author, have more expertise in this area, and secondly, the creation of apple has always been famous for its puzzling setting due to certificates and other obstacles.
As a sample for testing, let's take the good old projectwhich I am abused in each of my articles. And so, let's go.
After long diving right up to the 20th page of Google, I found three products that fit the description. Two of them did not start in principle, so we won’t even consider them, and the third one worked, but somehow: I couldn’t determine the appropriate version of Xcode, inexplicably long collected Carthage and Cocoapods (for the uninitiated - dependencies), and the impression was made I’ve never heard of a cache of any kind. And in the end, it flipped the assembly, as expected:
Oh, these tales. Without false modesty, I realized that I can do better now. Because in addition to one ambitious wording, there was also a clear understanding that competitors do not work where, how to fix it and make it generally better and faster.
Work
Inspired by the idea, along with colleagues who supported the idea, we sat down to develop a prototype. I will omit the pompous narrative of how we worked day and night under 'Eye of the tiger', and instead of juicy marketing, I’d better tell you what we achieved, what chips we implemented and what really works now.
The main goal was to fully automate the configuration of continuous integration for everyone, so that even the manager could build a normal business process in his company, having only a link to the repository and with a complete lack of technical knowledge. We even conducted an experiment asking a person who is not developing for iOS to set up continuous integration using our creation. We were very glad that he succeeded. It was a signal that we were moving in the right direction.
A small demo video on how everything happens.
If compared with Travis, then with equal resources, projects are assembled 4-5 times faster due to optimization of dependencies and the lack of the need to virtualize each individual worker and deliver him the missing software.
I forgot to say. The project decided to call Buildben. At first they just wanted Ben, in honor of our wonderful office cat, and then a pun was born between Big Ben and Build Ben.
But we were distracted. You probably all understood that the class does everything by itself, the cat is cool, we are great and all that. As they say - what will surprise us?
But with what. We have specially prepared some special features that you will like:
- Caching Carthage frameworks in the cloud. What is brought together once does not build again.
If someone has not encountered, then there is such a problem that many frameworks on github do not provide a compiled binary. And if provided, then only for a specific Swift. We have created our own library repository for each version of Swift, which is regularly updated automatically as we work. - Utility for independent configuration of the iOS developer cabinet. When deploying the application, you constantly have to think about certificates, profiles, bundle id, any entitlements, icloud, and others. We took all this upon ourselves and automated.
- Tool for preparing Crashlytics applications. Everyone who worked with Crashlytics knows that when you create a new application or at least just change its bundle id, you have to run it at least once in order to create it. And if you need to check the application with a new bundle id? Or just want to use the delights of distribution in krashlishik without trouble with the SDK?
So, now everything works as it should. It is enough to choose Crashlytics in the admin panel, then it is on.
We plan to make it possible to simply drive in the Fabric username or, if you do not have an account, then register it yourself without dancing with tambourines. - Asynchronous assembly. If something can be done at the same time, we do it. For example, Cocoapods & Carthage are installed in parallel on different machines, and then merged together. As a result, tasks that do not require powerful resources are performed by the corresponding micro-processes, allowing you to focus all the power on compilation.
The next step
At one point, we realized that we were cooking in our own boiler for too long, and almost no one had heard about the service except for a couple of acquaintances and friends. Yes, and the funds are not rubber, the cloud at Amazon will not pay for itself, you know.
It was decided to go look for an investor in the best traditions of startups. We didn’t have to look for a long time, which is surprising, we became interested and made an appointment.
They asked a lot of questions, asked about the experience as teams, what projects we developed,
"- Gentlemen, everything is great, the project is interesting. But there is one thing. You need users. Without them, there is no certainty that the project is really in demand and should be developed. Find at least a couple of dozen people and companies who want to use your services, and we will continue the conversation. “
And where can the techie find the techie who loves new products and is ready to go in the forefront? Of course, here at the Habr. My little acquaintance with the audience on several written articles has shown that the people here, although they like to criticize, are reasonable and inquisitive.
I not the seller and marketer and do not know how to speak beautifully about love, but if you read to this point, it means that you found it interesting topic, and we will be happy if you use it.
we have a smalllanding site where you can leave a request for beta testing, you are welcome. Each application triggers my alarm clock, and I run to respond immediately. But, unfortunately, our resources are now very limited, and we can not invite everyone, but we will try to the maximum.
Important Nuances
At the end of the post I’ll add a FAQ, which will be updated as we communicate in the comments.
Security issues can be found there now. This is a particularly important aspect that everyone understands.
I also want to say and even duplicate that the development of a boxed version that you can deploy in yourself and forget about paranoia is among our top priorities and we know about its necessity.
Conclusion
Perhaps this is all I wanted to tell. I will be glad to answer questions, listen to advice and sanctify any aspects in more detail.
Oh yes, by the way, this is the same cat I was talking about:
Miracle, isn’t it?
And by tradition I want to share useful links:
- Crashlytics is an incredibly convenient and free distribution system. Has good tutorials and a clear SDK for iOS & Android, I recommend.
- Mattermost is a chat for team communication, like Slack, only self-hosted and free. Easily integrates with everything you can. If you are still discussing business matters in a telegram, then here you are.
- Fastlane is a Swiss knife for continuous integration. If you are doing deployment yourself, then this is just a must have. True, it requires some knowledge of Ruby.
UPD1: Oops! Unexpectedly for us, we received dozens of applications and a confident list of bug reports.
Forced to suspend the admission of new participants until next Monday.
We will pay attention to all new users, thank you for being with us.
FAQ
Question: Why can I be sure that my code will not leak anywhere?
Answer: First of all, this is our main primary responsibility. We do not read your code and no one outside can access it, everything is hidden in an individual sandbox for each project. And even more than that, we immediately delete it after assembly, leaving only the logs. This is written in our confidentiality agreement, which you sign automatically by registering in our service. Those. We in the confidentiality agreement write that we delete the source code after assembly.
Q: Well, what about accessing iTunes Connect? How can I be sure that you will not break our application?
Answer:We recommend using a separate account for external access, regardless of whether you use our service or your own script. You can determine the border where you want our product to go. For our part, we are more than anyone interested in the safety of information.
Question: Do you plan to release a boxed version that you could deploy to yourself and not worry about security?
Answer: We plan very soon. This is one of the priority areas.