Interview with Moses Uretsky, co-founder and director of Digital Ocean
How did the DO idea come about? There have already been thousands of hosting providers on the market, not to mention such giants as Amazon, Google, Microsoft. Surely everyone said that your idea will fail?
Before DO, my brother and I had been hosting for many years, and at some point it became clear that everyone was moving towards the cloud. Many companies started much earlier than us - we ourselves then worked with various providers. They all built their clouds as they considered necessary and correct, but it turned out somehow unreasonably difficult.
So we decided that we would take care of the clouds and do our own thing, create our own version, which we would like ourselves - which, probably, was not very reasonable. The fact is that everyone with whom we discussed this said that it was a bad idea and we should not take it at all :).
So, perhaps, in this story there was no magic. There was an area well known to us in which we already worked a lot. We did not like the solutions presented on the market, and we wanted to make our own. So the idea came up to create something that we could use ourselves and see if anyone else wants to use it too.
Who were among the founders except you? Were you all programmers?
The founders were me, my brother Ben, Jeff Car, Alec Cartman and Mitch Weiner. We were all developers, but with different skill sets and different focuses. Ben is more knowledgeable about networking, system administration, and configuration management. Jeff is a specialist in backends, low-level programming. I have a mixture of front-end development and system administration, plus I have an artistic background, so I look after how our product looks. Mitch is our marketer, but even in the past he worked a lot in development and managed to found a small company producing a CRM system. Alec is very good at Rails.
At the very beginning, our team was enough to solve all the problems, but then, of course, new problems began to appear and had to expand. Still, building a cloud is much more difficult than just writing software and deploying it. For example, much is tied to physical infrastructure and a bunch of servers.
Initially, you wanted to take some ready-made technologies (like OpenStack)?
At the very beginning of development, we really looked at what could be found in the public domain. We chose between CloudStack, OpenStack, Eucalyptus and a few more projects, but not one of them caused delight. With any of them, I would have to redo everything for myself. Still, when you create a high-tech product focused on developers, all internal technologies and APIs are no less important than the interface. Even a novice developer sooner or later goes beyond the standard admin area, and then the fact that the product is “under the hood” comes into play.
Success Doesn't Come Right Away
How did you get the first users? Did it work right away?
We then worked in coworking and just put a bunch of flyers in the elevator and invited people to try out our new Digital Ocean service and get a free cloud server for testing. Of course, no one came to us, three people registered, and this could all have ended. In general, the launch was somehow not very successful :). However, we invested so much time and effort into DO that we decided to continue working.
We suddenly had the chance to showcase DO at the New York Tech Meetup hosted by Meetup.com. Among those present there were 700-800 very technically savvy people, all of them were delighted. I remember when we conducted the demonstration, we tried to test a couple of new things, for example, tried to build integration with GitHub, but everything was still not quite ready, so on stage we had to show the fake and present it as a beta version.
Then, during the after-party, people came up to us and asked questions, they were delighted with what we were doing. We have registered fifty people. So our platform was used not by five, but by fifty people. Yes, it has been a long way.
Have you turned to investors, tried something else?
Yes, for example, we decided to try participating in TechStar and even became finalists of New York City TechStars, but David Tisch told us: “I don’t quite understand what you are doing, guys, because I'm not a techie. So I'm not sure I can help you. ” Be that as it may, he advised us to take part in the Boulder program (boulder.me). We turned to them, and they agreed to conduct a technical examination. But in the end, we heard all the same things that other investors had told us before: that Amazon AWS is our largest competitor, that there will be no success, that we won’t succeed. We answered: “Okay, you“ really helped us ”” - and continued to do what we were doing.
And then, in January, a few months later, we still started. The response arose gigantic and almost instantly. Before that, we had five or six people connecting, and suddenly everything just exploded - hundreds of developers registered every day. We realized that we need to immediately register a company, hire support for users, make sure that there are enough servers in the data center, and so on. This process, in general, has spun and does not stop to this day. We very quickly increased the company’s staff to one hundred people, but, in fact, we are doing everything the same as before, just new challenges arise. At the same time, you should try not to upset customers who believe in us and love our product. We try to do our best to repay them the same and continue to please them further.
Digital Ocean Inside
How did your technology stack change over time? Are you really using Go for development?
Any company goes through the same phases. Everyone starts with a minimum product (MVP), they release a prototype that they could create with a small number of people at their disposal. In such a situation, they do not start from normal planning, but primarily from what a particular person wants to develop and how he will write it. Because of this, you really can't imagine when you run into a problem.
In particular, in January 2013, when everything began to spin and began to grow exponentially, we had to rebuild the entire system, because it was becoming more and more complicated. In particular, we distributed all services so that in the event of a fall in one of the services, the entire system does not crash.
In general, we killed a lot of time in order to rebuild the code and re-plan the entire architecture. And then Go came into our field of vision, because it is a very fast, new language, in which there is a lot of interesting things. I think it’s very rare to have an ideal test site for such things. And when you have thousands of servers scattered around the world, Go very naturally falls into this distributed system.
In fact, we went through all the agony of growth, our work was not limited to creating new chips and finishing the product. For example, planning looked as follows: we evaluated our scales, multiplied them by ten and relied on this figure while working with our architecture. And almost every time we practiced this “exercise”, Go fit perfectly. In addition, people who are interested in Go, as a rule, are very good developers, and they are interested in the problems that we are working on. Thanks to this, we were able to recruit a large team of excellent engineers, ready to both support existing products and make new ones.
Shark Sammy, the mascot of the service
What is now under the hood of Digital Ocean? Frankly, I am surprised that you use all of your own.
We really almost do not use third-party tools. The exception is some low-level tools, but they are few. Everything related to management, planning, events, account management, we do ourselves.
I don’t argue, OpenStack is an interesting thing. But his openness is greatly overestimated - after all, this project is being developed by a commercial organization, and this is somehow wrong. This is not a typical open source story in which two or three people got together and wrote something in their free time, and then the community supported them, helped them, and everyone began to use the product. There are many examples of projects that were not very supported by the community and developed mainly thanks to companies. In almost all cases, everything ends the same way: the developers try to please everyone at once, too many interested parties participate in the process, as a result - complete disorganization. We hear from all sides about problems with OpenStack and that after the deployment, all the time the developers spend on catching bugs and trying to make everything work fine. We believe,
What iron are you using?
Mostly Dell and SuperMicro. They supply reliable iron, so we have been working with them for several years. They treat us very well and try to support us in every possible way. We have to make purchases in various countries, and each new data center only adds to the difficulties with logistics, ordering servers, their delivery, assembly and so on. Therefore, it is very convenient to have a partner like Dell, who, as a rule, has a representative office in each country. For example, we recently opened a data center in Amsterdam, and it took 12 days to deliver a batch of servers. Previously, it took us about two months. Therefore, we would be happy to build our own servers, but now for us the main priority is logistics.
Security and other issues
For any PaaS and hosting, security is a headache. Especially when all sorts of nasty things begin to be done from your capacities. How do you track it, how do you fight?
Security is a big problem. We talked with different companies and startups, asked them what figures on abyuz and fraud they observe. We spoke with many large companies in New York. About 1-2% of their transactions are fraudulent. But a couple of percent is nothing, because we have to deal with 30-40%! That is, every third registered user of Digital Ocean is fake.
This is really a huge problem affecting the entire industry. If you have low-level access to the server, you can do anything: scan ports, DDoS, file phishing, send spam, and so on. That is, the list of possible nasty things is almost endless, but many attackers do not even understand that they are harming someone. Many of them just learn to write code and understand technologies, for this they are engaged in various dubious experiments. They do not understand that their actions cause a lot of problems, they are just curious. Our job is to identify such incidents and not interfere with “law-abiding” users. Unfortunately, this is more an art than a science.
But have you achieved any success?
Of course, we use many different technologies. Sometimes we cooperate with other companies that specialize in solving such problems, but we do a lot of automation ourselves. However, even companies for which this is the main work, where megaums, geniuses work ... even these companies will tell you: “Listen, we will gradually do this, we will work day after day, but still we do not guarantee you a 100% result. As soon as we find a solution, the attackers will figure out how to get around it. "
But we understand this, we treat this normally.
The most unpleasant situation is when, as a result of automated tests, a normal user is marked as an intruder.
Naturally, for them it becomes a complete surprise and leads to various problems. We try to communicate with such users, explain to them why we do this. Unfortunately, this, of course, spoils their entire user experience and afflicts them, and such users have every right to be angry and indignant. But, alas, if you do nothing, then there will be even more problems, as a result, all users will suffer in general. It is simply impossible to get around this issue. Therefore, you need to continue to work, try your best.
By the end of 2013, Digital Ocean surpassed AWS in terms of growth in the number of web servers - according to Netcraft. Naturally, the total number of instances is not taken into account here.
When it comes to large-scale projects such as DO, it may be silly to ask about the main problems that you have encountered. But still - what could you single out?
I think that the main problem is not in what language to write the product, but how to design the optimal architecture. In our case, the main problems are associated with the unification of distributed systems - you need to constantly optimize how the data is synchronized across different parts of your architecture. And the wider the geography of your project, the more acute this problem is.
We had an interesting case when we opened a representative office in Singapore. Before that, we had a central place where we stored data, since the delay between the East and West coasts of the USA and Amsterdam was the same and nothing particularly slowed down.
But when we arrived in Singapore, it became obvious that the delay to Singapore was clearly much higher than we expected. Plus, in addition to the lag problem, there is the problem of channel reliability - this is another problem that is solved within the framework of distributed architectures.
In a word, this is a very interesting set of problems.
When everything continues to grow, even the most commonplace problems reveal new aspects. Collecting analytics from virtual machines on such a scale is an interesting challenge, especially when you want to receive data in real time. At a minimum, you need to make sure that all notifications are sent correctly and in a timely manner. So, even realizing the most basic things, on such a scale you have to understand the smallest details.
I use the API quite actively. Over the past three years, he has changed twice. What was the problem with the original API?
There are mistakes that are somehow made; something turns out well, something worse. The first API was amazing, it was used by a huge number of people around the world. But our clients have grown up and over time have discovered a number of limitations. There were two main things that we wanted to change.
First: we really wanted to make the API completely RESTful. The original version could not boast of this, which created certain problems with Google, writing wrappers and so on. This did not satisfy many.
Second: users began to do things with our API that we ourselves would never have thought of. For example, people began to write mobile applications for working with DO. Our API did not support OAuth, so for authorization I had to copy the ID and the long key into the application - obviously not the best user-experience. All this we took into account in the new version.
The new version generally focuses on integration. This made it easier to think about developing new services and smart applications. Now they can be flexibly assigned various roles and access rights. In a word, we began to think of our users not as individual developers, but as a large unified ecosystem.
What is the coolest feature in DO, in your opinion?
I believe that the coolest feature of any product is not what you see, but what you do not see. In other words, the best features are those that we did not implement. This is especially important when it comes to a developer-oriented product. When you do something for advanced users, there is always a desire to cram more functions and settings. But the result is an uncomfortable interface and poor user experience. So I think our main advantage is the ability to find balance.
We always think about why to add this or that feature to the product, how can we get rid of it in case of something. If the product is already overloaded, we are thinking about how to simplify it, how to take part of the load and difficulties onto ourselves, and not force our users to deal with all this on their own.
Why is community important?
I love the way you work with your community and users. Each time you write to the caliper and get a response essentially from a person who is clearly in the subject. Around DO, a serious community has already formed; how did you manage to do this?
Community is undoubtedly one of our priorities. We have three main principles: love, simplicity and community. It all starts with love - we ourselves must love our product, because even if we don’t love it, why should someone else love it? You also need to love your users. When these conditions are met, you understand that you can do a lot to help people. I think we are concentrating so closely on this, because without the support that the community provides us, we would not have succeeded.
It's not even about the community around Digital Ocean, which, of course, is just great. The point is that, for example, without the possibilities that Linux provides today, there would be no DO. Linux provides virtual servers, without Linux we would not exist at all. We use programming languages for which we do not make any deductions, we do not pay any authors, copyright holders and so on.
It seems to me that for our generation and subsequent it is already a way of life and a given, once upon a time there simply was not such a thing. Without this, creating a company would be much more difficult. Now many say that the cost of creating a company has become much less, and this is largely due to open source. For example, if you decide to launch your own cloud hosting today, you can either pay a ton of money or turn to open source alternatives.
And what do you, for your part, do for the community?
We sponsor several conferences to personally communicate with users. You must always stay in touch with the community, because if you do not do this, very soon you will cease to understand what is important to them. They are wonderful, they give us so much love and support, inspire us to work further. And the great thing is that they are extremely honest. When we make mistakes, they almost immediately tell us: “Hey guys, you messed up,” and it’s very cool. After all, when you make a mistake, the best thing you can do is to admit that you made a mistake and work to correct it. This is a very high bar, but we have to compete with companies worth billions of dollars, they force us to compete at a high level and motivate us to do it in a positive way, and not sit aside, looking at these giants, and think: "We will never reach such a level." They literally force us to remain in good shape, to keep our promises. It seems to me that we owe our success precisely to this. If we continue in the same vein, I believe we will achieve even more.
First published in the Hacker magazine from 08/2014.
Subscribe to Hacker