Cassandra Cluster Rescue Experience
I happened to save the Cassandra cluster that had gone into oblivion. It was an interesting experience that I would like to share, because in a normal situation, most databases work the same way, but the level of stress during a fall can vary very much.
The service in which Cassandra is used must store N last events for each user. Events come much more often than the user can read them and, in most cases, the recorded data will never be read, but will simply be supplanted by newer events. There are not very many databases in the world that work well in write-intensive tasks, but Cassandra is one of them. Writing to a cluster (with minimal consistency) is much faster than reading. Of course, it helps that you need to select the data only by the primary key - the user id.
The person who launched the service did not take the documentation seriously enough and did not balance the ring. The fact is that with the automatic addition of a node, it is allocated half of the largest segment at the time of adding. As a result, out of five almost simultaneously launched nodes, we got a very bizarre configuration in which two servers were loaded much more strongly than the other three. The size of the hard drive on all five servers was 260GB. Two nodes fell due to running out of disk space and the entire cluster was drowned in load. The documentation warns many times that it is impossible to allow automatic selection of ring segments in production. Provides a formula and PHP code for the manual calculation of tokens.
Firstly, with Cassandra turned off, you can do anything with its data files. We moved one of the heavy and old files (30GB) to NFS and put symlink on it. We started the cluster, checked the service - it works. The total repair time from the moment the problem is detected is 15 minutes. Almost all this time it took to transfer the file to NFS.
Secondly, I immediately turned on caching in the database. Cassandra has a pretty decent caching mechanism that significantly reduces hard drive access. At least in our application, cache hits were between 80% and 90%.
Note: the cache size is specified in the "records", not bytes. You need to correctly understand the size of the average record so as not to miss. Of course, I missed, highlighted more available and after a few hours I got the first node that crashed from Out Of Memory. It is treated with a simple restart and a more careful cache size.
So, the service came to life, copes with the load, but of course it is impossible to have an unbalanced cluster with a part of the data on NFS. After reading the documentation, an attempt was made to use the nodetool move command. I tried to get her to work for almost a week. The essence of the problem was that the data between the nodes did not move. The streams directory appeared on the source node, which contained the data prepared for the transfer, but the transfer itself (which can be viewed using the notedool streams command) always hung. Sometimes even at the very beginning.
So for the first time I ran into bug 1221 . After reading the fix, I tried to upgrade to the latest version, but then bug 1760 overtook me. As a result, I still updated the cluster to 0.6.5, but it didn’t help much. The cluster is stuck in an unbalanced state.
I must say that the tools for managing the cluster are not just miserable, but rudimentary. You can give just a few commands and monitor their process by indirect signs. That's all.
To my great joy, at this point the management forked out a training seminar from Riptano . This company is Cassandra, they develop it and provide paid support. At this workshop, the Cassandra Tao opened to me.
Do not try to treat anything. Falls - finish, clean, turn on as new. That was what was said at the seminar. This explains the rudimentary nature of cluster management tools. The fact is that, according to the idea of the authors, management consists of two main operations - 1) add a node; 2) remove the node.
In this way, in the end, I managed to fix the cluster. The nodes were removed one after the other from the cluster, cleaned, and started with correctly counted tokens. Of course I had to sit behind the paper thinking up such a restart order that would allow me to do without downtime. It turned out to be an interesting, albeit not very difficult, combinatorics problem.
Of course, there were some bugs. I was haunted all the time by 1676. A loading node received 50GB of its new data and sat quietly on. Restarting the service resulted in the next 50GB. And so until everything comes.
It turned out to fix the cluster. My opinion about Cassandra has changed from “what a student handicraft, no tools” to “surprisingly stable database”. In fact, for two months the cluster worked with damaged servers - for two, the speed of accessing the HDD was slowed to the speed of NFS. And while the service as a whole lived and users did not really complain.
During this time, I learned a lot about the insides of this database, talked with its creators (surprisingly responsive and smart people), and even, towards the end of the process, I enjoyed it.
about the project
The service in which Cassandra is used must store N last events for each user. Events come much more often than the user can read them and, in most cases, the recorded data will never be read, but will simply be supplanted by newer events. There are not very many databases in the world that work well in write-intensive tasks, but Cassandra is one of them. Writing to a cluster (with minimal consistency) is much faster than reading. Of course, it helps that you need to select the data only by the primary key - the user id.
Something went wrong
The person who launched the service did not take the documentation seriously enough and did not balance the ring. The fact is that with the automatic addition of a node, it is allocated half of the largest segment at the time of adding. As a result, out of five almost simultaneously launched nodes, we got a very bizarre configuration in which two servers were loaded much more strongly than the other three. The size of the hard drive on all five servers was 260GB. Two nodes fell due to running out of disk space and the entire cluster was drowned in load. The documentation warns many times that it is impossible to allow automatic selection of ring segments in production. Provides a formula and PHP code for the manual calculation of tokens.
nodetool ring
161594151652226679147006417055137150248
X1 Up 106.92 GB 38459616957251446947579138011358024346 |<--|
X2 Up 261.58 GB 87228834825681839886276714150491220347 | ^
X3 Up 268.08 GB 136691754424709629435450736264931173936 v |
X4 Up 148.58 GB 151190524462851319585265604946253553766 | ^
X5 Up 72.71 GB 161594151652226679147006417055137150248 |-->|
Resuscitation
Firstly, with Cassandra turned off, you can do anything with its data files. We moved one of the heavy and old files (30GB) to NFS and put symlink on it. We started the cluster, checked the service - it works. The total repair time from the moment the problem is detected is 15 minutes. Almost all this time it took to transfer the file to NFS.
Secondly, I immediately turned on caching in the database. Cassandra has a pretty decent caching mechanism that significantly reduces hard drive access. At least in our application, cache hits were between 80% and 90%.
nodetool setcachecapacity App Feeds 200000 1000000Note: the cache size is specified in the "records", not bytes. You need to correctly understand the size of the average record so as not to miss. Of course, I missed, highlighted more available and after a few hours I got the first node that crashed from Out Of Memory. It is treated with a simple restart and a more careful cache size.
Attempted Treatment
So, the service came to life, copes with the load, but of course it is impossible to have an unbalanced cluster with a part of the data on NFS. After reading the documentation, an attempt was made to use the nodetool move command. I tried to get her to work for almost a week. The essence of the problem was that the data between the nodes did not move. The streams directory appeared on the source node, which contained the data prepared for the transfer, but the transfer itself (which can be viewed using the notedool streams command) always hung. Sometimes even at the very beginning.
So for the first time I ran into bug 1221 . After reading the fix, I tried to upgrade to the latest version, but then bug 1760 overtook me. As a result, I still updated the cluster to 0.6.5, but it didn’t help much. The cluster is stuck in an unbalanced state.
I must say that the tools for managing the cluster are not just miserable, but rudimentary. You can give just a few commands and monitor their process by indirect signs. That's all.
To my great joy, at this point the management forked out a training seminar from Riptano . This company is Cassandra, they develop it and provide paid support. At this workshop, the Cassandra Tao opened to me.
Tao Cassandra
Do not try to treat anything. Falls - finish, clean, turn on as new. That was what was said at the seminar. This explains the rudimentary nature of cluster management tools. The fact is that, according to the idea of the authors, management consists of two main operations - 1) add a node; 2) remove the node.
In this way, in the end, I managed to fix the cluster. The nodes were removed one after the other from the cluster, cleaned, and started with correctly counted tokens. Of course I had to sit behind the paper thinking up such a restart order that would allow me to do without downtime. It turned out to be an interesting, albeit not very difficult, combinatorics problem.
Of course, there were some bugs. I was haunted all the time by 1676. A loading node received 50GB of its new data and sat quietly on. Restarting the service resulted in the next 50GB. And so until everything comes.
Conclusion
It turned out to fix the cluster. My opinion about Cassandra has changed from “what a student handicraft, no tools” to “surprisingly stable database”. In fact, for two months the cluster worked with damaged servers - for two, the speed of accessing the HDD was slowed to the speed of NFS. And while the service as a whole lived and users did not really complain.
During this time, I learned a lot about the insides of this database, talked with its creators (surprisingly responsive and smart people), and even, towards the end of the process, I enjoyed it.
