How to get to Hell because of Helm, but grab onto a straw

- To tired hipsters we speak the truth.

All of us (that is, I) love to pull everything new and brilliant in production in order to finally replace some problems with others. This story is dedicated to us (that is, to me).

To understand and forgive further text, you will need superficial knowledge of how to work with Kubernetes at the user level and any rumors about Helm .

Let's first abstract, and then let someone deal with it. Imagine for a moment that we are a kind of Columbus in the world of smoothies, electric scooters and Kubernetes. Our people are crowding Europe in an overpopulated old woman in one of its small, endless states, deploying deployment with Kronjobes on a daily basis under the crunch of quadcopters. But astronomers have already found an explanation for the suspicious curvature of the horizon. And here is a feeling that we are the same Chuzen Van of modernity. (But no one asked.) And somewhere there, beyond the semicircular ocean, is the shortest way to India! Endless spaces, freedom from the burden of routine and many free flavored spices. You just need to bring our people there and finally release them! Pushing the oceans in front of you is too much even for such magnificence as we are. Therefore, you need to build ships, pack our deployments, Kronjobas and other demons, raise services instead of sails and steer straight into the light there. To steer need a steering wheel, that is Helm. He is a helmet. A helmet would surely come in handy, if we know ahead what awaits us ahead. But there is only a steering wheel. Building ships is not easy, so we need help, as it were, but our people are always busy with something very useless. Therefore, it is necessary as if by oneself and as always. Slowly we start, we build a small boat, we row to India, we see the vast expanses with our own eyes, we take one spice, we float back, we show people. People like it, they bless our attempts. Then we are preparing a great outcome, we put in order the configurations, we are building many ships already. People come up, are interested. Show, tell, educate, promise, promise, we promise ... The more ships - the more interest. Someone joins, helps. To all this was not idle, swim instaging nearby, there are no spices yet, it is impossible to live there, but tourists like it. And now, after weeks / months / best-years / sleeves he, THAT DAY, has come! It is time to release the flotilla and subdue the treasured terra incognita. With trepidation in our hearts, we sail our people to India, and there, nutyponel, America. And everything seems to be very similar, but this is a premonition ... Our people, of course, immediately come to taste (haha, they don't care at all anyway, in fact), they begin to explore the territory, cabins there, cabbage, all kinds of services are deployed. But here and there, something periodically disappears, as if through the ground. That floor will disappear, the coconut will not give birth. And we are mumbling with the growing roar “ wait, wait, I'm not ready yet, give only a day... ” And still so we squint attentively, and there, in front, behind every bush SUDDENLY, the Indians with tomahawks appear and look so ... unkindly or something ... And the chill runs straight through the inter-aural ganglion. And the people are like this: - “And what is it there? And where is my floor? ”And we are:“ My people! It’s not that I’ve misled you for 40 years, and no one, of course, will bury you back, but it’s urgent right away, *****, IT IS NECESSARY TO BUILD THE FREQUENCES !!!!!! ”And they are: -“ And , well, of course, as always, again. "

That's how I felt when I transferred a certain number of services in production from pure Kubernetes to Helm, and then ran into it .

Well, at the end of the promised saving straw. First, the light version, but with an explanation of the above delusions. Here is a demo script:

1. Suppose I deploy a chart for project: 1.5. For the first time with Helm, and before that was just Kubernetes.
2. Then I find that there is a bug in the release, and in version 1.4 it was not. And it would be necessary to roll back, but for her and Helm the chart was not there either. Therefore, I decide to do the old fashioned way: kubectl set image deployment/project project=registry.project.com/project:1.4 --record. For this and for the pack of other services that unfolded together.
3. Then it turns out that the bug is not in this service, but in the next one, and everything is fine with this and it is necessary to return 1.5. Now, if you call helm upgrade --install, then a big surprise is waiting ( more details ): the image will still be from 1.4, and the labels - from 1.5. And Helm shows that everything is fine, actually 1.5 has been deployed there and even the pods have been restarted (CI-build is green).

How to avoid it? If any change is made to any K8s resource controlled by Helm with pure kubectl commands on top of the Helm Chart deployed, then these changes should also be canceled with the kubectl commands. Helm can deploy a new chart. But he compares the new chart with the previous one, but not with the current state of the resources. And if you have edited the image, then the future version of Chart will probably contain another image and everything will be fine. But if you edited the environment variable, or launch arguments, or something else, then the new version of Chart most likely does not differ from the previous one. And your manual changes will remain in place after the update.

And for a snack, a heavy version of the saving strawfor those who cannot agree with this unpredictable state.

New technologies - the source of new sorrows.