Saydkik suicide

It just so happened that there was a need to prematurely stop the already started sidekik-worker . And, as everyone involved already knows, the launched task cannot be stopped by regular means - this is simply not provided for by the architecture. And when the sidekick task began to be completed, then nothing would stop it. Of course, the Internet immediately found a solution with the killing of the ruby ​​process and with the abolition of restarting it , but this solution, for obvious reasons, can not suit either application developers or sidekick developers.

As a result, the solution came from where they did not wait and it turned out to be extremely simple and obvious. In general, the idea can be formulated very briefly: the sidekick process must kill itself, and we only need to tell it when to do it. All the code that used to just run in the sidekick will be run in a separate trade inside the sidekick process. And, in parallel with him, we will launch a trade that will follow the instructions from the outside so that the sidekick process will kill itself. Having received instructions to kill himself, the tracking thread kills a neighbor and kills himself.

In general, this approach is not limited only to using sidekick and long-running garbage can be anywhere. Therefore, let us abstract from the sidekick and try to generalize our approach.

First, you need to decide how we will tell the killer thread the news that we need to die. And in the case of Ruby, it is best to use an external messaging mechanism and radishes in this case are ideal. And it is better not to use the message transfer mechanism built into the radish for several reasons, the main of which is the lack of delayed reading. If for some reason the trade killer misses the message, then our murder process may not take place at all. Suppose we ask the sidekick process to die very quickly - right after we start it. There is a chance that the trade killer will not exist at all.

So, the message verification process is quite simple and straightforward:

  until $redis.del("sidekiq:killer:#{self.identificator}") == 1 do
    sleep 0.1
  end
  main_thread.kill

And the method identificatorwill be responsible for the unique key component in the radish.

The main trade with the code name "victim" after its natural death should leave to make sure that the killer will not wait for him forever:

  begin
    self.perform_without_thread(*args)
  rescue
    $redis.set("sidekiq:killer:#{self.identificator}", 1)
  end

Again, like a real killer, our trade killer must be sure that the victim is dead:

  until !!main_thread.status == false do
    sleep 0.1
  end

Well, in the end, our common sidekick process is waiting for the completion of all two processes:

  [watcher_thread, main_thread].each(&:join)

Now, let's put the above code together to get a complete module for direct addition to existing long-running processes:

module SidekiqKiller
  def perform_with_thread(*args)
    main_thread = Thread.new do
      begin
        self.perform_without_thread(*args)
      rescue
        $redis.set("sidekiq:killer:#{self.identificator}", 1)
      end
    end
    watcher_thread = Thread.new do
      until $redis.del("sidekiq:killer:#{self.identificator}") == 1 do
        sleep 0.1
      end
      main_thread.kill
      until !!main_thread.status == false do
        sleep 0.1
      end
    end
    [watcher_thread, main_thread].each(&:join)
  end
  alias_method_chain :perform, :thread
end

In our system, the mechanism of ordered suicide of workers was used to force stop the worker of a separately launched instance of a staging server. The mechanism, of course, is trouble-free, but too expensive from memory. As a result, this part of the system was rewritten in pure ruby, optimizing and getting rid of sidekick and related libraries.

Which of these can be concluded? No, except that we are all mortal. Still, probably the fact that you should not do this if there is not a very urgent need.