August 21, 2014 at 01:19An instructive story about what can happen to a site on shared hosting
The working day was slowly but surely coming to an end. Sunlight streamed through the blinds and flooded the office with golden scarlet. Somewhere in the corner the coffee machine was buzzing, squeezing the rest of the coffee from the capsule. Our project discussed something animatedly with the designer, and I ruled the jambs kindly left to me by the junior programmer.
And everything seems to be fine if not for the message: “What about the T site?”.
One of my good friends noticed that one of our sites displayed incorrectly and let me know.
I dropped everything and loaded the page with the site. No problems were observed on it. Well, perhaps a small redesign was required ...
"Everything is in order with the site" - I wrote to a friend and again plunged into the fascinating world of code and bugs.
After some time, my brain nevertheless removed from my memory the alarm signal of my friend and I involuntarily climbed to re-check the site. For some reason, confident in the performance of the site, I experienced a slight self-confidence.
The main page of the site, sadly met me with a lost encoding and a complete lack of css. The black symbols of abracadabra on a virgin white background brought me back to reality. Self-confidence instantly disappeared and I began to feverishly push CTRL + F5 into the keyboard.
“It's just a cache ... Yes ... Just a cache ...” I repeated to myself over and over again.
When I realized that my tenth attempt to reset the browser cache does not give any results, and the main page of the site continues to change its appearance with every click on the treasured keys, the first thing that flashed through my head was “Hack ?!”. My fears began to grow stronger, when after the next page reload, I saw a red and white inscription saying that such and such a table was not found.
Hands themselves set out to change everything related to the site, accesses: admin panel, database, SSH.
After, I began to carefully study the logs. Although, the logs are loudly said. At my disposal were only reports on the operation of the web server. Logs of MySQL crashes and unsuccessful authorization attempts via SSH are not issued on shared hosting.
Nothing strange was found in the logs, and I went straight to the SSH console in order to connect to MySQL directly, because I clearly remember that the site cursed at the lack of tables in the database.
Very strange, but all the tables were in place (at least the ones the site was cursing for sure were in place).
As a CMS, on site T, 1C-Bitrix is used. We are very fond of this system and in every way admire it (with rare exceptions).
Imagine my surprise when, having entered the site settings, I saw data from a completely extraneous site R. The path to the root, domain name and some other settings were changed. I, stunned, looked at the monitor with eyes wide with surprise. At the same time, somewhere behind me, our project was already pouring valerian tincture. For some reason, I pressed F5. What happened after that plunged me into the deepest shock and despondency. The site settings returned to their normal values.
At this moment, for a moment, I doubted the adequacy of the entire IT industry as a whole and began to believe in miracles. My wife’s phone call got me out of a stupor. I picked up the phone. I still can’t remember what she was telling me, because the feeling of something mysterious did not leave me and my brain was frantically trying to find at least some explanation for what had happened.
I mumbled something into the phone and hung up. Well now, at least I'm sure of something. I am sure that at home I will find a lot of interesting information about myself.
But then, and now we need to understand what is happening.
After the next reload of the settings page, I again saw other people's values.
Judging by the absolute path to the root folder of the web server, we registered the settings for the site, which was located on the same hosting as us.
I instantly felt for the handset and called the technical support of the Hosting.
From the conversation it became clear that they are not able to solve our problems right now. The technical support officer politely made it clear to me that I should write an appeal by e-mail and my request will be considered within 24 hours, in the order of a live queue. After I, just as politely, expressed to them everything I think of them, I hung up and wrote them a letter full of epithets. Just in case, we also wrote a letter to tech support 1C-Bitrix.
The dead silence of the office was broken only by the ticking of a large wall clock and the barely audible noise of the system unit cooling system. A round-white snow-white clock, performed several functions in the office. Firstly, they were an element of the interior. The furniture in our office is not much, nothing more. Although, I absolutely do not mind buying a large and soft sofa, so that in between tasks, you could lie down with a cup of coffee in your hands and, legs stretched out, immerse yourself in the dream that someday we will move away from website development and start developing interesting and of course innovative, in terms of gameplay, games.
Secondly, this snow-white round clock did fulfill the function inherent in them - they showed time. If someone needed to find out the time, then he certainly turned to this watch. They have some inexplicable and almost hypnotic attraction. And nobody cared that the smartphone also had a watch on hand or on the monitor screen.
So at that moment, our pedantic overseer inexorably counted the time with each step of the second hand.
And we all waited for that, right now, right now, our mail server will receive the long-awaited response from some of the technical support.
But there was still no letter. There was no question of any work. Chasing gray matter, my brain tried to unravel this strange metamorphosis with site T.
Understanding that if a site in this state is seen by its visitors, they obviously will not be happy, I climbed to shut it down from public access. Moreover, if there is a hack, it is quite possible that the site is already full of injections in order to steal cookies, which means that visiting this site is absolutely contraindicated.
After pressing the treasured button in the settings of the main module, the site plunged into suspended animation.
Out of curiosity, I went to site R, whose settings were spelled out in the settings of our site.
“Site Under Construction” - flaunted on the main page of the site R.
The brain tried to establish a connection between the last two events and I again opened access to the site T.
Having again visited the site R, I saw that it had restored its working capacity.
Coincidence or dependency?
Repeating the manipulations in the settings of the main module, I was convinced that there is a dependence of the site R on our site T and vice versa.
But how can this be?
All that T and R sites are connected with is hosting.
“And let's call the phone number listed on the R website and try to explain the situation to them” - unexpectedly, our project suggested.
Before we had time to feel the genius of this idea, when suddenly a telephone rang in the office.
At the end of the tube there were guys who were developing the R site and are currently at a loss for events.
They handed me the phone.
After a short conversation, it turned out that they, like us, have no idea about the causes of what is happening.
“Apparently on the hosting, in some strange way, there is a symbolic link between our two bases,” I suggested.
Well, what other explanation could be found here? Access to both databases is different, but, in some strange way, changes in their database affect our and vice versa.
“Have you tried creating another database?” I continued.
"Yes, this is the third one," they answered me sadly in the voice at the other end of the phone.
Thus, the option with a symbolic link did not hold water.
Those guys also wrote an appeal to the technical support of the Hosting. We exchanged names, skype and phone numbers, agreeing to keep each other up to date.
There was no news from the technical support of the Hosting. I dialed their number again and began to explain to the technical support employee that the situation is stalemate and the same problem is already observed on two hosting accounts.
I did not hear anything intelligible in response. At the end of a meaningless and inconclusive conversation, I finally became convinced that the technical support of the Hosting would definitely not help us.
Well, at least it was clear that this was not a hack.
From that moment, I stopped hoping for technical support of the Hosting and took everything into my own hands.
Something told me that I would get answers to my questions by going to the list of tables through the 1C-Bitrix admin panel. Well, in fact, there were no other options from where to start the search.
I was surprised no less than the former when I discovered that the contents of the b_lang table (where the site settings are stored) and the contents of the admin page of the site settings are different.
What the hell is this?! How can this be ?!
“There are two of them!” - flashed through my head.
“There are two connections to the database,” I became more and more firmly entrenched in this thought.
How else to explain that the admin panel shows one thing, and the contents of the tables show another?
Having developed this idea a little more, I remembered about permanent connections to the database .
Although, judging by the description of the technology of persistent connections, they are delimited by at least a username and password when connecting to the database, and this data for sites R and T is different.
And yet, can the caching system somehow remember the connection and give it away with two completely different connections?
By this moment I was ready to believe in anything.
I turned off the permanent connections feature in the 1C-Bitrix settings and, at the same time, left the caching type empty (before that there was a value - “APC”).
I asked my colleagues in misfortune to do the same.
When everything was ready, I pressed F5. These were the longest and most exciting seconds of my life. Well, it can't be that simple.
Finally, the page loaded and ... the site worked! The site R, too, was all right. There was only one way to check if everything was fine. I went into the settings of the main module and again sent the site to the “Under Construction” state. Site T obediently followed this instruction, and site R remained operational. This fact told us that the problem was successfully solved and the bases are no longer interconnected.
But what is the problem? In persistent connections or in caching?
It was possible to find out that the problem was that sites that were not connected at all had an APC cache.
1C-Bitrix, in the dbconn.php file, forcibly caches the following tables:
The list may vary.
Among other tables, the same b_lang is clearly visible here, where site settings are stored. Therefore, T and R sites alternately recorded data in this table and cached it in APC. When site R cached a table, site T took a cached copy and vice versa.
But the main question is - how is it possible on a hosting with thousands of working accounts, mixing the cache?
After all, it turns out that any site using APC can disrupt the operation of any (almost) other site (also using APC), within the given hosting (more precisely, the server).
As a result, losses due to downtime of sites and, possibly, data theft, because you can cache anything.
Is such hosting logic normal?
After a short discussion with the developers of the R site, we came to the conclusion that you can simply set a different BX_CACHE_SID for our sites.
Obviously, there is a certain problem, since the T site has been working on this Hosting for about a year and a half and has not caused any complaints of this nature. And then suddenly such an incident.
Why did our cache mix with the R site?
Why is there no mechanism for delimiting the cache of different accounts on such a large hosting?
With these questions I went home. They did not leave my head until the evening.
At the threshold I was cordially greeted, having already missed me, my wife, and on the table was a hot and appetizing smelling supper.
“This misunderstanding is over,” I thought with relief.
Yes, someday we will definitely take up games ...
UPD 1: after thinking over the comments, I came to the conclusion that it is necessary to summarize a brief summary of all of the above.
Always set a unique cache ID and regularly make backups.
UPD 2: As a result of communication with the technical support of the Hosting, it turned out that they don’t provide account cache sharing at all.
They contacted the director of technical support, to which you can send your wishes for the hosting.
UPD 3: The head of the technical support service of the Hosting shared with me wonderful information that a new hosting environment is being developed, which will provide for the separation of account caches.
And everything seems to be fine if not for the message: “What about the T site?”.
One of my good friends noticed that one of our sites displayed incorrectly and let me know.
I dropped everything and loaded the page with the site. No problems were observed on it. Well, perhaps a small redesign was required ...
"Everything is in order with the site" - I wrote to a friend and again plunged into the fascinating world of code and bugs.
After some time, my brain nevertheless removed from my memory the alarm signal of my friend and I involuntarily climbed to re-check the site. For some reason, confident in the performance of the site, I experienced a slight self-confidence.
The main page of the site, sadly met me with a lost encoding and a complete lack of css. The black symbols of abracadabra on a virgin white background brought me back to reality. Self-confidence instantly disappeared and I began to feverishly push CTRL + F5 into the keyboard.
“It's just a cache ... Yes ... Just a cache ...” I repeated to myself over and over again.
When I realized that my tenth attempt to reset the browser cache does not give any results, and the main page of the site continues to change its appearance with every click on the treasured keys, the first thing that flashed through my head was “Hack ?!”. My fears began to grow stronger, when after the next page reload, I saw a red and white inscription saying that such and such a table was not found.
Hands themselves set out to change everything related to the site, accesses: admin panel, database, SSH.
After, I began to carefully study the logs. Although, the logs are loudly said. At my disposal were only reports on the operation of the web server. Logs of MySQL crashes and unsuccessful authorization attempts via SSH are not issued on shared hosting.
Nothing strange was found in the logs, and I went straight to the SSH console in order to connect to MySQL directly, because I clearly remember that the site cursed at the lack of tables in the database.
Very strange, but all the tables were in place (at least the ones the site was cursing for sure were in place).
As a CMS, on site T, 1C-Bitrix is used. We are very fond of this system and in every way admire it (with rare exceptions).
Imagine my surprise when, having entered the site settings, I saw data from a completely extraneous site R. The path to the root, domain name and some other settings were changed. I, stunned, looked at the monitor with eyes wide with surprise. At the same time, somewhere behind me, our project was already pouring valerian tincture. For some reason, I pressed F5. What happened after that plunged me into the deepest shock and despondency. The site settings returned to their normal values.
At this moment, for a moment, I doubted the adequacy of the entire IT industry as a whole and began to believe in miracles. My wife’s phone call got me out of a stupor. I picked up the phone. I still can’t remember what she was telling me, because the feeling of something mysterious did not leave me and my brain was frantically trying to find at least some explanation for what had happened.
I mumbled something into the phone and hung up. Well now, at least I'm sure of something. I am sure that at home I will find a lot of interesting information about myself.
But then, and now we need to understand what is happening.
After the next reload of the settings page, I again saw other people's values.
Judging by the absolute path to the root folder of the web server, we registered the settings for the site, which was located on the same hosting as us.
I instantly felt for the handset and called the technical support of the Hosting.
From the conversation it became clear that they are not able to solve our problems right now. The technical support officer politely made it clear to me that I should write an appeal by e-mail and my request will be considered within 24 hours, in the order of a live queue. After I, just as politely, expressed to them everything I think of them, I hung up and wrote them a letter full of epithets. Just in case, we also wrote a letter to tech support 1C-Bitrix.
The dead silence of the office was broken only by the ticking of a large wall clock and the barely audible noise of the system unit cooling system. A round-white snow-white clock, performed several functions in the office. Firstly, they were an element of the interior. The furniture in our office is not much, nothing more. Although, I absolutely do not mind buying a large and soft sofa, so that in between tasks, you could lie down with a cup of coffee in your hands and, legs stretched out, immerse yourself in the dream that someday we will move away from website development and start developing interesting and of course innovative, in terms of gameplay, games.
Secondly, this snow-white round clock did fulfill the function inherent in them - they showed time. If someone needed to find out the time, then he certainly turned to this watch. They have some inexplicable and almost hypnotic attraction. And nobody cared that the smartphone also had a watch on hand or on the monitor screen.
So at that moment, our pedantic overseer inexorably counted the time with each step of the second hand.
And we all waited for that, right now, right now, our mail server will receive the long-awaited response from some of the technical support.
But there was still no letter. There was no question of any work. Chasing gray matter, my brain tried to unravel this strange metamorphosis with site T.
Understanding that if a site in this state is seen by its visitors, they obviously will not be happy, I climbed to shut it down from public access. Moreover, if there is a hack, it is quite possible that the site is already full of injections in order to steal cookies, which means that visiting this site is absolutely contraindicated.
After pressing the treasured button in the settings of the main module, the site plunged into suspended animation.
Out of curiosity, I went to site R, whose settings were spelled out in the settings of our site.
“Site Under Construction” - flaunted on the main page of the site R.
The brain tried to establish a connection between the last two events and I again opened access to the site T.
Having again visited the site R, I saw that it had restored its working capacity.
Coincidence or dependency?
Repeating the manipulations in the settings of the main module, I was convinced that there is a dependence of the site R on our site T and vice versa.
But how can this be?
All that T and R sites are connected with is hosting.
“And let's call the phone number listed on the R website and try to explain the situation to them” - unexpectedly, our project suggested.
Before we had time to feel the genius of this idea, when suddenly a telephone rang in the office.
At the end of the tube there were guys who were developing the R site and are currently at a loss for events.
They handed me the phone.
After a short conversation, it turned out that they, like us, have no idea about the causes of what is happening.
“Apparently on the hosting, in some strange way, there is a symbolic link between our two bases,” I suggested.
Well, what other explanation could be found here? Access to both databases is different, but, in some strange way, changes in their database affect our and vice versa.
“Have you tried creating another database?” I continued.
"Yes, this is the third one," they answered me sadly in the voice at the other end of the phone.
Thus, the option with a symbolic link did not hold water.
Those guys also wrote an appeal to the technical support of the Hosting. We exchanged names, skype and phone numbers, agreeing to keep each other up to date.
There was no news from the technical support of the Hosting. I dialed their number again and began to explain to the technical support employee that the situation is stalemate and the same problem is already observed on two hosting accounts.
I did not hear anything intelligible in response. At the end of a meaningless and inconclusive conversation, I finally became convinced that the technical support of the Hosting would definitely not help us.
Well, at least it was clear that this was not a hack.
From that moment, I stopped hoping for technical support of the Hosting and took everything into my own hands.
Something told me that I would get answers to my questions by going to the list of tables through the 1C-Bitrix admin panel. Well, in fact, there were no other options from where to start the search.
I was surprised no less than the former when I discovered that the contents of the b_lang table (where the site settings are stored) and the contents of the admin page of the site settings are different.
What the hell is this?! How can this be ?!
“There are two of them!” - flashed through my head.
“There are two connections to the database,” I became more and more firmly entrenched in this thought.
How else to explain that the admin panel shows one thing, and the contents of the tables show another?
Having developed this idea a little more, I remembered about permanent connections to the database .
Although, judging by the description of the technology of persistent connections, they are delimited by at least a username and password when connecting to the database, and this data for sites R and T is different.
And yet, can the caching system somehow remember the connection and give it away with two completely different connections?
By this moment I was ready to believe in anything.
I turned off the permanent connections feature in the 1C-Bitrix settings and, at the same time, left the caching type empty (before that there was a value - “APC”).
I asked my colleagues in misfortune to do the same.
When everything was ready, I pressed F5. These were the longest and most exciting seconds of my life. Well, it can't be that simple.
Finally, the page loaded and ... the site worked! The site R, too, was all right. There was only one way to check if everything was fine. I went into the settings of the main module and again sent the site to the “Under Construction” state. Site T obediently followed this instruction, and site R remained operational. This fact told us that the problem was successfully solved and the bases are no longer interconnected.
But what is the problem? In persistent connections or in caching?
It was possible to find out that the problem was that sites that were not connected at all had an APC cache.
1C-Bitrix, in the dbconn.php file, forcibly caches the following tables:
- b_file
- b_file_bucket_size
- b_lang
- b_option
- b_lang_domain
- b_site_template
- b_event
- b_agent
The list may vary.
Among other tables, the same b_lang is clearly visible here, where site settings are stored. Therefore, T and R sites alternately recorded data in this table and cached it in APC. When site R cached a table, site T took a cached copy and vice versa.
But the main question is - how is it possible on a hosting with thousands of working accounts, mixing the cache?
After all, it turns out that any site using APC can disrupt the operation of any (almost) other site (also using APC), within the given hosting (more precisely, the server).
As a result, losses due to downtime of sites and, possibly, data theft, because you can cache anything.
Is such hosting logic normal?
After a short discussion with the developers of the R site, we came to the conclusion that you can simply set a different BX_CACHE_SID for our sites.
Obviously, there is a certain problem, since the T site has been working on this Hosting for about a year and a half and has not caused any complaints of this nature. And then suddenly such an incident.
Why did our cache mix with the R site?
Why is there no mechanism for delimiting the cache of different accounts on such a large hosting?
With these questions I went home. They did not leave my head until the evening.
At the threshold I was cordially greeted, having already missed me, my wife, and on the table was a hot and appetizing smelling supper.
“This misunderstanding is over,” I thought with relief.
Yes, someday we will definitely take up games ...
UPD 1: after thinking over the comments, I came to the conclusion that it is necessary to summarize a brief summary of all of the above.
Always set a unique cache ID and regularly make backups.
UPD 2: As a result of communication with the technical support of the Hosting, it turned out that they don’t provide account cache sharing at all.
They contacted the director of technical support, to which you can send your wishes for the hosting.
UPD 3: The head of the technical support service of the Hosting shared with me wonderful information that a new hosting environment is being developed, which will provide for the separation of account caches.