InterSystems Database Mirroring. Creating and testing mirrors. Part 2
In the previous part, mirror tuning was considered - the high-availability technology InterSystems Database Mirroring DBMS Caché.
In this article, scenarios of interruptions (failures) and the reaction of the mirror to them will be considered.
Breaks can be both planned and unplanned.
Scheduled ones are when we need to stop the server to update the operating system, DBMS version, application system version and we control the process.
Examples of scheduled breaks:
Unscheduled - when something happens that leads to the inability of the server to work with clients of the information system, and without our knowledge. The reason for this may be:
Automatic failover for the Caché mirror occurs only in the event of a failure / interruption on the Primary server, and in this article we will consider just such cases. Of course, there may be scenarios when a problem occurs on the Backup server, but in this case the IP continues to work, and there is no physical mirroring until the administrator manually restores the Backup server. Another scenario is possible when both servers failed immediately - but in this case, a break in the operation of the IP is inevitable, and its settlement is carried out “manually”.
The basic scenario for automatic failover:
What could go wrong?
Imagine a situation where the network interface fails on the Primary server, but not for long, however, it is enough for the Backup server to already “decide” to become the Primary server. The result is a split brain situation.
Split brain - braincancer is a state of complete desynchronization of cluster servers, when both servers are working and each of them believes that it is Primary at the moment.
Such a situation may arise, for example, if the Backup server “mistakenly” decided that the Primary server is in “down”, as a result, two Primary servers appear at once in the cluster, which try to simultaneously serve user requests - the cluster collapse.
For IP users, this situation is expressed in an indefinite interruption of work, which is restored manually by the administrator.
The fastest way to recover from this situation is to re-create the cluster by restoring data from a backup or from an asynchronous “mirror” .
The decisive factor for the behavior of the Caché mirror during failure / interruption is the value of the AgentContactRequiredForFailover = yes / no parameter.
In the case of AgentContactRequiredForFailover (ACRF) = yes (default) with failover, the ISC Agent on both servers must be in contact at the time of the break. This situation is possible for cases of unplanned failure of Cache, scheduled reboot of Cache, scheduled reboot of the OS. Therefore, in this mode, for all other cases, an automatic failover will not occur!
This mode of operation of mirroring allows failover to be performed without the presence of online contact between server agents. This mode may be needed to cover situations like Power Off primary-server, failure of network adapters or the network itself between servers, etc., when there is no connection with the Primary-server. In this mode, the decision to automatically switch the Backup server to the Primary server state is determined by the user function in the ZMIRROR program located in the (% SYS) area.
Of course, when building a failover solution, we expect to automatically overcome the failure for most situations, otherwise why all this. Currently, the Caché mirror is capable of automatically processing almost all possible scenarios.
We tested how the mirror behaves for several typical scenarios with different ACRF values.
We tested the latest version of Caché 2012.2 RC at the moment. Yes - means automatic failover for this scenario.
According to the test results, it turned out - an unpleasant scenario is to disable the network interface (No. 4). Therefore, it is recommended that a redundant network link be established between the mirror servers.
Example of server configuration:
If both servers fail simultaneously - scenarios 5-6 (long interruption in power supply, for example), it is important to follow the commissioning sequence - the Primary server must be activated first. Otherwise, the Backup server becomes Primary, and after the start of the former Primary server, there are two of them - as a result of the split brain.
As can be seen from the table, the Caché synchronous mirror with ACRF = yes (the default value) provides only those emergency situations when the network between the cluster servers is “alive”.
For other situations, there is ACRF = no mode, where the administrator, at his own risk, writes the user program ZMIRROR to determine if the Primary server fails.
It is theoretically possible to build such a hardware-software system for the final installation of the mirror, when for any scenario it is possible to uniquely determine in software state what state the Primary server is in.
If this situation occurs on your server, it is advisable to set the mode to ACRF = no. Then in case of a break, and when there is no communication with the ISC Agent Primary server, the system calls the function $$ IsOtherNodeDown ^ ZMIRROR (). The function returns 1 or 0. If there is no function or the function returns 0, the automatic failover process is terminated.
The task of the programmer when writing a custom implementation of the $$ IsOtherNodeDown ^ ZMIRROR () function is to reliably verify that the Primary server really failed and return 1 in this case.
Disclaimer! Using this code in no way guarantees an automatic failover in your production configuration for ACRF = No. For real use of this mode, thorough failover testing is required for each configuration individually!
A few recommendations from the InterSystems Database Mirroring operating experience
“You cannot“ manually ”clean the primary server log if the backup server is turned off”
Indeed, if you do this and the backup server does not manage to pick up the necessary logs, you can create a backup server again.
“You cannot keep the backup server turned off for a long time while the primary server is running”
In this case, the disk space on the primary server may run out due to overflow with unsynchronized logs.
“Time on the servers must be synchronized”
This is a simple condition, but it must be fulfilled. Otherwise, during desynchronization, for example, for several hours, the backup server “gets confused” with the relevance of the logs. As a result, we make the backup server again.
“The network between the servers must be unloaded”.
The network is constantly transmitting the log from the Primary server to backup. If the network between the servers is still busy with something, or the channel is not enough for data transfer, as a result, the Primary server will start to “slow down”, which will not have time to transfer data entered by IS users.
“System configuration settings, mappings, role and user settings must be synchronized manually”
The settings for mapping globals, classes, and packages, as well as all settings for roles, users, and system configuration for Primary and Backup servers, must also be kept in sync. Mirror synchronizes only the databases selected for mirroring. Therefore, the mirror administrator must have identical configuration, mappings, and user security policies and maintain this state for Primary and Backup servers.
The average runtime for automatic failover for Agent Contact Required for Failover = yes is 10-20 seconds. A pause of 10-20 seconds is the time when the application continues to work with the new server. In this case, the connection reconnects (if there was a direct connection to the server) or simply continue to work if the work was done through an ECP connection.
In the case of ACRF = no, the pause is an average of 35-40 seconds. In this case, ECP clients will lose connection, because default timeout for ECP = 30 sec. For systems using ECP and a mirror with ACRF = no on ECP client servers, in SYSTEM ^% ZSTART, it is advisable to write a code that increases the ECP timeout:
do $ system.ECP.SetProperty ("ECPRecoveryTimout", 90)
Answering this question, I say “Yes.” Even if you use what comes “out of the box” - namely, the ACRF = yes mode, it is possible to carry out scheduled work on updating the OS, Caché, Hardware, and so on. non-stop operation of the information system, as well as automatic overcoming of failures of the Caché DBMS. The remaining scenarios can be covered by known means using hardware failover clusters and uninterruptible power systems in the required number of “nines after the decimal point”.
In this article, scenarios of interruptions (failures) and the reaction of the mirror to them will be considered.
Breaks can be both planned and unplanned.
Scheduled ones are when we need to stop the server to update the operating system, DBMS version, application system version and we control the process.
Examples of scheduled breaks:
- rebooting the OS, for example, to install updates;
- DBMS reboot
- hardware server update.
Unscheduled - when something happens that leads to the inability of the server to work with clients of the information system, and without our knowledge. The reason for this may be:
- DBMS freezing
- OS freeze
- emergency (Reset, Power Off);
- server hardware failure;
- failure of network equipment;
war, epidemic, snowstorm, outer space black holes.
Failover
Automatic failover for the Caché mirror occurs only in the event of a failure / interruption on the Primary server, and in this article we will consider just such cases. Of course, there may be scenarios when a problem occurs on the Backup server, but in this case the IP continues to work, and there is no physical mirroring until the administrator manually restores the Backup server. Another scenario is possible when both servers failed immediately - but in this case, a break in the operation of the IP is inevitable, and its settlement is carried out “manually”.
The basic scenario for automatic failover:
- Primary-server serves IS clients that work with one of the mirrored databases, and the Backup-server stands “stand by”;
- A failure occurs on the Primary server;
- The backup server makes sure that the Primary server is in the “down”, automatically becomes the Primary server and continues to serve the clients of the information system;
- The former Primary server is restored (for example, by the administrator), becomes a new Backup server and enters “stand by” mode for the new Primary server.
What could go wrong?
Imagine a situation where the network interface fails on the Primary server, but not for long, however, it is enough for the Backup server to already “decide” to become the Primary server. The result is a split brain situation.
Split brain
Split brain - brain
Such a situation may arise, for example, if the Backup server “mistakenly” decided that the Primary server is in “down”, as a result, two Primary servers appear at once in the cluster, which try to simultaneously serve user requests - the cluster collapse.
For IP users, this situation is expressed in an indefinite interruption of work, which is restored manually by the administrator.
The fastest way to recover from this situation is to re-create the cluster by restoring data from a backup or from an asynchronous “mirror” .
AgentContactRequiredForFailover
The decisive factor for the behavior of the Caché mirror during failure / interruption is the value of the AgentContactRequiredForFailover = yes / no parameter.
AgentContactRequiredForFailover = yes
In the case of AgentContactRequiredForFailover (ACRF) = yes (default) with failover, the ISC Agent on both servers must be in contact at the time of the break. This situation is possible for cases of unplanned failure of Cache, scheduled reboot of Cache, scheduled reboot of the OS. Therefore, in this mode, for all other cases, an automatic failover will not occur!
AgentContactRequiredForFailover = no
This mode of operation of mirroring allows failover to be performed without the presence of online contact between server agents. This mode may be needed to cover situations like Power Off primary-server, failure of network adapters or the network itself between servers, etc., when there is no connection with the Primary-server. In this mode, the decision to automatically switch the Backup server to the Primary server state is determined by the user function in the ZMIRROR program located in the (% SYS) area.
Test results
Of course, when building a failover solution, we expect to automatically overcome the failure for most situations, otherwise why all this. Currently, the Caché mirror is capable of automatically processing almost all possible scenarios.
We tested how the mirror behaves for several typical scenarios with different ACRF values.
We tested the latest version of Caché 2012.2 RC at the moment. Yes - means automatic failover for this scenario.
| N | Scenarios | ACRF = Yes | ACRF = No |
|---|---|---|---|
| 1 | Planned or emergency reboot, shutdown of Caché DBMS on Primary | Yes | Yes |
| 2 | OS scheduled reboot | Yes | Yes |
| 3 | Emergency Stop OS, Reset, Power Off | Not | Yes |
| 4 | Disabling the network interface on Primary | Not | Yes |
| 5 | Power Off at the same time Primary and Backup, and then booting first Primary | Yes | Yes |
| 6 | Power Off at the same time Primary and Backup, and then boot first Backup | Not | Not |
According to the test results, it turned out - an unpleasant scenario is to disable the network interface (No. 4). Therefore, it is recommended that a redundant network link be established between the mirror servers.
Example of server configuration:
If both servers fail simultaneously - scenarios 5-6 (long interruption in power supply, for example), it is important to follow the commissioning sequence - the Primary server must be activated first. Otherwise, the Backup server becomes Primary, and after the start of the former Primary server, there are two of them - as a result of the split brain.
As can be seen from the table, the Caché synchronous mirror with ACRF = yes (the default value) provides only those emergency situations when the network between the cluster servers is “alive”.
For other situations, there is ACRF = no mode, where the administrator, at his own risk, writes the user program ZMIRROR to determine if the Primary server fails.
Zmirror
It is theoretically possible to build such a hardware-software system for the final installation of the mirror, when for any scenario it is possible to uniquely determine in software state what state the Primary server is in.
If this situation occurs on your server, it is advisable to set the mode to ACRF = no. Then in case of a break, and when there is no communication with the ISC Agent Primary server, the system calls the function $$ IsOtherNodeDown ^ ZMIRROR (). The function returns 1 or 0. If there is no function or the function returns 0, the automatic failover process is terminated.
The task of the programmer when writing a custom implementation of the $$ IsOtherNodeDown ^ ZMIRROR () function is to reliably verify that the Primary server really failed and return 1 in this case.
ZMIRROR Custom Function Example
ZMIRROR
; USE AT YOUR OWN RISK.
q
IsOtherNodeDown () public
{
d ## class (% Device) .Broadcast ("", "------------> IsOtherNodeDown ()")
try {
s ip = "192.168.169.186"; the address of another Failover node
s PingSuccessful = $ system.INetInfo.CheckAddressExist (ip)
d ## class (% Device) .Broadcast ("", "IP adress" _ip_ "is" _
$ s (PingSuccessful: "available", 1: "unavailable") _ ".")
if 'PingSuccessful s returnValue = 1 q; no ping
s url = " " _ip _ ": 57772 / csp / sys / UtilHome.csp"
s stat = $$ GetStatus (url)
d ## class (% Device).
s returnValue = 0
} catch (e) {
d ## class (% Library.Device) .Broadcast ("", e.DisplayString ())
s returnValue = 0
}
d ## class (% Device) .Broadcast ("" , "Returning" _returnValue_ "...")
d ## class (% Device) .Broadcast ("", "<------------ IsOtherNodeDown ()")
q returnValue
}
CheckBecomePrimaryOK () public
{
d ## class (% Device) .Broadcast ("", "------------> CheckBecomePrimaryOK ()")
try {
s stat = $$ MemberDescription ()
s returnValue = 1
d ## class (% Device) .Broadcast ("", "This instance is" _stat)
d ## class (% Device) .Broadcast ("","Returning" _returnValue_ "...")
} catch (e) {
d ## class (% Library.Device) .Broadcast ("", e.DisplayString ())
s returnValue = 0
}
d ## class (% Device) .Broadcast ("", "<------- ----- CheckBecomePrimaryOK () ")
q returnValue
}
NotifyBecomePrimary () public
{
d ## class (% Device) .Broadcast (" "," ------------> NotifyBecomePrimary () " )
try {
d ## class (% Device) .Broadcast ("", "This instance is" _ $$ MemberDescription ())
} catch (e) {
d ## class (% Library.Device) .Broadcast ("" , e.DisplayString ())
}
d ## class (% Device) .Broadcast ("", "<------------ NotifyBecomePrimary ()")
}
NotifyBecomePrimaryFailed () public
{
d ## class (% Device) .Broadcast ("", "------------> NotifyBecomePrimaryFailed ()")
try {
d ## class (% Device) .Broadcast ("", " This instance is "_ $$ MemberDescription ())
} catch (e) {
d ## class (% Library.Device) .Broadcast (" ", e.DisplayString ())
}
d ## class (% Device) .Broadcast ("", "<------------ NotifyBecomePrimaryFailed ()")
}
MemberDescription (); get mirror member status
q $ s ($ system.Mirror.IsMember (): $ s ($ system.Mirror.IsBackup (): "a backup",
$ system.Mirror.IsPrimary (): "a primary",
$ system .Mirror.IsAsyncMember (): "an async",
1: "an unknown") _ "member of" _ $ system.Mirror.MirrorName (),
1: "not a member")
GetStatus (url)
; url - address http: // localhost: 57772 / csp / sys / UtilHome.csp
S $ ZT = "HttpError"
N A, status, err, httprequest, Port, content,% objlasterror
Set httprequest = ## class (% Net.HttpRequest).% New ()
SA = $ P ($ p (url , ": //", 2), "/", 1) IA = "" Q "Error! URL"
IA [":" S Port = $ P (A, ":", 2), A = $ P (A, ":", 1) Set httprequest.Port = Port
Set httprequest.Server = A
S A = "/" _ $ P ($ p (url, ": //", 2), "/", 2,991)
Set httprequest.Timeout = 3
Do httprequest.Get (A)
S status = httprequest.HttpResponse.StatusLine
Do httprequest.% Close ()
i status = "", $ g (% objlasterror) '= "" g HttpError
q status
HttpError I $ ZE [""Q" Error! Server No Access? "_ $ ZE
do DecomposeStatus ^% apiOBJ (% objlasterror, .err)
Q" Error "_ $ G (err (err))
; USE AT YOUR OWN RISK.
q
IsOtherNodeDown () public
{
d ## class (% Device) .Broadcast ("", "------------> IsOtherNodeDown ()")
try {
s ip = "192.168.169.186"; the address of another Failover node
s PingSuccessful = $ system.INetInfo.CheckAddressExist (ip)
d ## class (% Device) .Broadcast ("", "IP adress" _ip_ "is" _
$ s (PingSuccessful: "available", 1: "unavailable") _ ".")
if 'PingSuccessful s returnValue = 1 q; no ping
s url = " " _ip _ ": 57772 / csp / sys / UtilHome.csp"
s stat = $$ GetStatus (url)
d ## class (% Device).
s returnValue = 0
} catch (e) {
d ## class (% Library.Device) .Broadcast ("", e.DisplayString ())
s returnValue = 0
}
d ## class (% Device) .Broadcast ("" , "Returning" _returnValue_ "...")
d ## class (% Device) .Broadcast ("", "<------------ IsOtherNodeDown ()")
q returnValue
}
CheckBecomePrimaryOK () public
{
d ## class (% Device) .Broadcast ("", "------------> CheckBecomePrimaryOK ()")
try {
s stat = $$ MemberDescription ()
s returnValue = 1
d ## class (% Device) .Broadcast ("", "This instance is" _stat)
d ## class (% Device) .Broadcast ("","Returning" _returnValue_ "...")
} catch (e) {
d ## class (% Library.Device) .Broadcast ("", e.DisplayString ())
s returnValue = 0
}
d ## class (% Device) .Broadcast ("", "<------- ----- CheckBecomePrimaryOK () ")
q returnValue
}
NotifyBecomePrimary () public
{
d ## class (% Device) .Broadcast (" "," ------------> NotifyBecomePrimary () " )
try {
d ## class (% Device) .Broadcast ("", "This instance is" _ $$ MemberDescription ())
} catch (e) {
d ## class (% Library.Device) .Broadcast ("" , e.DisplayString ())
}
d ## class (% Device) .Broadcast ("", "<------------ NotifyBecomePrimary ()")
}
NotifyBecomePrimaryFailed () public
{
d ## class (% Device) .Broadcast ("", "------------> NotifyBecomePrimaryFailed ()")
try {
d ## class (% Device) .Broadcast ("", " This instance is "_ $$ MemberDescription ())
} catch (e) {
d ## class (% Library.Device) .Broadcast (" ", e.DisplayString ())
}
d ## class (% Device) .Broadcast ("", "<------------ NotifyBecomePrimaryFailed ()")
}
MemberDescription (); get mirror member status
q $ s ($ system.Mirror.IsMember (): $ s ($ system.Mirror.IsBackup (): "a backup",
$ system.Mirror.IsPrimary (): "a primary",
$ system .Mirror.IsAsyncMember (): "an async",
1: "an unknown") _ "member of" _ $ system.Mirror.MirrorName (),
1: "not a member")
GetStatus (url)
; url - address http: // localhost: 57772 / csp / sys / UtilHome.csp
S $ ZT = "HttpError"
N A, status, err, httprequest, Port, content,% objlasterror
Set httprequest = ## class (% Net.HttpRequest).% New ()
SA = $ P ($ p (url , ": //", 2), "/", 1) IA = "" Q "Error! URL"
IA [":" S Port = $ P (A, ":", 2), A = $ P (A, ":", 1) Set httprequest.Port = Port
Set httprequest.Server = A
S A = "/" _ $ P ($ p (url, ": //", 2), "/", 2,991)
Set httprequest.Timeout = 3
Do httprequest.Get (A)
S status = httprequest.HttpResponse.StatusLine
Do httprequest.% Close ()
i status = "", $ g (% objlasterror) '= "" g HttpError
q status
HttpError I $ ZE ["
do DecomposeStatus ^% apiOBJ (% objlasterror, .err)
Q" Error "_ $ G (err (err))
Disclaimer! Using this code in no way guarantees an automatic failover in your production configuration for ACRF = No. For real use of this mode, thorough failover testing is required for each configuration individually!
Mirror Tips
A few recommendations from the InterSystems Database Mirroring operating experience
“You cannot“ manually ”clean the primary server log if the backup server is turned off”
Indeed, if you do this and the backup server does not manage to pick up the necessary logs, you can create a backup server again.
“You cannot keep the backup server turned off for a long time while the primary server is running”
In this case, the disk space on the primary server may run out due to overflow with unsynchronized logs.
“Time on the servers must be synchronized”
This is a simple condition, but it must be fulfilled. Otherwise, during desynchronization, for example, for several hours, the backup server “gets confused” with the relevance of the logs. As a result, we make the backup server again.
“The network between the servers must be unloaded”.
The network is constantly transmitting the log from the Primary server to backup. If the network between the servers is still busy with something, or the channel is not enough for data transfer, as a result, the Primary server will start to “slow down”, which will not have time to transfer data entered by IS users.
“System configuration settings, mappings, role and user settings must be synchronized manually”
The settings for mapping globals, classes, and packages, as well as all settings for roles, users, and system configuration for Primary and Backup servers, must also be kept in sync. Mirror synchronizes only the databases selected for mirroring. Therefore, the mirror administrator must have identical configuration, mappings, and user security policies and maintain this state for Primary and Backup servers.
Performance characteristics
The average runtime for automatic failover for Agent Contact Required for Failover = yes is 10-20 seconds. A pause of 10-20 seconds is the time when the application continues to work with the new server. In this case, the connection reconnects (if there was a direct connection to the server) or simply continue to work if the work was done through an ECP connection.
In the case of ACRF = no, the pause is an average of 35-40 seconds. In this case, ECP clients will lose connection, because default timeout for ECP = 30 sec. For systems using ECP and a mirror with ACRF = no on ECP client servers, in SYSTEM ^% ZSTART, it is advisable to write a code that increases the ECP timeout:
do $ system.ECP.SetProperty ("ECPRecoveryTimout", 90)
The bottom line or “do we need it”?
Answering this question, I say “Yes.” Even if you use what comes “out of the box” - namely, the ACRF = yes mode, it is possible to carry out scheduled work on updating the OS, Caché, Hardware, and so on. non-stop operation of the information system, as well as automatic overcoming of failures of the Caché DBMS. The remaining scenarios can be covered by known means using hardware failover clusters and uninterruptible power systems in the required number of “nines after the decimal point”.
