March 25, 2019 at 13:38Stream conversion of Firebird 2.5 databases to ODS12 format (Firebird 3.0)
Each version of Firebird has its own version of the format of database disk structures - O (n) D (isk) S (tructure). Prior to version 2.5 inclusive, the Firebird engine could work with ODS of previous versions, that is, databases from older versions were opened by the new version and worked in compatibility mode, but the Firebird 3.0 engine only works with a database in its own ODS version 12.0.
To switch to 3.0, the database from 2.5 must be converted to a new format via backup / restore. Of course, we assume that the database was previously prepared for conversion - i.e. metadata and requests have been tested for compatibility with Firebird 3.0.
If you follow the standard approach, this means that you need to backup on version 2.5, then install 3.0 and make a restore. This procedure is acceptable if there is enough time, but when migrating large databases, or while migrating several dozen databases, when time is running out, you can use stream conversion, which is 30-40% faster. How exactly to do this (under Windows and under Linux), read under the cat.
The general idea is that for acceleration we will use the pipeline:
Gbak from 2.5 generates a backup in a linear format and sends it to stdout, which immediately picks up gbak from 3.0 via stdin and creates a new database.
It is necessary to organize such a pipeline using the local (file) access method, since network access (even through localhost) will significantly slow down the process.
Below we look at the details for Windows and Linux.
Windows
In the case of Windows, the easiest way is to make a fully autonomous build of Firebird. To do this, take the Firebird 2.5 embed-archive , rename fbemded.dll to fbclient.dll, add gbak.exe utilities from the “usual” 2.5 archive and (optionally) isql.exe.
Firebird 3.0 uses a single assembly and does not require any modification.
The smallest option (which does not require the installation of VS2008 / VS2010 runtime libraries on the target system) contains the following files:
An experienced administrator may notice that intl / fbintl.dll and intl / fbintl.conf are not included in 2.5. This is true, since gbak does not use a connection char and does not convert data between chars, but on the "receiving" side of Firebird 3.0 these files are needed when creating indexes.
In firebird.conf, Firebird 3.0 recommends adding:
Also, it is advisable to set different IpcName values for 2.5 and 3.0.
When choosing the values of other parameters, firebird.conf proceeds from a simple consideration: at the data transfer stage, in one process, gbak runs 2.5, and in the other 3.0, then 2.5 finishes, and 3.0 starts building indexes.
To speed up the stage of building indexes in 3.0, it is recommended to increase the size of the TempCacheLimit parameter to ~ 40% RAM (if it is a dedicated server, of course).
For example, if the server has 16 GB of RAM, then you can put
Of course, this value can only be set for 64-bit Firebird 3, since any 32-bit process will not be able to allocate more than 2 gigabytes of memory.
At 2.5, this parameter does not need to be changed - it already cannot be more than 2 gigabytes, and it does not affect the speed during backup.
Before performing the operation, you need to check that the page cache in the database header is set to 0 (command
If the cache is specified explicitly in the database header, then it overrides the values from firebird.conf (and databases.conf in 3.0), and in case of inadequately large values it can lead to excessive memory consumption and go to the swap.
Next, copy the files to the target system.
Conversion is carried out after stopping the “system” Firebird 2.5 service, on the command line with increasing privileges to the local administrator (example):
This example uses “straight oblique” in quotation marks (valid “unix-style”), and “hat” (“^” character) escapes the line feed character, which is convenient when typing long commands. The -st (atus) option appeared in Firebird 2.5.8 and allows you to write data on the operating time of the gbak process to the protocol (see the documentation for details).
Linux
On Linux, Firebird 3 depends on the tommath library. In CentOS (RHEL), this library is in the epel repository, in Ubuntu (Debian) it is in the system one.
For CentOS, you must first connect the epel repository and only then do
Ubuntu does not need to connect additional repositories, but different versions of packages are installed on Ubuntu 16 and Ubuntu 18 - libtommath0 and libtommath1, respectively.
Firebird 3.0 is looking for tommath.so.0 and for Ubuntu 18 it is additionally required to create a link (symlink) from tommath.so.0 to tommath.so.1. To do this, you first need to find tommath.so.1.
The search path in Ubuntu is
The second problem is that prior to Firebird 3.0.1, inclusive, there was no easy way to install two different versions of the server. We do not consider the option “compile from source with the desired prefix” because of its relative complexity.
For Firebird 3.0.2 and higher, the assembly with –enable-binreloc is implementedand a separate installer option (-path path).
Assuming that the tommath library and, if necessary, the symlink for tommath.so.0 have been added to the system, you can add the actual (at the time of this writing) Firebird 3.0.4 distribution to, for example, / opt / fb3:
After that, you can stop the Firebird system service and start streaming conversion.
When stopping Firebird, it should be borne in mind that Firebid 2.5 processes in Classic mode usually start xinetd - therefore, it is either necessary to disable the firebird service for xinetd or to stop xinetd completely.
In firebird.conf for 3.0 on Linux, you do not need to set MaxUnflushed parameters (they work only on Windows) and change Firebird 2.5 settings.
In Linux, local (file) access of Firebird 2.5 is not equivalent to the embeded version for Windows - server 2.5 will work in the gbak process (without the network part), but access rights will be checked against the user base, which means that not only a login, but also a password will be required :
After successful conversion, you must first remove the "additional" Firebird 3.0, then the "main" Firebird 2.5, and only after that perform a clean installation of Firebird 3.0 - best of all from the regular tar.gz installer, and not through the repositories, because the version in the repositories may lag.
Also, after restoring the Linux database and reinstalling it, you need to check that the new database has the owner of the firebird user.
If this is not so, then it will be necessary to fix
Summary
In addition to saving time and disk space, streaming conversion has another important advantage - database conversion is done without deleting the existing Firebird 2.5, which greatly simplifies the rollback if the conversion is unsuccessful (most often due to lack of space or unexpected reboot during the migration process) .
Saving time is due to the fact that the "classic" conversion is "backup time" plus "recovery time". Recovery consists of two parts: reading data from a backup file and building an index.
In streaming conversion, the total time is obtained as “backup time plus five to ten percent” and “index building time”.
The specific results depend on the structure of the database, but on average, the recovery time is approximately equal to the double backup time. Therefore, if we take backup time per unit, then “classical conversion” - three units of time, flow - two units of time. An increase in TempCacheLimit also helps to reduce time.
In general, stream conversion in practice allows you to save 30-40% percent of the time of an alternate backup and restaurant.
Questions?
Please write all questions in the comments, or send to the author of the methodology and co-author of this article - Vasily Sidorov, iBase's leading systems engineer, at bs at ibase ru.
To switch to 3.0, the database from 2.5 must be converted to a new format via backup / restore. Of course, we assume that the database was previously prepared for conversion - i.e. metadata and requests have been tested for compatibility with Firebird 3.0.
If you follow the standard approach, this means that you need to backup on version 2.5, then install 3.0 and make a restore. This procedure is acceptable if there is enough time, but when migrating large databases, or while migrating several dozen databases, when time is running out, you can use stream conversion, which is 30-40% faster. How exactly to do this (under Windows and under Linux), read under the cat.
The general idea is that for acceleration we will use the pipeline:
gbak -b … база25 stdout | gbak -c … stdin база30Gbak from 2.5 generates a backup in a linear format and sends it to stdout, which immediately picks up gbak from 3.0 via stdin and creates a new database.
It is necessary to organize such a pipeline using the local (file) access method, since network access (even through localhost) will significantly slow down the process.
Below we look at the details for Windows and Linux.
Windows
In the case of Windows, the easiest way is to make a fully autonomous build of Firebird. To do this, take the Firebird 2.5 embed-archive , rename fbemded.dll to fbclient.dll, add gbak.exe utilities from the “usual” 2.5 archive and (optionally) isql.exe.
Firebird 3.0 uses a single assembly and does not require any modification.
The smallest option (which does not require the installation of VS2008 / VS2010 runtime libraries on the target system) contains the following files:
25/gbak.exe
25/fbclient.dll
25/firebird.conf
25/firebird.log
25/firebird.msg
25/ib_util.dll
25/icudt30.dll
25/icuin30.dll
25/icuuc30.dll
25/Microsoft.VC80.CRT.manifest
25/msvcp80.dll
25/msvcr80.dll
30/fbclient.dll
30/firebird.conf
30/firebird.msg
30/gbak.exe
30/ib_util.dll
30/icudt52.dll
30/icudt52l.dat
30/icuin52.dll
30/icuuc52.dll
30/msvcp100.dll
30/msvcr100.dll
30/intl/fbintl.conf
30/intl/fbintl.dll
30/plugins/engine12.dllAn experienced administrator may notice that intl / fbintl.dll and intl / fbintl.conf are not included in 2.5. This is true, since gbak does not use a connection char and does not convert data between chars, but on the "receiving" side of Firebird 3.0 these files are needed when creating indexes.
In firebird.conf, Firebird 3.0 recommends adding:
MaxUnflushedWrites = -1
MaxUnflushedWriteTime = -1Also, it is advisable to set different IpcName values for 2.5 and 3.0.
When choosing the values of other parameters, firebird.conf proceeds from a simple consideration: at the data transfer stage, in one process, gbak runs 2.5, and in the other 3.0, then 2.5 finishes, and 3.0 starts building indexes.
To speed up the stage of building indexes in 3.0, it is recommended to increase the size of the TempCacheLimit parameter to ~ 40% RAM (if it is a dedicated server, of course).
For example, if the server has 16 GB of RAM, then you can put
TempCacheLimit=6GOf course, this value can only be set for 64-bit Firebird 3, since any 32-bit process will not be able to allocate more than 2 gigabytes of memory.
At 2.5, this parameter does not need to be changed - it already cannot be more than 2 gigabytes, and it does not affect the speed during backup.
Before performing the operation, you need to check that the page cache in the database header is set to 0 (command
gstat -h databasename, see the line Page buffers). If the cache is specified explicitly in the database header, then it overrides the values from firebird.conf (and databases.conf in 3.0), and in case of inadequately large values it can lead to excessive memory consumption and go to the swap.
Next, copy the files to the target system.
Conversion is carried out after stopping the “system” Firebird 2.5 service, on the command line with increasing privileges to the local administrator (example):
set ISC_USER=владелец
"25/gbak" -z -b -g -v -st t -y 25.log база25 stdout|^
"30/gbak" -z -c -v -st t -y 30.log stdin база30This example uses “straight oblique” in quotation marks (valid “unix-style”), and “hat” (“^” character) escapes the line feed character, which is convenient when typing long commands. The -st (atus) option appeared in Firebird 2.5.8 and allows you to write data on the operating time of the gbak process to the protocol (see the documentation for details).
Linux
On Linux, Firebird 3 depends on the tommath library. In CentOS (RHEL), this library is in the epel repository, in Ubuntu (Debian) it is in the system one.
For CentOS, you must first connect the epel repository and only then do
yum install libtommathUbuntu does not need to connect additional repositories, but different versions of packages are installed on Ubuntu 16 and Ubuntu 18 - libtommath0 and libtommath1, respectively.
Firebird 3.0 is looking for tommath.so.0 and for Ubuntu 18 it is additionally required to create a link (symlink) from tommath.so.0 to tommath.so.1. To do this, you first need to find tommath.so.1.
The search path in Ubuntu is
/usr/lib/x86_64-linux-gnu/, but in other Debian-based distributions it may be different. The second problem is that prior to Firebird 3.0.1, inclusive, there was no easy way to install two different versions of the server. We do not consider the option “compile from source with the desired prefix” because of its relative complexity.
For Firebird 3.0.2 and higher, the assembly with –enable-binreloc is implementedand a separate installer option (-path path).
Assuming that the tommath library and, if necessary, the symlink for tommath.so.0 have been added to the system, you can add the actual (at the time of this writing) Firebird 3.0.4 distribution to, for example, / opt / fb3:
./install.sh -path /opt/fb3After that, you can stop the Firebird system service and start streaming conversion.
When stopping Firebird, it should be borne in mind that Firebid 2.5 processes in Classic mode usually start xinetd - therefore, it is either necessary to disable the firebird service for xinetd or to stop xinetd completely.
In firebird.conf for 3.0 on Linux, you do not need to set MaxUnflushed parameters (they work only on Windows) and change Firebird 2.5 settings.
In Linux, local (file) access of Firebird 2.5 is not equivalent to the embeded version for Windows - server 2.5 will work in the gbak process (without the network part), but access rights will be checked against the user base, which means that not only a login, but also a password will be required :
export ISC_USER=username ISC_PASSWORD=password
/opt/firebird/bin/gbak -b … база25 stdout\
|/opt/fb3/bin/gbak -c … stdin база30After successful conversion, you must first remove the "additional" Firebird 3.0, then the "main" Firebird 2.5, and only after that perform a clean installation of Firebird 3.0 - best of all from the regular tar.gz installer, and not through the repositories, because the version in the repositories may lag.
Also, after restoring the Linux database and reinstalling it, you need to check that the new database has the owner of the firebird user.
If this is not so, then it will be necessary to fix
chown firebird.firebird databaseSummary
In addition to saving time and disk space, streaming conversion has another important advantage - database conversion is done without deleting the existing Firebird 2.5, which greatly simplifies the rollback if the conversion is unsuccessful (most often due to lack of space or unexpected reboot during the migration process) .
Saving time is due to the fact that the "classic" conversion is "backup time" plus "recovery time". Recovery consists of two parts: reading data from a backup file and building an index.
In streaming conversion, the total time is obtained as “backup time plus five to ten percent” and “index building time”.
The specific results depend on the structure of the database, but on average, the recovery time is approximately equal to the double backup time. Therefore, if we take backup time per unit, then “classical conversion” - three units of time, flow - two units of time. An increase in TempCacheLimit also helps to reduce time.
In general, stream conversion in practice allows you to save 30-40% percent of the time of an alternate backup and restaurant.
Questions?
Please write all questions in the comments, or send to the author of the methodology and co-author of this article - Vasily Sidorov, iBase's leading systems engineer, at bs at ibase ru.
Only registered users can participate in the survey. Please come in.
What version of Firebird are you using?
- 33.3% Firebird 3.x 7
- 57.1% Firebird 2.5 12
- 4.7% Firebird 2.1 1
- 9.5% Firebird 2.0, 1.5 or 1.0 2