To help the webmaster: Linux bash script to translate the site to a new encoding

“It is better to lose a day, then fly in an hour” © Wings, legs, tail

Not so long ago I was “lucky” to transfer a medium-sized website from one encoding to another. To be more precise, from windows-1251 to UTF-8. Then one more - more, on the third I broke down, and following the correct principle of the above, I had to lose a lot of time writing a script to automate this process, but then, after an hour, I still flew by.

The script settings are as follows:
Initial parameters:
SDIR = "/ usr / local / apache2 / htdocs / site.ru /" - the source directory of the site with a slash / at the end of
SCP = "CP1251" - the source (from) code page for iconv
EXT = ". * \. (Htm [l] * | php [3] * | js | css) $" - file extensions for transcoding (such as .htm, .html, .php, .php3, will be involved here. js, .css)
FCS = “windows-1251” - the name of the source code page to replace the meta charset = in files

Target parameters:
DROP_STRUCT = true - takes the value false, true, adjusting the condition: should the target directory

DDIR = "be cleaned at startup /usr/local/apache2/htdocs/new.site.ru/ "- the target directory of the site with a slash / at the end (must exist)
DCP =" UTF-8 "- the target (to) code page for iconv
TCS =" UTF -8 "- the name of the target code page to replace the meta charset = files

A by itself and the script itself: Now, some more useful things.

#!/bin/bash

# --- CONFIG SECTION ---

# Source Dir's params

SDIR="/usr/local/apache2/htdocs/site.ru/"    # with slash '/' in the end
SCP="CP1251"        # codepage for 'iconv'
EXT=".*\.(htm[l]*|php[3]*|js|css)$"    #files extensions for coding
FCS="windows-1251"    # charset for replace

# Destination Dir's params

DROP_STRUCT=true    # false, true

DDIR="/usr/local/apache2/htdocs/new.site.ru/"    # with slash '/' in the end
DCP="UTF-8"        # codepage for 'iconv'
TCS="UTF-8"        # new charset

# --- END CONFIG SECTION ---

# Drop structure
#
if $DROP_STRUCT
then
    rm -dfr $DDIR*
fi

# Make new copy
#
cp -aR $SDIR* $DDIR

# Flush miscoded files
#
find $DDIR -type f | grep -E "$EXT" | xargs -i rm -f {}

# Convert From To
#
find $SDIR -type f | grep -E "$EXT" | sed "s#$SDIR##" | xargs -i echo {} | \
while read f
do
iconv -c -f $SCP -t $DCP -o "$DDIR$f" "$SDIR$f"

# Revert MODE & OWNER
chmod `find "$SDIR$f" -maxdepth 0 -printf "%m"` "$DDIR$f"
chown `find "$SDIR$f" -maxdepth 0 -printf "%u:%g"` "$DDIR$f"

# Replace strings
perl -pi -e "s#content\s*\=\s*[\"'].*?charset\s*=\s*$FCS.*?[\"']#content=\"text/html; charset=$TCS\"#g" "$DDIR$f"
done




1. Perhaps even after transcoding to UTF-8 and replacing meta content with charset = UTF-8, you still see abracadabra or not what you would like. The thing here is that for the new site in UTF-8, it is necessary to replace the default_charset parameter for PHP itself, because in global variables, it is explicitly set for a different code page (windows-1251). I do this in the virtual host settings (httpd.conf) via:

php_admin_value default_charset UTF-8

2. As a rule, now any website wants databases, which you will need to transfer to UTF-8. It doesn’t make much effort if you have phpMyAdmin or mysqldump at hand, in extreme cases, for giant databases, you probably have to write a conversion script and temporarily suspend the service. The simplicity of the idea should be clear: we do a database dump, recode it using the same iconv and replace everything that is connected with the code pages with the desired data, fill everything into a new database.

An even more correct option suggested by 4m @ t! C is to do this on the tested database using ALTER TABLE tbl_name CONVERT TO CHARACTER SET charset_name;

A small gibberish can also come out of the database, which is manifested in the incorrect display of Russian 's' and'. Here, the default code page for MySQL will play a joke with us. To fix this problem, after connecting to the database, you will have to add the following lines to your site code:

mysql_query ("SET NAMES 'utf8'");

Or change default-character-set and default-collation for MySQL, if this is permissible.

Remember !!! Take these translations seriously, first performing them on a parallel version of the site and test, test, test.

Successful translations!

Source: Notes at hand