Ude Cyrillic errors C# encoding detection

Fixing Cyrillic Encoding Detection Errors in C# with Ude

The Ude library in C# occasionally misidentifies the encoding of text files containing Cyrillic characters, incorrectly reporting 'Macintosh' instead of Windows-1251. This results in garbled text when reading—what should be readable Russian appears as unreadable 'mojibake'. The issue surfaces both in Notepad++ and in code using CharsetDetector.

In the GetFileEncoding method, Ude returns cdet.Charset as 'Macintosh', which causes errors during subsequent conversion:

public static Encoding GetFileEncoding(string filename)
{
    Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);

    using (FileStream fs = File.OpenRead(filename))
    {
        Ude.CharsetDetector cdet = new Ude.CharsetDetector();
        cdet.Feed(fs);
        cdet.DataEnd();

        if (cdet.Charset != null)
        {
            return Encoding.GetEncoding(cdet.Charset);
        }
    }

    return Encoding.GetEncoding(CodePages.Windows1251);
}

Applying this incorrect encoding in ChangeEncodingForMidFile worsens the problem: text is read improperly and written to a new file with corruption.

private static void ChangeEncodingForMidFile(string midFile, 
                                             Encoding midFileEncoding, 
                                             string newFullMidFileName)
{
    using StreamReader reader = new StreamReader(midFile, midFileEncoding);

    using StreamWriter writer = new StreamWriter(
        newFullMidFileName,
        false,
        Encoding.GetEncoding(CodePages.Windows1251)
    );

    writer.Write(reader.ReadToEnd());

    File.Delete(midFile);
    File.Move(newFullMidFileName, midFile);
}

Analyzing Differences Between MacCyrillic and Windows-1251

The MacCyrillic (CodePages.Macintosh) and Windows-1251 encodings use different byte ranges for Cyrillic characters. For typical Russian text without rare symbols, we can identify 'suspicious' bytes that are uncharacteristic of one encoding but common in the other:

• Suspicious for Macintosh: Bytes typical of Windows-1251 (0xAB, 0xB8–0xB9, 0xBB, 0xC0–0xC7, 0xC9, 0xCB–0xCF, 0xD2–0xDB, 0xFF).
• Suspicious for Windows-1251: Bytes typical of Macintosh (0x80–0x8F, 0x90–0x97, 0x99–0x9F).

These sets are used for heuristic revalidation.

private static readonly ArrayList<byte> StrangeForMacintoshBytes = new ArrayList<byte>
{
    0xAB, 
    0xB8, 0xB9, 0xBB,
    0xC0, 0xC1, 0xC2, 0xC3, 0xC4, 0xC5, 0xC6, 0xC9, 0xCB, 0xCC, 0xCD, 0xCE, 0xCF,
    0xD2, 0xD3, 0xD4, 0xD5, 0xD6, 0xD7, 0xD8, 0xD9, 0xDA, 0xDB,
    0xFF
};

private static readonly ArrayList<byte> StrangeForWindows1251Bytes = new ArrayList<byte>
{
    0x80, 0x81, 0x83, 0x84, 0x85, 0x86, 0x87, 0x88, 0x89, 0x8A, 0x8B, 0x8C, 0x8D, 0x8E, 0x8F,
    0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x99, 0x9A, 0x9B, 0x9C, 0x9D, 0x9E, 0x9F
};

Implementing CustomGetEncoding

The CustomGetEncoding method is triggered when Macintosh or MacCyrillic is detected:

if (cdet.Charset != null)
{
    var encoding = Encoding.GetEncoding(cdet.Charset);
    if (encoding.CodePage == CodePages.Macintosh || encoding.CodePage == CodePages.MacCyrillic)
        return CustomGetEncoding(encoding, filename);
}

Logic of CustomGetEncoding:

• Scan the file’s bytes.
• If a byte from StrangeForMacintoshBytes is found — return Windows-1251.
• Otherwise, if a byte from StrangeForWindows1251Bytes is found — return Macintosh.
• If no suspicious bytes are found — default to Windows-1251.

This resolves the case for plain Russian text but isn’t universally applicable.

Limitations and Recommendations

Drawbacks of this approach:

• Limited to Cyrillic text without rare characters.
• Doesn’t handle UTF-8 or other encodings.
• Relies on heuristics, not a reliable detection mechanism.

For users:

• Open the file in Notepad++.
• Try different encodings (Windows-1251, UTF-8) until no garbling occurs.
• Copy the text.
• Create a new file with the target encoding and paste the content.
• Save it.

Even this manual workaround doesn’t guarantee 100% success if Ude misidentifies the encoding despite visual correctness.

Other charset detection libraries exhibit similar issues. Automatic detection is inherently probabilistic and depends on byte frequency patterns in the text.

Key Takeaways

• Ude frequently confuses MacCyrillic and Windows-1251 for Cyrillic files.
• Using 'suspicious byte' heuristics fixes detection without modifying the library.
• The fix works only for basic Russian text.
• Always manually verify encoding in Notepad++ before processing.
• Charset detection is probabilistic—errors are unavoidable.

— Editorial Team