June 21, 2011 at 12:13How Text Recognition Technologies Help Fight Data Leaks
At interviews, we are often asked what the Department of Developer Products does. We briefly talk about the ABBYY FineReader Engine , but many job seekers know firsthand what the SDK is and how it can be used, and perceive our story as common words.
Today is a great example of how ABBYY FineReader Engine is used in a real product of a real company to solve real problems. Recently, the Russian company SECURIT has integrated the FineReader Engine into its data leak detection (DLP) products, including a product called Zgate. There was a press release about this , and we will carefully look at the technical side.
To detect data leaks, Zgate analyzes messages created by users in the process of work - mail messages (including those sent via webmail services), messages on social networks, forums, blogs and Internet pagers. To do this, it integrates with the mail server and proxy server and thus can control all traffic (usually it’s enough to control only outgoing traffic).
As soon as a suspicious message has been identified (the product has decided that the message contains confidential information), its transmission can be blocked, it can be quarantined until the end of the human check or the message can be immediately transferred and postponed for further verification. If the message flow is large and the suspension is not required or may be harmful for the company, you can put Zgate on a dedicated server and configure routing so that all traffic is duplicated to this server - Zgate will work completely independently and will not affect message transfer .
When searching for suspicious messages, the rules specified by the administrator are used. The search is performed using the dictionary, including using regular expressions and morphology (“rollback size” and “rollback size” should be identically reliable), or by comparing the transmitted message with sample documents - in this case, the same methods for identifying duplicates are used as in Google Search and other search engines. Text analysis methods include pleasant little things - for example, a product can consider letters from different alphabets with identical styles (“a” from Cyrillic and “Latin”) or numbers and certain combinations of letters (“w8ing” and “waiting”) to be equivalent.
This is a very simplified description, the product administrator’s manual alone takes about two hundred pages, and a good implementation of the product usually requires close involvement of the supplier in order to choose the right hardware, configure the system according to the needs of a particular customer. For example, you need to take into account the typical amount of traffic and the requirements for its control in order to select the number and capacity of servers sufficient to process messages as they arrive.
Zgate is a complex DLP product with well-understood functionality that could continue to work on its own. At the same time, before the product could only view digitally formatted documents — RTF, MS Word, PDF with a text layer, etc., but not all documents exist in organizations in this form. Documents can be in the form of graphic images (scans and photographs) or PDF without a text layer, and the transfer of documents in such formats may also need to be controlled.
After embedding the FR Engine, Zgate works exactly in the same scenarios, but now it can view and analyze image files. Accordingly, if before such files had to be either always skipped (unconditional trust) or always blocked (unconditional mistrust), now you can make an informed decision on each file.
Zgate extracts files from messages, passes them to the FR Engine for recognition, the recognized text is transmitted to the input of the same analysis methods as before. Thanks to the high recognition accuracy of texts in a variety of languages, the applicability of Zgate is expanding.
If there wasn’t an SDK, Zgate developers would have to do the recognition themselves, and this is not so easy (our company has been developing and improving recognition technologies for many years). Instead, they license our SDK and can, for example, simply write such code in C # (based on the example supplied with the SDK):
And that's it - FR Engine will open the image, recognize it with the default settings, export the result to a text file. If necessary, you can easily select the desired set of languages, other parameters. It doesn’t matter that several million lines of code of a wide variety of subsystems work inside - opening images, recognition, export. The FR Engine user is provided with a well-thought-out software interface that allows you to use all the features of the product.
Product features include, for example, almost two hundred recognition languages, many of them with dictionary support, the opening of a wide range of image formats, and very high recognition accuracy. The developers of the product may not do all this themselves, but license in the form of SDK. A text file with recognition results issued by FR Engine can be transferred to the input of text analysis methods and decide what to do with the image.
SECURIT Zgate is a great example of FR Engine integration. So part of the product’s functionality is created by the Zgate developers themselves, and partly licensed by us. This gives everyone the opportunity to do what he knows best.
A few words about Linux. Solutions like Zgate can allow themselves to work on one system - in this case, on Windows, because the cost of implementation is usually high enough so that the cost of a license for Windows does not cause concern - you just purchase the right hardware with the right operating system. For example, in a spherical company in a vacuum with several thousand users, the volume of outgoing traffic is usually about 20 gigabytes, a pair of servers of the HP ProLiant DL160 G6 E5620 level with a quad-core processor each are usually used for its processing. If the Zgate developers decide to switch to Linux, we have the FR Engine version for Linux .
Dmitry Meshcheryakov,
Department of Products for Developers
Today is a great example of how ABBYY FineReader Engine is used in a real product of a real company to solve real problems. Recently, the Russian company SECURIT has integrated the FineReader Engine into its data leak detection (DLP) products, including a product called Zgate. There was a press release about this , and we will carefully look at the technical side.
To detect data leaks, Zgate analyzes messages created by users in the process of work - mail messages (including those sent via webmail services), messages on social networks, forums, blogs and Internet pagers. To do this, it integrates with the mail server and proxy server and thus can control all traffic (usually it’s enough to control only outgoing traffic).
As soon as a suspicious message has been identified (the product has decided that the message contains confidential information), its transmission can be blocked, it can be quarantined until the end of the human check or the message can be immediately transferred and postponed for further verification. If the message flow is large and the suspension is not required or may be harmful for the company, you can put Zgate on a dedicated server and configure routing so that all traffic is duplicated to this server - Zgate will work completely independently and will not affect message transfer .
When searching for suspicious messages, the rules specified by the administrator are used. The search is performed using the dictionary, including using regular expressions and morphology (“rollback size” and “rollback size” should be identically reliable), or by comparing the transmitted message with sample documents - in this case, the same methods for identifying duplicates are used as in Google Search and other search engines. Text analysis methods include pleasant little things - for example, a product can consider letters from different alphabets with identical styles (“a” from Cyrillic and “Latin”) or numbers and certain combinations of letters (“w8ing” and “waiting”) to be equivalent.
This is a very simplified description, the product administrator’s manual alone takes about two hundred pages, and a good implementation of the product usually requires close involvement of the supplier in order to choose the right hardware, configure the system according to the needs of a particular customer. For example, you need to take into account the typical amount of traffic and the requirements for its control in order to select the number and capacity of servers sufficient to process messages as they arrive.
Zgate is a complex DLP product with well-understood functionality that could continue to work on its own. At the same time, before the product could only view digitally formatted documents — RTF, MS Word, PDF with a text layer, etc., but not all documents exist in organizations in this form. Documents can be in the form of graphic images (scans and photographs) or PDF without a text layer, and the transfer of documents in such formats may also need to be controlled.
After embedding the FR Engine, Zgate works exactly in the same scenarios, but now it can view and analyze image files. Accordingly, if before such files had to be either always skipped (unconditional trust) or always blocked (unconditional mistrust), now you can make an informed decision on each file.
Zgate extracts files from messages, passes them to the FR Engine for recognition, the recognized text is transmitted to the input of the same analysis methods as before. Thanks to the high recognition accuracy of texts in a variety of languages, the applicability of Zgate is expanding.
If there wasn’t an SDK, Zgate developers would have to do the recognition themselves, and this is not so easy (our company has been developing and improving recognition technologies for many years). Instead, they license our SDK and can, for example, simply write such code in C # (based on the example supplied with the SDK):
void processOneImage( FREngine.IEngine engine, string imageFilePath, string resultPath )
{
FREngine.FRDocument document = engine.CreateFRDocument();
try {
document.AddImageFile( imageFilePath, null, null );
document.Process( null, null, null );
document.Export( resultPath, FREngine.FileExportFormatEnum.FEF_Text, null );
} finally {
document.Close();
// кстати, у нас есть вакансии - www.abbyy.ru/vacancy
}
}
And that's it - FR Engine will open the image, recognize it with the default settings, export the result to a text file. If necessary, you can easily select the desired set of languages, other parameters. It doesn’t matter that several million lines of code of a wide variety of subsystems work inside - opening images, recognition, export. The FR Engine user is provided with a well-thought-out software interface that allows you to use all the features of the product.
Product features include, for example, almost two hundred recognition languages, many of them with dictionary support, the opening of a wide range of image formats, and very high recognition accuracy. The developers of the product may not do all this themselves, but license in the form of SDK. A text file with recognition results issued by FR Engine can be transferred to the input of text analysis methods and decide what to do with the image.
SECURIT Zgate is a great example of FR Engine integration. So part of the product’s functionality is created by the Zgate developers themselves, and partly licensed by us. This gives everyone the opportunity to do what he knows best.
A few words about Linux. Solutions like Zgate can allow themselves to work on one system - in this case, on Windows, because the cost of implementation is usually high enough so that the cost of a license for Windows does not cause concern - you just purchase the right hardware with the right operating system. For example, in a spherical company in a vacuum with several thousand users, the volume of outgoing traffic is usually about 20 gigabytes, a pair of servers of the HP ProLiant DL160 G6 E5620 level with a quad-core processor each are usually used for its processing. If the Zgate developers decide to switch to Linux, we have the FR Engine version for Linux .
Dmitry Meshcheryakov,
Department of Products for Developers
