Text Analyzer: Authorship Recognition (End)

This article is about the authorship recognition algorithm implemented in the project “Text Analyzer”. At the end of the article, we will look at how frequency characteristics are assembled, and in general terms we will get acquainted with the Hamming neural system. ( Beginning and continuation ).

Article structure:

1. Authorship analysis
2. Getting to know the code
3. TAuthoringAnalyser internals and text storage
4. Leveling by state machine on strategies
5. Frequency response collection
6. Hamming Neural Network and Authorship Analysis

Additional materials:

• Sources of the project "Text Analyzer" (Borland C ++ Builder 6.0)
• Testing the Hamming neural system in Excel ( [xls] )
• Transition table for spacecraft that breaks text into levels ( [xls] )
• Calculation of the harmony of individual letters ( [xls] )
• Presentation of the text analyzer diploma project ( [ppt] )
• Presentation of the “Map of Harmony” project ( [ppt] )
• All of these materials are compressed ( [zip] , [7z] , [rar] )

5. Collection of frequency characteristics

The frequency characteristics of individual characters, two-, three-letter combinations, word frequency tables, etc. are most useful for recognizing authorship. This program implements only the simplest: counting symbol frequencies. Of course, this is not enough for a comprehensive analysis. Now the recognition accuracy, admittedly, is not very high. As far as I remember, in tests the probability of the correct answer reached 60-70 percent, and even then under ideal conditions. While developing the program, I was hoping to someday rewrite it, adding a comprehensive analysis of authorship based on many methods. Who knows, maybe I'll take it ...

So, the collection of frequency characteristics of the characters. The TCharFrequencyCalculator ( [h] ) class creates a frequency table, performing one pass through the text. Template Class TFrequencyTable ([h] ) can store a frequency table for objects of any type.

Copy Source | Copy HTML

1. class TCharFrequencyCalculator
2. {
3. private:
4.  
5.     TFrequencyTable _FTable;
6.  
7. public:
8.  
9.     TFrequencyTable & operator ()(TTextStringWrapper & tWrapper)
10.     {
11.         _FTable << ftm_Clear;
12.  
13.     TUInt i;
14.         TTextStringWrapper d; // Эта переменная, видимо, просто мусор в коде...
15.         for (i=tWrapper.Begin(); i<=tWrapper.End(); i++)
16.             _FTable << (tWrapper[i]);
17.     return _FTable;
18.     };
19.  
20.     TFrequencyTable & operator ()(const TTextString & tTextString)
21.     {
22.         _FTable << ftm_Clear;
23.  
24.         for (TSInt i=1; i<=tTextString.Length(); i++)
25.             _FTable << tTextString[i];
26.     return _FTable;
27.     };
28.  
29.     TCharFrequencyCalculator(){};
30. };

// Usage:
TCharFrequencyCalculator calculator;
TFrequencyTable charFrequencyTable = calculator (text);


It is worth paying attention to the fact that there are already two functions that count frequencies. The first accepts TTextStringWrapper ( [cpp] , [h] ) - a wrapper class over a text string. The “wrapper” is also known as the Adapter pattern (Adapter, Wrapper, [1] , [2] , [3]) It converts the interface of one class to the interface of another. One could make a truly universal calculator, but one would have to abstract from objects whose frequencies we are counting. A calculator should not even care what lists, tables or arrays store data there, where they come from, how many elements, and in what order they are. Adapted to the interface needed by the calculator, the lists of objects would be processed in the same way ... Did you feel like deja vu? That's right, we already discussed this when we considered the state machine manager. There, we abstracted from event lists using the Iterator pattern, and here, from lists of elements using the Adapter. This is its atypical application, it is worse than iterators: we would have to spawn hierarchies of adapters and frequency tables, so that they can be quickly replaced. In the end, the adapter would cease to be itself, and turned into a kind of abstract container. It would look something like this:

Copy Source | Copy HTML

1. class TFrequencyCalculator
2. {
3.     TFrequencyTable * operator ()(TWrapper * tWrapper, TFrequencyTable *table)
4.     {
5.         table << ftm_Clear;
6.  
7.         for (int i=tWrapper->Begin(); i<=tWrapper->End(); ++i)
8.             table << (tWrapper->at(i));
9.     return table;
10.     };
11. }
12.  
13. class TWordWrapper : public TWrapper
14. {
15. // ......
16.     virtual int Begin() const;
17.     virtual int End() const;
18.     virtual Word at(const int &index) const;
19. // ......
20. };
21.  
22. class TSentenceWrapper : public TWrapper {/*......*/};
23. class TWordsFrequencyTable : public TFrequencyTable {/*......*/};
24. class TSentenceFrequencyTable : public TFrequencyTable {/*......*/};
25.  
26. // Использование:
27. TWordWrapper wordWrapper = TWordWrapper(wordsList)
28. TFrequencyCalculator wordCalc;
29. TWordsFrequencyTable wordFrequencyTable = wordCalc(&wordWrapper, &wordFrequencyTable);
30.  
31. TSentenceWrapper sentenceWrapper = TSentenceWrapper(sentencesMap)
32. TFrequencyCalculator sentenceCalc;
33. TSentenceFrequencyTable sentenceFrequencyTable = sentenceCalc(&sentenceWrapper, &sentenceFrequencyTable);
34.  

6. Hamming Neural Network and Authorship Analysis

Finally, we, tired and battered, got to the very last step. Hamming Neural Network ( [1] , [2]) takes as a basis a set of binary vectors of the same length. They are called samples and are stored in the sample matrix. A test vector of the same length is fed to the input of the neural network. The number of inputs is equal to the size of the vector; for data of large volume of inputs there can be a lot. One of the advantages of the Hamming ANN (over the Hopfield ANN) is that no matter what the dimension of the input vector is, the structure of the neural network will not change. Layers - two, with the first fictitious; there are exactly as many outputs and neurons in each layer as there are samples in the matrix. The neural network is fast. Her task is to find in the matrix of samples a vector that is most “similar” to the input. Similarity is determined by the so-called Hamming distance. The shorter this distance, the more “similar” two vectors. Relatively speaking, the Hamming distance shows how many bits in these vectors do not match. It’s easy to calculate the Hamming distance, and the entire neural network is just a convenient representation of a few simple formulas. Having calculated some values, the neural network converges to the result: a vector with all zeros will be received at the outputs - excluding any one output where a unit appears. The index of this yield indicates the desired sample in the matrix of samples.

To load text samples in the neural network (class THamNeuroSystem: [cpp] , [h] ), they need to be converted to binary. This is done by the template connector class (THamNSConnector: [h] ). It's funny to see the neural network and connector code: I understand that this can be done much, much easier.

Copy Source | Copy HTML

1. template void THamNSConnector::ByteToBinaryVector(T DataItem, TSInt SizeOfData, TSampleVector *DestinationVector)
2. {
3. vector BoolBits;
4. T NewDataItem = DataItem;
5.  
6.     for (TSInt i=1; i
7.     {
8.         BoolBits.clear();
9.  
10.         BoolBits.push_back( NewDataItem & bitOne );
11.         BoolBits.push_back( NewDataItem & bitTwo );
12.         BoolBits.push_back( NewDataItem & bitThree );
13.         BoolBits.push_back( NewDataItem & bitFour );
14.         BoolBits.push_back( NewDataItem & bitFive );
15.         BoolBits.push_back( NewDataItem & bitSix );
16.         BoolBits.push_back( NewDataItem & bitSeven );
17.         BoolBits.push_back( NewDataItem & bitEight );
18.  
19.         for (TUInt j= 0; j
20.                 {
21.             if (BoolBits[j]) DestinationVector->push_back(1);
22.             else DestinationVector->push_back( 0);
23.                 }
24.  
25.     NewDataItem = NewDataItem >> 8;
26.     };
27. };
28.  
29. template TSampleVector THamNSConnector::VectorToBinaryVector(TVector SourceVector)
30. {
31.     TSInt SizeOfData;
32.     T DataItem;
33.     TUInt i;
34.     TSampleVector ResVector;
35.  
36.     SizeOfData = sizeof(T);
37.  
38.     for (i= 0; i
39.     {
40.         DataItem = SourceVector[i];
41.         ByteToBinaryVector(DataItem, SizeOfData, &ResVector);
42.     };
43.  
44.     return ResVector;
45. };
46.  

That's all. Apart from my attempts to improve the neural network, there’s nothing more to talk about. The neural network returns the index of a sample of that text, the characteristics of which it considers more similar to the characteristics of the text of an unknown author. How true the answer is, you can figure it out yourself by compiling the program. I did not have enough for extensive tests either at the time of writing the diploma, or now. Hope the article was helpful.

Sincerely.