June 15, 2009 at 16:17USE from the inside (through the eyes of the admin)
Topic of the day, a lot of topics and Habré ( here and here )
It is not going to stop the experiment legality, content KIM (control test materials) and so on. I will try to concentrate directly on the technological aspect of preparing and conducting the exam.
Why I can be trusted is my experience of 2004-2007 in the Regional Center for the Study of Organizations of the Moscow Region (system administration, data collection, coordination with developers and regions, training, consultations, etc.). All this time we worked on the CROC technology (see below), in 2009 the technology changed by order of the DH. So, in order
Until recently, there were two technologies for organizing and conducting the USE: the Federal Testing Center (hereinafter CT), www.rustest.ru ) and CROC (CROC CJSC www.croc.ru , hereinafter simply CROC).
Technologies differ fundamentally. I will dwell on CROC technology as detailed as possible and in the end I will point out why it was better than CT (personal professional opinion of mine, colleagues and management).
CROC - MS SQL 2000 as the database server and WIN 2000 (2003) as the application server. The servers are distributed across different machines, on WIN 2000 there was an additional AD for the internal LAN.
On the application server there is the server part of the software package (hereinafter AIS), on the client machines, respectively, the client part, the user logs in via AD to the domain, work through the browser interface (IE6 required). According to security rules, no other software can be installed on a client (I also installed an office, rar and radmin are password-protected and only for a group of administrators). All software is 100% licensed (Windows, offices, server programs and the like).
As a supplement to the comments on the volumes on the disks: 7 gigs uncompressed dump base (full) and 90 gigs archive of pictures of all exams. The total number of man \ exams is about 120,000.
The entire local area network with a database and any exam data is PHYSICALLY disconnected from the Internet (to avoid data leakage).
At the AIS level, there is a distribution of user rights for access to the system (it is configured very flexibly, checkboxes indicate the necessary privileges for a group of users). Accordingly, you can log in to any client station through AD with the necessary rights and work as required.
The network structure for the exam for 50,000 people (we had the maximum load, the data must be processed and submitted for verification on the 4th day after the exam) consisted of:
1 - 3 servers (databases, AIS server, file archive of scanned pictures)
2 - 12-17 scan stations
3 - 20-25 verification stations
2 - 1-2 administrator stations for continuous monitoring of bottlenecks.
For the rest of the year, client-side scanning / verification stations work as normal for other users. At the time of exams, an image with a ready-made system was rolled up and withdrawn for our use.
Scanning stations - DS-300 high-speed scanners manufactured by LOMO, PANASONIC type KV-S3105C and low-power Fujitsu. DS worked through the network, KV and Fujitsu through skyies. The speed of the first two machines is impressive (first) - 90-120 sheets per minute in black and white about 200 dpi. When scanning, the form is cut into pieces (by reference marks), recognition passes instantly and files are transferred to the file server. All software is developed by CROC (did not intervene in their subcontracts).
Preparation for exams begins in October (approximately) and lasts all year. It includes the following steps:
1 - reconciliation of information on the educational institution, administrative directories, clarification of those responsible for the examinations (usually one person from the Moscow region, but if the district is large and / or people are little computer savvy - two, one techie, one manager). Municipal coordinators - in general, a separate issue, they would put a monument. Usually these are women from education authorities.
2 - collection of data on graduates of public institutions (the most complete map, name, passport, planned for exams). The collection is carried out by specialized client software (a program without installation) with a very clear interface. The capabilities of the software are much wider - in addition to basic data collection, this is also the collection of analytical data for stats. Analysis. For example - data on the teacher (experience, specialization, date of courses), annual assessments on the subject, textbook and much more. These data are filled in throughout the year (because a very large amount of information must be collected and entered).
3 - reconciliation of passport data of graduates (on paper, signed by the head of the public institution)
4 - from the publication of the exact dates of the exams, the final collection of data on registration for exams and subsequent verification (on paper, signed by the head of the public institution and the student's signature next to their own passport data and selected exams). The term of this stage is very short, because there is very little time from the order with the dates of exams and the deadline for submitting data to higher authorities. Why so strictly - to print the required amount of CMM
Very interesting things were going on with reconciliations. Pupils throughout the year should at least 2-3 times see the lists with their data and, if necessary, correct (name, passport). Not only did each reconciliation bring about 5-10% of changes to the database (this is all done by the center’s hands), so many forms with corrections came out on exams. Children are completely inattentive to their data, and because of this, they may also have problems obtaining results (data correction goes through the FTsT and this is very long and difficult for us).
So, all the data is collected, verified, go to the exam.
A few days before the exam, we receive from the FTsT the ordered number of CMMs (with a certain limit in case of marriage or a lack of audience). Everything is very serious - sealed, taken according to the inventory by the head of the center.
Printing materials for the exam
In the center sets are printed for each audience and PES (point of the exam). They printed on HP in wild streaming mode, from the second year they got the hang of simultaneously printing from two clients to two printers at once. The printed kit is removed from the printer, the statement is signed and sealed in the so-called security bag. After the packs are completed for each district \ PES and stored in a special room (with combination lock). Cartridges are replaced every 3-4 hours, dust is a pillar, the lungs slowly die from 3-4x simultaneously working HP-shnikov.
The day before the exam, municipal coordinators arrive - according to the inventory, packages are issued, the quantity and general data are verified. Hours at 18-19 the last package was issued, cognac and sleep, for the next 3-4 days as a nightmare.
From 8 a.m. (opening time of packages on the ground), the boss and I were in the center — answers to all possible questions during the exam and nervous expectation of marriage in KIM (there was such a thing, at the printing stage, marriage, the child has to fill out a change of option), errors registration of children for the exam (several times, municipal coordinators forgot to bring the student to the exam, but he calmly did not sign on the reconciliation statements). Some lull after 10am (the exam went).
From about 14 days, the delivery of materials from PES begins. Acceptance is as follows:
1 - reconciliation of packages by the head of the center
Reception and scanning after the exam
2 - the municipal coordinator and the operator of the scanning station “retire” in the hall and scan by package, after processing each audience package, data (machine processing) is checked against the sheets, not a single form should be lost. At this stage, several times there were cases that in the PES forgot the form. In this case, the reception of the area is suspended until clarified. At the client machine level, there is a strict count of the number of forms issued and scanned for each audience. Substitution is not possible. Forms are registered and with a unique code; during scanning, all fields are verified.
There was a curious case when in a PES a student went to the toilet with an unresolved part C and flushed the form into the toilet (he didn’t drain it well). Upon acceptance, they found out, they sent the messenger to the point, he caught a form in the toilet and brought it to us with his nose, we had to process it.
3 - I, as an administrator, in the interval between resolving current issues, check the processed audiences, if the number of issued / accepted forms matches, I transfer the audience to check.
Documents from a large area, Odintsovo district (colleagues recalled), are waiting for the line for scanning, in this bundle of forms are 2500 participants
4 - after this, the forms with the work are verified. Verification is a delicate matter - each client machine with a verification base is self-learning. If the operator confirms the value of one field 2 times, it falls into the dictionary and with a probability of 99% will be automatically worked out. Occupation is very tedious, because usually ends much later than scanning (2-6 hours at night). It is at this stage that the processing of "random" errors occurs. If the child has put in part A the answer in the 5th box (it is reserved), the extra \ service fields are filled - all this falls into the verifier.
5 - administrator - conducts general monitoring of the system, corrects any errors on the lists and so on. Cases of placing the “student did not show up” label and the completed form (the system displays such cases) are monitored, otherwise the work will not be counted. Lost blanks - the system was unable to recognize the scanned number and did not attach the blank to the audience.
6 - after the end of verification, you can print packages for the work of experts. They begin from 10 am the next day of the exam. Therefore, from 8-9 p.m. at least 2 printing stations of works C are launched. Issuance is strictly random - the forms are unnamed, the expert does not know who he is checking, each work is given to two experts, if the discrepancy in their estimates is higher than the specified mark - the work is shown as requiring additional verification and printed for the "third expert." The third expert cannot be one of the two who have already checked, is appointed manually with the instructions of the chairman of the inspection commission.
The verification of part C takes 3-4 days. As soon as all the data is processed and not a single form is lost, a CSV is generated with the results. This file is encrypted on the server (Crypto Pro, the keys are stored on a separate medium in a safe and issued for signature) and sent through an encrypted channel through a special program to the FTsT for evaluation.
I will not talk about scaling, who are interested in finding calculations. A fairly complex mathematical model, is built in the FTsT when obtaining results from all over Russia.
After 3-5 days, the processed results with the estimates come from the FTsT to CSV, are downloaded to the database and the protocols are printed (each signed by the seal of the Minister of Education of the region) and issued to each district. Recent years have begun to allow the publication of data through sites, this year we have done the same functionality on our Mosoblast site (which caused an increase in traffic and put the host server)).
In principle, in general terms, everything. In real time, when exams are held with an interval of 3-5 days and / or 2-3 exams on one day, this results in 2-3 weeks of 100% presence at work, without days off, 25 hours a day. But when you know that behind our backs of 50,000 normally passed children and satisfied parents, it’s worth it.
I can’t remember the appeals for the exam (although sometimes it would be worth it, but the children are tired and do not immediately understand the situation). According to the results - about 500 pieces per 50,000.
A special commission is going to appeal, the recognized forms are printed, copies of the originals, a reconciliation is made for machine recognition errors, and experts on objects with the child and parent walk through all the tasks step by step. Appeals are divided into categories: rejected and reviewed. 99% of the appeals reviewed are scored (rarely when justified, usually flowers, sweets, tears). A couple of times they were especially “insolent” lowering the score. About 40% remain with the same result, that is, the appeal is rejected. I would also like to tell the excellent students - if you don’t have 1-2 points at the university - be sure to file an appeal, such cases (there are not many) are always positively considered.
1 - banal inattention at the level of districts and schools. Coordinators lose / forget children, children do not pay attention to their passport data and the selected exam dates. It creates many problems for us in reconciling data and for children in case of complications.
2 - low computer literacy of the coordinators. In no case do not want to blame them, but the fact remains. Not every district can find a specialist to solve technical issues (usually consists of working with e-mail, excel, word, software for collecting information). From there, and my joke about the "red and blue Internet."
3 - lack of Internet in education management bodies. Many work through a modem; they send us data from home computers. I just keep silent about schools - rarely there is rarely a network, data exchange via floppy disks usually (!!!!)
4 - complete chaos in relations with FTsT. A couple of examples. They write and approve a database format with representatives of CROC and the ministry. In fact, we find a discrepancy - we approved one, uploaded the data to CSV, but they are not loaded into the FTsT database - the order of the fields has changed, the data format and so on. There is abuse, Krok frantically writes a hack for his base in order to fit the supposedly approved format to the new requirements (not expressed by anyone anywhere). Next - the output of the results, for some reason our file is not accepted by them (the format is erroneous or otherwise) - the silence is complete on their part, only by calling (and getting through to them oh how difficult) it turns out that there is a jamb. Again this time, the nerves of us and the children \ parents. When communicating, one gets the impression of a classic “shovel” store of jokes. Very, very annoying such facts,
5 - even greater chaos at a higher level. Inconsistency of orders of reality, one decree contradicts another and the like. Eternal glory to our boss, who spared no time for gray hair, raised such questions as quickly as possible (although not at all at our level)
6 - and, perhaps most importantly, IN CAREFULNESS and UNKNOWLEDGE of the rules during the exam in paragraph. We conduct training of specialists of each PES all year round, we are ready to constantly answer any question. After that it turns out that they don’t know which side to open the package and issue the statement.
1 - very, very friendly software (not without sins, but compared to the software FTsT - paradise on earth). Implementation specialists answered almost any question. Many own developments (sinful, I wrote a couple of phrases on php for the speed of the same procedures). A single complex, flexible and expandable, developed by one office. Soft CT was assembled from different pieces of different developers, based on ABBYY. The work is unintuitive, uncomfortable, and in my opinion just crooked.
2 - convenience of the final configuration of the system in the center, streamlined technology literally to the skin. When the center was transferred to the FTsT - the labor costs during the execution increased significantly (as did the level of risks)
3 - high data security. From the center and from the PES in the days of a single drop of information was not lost. In addition to the administrator, no one can physically make changes to the database; at the client level, everything is monitored and saved. Only the admin has the ability to make a “byaku”. A couple of times the truth came to me personally with a quiet offer of money :) but to no avail. As far as I know, similar cases did not pass at the district level either.
4 - almost 100% convenience for the end user - the student. A letterhead, indicating the school, PES, only put the signature and solve problems (the CFT also needs to enter the name and surname - another reason for error)
5 - huge opportunities for statistical analysis of not only the exam but also the entire educational system. The data array allowed this (provided 100% field coverage). We used SQL Analyze Server with web components (analytics module - development of the same CROC). Well, directly from the database requests were drawn particularly interesting indicators. You can compare almost anything - an assessment of the duration of the study of the subject in conjunction with the textbook, the correspondence of the assessment of the exam and ready-made indicators on the growth curve of the final score from the 5th grade. Many wonderful things, such data was not anywhere else. In practice, it was covered with a copper basin, although there was a separate center under the ministry for statistical work.
Results (from work experience) - percentage of machine processing errors - 0.1%, error rate of the verifier - 2 cases per 50,000 people \ exams. That is, everything is clean from our side.
I repeat - everything written - my personal opinion on the experience of working on this technology and the experience of the joint colleagues of our center during the period until 2007. And I'm sorry for the language of presentation - the Chukchi is not a writer, a Chukchi reader.
1 - what is written: in C # (ASP.Net) - app. server part, T-SQL - client part, Active-X (C ++, VB6, C #) + VBScript - client.
2 - how the options are issued for the audience: a fairly complex algorithm that spreads the KIM options fairly evenly across the PES audiences, trying not to give the same options to the \ "cross \" neighbors (left-right-up-down)
If you are interested in individual, undisclosed ones, aspects of technology - write, I will try to complement. Well, if there is no account on the Habré - write in the comments on the blog
Workplace admin))
It is not going to stop the experiment legality, content KIM (control test materials) and so on. I will try to concentrate directly on the technological aspect of preparing and conducting the exam.
Why I can be trusted is my experience of 2004-2007 in the Regional Center for the Study of Organizations of the Moscow Region (system administration, data collection, coordination with developers and regions, training, consultations, etc.). All this time we worked on the CROC technology (see below), in 2009 the technology changed by order of the DH. So, in order
Technology
Until recently, there were two technologies for organizing and conducting the USE: the Federal Testing Center (hereinafter CT), www.rustest.ru ) and CROC (CROC CJSC www.croc.ru , hereinafter simply CROC).
Technologies differ fundamentally. I will dwell on CROC technology as detailed as possible and in the end I will point out why it was better than CT (personal professional opinion of mine, colleagues and management).
CROC - MS SQL 2000 as the database server and WIN 2000 (2003) as the application server. The servers are distributed across different machines, on WIN 2000 there was an additional AD for the internal LAN.
On the application server there is the server part of the software package (hereinafter AIS), on the client machines, respectively, the client part, the user logs in via AD to the domain, work through the browser interface (IE6 required). According to security rules, no other software can be installed on a client (I also installed an office, rar and radmin are password-protected and only for a group of administrators). All software is 100% licensed (Windows, offices, server programs and the like).
As a supplement to the comments on the volumes on the disks: 7 gigs uncompressed dump base (full) and 90 gigs archive of pictures of all exams. The total number of man \ exams is about 120,000.
The entire local area network with a database and any exam data is PHYSICALLY disconnected from the Internet (to avoid data leakage).
At the AIS level, there is a distribution of user rights for access to the system (it is configured very flexibly, checkboxes indicate the necessary privileges for a group of users). Accordingly, you can log in to any client station through AD with the necessary rights and work as required.
The network structure for the exam for 50,000 people (we had the maximum load, the data must be processed and submitted for verification on the 4th day after the exam) consisted of:
1 - 3 servers (databases, AIS server, file archive of scanned pictures)
2 - 12-17 scan stations
3 - 20-25 verification stations
2 - 1-2 administrator stations for continuous monitoring of bottlenecks.
For the rest of the year, client-side scanning / verification stations work as normal for other users. At the time of exams, an image with a ready-made system was rolled up and withdrawn for our use.
Scanning stations - DS-300 high-speed scanners manufactured by LOMO, PANASONIC type KV-S3105C and low-power Fujitsu. DS worked through the network, KV and Fujitsu through skyies. The speed of the first two machines is impressive (first) - 90-120 sheets per minute in black and white about 200 dpi. When scanning, the form is cut into pieces (by reference marks), recognition passes instantly and files are transferred to the file server. All software is developed by CROC (did not intervene in their subcontracts).
Data collection.
Preparation for exams begins in October (approximately) and lasts all year. It includes the following steps:
1 - reconciliation of information on the educational institution, administrative directories, clarification of those responsible for the examinations (usually one person from the Moscow region, but if the district is large and / or people are little computer savvy - two, one techie, one manager). Municipal coordinators - in general, a separate issue, they would put a monument. Usually these are women from education authorities.
2 - collection of data on graduates of public institutions (the most complete map, name, passport, planned for exams). The collection is carried out by specialized client software (a program without installation) with a very clear interface. The capabilities of the software are much wider - in addition to basic data collection, this is also the collection of analytical data for stats. Analysis. For example - data on the teacher (experience, specialization, date of courses), annual assessments on the subject, textbook and much more. These data are filled in throughout the year (because a very large amount of information must be collected and entered).
3 - reconciliation of passport data of graduates (on paper, signed by the head of the public institution)
4 - from the publication of the exact dates of the exams, the final collection of data on registration for exams and subsequent verification (on paper, signed by the head of the public institution and the student's signature next to their own passport data and selected exams). The term of this stage is very short, because there is very little time from the order with the dates of exams and the deadline for submitting data to higher authorities. Why so strictly - to print the required amount of CMM
Very interesting things were going on with reconciliations. Pupils throughout the year should at least 2-3 times see the lists with their data and, if necessary, correct (name, passport). Not only did each reconciliation bring about 5-10% of changes to the database (this is all done by the center’s hands), so many forms with corrections came out on exams. Children are completely inattentive to their data, and because of this, they may also have problems obtaining results (data correction goes through the FTsT and this is very long and difficult for us).
So, all the data is collected, verified, go to the exam.
Conducting an exam
A few days before the exam, we receive from the FTsT the ordered number of CMMs (with a certain limit in case of marriage or a lack of audience). Everything is very serious - sealed, taken according to the inventory by the head of the center.
Printing materials for the exam
In the center sets are printed for each audience and PES (point of the exam). They printed on HP in wild streaming mode, from the second year they got the hang of simultaneously printing from two clients to two printers at once. The printed kit is removed from the printer, the statement is signed and sealed in the so-called security bag. After the packs are completed for each district \ PES and stored in a special room (with combination lock). Cartridges are replaced every 3-4 hours, dust is a pillar, the lungs slowly die from 3-4x simultaneously working HP-shnikov.The day before the exam, municipal coordinators arrive - according to the inventory, packages are issued, the quantity and general data are verified. Hours at 18-19 the last package was issued, cognac and sleep, for the next 3-4 days as a nightmare.
Exam day
From 8 a.m. (opening time of packages on the ground), the boss and I were in the center — answers to all possible questions during the exam and nervous expectation of marriage in KIM (there was such a thing, at the printing stage, marriage, the child has to fill out a change of option), errors registration of children for the exam (several times, municipal coordinators forgot to bring the student to the exam, but he calmly did not sign on the reconciliation statements). Some lull after 10am (the exam went).
From about 14 days, the delivery of materials from PES begins. Acceptance is as follows:
1 - reconciliation of packages by the head of the center
Reception and scanning after the exam
2 - the municipal coordinator and the operator of the scanning station “retire” in the hall and scan by package, after processing each audience package, data (machine processing) is checked against the sheets, not a single form should be lost. At this stage, several times there were cases that in the PES forgot the form. In this case, the reception of the area is suspended until clarified. At the client machine level, there is a strict count of the number of forms issued and scanned for each audience. Substitution is not possible. Forms are registered and with a unique code; during scanning, all fields are verified.There was a curious case when in a PES a student went to the toilet with an unresolved part C and flushed the form into the toilet (he didn’t drain it well). Upon acceptance, they found out, they sent the messenger to the point, he caught a form in the toilet and brought it to us with his nose, we had to process it.
3 - I, as an administrator, in the interval between resolving current issues, check the processed audiences, if the number of issued / accepted forms matches, I transfer the audience to check.
Documents from a large area, Odintsovo district (colleagues recalled), are waiting for the line for scanning, in this bundle of forms are 2500 participants
4 - after this, the forms with the work are verified. Verification is a delicate matter - each client machine with a verification base is self-learning. If the operator confirms the value of one field 2 times, it falls into the dictionary and with a probability of 99% will be automatically worked out. Occupation is very tedious, because usually ends much later than scanning (2-6 hours at night). It is at this stage that the processing of "random" errors occurs. If the child has put in part A the answer in the 5th box (it is reserved), the extra \ service fields are filled - all this falls into the verifier.5 - administrator - conducts general monitoring of the system, corrects any errors on the lists and so on. Cases of placing the “student did not show up” label and the completed form (the system displays such cases) are monitored, otherwise the work will not be counted. Lost blanks - the system was unable to recognize the scanned number and did not attach the blank to the audience.
6 - after the end of verification, you can print packages for the work of experts. They begin from 10 am the next day of the exam. Therefore, from 8-9 p.m. at least 2 printing stations of works C are launched. Issuance is strictly random - the forms are unnamed, the expert does not know who he is checking, each work is given to two experts, if the discrepancy in their estimates is higher than the specified mark - the work is shown as requiring additional verification and printed for the "third expert." The third expert cannot be one of the two who have already checked, is appointed manually with the instructions of the chairman of the inspection commission.
The verification of part C takes 3-4 days. As soon as all the data is processed and not a single form is lost, a CSV is generated with the results. This file is encrypted on the server (Crypto Pro, the keys are stored on a separate medium in a safe and issued for signature) and sent through an encrypted channel through a special program to the FTsT for evaluation.
I will not talk about scaling, who are interested in finding calculations. A fairly complex mathematical model, is built in the FTsT when obtaining results from all over Russia.
After 3-5 days, the processed results with the estimates come from the FTsT to CSV, are downloaded to the database and the protocols are printed (each signed by the seal of the Minister of Education of the region) and issued to each district. Recent years have begun to allow the publication of data through sites, this year we have done the same functionality on our Mosoblast site (which caused an increase in traffic and put the host server)).
In principle, in general terms, everything. In real time, when exams are held with an interval of 3-5 days and / or 2-3 exams on one day, this results in 2-3 weeks of 100% presence at work, without days off, 25 hours a day. But when you know that behind our backs of 50,000 normally passed children and satisfied parents, it’s worth it.
Appeals
I can’t remember the appeals for the exam (although sometimes it would be worth it, but the children are tired and do not immediately understand the situation). According to the results - about 500 pieces per 50,000.
A special commission is going to appeal, the recognized forms are printed, copies of the originals, a reconciliation is made for machine recognition errors, and experts on objects with the child and parent walk through all the tasks step by step. Appeals are divided into categories: rejected and reviewed. 99% of the appeals reviewed are scored (rarely when justified, usually flowers, sweets, tears). A couple of times they were especially “insolent” lowering the score. About 40% remain with the same result, that is, the appeal is rejected. I would also like to tell the excellent students - if you don’t have 1-2 points at the university - be sure to file an appeal, such cases (there are not many) are always positively considered.
Bottlenecks of the whole procedure:
1 - banal inattention at the level of districts and schools. Coordinators lose / forget children, children do not pay attention to their passport data and the selected exam dates. It creates many problems for us in reconciling data and for children in case of complications.
2 - low computer literacy of the coordinators. In no case do not want to blame them, but the fact remains. Not every district can find a specialist to solve technical issues (usually consists of working with e-mail, excel, word, software for collecting information). From there, and my joke about the "red and blue Internet."
3 - lack of Internet in education management bodies. Many work through a modem; they send us data from home computers. I just keep silent about schools - rarely there is rarely a network, data exchange via floppy disks usually (!!!!)
4 - complete chaos in relations with FTsT. A couple of examples. They write and approve a database format with representatives of CROC and the ministry. In fact, we find a discrepancy - we approved one, uploaded the data to CSV, but they are not loaded into the FTsT database - the order of the fields has changed, the data format and so on. There is abuse, Krok frantically writes a hack for his base in order to fit the supposedly approved format to the new requirements (not expressed by anyone anywhere). Next - the output of the results, for some reason our file is not accepted by them (the format is erroneous or otherwise) - the silence is complete on their part, only by calling (and getting through to them oh how difficult) it turns out that there is a jamb. Again this time, the nerves of us and the children \ parents. When communicating, one gets the impression of a classic “shovel” store of jokes. Very, very annoying such facts,
5 - even greater chaos at a higher level. Inconsistency of orders of reality, one decree contradicts another and the like. Eternal glory to our boss, who spared no time for gray hair, raised such questions as quickly as possible (although not at all at our level)
6 - and, perhaps most importantly, IN CAREFULNESS and UNKNOWLEDGE of the rules during the exam in paragraph. We conduct training of specialists of each PES all year round, we are ready to constantly answer any question. After that it turns out that they don’t know which side to open the package and issue the statement.
Positive aspects of technology
1 - very, very friendly software (not without sins, but compared to the software FTsT - paradise on earth). Implementation specialists answered almost any question. Many own developments (sinful, I wrote a couple of phrases on php for the speed of the same procedures). A single complex, flexible and expandable, developed by one office. Soft CT was assembled from different pieces of different developers, based on ABBYY. The work is unintuitive, uncomfortable, and in my opinion just crooked.
2 - convenience of the final configuration of the system in the center, streamlined technology literally to the skin. When the center was transferred to the FTsT - the labor costs during the execution increased significantly (as did the level of risks)
3 - high data security. From the center and from the PES in the days of a single drop of information was not lost. In addition to the administrator, no one can physically make changes to the database; at the client level, everything is monitored and saved. Only the admin has the ability to make a “byaku”. A couple of times the truth came to me personally with a quiet offer of money :) but to no avail. As far as I know, similar cases did not pass at the district level either.
4 - almost 100% convenience for the end user - the student. A letterhead, indicating the school, PES, only put the signature and solve problems (the CFT also needs to enter the name and surname - another reason for error)
5 - huge opportunities for statistical analysis of not only the exam but also the entire educational system. The data array allowed this (provided 100% field coverage). We used SQL Analyze Server with web components (analytics module - development of the same CROC). Well, directly from the database requests were drawn particularly interesting indicators. You can compare almost anything - an assessment of the duration of the study of the subject in conjunction with the textbook, the correspondence of the assessment of the exam and ready-made indicators on the growth curve of the final score from the 5th grade. Many wonderful things, such data was not anywhere else. In practice, it was covered with a copper basin, although there was a separate center under the ministry for statistical work.
Results (from work experience) - percentage of machine processing errors - 0.1%, error rate of the verifier - 2 cases per 50,000 people \ exams. That is, everything is clean from our side.
I repeat - everything written - my personal opinion on the experience of working on this technology and the experience of the joint colleagues of our center during the period until 2007. And I'm sorry for the language of presentation - the Chukchi is not a writer, a Chukchi reader.
UPD information from a leading system developer
1 - what is written: in C # (ASP.Net) - app. server part, T-SQL - client part, Active-X (C ++, VB6, C #) + VBScript - client.
2 - how the options are issued for the audience: a fairly complex algorithm that spreads the KIM options fairly evenly across the PES audiences, trying not to give the same options to the \ "cross \" neighbors (left-right-up-down)
If you are interested in individual, undisclosed ones, aspects of technology - write, I will try to complement. Well, if there is no account on the Habré - write in the comments on the blog
Workplace admin))