February 23, 2016 at 17:10ECMA-130 (Compact Disc) on the fingers
Three crossings is equal to one fire. Raking out an old box that smelled of acetone, with multilayer dust on the bottom (it’s good that my wife didn’t see it), I came across painfully familiar CDs. Here is one of my favorite childhood films ... but my once favorite arcade toy ... The
strange thing is curiosity. Here is a copy of the CD on the table - a data storage format hopelessly outdated in our 21st century ; but still I wonder how the data is stored there? .. What is the data storage stack itself? .. How are the errors corrected? .. What is the code redundancy? ..
As a child, I had enough knowledge about the laser beam, about some head, about "This spinning thing" and the mysterious pit .
No sooner said than done. Looking over the ECMA-130 standard(there is, by the way, the domestic standard: GOST 27667-88 ) found a lot of interesting details. For example, I guessed about redundancy, but I couldn’t even think that for the recording of 700 MB of data “in reality” 1943 MB are written (That is, 2.776 times more) ...
Schematically, the whole stack can be represented by a picture:
Consider the stack “from top to bottom”.
That is, from the moment of data transfer to the drive to the recording of the pits themselves.
It should be said that not the entire area of the disk is used to write / store / read information.
Compact disc is divided into zones ( areas ):
It is the information zone that is of interest to us, so we will talk about it in more detail. Information area is divided into the following "subzones":
All three translations are made by GOST ... So I will not translate from myself. What does the “program" have to do with this when it comes to data - I can’t put my mind to it. If someone on Habré can answer me why the user data zone was translated as a "program zone" I will be extremely grateful!
Phew ... I hope they finished the physical educational program. Of course, I missed many points related to widths and lengths; with coating method; beam wavelength; and other physical properties. However, firstly, this is not the purpose of this article; secondly, I myself have not yet figured out more than half of the information. Yes and no desire, to be honest ... My curiosity is purely "programmer" orientation;)
The first phase is a breakdown into “information tracks”. There are already two recording options, in digital form ( Digital Data Tracks, DDT ) and audio data ( Audio Tracks ).
In the future, we will consider only digital data. All of the following information is correct only for DDT .
Data is divided into 8 bits (one byte) and grouped into sectors .
It is funny that the number of sectors in the disk is not defined by the standard. It depends on how much data “get” written to disk ...
It seems a little ridiculous. Nevertheless, this absurdity was used by various companies to detect counterfeit disks in their era. (Since this is “sidestepping,” I wrote in more detail under the spoiler a bit lower at the end of the chapter.)
Of course, the track length is more or less fixed and in general QoS can be guaranteed.
There are three ways to (mode) write data in sectors:
Here are the pictures:
Useful information ( User Mode ) consists of 2048 bytes in mode 01 ; or 2336 bytes in 02 mode .
Which mode to choose? It all depends on what reliability requirement you require.
Sector Mode 01 is more reliable, as it uses additional EDC verification and PQ encoding.
Error Detection Code (EDC) , as the name implies, is intended only to detect, but not to correct errors.
Here is its polynomial: P (x) = (x 16 + x 15 + x 2 +1) (x 16 + x 2 + x + 1)
The eight bytes of the Intermediate field are filled with zero bytes ( 0x00 ).
I don’t know why they were left ... Maybe “in reserve” (they like it in IT standards),or maybe it's an insidious plan for steganographic data transfer.
Reed-Solomon Product-like Code (RSPC) , it is also P + Q encoding is used from 12 to 2075 bytes of data in mode 01 . I will omit the details; you can read them in Annex A of the ECMA-130 standard .
Bytes 12 through 2075 and verification bytes 2 076 through 2,351 are 2340 bytes of data. This data is divided into two blocks of 1170 bytes each. The division occurs as in school physical education lessons. " Pay off on the perrrvy-vtorrrroy! " That is, to odd and even bytes.
Next comes coding with external and internal codes. External is called P-coding , internal Q-coding .
The most difficult ECMA-130 stack to understand is passed. Now it will be much easier.
We pass to scrambling. This is how one scrambling sector looks like:
Each such sector is called the Scrambled Sector .
Each Scrambled Sector is divided into frames, 24 bytes long each.
These frames are called: F 1 frame .
Each Scrambled Sector we have consists of 2352 bytes.
Accordingly, each sector is divided into 98 frames.
Cross Interleaved Reed-Solomon (CIRC) coding is performed for each F 1 frame .
This is error correcting code with an input word length of 24 bytes and an output word length of 32 bytes.
Moreover, unlike EDC and RSPC encoding, CIRC encoding is used for all Sector Mode .
The resulting sequence of 32 bytes is called F 2 frames .
At the beginning of each F 2 frame , one check byte is added and an F 3 frame with a length of 33 bytes is obtained (32 + 1 = 33).
At this point, data bytes (8 bits) are converted to 14 bits of data. The conversion is carried out according to the table.
I will not give the whole table, you can find it in Annex D of the ECMA-130 standard .
Why 8-to-14 coding is necessary in the standard is not indicated. (The standard is not required to answer the questions WHY, the standard should contain answers to the questions in MAY) ...
I have one hypothesis. The fact is that the real world is not so "perfect" as programmers see it. For example, a drawn point is a small "blot", and a drawn line always has an area; otherwise, our eyes would not see a point and a line ... For this reason, I will venture to make a number of assumptions. I emphasize that I have never worked professionally with the manufacture of CDs. These are just assumptions. (Discussion in the comments is categorically welcome!).
Hypothesis.
Let's see how redundant the CD protocol is:
Multiplying everything, we get: 1.148 ⋅ 1.005 ⋅ 1.375 ⋅ 1.750 = 2.776 . Thus, as a result, 2.776 times more information is recorded on the disk itself than “useful information”.
For example, when the amount of “useful information” is 700 MB, 1943 MB of data is actually written to the disk .
Sector Mode 02 does not use PQ encoding. For this mode, the redundancy is: 1.007 ⋅ 1.005 ⋅ 1.375 ⋅ 1.750 = 2.435 .
There is a standard SCSI Multimedia Commands . It describes the raw data reading commands. The READ CD and WRITE CD commands read 2352 bytes of data from the entire sector. However, I did not find commands for reading F-fraim ... In principle, if you write redundant information that is not afraid of partial losses (for example, video, telematics),
you can do without F 1 -F 2 -F 3 frames by increasing the “payload” in 1.375 times.
There are also a number of unused areas in the CD (for example, the same Intermidiate ), which can also be used. For example, for the tasks of steganography.
Unfortunately, I did not find OpenSource code that implements these features ...
If there are specialists on this subject on the hub, I will be glad to get a link (plus a karma from me).
strange thing is curiosity. Here is a copy of the CD on the table - a data storage format hopelessly outdated in our 21st century ; but still I wonder how the data is stored there? .. What is the data storage stack itself? .. How are the errors corrected? .. What is the code redundancy? ..
As a child, I had enough knowledge about the laser beam, about some head, about "This spinning thing" and the mysterious pit .
No sooner said than done. Looking over the ECMA-130 standard(there is, by the way, the domestic standard: GOST 27667-88 ) found a lot of interesting details. For example, I guessed about redundancy, but I couldn’t even think that for the recording of 700 MB of data “in reality” 1943 MB are written (That is, 2.776 times more) ...
Schematically, the whole stack can be represented by a picture:
Consider the stack “from top to bottom”.
That is, from the moment of data transfer to the drive to the recording of the pits themselves.
It should be said that not the entire area of the disk is used to write / store / read information.
Compact disc is divided into zones ( areas ):
- Center Hole (Center Hole) - this is the “hole” with a diameter of 15 mm (± 0.1 mm), for which the disc itself is attached.
- First transition area - “ring”, between 15 and 20 mm from the center of the disk.
- Clamping area - as the name implies, this area is necessary so that the disk does not “skip” when reading / writing. (26-33 mm)
- Second transition area - The “second ring", between 33 and 44 mm from the center of the disk.
- Information area ( information area ) - it is "useful information" part of the CD. Located at a distance of 44 mm to 118 mm from the center.
- Rim area (rim) is the last area. It is a ring from 118 mm to 120 mm from the center.
It is the information zone that is of interest to us, so we will talk about it in more detail. Information area is divided into the following "subzones":
- inner buffer zone ( input area )
- user data zone ( program zone )
- outer buffer zone ( output area )
All three translations are made by GOST ... So I will not translate from myself. What does the “program" have to do with this when it comes to data - I can’t put my mind to it. If someone on Habré can answer me why the user data zone was translated as a "program zone" I will be extremely grateful!
Phew ... I hope they finished the physical educational program. Of course, I missed many points related to widths and lengths; with coating method; beam wavelength; and other physical properties. However, firstly, this is not the purpose of this article; secondly, I myself have not yet figured out more than half of the information. Yes and no desire, to be honest ... My curiosity is purely "programmer" orientation;)
Information Tracks
The first phase is a breakdown into “information tracks”. There are already two recording options, in digital form ( Digital Data Tracks, DDT ) and audio data ( Audio Tracks ).
In the future, we will consider only digital data. All of the following information is correct only for DDT .
Sector
Data is divided into 8 bits (one byte) and grouped into sectors .
It is funny that the number of sectors in the disk is not defined by the standard. It depends on how much data “get” written to disk ...
It seems a little ridiculous. Nevertheless, this absurdity was used by various companies to detect counterfeit disks in their era. (Since this is “sidestepping,” I wrote in more detail under the spoiler a bit lower at the end of the chapter.)
Of course, the track length is more or less fixed and in general QoS can be guaranteed.
There are three ways to (mode) write data in sectors:
- Sector Mode (00) is an empty sector filled with data, consisting of 0x00 bytes.
- Sector Mode (01) - use of EDC , PQ and CIRC coding. (more about this below)
- Sector Mode (02) - lack of P and Q encoding, only CIRC encoding
Here are the pictures:
Method for determining counterfeit disk
The method, like all ingenious, is simple.
Record a license key .
The procedure for checking the license key .
Record a license key .
- You must first write all the necessary information to disk.
- Then we select 2 sectors. For example 103123 and 120234 sectors. We denote these sectors both A and B .
- We select two bytes: one byte on each sector. For example, 4th byte of the first sector and 8th byte of the second
- Then you should calculate the angle between these bytes on the sector. How to do it? Suppose you have access to the low-level driver to read and you know the time of oborota.Togda should calculate the time spent on reading between the A and B . Dividing this time by the time of one revolution, you can calculate the angle with a certain error.
- The conception of a corner, rounded to a certain sign, is fed to the input of the hash function . A hash value takes several characters, for example, the last 3 characters.
- These three characters are written to the license key.
The procedure for checking the license key .
- The user is asked to enter a license key.
- Calculate the angle between the selected bytes A and B sectors
- We calculate the checked list of angles. For example, we calculated an angle of 33.343 °. Suppose rounding occurs to the nearest degree. We round and get 33 °. Assume that the error is ± 2 °. List of angles: [31 °, 32 °, 33 °, 34 °, 35 °].
- For each corner from the list, calculate the hash. We take some characters from a hash. For example, the last 3 characters.
If at least one hash from the list matches the hash from the license key, then we conclude that the disk is licensed. Otherwise, the counterfeit disk
Error Correction (Sector Mode 01 only)
Useful information ( User Mode ) consists of 2048 bytes in mode 01 ; or 2336 bytes in 02 mode .
Which mode to choose? It all depends on what reliability requirement you require.
Sector Mode 01 is more reliable, as it uses additional EDC verification and PQ encoding.
Edc
Error Detection Code (EDC) , as the name implies, is intended only to detect, but not to correct errors.
Here is its polynomial: P (x) = (x 16 + x 15 + x 2 +1) (x 16 + x 2 + x + 1)
Intermediate
The eight bytes of the Intermediate field are filled with zero bytes ( 0x00 ).
I don’t know why they were left ... Maybe “in reserve” (they like it in IT standards),
P and Q Encoding (RSPC)
Reed-Solomon Product-like Code (RSPC) , it is also P + Q encoding is used from 12 to 2075 bytes of data in mode 01 . I will omit the details; you can read them in Annex A of the ECMA-130 standard .
Bytes 12 through 2075 and verification bytes 2 076 through 2,351 are 2340 bytes of data. This data is divided into two blocks of 1170 bytes each. The division occurs as in school physical education lessons. " Pay off on the perrrvy-vtorrrroy! " That is, to odd and even bytes.
Next comes coding with external and internal codes. External is called P-coding , internal Q-coding .
Picture with P and Q encoding
For a better understanding: picture only with Q encoding
The most difficult ECMA-130 stack to understand is passed. Now it will be much easier.
Scrambling
We pass to scrambling. This is how one scrambling sector looks like:
Each such sector is called the Scrambled Sector .
What is scrambling, and why is it needed?
On the meaning of scrambling, briefly and succinctly in a comment on one of my posts wrote snapdragon
A scrambler is needed to make the signal spectrum uniform. Otherwise, with uniform data (for example, many repeating units or zeros), the signal energy will be concentrated in a narrow range.
F 1 , F 2 and F 3 frames
Each Scrambled Sector is divided into frames, 24 bytes long each.
These frames are called: F 1 frame .
Each Scrambled Sector we have consists of 2352 bytes.
Accordingly, each sector is divided into 98 frames.
CIRC Encoding (F 2 frame)
Cross Interleaved Reed-Solomon (CIRC) coding is performed for each F 1 frame .
This is error correcting code with an input word length of 24 bytes and an output word length of 32 bytes.
Moreover, unlike EDC and RSPC encoding, CIRC encoding is used for all Sector Mode .
The resulting sequence of 32 bytes is called F 2 frames .
Control byte
At the beginning of each F 2 frame , one check byte is added and an F 3 frame with a length of 33 bytes is obtained (32 + 1 = 33).
8-to-14 encoding
At this point, data bytes (8 bits) are converted to 14 bits of data. The conversion is carried out according to the table.
I will not give the whole table, you can find it in Annex D of the ECMA-130 standard .
... ...
00010000 10000000100000
00010001 10000010000000
00010010 10010010000000
00010011 00100000100000
00010100 01000010000000
00010101 00000010000000
00010110 00010010000000
... ...
Why 8-to-14 coding is necessary in the standard is not indicated. (The standard is not required to answer the questions WHY, the standard should contain answers to the questions in MAY) ...
I have one hypothesis. The fact is that the real world is not so "perfect" as programmers see it. For example, a drawn point is a small "blot", and a drawn line always has an area; otherwise, our eyes would not see a point and a line ... For this reason, I will venture to make a number of assumptions. I emphasize that I have never worked professionally with the manufacture of CDs. These are just assumptions. (Discussion in the comments is categorically welcome!).
Hypothesis.
- Pete is not “ideally” burned out on the surface of the disc, so some space is needed next to the pet, because a scorched pit may “pop in” over this space.
- Most likely, there are certain problems with the synchronization of the head itself. Too many consecutive zeros is bad.
- Perhaps a large number of units is an additional "load" on the read head. Therefore, their reduction will significantly increase the life of the CD drive. On average, we have 4 units for 8 bits of data. In 8-to-14 encoding , we have 1 or 2 units in the codeword . That is two times less.
Count redundancy
Let's see how redundant the CD protocol is:
- Depending on Sector Mode:
- Sector Mode 01 (PQ encoding) - At the input, a block of 2048 bytes, at the output 2352 . Hence redundancy is: 2352 / 2048 = 1.148
- Sector Mode 02 (without coding PQ) - 2352 / was 2336 = 1.007
- Scrambling is a trifle, but for the order we take into account: (12 + 2340) / 2340 = 1.005
- F 1 -F 2 -F 3 frames - 33 / 24 = 1.375
- 8-to-14 coding - 14 / 8 = 1.750
Multiplying everything, we get: 1.148 ⋅ 1.005 ⋅ 1.375 ⋅ 1.750 = 2.776 . Thus, as a result, 2.776 times more information is recorded on the disk itself than “useful information”.
For example, when the amount of “useful information” is 700 MB, 1943 MB of data is actually written to the disk .
Sector Mode 02 does not use PQ encoding. For this mode, the redundancy is: 1.007 ⋅ 1.005 ⋅ 1.375 ⋅ 1.750 = 2.435 .
Bonus: SCSI Multimedia Commands
There is a standard SCSI Multimedia Commands . It describes the raw data reading commands. The READ CD and WRITE CD commands read 2352 bytes of data from the entire sector. However, I did not find commands for reading F-fraim ... In principle, if you write redundant information that is not afraid of partial losses (for example, video, telematics),
you can do without F 1 -F 2 -F 3 frames by increasing the “payload” in 1.375 times.
There are also a number of unused areas in the CD (for example, the same Intermidiate ), which can also be used. For example, for the tasks of steganography.
Unfortunately, I did not find OpenSource code that implements these features ...
If there are specialists on this subject on the hub, I will be glad to get a link (plus a karma from me).