PyDERASN: as I wrote the ASN.1 library with slots and blobs
ASN.1 is a standard (ISO, ITU-T, GOST) of a language describing structured information, as well as encoding rules for this information. For me as a programmer, this is just another format for serializing and presenting data, along with JSON, XML, XDR and others. It is extremely common in our ordinary life, and many people come across it: in cellular, telephone, VoIP communications (UMTS, LTE, WiMAX, SS7, H.323), in network protocols (LDAP, SNMP, Kerberos), in everything Regarding cryptography (X.509, CMS, PKCS standards), in bank cards and biometric passports, and much more.
This article discusses PyDERASN : Python ASN.1 library actively used in projects related to cryptography in Atlas .
In fact, it is not worth recommending ASN.1 for cryptographic tasks: ASN.1 and its codecs are complex. This means that the code will not be simple, but it is always an extra attack vector. Just look at the list of vulnerabilities in ASN.1 libraries. Bruce Schneier, in his Cryptography engineering, also does not recommend using this standard because of its complexity: "The best-known TLV encoding is ASN.1, but it is incredibly complex and we shy away from it." But, unfortunately, today we have public key infrastructures that actively use X.509 certificates , CRL, OCSP, TSP, CMP, CMC , CMS messages , and a ton of PKCS standards. Therefore, you have to be able to work with ASN.1 if you are doing something related to cryptography.
ASN.1 can be encoded in a variety of ways / codecs:
and a number of others. But in cryptographic tasks in practice, two are used: BER and DER. Even in signed XML documents ( XMLDSig , XAdES ), there will still be Base64-encoded ASN.1 DER objects, as well as in Let's Encrypt's JSON-based ACME protocol . You can better understand all these codecs and BER / CER / DER coding principles in articles and books: ASN.1 in simple words , ASN.1 - Communication between heterogeneous systems by Olivier Dubuisson , ASN.1 Complete by Prof John Larmouth .
BER is a binary byte-oriented (for example, PER, popular in mobile communications - bit-oriented) TLV format. Each element is encoded as: tag ( Tag) identifying the type of encoded element (integer, string, date, etc.), the length ( L ength) of the content and the content itself ( V alue). BER optionally allows you to not specify a length value by setting a special indefinite length value and ending with an End-Of-Octets message label. In addition to length coding, BER has a lot of variability in the method of encoding data types, such as:
For all of the above, it is not always possible to encode data so that it is identical to the original form. Therefore, a subset of the rules was invented: DER is a strict regulation of only one valid coding method, which is critical for cryptographic tasks, where, for example, changing one bit will invalidate the signature or checksum. DER has a significant drawback: the lengths of all elements must be known in advance during encoding, which does not allow stream serialization of data. CER codec is free from this drawback, similarly guaranteeing an unambiguous presentation of the data. Unfortunately (or fortunately, we don’t have even more complex decoders?), It did not become popular. Therefore, in practice, we encounter a “mixed” use of BER and DER encoded data. Since both CER and DER are a subset of BER,
At work, we write many Python programs related to cryptography. And a few years ago there was practically no choice of free libraries: either these are very low-level libraries that allow you to simply encode / decode, for example, an integer and a structure header, or this is pyasn1. We lived on it for several years and at first were very pleased, since it allows you to work with ASN.1 structures as high-level objects: for example, a decoded X.509 certificate object allows you to access its fields through the dictionary interface: cert ["tbsCertificate"] ["SerialNumber"] will show us the serial number of this certificate. Similarly, you can "collect" complex objects by working with them as with lists, dictionaries, and then just call the pyasn1.codec.der.encoder.encode function and get a serialized representation of the document.
However, weaknesses, problems and limitations were revealed. There were and, unfortunately, still errors in pyasn1: at the time of writing, in pyasn1, one of the basic types, GeneralizedTime, is incorrectly decoded and encoded.
In our projects, to save space, we often store only the path to the file, the offset and length in bytes of the object we want to refer to. For example, an arbitrary signed file will most likely be located in the CMS SignedData ASN.1 structure:
and we can get the original signed file at an offset of 65 bytes, 751 bytes long. pyasn1 does not store this information in its decoded objects. The so-called TLVSeeker was written - a small library that allows you to decode the tags and lengths of objects, in the interface of which we ordered "go to the next tag", "go inside the tag" (go inside the SEQUENCE of the object), "go to the next tag", "tell your offset and the length of the object where we are. " It was a “manual” walk on ASN.1 DER-serialized data. But it wasn’t possible to work with BER-serialized data like this, since, for example, the byte string OCTET STRING could be encoded as several chunk-s.
Another drawback for our pyasn1 tasks is the inability to understand from decoded objects whether a given field was present in SEQUENCE or not. For example, if the structure contains the Field SEQUENCE OF Smth OPTIONAL field, then it could be completely absent in the received data (OPTIONAL), but could be present, but at the same time be of zero length (empty list). In general, this could not be clarified. And this is necessary for a rigorous check of the validity of the received data. Imagine that some certification authority would issue a certificate with "not entirely" valid data from the point of view of ASN.1 schemes! For example, the certification authority TÜRKTRUST Elektronik Sertifika Hizmet Sağlayıcısı in its root certificate went beyond the permissible RFC 5280boundaries of the length of the subject component - it cannot be honestly decoded according to the scheme DER codec requires that a field whose value is DEFAULT be not encoded during transmission - such documents are encountered in life, and the first version of PyDERASN even consciously allowed such invalid (from the DER point of view) behavior for the sake of backward compatibility.
Another limitation is the inability to easily find out in which form (BER / DER) one or another object was encoded in the structure. For example, the CMS standard says that the message is BER-encoded, but the signedAttrs field, over which the cryptographic signature is formed, should be in DER. If we decode with the DER, then we will fall on the processing of the CMS itself, if we decode with the BER, we will not know in what form signedAttrs was. As a result, the TLVSeeker (which has no analogue in pyasn1) will have to search for the location of each of the signedAttrs fields, and decode it DER separately from the serialized view.
The possibility of automatic processing of DEFINED BY fields, which are very common, was very desirable for us. After decoding the ASN.1 structure, we may have many ANY fields left, which should be processed further according to the scheme selected on the basis of the OBJECT IDENTIFIER specified in the structure field. In Python code, this means writing an if and then calling the decoder for the ANY field.
In Atlas, we regularly, having found any problems or modifying the used free programs, send patches to the top. In pyasn1, we sent improvements several times, but the pyasn1 code is not the easiest to understand, and sometimes incompatible API changes occurred in it, which hit us on the hands. Plus, we are used to writing tests with generative testing, which was not the case in pyasn1.
One fine day, I decided that I had to endure this and it was time to try writing my own library with __slot __s, offset s and perfectly displayed blobs! Just creating an ASN.1 codec would not be enough - you need to transfer all of our dependent projects to it, and this is hundreds of thousands of lines of code in which there is a lot of work with ASN.1-structures. That is one of the requirements for it: ease of translation of the current pyasn1 code. Having spent my entire vacation, I wrote this library, transferred all the projects to it. Since they have almost 100% coverage by tests, this also meant the library was fully operational.
PyDERASN, likewise, has almost 100% test coverage. Generative testing is used with the wonderful hypothesis library . Also conducted andfuzzing py-afl on 32 nuclear machines. Despite the fact that we have almost no Python2 code left, PyDERASN still maintains compatibility with it and because of this it has a single six dependency. In addition, it is tested against ASN.1: 2008 compliance test suite .
The principle of working with it is similar to pyasn1 - working with high-level Python objects. Description of ASN.1 circuits is similar.
However, PyDERASN has a semblance of strong typing. In pyasn1, if the field was of type CMSVersion (INTEGER), then it could be assigned an int or INTEGER. PyDERASN strictly requires that the assigned object be exactly CMSVersion. In addition to writing Python3 code, we use typing annotations , so our functions will not have incomprehensible arguments like def func (serial, contents), but def func (serial: CertificateSerialNumber, contents: EncapsulatedContentInfo), and PyDERASN helps to keep track of such code.
At the same time, PyDERASN has extremely convenient concessions for this very typing. pyasn1 did not allow SubjectKeyIdentifier (). subtype (implicitTag = Tag (...)) to assign a SubjectKeyIdentifier () object (without the required IMPLICIT TAG) and often had to copy and recreate objects only because of changed IMPLICIT / EXPLICIT tags. PyDERASN strictly observes only the basic type - it will automatically substitute tags from an existing ASN.1 structure. This greatly simplifies application code.
If an error occurs during decoding, then in pyasn1 it is not easy to understand exactly where it occurred. For example, in the already mentioned Turkish certificate we get this error: UTF8String (tbsCertificate: issuer: rdnSequence: 3: 0: value: DEFINED BY 2.5.4.10:utf8String) (at 138) unsatisfied bounds: 1 ⇐ 77 ⇐ 64 When writing ASN .1 structures people can make mistakes, and it helps easier to debug applications or find out problems of encoded documents of the opposite side.
The first version of PyDERASN did not support BER encoding. It appeared much later and the UTCTime / GeneralizedTime processing with time zones is still not supported. This will come in the future, because the project is written mainly in free time from work.
Also in the first version there was no work with DEFINED BY fields. A few months later this opportunity appeared and began to be actively used, significantly reducing the application code - in one decoding operation it was possible to get the entire structure disassembled to the very depth. To do this, in the scheme are set which fields that “determine”. For example, a description of a CMS schema:
says that if contentType contains an OID with id_signedData, then the content field (located in the same SEQUENCE) needs to be decoded using the SignedData scheme. Why are there so many brackets? A field can “define” several fields at the same time, as is the case in EnvelopedData structures. Defined fields are identified by the so-called decode path - it sets the exact location of any element in all structures.
It is not always desirable or not always possible to immediately introduce these defines into the circuit. There may be application-specific cases where OIDs and structures are known only in a third-party project. PyDERASN provides the ability to specify these defines right at the time of decoding the structure:
Here we say that in CMS SignedData for all attached certificates, decode all their extensions (AuthorityKeyIdentifier, BasicConstraints, SubjectSignTool, etc.). We indicate through the decode path which element to "substitute" defines, as if it were defined in the circuit.
Finally, PyDERASN has the ability to work from the command line to decode ASN.1 files and has rich pretty printing . You can decode an arbitrary ASN.1, or you can specify a clearly defined scheme and see something like this:
Displayed information: object offset, tag length, length length, content length, presence of EOC (end-of-octets), BER encoding flag, encoding indefinite-length flag, EXPLICIT tag length and offset (if any), object nesting depth in structures, IMPLICIT / EXPLICIT tag value, object name according to the scheme, its basic ASN.1 type, serial number inside SEQUENCE / SET OF, value CHOICE (if any), human-readable name INTEGER / ENUMERATED / BIT STRING according to the scheme, value of any basic type , DEFAULT / OPTIONAL flag from the circuit, a sign that the object was automatically decoded as DEFINED BY and after gm of OID-and it happened, chelovekochitaemy OID.
The pretty printing system is specially made so that it generates a sequence of PP objects that are already visualized by separate means. The screenshot shows renderer in plain colored text. There are renderers in JSON / HTML format, so that this can be seen with highlighting in the ASN.1 browser as in asn1js project.
This was not the goal, but PyDERASN was significantly faster than pyasn1. For example, decoding CRL files of megabyte sizes can take so long that you have to think about intermediate formats of data storage (fast) and change the architecture of applications. pyasn1 decodes CRL CACert.org on my laptop for more than 20 minutes, while PyDERASN in just 28 seconds! There is a project asn1cryptoaimed at quickly working with cryptographic structures: it decodes (completely, not lazy) the same CRL in 29 seconds, but it consumes almost twice as much RAM when running under Python3 (983 MiB versus 498 mi), and in 3.5 times under Python2 (1677 against 488), while pyasn1 consumes as much as 4.3 times more (2093 against 488).
asn1crypto, which I mentioned, we did not consider, because the project was just in its infancy, and we had not heard about it. Now they wouldn’t look in his direction either, since I immediately found out that he doesn’t take the same GeneralizedTime, and during serialization he silently removes fractions of a second. This is acceptable for working with X.509 certificates, but in general it will not work.
At the moment, PyDERASN is the most rigorous of the free Python / Go DER decoders I know. In the encoding / asn1 library of my favorite Go, there is no strict check on OBJECT IDENTIFIER and UTCTime / GeneralizedTime strings. Sometimes strictness can interfere (primarily because of backward compatibility with old applications that no one will fix), so in PyDERASN during decoding you can pass various settings weakening checks.
The project code tries to be as simple as possible. The entire library is a single file. The code is written with emphasis on ease of understanding, without unnecessary performance and DRY code optimizations. It does not, as I already said, support full-fledged BER decoding of UTCTime / GeneralizedTime strings, as well as REAL, RELATIVE OID, EXTERNAL, INSTANCE OF, EMBEDDED PDV, CHARACTER STRING data types. In all other cases, personally, I see no reason to use other libraries in Python.
Like all my projects, such as PyGOST , GoGOST , NNCP , GoVPN , PyDERASN is completely free software distributed under the terms of LGPLv3 + , and is available for free download. Examples of use arehere and in PyGOST tests .
Sergey Matveev , cipher bank , member of the Open Society Foundation Foundation , Python / Go-developer, chief specialist of FSUE “STC Atlas” .
This article discusses PyDERASN : Python ASN.1 library actively used in projects related to cryptography in Atlas .
In fact, it is not worth recommending ASN.1 for cryptographic tasks: ASN.1 and its codecs are complex. This means that the code will not be simple, but it is always an extra attack vector. Just look at the list of vulnerabilities in ASN.1 libraries. Bruce Schneier, in his Cryptography engineering, also does not recommend using this standard because of its complexity: "The best-known TLV encoding is ASN.1, but it is incredibly complex and we shy away from it." But, unfortunately, today we have public key infrastructures that actively use X.509 certificates , CRL, OCSP, TSP, CMP, CMC , CMS messages , and a ton of PKCS standards. Therefore, you have to be able to work with ASN.1 if you are doing something related to cryptography.
ASN.1 can be encoded in a variety of ways / codecs:
- BER (Basic Encoding Rules)
- CER (Canonical Encoding Rules)
- DER (Distinguished Encoding Rules)
- GSER (Generic String Encoding Rules)
- JER (JSON Encoding Rules)
- LWER (Light Weight Encoding Rules)
- OER (Octet Encoding Rules)
- PER (Packed Encoding Rules)
- SER (Signaling specific Encoding Rules)
- XER (XML Encoding Rules)
and a number of others. But in cryptographic tasks in practice, two are used: BER and DER. Even in signed XML documents ( XMLDSig , XAdES ), there will still be Base64-encoded ASN.1 DER objects, as well as in Let's Encrypt's JSON-based ACME protocol . You can better understand all these codecs and BER / CER / DER coding principles in articles and books: ASN.1 in simple words , ASN.1 - Communication between heterogeneous systems by Olivier Dubuisson , ASN.1 Complete by Prof John Larmouth .
BER is a binary byte-oriented (for example, PER, popular in mobile communications - bit-oriented) TLV format. Each element is encoded as: tag ( Tag) identifying the type of encoded element (integer, string, date, etc.), the length ( L ength) of the content and the content itself ( V alue). BER optionally allows you to not specify a length value by setting a special indefinite length value and ending with an End-Of-Octets message label. In addition to length coding, BER has a lot of variability in the method of encoding data types, such as:
- INTEGER, OBJECT IDENTIFIER, BIT STRING and the length of the element may not be normalized (not encoded in minimal form);
- BOOLEAN is true for any non-zero content;
- BIT STRING may contain "extra" zero bits;
- BIT STRING, OCTET STRING and all their derived string types, including date / time, can be divided into pieces (chunk) of variable length, the length of which during (de) encoding is not known in advance;
- UTCTime / GeneralizedTime can have different methods for setting the time zone offset and “extra” zero fractions of seconds;
- DEFAULT SEQUENCE values may or may not be encoded;
- Named values of the last bits in BIT STRING can be optionally encoded;
- SEQUENCE (OF) / SET (OF) can have an arbitrary order of elements.
For all of the above, it is not always possible to encode data so that it is identical to the original form. Therefore, a subset of the rules was invented: DER is a strict regulation of only one valid coding method, which is critical for cryptographic tasks, where, for example, changing one bit will invalidate the signature or checksum. DER has a significant drawback: the lengths of all elements must be known in advance during encoding, which does not allow stream serialization of data. CER codec is free from this drawback, similarly guaranteeing an unambiguous presentation of the data. Unfortunately (or fortunately, we don’t have even more complex decoders?), It did not become popular. Therefore, in practice, we encounter a “mixed” use of BER and DER encoded data. Since both CER and DER are a subset of BER,
Problems with pyasn1
At work, we write many Python programs related to cryptography. And a few years ago there was practically no choice of free libraries: either these are very low-level libraries that allow you to simply encode / decode, for example, an integer and a structure header, or this is pyasn1. We lived on it for several years and at first were very pleased, since it allows you to work with ASN.1 structures as high-level objects: for example, a decoded X.509 certificate object allows you to access its fields through the dictionary interface: cert ["tbsCertificate"] ["SerialNumber"] will show us the serial number of this certificate. Similarly, you can "collect" complex objects by working with them as with lists, dictionaries, and then just call the pyasn1.codec.der.encoder.encode function and get a serialized representation of the document.
However, weaknesses, problems and limitations were revealed. There were and, unfortunately, still errors in pyasn1: at the time of writing, in pyasn1, one of the basic types, GeneralizedTime, is incorrectly decoded and encoded.
In our projects, to save space, we often store only the path to the file, the offset and length in bytes of the object we want to refer to. For example, an arbitrary signed file will most likely be located in the CMS SignedData ASN.1 structure:
0 [1,3,1018] ContentInfo SEQUENCE
4 [1,1, 9] . contentType: ContentType OBJECT IDENTIFIER 1.2.840.113549.1.7.2 (id_signedData)
19-4 [0,0,1003] . content: [0] EXPLICIT [UNIV 16] ANY
19 [1,3, 999] . . DEFINED BY id_signedData: SignedData SEQUENCE
23 [1,1, 1] . . . version: CMSVersion INTEGER v3 (03)
26 [1,1, 19] . . . digestAlgorithms: DigestAlgorithmIdentifiers SET OF
[...]
47 [1,3, 769] . . . encapContentInfo: EncapsulatedContentInfo SEQUENCE
51 [1,1, 8] . . . . eContentType: ContentType OBJECT IDENTIFIER 1.3.6.1.5.5.7.12.2 (id_cct_PKIData)
65-4 [1,3, 751] . . . . eContent: [0] EXPLICIT OCTET STRING 751 bytes OPTIONAL
ТУТ СОДЕРЖИМОЕ ПОДПИСЫВАЕМОГО ФАЙЛА РАЗМЕРОМ 751 байт
820 [1,2, 199] . . . signerInfos: SignerInfos SET OF
823 [1,2, 196] . . . . 0: SignerInfo SEQUENCE
826 [1,1, 1] . . . . . version: CMSVersion INTEGER v3 (03)
829 [0,0, 22] . . . . . sid: SignerIdentifier CHOICE subjectKeyIdentifier
[...]
956 [1,1, 64] . . . . . signature: SignatureValue OCTET STRING 64 bytes
. . . . . . C1:B3:88:BA:F8:92:1C:E6:3E:41:9B:E0:D3:E9:AF:D8
. . . . . . 47:4A:8A:9D:94:5D:56:6B:F0:C1:20:38:D2:72:22:12
. . . . . . 9F:76:46:F6:51:5F:9A:8D:BF:D7:A6:9B:FD:C5:DA:D2
. . . . . . F3:6B:00:14:A4:9D:D7:B5:E1:A6:86:44:86:A7:E8:C9
and we can get the original signed file at an offset of 65 bytes, 751 bytes long. pyasn1 does not store this information in its decoded objects. The so-called TLVSeeker was written - a small library that allows you to decode the tags and lengths of objects, in the interface of which we ordered "go to the next tag", "go inside the tag" (go inside the SEQUENCE of the object), "go to the next tag", "tell your offset and the length of the object where we are. " It was a “manual” walk on ASN.1 DER-serialized data. But it wasn’t possible to work with BER-serialized data like this, since, for example, the byte string OCTET STRING could be encoded as several chunk-s.
Another drawback for our pyasn1 tasks is the inability to understand from decoded objects whether a given field was present in SEQUENCE or not. For example, if the structure contains the Field SEQUENCE OF Smth OPTIONAL field, then it could be completely absent in the received data (OPTIONAL), but could be present, but at the same time be of zero length (empty list). In general, this could not be clarified. And this is necessary for a rigorous check of the validity of the received data. Imagine that some certification authority would issue a certificate with "not entirely" valid data from the point of view of ASN.1 schemes! For example, the certification authority TÜRKTRUST Elektronik Sertifika Hizmet Sağlayıcısı in its root certificate went beyond the permissible RFC 5280boundaries of the length of the subject component - it cannot be honestly decoded according to the scheme DER codec requires that a field whose value is DEFAULT be not encoded during transmission - such documents are encountered in life, and the first version of PyDERASN even consciously allowed such invalid (from the DER point of view) behavior for the sake of backward compatibility.
Another limitation is the inability to easily find out in which form (BER / DER) one or another object was encoded in the structure. For example, the CMS standard says that the message is BER-encoded, but the signedAttrs field, over which the cryptographic signature is formed, should be in DER. If we decode with the DER, then we will fall on the processing of the CMS itself, if we decode with the BER, we will not know in what form signedAttrs was. As a result, the TLVSeeker (which has no analogue in pyasn1) will have to search for the location of each of the signedAttrs fields, and decode it DER separately from the serialized view.
The possibility of automatic processing of DEFINED BY fields, which are very common, was very desirable for us. After decoding the ASN.1 structure, we may have many ANY fields left, which should be processed further according to the scheme selected on the basis of the OBJECT IDENTIFIER specified in the structure field. In Python code, this means writing an if and then calling the decoder for the ANY field.
The advent of PyDERASN
In Atlas, we regularly, having found any problems or modifying the used free programs, send patches to the top. In pyasn1, we sent improvements several times, but the pyasn1 code is not the easiest to understand, and sometimes incompatible API changes occurred in it, which hit us on the hands. Plus, we are used to writing tests with generative testing, which was not the case in pyasn1.
One fine day, I decided that I had to endure this and it was time to try writing my own library with __slot __s, offset s and perfectly displayed blobs! Just creating an ASN.1 codec would not be enough - you need to transfer all of our dependent projects to it, and this is hundreds of thousands of lines of code in which there is a lot of work with ASN.1-structures. That is one of the requirements for it: ease of translation of the current pyasn1 code. Having spent my entire vacation, I wrote this library, transferred all the projects to it. Since they have almost 100% coverage by tests, this also meant the library was fully operational.
PyDERASN, likewise, has almost 100% test coverage. Generative testing is used with the wonderful hypothesis library . Also conducted andfuzzing py-afl on 32 nuclear machines. Despite the fact that we have almost no Python2 code left, PyDERASN still maintains compatibility with it and because of this it has a single six dependency. In addition, it is tested against ASN.1: 2008 compliance test suite .
The principle of working with it is similar to pyasn1 - working with high-level Python objects. Description of ASN.1 circuits is similar.
class TBSCertificate(Sequence):
schema = (
("version", Version(expl=tag_ctxc(0), default="v1")),
("serialNumber", CertificateSerialNumber()),
("signature", AlgorithmIdentifier()),
("issuer", Name()),
("validity", Validity()),
("subject", Name()),
("subjectPublicKeyInfo", SubjectPublicKeyInfo()),
("issuerUniqueID", UniqueIdentifier(impl=tag_ctxp(1), optional=True)),
("subjectUniqueID", UniqueIdentifier(impl=tag_ctxp(2), optional=True)),
("extensions", Extensions(expl=tag_ctxc(3), optional=True)),
)
However, PyDERASN has a semblance of strong typing. In pyasn1, if the field was of type CMSVersion (INTEGER), then it could be assigned an int or INTEGER. PyDERASN strictly requires that the assigned object be exactly CMSVersion. In addition to writing Python3 code, we use typing annotations , so our functions will not have incomprehensible arguments like def func (serial, contents), but def func (serial: CertificateSerialNumber, contents: EncapsulatedContentInfo), and PyDERASN helps to keep track of such code.
At the same time, PyDERASN has extremely convenient concessions for this very typing. pyasn1 did not allow SubjectKeyIdentifier (). subtype (implicitTag = Tag (...)) to assign a SubjectKeyIdentifier () object (without the required IMPLICIT TAG) and often had to copy and recreate objects only because of changed IMPLICIT / EXPLICIT tags. PyDERASN strictly observes only the basic type - it will automatically substitute tags from an existing ASN.1 structure. This greatly simplifies application code.
If an error occurs during decoding, then in pyasn1 it is not easy to understand exactly where it occurred. For example, in the already mentioned Turkish certificate we get this error: UTF8String (tbsCertificate: issuer: rdnSequence: 3: 0: value: DEFINED BY 2.5.4.10:utf8String) (at 138) unsatisfied bounds: 1 ⇐ 77 ⇐ 64 When writing ASN .1 structures people can make mistakes, and it helps easier to debug applications or find out problems of encoded documents of the opposite side.
The first version of PyDERASN did not support BER encoding. It appeared much later and the UTCTime / GeneralizedTime processing with time zones is still not supported. This will come in the future, because the project is written mainly in free time from work.
Also in the first version there was no work with DEFINED BY fields. A few months later this opportunity appeared and began to be actively used, significantly reducing the application code - in one decoding operation it was possible to get the entire structure disassembled to the very depth. To do this, in the scheme are set which fields that “determine”. For example, a description of a CMS schema:
class ContentInfo(Sequence):
schema = (
("contentType", ContentType(defines=((("content",), {
id_authenticatedData: AuthenticatedData(),
id_digestedData: DigestedData(),
id_encryptedData: EncryptedData(),
id_envelopedData: EnvelopedData(),
id_signedData: SignedData(),
}),))),
("content", Any(expl=tag_ctxc(0))),
)
says that if contentType contains an OID with id_signedData, then the content field (located in the same SEQUENCE) needs to be decoded using the SignedData scheme. Why are there so many brackets? A field can “define” several fields at the same time, as is the case in EnvelopedData structures. Defined fields are identified by the so-called decode path - it sets the exact location of any element in all structures.
It is not always desirable or not always possible to immediately introduce these defines into the circuit. There may be application-specific cases where OIDs and structures are known only in a third-party project. PyDERASN provides the ability to specify these defines right at the time of decoding the structure:
ContentInfo().decode(data, ctx={"defines_by_path": ((
(
"content", DecodePathDefBy(id_signedData),
"certificates", any, "certificate", "tbsCertificate",
"extensions", any, "extnID",
),
((("extnValue",), {
id_ce_authorityKeyIdentifier: AuthorityKeyIdentifier(),
id_ce_basicConstraints: BasicConstraints(),
[...]
id_ru_subjectSignTool: SubjectSignTool(),
}),),
),)})
Here we say that in CMS SignedData for all attached certificates, decode all their extensions (AuthorityKeyIdentifier, BasicConstraints, SubjectSignTool, etc.). We indicate through the decode path which element to "substitute" defines, as if it were defined in the circuit.
Finally, PyDERASN has the ability to work from the command line to decode ASN.1 files and has rich pretty printing . You can decode an arbitrary ASN.1, or you can specify a clearly defined scheme and see something like this:
Displayed information: object offset, tag length, length length, content length, presence of EOC (end-of-octets), BER encoding flag, encoding indefinite-length flag, EXPLICIT tag length and offset (if any), object nesting depth in structures, IMPLICIT / EXPLICIT tag value, object name according to the scheme, its basic ASN.1 type, serial number inside SEQUENCE / SET OF, value CHOICE (if any), human-readable name INTEGER / ENUMERATED / BIT STRING according to the scheme, value of any basic type , DEFAULT / OPTIONAL flag from the circuit, a sign that the object was automatically decoded as DEFINED BY and after gm of OID-and it happened, chelovekochitaemy OID.
The pretty printing system is specially made so that it generates a sequence of PP objects that are already visualized by separate means. The screenshot shows renderer in plain colored text. There are renderers in JSON / HTML format, so that this can be seen with highlighting in the ASN.1 browser as in asn1js project.
Other libraries
This was not the goal, but PyDERASN was significantly faster than pyasn1. For example, decoding CRL files of megabyte sizes can take so long that you have to think about intermediate formats of data storage (fast) and change the architecture of applications. pyasn1 decodes CRL CACert.org on my laptop for more than 20 minutes, while PyDERASN in just 28 seconds! There is a project asn1cryptoaimed at quickly working with cryptographic structures: it decodes (completely, not lazy) the same CRL in 29 seconds, but it consumes almost twice as much RAM when running under Python3 (983 MiB versus 498 mi), and in 3.5 times under Python2 (1677 against 488), while pyasn1 consumes as much as 4.3 times more (2093 against 488).
asn1crypto, which I mentioned, we did not consider, because the project was just in its infancy, and we had not heard about it. Now they wouldn’t look in his direction either, since I immediately found out that he doesn’t take the same GeneralizedTime, and during serialization he silently removes fractions of a second. This is acceptable for working with X.509 certificates, but in general it will not work.
At the moment, PyDERASN is the most rigorous of the free Python / Go DER decoders I know. In the encoding / asn1 library of my favorite Go, there is no strict check on OBJECT IDENTIFIER and UTCTime / GeneralizedTime strings. Sometimes strictness can interfere (primarily because of backward compatibility with old applications that no one will fix), so in PyDERASN during decoding you can pass various settings weakening checks.
The project code tries to be as simple as possible. The entire library is a single file. The code is written with emphasis on ease of understanding, without unnecessary performance and DRY code optimizations. It does not, as I already said, support full-fledged BER decoding of UTCTime / GeneralizedTime strings, as well as REAL, RELATIVE OID, EXTERNAL, INSTANCE OF, EMBEDDED PDV, CHARACTER STRING data types. In all other cases, personally, I see no reason to use other libraries in Python.
Like all my projects, such as PyGOST , GoGOST , NNCP , GoVPN , PyDERASN is completely free software distributed under the terms of LGPLv3 + , and is available for free download. Examples of use arehere and in PyGOST tests .
Sergey Matveev , cipher bank , member of the Open Society Foundation Foundation , Python / Go-developer, chief specialist of FSUE “STC Atlas” .
