JSON Schema and its use for validating JSON documents in C ++
This article describes the JSON Schema standard and its use for checking compliance with a given format in C ++ using the valijson library .
To begin with, let us recall what led to the widespread crowding out of XML by JSON and that this was bad. XML was originally created as a metalanguage for document markup , allowing the use of a unified code for the parser and document validator. Being the first standard of this kind, which dates back to the period of rapid implementation of digital corporate information systems, XML served as the basis for countless standards for data serialization and interaction protocols, i.e. storage and transmission of structured data. Whereas it was created primarily for marking up documents.
Being developed by committees , the XML standard has been supplemented by many extensions that allow, in particular, to avoid name conflicts and perform complex queries in XML documents. And, most importantly, since the resulting piling up of tags turned out to be completely unreadable by anyone, the XML Schema standard was developed and widely implemented , which allows the exact XML to strictly describe the valid content of each document for subsequent automatic verification.
Meanwhile, more and more developers, under the influence of nascent interactive web-technologies, began to get acquainted with the JavaScript language, and they began to realize that it was not necessary to study many hundreds of pages of XML specifications to represent structured objects in text form. And when Douglas Crockford proposed standardizing a subset of JavaScript for serializing objects (but not markup documents!) Regardless of language, the idea was supported by the community. Currently, JSON is one of two (along with XML) languages supported by all any popular programming technologies. The same YAML, designed to make JSON more convenient and human-readable, because of its complexity (i.e. breadth of possibilities) is not so widespread (my company had problems working with YAML from MATLAB not so long ago,
So, having massively started using JSON to represent data, developers are faced with the need to manually check the contents of documents, each time in each language, reinventing the validation logic. People familiar with XML Schema could not but enrage. And gradually, a similar JSON Schema standard was formed and lives at http://json-schema.org/ .
Consider an example of a simple, but exponential, scheme that defines a dictionary of 2D or 3D geometric points in the space (-1, 1) x (-1, 1) x (-1, 1) with keys consisting of numbers:
If you forgive Crockford of annoying quotes, from this document it should be clear that we agree to deal with an object (dictionary) whose keys must consist of numbers (see the regular expression), the values of which must have fields x, y, value, and can have field z, and value is a non-negative number, and x, y, z all have some identical type point_coord, corresponding to a number from -1 to +1. Even assuming that the JSON Schema does not provide other features (which is far from the truth), this should be enough for many use cases.
But this is the case if a validator is implemented for your language / platform. In the case of XML, such a question could hardly arise.
On the http://json-schema.org/ website you can find a list of softwarefor validation. And in this place the immaturity of JSON-Schema (and its website) makes itself felt. For C ++, one (seemingly interesting) libvariant library is specified , which is engaged in validation only part-time and also released under the malicious LGPL license (goodbye, iOS). For C, we also have one option , and also under LGPL.
However, an acceptable solution exists and is called valijson. This library has everything we need (circuit validation and a BSD license), and even more - independence from the JSON parser. Valijson allows you to use any json parser through an adapter (bundled adapters for jsoncpp, json11, rapidjson, picojson and boost :: property_tree), thus not requiring you to switch to a new json library (or drag one more one after yourself). Plus, it consists only of header files (header only) and does not require compilation. The obvious minus is only one, and then not for everyone - the dependence on boost. Although there is hope for a deliverance even from this shortcoming.
Let’s take a look at the example of a document compiling a JSON scheme and validating this document.
Suppose we have a table of some striped objects for which a specific striped coloring is given (in the form of a sequence of 0 and 1 corresponding to black and white).
Here we have a dictionary with numeric keys, to which the suffix “inv” can be assigned (for inverted barcodes). All values in the dictionary are objects and must have the fields “width”, “stripe_length” (strictly positive numbers) and “code” (a string of zeros and units of length 12).
We begin to draw up a diagram by specifying restrictions on the format of the names of top-level fields:
Here we used the patternProperties construct, which allows / specifies values whose keys satisfy the regular expression. We also indicated (additionalProperties = false) that unspecified keys are not allowed. Using additionalProperties, you can not only allow or prohibit fields that are not specified explicitly, but also impose restrictions on their values by specifying a type specifier as the value, for example, like this:
Next, we describe the type of value of each object in the dictionary:
Here we explicitly list the allowed fields (properties), requiring their presence (required), without prohibiting (by default) any additional properties. Numeric properties are strictly positive, and the string code must match the regular expression.
In principle, it remains only to insert a description of the type of an individual object in the above table layout. But before doing this, we note that we have duplicated the specification of the “width” and “stripe_length” fields. In the real code from which the example is taken, there are even more such fields, so it would be useful to define this type once, and then refer to it from everywhere. For this, there is a link mechanism ($ ref). Pay attention to the definitions section in the final diagram:
Save it to a file and proceed with writing the validator.
We use jsoncpp as the json parser . We have the usual function of loading a json document from a file:
The minimal validator function that tells us the location of all validation errors looks something like this:
Note that in this example, jsoncpp connects as
Now we can upload and validate documents:
That's it, we learned how to validate json documents. But note that now we have to think about where to store the circuits! Indeed, if a document changes every time and is obtained, for example, from a web request or from a command line argument, then the scheme is unchanged and must be delivered with the application. And for small programs without a developed mechanism for loading static resources, the need to introduce one represents a significant barrier to the implementation of validation through schemes. It would be great to compile the scheme with the program, because changing the scheme in any case will require changing the code that processes the document.
This is possible and even quite convenient if we have C ++ 11 at our disposal. The solution is primitive, but it works fine: we just define a string constant with our schema. And in order not to care about the quotes inside the string, we use raw string literal :
Thus, we have a convenient cross-platform cross-language mechanism for validating json documents, the use of which in C ++ does not require linking external libraries with inconvenient licenses, or fussing with paths to static resources. This thing can really save a lot of energy, and, importantly, help permanently kill XML as a format for representing objects, because it is inconvenient for both people and machines.
A bit of history
To begin with, let us recall what led to the widespread crowding out of XML by JSON and that this was bad. XML was originally created as a metalanguage for document markup , allowing the use of a unified code for the parser and document validator. Being the first standard of this kind, which dates back to the period of rapid implementation of digital corporate information systems, XML served as the basis for countless standards for data serialization and interaction protocols, i.e. storage and transmission of structured data. Whereas it was created primarily for marking up documents.
Being developed by committees , the XML standard has been supplemented by many extensions that allow, in particular, to avoid name conflicts and perform complex queries in XML documents. And, most importantly, since the resulting piling up of tags turned out to be completely unreadable by anyone, the XML Schema standard was developed and widely implemented , which allows the exact XML to strictly describe the valid content of each document for subsequent automatic verification.
Meanwhile, more and more developers, under the influence of nascent interactive web-technologies, began to get acquainted with the JavaScript language, and they began to realize that it was not necessary to study many hundreds of pages of XML specifications to represent structured objects in text form. And when Douglas Crockford proposed standardizing a subset of JavaScript for serializing objects (but not markup documents!) Regardless of language, the idea was supported by the community. Currently, JSON is one of two (along with XML) languages supported by all any popular programming technologies. The same YAML, designed to make JSON more convenient and human-readable, because of its complexity (i.e. breadth of possibilities) is not so widespread (my company had problems working with YAML from MATLAB not so long ago,
So, having massively started using JSON to represent data, developers are faced with the need to manually check the contents of documents, each time in each language, reinventing the validation logic. People familiar with XML Schema could not but enrage. And gradually, a similar JSON Schema standard was formed and lives at http://json-schema.org/ .
JSON Schema
Consider an example of a simple, but exponential, scheme that defines a dictionary of 2D or 3D geometric points in the space (-1, 1) x (-1, 1) x (-1, 1) with keys consisting of numbers:
{
"type": "object",
"patternProperties": {
"^[0-9]+$": {
"type": "object",
"properties": {
"value": {
"type": "number",
"minimum": 0
}
"x": { "$ref": "#/definitions/point_coord" },
"y": { "$ref": "#/definitions/point_coord" },
"z": { "$ref": "#/definitions/point_coord" }
},
"required": ["value", "x", "y"]
}
}
"additionalProperties": false,
"definitions": {
"point_coord": {
"type": "number",
"minimum": -1,
"maximum": 1
}
}
}
If you forgive Crockford of annoying quotes, from this document it should be clear that we agree to deal with an object (dictionary) whose keys must consist of numbers (see the regular expression), the values of which must have fields x, y, value, and can have field z, and value is a non-negative number, and x, y, z all have some identical type point_coord, corresponding to a number from -1 to +1. Even assuming that the JSON Schema does not provide other features (which is far from the truth), this should be enough for many use cases.
But this is the case if a validator is implemented for your language / platform. In the case of XML, such a question could hardly arise.
On the http://json-schema.org/ website you can find a list of softwarefor validation. And in this place the immaturity of JSON-Schema (and its website) makes itself felt. For C ++, one (seemingly interesting) libvariant library is specified , which is engaged in validation only part-time and also released under the malicious LGPL license (goodbye, iOS). For C, we also have one option , and also under LGPL.
However, an acceptable solution exists and is called valijson. This library has everything we need (circuit validation and a BSD license), and even more - independence from the JSON parser. Valijson allows you to use any json parser through an adapter (bundled adapters for jsoncpp, json11, rapidjson, picojson and boost :: property_tree), thus not requiring you to switch to a new json library (or drag one more one after yourself). Plus, it consists only of header files (header only) and does not require compilation. The obvious minus is only one, and then not for everyone - the dependence on boost. Although there is hope for a deliverance even from this shortcoming.
Let’s take a look at the example of a document compiling a JSON scheme and validating this document.
Schematic Example
Suppose we have a table of some striped objects for which a specific striped coloring is given (in the form of a sequence of 0 and 1 corresponding to black and white).
{
"0inv": {
"width": 0.11,
"stripe_length": 0.15,
"code": "101101101110"
},
"0": {
"width": 0.05,
"stripe_length": 0.11,
"code": "010010010001"
},
"3": {
"width": 0.05,
"stripe_length": 0.11,
"code": "010010110001"
},
...
}
Here we have a dictionary with numeric keys, to which the suffix “inv” can be assigned (for inverted barcodes). All values in the dictionary are objects and must have the fields “width”, “stripe_length” (strictly positive numbers) and “code” (a string of zeros and units of length 12).
We begin to draw up a diagram by specifying restrictions on the format of the names of top-level fields:
{
"comment": "Schema for the striped object specification file",
"type": "object",
"patternProperties": {
"^[0-9]+(inv)?$": { }
},
"additionalProperties": false
}
Here we used the patternProperties construct, which allows / specifies values whose keys satisfy the regular expression. We also indicated (additionalProperties = false) that unspecified keys are not allowed. Using additionalProperties, you can not only allow or prohibit fields that are not specified explicitly, but also impose restrictions on their values by specifying a type specifier as the value, for example, like this:
{
"additionalProperties": {
"type": "string",
"pattern": "^Comment: .*$"
}
}
Next, we describe the type of value of each object in the dictionary:
{
"type": "object",
"properties": {
"width": {
"type": "number",
"minimum": 0,
"exclusiveMinimum": true
},
"stripe_length": {
"type": "number",
"minimum": 0,
"exclusiveMinimum": true
},
"code": {
"type": "string",
"pattern": "^[01]{12}$"
}
},
"required": ["width", "stripe_length", "code"]
}
Here we explicitly list the allowed fields (properties), requiring their presence (required), without prohibiting (by default) any additional properties. Numeric properties are strictly positive, and the string code must match the regular expression.
In principle, it remains only to insert a description of the type of an individual object in the above table layout. But before doing this, we note that we have duplicated the specification of the “width” and “stripe_length” fields. In the real code from which the example is taken, there are even more such fields, so it would be useful to define this type once, and then refer to it from everywhere. For this, there is a link mechanism ($ ref). Pay attention to the definitions section in the final diagram:
{
"comment": "Schema for the striped object specification file",
"type": "object",
"patternProperties": {
"^[0-9]+(inv)?$": {
"type": "object",
"properties": {
"width": { "$ref": "#/definitions/positive_number" },
"stripe_length": { "$ref": "#/definitions/positive_number" },
"code": {
"type": "string",
"pattern": "^[01]{12}$"
}
},
"required": ["width", "stripe_length", "code"]
}
},
"additionalProperties": false,
"definitions": {
"positive_number": {
"type": "number",
"minimum": 0,
"exclusiveMinimum": true
}
}
}
Save it to a file and proceed with writing the validator.
Valijson application
We use jsoncpp as the json parser . We have the usual function of loading a json document from a file:
#include<json-cpp/json.h>
Json::Value load_document(std::stringconst& filename){
Json::Value root;
Json::Reader reader;
std::ifstream ifs(filename, std::ifstream::binary);
if (!reader.parse(ifs, root, false))
throwstd::runtime_error("Unable to parse " + filename + ": "
+ reader.getFormatedErrorMessages());
return root;
}
The minimal validator function that tells us the location of all validation errors looks something like this:
#include<valijson/adapters/jsoncpp_adapter.hpp>#include<valijson/schema.hpp>#include<valijson/schema_parser.hpp>#include<valijson/validation_results.hpp>#include<valijson/validator.hpp>voidvalidate_json(Json::Value const& root, Json::Value const& schema_js){
using valijson::Schema;
using valijson::SchemaParser;
using valijson::Validator;
using valijson::ValidationResults;
using valijson::adapters::JsonCppAdapter;
JsonCppAdapter doc(root);
JsonCppAdapter schema_doc(schema_js);
SchemaParser parser(SchemaParser::kDraft4);
Schema schema;
parser.populateSchema(schema_doc, schema);
Validator validator(schema);
validator.setStrict(false);
ValidationResults results;
if (!validator.validate(doc, &results))
{
std::stringstream err_oss;
err_oss << "Validation failed." << std::endl;
ValidationResults::Error error;
int error_num = 1;
while (results.popError(error))
{
std::string context;
std::vector<std::string>::iterator itr = error.context.begin();
for (; itr != error.context.end(); itr++)
context += *itr;
err_oss << "Error #" << error_num << std::endl
<< " context: " << context << std::endl
<< " desc: " << error.description << std::endl;
++error_num;
}
throwstd::runtime_error(err_oss.str());
}
}
Note that in this example, jsoncpp connects as
#include <json-cpp/json.h>, while valijson/adapters/jsoncpp_adapter.hppin the current version, valijson assumes jsoncpp connects as #include <json/json.h>. So do not be surprised if the compiler does not find it json/json.h, and just fix it valijson/adapters/jsoncpp_adapter.hpp. Now we can upload and validate documents:
Json::Value const doc = load_document("/path/to/document.json");
Json::Value const schema = load_document("/path/to/schema.json");
try
{
validate_json(doc, schema);
...
return0;
}
catch (std::exception const& e)
{
std::cerr << "Exception: " << e.what() << std::endl;
return1;
}
That's it, we learned how to validate json documents. But note that now we have to think about where to store the circuits! Indeed, if a document changes every time and is obtained, for example, from a web request or from a command line argument, then the scheme is unchanged and must be delivered with the application. And for small programs without a developed mechanism for loading static resources, the need to introduce one represents a significant barrier to the implementation of validation through schemes. It would be great to compile the scheme with the program, because changing the scheme in any case will require changing the code that processes the document.
This is possible and even quite convenient if we have C ++ 11 at our disposal. The solution is primitive, but it works fine: we just define a string constant with our schema. And in order not to care about the quotes inside the string, we use raw string literal :
// Схема как R"(raw string)"staticstd::stringconst MY_SCHEMA =
R"({
"comment": "Schema for pole json specification",
"type": "object",
"patternProperties": {
"^[0-9]+(inv)?$": {
...
...
}
}
...
})";
// Загрузка json из строки
Json::Value json_from_string(std::stringconst& str);
{
Json::Reader reader;
std::stringstreamschema_stream(str);
Json::Value doc;
if (!reader.parse(schema_stream, doc, false))
throwstd::runtime_error("Unable to parse the embedded schema: "
+ reader.getFormatedErrorMessages());
return doc;
}
// Собственно валидация документа doc (validate_json определена выше)
validate_json(doc, json_from_string(MY_SCHEMA));
Thus, we have a convenient cross-platform cross-language mechanism for validating json documents, the use of which in C ++ does not require linking external libraries with inconvenient licenses, or fussing with paths to static resources. This thing can really save a lot of energy, and, importantly, help permanently kill XML as a format for representing objects, because it is inconvenient for both people and machines.
