Universal Binary JSON is another binary JSON

The article is a free translation of the information provided on the official website.

Introduction

JSON is a widespread and popular data exchange format. Its elegance, ease of processing and a relatively rich type system have become a natural choice for many developers who need to quickly and easily save or randomly transfer data between systems.

Unfortunately, the process of packing and unpacking structures native to the programming language using a textual representation of data suitable for transmission over a network has significant resource costs. In heavily loaded systems, avoiding the JSON- formatted text processing stepmay lead to higher, better processing time information and a reduction in the size of stored data.

To achieve better results in such cases, it becomes useful to use the binary JSON format .

Why?

Attempts to make JSON use faster, faster with binary specifications such as BSON , BJSON, or Smile , exist, but they fail for two reasons:

1. Internal data types . The use of internal data types that are exclusively unique to binary formats and were not originally included in the JSON standard makes the above specifications unsuitable for widespread use, since, depending on the implementation, each such type can be interpreted differently.
2. The complexity of the implementation . Some formats allow for higher performance, while others provide a more compact presentation due to a more complex, confusing specification. Which, in turn, slows down or makes impossible their distribution and implementation. Ease of use is the engine of JSON success .

For example, BSON defines data types for regular expressions, blocks of JavaScript code, and other constructs that do not have a corresponding data type in JSON . BJSON also defines its data types, leaving ample room for errors related to the interpretation of types in two different implementations. Smile defines more complex data types, rules for generating and parsing in order to efficiently use space.

All existing binary JSON specifications suffer from incompatibility and implementation difficulties, which naturally ruins the main advantage of JSON , which made it so popular, is simplicity. JSON

simplicityallowed to create implementations in various programming languages ​​and made it convenient and understandable immediately for everyone who uses your data directly.

Any successful binary JSON specification must adopt these properties in order to be truly useful to the community as a whole.

Why not JSON + gzip?

JSON compression may be a better solution than using binary formats. But there are two problems:

1. The data processing speed drops by 50% .
2. There is no way to research data and work with them directly.

It turns out that the size of the transmitted data can be reduced by an average of 75%, but at the same time the processing overhead will increase significantly.

Goals

Universal Binary JSON specification is designed exclusively on the principles of full compatibility with JSON , simplicity, speed and accessibility for understanding. Reading and writing in this format are trivial in nature. As a side effect - reducing the space occupied by the data by an average of 30%.

1. Full compatibility . 100% JSON compatibility and exclusive use of data types supported by all modern programming languages. This allows you to efficiently convert data between JSON and Universal Binary JSON without developers using intricate data structures that may not be supported by the programming language.
2. Ease of use . This is achieved due to the fact that the JSON specification is taken as the basis and only one binary structure is used for a rational description of types. Thanks to this, we get the availability and ease of understanding by the developers.
3. Speed ​​and efficiency . The motivation for using binary formats is the speed and efficiency of data parsing. Moreover, as a side effect, a reduction in space consumption by 30%.

Data format

General view of the single byte structure in the specification used to describe all supported types

[type, 1-byte char] ([length, 1 or 4-byte integer]) ([data])

• type - 1 byte, ASCII character. Used to indicate the type of data that follows it.
• length (optional) - 1 or 4 bytes (integer value) depending on the length or size of the object. For an array, its length. For an object, the number of Key / Value pairs. If the length or number of elements is from 0 to 254 inclusive, then 1 byte is used. This field with a value of 255 is reserved for objects and arrays of unknown length.
• data (optional) - a sequence of bytes that directly represents the data of the object.

The length and data fields are used or not used depending on the data type. For example, a type 32 bit integer has a standard size of 4 bytes. To write a value of this type, you need 1 byte to indicate the type and 4 bytes to the value itself. In this case, the length field is not used due to uselessness.

Thanks to this presentation of information, the goals are achieved.

Opportunities

The Universal Binary JSON specification supports:

• basic data types
• arrays
• objects
• integers, arrays and objects of unknown length or size
• data streaming

Important features : values ​​of numeric types are written in byte order from high to low ( Big-Endian ) and the main encoding of text information is UTF-8 .

Community

• Official UBJSON Specification Website - Universal Binary JSON
• Took part and discussed
• Java implementation - Universal Binary JSON Java Library
• Simple C # Implementation - Ubjson.NET

Implementations in other programming languages ​​will be available as they appear on this link .

PS: The author of the specification and implementation in Java, Riyad Kalla and I, the author of the article and implementation in C #, we will be glad to any of your participation in the process of working on the specification.