Interconversions JSON, YAML, XML

JSON, YAML are now popular, and XML technologies are considered a relic of the past.


Let's try to use "retro technology" to work with data in the JSON and YAML format. And we will discuss the reasons for using them today.

There is a task - to put the logic of data transformation into the application configuration, preferably in a declarative style and unified for different formats. The data can be in various text serialization formats json, yaml, xml, java properties, ini file. But at the same time, Data Lake is too heavy artillery for this. Put the data in a document-oriented or object-relational database and try to fulfill requests for the data uploaded there too over engineering for the first stage of ETL transformation.

Jsonpathrepeats a subset of XPath, but only to JSON format. And writing a declarative query without programming will not work - there is no analogue of XQuery. Alternatively, you could use some embedded database in jvm with its declarative language query, but this is a topic for a separate publication and the original data model in json, yaml is not relational.

Approach to data requests from JSON / YAML

XQuery can be performed on data in the Document Object Model. How to convert data from JSON / YAML to a DOM object ... You can use camel-xmljson or json2xml . In these libraries, the data source is json only. Therefore, rush on his dom-transformation bike. This library can take Map <String, Object> as input and turn it into org.w3c.dom.Node, and there is also a reverse conversion.

It remains to learn how to turn JSON and YAML into Map <String, Object>. For example, this can be done using the com.fasterxml.jackson.databind.ObjectMapper class from jackson .

Turn JSON into Map:

ObjectMapper mapper = new ObjectMapper();
Map<String, Object> objectTree = mapper.readValue(yaml, new TypeReference<Map<String, Object>>() {});

Turning YAML into a Map:

ObjectMapper mapper = new ObjectMapper(new YAMLFactory());
Map<String, Object> objectTree = mapper.readValue(yaml, new TypeReference<Map<String, Object>>() {});

Turn Map into a Document Object Model by connecting a library to the project :

DomTransformer toDom = new DomTransformer(new TypeAutoDetect()).transform(objectTree.size() == 1 ? objectTree : Collections.singletonMap("root", objectTree));
Node document = toDom.translate(objectTree);

You can use any XQuery implementation to execute queries. I like basex as a still developing open source project. We connect the org.basex: basex: jar: 9.0 dependency to the project and execute the declarative request:

String yaml = IOUtils.toString(TranslateTest.class.getResource("/pipeline.yml").toURI(), StandardCharsets.UTF_8);
ObjectMapper mapper = new ObjectMapper(new YAMLFactory());
Map<String, Object> objectGraph = mapper.readValue(yaml, new TypeReference<Map<String, Object>>() {});
Node document = new DomTransformer(new TypeAutoDetect()).transform(
                        objectGraph.size() == 1 ? objectGraph : Collections.singletonMap("root", objectGraph));
try(QueryProcessor proc = new QueryProcessor("declare variable $extDataset external; " +
        " $extDataset//*[text()='git-repo']", new Context())) {
    proc.bind("extDataset", document);
    Value queryResult = proc.value(); // execute the query
    queryResult.iter().forEach(System.out::println);
}

The results for the data from pipeline.yml



If you need to convert DOM / XML to JSON / YAML using jackson, then transform (Node currentNode) can help with this.

findings

With XQuery, you can query not only XML data. What this query language still successfully manages and this "old man" will still live in java data transformation projects even in JSON and YAML formats.

Of course, poorly structured data is not only JSON, YAML and XML. And it's too early to put an end to the processing of everything in the world ...



I hope that the approach from the publication will help you to perform declarative queries on heterogeneous data in the application. Or you have faced a similar problem in the JVM and you have better ideas, share in the comments!