YAML service on Java/Vue: Fast data import

Accelerating Enterprise Data Import: A YAML-Based Service with Java and Vue

In modern enterprise systems, the task of quickly and efficiently importing large volumes of historical or initial data often presents a significant challenge. Traditional approaches, which demand cumbersome code or complex configurations, can prolong the process for days. This article explores an innovative method for creating a lightweight, YAML-oriented service using a Java and Vue.js stack, designed to reduce data loading times to mere minutes, thereby significantly simplifying how technical specialists interact with the platform and its content.

The development of the new JMatrixPlatform, built on the principle of "everything as code," highlighted a critical issue: the lack of a convenient tool for one-time data uploads. Unlike older systems where data import was a trivial task, the new architecture required writing Java code for every operation, making the process laborious and inflexible. Analysts lost the ability to manage data independently, becoming entirely reliant on developers.

Several standard approaches were considered, each with significant drawbacks:

• Augmenting REST API with batch services: This would necessitate complex conversion of Excel data into JSON arrays, including escaping quotes within Excel formulas, which is a non-trivial task in itself. Using external tools like Postman would further complicate the process.
• Using curl + json for each object: This would require additional scripting for token retrieval and would not offer error resilience, halting the import at the first incorrect record.
• Direct import via SQL: This was ruled out because all data interaction logic is implemented at the application level. Direct database intervention, bypassing the platform's business logic, could lead to data inconsistency and compromise system integrity.
• Developing native Excel import for each entity: This was deemed economically unfeasible and time-consuming for one-off imports.

The need for a fast and flexible tool, empowering technical specialists to manage data uploads independently, became evident.

Solution: A YAML-Oriented Service for Batch Uploads

Recognizing the limitations of existing approaches, the author concluded that a minimalist service was needed, featuring a single entry point (endpoint) capable of accepting structured data to perform various operations (create, modify, delete). Instead of JSON, YAML was chosen due to its conciseness and ease of manual generation or automated creation using simple Excel formulas or LLMs.

A YAML request example demonstrates its simplicity and readability:

#request
- createObject:
    type: ru.commons.matrix.schema.type.ATPPerson
    policy: ru.commons.matrix.schema.policy.ALCPerson
- createObject:
    type: ru.commons.matrix.schema.type.ATPPerson1
    policy: ru.commons.matrix.schema.policy.ALCPerson
- createObject:
    type: ru.commons.matrix.schema.type.ATPPerson
    policy: ru.commons.matrix.schema.policy.ALCPerson

And the corresponding response, including execution status and object ID or an error message:

#response
---
- createObject:
    status: 200
    message: null
    oid: "f4ba679e-9253-4a83-a390-44daf7ac7756"
- createObject:
    status: 500
    message: "Admin type ru.commons.matrix.schema.type.ATPPerson1 not found. Enter a correct name or contact the administrator."
- createObject:
    status: 200
    message: null
    oid: "d9c98e74-bd17-4b3c-ae07-d9c307151c74"

This format allows for easy command generation, for instance, from Excel, using simple formulas:

="- createObject:
    id: "&K2&"
    type: "&I2&"
    policy: "&J2&"
    code: "&B2&"
    title: '"&C2&"'"

This approach isn't a full-fledged DSL (Domain Specific Language) but leverages existing REST API DTOs (Data Transfer Objects) wrapped in action commands, ensuring flexibility and reusability.

Technical Implementation with Java and Spring Framework

The core of the service is the JQLController, which processes incoming YAML requests.

@RestController
@RequestMapping("jql")
@RequiredArgsConstructor
public class JQLController {

  /**
   * Executes a batch of commands.
   *
   * LinkedHashMap is used to preserve the order of input and output commands
   * - input commands are processed in the order they appear in YAML
   * - responses are returned in the same order as requests
   * - this is crucial for scenarios where execution order matters (create → connect)
   * Jackson uses LinkedHashMap by default, but explicit specification
   * protects against accidental implementation changes in the future.
   */
  @PostMapping(consumes = "application/yaml", produces = "application/yaml")
  public ResponseEntity<List<LinkedHashMap<String, JQLResponseData>>> promote(@JPathContextVariable JContext ctx,
      @RequestBody List<LinkedHashMap<String, Object>> commands) {

    List<LinkedHashMap<String, JQLResponseData>> results = new ArrayList<>(commands.size());
    for (Map<String, Object> command : commands) {
      Map<JQLEnum, IJQLDTO> parsed = fromYaml(command);
      //an error in a command does not cause the entire batch to fail
      //run accounts for this, so a try-catch here is not needed
      results.add(run(ctx, parsed));

    }

    return ResponseEntity.ok(results);
  }

  private static Map<JQLEnum, IJQLDTO> fromYaml(Map<String, Object> commands) {
    Map<JQLEnum, IJQLDTO> command = new LinkedHashMap<>();

    for (Map.Entry<String, Object> entry : commands.entrySet()) {
      String key = entry.getKey();
      Object value = entry.getValue();

      JQLEnum jqlEnum = JQLEnum.valueOf(key);
      Class<? extends IJQLDTO> dtoClass = jqlEnum.getDTOClass();

      IJQLDTO dto = JObjectJSON.MAPPER.convertValue(value, dtoClass);
      command.put(jqlEnum, dto);
    }

    return command;
  }

  /**
   * Executes commands and returns the result in the same format.
   *
   * Input: { "createObject": { "type": "...", "policy": "..." } }
   * Output: { "createObject": { "status": 200, "id": "..." } }
   *
   * or
   *
   * Output: { "createObject": { "status": 500, "message": "..." } }
   */
  private static LinkedHashMap<String, JQLResponseData> run(JContext ctx, Map<JQLEnum, IJQLDTO> commands) {
    LinkedHashMap<String, JQLResponseData> response = new LinkedHashMap<>();

    if (commands.isEmpty()) {
      return response;
    }

    try {
      ctx.getTxUpdate().executeWithoutResult(tx -> {
        for (Map.Entry<JQLEnum, IJQLDTO> entry : commands.entrySet()) {
          response.put(entry.getKey().name(), entry.getKey().execute(ctx, entry.getValue()));
        }
      });

    } catch (JMatrixLocalizedError ex) {
      commands.keySet().forEach(el -> {
        response.put(el.name(), new JQLResponseData(500, ex.getLocalizedMessage(ctx.getLocale())));
      });
    } catch (Exception ex) {
      commands.keySet().forEach(el -> {
        response.put(el.name(), new JQLResponseData(500, ex.getMessage()));
      });
    }

    return response;
  }
}

The JQLController uses Spring annotations to handle POST requests with an application/yaml content type. A key feature is the use of LinkedHashMap to preserve command order, which is critical for scenarios where operation sequence matters (e.g., creating an object before linking it). The fromYaml method is responsible for deserializing YAML objects into corresponding DTOs, using JQLEnum to determine the command type and DTO class. The run method executes commands within a transaction, handling errors and forming a structured YAML response.

Spring Configuration for YAML Support

Since Spring Framework does not natively support application/yaml as a content type, additional HttpMessageConverter configuration is required.

@Configuration
public class YamlConfig {

  @Bean
  public YamlHttpMessageConverter yamlHttpMessageConverter() {
    YAMLFactory factory = new YAMLFactory();
        //.disable(YAMLGenerator.Feature.SPLIT_LINES)
        //.disable(YAMLGenerator.Feature.WRITE_DOC_START_MARKER);

    YAMLMapper mapper = new YAMLMapper(factory);

    return new YamlHttpMessageConverter(mapper);
  }

  public static class YamlHttpMessageConverter extends AbstractJackson2HttpMessageConverter {
    public YamlHttpMessageConverter(ObjectMapper objectMapper) {
      super(objectMapper,
          MediaType.parseMediaType("application/yaml"),
          MediaType.parseMediaType("application/x-yaml"),
          MediaType.parseMediaType("text/yaml"));
    }
  }
}

YamlConfig defines a YamlHttpMessageConverter bean, which extends AbstractJackson2HttpMessageConverter and registers a YAMLMapper to handle YAML formats (application/yaml, application/x-yaml, text/yaml). This allows Spring to automatically marshal and unmarshal YAML data within controller objects.

DTO Structure and Command Enum

To standardize responses, a base DTO JQLResponseData was developed:

@Getter
public class JQLResponseData {
  private int status = 200;
  private String message = null;

  public JQLResponseData() {

  }

  public JQLResponseData(int status, String message) {
    this.status = status;
    this.message = message;
  }

}

This class provides a consistent format for execution status and error messages.

Initially, for the MVP, the set of commands was implemented via enum JQLEnum, which simplified development. However, in more mature systems, this could be replaced by a more flexible mechanism for registering command classes.

public enum JQLEnum {
  createObject {
    @Override
    JQLResponseData execute(JContext ctx, IJQLDTO value) {
      JDTODomainObject dto = (JDTODomainObject) value;

      JDomainObject object;
      if (dto.getId() == null) {
        object = new JDomainObject();
      } else {
        object = new JDomainObject(dto.getId());
      }
      dto.unmap(object);
      object.create(ctx, JModel.getRequiredAdminByName(dto.getType()), JModel.getRequiredAdminByName(dto.getPolicy()));

      return new CreateDomainRS(object.getId());
    }

    @Override
    public Class<? extends IJQLDTO> getDTOClass() {
      return JDTODomainObject.class;
    }

  },
  deleteObject {
    @Override
    JQLResponseData execute(JContext ctx, IJQLDTO value) {
      DeleteDomainRQ dto = (DeleteDomainRQ) value;
      new JDomainObject(dto.getId()).delete(ctx);

      return new JQLResponseData();
    }

    @Override
    public Class<? extends IJQLDTO> getDTOClass() {
      return DeleteDomainRQ.class;
    }

  },
  //etc.

  abstract JQLResponseData execute(JContext ctx, IJQLDTO value);

  public abstract Class<? extends IJQLDTO> getDTOClass();
}

Each JQLEnum element encapsulates the logic for executing a specific command (execute) and provides a method to retrieve the corresponding DTO class (getDTOClass). This allows for centralized management of available operations and their processing.

User Interface with Vue.js

To interact with the YAML service, a simple web interface was developed using Vue.js. It features two areas: one for entering YAML requests and another for displaying results. The ace-builds library provides convenient YAML code editing with syntax highlighting.

<template>
  <div class="jql-base-div">
    <div ref="refRequests" class="jql-requests-div" @keyup.alt.enter="handleAltEnter"></div>
    <div ref="refResults" class="jql-response-div"></div>
  </div>
</template>

<script setup>
import { useJServices } from '@/composables/useJServices';

const { serviceFetch } = useJServices()

import ace from 'ace-builds';

import 'ace-builds/src-noconflict/mode-yaml';
import 'ace-builds/src-noconflict/theme-chrome';

import { onMounted, ref } from 'vue';

ace.config.set('basePath', '/ace')
ace.config.set('workerPath', '/ace')
ace.config.set('themePath', '/ace')

const props = defineProps({
  routeParams: Object,
  routeQuery: Object,
  requestBody: Object,
  metaComponent: Object
})

const refRequests = ref(null)
const refResults = ref(null)

let aceEditorRequests = null
let aceEditorResponse = null

onMounted(() => {
  document.title = 'JMatrix: JQL'

  aceEditorRequests = ace.edit(refRequests.value)
  aceEditorRequests.setTheme("ace/theme/chrome")
  aceEditorRequests.session.setMode("ace/mode/yaml")
  aceEditorRequests.setOptions({
    fontSize: "13px",
    showPrintMargin: false,
    r

The interface provides an interactive environment for testing and executing YAML commands, enabling developers and analysts to quickly verify operation results and adjust requests. The use of ace-builds significantly enhances the convenience of working with YAML structures.

Advantages and Future Development

The implementation of the YAML service has drastically reduced the time required for historical data loading, from days to minutes. This solution has not only improved the efficiency of pre-sales customizations but also significantly expanded analysts' capabilities by providing them with a tool for independent data work without direct developer involvement.

Key advantages of this approach:

• Speed and Efficiency: Rapid data loading thanks to a simplified format and batch processing.
• Flexibility: Easy generation of requests from various sources (Excel, LLM) without complex coding.
• Analyst Autonomy: Reduced reliance on developers for routine data import operations.
• Architectural Cleanliness: Utilization of existing DTOs and preservation of business logic at the application level.

Future development considerations include:

• Refactoring JQLEnum to a more dynamic command registration mechanism, using reflection or configuration, to simplify the addition of new operations.
• Expanding the UI with YAML schema validation capabilities and more sophisticated response processing.
• Integration with version control systems to track changes in imported data.

This approach demonstrates how thoughtful architecture and the selection of appropriate technologies can solve complex data import challenges, significantly boosting team productivity and system flexibility.

Key Takeaways

• Problem: Traditional data import methods (REST API, SQL, native utilities) are inefficient for one-time historical data loading in "code-first" enterprise systems, often requiring days of work.
• Solution: Creation of a lightweight, YAML-oriented service using Java (Spring Framework) and Vue.js for batch command processing.
• Choice of YAML: YAML was preferred over JSON due to its conciseness, ease of generation (from Excel, LLM), and readability for technical specialists.
• Architecture: The service uses JQLController to process YAML requests, YamlHttpMessageConverter for Spring integration, standardized DTOs (JQLResponseData), and an enum for command execution.
• Result: Reduced data import time from days to minutes, increased analyst autonomy, and enhanced system flexibility while maintaining business logic integrity.

— Editorial Team