KTV. Rake on the way to marshaling

    I wrote about KTV , but it is one thing to come up with something incomprehensible, another to try to use it. In addition to the S2 style system, I plan to use KTV to work with the server instead of JSON. I have no plans to conquer the world, but I want to figure out whether it was more convenient or not. In order to communicate easily, you need to be able to parse objects from ktv files, and serialize them back to them.

    Swift, for which I am writing this, at the moment (Swift 2.x), is not intended for dynamic parsing at all, in any way, at all. Therefore, I had to come up with something a little strange and non-standard. Then this strange and non-standard had to be implemented.

    In the process, an infinite number of rakes were stepped on, about which I will tell. Perhaps someone will laugh at the thoughtless me, maybe they will help someone avoid similar things - I don’t know. It was useful for me to understand.

    If anyone sees how it is easier or better to solve these problems, write. I am pleased to learn more options, since all that are listed below are, to one degree or another, crutches. Suddenly there is something more pleasant.

    How to get the structure of an object?


    The first task that arises if we want to convert what came over the network into a native object (in my case, the Swift language) is to figure out the structure of the object. Recalling the experience of Java (where two hundred libraries have already been written for each one), I looked at several ways.

    Introspection of language objects


    Reflection, runtime or at least something similar. There are two directions in Swift that are developing in this direction:

    • Class Mirror . This is the thing that the debugger or Playgrounds uses to display information about objects. Accordingly, the information is what they need: type, field name, value, generic. There are no access parameters (private / public), no annotations, and the method itself is not defined for all objects.
    • function reflect , which returns exactly the same data in a slightly different format.

    Most likely, both of these methods converge somewhere in one, so the result is similar. This method is the coolest if you can use it. Alas, it now works only for reading, there is no record in any form. We are waiting for the expansion of "mirrors", we move on to the next method.

    Parsing the source code. Sourcekit


    The next way is to parse the source itself, which, obviously, indicates everything that is possible. It would be extremely difficult if Apple did not provide SourceKit. What is SourceKit? This is a framework (sourcekitd.framework) that can execute requests like “proparsi, please, this file, and tell us what you see there in the form of a syntax tree (AST, Abstract Syntax Tree)”. It is extremely useful that SourceKit also knows how to parse documentation for language elements, and not just identifiers and types.

    In addition to the framework itself, SourceKitten lives on the Github , which provides a Swift interface to sourcekitd.framework. Using it is very simple:

    let sourceKitFile = File(path:classFileName) // получаем файл
    let structure = Structure(file:sourceKitFile!).dictionary // вытаскиваем AST

    However, to connect both sourcekitd.framework and SourceKitten, I had to work hard. It turned out somehow like this:

    • download, connect sourceKitten source
    • don't forget the addiction, SWXMLHash.swift
    • connect sourcekitd.framework and libclang.dylyb
    • write a Bridging-Header in which to include the necessary headers:
      #import "Index.h"
      #import "Documentation.h"
      #import "CXCompilationDatabase.h"
      #import "sourcekitd.h"

    After that, the lines above will work. After receiving AST, the job is only to interpret it correctly (for this I wrote a class and structure with different fields and ran it, checked that I would get SourceKit. I was able to pull out the types of objects (though only if they are written explicitly, type inferrence is not supported), and access modifiers, and to understand where constants are, where not. And to take generics from regular and associative arrays.

    In addition, SourceKit also produces documentation for the code element. Here, however, is also not without a rake:

    • for some reason, it does Structure(file:String)n’t show the documentation for me (and inside the structure with the documentation there are not enough types), to see it, you need to run another command:
      let structureWithDocs = SwiftDocs (file: sourceKitFile !, arguments: [classFileName]) ?. docsDictionary
    • behavior is not very deterministic. For example, if you write in the comment @name, then it disappears from the AST. Maybe there are some other secret words, I don’t know.

    Ok, got the structure, what to do next?

    Work with the structure


    There are three ways to work with a structure:

    • generate code at the compilation stage, which will marshal objects.
    • write a common code, which (usually again with the help of reflexion or another method of introspection), right in the process of execution, puts the necessary values ​​into the fields of objects and picks them up, producing a format for transmission.
    • generate a marshaling code right in the execution process.

    The second method disappears because mirrors in Swift cannot write anything to objects. The third method is very cool, but with no chance at all (unlike Java, where bytecode can be generated on the fly), only the first remains. That is, by structure, we need to create methods that will receive KTV or JSON as an input, issuing the desired, filled object, or vice versa, to receive text in KTV or JSON formats from the object.

    The code needs to be generated not anyhow (just picked and assigned values ​​to the fields):

    • you need to work with all kinds of checks (considering Optional types, for example),
    • be able to work with nested objects (class hierarchies)
    • be able to customize (at least map the property to a field with a different name)
    • have the ability to custom parsers / serializers (for example, if you get a date in a non-standard format)
    • Be able to generate code that is accessible from Objective-C, and not just from Swift.

    There are many problems, and in order to solve them all, I had to seriously bother.

    Stage 1. Just Assignment


    Before parsing the code, let's take a look at what we are actually doing. We have a model class that should create a bunch of model objects using data from a KTV file. The diagram shows the participants. The scheme as we work is a little more complicated.
    image

    How to do this in the simplest case? Let's start by learning how to pull data from a KTV object. Read the file - we have already read and saved it in a KTV object, which in structure is a bit like the associative array that it returns NSJSONSerialization.

    So, you need to write a function (or functions) that would pull the key value from KTV, and then assign it to the property. The only difficulty was that the values ​​are Optional, and the types are many different. Let's go through the steps.

    First, we write a method that pulls out a KTV value using an arbitrary key. In this case, links are resolved, mixins are taken into account. I got something like that. I also use this method to get the link text in S2, so we return tuple.

    private func valueAndReferenceForKey(key:String, resolveReferences:Bool = true) 
            throws -> (value:KTVValue?, reference:String?) {
        var result:KTVValue? = properties[key]
        var reference:String? = nil
        if let result_ = result {
            switch result_ {
                case .reference(let link):
                    if resolveReferences || link.hasPrefix("~") {
                        result = try findObjectByReference(link)
                    } else {
                        result = nil
                        reference = link
                            .stringByReplacingOccurrencesOfString("@", withString:"")
                    }
                default:
                    break
            }
        }
        return (result, reference)
    }

    After which I thought it was worth writing a generalized method that I could call to get values ​​of different types.

    private func specificValueForKey(key:String, defaultValue:T?, 
            resolveReferences:Bool, valueResolver:(value:KTVValue) throws -> T?) 
            throws -> (value:T?, reference:String?) {
        let (resultValue, reference) = try valueAndReferenceForKey(key,
                resolveReferences:resolveReferences)
        var result = defaultValue
        if let result_ = resultValue {
            result = try valueResolver(value:result_)
        }
        return (result, reference)
    }

    Here the only interesting part is the resolver, which can get the value of a specific type from the KTV value. When such a method is written, you can either use it directly, or write several wrappers for standard types to make it more convenient to use.

    public func string(key key:String, defaultValue:String? = "") 
            throws -> String? {
        return try specificValueForKey(key, defaultValue:defaultValue,
                resolveReferences:true, valueResolver:KTVValue.stringResolver).value
    }

    Another generalized method I use to work with Optional types in the same way.

    private func deoptionizeValue(value:T?, defaultValue:T) -> T {
        if let value_ = value {
            return value_
        } else {
            return defaultValue
        }
    }

    As a result, the parser consists of very simple blocks, each of which is dedicated to one property.

    stringProperty = deoptionizeValue(value:string(key:"key"), defaultValue:"")

    We have to deal with objects a little less conveniently, but from the point of view of tricks, it is quite obvious. I got something like this:

    if let object_ = getGeneralObject(name:"object", ktv:ktv) {
        object = ChildObject(ktvLenient:object_)
    } else {
        object = nil
    }

    Stage 2. Dates, research


    The next task that needed to be solved was date formatters. There may also be several solutions.

    Use special methods of the model class to format fields. I used this option in the Objective-C version of the same one, and it worked pretty well. The only problem is the separation of the field declaration and its parameters. Inconveniently, you constantly forget, or forget to correct, if the name of the property has changed. Plus, there is still a problem with the structures

    The fact is that structures have an automatically created initializer with the names of all fields. This is convenient, and given the semantics of this type, it is economical. If we want to use some methods of an object or class, then we need to use selfone that requires initialization before the call. Therefore, the structure needs a (empty) default initializer init(). This is superfluous code that needs to be written in each model class / in each structure (it is impossible to generate it, it must be in the main class) and which will always be forgotten.

    Use special classes, or dummies, as types of model class property. That is, not String, but MappedString. This will allow them to be configured (right at creation), for example, like this:, var property:MappedDate = MappedDate(format:'dd-MM-yyyy')and it will be possible to use their methods for serializing / parsing values. You can use tuples instead of a class, it looks absolutely monstrous, but it also works. There are many disadvantages to this solution, the main one is that you will need to dodge somehow ( object.property.valuefor example) to access the props . Well, the record is wild.

    Use mappers. Having tried the above options, I came to this.

    Stage 3. Custom Mappers


    I call a mapper a class that knows how a certain type is converted from / to KTV. It is possible to send KTV-value, so he returned to the real type ( String, NSDate, ...), and can be on the contrary, give the type to he returned KTV-value (serialization).

    The usual mapper, we already implemented a couple of sections back, for the line. In principle, you can wrap this code in a class, and then select the classes depending on the type of property. At this point, an interesting feature of Swift is revealed.

    The fact is that in the end, we need some kind of factory that, by type (by line or by class) will produce the desired object. It would be nice if this factory produced a generalized facility. Then there will be no ghosts, and the logic of the call will be as simple as possible, somehow

    let stringMapper:Mapper = mapper.getFor("String")
    var value:String = stringMapper.valueFromKTV(ktvValue)

    The problem is how to make this method . Due to the nature of generics in the protocols (more precisely, their absence, associative types are used instead), for example, you cannot make an array of generalized objects. That is, it is impossible (it doesn’t matter if you specify a generic type or not) . The array must already have specific types. As a result, I had to cheat a little. The array of mappers inside the factory had to be made some kind of general non-generalized protocol-parent of all mappers. One could just make an array , but it somehow doesn’t work out very well.func getFor(className:String) -> Mappervar mappers:\[Mapper] = \[]

    AnyObject

    public protocol KTVModelAnyMapper {}
    public class KTVModelMapper: KTVModelAnyMapper { ... }
    public class KTVModelMapperFactory {
        private var _mappersByType = [String:KTVModelAnyMapper]()
        public func mapper(type:String, propertyName:String) 
                throws -> KTVModelMapper {
            if let mapper = _mappersByType[propertyName] as? KTVModelMapper {
                return mapper
            } else {
                throw KTVModelObjectParseableError.CantFindTypeMapper
            }
        }
    }

    I did not come up with solutions without type casting. Maybe some of the readers tell me?

    As a result, the parser code is as follows:

    let mappers = KTVModelMapperFactory()
    do { 
        _stringOrNil = try mappers
            .mapper("String", propertyName:mappers.ktvNameFor("stringOrNil"))
            .parseOptionalValue(ktv["stringOrNil"], defaultValue:_stringOrNil)
    } catch { 
        errors["stringOrNil"] = error 
    }

    The beauty of this solution is that for each model class you can substitute your own factory, and thus control how the ktv values ​​are mapped into the object.
    image

    The question remains, how to set this custom factory for the model object? I came up with two options:

    • Use a custom protocol. In the protocol name, encode the name of the custom factory, and in the process of creating the parser extension, pull out the name with SourceKit, the name of the factory from there, and connect it when parsing. The solution is working, I checked, but it's terribly crooked.
    • No less awkward solution (I don’t know any good ones), but at least beautiful - use comments.

    Stage 4. Annotations in the comments


    The fact is that, in addition to information about classes and structures, SourceKit also produces comments bound to elements. The very ones from which documentation is then obtained starting with ///or /** */. Thus, you can stuff anything there, parse anything you like, and do what we want. It is clear that there is no typification here, writing is solely on the conscience of the developer, but, having tried all of the above methods (and a couple of absolutely terrible ones), it turns out that this is the most adequate.

    A comment with a custom factory for the model class looks like this:

    /// @mapper = RootObjectMapperFactory
    public struct RootObject: KTVModelObject { ... }

    And so, for example, you can check the name of the property in KTV:

    /// @ktv = __date__
    var date:NSDate

    However, in this place the possibilities are endless, since we are moving away from Swift and just starting to parse arbitrary text. You can set date formats, you can set arbitrary code, which then fits into a parser or serializer.

    Result, Conclusions


    Swift is still a bad language for magic libraries. That is, those in which you put something simple, it is boiled there and another simple and beautiful is given out. At the same time, on the part of the developer, “nothing needs to be done” (allegedly). This type of library is always the most complex, but it is it that shows the power of the platform. This is Hibernate, this is Rails, this is CoreData, and so on. It’s insanely hard to write on Swift now and only SourceKit reduces complexity to acceptable, if it weren’t, I would have to parse classes with my hands, which, to put it mildly, is an ungrateful task.

    However, like charging for the mind, this code turned out to be great. I haven’t found so many rakes in one place for a long time. You can touch what happened here: https://github.com/bealex/KTVThis is a very lively, non-production code on which I study working with Swift, so please treat it as well.

    I hope you will be interested. If you have any questions, ask!

    Also popular now: