Data acquisition, part 2

In the first part of my story about data acquisition, I wrote about what tools are used to get HTML from the Internet. In this post I’ll talk in more detail about how to get the necessary data from this HTML, and how to transform this data into the format we need.

HTML Shaped

When you get HTML from some resource, you can have two options - either perfectly formed HTML that can be immediately converted to XML (that is, taken and used), or poorly formed HTML. Most HTML, unfortunately, is poorly formed. In this situation, there are two options: either use the HTML Agility Pack to pull out all the data you need, or use the same library to “adjust” the resulting HTML and make it more XML-like. Here is the most minimal example of how you can remove all unclosed items IMG:

var someHtml = "

hello

";

HtmlDocument doc = new HtmlDocument();

doc.LoadHtml(someHtml);

// fix images
foreach (var node in doc.DocumentNode.SelectNodes("//img"))

  if (!node.OuterHtml.EndsWith("/>"))

    node.Remove();

Console.WriteLine(doc.DocumentNode.OuterHtml);

Console.ReadLine();

It may seem to some that fixing HTML is an unnecessary task - after all, using the same method SelectNodes()you can get any element, even if this element is malformed. But there is one advantage that should not be forgotten - if you received the correct XML, then a) you can make (or generate) an XSD for this piece of XML; and b) after receiving XSD, you can generate mappings from the XML structure on POCO, which are much easier to work with.

Mappings

Data mapping typically appears in integration systems like BizTalk. The idea is to convert the dataset into anything - usually it's really just a different dataset. In fact, in many cases this is a one-to-one comparison, but different conversions are often needed - for example, all HTML is text, but to get a number, you need to do the conversion ( int.Parse()etc.). Let's take a look at how this is done.

Suppose we get the following (primitive) structure when parsing:

  

    

    

  

  

    

    

  

  

    

    

  

Alexander	RD
Sergey	MVP, RD
Dmitri	MVP

Now imagine that we need to map this data to the following structure:

class Person

{

  public string Name { get; set; }

  public bool IsMVP { get; set; }

  public bool IsRD { get; set; }

}



For this class, it is better to immediately create a collection class:

public class PersonCollection : Collection {}



Now we will generate XSD for the source data. The result looks something like this:



  

    

      

        

          

            

              

              

            

          

        

      

    

  





It's easy - probably too easy. What is more difficult is to get a diagram for our collection class. (Nb: instead of the scheme, you can use, for example, the database directly, but I will probably use XSD.) Warning, magic trick: we compile the assembly with the type PersonCollectionand then run the following command:

xsd -t:PersonCollection "04 Mapping.exe"



Do not believe it - this command generates XSD based on the CLR type! I note that launching XSDmakes sense only in the “bitness” of your system. Despite the fact that everything compiles for x86 for me, to make XSDit work I had to do a 64-bit assembly. It turned out the following XSD file with which you can do mapping:



  

  

    

      

    

  

  

    

      

      

      

    

  





Well, we have the left and right side of the mapping. Mapping itself can be created using an application such as Stylus Studio or MapForce. Mappings are created visually, but the creation process is not intuitive , so if you have never worked with visual mappings, you will have to suffer a bit at the beginning.

In order to create my mapping, I used the Altova MapForce program . In short, this program can do many different mappings, including XSD-on-XSD, which is what we need. Mappings are generated for the XSLT1 / 2, XQuery, Java, C #, and C ++ languages. Personally, I use XSLT2 for my purposes, and I use the free AltovaXML engine to run transformationsbecause all that Microsoft provides in .Net for XSLT is real misery. And XQuery in general in .Net is not present. And no, the Mvp.Xml library doesn’t help much either, although the prize is relied upon for the efforts of developers.

The first thing we do is visually describe mapping using the primitives available to us. The result looks something like this:




Now we generate XSLT for mapping. All that remains is to decide how to call it. Given that we use AltovaXML for transformation, the code itself looks like this:

public static string XsltTransform(string xml, string xslt)

{

  var app = new Application();

  var x = app.XSLT2;

  x.InputXMLFromText = xml;

  x.XSLFromText = xslt;

  return x.ExecuteAndGetResultAsString();

}



In order to deserialize XML into a collection, we use the following method:

public static T FromXml(string xml) where T : class

{

  var s = new XmlSerializer(typeof(T));

  using (var sr = new StringReader(xml))

  {

    return s.Deserialize(sr) as T;

  }

}



That’s all, actually - having received our XML, you can safely transform it:

string xml = File.ReadAllText("Input.xml");

string xslt = File.ReadAllText("../../output/MappingProjectMapToPersonCollection.xslt");

string result = XsltTransform(xml, xslt);

var pc2 = FromXml(result);



Mapping lyrics

Some might think that mapping is superfluous, and for simple cases, perhaps it really is. But I want to note that mapping, as an additional level of abstraction, allows you to better control the result and adapt it to changing conditions - and in the case of a changing site design, this is really true.

Mapping and working with XML is generally not free - Visual Studio (even 2010) does extremely poorly with it, so I used a specialized, paid program. Although no, I'm lying of course, because mappings are supported in BizTalk (and therefore in VS2008). And naturally, our task can be “transposed”, in a sense, on BizTalk. And what, for personal use, you can try if you sit on the MSDN-subscription.

That's all for today. Sources, as alwayshere . Comments welcome.