xNet - C # Web Library

Gradually, with the study of C # and the .NET Framework , I began to write various Helpers that hid routine code behind the call of just one method. After that, it turned into the development of a full-fledged library, which I want to introduce to you. This library is written completely 'from scratch'.

So what is xNet?

xNet is a class library for the .NET Framework , which includes:

• Classes for working with proxy servers: HTTP, Socks4 (a), Socks5, Chain .
• Classes for working with the HTTP 1.0 / 1.1 protocol: keep-alive, gzip, deflate, chunked, SSL, proxies and more .

Download: releases
Sources: github.com/X-rus/xNet The

article was significantly rewritten in connection with the release of the new version of the library. Last update 09/12/2015

The basics

HttpRequest is used to send requests . In it, you can set various settings: headers, timeouts, whether to follow forwarding, whether to keep a permanent connection, and others. When you send a request, it first accepts the response headers using HttpResponse , and then returns a link to this object so that you can load the message body. You can download the message body using one of the special methods.

Request example:

using (var request = new HttpRequest())
{
    request.UserAgent = Http.ChromeUserAgent();
    // Отправляем запрос.
    HttpResponse response = request.Get("habrahabr.ru");
    // Принимаем тело сообщения в виде строки.
    string content = response.ToString();
}

The using statement is used to close the connection to the server. You can do the same by calling the HttpRequest.Close method .

You can also get a link to HttpResponse using the HttpRequest.Response property . The returned HttpResponse is not unique. For each HttpRequest, there is only one HttpResponse (it is reused on every request).

Sending simple requests and loading the message body

GET request example:

using (var request = new HttpRequest())
{
    var urlParams = new RequestParams();
    urlParams["param1"] = "val1";
    urlParams["param2"] = "val2";
    string content = request.Get("habrahabr.ru", urlParams).ToString();
}

RequestParams is used to set URL parameters. In this case, this is equivalent to the following entry:

Get("habrahabr.ru/?param1=val1¶m2=val2")

You can set URL parameters using the HttpRequest.AddUrlParam method . These parameters will be erased after the first request.

Example GET request with temporary URL parameters:

using (var request = new HttpRequest())
{
    request.AddUrlParam("param1", "val1").AddUrlParam("param2", "val2");
    string content = request.Get("habrahabr.ru").ToString();
}

An example of a POST request:

using (var request = new HttpRequest())
{
    var reqParams = new RequestParams();
    reqParams["login"] = "neo";
    reqParams["password"] = "knockknock";
    string content = request.Post(
        "www.whitehouse.gov", reqParams).ToString();
}

You can set request parameters using HttpRequest.AddParam . These parameters will be erased after the first request.

An example of a POST request with temporary parameters:

using (var request = new HttpRequest())
{
    request.AddParam("login", "neo").AddParam("password", "knockknock");
    string content = request.Post("www.whitehouse.gov").ToString();
}

You can also send a string, an array of bytes, a file, or any object that inherits from HttpContent .

One of the five HttpResponse methods is used to load the message body : ToString , ToBytes , ToFile , ToStream or ToMemoryStream . If the message body is not needed, then the None method should be called .

If you want the request parameters to be encoded, say, in 1251 encoding, then you need to set the property HttpRequest.CharacterSet = Encoding.GetEncoding (1251)

Sending Multipart / form data

About Multipart / form data . To work with them, the MultipartContent class is used .

Multipart / form request example:

using (var request = new HttpRequest())
{
    var multipartContent = new MultipartContent()
    {
        {new StringContent("Bill Gates"), "login"},
        {new StringContent("qwerthahaha"), "password"},
        {new FileContent(@"C:\windows_9_alpha.rar"), "file1", "1.rar"}
    };
    request.Post("www.microsoft.com", multipartContent).None();
}

In order not to create an object of the MultipartContent class yourself , you can use two special HttpRequest methods : AddField and AddFile . These parameters will be erased after the first request.

Example of a Multipart / form request with time parameters:

using (var request = new HttpRequest())
{
    request
        .AddField("login", "Bill Gates")
        .AddField("password", "qwerthahaha")
        .AddFile("file1", @"C:\windows_9_alpha.rar");
    request.Post("www.microsoft.com").None();
}

Permanent connection and reconnection

HttpRequest supports persistent connections. They can be disabled using the HttpRequest.KeepAlive property .

Example:

using (var request = new HttpRequest("habrahabr.ru"))
{
    request.Get("/").None();
    request.Get("/feed").None();
    request.Get("/feed/posts");
}

All three requests will be executed using the same connection, but this is only if the server does not close the connection earlier. Note that in the last request, I do not call the None method . The fact is that it loads the message body so that the next request can be correctly executed when using a permanent connection, and since this is the last request, you can not download the message body, because the connection will still be closed.

A little more about the None method . If a persistent connection is not used, then this method simply closes the connection. If you do not call this method, it will be called automatically at the next request. I recommend calling this method yourself.

Keep in mind that when working with persistent connections, disconnections from the server may occur. This can happen because the server has reached the limit of requests for one connection, or the timeout for the next request has expired. HttpRequest can handle such situations by trying to reconnect (this behavior is not related to Reconnect ). If it fails to connect, it will throw an exception. To fine-tune persistent connections, you can use the KeepAliveTimeout and MaximumKeepAliveRequests properties .

Additionally, you can manage reconnections using the Reconnect , ReconnectLimit, and ReconnectDelay properties.. By default, this behavior is disabled. HttpRequest will try to reconnect if an error occurs while connecting to the server, or if an error occurs during data upload / download. But reconnection will not work if an error occurs while loading the message body (in the ToString () method, for example). This behavior may come in handy when working with proxies or if you have an unstable internet connection.

Set cookies, headers and more

Additional headers are set using a special indexer. Keep in mind that some headers can only be set using special properties. Their list can be found in the documentation. The headers specified through the indexer are sent in all requests. If you need to set a temporary header for one request, then use the HttpRequest.AddHeader method . Such headers overlap the headers specified through the indexer.

Cookies are set using the HttpRequest.Cookies property . Cookies may vary based on server response. To prevent this, set the value of the CookieDictionary.IsLocked property to true .

Example:

using (var request = new HttpRequest("habrahabr.ru"))
{
    request.Cookies = new CookieDictionary()
    {
        {"hash", "yrttsumi"},
        {"super-hash", "df56ghd"}
    };
    request[HttpHeader.DNT] = "1";
    request["X-Secret-Param"] = "UFO";
    request.AddHeader("X-Tmp-Secret-Param", "42");
    request.AddHeader(HttpHeader.Referer, "http://site.com");
    request.Get("/");
}

At the HttpRequest and HttpResponse has additional methods for working with cookies and headers: ContainsHeader , EnumerateHeaders , clearAllHeaders , ContainsCookie , ContainsRawCookie and others.

To send requests, you need to set the User-Agent. It can be generated by one of the Http methods : IEUserAgent , OperaUserAgent , ChromeUserAgent , FirefoxUserAgent or OperaMiniUserAgent .
And ask like this:

request.UserAgent = Http.ChromeUserAgent();

There used to be a RandomUserAgent method , but I deleted it, since using it can lead to some problems. Let's say a site for Google Chrome produces one content, but for IE it’s a little different, or completely different. Therefore, with caution use User Agents from various browsers.

In HttpRequest there is still a lot of different properties with which you can manipulate sending headers. See the documentation.

Proxy Connection

xNet supports the following types of proxies:

• HTTP - class HttpProxyClient
• Socks4 (a) - class Socks4ProxyClient (a)
• Socks5 - class Socks5ProxyClient

All of these classes inherit from the ProxyClient class . There is an additional class ChainProxyClient , which allows you to create a connection through a chain of proxies.

Example:

var proxyClient = HttpProxyClient.Parse("127.0.0.1:8080");
var tcpClient = proxyClient.CreateConnection("habrahabr.ru", 80);
// Далее работаем с соединением.

HttpRequest supports work through a proxy server. To do this, set the HttpRequest.Proxy property .

Error processing

If an error occurs when working with the HTTP protocol, an HttpException will be thrown , and if an error occurs when working with a proxy server, a ProxyException will be thrown . Both of these exceptions inherit from the NetException class .

Example:

try
{
    using (var request = new HttpRequest())
    {
        request.Proxy = Socks5ProxyClient.Parse("127.0.0.1:1080");
        request.Get("habrahabr.ru");
    }
}
catch (HttpException ex)
{
    Console.WriteLine("Произошла ошибка при работе с HTTP-сервером: {0}", ex.Message);
    switch (ex.Status)
    {
        case HttpExceptionStatus.Other:
            Console.WriteLine("Неизвестная ошибка");
            break;
        case HttpExceptionStatus.ProtocolError:
            Console.WriteLine("Код состояния: {0}", (int)ex.HttpStatusCode);
            break;
        case HttpExceptionStatus.ConnectFailure:
            Console.WriteLine("Не удалось соединиться с HTTP-сервером.");
            break;
        case HttpExceptionStatus.SendFailure:
            Console.WriteLine("Не удалось отправить запрос HTTP-серверу.");
            break;
        case HttpExceptionStatus.ReceiveFailure:
            Console.WriteLine("Не удалось загрузить ответ от HTTP-сервера.");
            break;
    }
}

Additional classes

The WinInet class allows you to interact with the network settings of the Windows operating system. Using it, you can find out if the computer is connected to the Internet or get the value of the Internet Explorer proxy server.

The Html class is designed to help you work with HTML and other text data. It contains a method that replaces HTML entities in the text with the characters that represent them. Example: & _quot; test & _quot; in the "test". A method that replaces Unicode entities with the characters that represent them. Example: \ u2320 or \ U044F. And methods for extracting strings: Substring, LastSubstring, Substrings.

Related Links

• HTTP
• RFC2616 - HTTP / 1.1
• HTTP headers
• HTTP cookie
• Multipart / form-data
• Proxy server