Using the InternetTools FPC library in Delphi
In fact, the article is somewhat broader - it describes a method that allows you to transparently use many other libraries (and not only from the world of Free Pascal ), and InternetTools is chosen because of its remarkable feature - this is the case when (surprisingly) is missing Delphi-version with the same broad capabilities and ease of use.
This library is designed to extract information (parsing) from web documents (XML and HTML), allowing you to use both high-level query languages such as XPath and XQuery to specify the necessary data , and, as one of the options, providing direct access to the elements of the tree, built on the document.
Further material will be illustrated on the basis of a fairly simple task, which involves obtaining those elements of bulleted and numbered lists of this article that contain references, for which, if you refer to the documentation , such a small code is enough (it is based on the penultimate example with minor, minor changes) ):
However, now this compact and object-oriented code can only be written in Free Pascal, we also need to be able to use everything that this library provides in a Delphi application, preferably in a similar style, with the same facilities; It is also important to note that InternetTools is thread-safe (it can be accessed from many streams at the same time), so our option should provide this.
If we approach the task as far as possible from a distance, then there are several ways to use something written in another PL - they will be 3 large groups:
Further, in order to provide the reader with the opportunity to feel the difference, there are 2 options that are notable for their ease of use.
Let us first try to use InternetTools in a procedural style dictated by the very nature of a dynamic library, capable of exporting only functions and procedures; We will make the style of communication with the DLL look like WinAPI, when the handle of a certain resource is first requested, after which the useful work is performed, and then the destruction of the received handle occurs. It is not necessary to consider this variant as a role model in everything - it is chosen only for demonstration and subsequent comparison with the second one - a kind of poor relative.
The composition and ownership of the files of the proposed solution will look like this (arrows show dependencies):
Since in this case both Delphi and Free Pascal are very similar, it is quite reasonable to select such a common module containing the types used in the DLL export list in order to not duplicate their definition in the application
In this implementation, only one shy type is defined, but later the module will “mature” and its utility will become unquestionable.
The composition and procedures of DLL functions to select the minimum, but sufficient for the fulfillment of the above objectives :
Due to the demonstration nature of the current implementation, the full code is not given - much more important is how this simplest API will be used further. Here you just do not need to forget about the requirement of thread safety, which, although it will require some effort, but will not be something complicated.
Thanks to the previous preparations, it became possible to rewrite the example with lists in Delphi:
If you do not take into account the prototypes of functions and procedures from the dynamic library, then you can’t say that the code is catastrophically heavy compared to the Free Pascal version, but what if we complicate the task a little bit and try to filter some elements and output the addresses of links remaining:
It is possible to do this with the current API DLL, but the verbosity of the resulting is already very large, which not only greatly reduces the readability of the code, but also (and this is no less important) removes it from the above:
Obviously - in real, more complex cases, the volume of what has been written will only grow rapidly, and therefore we proceed to a solution that is free from such problems.
The procedural style of working with the library, as just shown, is possible, but has significant drawbacks. Due to the fact that the DLL as such supports the use of interfaces (as received and returned data types), you can organize work with InternetTools in the same convenient manner as when used with Free Pascal. In this case, the composition of the files should be slightly changed in order to distribute the declaration and implementation of interfaces into separate modules:
As before, we will consistently consider each of the files.
Declares the interfaces to be implemented in a DLL:
Conditional compilation directives are necessary due to the use of the module in an unchanged form in both Delphi and the FPC project.
The interface
The need to set the call type for all methods
The first one, both in importance and in scope - it is he, as reflected in the title, will contain the implementation of interfaces from the previous one: for both of them, a single class is assigned responsible
It is worthwhile to dwell on the method
In addition, this solution is thread-safe - provided that
Because of the work done in the modules above, the DLL only needs to export a single function (compare with the variant where the procedural style was used):
The procedure call is
If we now arrange the Delphi-solution of the original example based on the proposed interfaces, then it will hardly differ from that on Free Pascal, which means that the task set at the very beginning of the article can be considered completed:
The module is
A slightly complicated example likewise has minimal differences in Delphi:
If you look at the full capabilities of the interface IXQValue of InternetTools, it will be seen that the corresponding interface of
If you want to use a slightly different functionality, for example, managing cookies, the sequence of steps is very similar:
Although the InternetTools library is based on interfaces that imply automatic lifetime management, there is one non-obvious nuance that would seem to lead to memory leaks - if you run the next console application (created in Delphi, nothing will change in the case of FPC), then each time you press the enter key, the memory consumed by the process will increase:
There are no errors with the use of interfaces. The problem is that InternetTools does not release its internal resources allocated when analyzing a document (in a method
After its activation, the loss of resources will stop:
It is important to understand the following: a call
This library is designed to extract information (parsing) from web documents (XML and HTML), allowing you to use both high-level query languages such as XPath and XQuery to specify the necessary data , and, as one of the options, providing direct access to the elements of the tree, built on the document.
Brief introduction to InternetTools
Further material will be illustrated on the basis of a fairly simple task, which involves obtaining those elements of bulleted and numbered lists of this article that contain references, for which, if you refer to the documentation , such a small code is enough (it is based on the penultimate example with minor, minor changes) ):
uses
xquery;
const
ArticleURL = 'https://habr.com/post/415617';
ListXPath = '//div[@class="post__body post__body_full"]//li[a]';
var
ListValue: IXQValue;
begin
for ListValue in xqvalue(ArticleURL).retrieve.map(ListXPath) do
Writeln(ListValue.toString);
end.However, now this compact and object-oriented code can only be written in Free Pascal, we also need to be able to use everything that this library provides in a Delphi application, preferably in a similar style, with the same facilities; It is also important to note that InternetTools is thread-safe (it can be accessed from many streams at the same time), so our option should provide this.
Ways of implementation
If we approach the task as far as possible from a distance, then there are several ways to use something written in another PL - they will be 3 large groups:
- Placing the library in a separate process , the executable file of which is created by forces, in this case, FPC . This method can also be divided into two categories where network communication is possible:
- The area of interprocess communication is limited to only one computer. The most productive means (conditionally low level), intended for this kind of exchange, include shared memory and unnamed channels .
- Processes are accessible via the network — this is provided by the named pipes , RPC , DCOM, and so on.
- Encapsulating a library in a DLL (hereinafter sometimes referred to as a “dynamic library”), working, by definition, within a single process. Although COM objects can be placed in a DLL, the article will consider a simpler and less time consuming method, which, with all this, gives the same comfort when calling the library functionality.
- Porting . As in the previous cases, the appropriateness of this approach - rewriting code into another language - is determined by the balance between its pros and cons, but in the situation with InternetTools the disadvantages of porting are much more, namely: due to the considerable amount of library code, you need to do some serious work (even taking into account the similarity of programming languages), and also periodically, due to the development of the ported one , the task of transferring patches and new features to Delphi will appear.
Dll
Further, in order to provide the reader with the opportunity to feel the difference, there are 2 options that are notable for their ease of use.
"Classic" implementation
Let us first try to use InternetTools in a procedural style dictated by the very nature of a dynamic library, capable of exporting only functions and procedures; We will make the style of communication with the DLL look like WinAPI, when the handle of a certain resource is first requested, after which the useful work is performed, and then the destruction of the received handle occurs. It is not necessary to consider this variant as a role model in everything - it is chosen only for demonstration and subsequent comparison with the second one - a kind of poor relative.
The composition and ownership of the files of the proposed solution will look like this (arrows show dependencies):
InternetTools.Types module
Since in this case both Delphi and Free Pascal are very similar, it is quite reasonable to select such a common module containing the types used in the DLL export list in order to not duplicate their definition in the application
InternetToolsUsage, including functional prototypes from the dynamic library:unit InternetTools.Types;
interface
type
TXQHandle = Integer;
implementation
end.In this implementation, only one shy type is defined, but later the module will “mature” and its utility will become unquestionable.
InternetTools Dynamic Library
The composition and procedures of DLL functions to select the minimum, but sufficient for the fulfillment of the above objectives :
library InternetTools;
uses
InternetTools.Types;
function OpenDocument(const URL: WideString): TXQHandle; stdcall;
begin
...
end;
procedure CloseHandle(const Handle: TXQHandle); stdcall;
begin
...
end;
function Map(const Handle: TXQHandle; const XQuery: WideString): TXQHandle; stdcall;
begin
...
end;
function Count(const Handle: TXQHandle): Integer; stdcall;
begin
...
end;
function ValueByIndex(const Handle: TXQHandle; const Index: Integer): WideString; stdcall;
begin
...
end;
exports
OpenDocument,
CloseHandle,
Map,
Count,
ValueByIndex;
begin
end.Due to the demonstration nature of the current implementation, the full code is not given - much more important is how this simplest API will be used further. Here you just do not need to forget about the requirement of thread safety, which, although it will require some effort, but will not be something complicated.
InternetToolsUsage application
Thanks to the previous preparations, it became possible to rewrite the example with lists in Delphi:
program InternetToolsUsage;
...
uses
InternetTools.Types;
const
DLLName = 'InternetTools.dll';
function OpenDocument(const URL: WideString): TXQHandle; stdcall; external DLLName;
procedure CloseHandle(const Handle: TXQHandle); stdcall; external DLLName;
function Map(const Handle: TXQHandle; const XQuery: WideString): TXQHandle; stdcall; external DLLName;
function Count(const Handle: TXQHandle): Integer; stdcall; external DLLName;
function ValueByIndex(const Handle: TXQHandle; const Index: Integer): WideString; stdcall; external DLLName;
const
ArticleURL = 'https://habr.com/post/415617';
ListXPath = '//div[@class="post__body post__body_full"]//li[a]';
var
RootHandle, ListHandle: TXQHandle;
I: Integer;
begin
RootHandle := OpenDocument(ArticleURL);
try
ListHandle := Map(RootHandle, ListXPath);
try
for I := 0 to Count(ListHandle) - 1 do
Writeln( ValueByIndex(ListHandle, I) );
finally
CloseHandle(ListHandle);
end;
finally
CloseHandle(RootHandle);
end;
ReadLn;
end.If you do not take into account the prototypes of functions and procedures from the dynamic library, then you can’t say that the code is catastrophically heavy compared to the Free Pascal version, but what if we complicate the task a little bit and try to filter some elements and output the addresses of links remaining:
uses
xquery;
const
ArticleURL = 'https://habr.com/post/415617';
ListXPath = '//div[@class="post__body post__body_full"]//li[a]';
HrefXPath = './a/@href';
var
ListValue, HrefValue: IXQValue;
begin
for ListValue in xqvalue(ArticleURL).retrieve.map(ListXPath) do
if {Условие обработки элемента списка} then
for HrefValue in ListValue.map(HrefXPath) do
Writeln(HrefValue.toString);
end.It is possible to do this with the current API DLL, but the verbosity of the resulting is already very large, which not only greatly reduces the readability of the code, but also (and this is no less important) removes it from the above:
program InternetToolsUsage;
...
const
ArticleURL = 'https://habr.com/post/415617';
ListXPath = '//div[@class="post__body post__body_full"]//li[a]';
HrefXPath = './a/@href';
var
RootHandle, ListHandle, HrefHandle: TXQHandle;
I, J: Integer;
begin
RootHandle := OpenDocument(ArticleURL);
try
ListHandle := Map(RootHandle, ListXPath);
try
for I := 0 to Count(ListHandle) - 1 do
if {Условие обработки элемента списка} then
begin
HrefHandle := Map(ListHandle, HrefXPath);
try
for J := 0 to Count(HrefHandle) - 1 do
Writeln( ValueByIndex(HrefHandle, J) );
finally
CloseHandle(HrefHandle);
end;
end;
finally
CloseHandle(ListHandle);
end;
finally
CloseHandle(RootHandle);
end;
ReadLn;
end.Obviously - in real, more complex cases, the volume of what has been written will only grow rapidly, and therefore we proceed to a solution that is free from such problems.
Interface implementation
The procedural style of working with the library, as just shown, is possible, but has significant drawbacks. Due to the fact that the DLL as such supports the use of interfaces (as received and returned data types), you can organize work with InternetTools in the same convenient manner as when used with Free Pascal. In this case, the composition of the files should be slightly changed in order to distribute the declaration and implementation of interfaces into separate modules:
As before, we will consistently consider each of the files.
InternetTools.Types module
Declares the interfaces to be implemented in a DLL:
unit InternetTools.Types;
{$IFDEF FPC}
{$MODE Delphi}
{$ENDIF}
interface
type
IXQValue = interface;
IXQValueEnumerator = interface
['{781B23DC-E8E8-4490-97EE-2332B3736466}']
function MoveNext: Boolean; safecall;
function GetCurrent: IXQValue; safecall;
property Current: IXQValue read GetCurrent;
end;
IXQValue = interface
['{DCE33144-A75F-4C53-8D25-6D9BD78B91E4}']
function GetEnumerator: IXQValueEnumerator; safecall;
function OpenURL(const URL: WideString): IXQValue; safecall;
function Map(const XQuery: WideString): IXQValue; safecall;
function ToString: WideString; safecall;
end;
implementation
end.Conditional compilation directives are necessary due to the use of the module in an unchanged form in both Delphi and the FPC project.
The interface
IXQValueEnumeratoris not necessary in principle, however, in order to be able to use the cycles of the form " for ... in ..." as an example , one cannot do without it; the second interface is the main one and is an analog wrapper over IXQValuefrom InternetTools (it is specially made with the same name, to make it easier to correlate the future Delphi code with the library documentation on Free Pascal). If we consider the module in terms of design patterns, then the interfaces declared in it are adapters , albeit with a small feature — their implementation is located in the dynamic library. The need to set the call type for all methods
safecallwell described here . The obligation to use WideStringinstead of “native” strings will also not be justified, because the topic of exchanging dynamic data structures with a DLL is beyond the scope of the article.InternetTools.Realization Module
The first one, both in importance and in scope - it is he, as reflected in the title, will contain the implementation of interfaces from the previous one: for both of them, a single class is assigned responsible
TXQValue, whose methods are so simple that almost all consist of one line of code (this is quite expected) , because all the necessary functionality is already contained in the library - here you just need to refer to it):unit InternetTools.Realization;
{$MODE Delphi}
interface
uses
xquery,
InternetTools.Types;
type
IOriginalXQValue = xquery.IXQValue;
TXQValue = class(TInterfacedObject, IXQValue, IXQValueEnumerator)
private
FOriginalXQValue: IOriginalXQValue;
FEnumerator: TXQValueEnumerator;
function MoveNext: Boolean; safecall;
function GetCurrent: IXQValue; safecall;
function GetEnumerator: IXQValueEnumerator; safecall;
function OpenURL(const URL: WideString): IXQValue; safecall;
function Map(const XQuery: WideString): IXQValue; safecall;
function ToString: WideString; safecall; reintroduce;
public
constructor Create(const OriginalXQValue: IOriginalXQValue); overload;
function SafeCallException(ExceptObject: TObject; ExceptAddr: CodePointer): HResult; override;
end;
implementation
uses
sysutils, comobj,
w32internetaccess;
function TXQValue.MoveNext: Boolean;
begin
Result := FEnumerator.MoveNext;
end;
function TXQValue.GetCurrent: IXQValue;
begin
Result := TXQValue.Create(FEnumerator.Current);
end;
function TXQValue.GetEnumerator: IXQValueEnumerator;
begin
FEnumerator := FOriginalXQValue.GetEnumerator;
Result := Self;
end;
function TXQValue.OpenURL(const URL: WideString): IXQValue;
begin
FOriginalXQValue := xqvalue(URL).retrieve;
Result := Self;
end;
function TXQValue.Map(const XQuery: WideString): IXQValue;
begin
Result := TXQValue.Create( FOriginalXQValue.map(XQuery) );
end;
function TXQValue.ToString: WideString;
begin
Result := FOriginalXQValue.toJoinedString(LineEnding);
end;
constructor TXQValue.Create(const OriginalXQValue: IOriginalXQValue);
begin
FOriginalXQValue := OriginalXQValue;
end;
function TXQValue.SafeCallException(ExceptObject: TObject; ExceptAddr: CodePointer): HResult;
begin
Result := HandleSafeCallException(ExceptObject, ExceptAddr, GUID_NULL, ExceptObject.ClassName, '');
end;
end.It is worthwhile to dwell on the method
SafeCallException- its overlap, by and large, is not vital (operability TXQValuewill not suffer at all), however, the code given here allows you to pass to the Delphi-side the text of exceptions that will arise in the safecall methods (details can be found in an article already cited recently ). In addition, this solution is thread-safe - provided that
IXQValue, obtained, for example, through OpenURL, is not transferred between streams. This is due to the fact that the implementation of the interface only redirects calls to the already thread-safe InternetTools.InternetTools Dynamic Library
Because of the work done in the modules above, the DLL only needs to export a single function (compare with the variant where the procedural style was used):
library InternetTools;
uses
InternetTools.Types, InternetTools.Realization;
function GetXQValue: IXQValue; stdcall;
begin
Result := TXQValue.Create;
end;
exports
GetXQValue;
begin
SetMultiByteConversionCodePage(CP_UTF8);
end.The procedure call is
SetMultiByteConversionCodePagedesigned to work correctly with Unicode strings.InternetToolsUsage application
If we now arrange the Delphi-solution of the original example based on the proposed interfaces, then it will hardly differ from that on Free Pascal, which means that the task set at the very beginning of the article can be considered completed:
program InternetToolsUsage;
...
uses
System.Win.ComObj,
InternetTools.Types;
const
DLLName = 'InternetTools.dll';
function GetXQValue: IXQValue; stdcall; external DLLName;
const
ArticleURL = 'https://habr.com/post/415617';
ListXPath = '//div[@class="post__body post__body_full"]//li[a]';
var
ListValue: IXQValue;
begin
for ListValue in GetXQValue.OpenURL(ArticleURL).Map(ListXPath) do
Writeln(ListValue.ToString);
ReadLn;
end.The module is
System.Win.ComObjconnected not by chance - without it, the text of all safecall exceptions will become a faceless Exception in safecall method, and with it the original value generated in the DLL. A slightly complicated example likewise has minimal differences in Delphi:
...
const
ArticleURL = 'https://habr.com/post/415617';
ListXPath = '//div[@class="post__body post__body_full"]//li[a]';
HrefXPath = './a/@href';
var
ListValue, HrefValue: IXQValue;
begin
for ListValue in GetXQValue.OpenURL(ArticleURL).Map(ListXPath) do
if {Условие обработки элемента списка} then
for HrefValue in ListValue.Map(HrefXPath) do
Writeln(HrefValue.ToString);
ReadLn;
end.Remaining library functionality
If you look at the full capabilities of the interface IXQValue of InternetTools, it will be seen that the corresponding interface of
InternetTools.Typesdefines only 2 methods ( Mapand ToString) all of the rich set; adding the remaining ones, which the reader deems necessary in his particular case, is performed in exactly the same way and simple: the necessary methods are written in InternetTools.Types, after which InternetTools.Realizationthey are added to the module by code (most often as a single line). If you want to use a slightly different functionality, for example, managing cookies, the sequence of steps is very similar:
- A new interface is announced in
InternetTools.Types:
... ICookies = interface ['{21D0CC9A-204D-44D2-AF00-98E9E04412CD}'] procedure Add(const URL, Name, Value: WideString); safecall; procedure Clear; safecall; end; ... - Then it is implemented in the module
InternetTools.Realization:
... type TCookies = class(TInterfacedObject, ICookies) private procedure Add(const URL, Name, Value: WideString); safecall; procedure Clear; safecall; public function SafeCallException(ExceptObject: TObject; ExceptAddr: CodePointer): HResult; override; end; ... implementation uses ..., internetaccess; ... procedure TCookies.Add(const URL, Name, Value: WideString); begin defaultInternet.cookies.setCookie( decodeURL(URL).host, decodeURL(URL).path, Name, Value, [] ); end; procedure TCookies.Clear; begin defaultInternet.cookies.clear; end; ... - After that, a new exported function is returned to the DLL, which returns this interface:
... function GetCookies: ICookies; stdcall; begin Result := TCookies.Create; end; exports ..., GetCookies; ...
Resource Release
Although the InternetTools library is based on interfaces that imply automatic lifetime management, there is one non-obvious nuance that would seem to lead to memory leaks - if you run the next console application (created in Delphi, nothing will change in the case of FPC), then each time you press the enter key, the memory consumed by the process will increase:
...
const
ArticleURL = 'https://habr.com/post/415617';
TitleXPath = '//head/title';
var
I: Integer;
begin
for I := 1 to 100 do
begin
Writeln( GetXQValue.OpenURL(ArticleURL).Map(TitleXPath).ToString );
Readln;
end;
end.There are no errors with the use of interfaces. The problem is that InternetTools does not release its internal resources allocated when analyzing a document (in a method
OpenURL) —it needs to be done explicitly after it’s finished; For these purposes, the library module xqueryprovides a procedure freeThreadVarsthat it is logical to provide a call from the Delphi application by expanding the export list of the DLL:...
procedure FreeResources; stdcall;
begin
freeThreadVars;
end;
exports
...,
FreeResources;
...After its activation, the loss of resources will stop:
for I := 1 to 100 do
begin
Writeln( GetXQValue.OpenURL(ArticleURL).Map(TitleXPath).ToString );
FreeResources;
Readln;
end;It is important to understand the following: a call
FreeResourcesleads to the fact that all previously obtained interfaces become meaningless and any attempts to use them are unacceptable.