June 14, 2011 at 19:24Parsing in Python. How to build a Dovecote archive
The article describes the development of a script in Python . The script parses HTML-code , compiling a list of site materials, downloading articles and preliminary cleaning of the article’s text from "extraneous" elements. The libraries urllib (retrieving HTML pages), lxml (parsing HTML code, deleting elements and saving a "cleaned" article), re (working with regular expressions), configobj (reading configuration files) are used.
To write a script, basic knowledge of the Python language, programming skills and code debugging are sufficient.
The article provides explanations on the use of libraries on the example of compiling a list of publications S.M. Golubitsky, a link to a working script.
I can hardly be mistaken in saying that many hawkers are familiar with the indefatigable work of Sergei Golubitsky . For nearly 15 years of near-computer journalism, Namerek published 433 articles on the mountain in the paperless Komputer, prematurely rested in the Bose, and more than 300 Dovecats on the Computerra-online portal. And this is not counting the analytical research about the heroes of foreign gesheft in the Business Journal, the opening of the curtain on the secrets of creativity in the “Home Computer”, articles in the “Russian Journal”, “D`” and so on. and so on. Those interested in completeness reviews of life-creation will be found at the above links.
Last year, the author’s project “ Old Pigeon House and His Friends ” began work”, Which was conceived (and became) in particular by a constantly updated archive of publications of the author himself and a platform for conducting cultural-povidlian discussions. As a person who is not indifferent to the topics of network life, social mythology and self-development, masterfully revealed by the author, as well as eager for quality leisure reading, I once became a regular get-together at the Dovecote. To the best of my ability I try not only to keep the project in sight, but also somehow participate in its development.
As I was working in the field of proofreading of articles transferred to the archive from the Computerra-online portal, the first thing I decided to do was make an inventory of all the Pigeons.
So, the task, on the example of which we will consider parsing sites in Python, consisted of the following:
In terms of the programming language, my choice immediately and unequivocally fell on Python. Not only because I studied it at my leisure a few years ago (and then used the Active Python shell for some time as an advanced calculator), but also for the abundance of libraries, source code examples and the ease of writing and debugging scripts. Not least, they were also interested in the prospects for the further use of the acquired skills to solve immediate problems: integration with the Google Docs API, word processing automation, etc.
Solving a very specific task practically from scratch, the toolkit was selected so as to spend minimal time reading documentation, comparing libraries, and, ultimately, implementing. On the other hand, the solution should have the versatility sufficient for easy adaptation to other sites and similar tasks. Perhaps some tools and libraries are imperfect, but they allowed to ultimately bring the plan to the end.
So, the choice of tools began to determine the appropriate version of Python. I initially tried using Python 3.2, but in the process of experimenting, I stopped at Python 2.7 , because some examples on the “troika” did not go.
To simplify the installation of additional libraries and packages, I used setuptools - a tool for downloading, building and installing packages.
Additionally libraries were installed:
As improvised means were used:
The articles and discussions on Habré provided invaluable help:
As well as manuals, examples and documentation:
And of course, the Python Programming Language book
The task includes four similar procedures for downloading materials from four different sites. Each of them has one or several pages with a list of articles and links to material. In order not to spend a lot of time on formalizing and unifying the procedure, a basic script was written, on the basis of which a custom script was developed for each site, taking into account the peculiarities of the structure of the list of materials and the composition of HTML pages. So, parsing materials on Internettrading.net, where HTML was apparently generated manually, required a lot of additional checks and page parsing scenarios, while those formed by CMS Drupal (“Old Pigeon House and His Friends”) and Bitrix (“Computerra-online”, archives Computers' paper) pages contained a minimum of features.
In the future, I will refer to the details of the historically most recent script for parsing the portal of the Old Blueberry.
The list of articles is displayed in the “ Photographer ” section . Here is the name, link to the article and synopsis. The list is divided into several pages. You can go to the next page by changing the parameter in the address bar (? Page = n) in a loop, but it seemed to me more elegant to get a link to the next page from the HTML text.
On the page of the article there is a publication date in DD format YYYY Month, its own text and an indication of the source in the signature.
To work with various data types, two objects were created: MaterialList (object) - a list of articles (contains the method of parsing a separate page of the _ParseList listand the method for obtaining the URL of the next page _GetNextPage , stores a list of materials and their identifiers) and Material (object) - the article itself (contains a method for generating an identifier based on the _InitID date , a parsing method for the _ParsePage page , a method for determining the publication source _GetSection, and article attributes such as date publications, type of material, etc.)
The functions of working with elements of the document tree are additionally defined:
And the get_month_by_name (month) function , which returns the month number by its name to parse the date.
The main code (the main () procedure ) contains loading the configuration from the file, going through the pages of the list of materials with loading the contents into memory and further saving both the list (in CSV format) and the texts of the articles (in HTML, the file name is written to based on material identifier).
The configuration file stores the URL of the start page of the content list, all XPath paths for the content pages and article list, file names, and the directory path for saving articles.
In this part I will discuss the main points of the code that somehow caused difficulties or smoking manuals.
To simplify the debugging of the paths inside the documents and to facilitate the reading of the code, all XPath have been moved to a separate configuration file. The configobj library was quite suitable for working with the configuration file . The configuration file has the following structure: The nesting of subsections can be arbitrary, comments on sections and specific variables are allowed. An example of working with a configuration file: Downloading an html page is implemented using the urllib library . Using lxml, we transform the document into a tree and fix the relative links: When parsing the list of publications, we need to loop through all the elements of the list. The method is suitable for this.
lxml.html.HtmlElement.findall (path) . For example: Now is the time to make a comment about the FirePath plugin and its use for building XPath. Indeed, as already written on Habré, FirePath gives paths that differ from paths in lxml. Slightly, but there is a difference. Pretty soon, these differences were identified and later used FirePath with amendments, for example, replace the tbody tag with * (the most common problem). At the same time, paths adjusted in this way can be checked in FirePath, which greatly speeds up the process. While it returns a list of items, there is a find (path) method to get a single item. For example: The find and findall methods work only with simple paths that do not contain logical expressions in conditions, for example:
In order to use more complex conditions, for example, of a view, you
will need the xpath (path) method, which returns a list of elements. Here is a sample code that cleans selected elements from the tree (I still don’t understand how this magic works, but the elements are really deleted from the tree): This fragment also uses the lxml.html.tostring method, which saves the tree (without any extra elements!) to a string in the specified encoding. In conclusion, I will give two examples of working with the re expression library re . The first example implements a date parsing in the format “DD YYYY Month”: The re.split function is used (regexp, string, start, options)
, which forms a list of line items separated by a specific mask (in this case, by space). The re.U option allows you to work with strings containing Russian characters in Unicode. The zfill (n) function finishes the string with zeros on the left to the specified number of characters.
The second example shows how to use regular expressions to find a substring. The above example shows the code for the function _GetSection (item, path) , to which a subtree is passed that contains an indication of the publication source, for example, “First published in the Business Journal”. Notice the regex snippet ? P
The parser source code is uploaded to Google Docs . To protect the site of the Old Pigeon from the stream of parsers, I post only the code, without the configuration file with links and paths.
The result of applying the technology was an archive of articles from four sites on the hard drive and lists of all publications of Dovecats. Lists were manually uploaded to the Google Docs spreadsheet, and articles from the archive were also manually transferred for editing to Google documents.
The plans for solving problems:
PS Many thanks to all for the comments, support and constructive criticism. I hope that most of the comments will be useful to me in the future after careful study.
To write a script, basic knowledge of the Python language, programming skills and code debugging are sufficient.
The article provides explanations on the use of libraries on the example of compiling a list of publications S.M. Golubitsky, a link to a working script.
Preface or some lyrics
I can hardly be mistaken in saying that many hawkers are familiar with the indefatigable work of Sergei Golubitsky . For nearly 15 years of near-computer journalism, Namerek published 433 articles on the mountain in the paperless Komputer, prematurely rested in the Bose, and more than 300 Dovecats on the Computerra-online portal. And this is not counting the analytical research about the heroes of foreign gesheft in the Business Journal, the opening of the curtain on the secrets of creativity in the “Home Computer”, articles in the “Russian Journal”, “D`” and so on. and so on. Those interested in completeness reviews of life-creation will be found at the above links.
Last year, the author’s project “ Old Pigeon House and His Friends ” began work”, Which was conceived (and became) in particular by a constantly updated archive of publications of the author himself and a platform for conducting cultural-povidlian discussions. As a person who is not indifferent to the topics of network life, social mythology and self-development, masterfully revealed by the author, as well as eager for quality leisure reading, I once became a regular get-together at the Dovecote. To the best of my ability I try not only to keep the project in sight, but also somehow participate in its development.
As I was working in the field of proofreading of articles transferred to the archive from the Computerra-online portal, the first thing I decided to do was make an inventory of all the Pigeons.
Formulation of the problem
So, the task, on the example of which we will consider parsing sites in Python, consisted of the following:
- Make a list of all Dovecats hosted on Computerre-online. The list should include the title of the article, date of publication, information about the contents of the article (text only, pictures, videos), Synopsis, link to the source.
- Supplement the list with materials published in paper Computerra, find duplicates.
- Supplement the list with materials from the archive of the website Internettrading.net
- Download a list of articles already published on the “Old Pigeon” portal
- Download articles to a local disk for further processing, if possible automatically clearing the text of unnecessary elements.
Toolkit selection
In terms of the programming language, my choice immediately and unequivocally fell on Python. Not only because I studied it at my leisure a few years ago (and then used the Active Python shell for some time as an advanced calculator), but also for the abundance of libraries, source code examples and the ease of writing and debugging scripts. Not least, they were also interested in the prospects for the further use of the acquired skills to solve immediate problems: integration with the Google Docs API, word processing automation, etc.
Solving a very specific task practically from scratch, the toolkit was selected so as to spend minimal time reading documentation, comparing libraries, and, ultimately, implementing. On the other hand, the solution should have the versatility sufficient for easy adaptation to other sites and similar tasks. Perhaps some tools and libraries are imperfect, but they allowed to ultimately bring the plan to the end.
So, the choice of tools began to determine the appropriate version of Python. I initially tried using Python 3.2, but in the process of experimenting, I stopped at Python 2.7 , because some examples on the “troika” did not go.
To simplify the installation of additional libraries and packages, I used setuptools - a tool for downloading, building and installing packages.
Additionally libraries were installed:
- urllib - receiving HTML-pages of sites;
- lxml - a library for parsing XML and HTML code;
- configobj is a library for reading configuration files.
As improvised means were used:
- Notepad ++ is a text editor with syntax highlighting:
- FireBug - FireFox browser plugin that allows you to view the source code of HTML pages
- FirePath - FireFox browser plugin for XPath analysis and testing:
- Built-in Python GUI for debugging code.
The articles and discussions on Habré provided invaluable help:
- Approaches to Retrieving Data from Web Resources
- Examples of xpath html requests
- “LXML” or how to parse HTML with ease
- What library do you parse sites
- Easy website parsing with Beautiful Soup
As well as manuals, examples and documentation:
- Immersion in Python 3 (Pilgrim) / XML
- Package lxml. API reference
- XPath examples in the MSDN library
- The ElementTree XML API in Pyton documentation
- The lxml.etree Tutorial
And of course, the Python Programming Language book
Solution Overview
The task includes four similar procedures for downloading materials from four different sites. Each of them has one or several pages with a list of articles and links to material. In order not to spend a lot of time on formalizing and unifying the procedure, a basic script was written, on the basis of which a custom script was developed for each site, taking into account the peculiarities of the structure of the list of materials and the composition of HTML pages. So, parsing materials on Internettrading.net, where HTML was apparently generated manually, required a lot of additional checks and page parsing scenarios, while those formed by CMS Drupal (“Old Pigeon House and His Friends”) and Bitrix (“Computerra-online”, archives Computers' paper) pages contained a minimum of features.
In the future, I will refer to the details of the historically most recent script for parsing the portal of the Old Blueberry.
The list of articles is displayed in the “ Photographer ” section . Here is the name, link to the article and synopsis. The list is divided into several pages. You can go to the next page by changing the parameter in the address bar (? Page = n) in a loop, but it seemed to me more elegant to get a link to the next page from the HTML text.
On the page of the article there is a publication date in DD format YYYY Month, its own text and an indication of the source in the signature.
To work with various data types, two objects were created: MaterialList (object) - a list of articles (contains the method of parsing a separate page of the _ParseList listand the method for obtaining the URL of the next page _GetNextPage , stores a list of materials and their identifiers) and Material (object) - the article itself (contains a method for generating an identifier based on the _InitID date , a parsing method for the _ParsePage page , a method for determining the publication source _GetSection, and article attributes such as date publications, type of material, etc.)
The functions of working with elements of the document tree are additionally defined:
- get_text (item, path) - getting the text of the element along the path path in the document item
- get_value (item) - get node text in item document
- get_value_path (item, path) - get the node text in the item document along the path
- get_attr_path (item, path, attr) - get the attribute of the element along the path path in the item document
And the get_month_by_name (month) function , which returns the month number by its name to parse the date.
The main code (the main () procedure ) contains loading the configuration from the file, going through the pages of the list of materials with loading the contents into memory and further saving both the list (in CSV format) and the texts of the articles (in HTML, the file name is written to based on material identifier).
The configuration file stores the URL of the start page of the content list, all XPath paths for the content pages and article list, file names, and the directory path for saving articles.
Implementation details
In this part I will discuss the main points of the code that somehow caused difficulties or smoking manuals.
To simplify the debugging of the paths inside the documents and to facilitate the reading of the code, all XPath have been moved to a separate configuration file. The configobj library was quite suitable for working with the configuration file . The configuration file has the following structure: The nesting of subsections can be arbitrary, comments on sections and specific variables are allowed. An example of working with a configuration file: Downloading an html page is implemented using the urllib library . Using lxml, we transform the document into a tree and fix the relative links: When parsing the list of publications, we need to loop through all the elements of the list. The method is suitable for this.
# Comment
[ Section_1 ]
# Comment
variable_1 = value_1
# Comment
variable_2 = value_2
[[Subsection_1]]
variable_3 = value_3
[[Subsection_2]]
[ Section_2 ]from configobj import ConfigObj
# Загрузить файл конфигурации
cfg = ConfigObj('sgolub-list.ini')
# Получить значение параметра url из секции sgolub
url = cfg['sgolub']['url']
import urllib
from lxml.html import fromstring
# Загрузка html-документа в строку
html = urllib.urlopen(url).read();
# Преобразование документа к типу lxml.html.HtmlElement
page = fromstring(html)
# Преобразование относительных ссылок внутри документа в абсолютные
page.make_links_absolute(url)
lxml.html.HtmlElement.findall (path) . For example: Now is the time to make a comment about the FirePath plugin and its use for building XPath. Indeed, as already written on Habré, FirePath gives paths that differ from paths in lxml. Slightly, but there is a difference. Pretty soon, these differences were identified and later used FirePath with amendments, for example, replace the tbody tag with * (the most common problem). At the same time, paths adjusted in this way can be checked in FirePath, which greatly speeds up the process. While it returns a list of items, there is a find (path) method to get a single item. For example: The find and findall methods work only with simple paths that do not contain logical expressions in conditions, for example:
for item in page.findall(path):
url = get_attr_path(item,cfg['sgolub']['list']['xpath_link'],'href')
page.findall(path)content = page.find(cfg['sgolub']['doc']['xpath_content'])xpath_blocks = './/*[@id='main-region']/div/div/div/table/*/tr/td'
xpath_nextpage = './/*[@id='main-region']/div/div/div/ul/li[@class="pager-next"]/a[@href]'
In order to use more complex conditions, for example, of a view, you
xpath_purifytext = './/*[@id="fin" or @class="info"]'will need the xpath (path) method, which returns a list of elements. Here is a sample code that cleans selected elements from the tree (I still don’t understand how this magic works, but the elements are really deleted from the tree): This fragment also uses the lxml.html.tostring method, which saves the tree (without any extra elements!) to a string in the specified encoding. In conclusion, I will give two examples of working with the re expression library re . The first example implements a date parsing in the format “DD YYYY Month”: The re.split function is used (regexp, string, start, options)
from lxml.html import tostring
for item in page.xpath(cfg['computerra']['doc']['xpath_purifytext']):
item.drop_tree()
text=tostring(page,encoding='cp1251')
import re
import datetime
# content имеет тип lxml.html.HtmlElement
# и является частью страницы, содержащей непосредственно статью
datestr=get_text(content,cfg['sgolub']['doc']['xpath_date'])
if len(datestr)>0:
datesplit=re.split('\s+',datestr,0,re.U)
self.id = self._InitID(list,datesplit[2].zfill(4)+str(get_month_by_name(datesplit[1])).zfill(2)+datesplit[0].zfill(2))
self.date = datetime.date(int(datesplit[2]),get_month_by_name(datesplit[1]),int(datesplit[0]))
else:
self.id = self._InitID(list,list.lastid[0:8])
self.date = datetime.date(1970,1,1)
, which forms a list of line items separated by a specific mask (in this case, by space). The re.U option allows you to work with strings containing Russian characters in Unicode. The zfill (n) function finishes the string with zeros on the left to the specified number of characters.
The second example shows how to use regular expressions to find a substring. The above example shows the code for the function _GetSection (item, path) , to which a subtree is passed that contains an indication of the publication source, for example, “First published in the Business Journal”. Notice the regex snippet ? P
def _GetSection(item, path):
# рекомендуется компилировать регулярные выражения
reinfo = re.compile(r'.*«(?P.*)».*',re.LOCALE)
for info in item.xpath(path):
src=get_value_path(info,'.').strip('\n').strip().encode('cp1251')
if src.startswith('Впервые опубликовано'):
parser = self.reinfo.search(src)
if parser is not None:
if parser.group('gsource')=='Бизнес-журнале':
return 'Бизнес-журнал'
else:
return parser.group('gsource')
break
return ''
The parser source code is uploaded to Google Docs . To protect the site of the Old Pigeon from the stream of parsers, I post only the code, without the configuration file with links and paths.
Conclusion
The result of applying the technology was an archive of articles from four sites on the hard drive and lists of all publications of Dovecats. Lists were manually uploaded to the Google Docs spreadsheet, and articles from the archive were also manually transferred for editing to Google documents.
The plans for solving problems:
- Writing a service that automatically tracks new posts
- Integration with the Google Docs API to automatically list new posts
- Converting archived articles from HTML to XML format with automatic correction of some errors and uploading to Google Docs
PS Many thanks to all for the comments, support and constructive criticism. I hope that most of the comments will be useful to me in the future after careful study.
