April 23, 2019 at 17:06Recognizing dates written in natural language using Python3
We at the company create a service that extracts legal facts from the correspondence of the client and customer. The service grew out of one simple idea - my regular customers decided to simplify the work of managers and create a “contract generator”. The first task - to pull the details of the client and customer into the contract, we decided easily.
A second idea came up - to look for dates in the correspondence and insert them into the terms of reference, documents automatically.
However, people rarely write dates in chats and instant messengers so that the algorithm can easily recognize them.
“We start next week”, “this friday” - if managers are relatively easy to learn how to write dates in the correct format, then you cannot make such demands on clients.
I write in python and python immediately came to the rescue NLP-library spaCy * - NER (name entity recognition) module which easily retrieved dates from correspondence with English-speaking clients. As a result, we got a bunch of relative dates: “in two weeks”, “friday”, “this friday”, “this noon”, “wensday noon”.
But how to translate these dates into objects that the service perceives (datetime object)?
A long journey began in the area of “human readable dates parsing”. Surveying the surroundings, I found only three python libraries that worked immediately and without torment: these are timefhuman **, dateparser *** and datefinder ***.
The picture shows a comparison of three libraries:
In general, it is clear that the timefhuman, not spoiled by attention, turned out to be more convenient, although with a large number of false positives, than the dateparser that most people use.
An interesting logic of developers parsing “Friday” gave the future Friday in the case of timefhuman and last Friday in the dateparser.
In general, timefhuman turned out to be more alive and was selected for further development and testing of the prototype.
* spacy.io
** github.com/alvinwan/timefhuman
*** github.com/scrapinghub/dateparser
**** github.com/akoumjian/datefinder
A second idea came up - to look for dates in the correspondence and insert them into the terms of reference, documents automatically.
However, people rarely write dates in chats and instant messengers so that the algorithm can easily recognize them.
“We start next week”, “this friday” - if managers are relatively easy to learn how to write dates in the correct format, then you cannot make such demands on clients.
I write in python and python immediately came to the rescue NLP-library spaCy * - NER (name entity recognition) module which easily retrieved dates from correspondence with English-speaking clients. As a result, we got a bunch of relative dates: “in two weeks”, “friday”, “this friday”, “this noon”, “wensday noon”.
But how to translate these dates into objects that the service perceives (datetime object)?
A long journey began in the area of “human readable dates parsing”. Surveying the surroundings, I found only three python libraries that worked immediately and without torment: these are timefhuman **, dateparser *** and datefinder ***.
The picture shows a comparison of three libraries:
In general, it is clear that the timefhuman, not spoiled by attention, turned out to be more convenient, although with a large number of false positives, than the dateparser that most people use.
An interesting logic of developers parsing “Friday” gave the future Friday in the case of timefhuman and last Friday in the dateparser.
In general, timefhuman turned out to be more alive and was selected for further development and testing of the prototype.
* spacy.io
** github.com/alvinwan/timefhuman
*** github.com/scrapinghub/dateparser
**** github.com/akoumjian/datefinder
