Hackathon and winter science school for deep learning and question-answer systems

    Today, machines easily “bind two words” ( 1 , 2 ), but so far they are not able to guarantee a dialogue on common topics. However, tomorrow you will ask them to write a resume correctly and choose the best chess section for your children near your home. Want to understand in more detail how scientists from Facebook, Google and others work in this direction? Come listen to them.


    From January 31 to February 5, the second international hackathon for deep learning and machine intelligence DeepHack.Q & A will be held at the Moscow Institute of Physics and Technology with scientific support from the Laboratory of Neural Systems and Deep Learning. At the Hackathon School of Science, world leading experts from Google Brain, Facebook AI Research, OpenAI, Skoltech, University of Oxford, Courant Institute of Mathematical Sciences in NYU will give a series of lectures on deep learning and its application to natural language processing tasks. Most of them are remote, but Rob Fergus and Phil Blunsom are expected to attend in person.

    About the topic. Hackathon will be dedicated to The Allen AI Science Challenge. Participants are required to develop a program that can independently learn to answer questions at the 8th grade level of an American school. For this, the participants were given a training (2,500 questions) and validation (8,132 questions) question sets in a csv file format with 4 possible answers. The correct answers are known for the test suite. The validation set is necessary in order to determine the level of accuracy of the answers of your system and, accordingly, to rank the declared solutions according to this criterion. The sets contain questions on the main subjects of the school curriculum: physics, biology, geography, etc.

    The statement of the problem looks so general that it seems that it is impossible to approach its solution without having in-depth expertise in the field of natural language processing. However, this is not quite true. It is possible to apply already developed neural network methods in several lines of code and get the result in 32% accuracy.

    This is exactly what the guys from the 5vision team did (by the way, the winners of the summer hackathon) and published their decision on the Kaggle and Github forums . You will find installation instructions here . If you suddenly have an irresistible desire to use Linux, but you don’t have it at hand, you can register for free at koding.com(or many more where) and run everything there. Now I would like to dwell in more detail on what this decision does.

    It is based on one of the implementations of the representation of words in a multidimensional vector space - GloVe (Global Vector for Word Representation), where words that are similar in meaning are mapped to those that are close in the Euclidean metric of the vector. The better-known implementation of word2vec has already been covered on Habré ( 1 , 2 , 3 , etc.) (and one of the authors of this implementation, Tomas Mikolov, will give lectures on the hackathon).

    Applying GloVe ( glove_predict.py ) to the question is as follows:
    1. Question preprocessing:
      • we throw out everything except the uppercase and lowercase letters of the English alphabet and a space.
      • we throw out the so-called “stop” words (words that practically do not affect the meaning of the sentence).
    2. We set the zero vector representation of the question q = 0.
    3. We loop through all the remaining words of the question and add to q the vector representation of each word.
    4. We carry out similar calculations of the vector representation of all four answers.
    5. We choose the answer whose distance from the vector q is the smallest.

    This implementation provides 32% accuracy on a validation set.

    5vision has another more “classic” implementation ( ck12_wiki_predict.py ) based on the use of the TF-IDF measure . It works like this:
    1. We parse the keywords on the main topics from the site www.ck12.org (for example, the topic “Physics”: http://www.ck12.org/physics/ ).
    2. Download documents from Wikipedia for the collected keywords.
    3. We calculate the measures TF and IDF for this collection of documents and words.
    4. For each question, select the most relevant article from the collection.
    5. For each answer, we consider its relevance to this article and select the one that gives the maximum value.

    This implementation provides 36% accuracy. To present the scale of the results, it is worth noting that currently the first place in the ranking answers correctly 56% of the questions.

    The competition has a number of features ( here you can find a review-squeeze from the competition forum) - for example, the final solution should work without access to the Internet. In general, reading the competition forum can provide a lot of useful information.

    Schedule of the event is the same as in the past DeepHack'e.

    Sunday: gathering participants, organizational meeting, team building, setting up the environment, introductory lecture and the first night run of the calculation.

    Monday-Friday: discussion of intermediate results, programming of new solutions developed in a team, lectures, launch of calculations.

    Saturday: summing up, rewarding the winners.

    All participants will also be provided with accommodation and computing resources . But you need to bring your laptop with you.

    In the near future we plan to publish a review of neural network architectures in the field of natural language processing, from which ideas can be drawn to improve current solutions.

    Only registered users can participate in the survey. Please come in.

    How interested are you in natural language processing?

    • 14.4% I do natural language processing professionally. 14
    • 14.4% I handle natural language as a hobby. 14
    • 59.7% I would like to deal with natural language processing. 58
    • 11.3% I do not plan to deal with natural language processing. eleven

    Also popular now: