October 19, 2015 at 13:17Big data, beeline and cococo
A couple of days ago, accidentally entering the Habr without an ad block, I saw a banner: "Beeline, be a man - solve the shaitan problem." Challenge sounded interesting, determine the age by a set of parameters such as region, tariff plan, etc.
It should be noted that I have no experience in solving such a plan of tasks. All the ideas that I had were built from review articles that were readable diagonally and on laboratory work at the institute. Within the framework of that lab, a neural network was sawn up and trained on a dolphin to determine the letters of the alphabet from b / w pictures of size 64x64.
I have long been eager to plunge into the bigdata, and here the case turned up. I began to look for an entry point to this technology. He quickly turned on statues, tutors, and all kinds of examples that tried to chew on basic things and demonstrate some tools to work with all this. I had to plunge into the python.
Not very upset, all the python is not R. And with serious intentions and great fervor he deflated the trial of JetBrains PyCharm. Digging through a few examples, I approximately understood that I was daring to finally take up the beeline.
Anticipating how I will superimpose my social picture of the world on the data provided, trying to understand who and how many years I started downloading the task. After downloading the data, I felt strongly deceived. Instead of the promised tariffs and other nishtyakov, I saw a set of disconnected columns 1-61 and the values in the table in the form of hashes and some kind of crazy numbers. Hi, harsh cyberpunkism.
Having developed a solution for such a statement of the problem, it becomes completely unclear, but what will it really do? To recommend lace panties to Japanese chan or to pass mass court sentences in China. Discarding reflection and focusing on the fact that this is a harmless competition, I started picking data. The first thing I did randomly scattered the 3 most popular groups and uploaded the result, which was 27.03%, ok, this is where we will start.
It quickly turned out that the task goes far beyond any tutorial and is not so easy to solve. At the same time, pycharm fails and did not give the promised auto-sets because of this, at every step I had to crawl into the docks of the Pitonwe libs, decorated in the style of early 00, which also did not bring joy. The graphics drawn by pylab looked even more lousy.
But the last straw was the realization of how much I spend time coding, some nonsense, although I just need one schedule! And at this moment of pain and discomfort from all these tools, refusing to believe that everything is so bad, for some reason, I remembered the machine learning tab of azure, although I never opened it.
I open ML and they immediately offer me a tutor - I agree. It turns out that they are not vidos or tooltips with black masks, but naturally in the environment they create an example, at the same time telling and showing what for what and how. Everything is extremely simple, there are a bunch of modules divided into types: sources, data transformers, algorithms, training, evaluation, etc. Of course, I had to read what modules are available, what they are for and what parameters are responsible for what, at the same time I got into the theory. But, how is everything done humanly here? After studying the docks, it took 5 minutes to concoct everything and get the first real result.
Having chosen the most accurate algorithm, I decided to use the parameter selection module. The captain's module after sorting through all the options reported that the more, the better. Without hesitation, he increased the basic parameters by 100 times and left the whole thing to learn. The most inconvenient in this process is the lack of any progress of the bar or at least some estimate and in fact it remains only to guess when the process will end. Several times there were thoughts of stopping, but I endured until the end and it took 13 hours.
As a result, I got:
This is only 0.9% less than the top25, taking into account the fact that in the top25 the spread is 0.6% and to the bottom it increases, then in the top40, I think I got it. I think this is an excellent result for a person who yesterday did not know anything in this area.
Speaking of the material side, all this pampering in the ML studio cost me 24.38 Hours in RUB1,219.04.
It should be noted that I have no experience in solving such a plan of tasks. All the ideas that I had were built from review articles that were readable diagonally and on laboratory work at the institute. Within the framework of that lab, a neural network was sawn up and trained on a dolphin to determine the letters of the alphabet from b / w pictures of size 64x64.
I have long been eager to plunge into the bigdata, and here the case turned up. I began to look for an entry point to this technology. He quickly turned on statues, tutors, and all kinds of examples that tried to chew on basic things and demonstrate some tools to work with all this. I had to plunge into the python.
Not very upset, all the python is not R. And with serious intentions and great fervor he deflated the trial of JetBrains PyCharm. Digging through a few examples, I approximately understood that I was daring to finally take up the beeline.
Anticipating how I will superimpose my social picture of the world on the data provided, trying to understand who and how many years I started downloading the task. After downloading the data, I felt strongly deceived. Instead of the promised tariffs and other nishtyakov, I saw a set of disconnected columns 1-61 and the values in the table in the form of hashes and some kind of crazy numbers. Hi, harsh cyberpunkism.
Having developed a solution for such a statement of the problem, it becomes completely unclear, but what will it really do? To recommend lace panties to Japanese chan or to pass mass court sentences in China. Discarding reflection and focusing on the fact that this is a harmless competition, I started picking data. The first thing I did randomly scattered the 3 most popular groups and uploaded the result, which was 27.03%, ok, this is where we will start.
It quickly turned out that the task goes far beyond any tutorial and is not so easy to solve. At the same time, pycharm fails and did not give the promised auto-sets because of this, at every step I had to crawl into the docks of the Pitonwe libs, decorated in the style of early 00, which also did not bring joy. The graphics drawn by pylab looked even more lousy.
But the last straw was the realization of how much I spend time coding, some nonsense, although I just need one schedule! And at this moment of pain and discomfort from all these tools, refusing to believe that everything is so bad, for some reason, I remembered the machine learning tab of azure, although I never opened it.
I open ML and they immediately offer me a tutor - I agree. It turns out that they are not vidos or tooltips with black masks, but naturally in the environment they create an example, at the same time telling and showing what for what and how. Everything is extremely simple, there are a bunch of modules divided into types: sources, data transformers, algorithms, training, evaluation, etc. Of course, I had to read what modules are available, what they are for and what parameters are responsible for what, at the same time I got into the theory. But, how is everything done humanly here? After studying the docks, it took 5 minutes to concoct everything and get the first real result.
Having chosen the most accurate algorithm, I decided to use the parameter selection module. The captain's module after sorting through all the options reported that the more, the better. Without hesitation, he increased the basic parameters by 100 times and left the whole thing to learn. The most inconvenient in this process is the lack of any progress of the bar or at least some estimate and in fact it remains only to guess when the process will end. Several times there were thoughts of stopping, but I endured until the end and it took 13 hours.
As a result, I got:
This is only 0.9% less than the top25, taking into account the fact that in the top25 the spread is 0.6% and to the bottom it increases, then in the top40, I think I got it. I think this is an excellent result for a person who yesterday did not know anything in this area.
Speaking of the material side, all this pampering in the ML studio cost me 24.38 Hours in RUB1,219.04.
