Notebook Market Analysis Using Python

butaji August 31, 2009 at 09:41

Notebook Market Analysis Using Python

Introduction

In this article, I will talk about the state of today's Russian laptop market. We will conduct all analytics using python code. I think it will be useful both for those who are looking for a laptop, and for those who want to practice writing in python.

Let's start

For analysis, we need a data set, unfortunately I could not find web services at Russian online laptop stores, so I had to download the price list of one of them (I will not name it) and get prices and main parameters out of it ( in my opinion, these are: processor frequency, monitor diagonal, RAM size, hard disk size and the amount of memory on the video card). Then I did some analysis on the following issues:

The average cost of a laptop
Average hardware on laptops
The most expensive / cheapest laptop configuration
Which of the configuration parameters most affects its price
Price prediction of a specified configuration
Graph of distribution of configurations and prices

Lets code

I saved the price list that I managed to get in CSV format, to work with it you need to connect the csv module:

import csv
import re
import random We

also connect the module for working with random numbers and regular expressions, which we will need later.

Next, create a method for reading and receiving laptops:

def get_notebooks ():
    reader = csv.reader (open ('data.csv'), delimiter = ';', quotechar = '|')
    return filter ( lambda x: x! = None, map (create_notebook, reader))

everything is simple here, we read on the data.csv data file and filter by the result of the create_notebook function, because not all positions in the price are laptops, but by the way, it is:

def create_notebook (raw):
    try :
        notebook = Notebook ()
        notebook.vendor = raw [0] .split ('') [0]
        notebook.model = raw [ 0] .split ('') [1]
        notebook.cpu = getFloat (r "(\ d +) \, (\ d +) \ s \ Г", raw [0] .split ('/') [0])
        notebook.monitor = getFloat (r "(\ d +) \. (\ d +) \ ''", raw [0] .split ('/') [1])
        notebook.ram = getInt (r "(\ d +) \ Mb ", raw [0] .split ('/') [2])
        notebook.hdd = getInt (r" (\ d +) Gb ", raw [0] .split ('/'

        notebook.price = getInt (r "(\ d +) \ s \ rub.", raw [1])
        return notebook
    except Exception , e:
        return None

As you can see, I decided not to pay attention to the vendor, model and processor type (here, of course, not everything is so simple, but nonetheless), and also - in this method my custom helper functions are present:

def getFloat (regex, raw):
    m = re.search (regex, raw) .groups ()
    return float (m [0] + '.' + m [1])

def getInt (regex, raw):
    m = re.search (regex, raw) .groups ()
    return int (m [0])

I want to note that writing for python is best done in the style of data sets, rather than OOP structures, due to the fact that the language is more suited to this style, however, to bring some order in our domain area (laptops), I introduced a class like you may have noticed above (notebook = Notebook ())

class Notebook:
   pass

Great, now we have a structure in memory and it is ready for analysis ( 2005 different configurations and their cost ), so let's start:

Average laptop cost:

def get_avg_price () :
    print sum ([n.price for n in get_notebooks ()]) / len (get_notebooks ()) We execute the

code and see that 1K $, as the standard for computers tera still valid:

>> get_avg_price ()
34574

Average hardware parameters on laptops

def get_avg_parameters ():
    print “cpu {0}”. format (sum ([n.cpu for n in get_notebooks ()]) / len (get_notebooks ()))
    print “ monitor {0} ". format (sum ([n.monitor for n in get_notebooks ()]) / len (get_notebooks ()))
    print " ram {0} ". format (sum ([n.ram for n in get_notebooks ()]) / len (get_notebooks ()))
    print “hdd {0}”. format (sum ([n.hdd for n in get_notebooks ()]) / len (get_notebooks ()))
    print "Video {0}". Format (sum ([n.video for n in get_notebooks ()]) / len (get_notebooks ()))

Yes, and we have an averaged configuration in our hands:

>> get_avg_parameters ()
cpu 2.0460798005
monitor 14.6333167082
ram 2448
hdd 243
video 289

The most expensive / cheapest laptop configuration:

The functions are identical, except for the functions min / max

def get_max_priced_notebook ():
    maxprice = max ([n.price for n in get_notebooks ()])
    maxconfig = filter ( lambda x : x.price == maxprice, get_notebooks ()) [0]
    print "cpu {0}". format (maxconfig.cpu)
    print "monitor {0}". format (maxconfig.monitor)
    print "ram {0}". format (maxconfig.ram)
    print "hdd {0}". format (maxconfig.hdd)
    print "video {0}". format (maxconfig.video)
    print "price {0}". format (maxconfig.price)

>> get_max_priced_notebook ()
cpu 2.26
monitor 18.4
ram 4096
hdd 500
video 1024
price 181660

>> get_min_priced_notebook ()
cpu 1.6
monitor 8.9
ram 512
hdd 8
video 128
price 8090

Which of the configuration parameters most affects its price

It would be very interesting to find out which of the configuration parameters we pay the most money for. Having estimated, I suggested that most likely this is the diagonal of the monitor and the frequency of the processor, well, I think that it’s worth checking this.

To begin with, our set of configuration parameters should be slightly modified. Due to the fact that the units of various parameters are different in their order, we need to bring them to the same denominator, i.e. normalize them. So     here we go :

def normalized_set_of_notebooks ():
notebooks = get_notebooks ()
    cpu = max ([n.cpu for n in notebooks])
    monitor = max ([n.monitor for n in notebooks])
    ram = max ([n.ram for n in notebooks])
    hdd = max ([n.hdd for n in notebooks])
    video = max ([n.video for n in notebooks])
    rows = map ( lambda n: [ n.cpu / cpu, n.monitor / monitor, float (n.ram) / ram, float (n.hdd) / hdd, float (n.video) / video, n.price], notebooks)
    return rows

In this function, I find the maximum values for each of the parameters, after which I form the resulting list of laptops, in which each of the parameters is presented as a coefficient (its value will range from 0 to 1), showing the ratio of its parameter to the maximum value in the set, to For example, a memory in 2048Mb will give the configuration a coefficient of ram = 0.5 (2048/4056).

The contribution of each of the parameters we will consider in rubles, for clarity, we will store these weights in the set:

#cpu, monitor, ram, hdd, video
koes = [0, 0, 0, 0, 0]

I propose to calculate these coefficients for each configuration, and then determine the average value of all coefficients, which will give us averaged data on the weight of each of the configuration items.

def analyze_params (parameters):
    koeshistory = []
    # our notebooks
    notes = normalized_set_of_notebooks ()
    for i in range (len (notes)):
        koes = [0, 0, 0, 0, 0]
        # set the coefficients
        set_koes (notes [i] , koes)
        # save the coefficient history
        koeshistory.extend (koes)
        # show the progress
        if (i% 100 == 0):
            print i
            print koes

How will we set the coefficients for each configuration item? My way is this:

we need to randomly increase or decrease the value of one of the coefficients
then analyze whether we are close to the price for the configuration, when multiplying the parameter vector by the coefficient vector (I recall that in our case it is rubles)
if the approximation has taken place, we repeat this action, if not, then cancel it
repeat this order to the extent that we do not get closer to our price with the accuracy we set

Here is the implementation of this algorithm:

def set_koes (note, koes, error = 500):
    price = get_price (note, koes)
    lasterror = abs (note [5] - price)
    while (lasterror> error):
        k = random.randint (0 , 4)
        # change the coefficient
        inc = (random.random () * 2 - 1) * (error * (1 - error / lasterror))
        koes [k] + = inc
        # do not let the coefficient become less than zero
        if (koes [k ] <0): koes [k] = 0
        # we get the price when taking into account the coefficients
        price = get_price (note, koes)
        # we get the current error
        curerror = abs (note [5] - price)
        # we check whether we are close to the price indicated in the price list
        if (lasterror <curerror):
            koes [k] - = inc
        else :
            lasterror = curerror

inc - a variable responsible for increasing / decreasing the coefficient, the method of calculating it is explained by the fact that this value should be the larger, the greater the difference in error, for quick and more accurate approximation to the desired result.

Multiplication of vectors to get the price is as follows:

def get_price (note, koes):
    return sum ([note [i] * koes [i] for i in range (5)])

It's time to analyze:

>> analyze_params ()
cpu , monitor, ram, hdd, video

[15455.60675667684, 20980.560483811361, 12782.535270304281, 17819.904629585861, 14677.889529808042]

Данный набор мы получили, благодаря усреднению коэффициентов, полученных для каждой из конфигураций:

def get_avg_koes(koeshistory):
    koes = [0, 0, 0, 0, 0]
    for row in koeshistory:
        for i in range(5):
            koes[i] += koeshistory[i]
    for i in range(5):
        koes[i] /= len(koeshistory)
    return koes

Итак, у нас получился желаемый набор, что же мы можем сказать из этих цифр, а можем мы составить рейтинг параметров:

Диагональ монитора
Объем жесткого диска
Частота процессора
Объем видео-карточки
RAM size

I would like to note that this is far from an ideal option, and you may get different results, however, my assumption that the processor frequency and display diagonal are the most important parameters in the configuration was partially confirmed.

Predicting the price of the specified configuration It

would be cool if we had such a rich data set to be able to predict the price of a given configuration. This is what we will do.

First, we will convert our collection of laptops to a list:

def get_notebooks_list ():
    return map ( lambda n: [n.cpu, n.monitor, n.ram, n.hdd, n.video, n.price], get_notebooks ())

Next, we need a function that can determine the distance between two vectors, a good option I see is the Euclidean distance function:

def euclidean (v1, v2):
    d = 0.0
    for i in range (len (v1)):
        d + = (v1 [i] - v2 [i]) ** 2;
    return math.sqrt (d)

The root of the sum of the squares of the differences quite clearly and effectively shows us how much one vector is different from another. What is this feature useful for us? It's simple, when we get a vector with the parameters we are interested in, we will go over the entire collection of our set and find the nearest neighbor, and we already know its value, great! Here's how we do it:

def getdistances (data, vec1):
    distancelist = []
    for i in range (len (data)):
        vec2 = data [i]
        distancelist.append ((euclidean (vec1, vec2), i))
    distancelist.sort ()
    return distancelist

Next, you can complicate the task a bit, as well as the accuracy of the data provided. To do this, we introduce a function using the classification by the method of k weighted nearest neighbors :

weighted nearest neighbors is a metric classification algorithm based on assessing the similarity of objects. The classified object belongs to the class to which the objects of the training sample closest to it belong .

Well, take the average value among a number of nearest neighbors, which will negate the influence of vendor prices or configuration specificity:

def knnestimate (data, vec1, k = 3):
    dlist = getdistances (data, vec1)
    avg = 0.0
    for i in range (k):
        idx = dlist [i] [1]
        avg + = data [idx] [5]
    avg / = k
    return avg

* the last 3 algorithms are taken from Segeran Toby 's book “Programming the collective mind”

And what do we get:

> > knnestimate (get_notebooks_list (), [2.4, 17, 3062, 250, 512])
31521.0

>> knnestimate (get_notebooks_list (), [2.0, 15, 2048, 160, 256])
27259.0
>> knnestimate (get_notebooks_list (), [2.0, 15, 2048, 160, 128])
20848.0 The

prices are market and this is quite enough, although we absolutely do not take into account in this implementation, for example, processor frequency and monitor diagonal (for this we need to add to the function of comparing the vectors of their weight, which we calculated in the previous paragraph) The

graph of the distribution of configurations and prices

I would like to embrace the entire distribution pattern, i.e. draw the distribution of configurations and prices in the market. Ok, let's do it.

First you need to put the matplotlib library . Next, connect it to our project:

from pylab import *

We also need to create two data sets, for the abscissa and ordinate:

def power_of_notebooks_config ():
    return map ( lambda x: x [0] * x [1] * x [2] * x [3] * x [4], normalized_set_of_notebooks ())
def config_prices ():
    return map ( lambda x: x [5], normalized_set_of_notebooks ())

And the function in which we plot the distribution:

def draw_market ():
    plot (config_prices (), power_of_notebooks_config (), 'bo', linewidth = 1.0)

    xlabel ('price (Rub)' )
    ylabel ('config_power')
    title ('Russian Notebooks Market')
    grid (True)
    show ()

And what do we get:

In conclusion

So, we managed to conduct a small analysis of the Russian laptop market, as well as play a little with python.

The source code of the project is available at:

http://code.google.com/p/runm/source/checkout

I apologize for the slightly important syntax highlighting, my engine ( pygments ) did not want to be perceived by the hubr.

Tags: