Combining R and Python: Why, When, and How? / Open Data Science Blog

Efaldgent November 8, 2018 at 14:00

Combining R and Python: Why, When, and How?

Probably, many of those who are engaged in data analysis have ever thought about whether it is possible to use R and Python at the same time. And if so, then why might this be necessary? When will it be useful and effective for projects? And how do you choose the best way to combine languages, if Google gives out about 100,500 options?

Let's try to understand these issues.

What for

First of all, it is an opportunity to take advantage of the two most popular programming languages for data analysis today. Combining the most powerful and stable R and Python libraries in some cases, you can increase the efficiency of calculations or avoid the invention of bicycles for implementing any statistical models.
Secondly, this is an increase in the speed and convenience of projects , if different people in your team (or yourself) have good knowledge of different languages. A reasonable combination of existing R and Python programming skills can help here.

Let's try to talk in more detail about the first paragraph. The summary that follows is certainly subjective, and can be supplemented. It is created on the basis of the systematization of key articles on the benefits of languages and personal experience. But the world, as we know, is changing very quickly.

Python was created by clever programmers and is a general-purpose language, only later - with the development of data science - adapted for specific tasks of data analysis. Hence the main advantages of this language follow. When analyzing data, its use is optimal for:

Web scraping and crawling (beautifulsoup, Scrapy, etc.)
Effective work with databases and applications (sqlachemy, etc.)
Implementations of classic ML algorithms (scikit-learn, pandas, numpy, scipy, etc.)
Computer Vision Tasks

The main thing in R is an extensive collection of libraries. This language, especially at the initial stage, developed mainly due to the efforts of statisticians, not programmers. The statisticians tried hard and their achievements are difficult to challenge.

If suddenly you are thinking about trying a new tasty statmodel that you recently heard about at a conference, before you sit down to write it from scratch, google it first . Your chances of success are very great! So, an undoubted advantage of R are the capabilities of advanced statistical analysis. In particular, for a number of specific areas of science and practice (econometrics, bioinformatics, etc.). In my opinion, in R at the moment, the analysis of time series is still much more developed.R package

Another key and yet undeniable advantage of R over Python is interactive graphics. The possibilities for creating and customizing dashboards and simple applications for people without JS knowledge are truly huge. Do not believe me - take the time to explore the possibilities of libraries pair from the list: htmlwidgets, flexdashboard, shiny, slidify. For example, initially, the materials for this article were compiled as an interactive presentation on slidify .

But no matter how hard the statisticians try, they are not strong in everything. They failed to achieve such high efficiency of memory management, as in Python. Rather, in R, good and fast code running on large amounts of data is quite possible. But with much more effort and self-control than in Python.

Gradually, all differences are erased, and both languages are becoming increasingly interchangeable. In Python develop visualization (a big step forward has become seaborn) and added ~~do not always work~~ econometric libraries ( puFlux, pymaclab, etc.), Into R - growing storage management efficiency and improve data processing capabilities ( data.table). Here, for example, you can see examples of basic operations with data in R and Python . So is there an advantage in combining languages for your project, it's up to you.

As for the second point on improving the speed and convenience of project implementation, here we are talking mainly about the organization of the project. For example, you have two people on a project, one of whom is bigger and stronger on R, the other on Python. Provided that you can provide code review and other controls for both languages, you can try to distribute the tasks so that each participant uses their best skills. Of course, your experience in solving specific problems in various languages is also important.

Although it should be clarified that we are talking about research projects working with data. Other criteria are important for decision making. Combining, most likely, will not be useful for the stability and scalability of the calculations. So we smoothly and move on to the question of when it is more convenient to combine languages.

When

Given the features of both languages, you can benefit from combining R and Python with:

Exploratory data analysis
Prototyping
Implementation of a project / set of tasks with a wide coverage in various scientific and practical fields

I will give examples for possible combination of languages:

A study of regional labor markets . Connecting to the official HH.ru API using Python, researching trends and dependencies ( xgboost, xgboostExplainer) + visualization (Markdown reports) using R
Application based on Bloomberg data . Connect to the official API and process data in Python ( numpy, pandas) + output the result to a dashboard or shiny application in R ( flexdashboard, htmlwidgets)
Social media data analysis . Parsing and ML stack using Python + econometrics, network analysis, visualization and website - using R
A model for predicting demand for products at individual points . Unit for forecasting demand for goods in individual locations using Python (ML algorithms) + macroeconomic forecast using R (general equilibrium models and structural var models)
Analysis of the news flow . Parsing on R ( rvest) + NLP in Python + parameterized report on R ( RMarkdown Parameterized Reports)

All examples are real projects.

I repeat that even though 2 years have passed since my first presentation on the possibilities of combining R and Python, I still do not dare to recommend combining languages in production. Unless it is almost 2 separate entities / models that are not critically tied to each other.

If there are any lucky ones who have washed down R + Python production something - please share in the comments!

how

Now directly about the chairs. Among the approaches to combining R and Python, there are three main categories:

Command line tools . Execution of scripts using the command line + intermediate storage of files on disk (filling air gap)
Interfacing R and Python . Running R and Python processes simultaneously and transferring data between them in random access memory (in-memory)
Other approaches

Let us consider in more detail each of the approaches.

Command line tools

The point is to divide the project into separate relatively independent parts, executed in R or Python, and transfer data through the disk in some format convenient for both languages.
By syntax, everything is extremely simple. Scripts are executed using the command line as follows:

- a command to execute an R or Python script using the command line
- a directory for the location of the script
- a list of arguments to enter the script

The table below shows in more detail diagrams of executing scripts from the command line and reading the passed arguments. The comments indicate the type of object into which the argument list is written.

Command	Python	R
Cmd	`python path/to/myscript.py arg1 arg2 arg3`	`Rscript path/to/myscript.R arg1 arg2 arg3`
Fetch arguments	`# list, 1st el. - file executed import sys my_args = sys.argv`	`# character vector of args myArgs <- commandArgs(trailingOnly = TRUE)`

For those interested, there are very detailed examples below.

R script from Python

First, let's create a simple R script to determine the maximum number from the list and call it max.R.

# max.R
randomvals <- rnorm(75, 5, 0.5)
par(mfrow = c(1, 2))
hist(randomvals, xlab = 'Some random numbers')
plot(randomvals, xlab = 'Some random numbers', ylab = 'value', pch = 3)

Now let's execute it in Python, using cmd and passing a list of numbers to find the maximum value.

# calling R from Python
import subprocess
# Define command and arguments
command = 'Rscript'
path2script = 'path/to your script/max.R'
# Variable number of args in a list
args = ['11', '3', '9', '42']
# Build subprocess command
cmd = [command, path2script] + args
# check_output will run the command and store to result
x = subprocess.check_output(cmd, universal_newlines=True)
print('The maximum of the numbers is:', x)

Python script from R

First, create a simple Python script to split the text string into parts and call it `splitstr.py`.

# splitstr.py
import sys
# Get the arguments passed in
string = sys.argv[1]
pattern = sys.argv[2]
# Perform the splitting
ans = string.split(pattern)
# Join the resulting list of elements into a single newline
# delimited string and print
print('\n'.join(ans))

Now let's execute it on R, using cmd and passing a text string to delete the desired pattern.

# calling Python from R
command = "python"
# Note the single + double quotes in the string (needed if paths have spaces)
path2script ='"path/to your script/splitstr.py"'
# Build up args in a vector
string = "3523462---12413415---4577678---7967956---5456439"
pattern = "---"
args = c(string, pattern)
# Add path to script as first arg
allArgs = c(path2script, args)
output = system2(command, args=allArgs, stdout=TRUE)
print(paste("The Substrings are:\n", output))

For intermediate file storage when transferring information from one script to another, you can use many formats - depending on your goals and preferences. Both languages have libraries (and not one each) for working with each of the formats.

File Formats for R and Python

Medium storage	Python	R
Flat files
csv	csv pandas	readr, data.table
json	json	jsonlite
yaml	PyYaml	yaml
Databases
SQL	sqlalchemy, pandasql, pyodbc	sqlite, RODBS, RMySQL, sqldf, dplyr
NoSQL	Pymongo	Rmongo
Feather
for data frames	feather	feather
Numpy
for numpy objects	numpy	Rcppcnpy

The classic format is, of course, flat files. Often csv is the most simple, convenient and reliable. If you want to structure or store information for a relatively long time, then probably the best choice would be storage in databases (SQL / NoSQL).

For fast transfer numpyof objects in R and back have a fast and stable library RCppCNPy. An example of its use can be found here .

There is also a formatfeatherdeveloped specifically for passing date frames between R and Python. The initial feature of the format is sharpening in R and Python, ease of processing in both languages and very fast writing and reading. The idea is great, but with the implementation, as it sometimes happens, there are nuances. The developers of the format have repeatedly noted that it is not yet suitable for long-term solutions. When updating libraries for working with the format, the whole process may break down and require significant code changes.

But at the same time, writing and reading feathers in R and Python is really fast. A test comparison of the read-write speed of a file of 10 million lines is presented in the figure below. In both cases, it is feathersignificantly faster than the key libraries for working with the classic csv format.

Comparison of speed with feather and CSV formats for R and Python

File : data frame, 10 million lines. 10 attempts for each library.

R : CSV read / write through data.tableand packets dplyr. Working with feather turned out to be the fastest, but the speed bypass is data.tablenot high. At the same time, there are certain difficulties with setting up and working with feather for R. And support is doubtful. Пакет featherLast updated a year ago.

Python : CSV read / write using pandas. The feather gain in speed turned out to be significant, there were no problems with working with the format in Python 3.5.

It is important to note here that speed can only be compared separately for R and separately for Python. Between the languages will be incorrect, because the whole test was carried out from R - for the convenience of forming the final figure.

To summarize

Benefits

A simple method and therefore often the fastest
Just see the intermediate result
The ability to read / write most formats is implemented in both languages

disadvantages

The design quickly becomes cumbersome and difficult to manage as the number of transitions between languages increases.
Significant loss in file write / read speed with increasing data volume
The need to agree in advance the scheme of interaction between languages and the format of intermediate files

Interfacing R and Python

This approach consists in directly launching one language from another and provides for internal (in-memory) transmission of information.

During the time during which people thought about combining R and Python instead of juxtaposing them, quite a lot of libraries were created. Of these, only two are successful and resistant to various parameters, including the use of the Windows operating system. But their presence already opens up new horizons, greatly facilitating the process of combining languages.

To call each language through another, three libraries are presented below - in decreasing order of quality.

R from Python

The most popular, stable and stable library is this rpy2. It works quickly, has a good description as part of the official documentationpandas , and on a separate site . The main feature is integration with pandas. The key object for transmitting information between languages is data frame. Direct support for the most popular R visualization package is also declared ggplot2. Those. write code in python, see the graph directly in the Python IDE. Theme with ggplot2, however, junk for Windows.

Lack ofrpy2perhaps one - the need to spend some time studying the tutorials. For correct work with two languages, this is necessary, since there are unobvious nuances in the syntax and mapping of types of objects during transmission. For example, when transferring a number from Python to entering R, you get not a number, but a vector from one element.

The key advantage of the library pipe, which is in second place in the table below, is speed. Implementation through pipes really speeds up the work on average (there is even an article in JSS on this subject), and the availability of support for working with pandas objects in R-Python inspires hope. But the available disadvantages reliably shift the library to second place. A key drawback is poor support for installing libraries in R via Python. If you want to use some non-standard library in R (and R is often needed just for this), then to install it and work, you need to sequentially (!!!) download all its dependencies. And for some libraries there can be about 100500 and a small cart. The second important drawback is the inconvenient work with graphics. The constructed graph can be viewed only by writing it to a file on disk. The third drawback is poor documentation. If a jamb happens just outside the boundaries of the standard set, the solution is often not found either in the documentation or on stackoverflow.

Librarypyrserveeasy to use, but also significantly limited in functionality. For example, it does not support table transfer. Updating and supporting the library by developers also leaves much to be desired. Version 0.9.1 is the last available for more than 2 years.

Libraries	Comments
rpy2	- C-level interface - direct pandas support - graphics support (+ ggplot2) - weak Windows support
pyper	- Python code - use of pipes (faster on average) - indirect pandas support - limited graphics support - poor documentation
pyrserve	- Python code - use of pipes (faster on average) - indirect pandas support - limited graphics support - poor documentation - low level of project support

Python from R

The best library at the moment is the official development of RStudio reticulate. Cons in it are not yet visible. But the advantages are enough: active support, clear and convenient documentation, output of the results of scripts and errors immediately to the console, easy transfer of objects (as in rpy2, the main object is the date frame). Regarding active support, I will give an example from personal experience: the average speed of answering a question in stackoverflow and github / issues is about 1 minute. To work, you can learn the syntax from the tutorial and then connect individual python modules and write functions. Or, execute separate pieces of code in python through functions py_run. They make it easy to execute a python script from R, passing the necessary arguments and getting an object with all the script output.

The second place in quality is in the library rPython. The key advantage is the simplicity of the syntax. Only 4 teams and you are a wizard using R and Python. The documentation also does not lag behind: everything is stated clearly, simply and easily. The main drawback is the curve implementation of the data frame transmission. In both R and Python, an additional step is required for the passed object to become a table. The second important drawback is the lack of outputting the result of the function to the console. When you run some kind of python-command from R directly from R, you will not be able to quickly understand whether it was successfully executed or not. Even in the documentation there was a good passage from the authors, that in order to see the result of python code execution, you need to go into python itself and see. Working with a package from Windows is only possible through pain, but it is possible. Useful links are in the table.

Third place at the library Rcpp. This is actually a very good option. As a rule, what is implemented in R using C ++ works stably, efficiently and quickly. But it takes time to figure it out. Therefore, it is indicated here at the end of the list.

Outside the table, you can mention RSPython . The idea was good - a single platform on both sides with a single logic and syntax - but the implementation failed. The package has not been supported since 2005. Although in general, you can start and poke the old version with a wand.

Libraries	Comments
reticulate	- good documentation - active development (since August 2016) - magic function `py_run_file("script.py")`
rPython	- data transfer via json - indirect transfer of tables - good documentation - poor Windows support - often crashes with Anaconda
Rcpp	- through C ++ ( Boost.Python and Rcpp ) - you need specific skills - a good example

For the two most popular libraries below are detailed usage examples.

Interfacing R from Python: rpy2

For simplicity, we will use the R script specified in the first examples - max.R.

from rpy2.robjects import pandas2ri # loading rpy2
from rpy2.robjects import r
pandas2ri.activate() # activating pandas module
df_iris_py = pandas2ri.ri2py(r['iris']) # from r data frame to pandas
df_iris_r = pandas2ri.py2ri(df_iris_py) # from pandas to r data frame 
plotFunc = r("""
   library(ggplot2)
   function(df){
   p <- ggplot(iris, aes(x = Sepal.Length, y = Petal.Length)) 
          + geom_point(aes(color = Species))
   print(p)
   ggsave('iris_plot.pdf', plot = p, width = 6.5, height = 5.5)
 }
""") # ggplot2 example
gr = importr('grDevices') # necessary to shut the graph off
plotFunc(df_iris_r)
gr.dev_off()

Interfacing Python from R: reticulate

For simplicity, we will use the Python script specified in the first examples - splitstr.py.

library(reticulate)
# aliasing the main module
py <- import_main()
# set parameters for Python directly from R session
py$pattern <- "---"
py$string = "3523462---12413415---4577678---7967956---5456439"
# run splitstr.py from the slide 11
result <- py_run_file('splitstr.py')
# read Python script result in R
result$ans
# [1] "3523462"  "12413415" "4577678"  "7967956"  "5456439"

To summarize

Benefits

Flexible and interactive way
Fast mutual transfer of objects within the framework of random access memory

disadvantages

Need to read tutorials!
Subtleties in comparability of objects and methods of their transfer between languages
Transferring large amounts of data can also be difficult.
Weak Windows Support
Instability between library versions

Other approaches

As a separate category, I would like to give a few more ways to combine languages: using the special functionality of laptops and individual platforms. Please note that most of the links below are practical examples.

When choosing laptops, you can use:

Классический jupyter и синтаксис R majic вместе с обсуждавшейся выше rpy2. Несложно, в целом, но нужно опять же потратить некоторое время на изучение синтаксиса. Другой вариант: установка IRKernel. Но в этом случае получится только запускать кернелы отдельно, передавая файлы через запись на диске.
R Notebook. Достаточно просто установить уже упомянутую библиотеку reticulate и далее указывать для каждой ячейки ноутбука язык, на котором вы будете творить. Удобная штука!
Apache Zeppelin. Говорят, достойно и также в использовании удобно.

There was also a wonderful Beaker Notebooks project that made it very convenient to navigate between languages in an interface similar to both Jupyter and Zeppelin. To transfer in-memory objects, the authors wrote a separate library beaker. Plus, you could immediately conveniently publish the result (to the public). But for some reason, all this is no longer there, even the old publications are deleted - the developers focused on the BeakerX project .

Among the special software that makes it possible to combine R and Python should be highlighted:

An excellent open-source H2O platform . The product requires separate libraries connectors for Python , and for R .
SAS Features
Cloudera

Extended example

At the end of the article, I would also like to analyze a detailed example of solving a small research problem in which it is convenient to use R and Python.

Suppose the task is set : to compare inflation rates and economic growth rates in countries that implement inflation targeting (or a close analogue) as a monetary policy regime since 2013 - an approximate start to the introduction of this regime in Russia.

We need to solve the problem quickly, but we remember fast loading only with Python, processing and visualization with R.

Therefore, with shaking hands we start R Notebook, write `` python '' in the upper cell and download data from the World Bank website. Data will be transmitted via CSV.

Python: loading data from World Bank website

import pandas as pd
import wbdata as wd
# define a period of time
start_year = 2013
end_year = 2017
# list of countries under inflation targeting monetary policy regime
countries = ['AM', 'AU', 'AT', 'BE', 'BG', 'BR', 'CA', 'CH', 'CL', 'CO', 'CY', 'CZ', 'DE', 'DK', 'XC', 'ES', 'EE', 'FI', 'FR', 'GB', 'GR', 'HU', 'IN', 'IE', 'IS', 'IL', 'IT', 'JM', 'JP', 'KR', 'LK', 'LT', 'LU', 'LV', 'MA', 'MD', 'MX', 'MT', 'MY', 'NL', 'NO', 'NZ', 'PK', 'PE', 'PH', 'PL', 'PT', 'RO', 'RU', 'SG', 'SK', 'SI', 'SE', 'TH', 'TR', 'US', 'ZA']
# set dictionary for wbdata
inflation = {'FP.CPI.TOTL.ZG': 'CPI_annual', 'NY.GDP.MKTP.KD.ZG': 'GDP_annual'}
# download wb data
df = wd.get_dataframe(inflation, country = countries, data_date = (pd.datetime(start_year, 1, 1), pd.datetime(end_year, 1, 1)))
print(df.head())
df.to_csv('WB_data.csv', index = True)

Next, data is preprocessed on R: the spread of inflation in different countries (min / max / mean) and the average rate of economic growth (in terms of real GDP growth). We also change the names of some countries to shorter ones - so that later it will be more convenient to render.

R: data preprocessing

library(tidyverse)
library(data.table)
library(DT)
# get df with python results
cpi <- fread('WB_data.csv')
cpi <- cpi %>% group_by(country) %>% summarize(cpi_av = mean(CPI_annual), cpi_max = max(CPI_annual),
                                               cpi_min = min(CPI_annual), gdp_av = mean(GDP_annual)) %>% ungroup
cpi <- cpi %>% mutate(country = replace(country, country %in% c('Czech Republic', 'Korea, Rep.', 'Philippines',
                                                                'Russian Federation', 'Singapore', 'Switzerland',
                                                                'Thailand', 'United Kingdom', 'United States'),
                                        c('Czech', 'Korea', 'Phil', 'Russia', 'Singap', 'Switz', 'Thai', 'UK', 'US')),
                      gdp_sign = ifelse(gdp_av > 0, 'Positive', 'Negative'),
                      gdp_sign = factor(gdp_sign, levels = c('Positive', 'Negative')),
                      country = fct_reorder(country, gdp_av),
                      gdp_av = abs(gdp_av),
                      coord = rep(ceiling(max(cpi_max)) + 2, dim(cpi)[1])
                     )
print(head(data.frame(cpi)))

Then, using a small code and R, you can create a readable and enjoyable chart that allows you to answer the original question about comparing inflation and GDP in countries that use the inflation targeting mode.

R: visualization

library(viridis)
library(scales)
ggplot(cpi, aes(country, y = cpi_av)) +
  geom_linerange(aes(x = country, y = cpi_av, ymin = cpi_min, ymax = cpi_max, colour = cpi_av), size = 1.8, alpha = 0.9) + 
  geom_point(aes(x = country, y = coord, size = gdp_av, shape = gdp_sign), alpha = 0.5) +
  scale_size_area(max_size = 8) +
  scale_colour_viridis() +
  guides(size = guide_legend(title = 'Average annual\nGDP growth, %', title.theme = element_text(size = 7, angle = 0)),
         shape = guide_legend(title = 'Sign of\nGDP growth, %', title.theme = element_text(size = 7, angle = 0)),
         colour = guide_legend(title = 'Average\nannual CPI, %', title.theme = element_text(size = 7, angle = 0))) +
  ylim(floor(min(cpi$cpi_min)) - 2, ceiling(max(cpi$cpi_max)) + 2) +
  labs(title = 'Average Inflation and GDP Rates in Inflation Targeting Countries',
       subtitle = paste0('For the period 2013-2017'),
       x = NULL, y = NULL) +
  coord_polar() +
  theme_bw() +
  theme(legend.position = 'right',
        panel.border = element_blank(),
        axis.text.x = element_text(colour = '#442D25', size = 6, angle = 21, vjust = 1))
ggsave('IT_countries_2013_2017.png', width = 11, height = 5.7)

The resulting graph in the polar coordinate system shows the scatter of inflation values from 2013 to 2017, as well as the average values of GDP (over inflation). A circle means a positive GDP growth rate, a triangle means a negative one.

The figure as a whole allows us to draw some preliminary conclusions about the success of the inflation targeting regime in Russia relative to other countries. But this is beyond the scope of this article. And if interested, I can give links to various materials on the topic.

conclusions

It is possible and necessary to combine R and Python!
Especially for research data analysis and prototyping
Pay attention to the choice of combination methods , based on the objectives of the project:
- Command line tools: simple, clear and will definitely work.
- Great way to get started!
- Attention to feather format!
Interfacing R and Python: fast, flexible, and at times interactive .
- Specific syntax - you need to read the tutorials, a more complex setup.
- Use proven and stable libraries: rpy2and reticulate.
Other approaches. Attention to laptops!

Tags:

Combining R and Python: Why, When, and How? / Open Data Science Blog