Tabs or spaces? Analysis of 400 thousand GitHub repositories, billion files, 14 TB of code

For inquisitive developers, the issue of using tabs and spaces to format code is still relevant. Can they be used interchangeably: for example, 2 spaces per tab or 4? But there is no single standard, so sometimes there is a misunderstanding between developers. In addition, various IDEs and their compilers handle tabs in their own way.

The solution to the question usually becomes an agreement on formatting rules within a project or a programming language as a whole.

A team of developers from Google explored projects in the Github repository. They analyzed code written in 14 programming languages. The purpose of the study was to identify the ratio of tabs and spaces - that is, the most popular way to format text for each language.

Implementation

For analysis, we used the already existing table [bigquery-public-data: github_repos.sample_files], in which the names of the Github repositories are written.

Recall that about two months ago, all open Github code became available in the form of BigQuery tables.

However, not all repositories were selected for analysis, but only the top 400 thousand repositories with the largest number of stars that they received for the period from January to May 2016.



Files containing code in the 14 most popular programming languages ​​were selected from this table. For this, extensions of the corresponding files were specified as parameters of the sql query - .java, .h, .js, .c, .php, .html, .cs, .json, .py, .cpp, .xml, .rb, .cc, .go.

SELECT a.id id, size, content, binary, copies, sample_repo_name , sample_path
FROM (
  SELECT id, FIRST(path) sample_path, FIRST(repo_name) sample_repo_name 
  FROM [bigquery-public-data:github_repos.sample_files] 
  WHERE REGEXP_EXTRACT(path, r'\.([^\.]*)$') IN ('java','h','js','c','php','html','cs','json','py','cpp','xml','rb','cc','go')
  GROUP BY id
) a
JOIN [bigquery-public-data:github_repos.contents] b
ON a.id = b.id

864.6s elapsed, 1.60 TB processed

The request has been running for quite some time. And this is not surprising, since it was necessary to perform the join operation of a table of 190 million rows with a table of 70 million rows. A total of 1.6 TB of data was processed. Query results are available at this address .

The table [contents] contains files without their duplicates. Below is the total number of unique files and their total size. Duplicate files were not taken into account during the analysis.



After that, it only remained to form and launch the final request.

SELECT ext, tabs, spaces, countext, LOG((spaces+1)/(tabs+1)) lratio
FROM (
  SELECT REGEXP_EXTRACT(sample_path, r'\.([^\.]*)$') ext, 
         SUM(best='tab') tabs, SUM(best='space') spaces, 
         COUNT(*) countext
  FROM (
    SELECT sample_path, sample_repo_name, IF(SUM(line=' ')>SUM(line='\t'), 'space', 'tab') WITHIN RECORD best,
           COUNT(line) WITHIN RECORD c
    FROM (
      SELECT LEFT(SPLIT(content, '\n'), 1) line, sample_path, sample_repo_name 
      FROM [fh-bigquery:github_extracts.contents_top_repos_top_langs]
      HAVING REGEXP_MATCH(line, r'[ \t]')
    )
    HAVING c>10 # at least 10 lines that start with space or tab
  )
  GROUP BY ext
)
ORDER BY countext DESC
LIMIT 100

16.0s elapsed, 133 GB processed

Analysis of each line of 133 GB of code took 16 seconds. The same BigQuery helped to achieve this speed.


Most often, tabs are found in C, and spaces are found in Java.

Although for some, the ratio of certain control characters does not matter, and the debate on this topic seems far-fetched. This does not matter for some IDEs that retain tabs as a number of spaces. There are also IDEs in which this amount can be configured manually.

Some time ago, this problem was beaten in the series "Silicon Valley." The guy and the girl did not agree on the issue of formatting. As a result, the old holivar not only led to professional misunderstandings, but also created problems in their personal relationships.