Today we want to talk about some of the latest updates of the Sherlock system [this is a high-performance cluster at Stanford University - approx. per.], which greatly speeds up the listing of files in directories with a large number of entries.

Unlike regular articles, this is more of an insider report on how regular work on Sherlock is going on in order to maintain it in the best possible way for our users. We hope to publish more such articles in the future.

Listing many files takes time

It all started with a tech support question from the user. He reported a problem that execution lstook several minutes in a directory with more than 15,000 entries in $SCRATCH[directory for temporary files - approx. trans.].

Thousands of files in one directory usually create difficulties for the file system and this is definitely not recommended. The user knew this and admitted that it was not good, but mentioned that listing on his laptop was 1000 times faster than in Sherlock. Of course, it hit us. Therefore, we looked deeper.

Because ls looks beautiful

We looked at what the lslisting actually does and why the process takes so long. Most modern distributions lsdefault to as ls --color=autobecause everyone likes the color scheme.

But beautiful colors have a price: for each file I lsneed to get information about the type of file, its resolutions, flags, advanced attributes and the like, in order to select the appropriate color.

One of the simple solutions to the problem is to disable color in ls altogether, but imagine the indignation of users. In no case can you take the color output, we are not monsters.

Therefore, we looked deeper. lspaints records through an environment variable LS_COLORS, which sets dircolors(1)based on the configuration filedir_colors(5). Yes, the executable reads the configuration file to create an environment variable, which ls then uses (and if you do not know about the door (do) files , then dir_colors will work , no matter what).

We will understand in more detail

To determine which of the colorization schemes causes a slowdown, we created an experimental environment:

$ mkdir $SCRATCH/dont
$ touch $SCRATCH/dont/{1..10000} # don't try this at home!
$ time ls --color=always $SCRATCH/dont | wc -l
10000
real    0m12.758s
user    0m0.104s
sys     0m0.699s

12.7 seconds for 10,000 files, not very good.

By the way, you need a flag --color=always: although it accesses ls --color=auto, it lsdetects when it is not connected to the terminal (for example, via a channel or with output redirection) and disables coloring if the value is set auto. Smart guy.

So what takes so long? We looked with strace:

$ strace -c ls --color=always $SCRATCH/dont | wc -l
10000
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 44.21    0.186617          19     10000           lstat
 42.60    0.179807          18     10000     10000 getxattr
 12.19    0.051438           5     10000           capget
  0.71    0.003002          38        80           getdents
  0.07    0.000305          10        30           mmap
  0.05    0.000217          12        18           mprotect
  0.03    0.000135          14        10           read
  0.03    0.000123          11        11           open
  0.02    0.000082           6        14           close
[...]

Wow: 10,000 calls lstat(), 10,000 calls getxattr()(which all fail because in our environment there are no attributes that ls is looking for), 10,000 calls capget().

Surely this can be optimized.

Capabilities attribute Nope

Following the advice of a bug 10 years ago , we tried to disable the checking of the capabilities attribute :

$ eval $(dircolors -b | sed s/ca=[^:]*:/ca=:/)
$ time strace -c ls --color=always $SCRATCH/dont | wc -l
10000
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 98.95    0.423443          42     10000           lstat
  0.78    0.003353          42        80           getdents
  0.04    0.000188          10        18           mprotect
  0.04    0.000181           6        30           mmap
  0.02    0.000085           9        10           read
  0.02    0.000084          28         3           mremap
  0.02    0.000077           7        11           open
  0.02    0.000066           5        14           close
[...]
------ ----------- ----------- --------- --------- ----------------
100.00    0.427920                 10221         6 total
real    0m8.160s
user    0m0.115s
sys     0m0.961s

Wow, acceleration up to 8 seconds! We got rid of all these expensive calls getxattr(), and the calls capget()also disappeared, great.

But still these annoying challenges remained lstat(), though ...

How many flowers do you need?

Therefore, we examined in more detail LS_COLORS.

First, they simply turned off this variable:

$ echo $ LS_COLORS
rs = 0: di = 01; 34: ln = 01; 36: mh = 00: pi = 40; 33: so = 01; 35: do = 01; 35: bd = 40; 33; 01: cd = 40; 33; 01: or = 40; 31; 01: su = 37; 41: sg = 30; 43: ca =: tw = 30; 42: ow = 34; 42: st = 37; 44: ex = 01; 32 : *. tar = 01; 31: *. tgz = 01; 31: *. arc = 01; 31: *. arj = 01; 31: *. taz = 01; 31: *. lha = 01; 31: * .lz4 = 01; 31: *. lzh = 01; 31: *. lzma = 01; 31: *. tlz = 01; 31: *. txz = 01; 31: *. tzo = 01; 31: *. t7z = 01; 31: *. Zip = 01; 31: *. Z = 01; 31: *. Z = 01; 31: *. Dz = 01; 31: *. Gz = 01; 31: *. Lrz = 01 ; 31: *. Lz = 01; 31: *. Lzo = 01; 31: *. Xz = 01; 31: *. Bz2 = 01; 31: *. Bz = 01; 31: *. Tbz = 01; 31 : *. tbz2 = 01; 31: *. tz = 01; 31: *. deb = 01; 31: *. rpm = 01; 31: *. jar = 01; 31: *. war = 01; 31: * .ear = 01; 31: *. sar = 01; 31: *. rar = 01; 31: *. alz = 01; 31: *. ace = 01; 31: *. zoo = 01; 31: *. cpio = 01; 31: *. 7z = 01; 31: *. Rz = 01; 31: *. Cab = 01; 31: *. Jpg = 01; 35: *. Jpeg = 01; 35: *. Gif = 01 ; 35: *. Bmp = 01; 35: *. Pbm = 01; 35: *. Pgm = 01; 35: *. Ppm = 01; 35: *. Tga = 01; 35: *. Xbm = 01; 35 : *. xpm = 01; 35: *. tif = 01; 35: *. tiff = 01; 35: *. png = 01; 35: *. svg = 01; 35: *. svgz = 01; 35: * .mng = 01; 35: *. pcx = 01; 35: *. mov = 01; 35: *.mpg = 01; 35: *. mpeg = 01; 35: *. m2v = 01; 35: *. mkv = 01; 35: *. webm = 01; 35: *. ogm = 01; 35: *. mp4 = 01; 35: *. M4v = 01; 35: *. Mp4v = 01; 35: *. Vob = 01; 35: *. Qt = 01; 35: *. Nuv = 01; 35: *. Wmv = 01; 35: *. Asf = 01; 35: *. Rm = 01; 35: *. Rmvb = 01; 35: *. Flc = 01; 35: *. Avi = 01; 35: *. Fli = 01; 35: * .flv = 01; 35: *. gl = 01; 35: *. dl = 01; 35: *. xcf = 01; 35: *. xwd = 01; 35: *. yuv = 01; 35: *. cgm = 01; 35: *. emf = 01; 35: *. axv = 01; 35: *. anx = 01; 35: *. ogv = 01; 35: *. ogx = 01; 35: *. aac = 00; 36: *. Au = 00; 36: *. Flac = 00; 36: *. Mid = 00; 36: *. Midi = 00; 36: *. Mka = 00; 36: *. Mp3 = 00; 36: *. Mpc = 00; 36: *. Ogg = 00; 36: *. Ra = 00; 36: *. Wav = 00; 36: *. Axa = 00; 36: *. Oga = 00; 36: * .spx = 00; 36: *. xspf = 00; 36:emf = 01; 35: *. axv = 01; 35: *. anx = 01; 35: *. ogv = 01; 35: *. ogx = 01; 35: *. aac = 00; 36: *. au = 00; 36: *. Flac = 00; 36: *. Mid = 00; 36: *. Midi = 00; 36: *. Mka = 00; 36: *. Mp3 = 00; 36: *. Mpc = 00; 36: *. Ogg = 00; 36: *. Ra = 00; 36: *. Wav = 00; 36: *. Axa = 00; 36: *. Oga = 00; 36: *. Spx = 00; 36: * .xspf = 00; 36:emf = 01; 35: *. axv = 01; 35: *. anx = 01; 35: *. ogv = 01; 35: *. ogx = 01; 35: *. aac = 00; 36: *. au = 00; 36: *. Flac = 00; 36: *. Mid = 00; 36: *. Midi = 00; 36: *. Mka = 00; 36: *. Mp3 = 00; 36: *. Mpc = 00; 36: *. Ogg = 00; 36: *. Ra = 00; 36: *. Wav = 00; 36: *. Axa = 00; 36: *. Oga = 00; 36: *. Spx = 00; 36: * .xspf = 00; 36:
$ unset LS_COLORS
$ echo $ LS_COLORS
$ time ls --color = always $ SCRATCH / dont | wc -l
10,000
real 0m13.037s
user 0m0.077s
sys 0m1.092s

What!?! Still 13 seconds?

It turns out that when an environment variable is LS_COLORSnot defined or if only one of its elements is missing =color:, it uses the built-in database by default and still uses colors. Therefore, if you want to disable coloring for a certain type of file, you need to override it with =:or 00in the file DIR_COLORS.

After a lot of trial and error, we narrowed our search to this:

EXEC 00
SETUID 00
SETGID 00
CAPABILITY 00

what is written as

LS_COLORS='ex=00:su=00:sg=00:ca=00:'

This means: do not color the files either by the atrubut capabilities , or by bits setuid/setgid, or by the executable flag .

Speed ​​up ls

And if you do not do any of these checks, then the calls lstat()disappear, and now a completely different thing:

$ export LS_COLORS='ex=00:su=00:sg=00:ca=00:'
$ time strace -c ls --color=always $SCRATCH/dont | wc -l
10000
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 63.02    0.002865          36        80           getdents
  8.10    0.000368          12        30           mmap
  5.72    0.000260          14        18           mprotect
  3.72    0.000169          15        11           open
  2.79    0.000127          13        10           read
[...]
------ ----------- ----------- --------- --------- ----------------
100.00    0.004546                   221         6 total
real    0m0.337s
user    0m0.032s
sys     0m0.029s

0.3 seconds on a list of 10,000 files, a record.

Configure Sherlock

From 13 seconds with default settings to 0.3 seconds with a small setting LS_COLORSmeans 40-fold acceleration due to the absence of setuid / setgidand colored executable files. Not such a big loss.

Of course, now this is configured in Sherlock for each user.

But if you want to return the coloring, you can just go back to the default settings:

$ unset LS_COLORS

But then on directories with a large number of files, be sure to make coffee while it works ls.