Common Bash programming errors
- Transfer
The quality of the scripts used to automate and optimize the system is the key to its stability and longevity, and also saves the time and nerves of the administrator of this system. Despite the seeming primitiveness of bash as a programming language, it is full of pitfalls and tricky trends that can significantly spoil the mood for both the developer and the administrator.
Most of the manuals available are dedicated to how to write. I’ll tell you how to NOT write :-)
This text is a free translation of the Bash pitfalls wiki page as of December 13, 2008. Due to the wiki nature of the source, this translation may differ from the original. Since the volume of the text is too large to be published in its entirety, it will be published in parts.
One of the most common errors in bash scripts is loops like this:
This will not work if there are spaces in the name of one of the files, as the result of the substitution of the team
Quoting the entire command also does not work:
The entire output is now considered as one word, and instead of going through each of the files in the list, the cycle will be executed only once, and at the same time it
In fact, use is
Let the bash substitute the file names. Such a substitution will not lead to the division of the string into words. Each file name that matches the pattern
For more information, see paragraph 20 of the Bash FAQ .
An attentive reader should have noticed the quotation marks in the second line of the above example. This smoothly brings us to the trick number 2.
What is wrong with this team? It seems to be nothing special, unless you absolutely know for sure that the variables
But if you don’t know what kind of files you come across, or you are paranoid, or simply trying to follow a good bash programming style, then you will enclose the names of your variables in quotation marks so as not to put them into words.
Without double quotes, the script will execute the command
One workaround is to insert a double hyphen (
However, you may come across one of the systems in which such a trick does not work. Or the command you are trying to execute does not support the option
Another way is to make sure that file names always begin with the directory name (including
Even if we have a file whose name begins with "-", the template substitution mechanism ensures that the variable will contain something
In this example, quotation marks are not correctly aligned: in bash, there is no need to enclose a string literal in quotation marks; but you should definitely quote the variable if you are not sure that it does not contain spaces or wildcards.
This code is erroneous for two reasons:
1. If the variable used in the condition
will be perceived as
which will cause a "unary operator expected" error. (The operator "=" is binary, not unary, so the command
2. If the variable contains a space inside itself, it will be divided into different words before it is processed by the command
Even if you personally think that this is normal, this syntax is erroneous.
It will be correct like this:
But this option will not work if $ foo starts with
In bash, a keyword can be used to solve this problem
Inside
You may have seen code like this:
A hack
If one of the parts of the expression is a constant, you can do this:
The team
So far we are basically talking about the same thing. In the same way as with the expansion of the values of variables, the result of the command substitution is split into words and the expansion of file names (pathname expansion). Therefore, we must enclose the command in quotation marks:
What is not entirely obvious here is the sequence of quotation marks. A C programmer might suggest that the first and second quotation marks are grouped, as well as the third and fourth. However, in this case this is not so. Bash treats double quotes inside a command as the first pair, and outer quotes as the second.
In other words, the parser treats back quotes (
The same effect can be achieved using the preferred syntax
The quotes inside
Cannot be used
Please note that we swapped the constant and the variable inside
The same applies to
If the operator is
Therefore, the operators> and < cannot be used to compare numbers inside
If you want to compare two numbers, use
If you are writing for Bourne Shell (sh) and not for bash, the correct way is this:
Note that a command
Double square brackets also support this syntax:
At first glance, this code looks fine. But in fact, the variable
Each command in the pipeline is executed in a separate subshell, and changes to the variable inside the subshell do not affect the value of this variable in the parent shell instance (i.e., in the script that called this code).
In this case, the loop
There are several ways around this.
You can execute the loop in your subshell (a bit crooked, but it’s simpler and more understandable and works in sh):
To completely avoid creating a subshell, use redirection (in Bourne shell (sh), a subshell is also created for redirection, so be careful, this trick will work only in bash):
The previous method works only for files, but what if I need to process the output of a command line by line? Use process substitution:
A couple more interesting ways to solve the problem with sub-shells are discussed in the Bash FAQ # 24 .
Many people are embarrassed by the practice of putting square brackets after
However, such an opinion is a mistake! An opening square bracket (
Syntax
As you can see, there are no
Once again,
If you need to make a decision depending on the output of the command
Note that we discard the standard output
As explained in the previous paragraph,
Again the same mistake.
If you want to implement a difficult condition, here is the correct way:
Notice that here we have two commands after if, combined by the && operator. This code is equivalent to a command like this:
If the first test command returns a value
To be continued.
The first publication of this translation took place on the pages of my blog .
Most of the manuals available are dedicated to how to write. I’ll tell you how to NOT write :-)
This text is a free translation of the Bash pitfalls wiki page as of December 13, 2008. Due to the wiki nature of the source, this translation may differ from the original. Since the volume of the text is too large to be published in its entirety, it will be published in parts.
1. for i in `ls * .mp3`
One of the most common errors in bash scripts is loops like this:
for i in `ls * .mp3`; do # False! some command $ i # False! done
This will not work if there are spaces in the name of one of the files, as the result of the substitution of the team
ls *.mp3
is divided into words. Suppose we have a file in the current directory 01 - Don't Eat the Yellow Snow.mp3
. The cycle for
will be held on every word of the name of the file and $i
will take the values: "01"
, "-"
, "Don't"
, "Eat"
, "the"
, "Yellow"
, "Snow.mp3"
. Quoting the entire command also does not work:
for i in "` ls * .mp3` "; do # False! ...
The entire output is now considered as one word, and instead of going through each of the files in the list, the cycle will be executed only once, and at the same time it
i
will take on the value, which is a concatenation of all file names through a space. In fact, use is
ls
completely redundant: it is an external command that is simply not needed in this case. How is it right then? And like this:for i in * .mp3; do # Much better, but ... some command "$ i" # ... see catch # 2 done
Let the bash substitute the file names. Such a substitution will not lead to the division of the string into words. Each file name that matches the pattern
*.mp3
will be considered as one word, and a cycle will go through each file name once. For more information, see paragraph 20 of the Bash FAQ .
An attentive reader should have noticed the quotation marks in the second line of the above example. This smoothly brings us to the trick number 2.
2. cp $ file $ target
What is wrong with this team? It seems to be nothing special, unless you absolutely know for sure that the variables
$file
and $target
do not contain spaces or wildcards. But if you don’t know what kind of files you come across, or you are paranoid, or simply trying to follow a good bash programming style, then you will enclose the names of your variables in quotation marks so as not to put them into words.
cp "$ file" "$ target"
Without double quotes, the script will execute the command
cp 01 - Don't Eat the Yellow Snow.mp3 /mnt/usb
, and you will get a lot of type errors cp: cannot stat `01': No such file or directory
. If the variable values $file
either $target
contain the *,?, [..] or (..) characters used in wildmats, then if there are files matching the template, the variable values will be converted to the names of these files . Double quotes solve this problem, unless "$file"
it starts with a hyphen -
, in which case it cp
thinks that you are trying to give it another command line option. One workaround is to insert a double hyphen (
--
) between the command cp
and its arguments. A double hyphen tells cp
you to stop searching for options:cp - "$ file" "$ target"
However, you may come across one of the systems in which such a trick does not work. Or the command you are trying to execute does not support the option
--
. In this case, read on. Another way is to make sure that file names always begin with the directory name (including
./
for the current one). For instance:for i in ./*.mp3; do cp "$ i" / target ...
Even if we have a file whose name begins with "-", the template substitution mechanism ensures that the variable will contain something
./-foo.mp3
that is absolutely safe to use with cp
.3. [$ foo = "bar"]
In this example, quotation marks are not correctly aligned: in bash, there is no need to enclose a string literal in quotation marks; but you should definitely quote the variable if you are not sure that it does not contain spaces or wildcards.
This code is erroneous for two reasons:
1. If the variable used in the condition
[
does not exist or is empty, the line[$ foo = "bar"]
will be perceived as
[= "bar"]
which will cause a "unary operator expected" error. (The operator "=" is binary, not unary, so the command
[
will be shocked by this syntax) 2. If the variable contains a space inside itself, it will be divided into different words before it is processed by the command
[
:[multiple words here = "bar"]
Even if you personally think that this is normal, this syntax is erroneous.
It will be correct like this:
["$ foo" = bar] # is close!
But this option will not work if $ foo starts with
-
. In bash, a keyword can be used to solve this problem
[[
, which includes and greatly extends the old command test
(also known as [
)[[$ foo = bar]] # right!
Inside
[[
and it is ]]
no longer necessary to quote the names of variables, since variables are no longer broken into words and even empty variables are processed correctly. On the other hand, even if you put them in quotes once again, this will not hurt anything. You may have seen code like this:
[x "$ foo" = xbar] # right too!
A hack
x"$foo"
is required in code that should work in non-supporting shells [[
, because if it $foo
starts with -
, the command [
will be disoriented. If one of the parts of the expression is a constant, you can do this:
[bar = "$ foo"] # so right too!
The team
[
does not care that the expression to the right of the = sign starts with -
. She just uses this expression as a string. Only the left side requires such close attention.4. cd `dirname" $ f "`
So far we are basically talking about the same thing. In the same way as with the expansion of the values of variables, the result of the command substitution is split into words and the expansion of file names (pathname expansion). Therefore, we must enclose the command in quotation marks:
cd "` dirname "$ f" `"
What is not entirely obvious here is the sequence of quotation marks. A C programmer might suggest that the first and second quotation marks are grouped, as well as the third and fourth. However, in this case this is not so. Bash treats double quotes inside a command as the first pair, and outer quotes as the second.
In other words, the parser treats back quotes (
`
) as a level of nesting, and the quotes inside it are separated from external ones. The same effect can be achieved using the preferred syntax
$()
:cd "$ (dirname" $ f ")"
The quotes inside
$()
are grouped.5. ["$ foo" = bar && "$ bar" = foo]
Cannot be used
&&
inside the "old" team test
or its equivalent [
. The bash parser sees &&
outside the brackets and breaks your command into two, before and after &&
. Better use one of the options:[bar = "$ foo" -a foo = "$ bar"] # Right! [bar = "$ foo"] && [foo = "$ bar"] # Also right! [[$ foo = bar && $ bar = foo]] # That's right too!
Please note that we swapped the constant and the variable inside
[
- for the reasons discussed in the previous paragraph. The same applies to
||
. Use [[
either -o
or two commands [
.6. [[$ foo> 7]]
If the operator is
>
used internally [[ ]]
, it is considered as an operator for comparing strings, not numbers. In some cases, this may or may not work (and this will happen just when you least expect it). If it >
is inside [ ]
, it’s even worse: in this case, it is redirecting the output from the file descriptor with the specified number. An empty file with a name will appear in the current directory 7
, and the command test
will succeed unless the variable $foo
is empty. Therefore, the operators> and < cannot be used to compare numbers inside
[ .. ]
or [[ .. ]]
. If you want to compare two numbers, use
(( ))
:((foo> 7)) # That's right!
If you are writing for Bourne Shell (sh) and not for bash, the correct way is this:
[$ foo -gt 7] # That's right too!
Note that a command
test ... -gt ...
will throw an error if at least one of its arguments is not an integer. Therefore, it no longer matters whether the quotation marks are correctly placed: if the variable is empty, or contains spaces, or its value is not an integer, an error will occur in any case. Just check the value of the variable carefully before using it in a command test
. Double square brackets also support this syntax:
[[$ foo -gt 7]] # That's right too!
7. count = 0; grep foo bar | while read line; do ((count ++)); done; echo "number of lines: $ count"
At first glance, this code looks fine. But in fact, the variable
$count
will remain unchanged after exiting the loop, much to the surprise of the bash developer. Why it happens? Each command in the pipeline is executed in a separate subshell, and changes to the variable inside the subshell do not affect the value of this variable in the parent shell instance (i.e., in the script that called this code).
In this case, the loop
for
is part of the pipeline and runs in a separate subshell with its copy of the variable $count
, initialized by the value of the variable $count
from the parent shell: “0”. When the loop ends, the copy used in the loop is $count
discarded and the command echo
shows the unchanged initial value $count
("0").There are several ways around this.
You can execute the loop in your subshell (a bit crooked, but it’s simpler and more understandable and works in sh):
# POSIX compatible count = 0 cat / etc / passwd | ( while read line; do count = $ ((count + 1)) done echo "total number of lines: $ count" )
To completely avoid creating a subshell, use redirection (in Bourne shell (sh), a subshell is also created for redirection, so be careful, this trick will work only in bash):
# only for bash! count = 0 while read line; do count = $ (($ count + 1)) done </ etc / passwd echo "total number of lines: $ count"
The previous method works only for files, but what if I need to process the output of a command line by line? Use process substitution:
while read LINE; do echo "-> $ LINE" done <<(grep PATH / etc / profile)
A couple more interesting ways to solve the problem with sub-shells are discussed in the Bash FAQ # 24 .
8. if [grep foo myfile]
Many people are embarrassed by the practice of putting square brackets after
if
and beginners often get the false impression that they [
are part of the conditional syntax, just like the brackets in the conditional constructs of C. However, such an opinion is a mistake! An opening square bracket (
[
) is not part of the syntax, but a command that is the equivalent of a command test
, except that the closing argument must be the last argument to this command ]
. Syntax
if
:if COMMANDS then COMMANDS elif COMMANDS # optional then COMMANDS else # optional COMMANDS fi
As you can see, there are no
[
or ors in the if syntax [[
. Once again,
[
this is a command that takes arguments and returns a return code; like all normal commands, it can display error messages, but, as a rule, does not produce anything in STDOUT. if
executes the first set of commands, and depending on the return code of the last command from this set, determines whether the block of commands from the "then" section will be executed or the script will continue to execute. If you need to make a decision depending on the output of the command
grep
, you don’t need to enclose it in parentheses, brackets or braces, backticks, or any other syntax element. Just write grep
as a command after if
:if grep foo myfile> / dev / null; then ... fi
Note that we discard the standard output
grep
: we do not need the search result, we just want to know if a line is present in the file. If it grep
finds a string, it returns 0, and the condition is satisfied; otherwise (there is no line in the file) it grep
returns a value other than 0. In GNU grep, redirection >/dev/null
can be replaced with an option -q
that says grep
' y that you do not need to output anything.9. if [bar = "$ foo"]
As explained in the previous paragraph,
[
this is a team. As with any other command, bash assumes that the command is followed by a space, then the first argument, then again a space, etc. Therefore, you can’t write everything without spaces! That's right like this:if [bar = "$ foo"]
bar
, =
, "$foo"
(After the substitution, but without division into words) and ]
are the arguments of the team [
, so between each pair of arguments necessary space must be present to shell could determine where any argument begins and ends.10. if [[a = b] && [c = d]]
Again the same mistake.
[
- a command, not a syntax element between a if
condition and, moreover, not a means of grouping. You cannot take C syntax and convert it to bash syntax by simply replacing parentheses with square ones. If you want to implement a difficult condition, here is the correct way:
if [a = b] && [c = d]
Notice that here we have two commands after if, combined by the && operator. This code is equivalent to a command like this:
if test a = b && test c = d
If the first test command returns a value
false
(any nonzero number), the condition body is skipped. If it returns true
, the second condition is true; if it returns true
, then the condition body is satisfied. To be continued.
The first publication of this translation took place on the pages of my blog .