Bash scripts, part 4: input and output

Bash scripts: start of
Bash scripts, part 2: cycles
Bash scripts, part 3: parameters and command line keys
Bash scripts, part 4: input and output
Bash scripts, part 5: signals, background tasks, managing
Bash scripts scripts, part 6: functions and library development
Bash scripts, part 7: sed and word processing
Bash scripts, part 8: awk data processing language
Bash scripts, part 9: regular expressions
Bash scripts, part 10: practical examples
Bash scripts, part 11: expect and automation of interactive utilities

Last time, in the third part of this bash script series, we talked about command line options and keys. Our topic today is input, output, and everything connected with it.

You are already familiar with two methods of working with what command line scripts output:

• Display output data on the screen.
• Redirecting output to a file.

Sometimes you need to show something on the screen and write something to a file, so you need to understand how input and output are handled in Linux, which means learning how to send the results of scripts to where you need to. Let's start by talking about standard file descriptors.

Standard File Descriptors

Everything on Linux is files, including input and output. The operating system identifies files using descriptors.

Each process is allowed to have up to nine open file descriptors. The bash shell reserves the first three descriptors with identifiers 0, 1, and 2. This is what they mean.

• 0, STDIN — standard input stream.
• 1, STDOUT — standard output stream.
• 2, STDERR — standard error stream.

These three special descriptors handle the input and output of data in a script.
You need to properly understand the standard threads. They can be compared with the foundation on which scripts interact with the outside world. Consider the details about them.

Stdin

STDIN — this is a standard shell input stream. For the terminal, the standard input is the keyboard. When scripts use the input redirection character - <, Linux replaces the standard input file descriptor with the one specified in the command. The system reads the file and processes the data as if they were entered from the keyboard.

Many bash commands accept input from STDINif the command line does not specify the file from which to take data. For example, this is true for the team cat.

When you enter a command caton the command line without specifying parameters, it takes input from STDIN. After you enter the next line, it catsimply displays it on the screen.

STDOUT

STDOUT — standard shell output stream. By default, this is the screen. Most bash commands output data to STDOUT, which causes them to appear in the console. Data can be redirected to a file by attaching it to its contents, a command is used for this >>.

So, we have a certain data file to which we can add other data using this command:

pwd >> myfile

What will pwdbe output will be added to the file myfile, while the data already in it will not go anywhere.


Redirecting the output of a command to a file

So far, everything is fine, but what if you try to do something like the one shown below, referring to a nonexistent file xfile, thinking of all this so that myfilean error message appears in the file .

ls –l xfile > myfile

After executing this command, we will see error messages on the screen.


Attempting to access a nonexistent file

An attempt to access a nonexistent file generates an error, but the shell did not redirect error messages to the file, displaying them on the screen. But we wanted error messages to go to the file. What to do? The answer is simple - use the third standard descriptor.

Stderr

STDERR represents a standard shell error stream. By default, this descriptor indicates the same thing that it indicates STDOUT, which is why when an error occurs, we see a message on the screen.

So, suppose you need to redirect error messages, say, to a log file, or somewhere else, instead of displaying them on the screen.

▍ Error flow redirection

As you already know, file descriptor STDERR — 2. We can redirect errors by placing this descriptor before the redirection command:

ls -l xfile 2>myfile
cat ./myfile

The error message will now go to the file myfile.


Redirecting error message to file

▍ Redirecting error and output streams

When writing command line scripts, a situation may arise when you need to organize both redirection of error messages and redirection of standard output. In order to achieve this, you need to use the redirection commands for the corresponding descriptors with the files, where errors and standard output should go:

ls –l myfile xfile anotherfile 2> errorcontent 1> correctcontent

Redirecting Errors and Standard Output

The shell will redirect what the command lsusually sends STDOUTto the file correctcontentdue to the design 1>. Error messages that would get in STDERRappear in the file errorcontentdue to the redirect command 2>.

If necessary, and STDERR, and STDOUTcan be redirected to the same file using the command &>:


Redirecting STDERR and STDOUT to the same file

After executing the command, what is intended for STDERRand STDOUTappears in the file content.

Script output redirection

There are two methods for redirecting output in command line scripts:

• Temporary redirection, or redirection of output of one line.
• Permanent redirection, or redirection of all output in a script or in some part of it.

▍ Temporary output redirection

In the script, you can redirect the output of a single line to STDERR. In order to do this, it is enough to use the redirection command, specifying the descriptor STDERR, while the ampersand ( &) character must be placed before the descriptor number :

#!/bin/bash
echo "This is an error" >&2
echo "This is normal output"

If you run the script, both lines will appear on the screen, since, as you already know, by default errors are output to the same place as normal data.


Temporary redirection

Run the script so that the output STDERRfalls into the file.

./myscript 2> myfile

As you can see, now the usual output is made to the console, and error messages get to the file.


Error messages are written to a file.

▍ Permanent output redirection

If the script needs to redirect a lot of data displayed on the screen, adding the appropriate command to each call is echoinconvenient. Instead, you can set the output to be redirected to a specific descriptor for the duration of the script by using the command exec:

#!/bin/bash
exec 1>outfile
echo "This is a test of redirecting all output"
echo "from a shell script to another file."
echo "without having to redirect every line"

Run the script.


Redirecting all output to a file

If you look at the file specified in the output redirection command, it turns out that everything that was output by the commands echofell into this file.

The command execcan be used not only at the beginning of the script, but also in other places:

#!/bin/bash
exec 2>myerror
echo "This is the start of the script"
echo "now redirecting all output to another location"
exec 1>myfile
echo "This should go to the myfile file"
echo "and this should go to the myerror file" >&2

This is what happens after running the script and viewing the files to which we redirected the output.


Redirecting output to different files

First, the command execsets the redirection of output from STDERRto a file myerror. Then the output of several commands is echosent to STDOUTand displayed on the screen. After that, the command execsets the sending of what gets into the STDOUTfile myfile, and finally, we use the redirection command STDERRin the command echo, which leads to writing the corresponding line to the file. myerror.

Having mastered this, you can redirect the output to where you need to. Now let's talk about input redirection.

Script input redirection

To redirect input, you can use the same technique that we used to redirect output. For example, the command execallows you to make a data source for STDINsome file:

exec 0< myfile

This command tells the shell that the source should be a file myfile, not a regular one STDIN. Let's look at input redirection in action:

#!/bin/bash
exec 0< testfile
count=1
while read line
do
echo "Line #$count: $line"
count=$(( $count + 1 ))
done

This is what appears on the screen after running the script.


Redirecting input

In one of the previous articles, you learned how to use the command readto read data entered by the user from the keyboard. If you redirect input, making the data source a file, then the command read, when you try to read data from STDIN, will read it from the file, and not from the keyboard.

Some Linux administrators use this approach to read and subsequently process log files.

Create your own output redirection

By redirecting input and output in scripts, you are not limited to three standard file descriptors. As already mentioned, you can have up to nine open descriptors. The remaining six, with numbers from 3 to 8, can be used to redirect input or output. Any of them can be assigned to a file and used in the script code.

You can assign a descriptor for data output using the command exec:

#!/bin/bash
exec 3>myfile
echo "This should display on the screen"
echo "and this should be stored in the file" >&3
echo "And this should be back on the screen"

After the script is launched, part of the output will be displayed on the screen, and part - in the file with the descriptor 3.


Redirecting output using a native descriptor

Creating file descriptors for data entry

You can redirect the input in the script in the same way as the output. Save STDINin a different descriptor before redirecting data entry.

After reading the file, you can restore STDINand use it as usual:

#!/bin/bash
exec 6<&0
exec 0< myfile
count=1
while read line
do
echo "Line #$count: $line"
count=$(( $count + 1 ))
done
exec 0<&6
read -p "Are you done now? " answer
case $answer in
y) echo "Goodbye";;
n) echo "Sorry, this is the end.";;
esac

Let's test the script.


Input redirection

In this example, file descriptor 6 was used to store the link to STDIN. Then the input was redirected, the STDINfile became the data source for . After that, the input for the command readcame from the redirected STDIN, that is, from the file.

After reading the file, we return STDINto its original state, redirecting it to the descriptor 6. Now, in order to verify that everything works correctly, the script asks the user a question, waits for keyboard input, and processes what is entered.

Closing File Descriptors

The shell automatically closes the file descriptors after the script is finished. However, in some cases it is necessary to close the descriptors manually before the script finishes work. In order to close the handle, it must be redirected to &-. It looks like this:

#!/bin/bash
exec 3> myfile
echo "This is a test line of data" >&3
exec 3>&-
echo "This won't work" >&3

After executing the script, we will receive an error message.


Attempting to access a private file descriptor The

thing is that we tried to access a nonexistent descriptor.

Be careful when closing file descriptors in scripts. If you sent data to a file, then closed the handle, then opened it again, the shell will replace the existing file with a new one. That is, everything that was written to this file earlier will be lost.

Retrieving information about open descriptors

In order to get a list of all open descriptors in Linux, you can use the command lsof. On many distributions, like Fedora, the utility lsofis located at /usr/sbin. This command is very useful since it displays information about each descriptor open on the system. This includes what processes running in the background have discovered, and what is open by users who are logged in.

This team has many keys, consider the most important.

• -pAllows you to specify the IDprocess.
• -d Allows you to specify the number of the descriptor about which you want to get information.

In order to find out the PIDcurrent process, you can use a special environment variable $$in which the shell writes the current one PID.

The key is -aused to perform a logical operation Иon the results returned through the use of two other keys:

lsof -a -p $$ -d 0,1,2

Displays information about open handles

file type associated with STDIN, STDOUTand STDERR — CHR (character mode, character mode). Since they all point to a terminal, the file name matches the name of the device assigned to the terminal. All three standard files are readable and writable.

Let's look at a command call lsoffrom a script in which, in addition to the standard ones, other descriptors are opened:

#!/bin/bash
exec 3> myfile1
exec 6> myfile2
exec 7< myfile3
lsof -a -p $$ -d 0,1,2,3,6,7

Here's what happens if you run this script.


Viewing the file descriptor open script

script opened two descriptors for output ( 3and 6) and one - input ( 7). The paths to the files used to configure the descriptors are also shown here.

Output suppression

Sometimes it is necessary to make sure that the commands in the script, which, for example, can be executed as a background process, do not display anything on the screen. To do this, you can redirect the output to /dev/null. This is a kind of black hole.

Here, for example, how to suppress the output of error messages:

ls -al badfile anotherfile 2> /dev/null

The same approach is used if, for example, you need to clear the file without deleting it:

cat /dev/null > myfile

Summary

Today you learned how input and output work in command line scripts. Now you know how to handle file descriptors, create, view and close them, you know about redirecting input, output and error streams. All this is very important in the development of bash scripts.

Next time we’ll talk about Linux signals, how to handle them in scripts, about running scheduled tasks, and about background tasks.

Dear readers! This material provides the basics of working with input, output, and error streams. We are sure that among you there are professionals who can tell about all this that comes only with experience. If so - pass the word to you.