February 19, 2015 at 14:48Remote debugging on Linux using the GDB-gdbserver bundle
As we all know, the debugging process is such a thing, the importance of which is difficult to overestimate. Moreover, understanding the importance of such methods as LED blinking and outputting debug messages to the port, I remain of the opinion that nothing has been invented so far as step-by-step debugging. However, the step-by-step debugging task becomes less trivial when programming under Linux on embedded systems (such as rasbery pi, virt2real or industrial processor modules).
This task in Linux is intended to be solved by a standard bundle of GDB and gdbserver programs . The idea is that you write a program on your computer ( host in GDB terminology), compile it and upload it to the target device ( target) Next, run the debugged file and gdbserver on the target device ( target ) , and on to the GDB host . The interaction scheme of such debugging can be briefly represented as follows:
An executable file is the very program that we want to debug remotely. For correct operation, the program instance must be located not only on the target device, but also on the computer (host) from which debugging is in progress.
GDB is a program that directly executes the debugging process. Usually, it is part of the GCC toolchain and is located where the compiler is.
gdbserver is the reciprocal part of GDB, which runs the executable in debug mode. Since gdbserver runs on the remote side (target), it must be built for the target device using a cross-compiler. Actually, this article is devoted mainly to the assembly of gdbserver from the sources.
I have a virt2real board at my disposal, as well as a processor module based on the processor from the TI series AM335x . Below we will show the sequence of actions using the example of virt2real , however, all the same I have successfully (and importantly - similarly) done with the AM335x chip .
Note: the operating system installed on the host is Ubuntu.12.04.
We create the gdb folder in our home directory , in which we will perform all our manipulations. Inside, create a downloads subfolder :
Download the sources we need and unpack them. We need GDB itself , as well as the termcap library (it is used by GDB).
Usually collected directly in the source folder, but we will not do this in order to leave the source intact. This is useful in case you have not one, but several varieties of target platforms. Then the same downloaded sources can be used several times.
Accordingly, we start with the termcap library , because it will be needed later when building gdbserver itself . We create the builds folder in which we will collect our programs. Inside we create the v2r folder , we put all the results for the virt2real platform in it . Well, there is already a termcap folder for building the termcap library. Go to the created directory.
We indicate to the system the compiler and ranlib which we will use. In my case, this is:
Now we configure. You will need to specify 2 parameters: --host and --prefix.
If everything is done correctly and without errors, a Makefile will be created in the ~ / gdb / builds / v2r / termcap folder.
Next, we collect the library:
Create a folder in which we will build the server and go to it:
Specify where to get the termcap library:
We configure similarly to termcap. It is important to note here that we are building gdbserver (and not GDB), therefore we specify the configure file from the /gdb-7.8/gdb/gdbserver folder:
If everything is correct, a Makefile will be created. Further standard:
To test the debugging process, create a short Hello world and compile it for the target platform.
Download source package hello.cpp :
Fill the hello executable and our gdbserver onto the target board.
I upload to the / usr / folder using SCP:
Now run the second instance of the terminal and connect to the target board via ssh and go to the / usr / folder :
We run gdbserver on the target board, and with it, our hello executable (debugged) file. Then we open the debut session on the port we liked:
We return to host and run the debugged hello file using GDB
In response, you should see an invitation from GDB:
Connect to the remote gdbserver using the above port:
We get a response and an invitation to enter the following command:
At the same time, the remote side (gdbserver on target in the face of virt2real) should see the installation of the debug session:
This completes the step-by-step debugging example. I note:
For GDB to work correctly, it needs the so-called debugging symbols , which can be read from the remote OS libraries (target). Their absence is, for example, the reason for such messages:
And potentially cause other unpleasant debugging problems.
To fix the GDB problem on the host, you need to specify where to get these very libraries using the command:
If you have somewhere on your host the Linux image of your target is overloaded, then you must specify the path to the library folder.
Or preload these libraries on host:
We return to the folder with the project, run GDB again and specify the path to the libraries:
Now, messages such as:
View the list of currently used libraries as follows:
PS: this article does not claim to be scientific novelty, but is an attempt to gather in one place detailed information on the assembly and configuration of the GDB-gdbserver bundle. It is likely that for advanced linuxoids the material will seem trite and clumsy, but for those people who are more hardware, or for microcontroller programmers, switching to Linux is not so simple (I judge by myself). So, hopefully, the material will serve someone useful.
This task in Linux is intended to be solved by a standard bundle of GDB and gdbserver programs . The idea is that you write a program on your computer ( host in GDB terminology), compile it and upload it to the target device ( target) Next, run the debugged file and gdbserver on the target device ( target ) , and on to the GDB host . The interaction scheme of such debugging can be briefly represented as follows:
An executable file is the very program that we want to debug remotely. For correct operation, the program instance must be located not only on the target device, but also on the computer (host) from which debugging is in progress.
GDB is a program that directly executes the debugging process. Usually, it is part of the GCC toolchain and is located where the compiler is.
gdbserver is the reciprocal part of GDB, which runs the executable in debug mode. Since gdbserver runs on the remote side (target), it must be built for the target device using a cross-compiler. Actually, this article is devoted mainly to the assembly of gdbserver from the sources.
I have a virt2real board at my disposal, as well as a processor module based on the processor from the TI series AM335x . Below we will show the sequence of actions using the example of virt2real , however, all the same I have successfully (and importantly - similarly) done with the AM335x chip .
Note: the operating system installed on the host is Ubuntu.12.04.
Training
We create the gdb folder in our home directory , in which we will perform all our manipulations. Inside, create a downloads subfolder :
mkdir -p ~/gdb/downloads
cd ~/gdb/downloads
Download the sources we need and unpack them. We need GDB itself , as well as the termcap library (it is used by GDB).
wget ftp://ftp.gnu.org/gnu/termcap/termcap-1.3.1.tar.gz
wget ftp://ftp.gnu.org/gnu/gdb/gdb-7.8.tar.gz
tar xvzf termcap-1.3.1.tar.gz
tar xvzf gdb-7.8.tar.gz
Usually collected directly in the source folder, but we will not do this in order to leave the source intact. This is useful in case you have not one, but several varieties of target platforms. Then the same downloaded sources can be used several times.
Build the termcap library
Accordingly, we start with the termcap library , because it will be needed later when building gdbserver itself . We create the builds folder in which we will collect our programs. Inside we create the v2r folder , we put all the results for the virt2real platform in it . Well, there is already a termcap folder for building the termcap library. Go to the created directory.
mkdir -p ~/gdb/builds/v2r/termcap
cd ~/gdb/builds/v2r/termcap
We indicate to the system the compiler and ranlib which we will use. In my case, this is:
export CC=/opt/virt2real-sdk/codesourcery/arm-2013.05/bin/arm-none-linux-gnueabi-gcc
export RANLIB=/opt/virt2real-sdk/codesourcery/arm-2013.05/bin/arm-none-linux-gnueabi-ranlib
Now we configure. You will need to specify 2 parameters: --host and --prefix.
What to specify as parameters
- I admit, I still do not understand whether it matters what to specify in --host. I got the impression that no. In any case, I could not find a clear explanation of what exactly should be specified in this parameter, therefore I indicated the compiler prefix. Those. my compiler is called arm-none-linux-gnueabi-gcc, so I specified arm-none-linux-gnueabi as --host.
- As the --prefix parameter, specify the folder in which we want to put the results of the work (configuration), i.e. the folder in which we are now.
../../../downloads/termcap-1.3.1/configure --host=arm-none-linux-gnueabi --prefix=~/gdb/builds/v2r/termcap
If everything is done correctly and without errors, a Makefile will be created in the ~ / gdb / builds / v2r / termcap folder.
Check Makefile
I recommend you to play it safe, open the Makefile and see that the variables CC and RANLIB are exactly those that we indicated above. The fact is that during my first attempts to build gdbserver, I was stuck for a long time because of this point, because everything seemed to go fine, but the resulting file didn’t want to run on the target board. It turned out that for reasons unknown to me, the standard compiler persistently used in the Makefile. If you find that the compiler in the Makefile is not the same, you can simply replace it with your hands (it helped me). Although, when I repeated the whole procedure before writing the article, I did not have such a problem.
Next, we collect the library:
make
make install
Collecting gdbserver
Create a folder in which we will build the server and go to it:
mkdir -p ~/gdb/builds/v2r/gdbserver
cd ~/gdb/builds/v2r/gdbserver
Specify where to get the termcap library:
export LDFLAGS="-static -L~/gdb/builds/v2r/termcap/lib"
export CFLAGS="-g -O2 -I~/gdb/builds/v2r/termcap/include"
We configure similarly to termcap. It is important to note here that we are building gdbserver (and not GDB), therefore we specify the configure file from the /gdb-7.8/gdb/gdbserver folder:
../../../downloads/gdb-7.8/gdb/gdbserver/configure --host=arm-none-linux-gnueabi --prefix=/home/den1s/gdb/builds/v2r/gdbserver --disable-werror # &&
Note
No matter how I tried to never indicate absolute paths, so that the instruction could be used without any adjustment for myself, I did not succeed. Because the configure file swears at an attempt to specify a path starting with ~ as --prefix. Therefore, here you will need to enter your username.
If everything is correct, a Makefile will be created. Further standard:
make
make install
Try
To test the debugging process, create a short Hello world and compile it for the target platform.
mkdir -p ~/virt2real/hello_test/
cd ~/virt2real/hello_test/
/opt/virt2real-sdk/codesourcery/arm-2013.05/bin/arm-none-linux-gnueabi-g++ hello.cpp -O0 -g -o hello
Download source package hello.cpp :
#include
int main(int argc, char* argv[])
{
std::cout << "Hello,\n" << std::endl;
std::cout << "debugged\n" << std::endl;
std::cout << "World!!!" << std::endl;
return 0;
}
Fill the hello executable and our gdbserver onto the target board.
I upload to the / usr / folder using SCP:
scp ~/gdb/builds/v2r/gdbserver/bin/gdbserver root@192.168.3.1:/usr/
scp ~/virt2real/hello_test/hello root@192.168.3.1:/usr/
Now run the second instance of the terminal and connect to the target board via ssh and go to the / usr / folder :
ssh root@192.168.3.1
cd /usr/
We run gdbserver on the target board, and with it, our hello executable (debugged) file. Then we open the debut session on the port we liked:
[root@virt2real usr]# ./gdbserver :123 hello
Process hello created; pid = 5680;
listening on port 123
We return to host and run the debugged hello file using GDB
/opt/virt2real-sdk/codesourcery/arm-2013.05/bin/arm-none-linux-gnueabi-gdb hello
Headline
In order to be able not to specify the full path to GDB all the time, you can write an environment variable:
export PATH=$PATH:/opt/virt2real-sdk/codesourcery/arm-2013.05/bin/
In response, you should see an invitation from GDB:
GNU gdb (Sourcery CodeBench Lite 2013.05-24) 7.4.50.20120716-cvs
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-pc-linux-gnu --target=arm-none-linux-gnueabi".
For bug reporting instructions, please see:
...
Reading symbols from /home/den1s/virt2real/hello_test/hello...done.
(gdb)
Connect to the remote gdbserver using the above port:
(gdb) target remote 192.168.3.1:123
We get a response and an invitation to enter the following command:
Remote debugging using 192.168.3.1:123
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
0x36fd7e30 in ?? ()
(gdb)
At the same time, the remote side (gdbserver on target in the face of virt2real) should see the installation of the debug session:
Remote debugging from host 192.168.3.10 # где 192.168.3.10 - это IP нашего host'а с которого ведется отладка
| A comment | host | target |
|---|---|---|
| We put breakpoint on main | | |
| Go to Breakpoint | | |
| Go to the next line | | |
| Go to the next line | | |
| Go to the next line | | |
| Go to the next line (which, however, is not) | | |
| Run the program to the end, which leads to its completion | | |
This completes the step-by-step debugging example. I note:
- for a complete list of commands, you can use the help command, as well as read a book on debugging using GDB (see the link at the end of the article)
- “Manual” debugging using GDB is very tedious, so I recommend using, for example, Eclipse for these purposes. Unfortunately, the description of such debugging within the framework of this article would increase it to indecent dimensions. At the end of the article there is a link to a very good English-language description of this topic.
Sysroot installation
For GDB to work correctly, it needs the so-called debugging symbols , which can be read from the remote OS libraries (target). Their absence is, for example, the reason for such messages:
(gdb) next
Cannot access memory at address 0x0
Cannot access memory at address 0x0
0x36d1f040 in ?? ()
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?
And potentially cause other unpleasant debugging problems.
To fix the GDB problem on the host, you need to specify where to get these very libraries using the command:
set sysroot [Directory]
If you have somewhere on your host the Linux image of your target is overloaded, then you must specify the path to the library folder.
Or preload these libraries on host:
mkdir -p ~/virt2real/sysroot
cd ~/virt2real/sysroot
scp -r root@192.168.3.1:/lib lib
We return to the folder with the project, run GDB again and specify the path to the libraries:
cd ~/virt2real/hello_test/
arm-none-linux-gnueabi-gdb hello
(gdb) set sysroot ~/virt2real/hello_test/
Now, messages such as:
Reading symbols from ~/virt2real/sysroot/lib/libgcc_s.s...done.
View the list of currently used libraries as follows:
(gdb) info sharedlibrary
From To Syms Read Shared Object Library
0x36fd77a0 0x36ff24d0 Yes (*) ~/virt2real/sysroot/lib/ld-linux.so.3
No /usr/lib/libstdc++.so.6
0x36e70e00 0x36ed107c Yes (*) ~/virt2real/sysroot/lib/libm.so.6
0x36e51028 0x36e63154 Yes (*) ~/virt2real/sysroot/lib/libgcc_s.so.1
0x36d1cb00 0x36e11ae8 Yes (*) ~/virt2real/sysroot/lib/libc.so.6
(*): Shared library is missing debugging information.
Bibliography"
- Debugging methods in Embedded Linux. In Russian.
- Gdbserver assembly description. In Russian.
- Gdbserver assembly description. In English.
- Install sysroot. In English.
- R. Stolman. Debugging with GDB. In English.
- Configuring GDB in Code Composer Studio (the same Eclipse). In English.
- Instruction manual1) cd ~/gdb/downloads
2) wget ftp.gnu.org/gnu/termcap/termcap-1.3.1.tar.gz
3) wget ftp.gnu.org/gnu/gdb/gdb-7.8.tar.gz
4) tar xvzf termcap-1.3.1.tar.gz
5) tar xvzf gdb-7.8.tar.gz
6) mkdir -p ~/gdb/builds/v2r/termcap
7) cd ~/gdb/builds/v2r/termcap
8) export CC=/opt/virt2real-sdk/codesourcery/arm-2013.05/bin/arm-none-linux-gnueabi-gcc
9) export RANLIB=/opt/virt2real-sdk/codesourcery/arm-2013.05/bin/arm-none-linux-gnueabi-ranlib
10) ../../../downloads/termcap-1.3.1/configure --host=arm-none-linux-gnueabi --prefix=~/gdb/builds/v2r/termcap
11) make
12) make install
13) mkdir -p ~/gdb/builds/v2r/gdbserver
14) cd ~/gdb/builds/v2r/gdbserver
15) export LDFLAGS="-static -L~/gdb/builds/v2r/termcap/lib"
16) export CFLAGS="-g -O2 -I~/gdb/builds/v2r/termcap/include"
17) ../../../downloads/gdb-7.8/gdb/gdbserver/configure --host=arm-none-linux-gnueabi --prefix=/home/den1s/gdb/builds/v2r/gdbserver --disable-werror # &&
18) make
19) make install
PS: this article does not claim to be scientific novelty, but is an attempt to gather in one place detailed information on the assembly and configuration of the GDB-gdbserver bundle. It is likely that for advanced linuxoids the material will seem trite and clumsy, but for those people who are more hardware, or for microcontroller programmers, switching to Linux is not so simple (I judge by myself). So, hopefully, the material will serve someone useful.