Remote debugging on Linux using the GDB-gdbserver bundle

As we all know, the debugging process is such a thing, the importance of which is difficult to overestimate. Moreover, understanding the importance of such methods as LED blinking and outputting debug messages to the port, I remain of the opinion that nothing has been invented so far as step-by-step debugging. However, the step-by-step debugging task becomes less trivial when programming under Linux on embedded systems (such as rasbery pi, virt2real or industrial processor modules).

This task in Linux is intended to be solved by a standard bundle of GDB and gdbserver programs . The idea is that you write a program on your computer ( host in GDB terminology), compile it and upload it to the target device ( target) Next, run the debugged file and gdbserver on the target device ( target ) , and on to the GDB host . The interaction scheme of such debugging can be briefly represented as follows:


An executable file is the very program that we want to debug remotely. For correct operation, the program instance must be located not only on the target device, but also on the computer (host) from which debugging is in progress.
GDB is a program that directly executes the debugging process. Usually, it is part of the GCC toolchain and is located where the compiler is.
gdbserver is the reciprocal part of GDB, which runs the executable in debug mode. Since gdbserver runs on the remote side (target), it must be built for the target device using a cross-compiler. Actually, this article is devoted mainly to the assembly of gdbserver from the sources.

I have a virt2real board at my disposal, as well as a processor module based on the processor from the TI series AM335x . Below we will show the sequence of actions using the example of virt2real , however, all the same I have successfully (and importantly - similarly) done with the AM335x chip .

Note: the operating system installed on the host is Ubuntu.12.04.


We create the gdb folder in our home directory , in which we will perform all our manipulations. Inside, create a downloads subfolder :

mkdir -p ~/gdb/downloads
cd ~/gdb/downloads

Download the sources we need and unpack them. We need GDB itself , as well as the termcap library (it is used by GDB).

tar xvzf termcap-1.3.1.tar.gz
tar xvzf gdb-7.8.tar.gz

Usually collected directly in the source folder, but we will not do this in order to leave the source intact. This is useful in case you have not one, but several varieties of target platforms. Then the same downloaded sources can be used several times.

Build the termcap library

Accordingly, we start with the termcap library , because it will be needed later when building gdbserver itself . We create the builds folder in which we will collect our programs. Inside we create the v2r folder , we put all the results for the virt2real platform in it . Well, there is already a termcap folder for building the termcap library. Go to the created directory.

mkdir -p ~/gdb/builds/v2r/termcap
cd ~/gdb/builds/v2r/termcap

We indicate to the system the compiler and ranlib which we will use. In my case, this is:

export CC=/opt/virt2real-sdk/codesourcery/arm-2013.05/bin/arm-none-linux-gnueabi-gcc
export RANLIB=/opt/virt2real-sdk/codesourcery/arm-2013.05/bin/arm-none-linux-gnueabi-ranlib

Now we configure. You will need to specify 2 parameters: --host and --prefix.
What to specify as parameters
  • I admit, I still do not understand whether it matters what to specify in --host. I got the impression that no. In any case, I could not find a clear explanation of what exactly should be specified in this parameter, therefore I indicated the compiler prefix. Those. my compiler is called arm-none-linux-gnueabi-gcc, so I specified arm-none-linux-gnueabi as --host.
  • As the --prefix parameter, specify the folder in which we want to put the results of the work (configuration), i.e. the folder in which we are now.

../../../downloads/termcap-1.3.1/configure --host=arm-none-linux-gnueabi --prefix=~/gdb/builds/v2r/termcap

If everything is done correctly and without errors, a Makefile will be created in the ~ / gdb / builds / v2r / termcap folder.
Check Makefile
I recommend you to play it safe, open the Makefile and see that the variables CC and RANLIB are exactly those that we indicated above. The fact is that during my first attempts to build gdbserver, I was stuck for a long time because of this point, because everything seemed to go fine, but the resulting file didn’t want to run on the target board. It turned out that for reasons unknown to me, the standard compiler persistently used in the Makefile. If you find that the compiler in the Makefile is not the same, you can simply replace it with your hands (it helped me). Although, when I repeated the whole procedure before writing the article, I did not have such a problem.

Next, we collect the library:

make install

Collecting gdbserver

Create a folder in which we will build the server and go to it:

mkdir -p ~/gdb/builds/v2r/gdbserver
cd ~/gdb/builds/v2r/gdbserver

Specify where to get the termcap library:

export LDFLAGS="-static -L~/gdb/builds/v2r/termcap/lib"
export CFLAGS="-g -O2 -I~/gdb/builds/v2r/termcap/include"

We configure similarly to termcap. It is important to note here that we are building gdbserver (and not GDB), therefore we specify the configure file from the /gdb-7.8/gdb/gdbserver folder:

../../../downloads/gdb-7.8/gdb/gdbserver/configure --host=arm-none-linux-gnueabi --prefix=/home/den1s/gdb/builds/v2r/gdbserver --disable-werror # &&

No matter how I tried to never indicate absolute paths, so that the instruction could be used without any adjustment for myself, I did not succeed. Because the configure file swears at an attempt to specify a path starting with ~ as --prefix. Therefore, here you will need to enter your username.

If everything is correct, a Makefile will be created. Further standard:

make install


To test the debugging process, create a short Hello world and compile it for the target platform.

mkdir -p ~/virt2real/hello_test/
cd ~/virt2real/hello_test/
/opt/virt2real-sdk/codesourcery/arm-2013.05/bin/arm-none-linux-gnueabi-g++ hello.cpp -O0 -g -o hello

Download source package hello.cpp :

int main(int argc, char* argv[])
   std::cout << "Hello,\n" << std::endl;
   std::cout << "debugged\n" << std::endl;
   std::cout << "World!!!" << std::endl;
   return 0;

Fill the hello executable and our gdbserver onto the target board.
I upload to the / usr / folder using SCP:

scp ~/gdb/builds/v2r/gdbserver/bin/gdbserver root@
scp ~/virt2real/hello_test/hello root@

Now run the second instance of the terminal and connect to the target board via ssh and go to the / usr / folder :

ssh root@
cd /usr/

We run gdbserver on the target board, and with it, our hello executable (debugged) file. Then we open the debut session on the port we liked:

[root@virt2real usr]# ./gdbserver :123 hello
Process hello created; pid = 5680;
listening on port 123

We return to host and run the debugged hello file using GDB

/opt/virt2real-sdk/codesourcery/arm-2013.05/bin/arm-none-linux-gnueabi-gdb hello

In order to be able not to specify the full path to GDB all the time, you can write an environment variable:
export PATH=$PATH:/opt/virt2real-sdk/codesourcery/arm-2013.05/bin/

In response, you should see an invitation from GDB:

GNU gdb (Sourcery CodeBench Lite 2013.05-24)
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-pc-linux-gnu --target=arm-none-linux-gnueabi".
For bug reporting instructions, please see:
Reading symbols from /home/den1s/virt2real/hello_test/hello...done.

Connect to the remote gdbserver using the above port:

(gdb) target remote

We get a response and an invitation to enter the following command:

Remote debugging using
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
0x36fd7e30 in ?? ()

At the same time, the remote side (gdbserver on target in the face of virt2real) should see the installation of the debug session:

Remote debugging from host      # где - это IP нашего host'а с которого ведется отладка

A commenthosttarget
We put breakpoint on main
(gdb) break main
Cannot access memory at address 0x0
Breakpoint 1 at 0x8748: file hello.cpp, line 5.
Go to Breakpoint
(gdb) cont
warning: Could not load shared library symbols for 5 libraries, e.g. /usr/lib/
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?
Breakpoint 1, main (argc=1, argv=0x3efffb44) at hello.cpp:5
5       std::cout << "Hello\n" << std::endl;
Go to the next line
(gdb) next
6       std::cout << "debugged\n" << std::endl;
Go to the next line
(gdb) next
7       std::cout << "World!!!" << std::endl;
Go to the next line
(gdb) next
8       return 0;
Go to the next line (which, however, is not)
(gdb) next
Cannot access memory at address 0x0
Cannot access memory at address 0x0
0x36d1f040 in ?? ()
Run the program to the end, which leads to its completion
(gdb) cont
[Inferior 1 (process 1514) exited normally]
Child exited with status 0
GDBserver exiting
[root@virt2real usr]# 

This completes the step-by-step debugging example. I note:
  • for a complete list of commands, you can use the help command, as well as read a book on debugging using GDB (see the link at the end of the article)
  • “Manual” debugging using GDB is very tedious, so I recommend using, for example, Eclipse for these purposes. Unfortunately, the description of such debugging within the framework of this article would increase it to indecent dimensions. At the end of the article there is a link to a very good English-language description of this topic.

Sysroot installation

For GDB to work correctly, it needs the so-called debugging symbols , which can be read from the remote OS libraries (target). Their absence is, for example, the reason for such messages:

(gdb) next
Cannot access memory at address 0x0
Cannot access memory at address 0x0
0x36d1f040 in ?? ()
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?

And potentially cause other unpleasant debugging problems.
To fix the GDB problem on the host, you need to specify where to get these very libraries using the command:

set sysroot [Directory]

If you have somewhere on your host the Linux image of your target is overloaded, then you must specify the path to the library folder.
Or preload these libraries on host:

mkdir -p ~/virt2real/sysroot
cd ~/virt2real/sysroot
scp -r root@ lib

We return to the folder with the project, run GDB again and specify the path to the libraries:

cd ~/virt2real/hello_test/
arm-none-linux-gnueabi-gdb hello
(gdb) set sysroot ~/virt2real/hello_test/

Now, messages such as:

Reading symbols from ~/virt2real/sysroot/lib/libgcc_s.s...done.

View the list of currently used libraries as follows:

(gdb) info sharedlibrary
From        To          Syms Read   Shared Object Library
0x36fd77a0  0x36ff24d0  Yes (*)     ~/virt2real/sysroot/lib/
                        No          /usr/lib/
0x36e70e00  0x36ed107c  Yes (*)     ~/virt2real/sysroot/lib/
0x36e51028  0x36e63154  Yes (*)    ~/virt2real/sysroot/lib/
0x36d1cb00  0x36e11ae8  Yes (*)     ~/virt2real/sysroot/lib/
(*): Shared library is missing debugging information.


PS: this article does not claim to be scientific novelty, but is an attempt to gather in one place detailed information on the assembly and configuration of the GDB-gdbserver bundle. It is likely that for advanced linuxoids the material will seem trite and clumsy, but for those people who are more hardware, or for microcontroller programmers, switching to Linux is not so simple (I judge by myself). So, hopefully, the material will serve someone useful.

Also popular now: