OpenAI Gym + ROS + Gazebo: training a standalone robot at home. Part 1
Over the past few years, we have witnessed the introduction of artificial intelligence technologies in our daily lives - from robotic vacuum cleaners to unmanned drones. All of them, controlled by artificial intelligence, are already commonplace for us. But despite this, the development process, from design to implementation, takes years and is not cheap. In addition, machine learning algorithms require big data and there is no guarantee that everything will work out in the long run.
Sooner or later, every developer comes to the conclusion that before creating a real robot, you need to test the concept in a simulation, debug all systems and, in the end, understand whether the development path has been chosen.
Dr. Andrew Howard had similar thoughts when he, with his student Nathon Koenig, began in 2002 the development of a 3D simulator Gazebo at the University of Southern California. The concept of a high-precision simulator arose because of the need to test robots in various difficult outdoor conditions. In the first stages of development, this approach saves time and money on the purchase of the necessary equipment.
In this series of articles, I would like to share my experience in simulating and training an unmanned vehicle with only one ordinary camera as a sensor.
Hello. I am a Master of Science (Computer Science, MS) in Applied Informatics at the Berlin University of Applied Sciences (HTW-Berlin).
As part of my course, I work on methods for training autonomous robots in a simulated environment using ROS [1], Gazebo [2] and OpenAI Gym [3]. This course work is divided into two parts.
The first part is devoted to the development of a framework that would simplify the testing of Reinforcement Learning (training with reinforcement) algorithms on various autonomous robots inside the Gazebo simulation.
In the second part, I will concentrate on the implementation of the algorithm for autonomous driving. I will be working on this for the next semester, so this and the next articles will focus mainly on the implementation of the OpenAI Gym environment for Gazebo. Reinforcement Learning (RL) itself will be affected superficially.
The full project code can be found here .
We will use the ROS version of Melodic. This version has several limitations and the most significant of them is Ubuntu 18 (it is also possible to install ROS on Windows WSL and on Ubuntu Headless, but more on that another time). Additionally, ROS still does not support Python 3.
In this pilot part, I will briefly talk about the technologies that will be used to create the environment. I will also describe the installation process and in the end we will launch a full-fledged simulation, which can be controlled through the Gym API.
Robot Operating System (ROS) is a framework for programming robots. ROS is a “software glue” that enables developers to focus on their specific task. Although ROS is not an operating system, it does provide services such as hardware abstraction, low-level device management, implementing frequently used functions, passing messages between processes, and managing packages (plugins).
ROS is designed as a loosely coupled system in which a process called a node must be responsible for one task. Nodes communicate with each other using messages passing through logical channels called topics. Each node can send or receive data from another node using the publish – subscribe pattern.
For ROS, drivers are already implemented that allow working in a unified way with many devices, such as controllers, GPS, cameras, laser rangefinders, etc. etc.
And so, install ROS Melodic ( full instructions )
For some reason, the full installation package does not include some necessary packages. Install them separately:
We will not dwell on them in detail. They are needed to control our robot, which will be discussed later.
Gazebo is an open source dynamic 3D simulator that is developed by the Open Source Robotic Foundation and interacts quite closely with ROS. Gazebo allows you to accurately and efficiently simulate robots both in difficult indoor and outdoor conditions.
The simulator consists of the gzserver server , which is involved in calculating physics, collisions and simulating sensors. Clients can connect to the server, for example gzclient (for the desktop) and gzweb (for the browser). They are the ones who render the models.
All this makes it possible to test complex robotic systems in virtual space much faster and without risk to damage expensive real robots.
Gazebo is included in the full ROS installation package, so you don’t need to install anything else. For headless configuration gzweb is required . I will talk about him in the following parts.
OpenAI is a nonprofit artificial intelligence research company co-founded by Ilon Max.
OpenAI Gym is a Python library that provides an API for developing and comparing RL algorithms with a huge number of virtual environments and a common API. Gym already has many ready-made environments, including for Atari games.
My project is based on the openai_ros package . This package implements the architecture that was proposed by The Construct team . The Construct guys have developed a common framework that implements the necessary API for managing the simulation and describes an API for integrating Gazebo into Gym, requiring a minimal implementation. The proposed structure consists of three logical layers (you can actually use any number), the lowest of them is another “glue” that connects the Gym API with Gazebo. More about this in the next part, when we will parse the code.
First we need to create a working directory for catkin. catkin is a build automation system like CMake, which is included by default in the ROS installation package:
and clone the necessary ROS packages.
We need a robot model developed by MIT with all sensors.
Well, actually, the project itself
Next, you need to compile all these packages
You also need to install all the necessary python libraries.
To run any ROS package, we first need to load into the current terminal session all the ROS packages that we compiled:
Now we can run the Racecar robot simulation:
This command will launch the ROS master server, the Gazebo server along with its UI (on the desktop), load the robot and the world for it.
Screenshot:
And the same thing in the second terminal for our RL algorithm:
This team will launch our neuroracer_gym_rl package, which implements a simple Q-Learning algorithm in our Gym environment. We will analyze it in the next part.
Those who can't wait can find the source code here . Looking ahead, I’ll say that after several days of training on a server with a GeForce GTX 1080 Ti 11GB RAM card, this simple algorithm really did not learn anything. In the next part I will list the possible reasons, but now you can experiment with your implementation.
And so, we have a fully working simulation of an autonomous car and a ready-made RL algorithm template. In the next part, we will analyze openai_ros in more detail and discuss the sensors installed on the robot. After that I will show how to implement my algorithms and what problems may arise during training.
PS: This is my first experience writing an article. Therefore, do not hesitate to write if you find mistakes or you have ideas, or constructive criticism. I will try to take this into account in the following parts.
[1] (M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, and AY Ng. Ros: an open-source robot operating system. In ICRA workshop on open source software, volume 3, page 5. Kobe, Japan, 2009.
[2] NP Koenig and A. Howard. Design and use paradigms for gazebo, an open-source multi-robot simulator. In IROS, volume 4, pages 2149 -2154 . Citeseer, 2004.
[3] LPJSJSJTWZ Greg Brockman, Vicki Cheung. Openai gym, 2016, arXiv: 1606.01540
Sooner or later, every developer comes to the conclusion that before creating a real robot, you need to test the concept in a simulation, debug all systems and, in the end, understand whether the development path has been chosen.
Dr. Andrew Howard had similar thoughts when he, with his student Nathon Koenig, began in 2002 the development of a 3D simulator Gazebo at the University of Southern California. The concept of a high-precision simulator arose because of the need to test robots in various difficult outdoor conditions. In the first stages of development, this approach saves time and money on the purchase of the necessary equipment.
In this series of articles, I would like to share my experience in simulating and training an unmanned vehicle with only one ordinary camera as a sensor.
Hello. I am a Master of Science (Computer Science, MS) in Applied Informatics at the Berlin University of Applied Sciences (HTW-Berlin).
As part of my course, I work on methods for training autonomous robots in a simulated environment using ROS [1], Gazebo [2] and OpenAI Gym [3]. This course work is divided into two parts.
The first part is devoted to the development of a framework that would simplify the testing of Reinforcement Learning (training with reinforcement) algorithms on various autonomous robots inside the Gazebo simulation.
In the second part, I will concentrate on the implementation of the algorithm for autonomous driving. I will be working on this for the next semester, so this and the next articles will focus mainly on the implementation of the OpenAI Gym environment for Gazebo. Reinforcement Learning (RL) itself will be affected superficially.
The full project code can be found here .
System requirements
- Ubuntu 18.04 or Windows WSL Ubuntu 18.04
- Python 2.7
- pip
- Tenserflow CPU or GPU
We will use the ROS version of Melodic. This version has several limitations and the most significant of them is Ubuntu 18 (it is also possible to install ROS on Windows WSL and on Ubuntu Headless, but more on that another time). Additionally, ROS still does not support Python 3.
Part One: Introduction
In this pilot part, I will briefly talk about the technologies that will be used to create the environment. I will also describe the installation process and in the end we will launch a full-fledged simulation, which can be controlled through the Gym API.
ROS: “don't reinvent the wheel”
Robot Operating System (ROS) is a framework for programming robots. ROS is a “software glue” that enables developers to focus on their specific task. Although ROS is not an operating system, it does provide services such as hardware abstraction, low-level device management, implementing frequently used functions, passing messages between processes, and managing packages (plugins).
ROS is designed as a loosely coupled system in which a process called a node must be responsible for one task. Nodes communicate with each other using messages passing through logical channels called topics. Each node can send or receive data from another node using the publish – subscribe pattern.
For ROS, drivers are already implemented that allow working in a unified way with many devices, such as controllers, GPS, cameras, laser rangefinders, etc. etc.
And so, install ROS Melodic ( full instructions )
sudo sh -c 'echo "deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list'
sudo apt-key adv --keyserver hkp://ha.pool.sks-keyservers.net:80 --recv-key 421C365BD9FF1F717815A3895523BAEEB01FA116
sudo apt update
sudo apt install ros-melodic-desktop-full
sudo rosdep init
rosdep update
echo "source /opt/ros/melodic/setup.bash" >> ~/.bashrc
source /opt/ros/melodic/setup.bashFor some reason, the full installation package does not include some necessary packages. Install them separately:
sudo apt install ros-melodic-ackermann-msgs
sudo apt install ros-melodic-effort-controllers
sudo apt install ros-melodic-joy
sudo apt install ros-melodic-tf2-sensor-msgsWe will not dwell on them in detail. They are needed to control our robot, which will be discussed later.
Gazebo
Gazebo is an open source dynamic 3D simulator that is developed by the Open Source Robotic Foundation and interacts quite closely with ROS. Gazebo allows you to accurately and efficiently simulate robots both in difficult indoor and outdoor conditions.
The simulator consists of the gzserver server , which is involved in calculating physics, collisions and simulating sensors. Clients can connect to the server, for example gzclient (for the desktop) and gzweb (for the browser). They are the ones who render the models.
All this makes it possible to test complex robotic systems in virtual space much faster and without risk to damage expensive real robots.
Gazebo is included in the full ROS installation package, so you don’t need to install anything else. For headless configuration gzweb is required . I will talk about him in the following parts.
OpenAI Gym
OpenAI is a nonprofit artificial intelligence research company co-founded by Ilon Max.
OpenAI Gym is a Python library that provides an API for developing and comparing RL algorithms with a huge number of virtual environments and a common API. Gym already has many ready-made environments, including for Atari games.
openai_ros
My project is based on the openai_ros package . This package implements the architecture that was proposed by The Construct team . The Construct guys have developed a common framework that implements the necessary API for managing the simulation and describes an API for integrating Gazebo into Gym, requiring a minimal implementation. The proposed structure consists of three logical layers (you can actually use any number), the lowest of them is another “glue” that connects the Gym API with Gazebo. More about this in the next part, when we will parse the code.
Putting it all together
First we need to create a working directory for catkin. catkin is a build automation system like CMake, which is included by default in the ROS installation package:
cd ~
mkdir catkin_ws
cd catkin_ws/
mkdir src
cd src/and clone the necessary ROS packages.
Mit racecar
We need a robot model developed by MIT with all sensors.
git clone https://github.com/mit-racecar/racecar.git
git clone https://github.com/mit-racecar/racecar-simulator.gitopenai_ros
git clone https://bitbucket.org/theconstructcore/openai_ros.gitneuroracer
Well, actually, the project itself
git clone https://github.com/karray/neuroracer.gitNext, you need to compile all these packages
cd ~/catkin_ws
catkin_makeYou also need to install all the necessary python libraries.
pip install -U numpy tensorflow gym kerasRunning simulation
To run any ROS package, we first need to load into the current terminal session all the ROS packages that we compiled:
source ~/catkin_ws/devel/setup.bashNow we can run the Racecar robot simulation:
roslaunch racecar_gazebo racecar_tunnel.launchThis command will launch the ROS master server, the Gazebo server along with its UI (on the desktop), load the robot and the world for it.
Screenshot:
Spoiler heading
And the same thing in the second terminal for our RL algorithm:
source ~/catkin_ws/devel/setup.bash
roslaunch neuroracer_gym_rl qlearning.launchThis team will launch our neuroracer_gym_rl package, which implements a simple Q-Learning algorithm in our Gym environment. We will analyze it in the next part.
Those who can't wait can find the source code here . Looking ahead, I’ll say that after several days of training on a server with a GeForce GTX 1080 Ti 11GB RAM card, this simple algorithm really did not learn anything. In the next part I will list the possible reasons, but now you can experiment with your implementation.
Conclusion
And so, we have a fully working simulation of an autonomous car and a ready-made RL algorithm template. In the next part, we will analyze openai_ros in more detail and discuss the sensors installed on the robot. After that I will show how to implement my algorithms and what problems may arise during training.
PS: This is my first experience writing an article. Therefore, do not hesitate to write if you find mistakes or you have ideas, or constructive criticism. I will try to take this into account in the following parts.
References
[1] (M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, and AY Ng. Ros: an open-source robot operating system. In ICRA workshop on open source software, volume 3, page 5. Kobe, Japan, 2009.
[2] NP Koenig and A. Howard. Design and use paradigms for gazebo, an open-source multi-robot simulator. In IROS, volume 4, pages 2149 -2154 . Citeseer, 2004.
[3] LPJSJSJTWZ Greg Brockman, Vicki Cheung. Openai gym, 2016, arXiv: 1606.01540