March 8, 2010 at 20:56FreeBSD Netgraph on the example of Ethernet tunnel
Hello.
I think many system administrators working with FreeBSD are aware of the existence of the Netgraph kernel subsystem. But not many people know / understand how it works, and what can be built from it.
I’ll tell you what it is, and I will also analyze the assembly of an Ethernet bridge over the Internet using a simple example.
Wikipedia tells http://ru.wikipedia.org/wiki/Netgraph
Netgraph - a modular network subsystem of the FreeBSD kernel based on the principle of graphs. In Netgraph, a graph is constructed from nodes of various types, a node of each type has a number of inputs / outputs (hereinafter hooks). The netgraph node allows you to perform certain actions on the package passing through it. Some Netgraph nodes provide support for various protocols, encapsulations, such as L2TP, PPTP, PPPoE, PPP, ATM, bluetooth, others serve for bundling modules and sorting / routing between netgraph nodes, for example bpf, split.
The Netgraph subsystem is a set of modules, each with its own specific task. This is such a handful of lego blocks that you can always connect.
A vivid example of the use of many netgraph modules is the mpd daemon massively used by small Russian providers to terminate client tunnels such as PPPoE, PPTP, PPP, etc. In versions 3 of the mpd branch, he did many operations himself, and by the current version 5 he is mainly engaged in switching netgraph modules.
Currently there are 58 modules + 1 main module netgraph.ko (FreeBSD 7.3). All of them can be built into the kernel statically, or loaded dynamically. Since the modules are nuclear, they also work directly in the kernel, due to which you can get amazing results in performance.
There is a man for each module, which describes the module's capabilities, inputs / outputs (hooks, hooks, hooks, I will call them “hooks” below), received control messages and their actions.
To manage the netgraph subsystem, there is a ngctl utility that accepts various commands, such as list, mkpeer, connect, name, etc., you can read about them separately in man ngctl.
We pose a simple task: you need to connect two physical Ethernet networks via the Internet.
We have 2 servers.
BSD1
2 network interfaces
Interface 1 em0 - external IP 1.1.1.1
Interface 2 em1 - internal IP 192.168.1.1
BSD2
2 network interfaces
Interface 1 em0 - external IP 2.2.2.2
Interface 2 em1 - internal IP 192.168.1.2
Between servers there is a routed Internet connection via external interfaces.
You need to connect the network on the em1 interfaces of both servers.
We need:
netgraph.ko - the basis.
ng_ether.ko - module for connecting to physical network interfaces. When loading this module in the graph space, one node is automatically created for each physical interface that has the name of this interface. In our case, it will be “em0:”, “em1:”. Each ng_ether node has 3 hooks: lower, upper, orphans. The lower hook is a direct exit to the driver location of the network interface, from where packets are sent or received by the device. The upper hook is a direct exit to the place of the kernel of the system, from where packets should be sent or received by the network device driver. The orphans hook is the same lower, only erroneous, damaged packets from the network device get into it. We do not take it into account.
In simple terms: when you connect ng_ether, a gap appears in the exchange between the kernel and the network card driver. On the kernel side, this gap is called upper, on the network device driver side lower. This gap is connected until something from the netgraph subsystem connects to lower or upper.
ng_bridge.ko- module of a real Ethernet switch. Everyone, I think, can imagine (and someone still sees it), a box with Ethernet ports, blinking light bulbs, standing on a table and connecting computers to a network. ng_bridge is the same box in the netgraph subsystem. ng_bridge is a simple Ethernet switch implementation, equipped with an arp table and a simple loop detection algorithm. Ng_bridge hooks are called link0, link1, etc. We will connect hooks of our ng_ether to them. The module receives control messages, such as setconfig, getconfig, reset, getstats, clrstats, getclrstats, gettable. Message actions are intuitive, details in man ng_bridge.
ng_ksocket.ko- a module that allows you to open a socket for listening directly from the kernel (any socket supported by the socket () system function) and connect to another socket. Accepts a single hook connection. "//". The man socket tells what family, type, proto are. In our case it will be“ inet / dgram / udp. ”Inet is ipv4, dgram are datagrams, proto is udp. Also, the module receives several control messages, such as bind , listen, connect, accept, getname, getpeername, setopt, getopt. You can read more in man ng_ksocket. Using the message "connect" we can connect our incoming hook "inet / dgram / udp" to the remote one, which was created similarly, but with the command "Bind".
For a better understanding of the operation of your graph system, it is advisable to draw them before assembling in the system.
1. Connect both hooks of the ng_ether module of the em1 interface to ng_bridge so that the system sees both the physical network connected to the interface and our virtual one. It is done the same on both servers.
2. The next free link ng_bridge is connected to ng_ksocket with the parameters inet / dgram / udp. (Why did I choose UDP? Because no one guarantees successful delivery of a signal in a cable or radio network, just like in the IP protocol no one guarantees UDP delivery.) It is done the same on both servers.
3. We command the ng_ksocket module to take a specific port, at a specific IP address, and also connect to the remote IP on the desired port. We do this on both servers, in the difference that the IP addresses and ports change to opposite to each other.
I have already mentioned the ngctl utility above. With its help, we will create our modules in the system and link them.
We will need the mkpeer, connect, name, msg commands. I will describe them in simple terms.
Mkpeer team.
Syntax: ngctl mkpeer module1 module_type2 hook1 hook2
Creates a "module2" with the specified type and connects its hook "hook2" to the "hook1" of module "module1"
The connect command.
Syntax: ngctl connect module1 module2 hook1 hook2
Connects hook1 of module module1 to hook2 of module module2.
Name command.
Syntax: ngctl name module: hook name
Assigns a name to a module created through mkpeer.
Msg command
Syntax: ngctl msg module: message parameters
Sends a control “message” to the “module” with “parameters”.
Well, now what our graph will look like live for the “bsd1” server. For the tunnel we will use udp port 7777.
Create a ng_bridge node and connect the hook of the network interface “em1” “lower” to its hook “link0”.
ngctl mkpeer em1: bridge lower link0
We name the newly created node by the name “switch”, it can be found on the path “em1: lower”.
ngctl name em1: lower switch We
connect to the "link1" of our "switch" upper network interface "em1".
ngctl connect switch: em1: link1 upper
Create the ng_ksocket node and connect to its hook “inet / dgram / udp” “link2” of our “switch”
ngctl mkpeer switch: ksocket link2 inet / dgram / udp
We call the newly created ksocket “switch_socket”, it can be found on the path “switch: link2”
ngctl name switch: link2 switch_socket We
send the “bind” command to our “switch_socket”, with parameters. ksocket will take port 7777 on IP 1.1.1.1.
ngctl msg switch_socket: bind inet / 1.1.1.1: 7777
Send the “connect” command to our “switch_socket”, with parameters. ksocket will connect to port 7777 at IP address 2.2.2.2.
ngctl msg switch_socket: connect inet / 2.2.2.2: 7777
Send a command to the ng_ether module of the network interface em1 to switch to the mode of wiretapping of packets addressed not to it. After all, we now need to receive packets for devices located in our virtual network.
ngctl msg em1: setpromisc 1
ngctl msg em1: setautosrc 0
For the “bsd2” server, we just need to swap the parameters of the bind and connect commands.
For ease of use, I designed all this in sh script.
The script uses another ngctl shutdown command. This command sends a special control message to the module specified in the parameter. This message is received by each module, more in the "man module". Usually this command causes the destruction of the module and the break of all its connections.
Let's see what happened. The "ngctl16408" module with the socket type is used by ngctl for control, do not pay attention to it. Sachets run:
I wrote a lot of text, I hope, understandable. In practice, such a tunnel will seem unprotected to someone, because packets will run open in the Internet, but no one bothers you to forward this tunnel inside the VPN connection. For example, I use this bunch to forward IPTV multicast to the working network.
In the next article I will describe a few more netgraph modules - ng_netflow traffic count, ng_nat traffic, add shaping via ng_car to the resulting scheme today, or maybe I’ll come up with something else interesting.
Thanks for attention.
I think many system administrators working with FreeBSD are aware of the existence of the Netgraph kernel subsystem. But not many people know / understand how it works, and what can be built from it.
I’ll tell you what it is, and I will also analyze the assembly of an Ethernet bridge over the Internet using a simple example.
A bit of theory.
Wikipedia tells http://ru.wikipedia.org/wiki/Netgraph
Netgraph - a modular network subsystem of the FreeBSD kernel based on the principle of graphs. In Netgraph, a graph is constructed from nodes of various types, a node of each type has a number of inputs / outputs (hereinafter hooks). The netgraph node allows you to perform certain actions on the package passing through it. Some Netgraph nodes provide support for various protocols, encapsulations, such as L2TP, PPTP, PPPoE, PPP, ATM, bluetooth, others serve for bundling modules and sorting / routing between netgraph nodes, for example bpf, split.
The Netgraph subsystem is a set of modules, each with its own specific task. This is such a handful of lego blocks that you can always connect.
A vivid example of the use of many netgraph modules is the mpd daemon massively used by small Russian providers to terminate client tunnels such as PPPoE, PPTP, PPP, etc. In versions 3 of the mpd branch, he did many operations himself, and by the current version 5 he is mainly engaged in switching netgraph modules.
Let's look at our cubes.
[root@bsd1] /boot/kernel/> ls | grep ng_
ng_UI.ko
ng_async.ko
ng_atm.ko
ng_atmllc.ko
ng_bluetooth.ko
ng_bpf.ko
ng_bridge.ko
ng_bt3c.ko
ng_btsocket.ko
ng_car.ko
ng_ccatm.ko
ng_cisco.ko
ng_deflate.ko
ng_device.ko
ng_echo.ko
ng_eiface.ko
ng_etf.ko
ng_ether.ko
ng_fec.ko
ng_frame_relay.ko
ng_gif.ko
ng_gif_demux.ko
ng_h4.ko
ng_hci.ko
ng_hole.ko
ng_hub.ko
ng_iface.ko
ng_ip_input.ko
ng_ipfw.ko
ng_ksocket.ko
ng_l2cap.ko
ng_l2tp.ko
ng_lmi.ko
ng_mppc.ko
ng_nat.ko
ng_netflow.ko
ng_one2many.ko
ng_ppp.ko
ng_pppoe.ko
ng_pptpgre.ko
ng_pred1.ko
ng_rfc1490.ko
ng_socket.ko
ng_source.ko
ng_split.ko
ng_sppp.ko
ng_sscfu.ko
ng_sscop.ko
ng_sync_ar.ko
ng_sync_sr.ko
ng_tag.ko
ng_tcpmss.ko
ng_tee.ko
ng_tty.ko
ng_ubt.ko
ng_uni.ko
ng_vjc.ko
ng_vlan.ko
[root@bsd1] /boot/kernel/> ls | grep ng_ | wc -l
58
[root@bsd1] /boot/kernel/>
Currently there are 58 modules + 1 main module netgraph.ko (FreeBSD 7.3). All of them can be built into the kernel statically, or loaded dynamically. Since the modules are nuclear, they also work directly in the kernel, due to which you can get amazing results in performance.
There is a man for each module, which describes the module's capabilities, inputs / outputs (hooks, hooks, hooks, I will call them “hooks” below), received control messages and their actions.
To manage the netgraph subsystem, there is a ngctl utility that accepts various commands, such as list, mkpeer, connect, name, etc., you can read about them separately in man ngctl.
Task.
We pose a simple task: you need to connect two physical Ethernet networks via the Internet.
We have 2 servers.
BSD1
2 network interfaces
Interface 1 em0 - external IP 1.1.1.1
Interface 2 em1 - internal IP 192.168.1.1
BSD2
2 network interfaces
Interface 1 em0 - external IP 2.2.2.2
Interface 2 em1 - internal IP 192.168.1.2
Between servers there is a routed Internet connection via external interfaces.
You need to connect the network on the em1 interfaces of both servers.
Let's look at our cubes.
We need:
netgraph.ko - the basis.
ng_ether.ko - module for connecting to physical network interfaces. When loading this module in the graph space, one node is automatically created for each physical interface that has the name of this interface. In our case, it will be “em0:”, “em1:”. Each ng_ether node has 3 hooks: lower, upper, orphans. The lower hook is a direct exit to the driver location of the network interface, from where packets are sent or received by the device. The upper hook is a direct exit to the place of the kernel of the system, from where packets should be sent or received by the network device driver. The orphans hook is the same lower, only erroneous, damaged packets from the network device get into it. We do not take it into account.In simple terms: when you connect ng_ether, a gap appears in the exchange between the kernel and the network card driver. On the kernel side, this gap is called upper, on the network device driver side lower. This gap is connected until something from the netgraph subsystem connects to lower or upper.
ng_bridge.ko- module of a real Ethernet switch. Everyone, I think, can imagine (and someone still sees it), a box with Ethernet ports, blinking light bulbs, standing on a table and connecting computers to a network. ng_bridge is the same box in the netgraph subsystem. ng_bridge is a simple Ethernet switch implementation, equipped with an arp table and a simple loop detection algorithm. Ng_bridge hooks are called link0, link1, etc. We will connect hooks of our ng_ether to them. The module receives control messages, such as setconfig, getconfig, reset, getstats, clrstats, getclrstats, gettable. Message actions are intuitive, details in man ng_bridge. ng_ksocket.ko- a module that allows you to open a socket for listening directly from the kernel (any socket supported by the socket () system function) and connect to another socket. Accepts a single hook connection. "We make a graph.
For a better understanding of the operation of your graph system, it is advisable to draw them before assembling in the system.
1. Connect both hooks of the ng_ether module of the em1 interface to ng_bridge so that the system sees both the physical network connected to the interface and our virtual one. It is done the same on both servers.
2. The next free link ng_bridge is connected to ng_ksocket with the parameters inet / dgram / udp. (Why did I choose UDP? Because no one guarantees successful delivery of a signal in a cable or radio network, just like in the IP protocol no one guarantees UDP delivery.) It is done the same on both servers.
3. We command the ng_ksocket module to take a specific port, at a specific IP address, and also connect to the remote IP on the desired port. We do this on both servers, in the difference that the IP addresses and ports change to opposite to each other.
We collect the graph in the system.
I have already mentioned the ngctl utility above. With its help, we will create our modules in the system and link them.
We will need the mkpeer, connect, name, msg commands. I will describe them in simple terms.
Mkpeer team.
Syntax: ngctl mkpeer module1 module_type2 hook1 hook2
Creates a "module2" with the specified type and connects its hook "hook2" to the "hook1" of module "module1"
The connect command.
Syntax: ngctl connect module1 module2 hook1 hook2
Connects hook1 of module module1 to hook2 of module module2.
Name command.
Syntax: ngctl name module: hook name
Assigns a name to a module created through mkpeer.
Msg command
Syntax: ngctl msg module: message parameters
Sends a control “message” to the “module” with “parameters”.
Well, now what our graph will look like live for the “bsd1” server. For the tunnel we will use udp port 7777.
Create a ng_bridge node and connect the hook of the network interface “em1” “lower” to its hook “link0”.
ngctl mkpeer em1: bridge lower link0
We name the newly created node by the name “switch”, it can be found on the path “em1: lower”.
ngctl name em1: lower switch We
connect to the "link1" of our "switch" upper network interface "em1".
ngctl connect switch: em1: link1 upper
Create the ng_ksocket node and connect to its hook “inet / dgram / udp” “link2” of our “switch”
ngctl mkpeer switch: ksocket link2 inet / dgram / udp
We call the newly created ksocket “switch_socket”, it can be found on the path “switch: link2”
ngctl name switch: link2 switch_socket We
send the “bind” command to our “switch_socket”, with parameters. ksocket will take port 7777 on IP 1.1.1.1.
ngctl msg switch_socket: bind inet / 1.1.1.1: 7777
Send the “connect” command to our “switch_socket”, with parameters. ksocket will connect to port 7777 at IP address 2.2.2.2.
ngctl msg switch_socket: connect inet / 2.2.2.2: 7777
Send a command to the ng_ether module of the network interface em1 to switch to the mode of wiretapping of packets addressed not to it. After all, we now need to receive packets for devices located in our virtual network.
ngctl msg em1: setpromisc 1
ngctl msg em1: setautosrc 0
For the “bsd2” server, we just need to swap the parameters of the bind and connect commands.
For ease of use, I designed all this in sh script.
The script uses another ngctl shutdown command. This command sends a special control message to the module specified in the parameter. This message is received by each module, more in the "man module". Usually this command causes the destruction of the module and the break of all its connections.
#! / bin / sh
self = 1.1.1.1
peer = 2.2.2.2
port = 7777
if = em1
case "$ 1" in
start)
echo "Starting netgraph switch."
ngctl mkpeer $ {if}: bridge lower link0
ngctl name $ {if}: lower switch
ngctl connect switch: $ {if}: link1 upper
ngctl mkpeer switch: ksocket link2 inet / dgram / udp
ngctl name switch: link2 switch_socket
ngctl msg switch_socket: bind inet / $ {self}: $ {port}
ngctl msg switch_socket: connect inet / $ {peer}: $ {port}
ngctl msg $ {if}: setpromisc 1
ngctl msg $ {if}: setautosrc 0
echo "Ok."
exit 0
;;
stop)
echo "Stopping netgraph switch."
ngctl shutdown switch_socket:
ngctl shutdown switch:
ngctl shutdown $ {if}:
echo "Ok."
exit 0
;;
restart)
sh $ 0 stop
sh $ 0 start
;;
*)
echo "Usage:` basename $ 0` {start | stop | restart} "
exit 64
;;
esac
Let's see what happened. The "ngctl16408" module with the socket type is used by ngctl for control, do not pay attention to it. Sachets run:
[root@bsd1] /usr/local/etc/rc.d/> ngctl list
There are 5 total nodes:
Name: em0 Type: ether ID: 00000001 Num hooks: 0
Name: em1 Type: ether ID: 00000002 Num hooks: 2
Name: switch Type: bridge ID: 000000f6 Num hooks: 3
Name: ngctl16408 Type: socket ID: 00000100 Num hooks: 0
Name: switch_socket Type: ksocket ID: 000000fa Num hooks: 1
[root @ bsd1] / root /> ping 192.168.1.2
PING 192.168.1.2 (192.168.1.2): 56 data bytes
64 bytes from 192.168.1.2: icmp_seq = 0 ttl = 64 time = 3.760 ms
64 bytes from 192.168.1.2: icmp_seq = 1 ttl = 64 time = 3.527 ms
64 bytes from 192.168.1.2: icmp_seq = 2 ttl = 64 time = 3.479 ms
64 bytes from 192.168.1.2: icmp_seq = 3 ttl = 64 time = 4.052 ms
[root @ bsd1] / root /> ngctl msg switch: getstats 2
Rec'd response "getstats" (4) from "[f6]:":
Args: {recvOctets = 49333 recvPackets = 532 recvMulticast = 467 recvBroadcast = 63 xmitOctets = 580 xmitPackets = 12 xmitMulticasts = 10 xmitBroadcasts = 1}
the end
I wrote a lot of text, I hope, understandable. In practice, such a tunnel will seem unprotected to someone, because packets will run open in the Internet, but no one bothers you to forward this tunnel inside the VPN connection. For example, I use this bunch to forward IPTV multicast to the working network.
In the next article I will describe a few more netgraph modules - ng_netflow traffic count, ng_nat traffic, add shaping via ng_car to the resulting scheme today, or maybe I’ll come up with something else interesting.
Thanks for attention.
