November 10, 2015 at 14:19Another virtual interface
In a previous note , a code sketch of the Linux kernel module was shown to create an additional virtual network interface. This was a simplified fragment from a real project that worked for several years without failures and complaints, so it could well serve as a template for further improvement, correction and development.
But this approach to implementation is, firstly, not the only one, and, secondly, in some situations it may be unacceptable (for example, in an embedded system with a kernel younger than 2.6.36, where netdev_rx_handler_register () has not yet been called). Below we will consider an alternative with the same functionality, but implementing it on a completely different layer of the TCP / IP network stack.
Quite a lot, so as not to be repeated, it is written that the levels (layers) of the TCP / IP network stack do not correspond unambiguously to the 7 levels of the OSI / ISO open source system interaction model (or, to be more honest, the OSI model, which is close to the heart of academia, turned out to be inadequate really evolving TCP / IP network). The creation of the virtual interface, in the previous implementation discussed , was carried out at the interface level (L2, Level 2 - very roughly corresponding to the OSI channel level). The current implementation uses the capabilities of the network layer (L3).
It seems advisable to consider a certain minimum with respect to the network level tools, in the volume even slightly wider than necessary for the current task, for the possibilities of its further expansion. At the network level, the network protocol stack (TCP / IP, but not only - all other protocol families are supported here, but today they seem to be of little relevance) the processing of such protocols as: IP / IPv4 / IPv6, IPX, IGMP, RIP, OSPF, ARP, or the addition of original user protocols. A network-level API is provided for installing network-level handlers ():
In fact, in the protocol modules of the kernel, we need to add a filter through which socket buffers pass from the incoming interface stream (the outgoing stream is implemented easier, as shown in the previous implementation). The dev_add_pack () function adds another new handler for packages of a given type, implemented by the func () function. The function adds but does not replace the existing handler (including the default handler of the Linux network system). For processing, the function selects (gets) those socket buffers that satisfy the criteria laid down in the structure of the struct packet_type (by the type of protocol type and the network interface dev).
Note:According to the same scheme (setting the filter function), new protocols are added at a higher transport level of the network stack (on which, for example, UDP, TCP, SCTP protocols are processed). All higher levels (more or less similar to the OSI model levels) are not represented in the kernel, and are served in the user space by BSD socket programming techniques. But all these higher-level details will no longer be considered in the text.
If we would like to add a new protocol (proprietary), we would have to redefine its type:
The problem would be that the standard IP stack does not know such a protocol, and we will have to take care of all its processing. But our goal is only to redefine the processing of some packets, then for this we use the constant ETH_P_ALL, indicating that all protocols should pass through the filter (and if the dev field is NULL, then all network interfaces).
For comparison and concretization, a large number of protocol identifiers (Ethernet Protocol ID's) are found in, here are some of them, for example:
In this case, the type field is not an abstract numerical value in the program code, this value in binary form will be entered in the Ethernet header of the frame physically sent to the distribution environment:
(We will need the same description in the code when filling out the struct packet_type structure in the module).
The filter function itself (the func field), which we have yet to write, maybe, in its simplest form, is something like this:
The function is shown here mainly because of the mandatory call to kfree_skb (). It, unlike the seemingly close in meaning dev_kfree_skb () in the transmitting channel, does not destroy the socket buffer, but only decrements its usage counter (users field). When each additional protocol filter is set by calling dev_add_pack (), this socket buffer field will be incremented. You can install several network-level filters (in the same one, or several loadable modules) and they will work allin the reverse order of their installation, but each of them must execute kfree_skb (). Otherwise, you will have a slow but steady memory leak in the network stack, so its result, like a system crash, will be detected only after a few hours of continuous operation.
This is quite an interesting and not obvious place, so much so that it makes sense to distract and look at the source code for the implementation of kfree_skb () (file net / core / skbuff.c):
Calling kfree_skb () will actually release the socket buffer only if skb-> users == 1, for all other values it will only decrement skb-> users (usage counter).
Now we have enough details to organize the operation of the virtual interface, but using, this time, the network layer of the IP stack.
We will proceed as before : we will create two versions of the module - a simplified version of virtl.ko, whose network interface (virt0) replacesthe parent network interface, and the complete virt.ko version, which analyzes network protocol frames (ARP and IP4), and affects only the traffic that relates to its interface. The difference is that during the loading of the simplified module, the parent interface temporarily stops working (until the virtl.ko module is unloaded), and when the full version is loaded, both interfaces can work in parallel and independently. The code for the complete module is noticeably more cumbersome, but it does not add anything to understand the principles. Next, a simplified version showing the principles is considered in detail, and only later will we touch on the full version minimally (its code and test report are given in the examples archive):
Everything is quite transparent:
Here's how it works:
Now briefly, in a nutshell, on how to make a full-fledged virtual interface that works only with its own traffic and does not interfere with the operation of the parent interface (what the full version of the module does in the archive). To do this, you must:
Using such a full-weight module, you can open, for example, two parallel SSH sessions on different interfaces (using different IP), which will actually use a single common physical interface in parallel:
The last command shown (who) is already executed in an SSH session, that is, on that very remote host, to which two independent connections from two different subnets are fixed (the last two lines of output), which actually represent one host, but from the point of view its various network interfaces.
In preparing and debugging sample modules, to clarify the details, this (fairly recent) book was actively used: Rami Rosen: “Linux Kernel Networking: Implementation and Theory”, Apress, 650 pages, 2014, ISBN-13: 978-1-4302 -6196-4.
The author kindly provided it for free download before the book went on sale (2013-12-22). You can download it on this page .
Everyone who is interested in issues such as those discussed in this article will be able to find many ideas in this publication for the further development of the technique of their own use of network interfaces.
The archive of codes mentioned in the text for experiments and further development can be taken here or here .
But this approach to implementation is, firstly, not the only one, and, secondly, in some situations it may be unacceptable (for example, in an embedded system with a kernel younger than 2.6.36, where netdev_rx_handler_register () has not yet been called). Below we will consider an alternative with the same functionality, but implementing it on a completely different layer of the TCP / IP network stack.
Network layer protocols
Quite a lot, so as not to be repeated, it is written that the levels (layers) of the TCP / IP network stack do not correspond unambiguously to the 7 levels of the OSI / ISO open source system interaction model (or, to be more honest, the OSI model, which is close to the heart of academia, turned out to be inadequate really evolving TCP / IP network). The creation of the virtual interface, in the previous implementation discussed , was carried out at the interface level (L2, Level 2 - very roughly corresponding to the OSI channel level). The current implementation uses the capabilities of the network layer (L3).
It seems advisable to consider a certain minimum with respect to the network level tools, in the volume even slightly wider than necessary for the current task, for the possibilities of its further expansion. At the network level, the network protocol stack (TCP / IP, but not only - all other protocol families are supported here, but today they seem to be of little relevance) the processing of such protocols as: IP / IPv4 / IPv6, IPX, IGMP, RIP, OSPF, ARP, or the addition of original user protocols. A network-level API is provided for installing network-level handlers (
struct packet_type {
__be16 type; /* This is really htons(ether_type). */
struct net_device *dev; /* NULL is wildcarded here */
int (*func) (struct sk_buff*, struct net_device*, struct packet_type*, struct net_device*);
...
struct list_head list;
};
extern void dev_add_pack( struct packet_type *pt );
extern void dev_remove_pack( struct packet_type *pt );
In fact, in the protocol modules of the kernel, we need to add a filter through which socket buffers pass from the incoming interface stream (the outgoing stream is implemented easier, as shown in the previous implementation). The dev_add_pack () function adds another new handler for packages of a given type, implemented by the func () function. The function adds but does not replace the existing handler (including the default handler of the Linux network system). For processing, the function selects (gets) those socket buffers that satisfy the criteria laid down in the structure of the struct packet_type (by the type of protocol type and the network interface dev).
Note:According to the same scheme (setting the filter function), new protocols are added at a higher transport level of the network stack (on which, for example, UDP, TCP, SCTP protocols are processed). All higher levels (more or less similar to the OSI model levels) are not represented in the kernel, and are served in the user space by BSD socket programming techniques. But all these higher-level details will no longer be considered in the text.
If we would like to add a new protocol (proprietary), we would have to redefine its type:
#define PROTO_ID 0x1234
static struct packet_type test_proto = {
__constant_htons( PROT_ID ),
...
}
The problem would be that the standard IP stack does not know such a protocol, and we will have to take care of all its processing. But our goal is only to redefine the processing of some packets, then for this we use the constant ETH_P_ALL, indicating that all protocols should pass through the filter (and if the dev field is NULL, then all network interfaces).
For comparison and concretization, a large number of protocol identifiers (Ethernet Protocol ID's) are found in
#define ETH_P_LOOP 0x0060 /* Ethernet Loopback packet */
#define ETH_P_IP 0x0800 /* Internet Protocol packet */
#define ETH_P_ARP 0x0806 /* Address Resolution packet */
#define ETH_P_PAE 0x888E /* Port Access Entity (IEEE 802.1X) */
#define ETH_P_ALL 0x0003 /* Every packet (be careful!!!) */
...
In this case, the type field is not an abstract numerical value in the program code, this value in binary form will be entered in the Ethernet header of the frame physically sent to the distribution environment:
struct ethhdr {
unsigned char h_dest[ETH_ALEN]; /* destination eth addr */
unsigned char h_source[ETH_ALEN]; /* source ether addr */
__be16 h_proto; /* packet type ID field */
} __attribute__((packed));
(We will need the same description in the code when filling out the struct packet_type structure in the module).
The filter function itself (the func field), which we have yet to write, maybe, in its simplest form, is something like this:
int test_pack_rcv( struct sk_buff *skb, struct net_device *dev,
struct packet_type *pt, struct net_device *odev ) {
LOG( "packet received with length: %u\n", skb->len );
kfree_skb( skb );
return skb->len;
};
The function is shown here mainly because of the mandatory call to kfree_skb (). It, unlike the seemingly close in meaning dev_kfree_skb () in the transmitting channel, does not destroy the socket buffer, but only decrements its usage counter (users field). When each additional protocol filter is set by calling dev_add_pack (), this socket buffer field will be incremented. You can install several network-level filters (in the same one, or several loadable modules) and they will work allin the reverse order of their installation, but each of them must execute kfree_skb (). Otherwise, you will have a slow but steady memory leak in the network stack, so its result, like a system crash, will be detected only after a few hours of continuous operation.
This is quite an interesting and not obvious place, so much so that it makes sense to distract and look at the source code for the implementation of kfree_skb () (file net / core / skbuff.c):
void kfree_skb(struct sk_buff *skb) {
if (unlikely(!skb))
return;
if (likely(atomic_read(&skb->users) == 1))
smp_rmb();
else if (likely(!atomic_dec_and_test(&skb->users)))
return;
trace_kfree_skb(skb, __builtin_return_address(0));
__kfree_skb(skb);
}
Calling kfree_skb () will actually release the socket buffer only if skb-> users == 1, for all other values it will only decrement skb-> users (usage counter).
Now we have enough details to organize the operation of the virtual interface, but using, this time, the network layer of the IP stack.
Virtual interface module
We will proceed as before : we will create two versions of the module - a simplified version of virtl.ko, whose network interface (virt0) replacesthe parent network interface, and the complete virt.ko version, which analyzes network protocol frames (ARP and IP4), and affects only the traffic that relates to its interface. The difference is that during the loading of the simplified module, the parent interface temporarily stops working (until the virtl.ko module is unloaded), and when the full version is loaded, both interfaces can work in parallel and independently. The code for the complete module is noticeably more cumbersome, but it does not add anything to understand the principles. Next, a simplified version showing the principles is considered in detail, and only later will we touch on the full version minimally (its code and test report are given in the examples archive):
here is a fairly long code
#include
#include
#include
#include
#include
#include
#include
#include
#define ERR(...) printk( KERN_ERR "! "__VA_ARGS__ )
#define LOG(...) printk( KERN_INFO "! "__VA_ARGS__ )
#define DBG(...) if( debug != 0 ) printk( KERN_INFO "! "__VA_ARGS__ )
static char* link = "eth0";
module_param( link, charp, 0 );
static char* ifname = "virt";
module_param( ifname, charp, 0 );
static int debug = 0;
module_param( debug, int, 0 );
static struct net_device *child = NULL;
static struct net_device_stats stats; // статическая таблица статистики интерфейса
static u32 child_ip;
struct priv {
struct net_device *parent;
};
static char* strIP( u32 addr ) { // диагностика IP в точечной нотации
static char saddr[ MAX_ADDR_LEN ];
sprintf( saddr, "%d.%d.%d.%d",
( addr ) & 0xFF, ( addr >> 8 ) & 0xFF,
( addr >> 16 ) & 0xFF, ( addr >> 24 ) & 0xFF
);
return saddr;
}
static int open( struct net_device *dev ) {
struct in_device *in_dev = dev->ip_ptr;
struct in_ifaddr *ifa = in_dev->ifa_list; /* IP ifaddr chain */
LOG( "%s: device opened", dev->name );
child_ip = ifa->ifa_address;
netif_start_queue( dev );
if( debug != 0 ) {
char sdebg[ 40 ] = "";
sprintf( sdebg, "%s:", strIP( ifa->ifa_address ) );
strcat( sdebg, strIP( ifa->ifa_mask ) );
DBG( "%s: %s", dev->name, sdebg );
}
return 0;
}
static int stop( struct net_device *dev ) {
LOG( "%s: device closed", dev->name );
netif_stop_queue( dev );
return 0;
}
static struct net_device_stats *get_stats( struct net_device *dev ) {
return &stats;
}
// передача фрейма
static netdev_tx_t start_xmit( struct sk_buff *skb, struct net_device *dev ) {
struct priv *priv = netdev_priv( dev );
stats.tx_packets++;
stats.tx_bytes += skb->len;
skb->dev = priv->parent; // передача в родительский (физический) интерфейс
skb->priority = 1;
dev_queue_xmit( skb );
DBG( "tx: injecting frame from %s to %s with length: %u",
dev->name, skb->dev->name, skb->len );
return 0;
return NETDEV_TX_OK;
}
static struct net_device_ops net_device_ops = {
.ndo_open = open,
.ndo_stop = stop,
.ndo_get_stats = get_stats,
.ndo_start_xmit = start_xmit,
};
// приём фрейма
int pack_parent( struct sk_buff *skb, struct net_device *dev,
struct packet_type *pt, struct net_device *odev ) {
skb->dev = child; // передача фрейма в виртуальный интерфейс
stats.rx_packets++;
stats.rx_bytes += skb->len;
DBG( "tx: injecting frame from %s to %s with length: %u",
dev->name, skb->dev->name, skb->len );
kfree_skb( skb );
return skb->len;
};
static struct packet_type proto_parent = {
__constant_htons( ETH_P_ALL ), // перехватывать все пакеты: ETH_P_ARP & ETH_P_IP
NULL,
pack_parent,
(void*)1,
NULL
};
int __init init( void ) {
void setup( struct net_device *dev ) { // вложенная функция (расширение GCC)
int j;
ether_setup( dev );
memset( netdev_priv( dev ), 0, sizeof( struct priv ) );
dev->netdev_ops = &net_device_ops;
for( j = 0; j < ETH_ALEN; ++j ) // заполнить MAC фиктивным адресом
dev->dev_addr[ j ] = (char)j;
}
int err = 0;
struct priv *priv;
char ifstr[ 40 ];
sprintf( ifstr, "%s%s", ifname, "%d" );
#if (LINUX_VERSION_CODE < KERNEL_VERSION(3, 17, 0))
child = alloc_netdev( sizeof( struct priv ), ifstr, setup );
#else
child = alloc_netdev( sizeof( struct priv ), ifstr, NET_NAME_UNKNOWN, setup );
#endif
if( child == NULL ) {
ERR( "%s: allocate error", THIS_MODULE->name ); return -ENOMEM;
}
priv = netdev_priv( child );
priv->parent = dev_get_by_name( &init_net, link ); // родительский интерфейс
if( !priv->parent ) {
ERR( "%s: no such net: %s", THIS_MODULE->name, link );
err = -ENODEV; goto err;
}
if( priv->parent->type != ARPHRD_ETHER && priv->parent->type != ARPHRD_LOOPBACK ) {
ERR( "%s: illegal net type", THIS_MODULE->name );
err = -EINVAL; goto err;
}
memcpy( child->dev_addr, priv->parent->dev_addr, ETH_ALEN );
memcpy( child->broadcast, priv->parent->broadcast, ETH_ALEN );
if( ( err = dev_alloc_name( child, child->name ) ) ) {
ERR( "%s: allocate name, error %i", THIS_MODULE->name, err );
err = -EIO; goto err;
}
register_netdev( child ); // зарегистрировать новый интерфейс
proto_parent.dev = priv->parent;
dev_add_pack( &proto_parent ); // установить обработчик фреймов для родителя
LOG( "module %s loaded", THIS_MODULE->name );
LOG( "%s: create link %s", THIS_MODULE->name, child->name );
return 0;
err:
free_netdev( child );
return err;
}
void __exit virt_exit( void ) {
struct priv *priv= netdev_priv( child );
dev_remove_pack( &proto_parent ); // удалить обработчик фреймов
unregister_netdev( child );
dev_put( priv->parent );
free_netdev( child );
LOG( "module %s unloaded", THIS_MODULE->name );
LOG( "=============================================" );
}
module_init( init );
module_exit( virt_exit );
MODULE_AUTHOR( "Oleg Tsiliuric" );
MODULE_LICENSE( "GPL v2" );
MODULE_VERSION( "3.7" );
Everything is quite transparent:
- After registering a new network interface (virt0), it makes a call to dev_add_pack (), which sets the packet filter for the parent interface;
- The dev field is previously set in the packet_type structure to the parent interface pointer: only from this interface will incoming traffic be intercepted by the pack_parent () function defined in the structure;
- This function captures interface statistics and, most importantly, replaces the parent interface pointer with a virtual one in the socket buffer.
- Reverse substitution (virtual to physical) occurs in the send function of the start_xmit () frame.
Here's how it works:
- On the tested computer, load the module and configure it on a separate new subnet:
$ ip address ... 2: eth0:
(This shows statistics with zero bytes received on the interface). - On the computer we are testing from, we create an alias IP for the new subnet (192.168.50.0/24) and we can carry out traffic to the created interface:
$ sudo ifconfig vboxnet0:1 192.168.50.1 $ ping 192.168.50.19 PING 192.168.50.19 (192.168.50.19) 56(84) bytes of data. 64 bytes from 192.168.50.19: icmp_req=1 ttl=64 time=0.627 ms 64 bytes from 192.168.50.19: icmp_req=2 ttl=64 time=0.305 ms 64 bytes from 192.168.50.19: icmp_req=3 ttl=64 time=0.326 ms ^C --- 192.168.50.19 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2000ms rtt min/avg/max/mdev = 0.305/0.419/0.627/0.148 ms - On the same (testing) computer (response side) it is very informational to observe traffic (in a separate terminal), fixed by tcpdump:
$ sudo tcpdump -i vboxnet0 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vboxnet0, link-type EN10MB (Ethernet), capture size 65535 bytes ... 18:41:01.740607 ARP, Request who-has 192.168.50.19 tell 192.168.50.1, length 28 18:41:01.741104 ARP, Reply 192.168.50.19 is-at 08:00:27:0f:13:6d (oui Unknown), length 28 18:41:01.741116 IP 192.168.50.1 > 192.168.50.19: ICMP echo request, id 8402, seq 1, length 64 18:41:01.741211 IP 192.168.50.19 > 192.168.50.1: ICMP echo reply, id 8402, seq 1, length 64 18:41:02.741164 IP 192.168.50.1 > 192.168.50.19: ICMP echo request, id 8402, seq 2, length 64 18:41:02.741451 IP 192.168.50.19 > 192.168.50.1: ICMP echo reply, id 8402, seq 2, length 64 18:41:03.741163 IP 192.168.50.1 > 192.168.50.19: ICMP echo request, id 8402, seq 3, length 64 18:41:03.741471 IP 192.168.50.19 > 192.168.50.1: ICMP echo reply, id 8402, seq 3, length 64 18:41:06.747701 ARP, Request who-has 192.168.50.1 tell 192.168.50.19, length 28 18:41:06.747715 ARP, Reply 192.168.50.1 is-at 0a:00:27:00:00:00 (oui Unknown), length 28
Expanding opportunities
Now briefly, in a nutshell, on how to make a full-fledged virtual interface that works only with its own traffic and does not interfere with the operation of the parent interface (what the full version of the module does in the archive). To do this, you must:
- Declare two separate protocol handlers (for ARP name resolution protocols and for the IP protocol itself):
// обработчик фреймов ETH_P_ARP int arp_pack_rcv( struct sk_buff *skb, struct net_device *dev, struct packet_type *pt, struct net_device *odev ) { ... return skb->len; }; static struct packet_type arp_proto = { __constant_htons( ETH_P_ARP ), NULL, arp_pack_rcv, // фильтр пртокола ETH_P_ARP (void*)1, NULL }; // обработчик фреймов ETH_P_IP int ip4_pack_rcv( struct sk_buff *skb, struct net_device *dev, struct packet_type *pt, struct net_device *odev ) { ... return skb->len; }; static struct packet_type ip4_proto = { __constant_htons( ETH_P_IP ), NULL, ip4_pack_rcv, // фильтр пртокола ETH_P_IP (void*)1, NULL }; - Both of them are sequentially registered in the module initialization function:
arp_proto.dev = ip4_proto.dev = priv->parent; // перехват только с родительского интерфейса dev_add_pack( &arp_proto ); dev_add_pack( &ip4_proto ); - Each of the installed filters should perform interface spoofing only for those frames whose recipient IP matches the interface IP ...
- Two separate handlers are convenient in that the headers of the ARP and IP frames have a completely different format, and it is necessary to allocate destination IPs in them differently (all the full code is shown in the example archive).
Using such a full-weight module, you can open, for example, two parallel SSH sessions on different interfaces (using different IP), which will actually use a single common physical interface in parallel:
$ ssh [email protected]
[email protected]'s password:
Last login: Mon Jul 16 15:52:16 2012 from 192.168.1.9
...
$ ssh [email protected]
[email protected]'s password:
Last login: Mon Jul 16 17:29:57 2012 from 192.168.50.1
...
$ who
olej tty1 2012-07-16 09:29 (:0)
olej pts/0 2012-07-16 09:33 (:0.0)
...
olej pts/6 2012-07-16 17:29 (192.168.50.1)
olej pts/7 2012-07-16 17:31 (192.168.56.1)
The last command shown (who) is already executed in an SSH session, that is, on that very remote host, to which two independent connections from two different subnets are fixed (the last two lines of output), which actually represent one host, but from the point of view its various network interfaces.
Further refinement
In preparing and debugging sample modules, to clarify the details, this (fairly recent) book was actively used: Rami Rosen: “Linux Kernel Networking: Implementation and Theory”, Apress, 650 pages, 2014, ISBN-13: 978-1-4302 -6196-4.
The author kindly provided it for free download before the book went on sale (2013-12-22). You can download it on this page .
Everyone who is interested in issues such as those discussed in this article will be able to find many ideas in this publication for the further development of the technique of their own use of network interfaces.
The archive of codes mentioned in the text for experiments and further development can be taken here or here .
