August 6, 2015 at 15:23How We Overcome USB Data Transfer
How it all began
So, in our wonderful Bercut-MMT device (based on PXA320 and GNU / Linux) there is no less remarkable OTDR module (based on STM32 and NutOS), which is a pulsed optical reflectometer. This bundle works as follows: the user clicks on the screen on various elements of the UI, a little magic happens in the device, and the user's wishes are transformed into commands of the “duration 300” type, which go to the measuring module. Specifically, this command sets the measurement duration to 300 seconds. The module is connected to the device via USB, CDC-ACM is raised to transmit commands over USB.
Briefly - CDC-ACM allows you to emulate a serial port via USB. So for the top level, our measuring module in the system is available as / dev / ttyACM0. CDC-ACM is used to transmit commands to the module or read the current settings / status of the module. The USB Bulk interface was used to transmit the trace itself, which devoted all its time to only one thing - the transfer of trace data from the module to the device, as a binary data stream. At some point, we noticed that the trace did not come to us completely. So we discovered that USB can lose data.
Schematically, it looked like this:
b5-cardifaced is a daemon that accepts D-Bus commands and sends them to the card via the CDC-ACM interface. The execution result is sent back via D-BUS.
usbgather is a small program that works on the basis of libusb and is engaged in the fact that it scoops out a trace from the module via USB Bulk and issues it to stdout.
Crutches and bicycles
We sat down and thought - we need to understand whether the entire trace has come to us to be able to skip incomplete reflectograms. We began to come up with various tricky headlines, checksums, etc. Then we realized that we were inventing TCP. And then a strong-willed decision was made instead of USB Bulk to get TCP / IP on top of CDC-EEM. Why is CDC-EEM? Because CDC-EEM allows you to most easily use USB as a transport to transmit network traffic. On the device itself, there is CDC-ECM support in the kernel, and we use NutOS as the operating system modules and NutOS had support for CDC-EEM and TCP / IP stack.
Fix-long life 3 months
It would seem that nothing portended trouble. Raised CDC-EEM, configured IP addresses. Ping? Have a ping! Hooray. We changed the mechanism for transferring data from USB Bulk to transferring data through a TCP socket. Happiness was about to come, but then suddenly during testing the network fell shouting to dmesg about its difficult life, our crooked hands and the queue for sending for our network interface that got up a stake. More or less like this:
[ 118.289339] ------------[ cut here ]------------
[ 118.293978] WARNING: at net/sched/sch_generic.c:258 dev_watchdog+0x184/0x298()
[ 118.301163] NETDEV WATCHDOG: usb2 (cdc_eem): transmit queue 0 timed out
[ 118.307726] Modules linked in: cdc_eem usbnet cdc_acm wm97xx_ts ucb1400_ts ipv6 cards button pmmct
[ 118.318671] [] (unwind_backtrace+0x0/0xec) from [] (warn_slowpath_common+0x4c)
[ 118.328017] [] (warn_slowpath_common+0x4c/0x7c) from [] (warn_slowpath_fmt+0x)
[ 118.337536] [] (warn_slowpath_fmt+0x30/0x40) from [] (dev_watchdog+0x184/0x29)
[ 118.346552] [] (dev_watchdog+0x184/0x298) from [] (run_timer_softirq+0x18c/0x)
[ 118.355731] [] (run_timer_softirq+0x18c/0x26c) from [] (__do_softirq+0x84/0x1)
[ 118.364819] [] (__do_softirq+0x84/0x114) from [] (irq_exit+0x44/0x64)
[ 118.372959] [] (irq_exit+0x44/0x64) from [] (asm_do_IRQ+0x74/0x94)
[ 118.380843] [] (asm_do_IRQ+0x74/0x94) from [] (__irq_svc+0x44/0xcc)
[ 118.388792] Exception stack(0xc0425f78 to 0xc0425fc0)
[ 118.393819] 5f60: 00000001 c606b300
[ 118.401958] 5f80: 00000000 60000013 c0424000 c0428334 c0454bac c0428328 a0022438 69056827
[ 118.410100] 5fa0: a0022368 00000000 c0425f98 c0425fc0 c002b04c c002b058 60000013 ffffffff
[ 118.418232] [] (__irq_svc+0x44/0xcc) from [] (default_idle+0x34/0x40)
[ 118.426367] [] (default_idle+0x34/0x40) from [] (cpu_idle+0x54/0xb0)
[ 118.434425] [] (cpu_idle+0x54/0xb0) from [] (start_kernel+0x28c/0x2f8)
[ 118.442653] [] (start_kernel+0x28c/0x2f8) from [] (0xa0008034)
[ 118.450180] ---[ end trace d7e298087ff4c373 ]---
If briefly, literally in a nutshell - each network interface has a timer that records the time from each last data sending and if it exceeded a certain interval - we see this message. Then everything depends on the driver - some work fine after that, some do not.
Root of evil
All of the above was compounded by messages in dmesg about unknown link cmd. We added more debug and found out that we received an echo request response to the USB host, which we did not send.
When nothing works, it's time to read the documentation. So we got a dock on CDC-EEM, but not from anywhere, but directly from usb.org. It turns out that the first EEM packet is not only a bunch of data, but also an EEM header that contains the type of packet (control or data) and the length of the data. And yes, CDC-EEM has its own echo request / echo response.
But our happiness would not be complete if it weren’t for one more detail - when receiving service packets, the module would hang. Tightly. Together with CDC-ACM.
In our USB module, it was configured so that the transmission went in packets of 64 bytes. Accordingly, one Ethernet packet was beaten on N packets of 64 and transmitted via USB. Like this:
After a very long study of the situation, we came to the conclusion that this is what happens: we lose part of the EEM package (yes, USB does not guarantee delivery). But we read the length from the header and rely on it. Accordingly, we subtract N lost bytes from the next packet, and perceive the following data as the beginning of a new EEM packet and interpret the first 2 bytes as a header. And there may be anything. Up to cocked into 1 bit, which indicates that this is a service package. In very bad cases, we catch data that, when interpreted as an EEM header, gives us a huge echo response. Much more than our RAM. So we realized that our implementation of usbnet in NutOS requires serious improvements.
More checks good and different
In the process of picking usbnet in NutOS, it was found out that the current version is not at all ready to receive service packets. From the word at all. We made a new option, which became able to correctly process service packages, namely: we looked at the type of package, because we are obliged to answer echo by standard; checked the length - if it is more than MTU - then we obviously caught garbage. We also found oddity in the function that starts the data transfer via endpoint: we checked whether the zero endpoint was busy right now, and if it was busy, that was all. The caller of this function always believed that data transfer was started, and often it turned out that not. As a result, we lost data, and in both directions.
There were wars with the TCP socket - sometimes the data was not transmitted and we did not see why. I don’t know what led the NutOS developers, but many functions that had an int returned in any unclear situation returned -1. Some of them wrote down the actual error code in the socket information, some did not. So I had to work on dragging the return codes from the bottom, like the function of sending data from the network card, to the very top - functions like NutTcpDeviceWrite? (). After that, we were able to see where the plug happened.
Then there were all sorts of finishing and adjusting timeouts in the socket, adding static records to the ARP tables on the module and on the device itself: there are only 2 devices in our network: the device and the module, there is no sense in obsolete entries in the ARP table.
Summary
Our map now has a small TCP server, which serves to transfer data from the card to the device. Before starting the measurements, the TCP server starts in the card and the card starts to wait for incoming connections. After the client connects to the TCP server, the card starts measuring and sends the results to the device via the TCP server.
Now the operation of the device with the module is schematically something like this:
Actors:
b5-cardifaced - the same as before - transfers commands from the D-Bus to the card and sends the result back to the D-Bus;
nc - actually netcat, reads trace data from the socket and sends it to stdout for further processing.
After all these adventures, we now have a network reflectometer. Networking, however, is not 100% complete - control is via CDC-ACM, and data collection from the module is done via TCP / IP via CDC-EEM. We still have a slight data loss, but due to the use of TCP / IP at the output, we always get a full trace. We learned a lot about USB in general and CDC-EEM in particular, and I began to love USB a little less than before.
The load test showed that our module based on STM32F103 can pump 220 kilobytes of data per second over TCP / IP over CDC-EEM, while the module is doing useful work at this time and USB is working without DMA.
