April 16, 2014 at 09:06What should we fix a bug that is “not”
So, we have a task: to fix a bug, the manufacturer from which it rejects, customers do not notice, but I want to live. There is a camera, the stream from it to UDP simply breaks hellishly, the stream to TCP works, but the connections are constantly torn (and every video breaks for 3-5 seconds). Everyone is guilty of the problem (both the camera and software), but both sides claim that they are all hurt, that is, the situation is normal: do you see the bug? not. But he is . Since software is updated much more often than the camera, it makes sense to edit a place that you don’t have to touch later. So, we will fix from the side of the camera.
Bridgehead research
First things first, take the latest firmware (in my case, firmware_TS38ABFG031-ONVIF-P2P-V2.5.0.6_20140126120110.bin), and find out what it is:
$ file firmware_TS38ABFG006-ONVIF-P2P-V2.5.0.6_20140126120110.bin firmware_TS38ABFG006-ONVIF-P2P-V2.5.0.6_20140126120110.bin: data $ du -b firmware_TS38ABFG006-ONVIF-P2P-V2.5.0.6_20140126120110.bin 15222724 firmware_TS38ABFG006-ONVIF-P2P-V2.5.0.6_20140126120110.bin $ xxd firmware_TS38ABFG006-ONVIF-P2P-V2.5.0.6_20140126120110.bin | head 0000000: 4649 524d 5741 5245 6481 db15 c447 e800 FIRMWAREd .... G .. 0000010: 0300 0000 1406 0000 b0f1 1b00 4c21 815d ............ L !.] 0000020: 5453 3338 4f45 4d41 4246 475f 4c49 4e55 TS38OEMABFG_LINU 0000030: 5800 0000 0000 0000 0000 0000 0000 0000 X ............... 0000040: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000050: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000060: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000070: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000080: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000090: 0000 0000 0000 0000 0000 0000 0000 0000 ................ $ binwalk firmware_TS38ABFG006-ONVIF-P2P-V2.5.0.6_20140126120110.bin DECIMAL HEX DESCRIPTION -------------------------------------------------- -------------------------------------------------- --- 1556 0x614 uImage header, header size: 64 bytes, header CRC: 0xB21E2C9F, created: Sun Sep 22 11:07:02 2013, image size: 1831280 bytes, Data Address: 0x80008000, Entry Point: 0x80008000, data CRC: 0x1F4EFBAB, OS : Linux, CPU: ARM, image type: OS Kernel Image, compression type: none, image name: "Linux-2.6.18_pro500-davinci_IPNC" 14468 0x3884 gzip compressed data, from Unix, last modified: Sun Sep 22 11:07:02 2013, max compression 1832900 0x1BF7C4 CramFS filesystem, little endian size 13389824 version # 2 sorted_dirs CRC 0xc832a8c3, edition 0, 7334 blocks, 2607 files
So, its format is unknown, the start label “FIRMWARE” evokes thoughts that it is something of its own, the presence of a kernel and cramfs inside uImage suggests that in fact it is something simple. The presence of the line TS38OEMABFG_LINUX suggests that this is something even reminiscent of some version of the archive.
Since now we just need to find where to look - just pull out the file system from there, and look for the culprit module:
$ dd if = firmware_TS38ABFG006-ONVIF-P2P-V2.5.0.6_20140126120110.bin bs = 1832900 skip = 1 of = cramfs 7 + 1 records received 7 + 1 records sent 13389824 bytes (13 MB) copied, 0.161755 s, 82.8 MB / s $ fakeroot cramfsck -x fs cramfs $ grep LIVE555 -R fs / The fs / opt / topsee / rtsp_streamer binary matches $ strings fs / opt / topsee / rtsp_streamer | grep TCP sendRTPOverTCP 12RTCPInstance sendRTPOverTCP failed, sock:% d, chn:% d is not a RTCP instance RTCPInstance :: RTCPInstance error: totSessionBW parameter should not be zero! RTP / AVP / TCP % sTransport: RTP / AVP / TCP; unicast; destination =% s; source =% s; interleaved =% d-% d / TCP; unicast Failed to create RTCP socket (port% d) MediaSession :: initiate (): unable to create RTP and RTCP sockets Failed to create RTCP instance Received RTCP "BYE" on " 18RTCPMemberDatabase
Hohoho! “SendRTPOverTCP failed, sock:% d, chn:% d” tells us that the code is translated with debugging prints, which means that the amount of work is reduced by orders of magnitude!
So, we have a module containing the desired error, a module with a bunch of debugging lines inside, which means that the process of sizing is greatly simplified.
Localization and fix problems
We load the module into a disassembler, look for the debug line using OverTCP, look for the code from it to print out => we found the sendRTPOverTCP function.
Looking through it, we see two calls to the send () function - one with 4 bytes, one with a buffer passed to the input. This means that we didn’t get the oldest version, but when we had already unified the buffer, but we hadn’t yet done sendDataOverTCP functions (for more details on the differences in the implementation, see <post .
Now the question arises how to fix the bug in binary form when there is practically no margin in place (there is no empty space inside the file.)
We go to the function above that calls sendDataOverTCP - sendPacket. Its code has not changed from version to version, it is essentially the same - foreach (streams) {sendDataOverTCP (packet, stream)}.
Fortunately, the code was generously crammed with debugging fprints, and that saves us! Here's how this loop looks in binary form:
loop_next_5FAE4:
LDR R4, [R4,#4]
CMP R4, #0
BEQ loc_5FB68
loop_body_5FAF0:
MOV R3, R4
MOV R1, R5
MOV R2, R7
MOV R0, R6
BL sendRTPOverTCP
CMP R0, #0
BGE loop_next_5FAE4
MOV R1, #0
LDR R2, =aS_10 ; "%s(): "
LDR R3, =aSendpacket ; "sendPacket"
MOV R0, #STDERR_FILENO
BL fprintf_0
LDRB R12, [R4,#0xC]
LDR R3, [R4,#8]
MOV R1, #7
LDR R2, =aSendrtpovert_0 ; "sendRTPOverTCP failed, sock: %d, chn: %"...
MOV R0, #STDERR_FILENO
STR R12, [SP,#0x350+var_350]
BL fprintf_0
......This is actually salvation! Just cutting out the debug piece gives us space in the amount of 12 instructions (for ARM, all instructions are exactly 4 bytes in length, and this is very good).
So, we have a place in 12 instructions to do something to improve the situation. But what? It will be very difficult to completely push the sendDataOverTCP code from the latest version here ...
Although stop. What for? I seem to have described in detail that even using the correct sendDataOverTCP form is still bad ... And if there is no difference, why not just wrap the call in makeSocketBlocking () .. makeSocketNonBlocking ()?
Indeed, if there is a place in the system buffer, send () will execute instantly. If there is no space, then their implementation of sendDataOverTCP will still stick (why it will stick and not fall out immediately with zero - see the previous post).
Excellent! By quickly rewinding from the fcntl function we find makeSocketBlocking and makeSocketNonBlocking, after which we draw what code should turn out:
loop_next_5FAE4:
LDR R4, [R4,#4]
CMP R4, #0
BEQ loc_5FB68
loop_body_5FAF0:
; Сперва сделаем сокет блокирующим
LDR R0, [R4,#8]
BL makeSocketBlocking
; Затем остаётся прежний вызов отправки
MOV R3, R4
MOV R1, R5
MOV R2, R7
MOV R0, R6
BL sendRTPOverTCP
; Сохраним результат вызова функции
STMFD SP!, {R0}
; Сделаем сокет снова неблокирующим
LDR R0, [R4,#8]
BL makeSocketNonBlocking
; Восстановим результат вызова функции отправки
LDMFD SP!, {R0}
; Зацикливание остаётся как и раньше
CMP R0, #0
BGE loc_5FAE4
; Ну и занопливаем остатки отладочной распечатки
NOP
NOP
NOP
NOP
NOP
NOPIn order to patch, either open the documentation for the arm and translate it in your mind, or write the code in a separate file and translate it (do not forget to set the exact addresses with the ORGs so that all transitions (BL / BGE / ITD) are counted correctly), and I just found the appropriate instructions in the code and based on them calculated the necessary opcodes (I’m editing ARM for the first time, sorry).
As a result, we get rtsp_streamer with a patch superimposed on it that protects the TCP stream from corruption.
Blast soldering, sober assembly
So, we have a new rtsp_streamer, and there is firmware ... bin in which it needs to be built. Well, it seems, everything is simple: you need to unpack cramfs, replace the file, pack it back, replace it inside bin:
$ fakeroot -s .fakeroot cramfsck -x repack cramfs $ cp rtsp_streamer repack / opt / topsee / $ fakeroot -i .fakeroot mkcramfs repack newcramfs $ dd if = firmware_TS38ABFG006-ONVIF-P2P-V2.5.0.6_20140126120110.bin of = firmware_new.bin bs = 1832900 count = 1 $ cat newcramfs >> firmware_new.bin
Pour the obtained firmware_new.bin into the camera ... 0 effect. The camera eats the firmware, but nothing happens. Unpleasant. So, you need to understand the format of this .bin.
Let's open it in the hex editor, and begin to kumack:
(0) "FIRMWARE" - 100% header, 8 bytes.
(8) 64 81 DB 15 - 4 bytes, the purpose is not clear. by smell - checksum
(12) 0x00E847C4 = 15222724 - yeah, 4 bytes, firmware size. we check _new.bin - no, the size has not changed, which means that it has nothing to do with it.
(16) 0x00000003 - 4 bytes xs what. header version can?
(20) 0x00000614 = 1556 - so, and this is the displacement to the core inside
(24) 0x001BF1B0 = 1831344 - and this is the size of the core (1831344 + 1556 = 1832900)
(28)4C 21 81 5D - hmm. again something similar to a checksum.
(32) “TS38OEMABFG_LINUX” and a bunch of zeros later - 100h bytes, obviously a place for the section name
(288) 0x001BF7C4 = 1832900 - yeah, offset to the next section
(292) 0x00CC5000 = 13389824 - yeah, section size
(296) “TS38OEMABFG_V2.5.0 .6 ”and a bunch of zeros are opaques. 100h bytes, clearly under the section name.
But there is no checksum in front of it O_O
(552-1556) - something of an unknown appearance.
So, roughly the point is clear. The size of our cramfs has not changed, which means that these are not sizes, and therefore, checksums.
We extract the kernel, and consider its checksum of 4 bytes in length:
$ dd if = firmware_TS38ABFG006-ONVIF-P2P-V2.5.0.6_20140126120110.bin of = kernel bs = 1 skip = 1556 count = 1831344 $ crc32 kernel 5d81214c
Well, well ... at offset 28 just 0x5D81214C. Lucky is the standard CRC32. It was lucky because, according to the standard tool, it can only count it. Otherwise, you would have to run the python and consider it already “more difficult” :)
So, the checksums we have are crc32. And what is the checksum of the original cramfs? .. 37499eef. So so so. At offset 552, 0x37499eef is just written. So, for some reason, the checksum AFTER the partition name is written for the file system. Well OK, what do we need, we are not proud. We update the plate:
(28) 0x5D81214C - crc32 of the kernel section
(552) 0x37499eef - crc32 of the FS section
(556-1556) - something of an unknown appearance
Recalculate crc32 newcramfs, use the editor to write it at offset 552 into the binar, fill it into the camera.
And ... nothing O_O. So, the flair did not disappoint - at offset 8 it really is crc32, but from what?
Here we act simply - we start brute-forcing.
$ python
>>> from zlib import crc32
>>> d = open ("firmware_TS38ABFG006-ONVIF-P2P-V2.5.0.6_20140126120110.bin", "r"). read ()
>>> hex (crc32 (d [12:]))
'-0x781aca29' # no
>>> hex (crc32 (d [12: 1556]))
'-0x6f8f1744' # no
>>> hex (crc32 (d [0: 8] + d [12: 1556])
'0x6d29f056' # no
>>> hex (crc32 (d [0: 8] + d [12:]))
'-0x652ac4fd' # no
>>> hex (crc32 (d [0: 8] + "\ 0 \ 0 \ 0 \ 0" + d [12: 1556]))
'0x15db8164' # Oops! It!Well done quickly. So, we update the plate:
(8) 0x15DB8164 - CRC32 header (the first 1556 bytes), first nullifying this field
So, right there in python we quickly recount crc32 from the header firmware_new.bin and write it in the hex editor to the beginning.
Pour into the camera ... She goes into reboot. And does not answer ... does not answer ... another brick ... Oh! Pings go! Fuuuh.
We take cam-resync.py, and again we stick our camera with a stick. And ... and the stream does not break! Right there, on the first try! Wiiii :)
Smeared on bread, and it’s already possible to eat, but something is not right
The previously mentioned Andrei Syomochkin, meanwhile, collected his firmware with the rtsp_streamer I corrected and uploaded it to one of the problem cameras, from which there were many breaks. As a result, testing showed that the stream does not break, however, video artifacts began, the same as when losing packets on UDP. Since I didn’t embed anything like this, I was curious what it was - I had to look into the code again. To begin with, look at strings, we have a bunch of debugging lines. "CheckBufferTimeout for% d seconds !!!", "buffered data more than% d ms, drop all the buffered data !!!".
Yeah! It turns out that manufacturers have made overflow protection! And if for some reason the synchronous send () is still hanging up more than necessary (by default - 1 second), it drops the excess. This protects against OOM and video lag if it does not fit into a thin channel. But the code had obviously not worked before due to the use of non-blocking sockets and send () 's.
After wrapping in Blocking ... NonBlocking - the code started working :)
However, there is a small problem: 1 sec is not enough. If the channel starts to fail, but is thick enough, then the probability of a drop becomes stronger. After any such drop, the video is restored only after the keyframe. Usually a keyframe is quite rare (every 5-10 seconds) ... And it turns out an unpleasant situation - if there was a failure, then you need to wait 5-10 seconds before the next keyframe so that the video is repaired. If you reduce the frequency of the keyframe, this automatically increases the amount of transmitted data, since the keyframes are quite thick, which means it increases the frequency of the channel dying. Vicious circle.
In general, I additionally increased the buffering timeout to 10 seconds - this should not be enough to catch OOM, but enough to calmly go through retransmits and lags on non-superstable channels.
By the way, the "cunning" treatment algorithm
In a last article, I promised to tell how easy it is to fix the situation. The solution is just like felt boots - since we send RTP packets, we each have a timestamp. It is enough to check in sendRTPorRTCPPacketOverTCP before sending the packet lifetime, and if it is less than the configured one (I still think that 1 sec is not enough on TCP, it should be 6-10 sec) then send it, otherwise just do not send it silently.
Automate assembly-disassembly
The only thing left is to automate the assembly and disassembly of the firmware.
unpack.sh
Hidden text
fw=${1?Please give firmware bin as argument}
if [ -e $fw.unpack ]; then
echo "Already exists: $fw.unpack"
exit 1
fi
# Check format
if [ "$(dd if=$1 bs=8 count=1 2>/dev/null)" != "FIRMWARE" ]; then
echo "Wrong file"
exit 1
fi
mkdir -p $fw.unpack
echo "Extract header..."
dd if=$1 of=$fw.unpack/00header bs=1556 count=1 2>/dev/null
echo "Extract kernel..."
ksize=$(dd if=$1 bs=1 count=4 skip=24 2>/dev/null | perl -e 'print unpack("l", <>);')
dd if=$1 of=$fw.unpack/01kernel bs=1 skip=1556 count=$ksize 2>/dev/null
echo "Extract filesystem..."
foff=$(dd if=$1 bs=1 count=4 skip=288 2>/dev/null | perl -e 'print unpack("l", <>);')
dd if=$1 of=$fw.unpack/02cramfs bs=$foff skip=1 2>/dev/null
echo "Unpack filesystem..."
cd $fw.unpack
fakeroot -s .fakeroot cramfsck -x root 02cramfs
chmod +r -R root/
echo "Done"
Since we don’t know how to fill in a piece of the header, we cannot collect arbitrary ones so as not to kill the camera, so we save all the pieces as is.
Since we have block and other devices inside the firmware, we cannot work with it from a simple user. But then fakeroot comes to the rescue, which can save state to an external file. Therefore, unpack using fakeroot.
However, there is a micro-problem with it - as a result, the file should be available for reading to the current user. If you are a “real” root, then you can easily read the file, even if it is “chmod -r”. But fakeroot breaks on such a file. Therefore, immediately after unpacking, I change the read permissions for all files. BUT, the correct access rights are saved in the fakeroot status dump, so the reassembly is a great success.
The rest of the unpacking does not have any interesting points.
pack.sh
Hidden text
dir=${1?Please give path to a directory with unpacked firmware}
nfw=${2?Please give name for a newly packed firmware}
if [ ! -e $dir ]; then
echo "Directory not exists: $dir"
exit 1
fi
if [ -e $nfw ]; then
echo "Firmware already exists: $nfw"
exit 1
fi
# repack cramfs
if [ ! -e $dir/02cramfs.bak ]; then
mv $dir/02cramfs $dir/02cramfs.bak 2>/dev/null
fi
fakeroot -i $dir/.fakeroot mkcramfs $dir/root/ $dir/02cramfs
# construct new firmware
dd if=$dir/00header bs=1556 of=$nfw conv=notrunc 2>/dev/null
# remove old header crc32
dd if=/dev/zero bs=1 seek=8 count=4 of=$nfw conv=notrunc 2>/dev/null
# save kernel size
if [ $(stat -c %s $dir/01kernel) -ge 2097152 ]; then
echo "WARN: size of kernel is more than 0x200000. FW probably will not flash"
fi
perl -e 'print pack("l", -s "'$dir/01kernel'")' | dd bs=1 seek=24 count=4 of=$nfw conv=notrunc 2>/dev/null
# save kernel crc32
crc32 $dir/01kernel | perl -e 'print pack("l", oct("0x".<>));' | dd bs=1 seek=28 count=4 of=$nfw conv=notrunc 2>/dev/null
# save fs offset
perl -e 'print pack("l", 1556+(-s "'$dir/01kernel'"))' | dd bs=1 seek=288 count=4 of=$nfw conv=notrunc 2>/dev/null
# save fs size
if [ $(stat -c %s $dir/02cramfs) -lt 8388608 ]; then
echo "WARN: size of filesystem is less than 0x800000. FW probably will not flash"
fi
if [ $(stat -c %s $dir/02cramfs) -ge 15728640 ]; then
echo "WARN: size of filesystem is more than 0xF00000. FW probably will not flash"
fi
perl -e 'print pack("l", -s "'$dir/02cramfs'")' | dd bs=1 seek=292 count=4 of=$nfw conv=notrunc 2>/dev/null
# save fs crc32
crc32 $dir/02cramfs | perl -e 'print pack("l", oct("0x".<>));' | dd bs=1 seek=552 count=4 of=$nfw conv=notrunc 2>/dev/null
# save full FW size
perl -e 'print pack("l", 1556+(-s "'$dir/02cramfs'")+(-s "'$dir/01kernel'"))' | dd bs=1 seek=12 count=4 of=$nfw conv=notrunc 2>/dev/null
# Update header crc32
crc32 $nfw | perl -e 'print pack("l", oct("0x".<>));' | dd bs=1 seek=8 count=4 of=$nfw conv=notrunc 2>/dev/null
# concat rest
cat $dir/01kernel >> $nfw
cat $dir/02cramfs >> $nfw
echo "Done"
But the packaging is already a bit more complicated. We need to pack back cramfs, update the length of individual files and the total length in the header; recount checksums, including the heading, and only then merge everything together.
In general, the camera checks the boundary values myself, however, for convenience, I added a check on the boundaries of the file system and kernel sizes, so that if during assembly its sizes creep out, get a warning and remove extra tails from the firmware.
Result of labor
So, I collected the following patched firmware of the latest version 2.5.0.6:
- firmware_TS38ABFG006-ONVIF-P2P-V2.5.0.6_20140126120110-TCPFIX.bin
- firmware_TS38CD-ONVIF-P2P-V2.5.0.6_20140126121011-TCPFIX.bin
- firmware_TS38HI-ONVIF-P2P-V2.5.0.6_20140126121444-TCPFIX.bin
- firmware_TS38LM-ONVIF-P2P-V2.5.0.6_20140126121913-TCPFIX.bin
- firmware_HI3518C-V4-ONVIF-V2.5.0.6_20140126124339-TCPFIX.bin
If you suddenly need a fix on any other module of the same manufacturer - write in the comments, I will see if possible.
ps: everyone who needs something else from these firmware has posted the scripts in a more convenient place - on the github .
