July 1, 2014 at 20:44How to patch 11 different firmware and not go crazy with a variety
- Tutorial
If any operation turns into a routine, automate it. Even if you spend more time, then you were not engaged in a routine, but in an interesting business. It was under this sign that, instead of just patching the new 11 rtsp_streamer versions for cameras from TopSee , I decided to draw an autopatch. I consider python to be the ideal language for any knee-high products - succinctly enough, quite tough on readability (although I still manage to make it unreadable). In general, now I’ll tell you how to learn how to draw auto patchers in one evening using a stick and a rope.So, the main requirements for scripts on the knee are maximum compliance with expectations. It must either work or report that something is not right. The main error of such scripts is any actions without checking for compliance with expectations. Since otherwise you may not notice that something has changed and human intervention is required.
So, let's remember what we did last time , and go through the entire path again manually:
1) We load the file into the disassembler
2) We find the function fctnl
3) We go through the calls in search of using fcntl with O_NONBLOCK - we find two functions, makeSocketBlocking and makeSocketNonBlocking
Hidden text
Boolean makeSocketNonBlocking(int sock) {
#if defined(__WIN32__) || defined(_WIN32)
unsigned long arg = 1;
return ioctlsocket(sock, FIONBIO, &arg) == 0;
#elif defined(VXWORKS)
int arg = 1;
return ioctl(sock, FIONBIO, (int)&arg) == 0;
#else
int curFlags = fcntl(sock, F_GETFL, 0);
return fcntl(sock, F_SETFL, curFlags|O_NONBLOCK) >= 0;
#endif
}
Boolean makeSocketBlocking(int sock) {
#if defined(__WIN32__) || defined(_WIN32)
unsigned long arg = 0;
return ioctlsocket(sock, FIONBIO, &arg) == 0;
#elif defined(VXWORKS)
int arg = 0;
return ioctl(sock, FIONBIO, (int)&arg) == 0;
#else
int curFlags = fcntl(sock, F_GETFL, 0);
return fcntl(sock, F_SETFL, curFlags&(~O_NONBLOCK)) >= 0;
#endif
}
4) We are looking for the sendPacket () function in the code (from the past we know that debug printfs are written in it, by which it is not difficult to find)
Hidden text
Boolean RTPInterface::sendPacket(unsigned char* packet, unsigned packetSize) {
Boolean success = True; // we'll return False instead if any of the sends fail
for (tcpStreamRecord* streams = fTCPStreams; streams != NULL;
streams = streams->fNext) {
if (!sendRTPOverTCP(packet, packetSize, streams->fStreamSocketNum, streams->fStreamChannelId)) {
printf("%s(): ", "sendPacket");
printf("sendRTPOverTCP failed, sock: %d, chn: %d\r\n", streams->socket, streams->fStreamChannelId);
success = False;
}
}
return success;
}
5) Patch function
Hidden text
Boolean RTPInterface::sendPacket(unsigned char* packet, unsigned packetSize) {
Boolean success = True; // we'll return False instead if any of the sends fail
for (tcpStreamRecord* streams = fTCPStreams; streams != NULL;
streams = streams->fNext) {
makeSocketBlocking(streams->socket);
Boolean res = sendRTPOverTCP(packet, packetSize, streams->fStreamSocketNum, streams->fStreamChannelId);
makeSocketNonBlocking(streams->socket);
if (!res) {
success = False;
}
}
return success;
}
So, this is all we need to automate so that it launches and receives. Moreover, in different versions of the firmware fcntl is used somewhere, fcntl64 somewhere; depending on the compiler options, different registers are used, other small differences in the translated code are observed.
So, let's start writing our script. Clear business, since we will patch different versions, the name of the file to be patched must be passed as an argument. So the script started:
import sys
fname = sys.argv[1]
Since we make a script for ourselves, the files are small (~ a meter in weight), there is a lot of memory, so we won’t bother running around the file - we will load everything immediately into memory:
f = open(fname, "r+b")
f.seek(0, 2)
size = f.tell()
f.seek(0, 0)
fw = f.read(size)
f.close()
Let's start looking for makeSocketBlocking / makeSocketNonBlocking functions. The fcntl functions are used a lot where, so the real clue is O_NONBLOCK (= 0x800). On the other hand, these are library functions that no one touches, they go in a row, you can just find functions "as is":
.text:0003C554 makeSocketBlocking
.text:0003C554 10 40 2D E9 STMFD SP!, {R4,LR}
.text:0003C558 03 10 A0 E3 MOV R1, #F_GETFL ; cmd
.text:0003C55C 00 20 A0 E3 MOV R2, #0
.text:0003C560 00 40 A0 E1 MOV R4, R0
.text:0003C564 32 39 FF EB BL fcntl
.text:0003C568 04 10 A0 E3 MOV R1, #F_SETFL ; cmd
.text:0003C56C 02 2B C0 E3 BIC R2, R0, #O_NONBLOCK
.text:0003C570 04 00 A0 E1 MOV R0, R4 ; fd
.text:0003C574 2E 39 FF EB BL fcntl
.text:0003C578 00 00 E0 E1 MVN R0, R0
.text:0003C57C A0 0F A0 E1 MOV R0, R0,LSR#31
.text:0003C580 10 80 BD E8 LDMFD SP!, {R4,PC}
.text:0003C580 ; End of function makeSocketBlocking
.text:0003C584 makeSocketNonblocking ; CODE XREF: sub_43524+40p
.text:0003C584 ; .text:00043608p ...
.text:0003C584 10 40 2D E9 STMFD SP!, {R4,LR}
.text:0003C588 03 10 A0 E3 MOV R1, #3 ; cmd
.text:0003C58C 00 20 A0 E3 MOV R2, #0
.text:0003C590 00 40 A0 E1 MOV R4, R0
.text:0003C594 26 39 FF EB BL fcntl
.text:0003C598 04 10 A0 E3 MOV R1, #4 ; cmd
.text:0003C59C 02 2B 80 E3 ORR R2, R0, #0x800
.text:0003C5A0 04 00 A0 E1 MOV R0, R4 ; fd
.text:0003C5A4 22 39 FF EB BL fcntl
.text:0003C5A8 00 00 E0 E1 MVN R0, R0
.text:0003C5AC A0 0F A0 E1 MOV R0, R0,LSR#31
.text:0003C5B0 10 80 BD E8 LDMFD SP!, {R4,PC}
.text:0003C5B0 ; End of function makeSocketNonblocking
We copy the opcodes on this forehead, and replace all the parameters that differ from the version, address, compiler settings, etc.
blockMask = """
; makeSocketBlocking
mm
?? ?? 2D E9 ; STMFD SP!, ....
03 10 A0 E3 ; MOV R1, #3 ; cmd
00 20 A0 E3 ; MOV R2, #0
00 ?? A0 E1 ; MOV Rx, R0 ; save fd
?? ?? FF EB ; BL fcntl
04 10 A0 E3 ; MOV R1, #4 ; cmd
02 2B C0 E3 ; clear O_NONBLOCK
?? 00 A0 E1 ; restore fd
?? ?? FF EB ; BL fcntl
00 00 E0 E1 ; MVN R0, R0
A0 0F A0 E1 ; MOV R0, R0,LSR#31
?? ?? BD E8 ; LDMFD SP!, ....
mm
; makeSocketNonblocking
?? ?? 2D E9
03 10 A0 E3
00 20 A0 E3
00 ?? A0 E1
?? ?? FF EB
04 10 A0 E3
02 2B 80 E3
?? 00 A0 E1
?? ?? FF EB
00 00 E0 E1
A0 0F A0 E1
?? ?? BD E8
"""
I additionally put labels (“mm”) to the beginning of the functions; all non-opcodes commented out ";" and left it in the code as it is (for the future, to remember what is what).
Now we need to turn this line into a mask for the search. Of course, manual search is not at all a pleasure, therefore, we will search through regexps, since they are debugged and optimized, so do not play around. And we read the entire file into memory as one large line - which is also convenient. Therefore, we write a function that converts this mask to regexp:
import re
def maskToRegex(mask):
mask = re.sub( ";.*$", "", mask, flags=re.MULTILINE)
mask = re.sub( "\s+", "", mask, flags=re.MULTILINE)
masks = re.findall( "..", mask)
rgx = ""
for m in masks:
if m == "??":
rgx += "."
elif m == "mm":
rgx += "()"
else:
rgx += "\\x"+m
return rgx
The behavior is simple - delete all comments (everything from; to the end of the line), delete all spaces, cut the remaining characters into pairs, and look - if this is ?? - then replace it with a dot (any character), if “mm” - insert a label for which we will remember the position, otherwise we generate a symbol code (assigning "\ x" to this pair).
So, we have a mask and we can get a regexp from it, let's finally find these functions:
### 1. Find offset of makeSocketBlocking and makeSocketNonblocking
makeBlock = None
makeNonBlock = None
for find in re.finditer(maskToRegex(blockMask), fw, re.DOTALL):
if makeBlock is None and makeNonBlock is None:
makeBlock = find.start(1)
makeNonBlock = find.start(2)
print "Found makeNonBlock at ", hex(makeNonBlock)
print "Found makeBlock at ", hex(makeBlock)
else:
print "Non-unqiue makeNonBlock/makeBlocking functions found"
break
if makeBlock is None or makeNonBlock is None:
print "makeNonBlock/makeBlocking functions not found"
There are two important points in this code. First, global, meeting expectations. We verify that only one block of code is covered by this mask, and that it really does. At the time of debugging, we add a listing of the displacements of the found places. Secondly, it is important not to forget about re.DOTALL, so that any byte falls under the dot, we work with a binary string.
So now we need to find the sendPacket function. Let's look into the disassembled:
.text:0006C9A0 SendPacket ; CODE XREF: sub_69FB8+144p
.text:0006C9A0 ; .text:0006D144p ...
.text:0006C9A0 F0 4F 2D E9 STMFD SP!, {R4-R11,LR}
.text:0006C9A4 00 60 A0 E1 MOV R6, R0
.....
.text:0006CB78 C3 FF FF 1A BNE loc_6CA8C
.text:0006CB7C E2 FF FF EA B loc_6CB0C
.text:0006CB7C ; End of function SendPacket
.text:0006CB80 BC 92 0A 00 off_6CB80 DCD aS_10 ; DATA XREF: SendPacket+7Cr, SendPacket+F0r
.text:0006CB80 ; "%s(): "
.text:0006CB84 E4 6B 0A 00 off_6CB84 DCD aSendpacket ; DATA XREF: SendPacket+80r, SendPacket+F4r
.text:0006CB84 ; "sendPacket"
.text:0006CB88 00 9B 0A 00 off_6CB88 DCD aSendrtpovert_0 ; DATA XREF: SendPacket+98r
.text:0006CB88 ; "sendRTPOverTCP failed, sock: %d, chn: %"...
.text:0006CB8C 2C 9B 0A 00 off_6CB8C DCD aRemovestreamso ; DATA XREF: SendPacket+110r
.text:0006CB8C ; "removeStreamSocket, sock: %d, chnid: %d"...
Yeah, the link to this line lies immediately after the function code. So, to find the function, we need to: find the line with the name of the function (thanks to the debugging macros, it lies as a separate independent line), convert the offset to an address, find this address, in the code, rewind this address up to the STMFD SP! Instruction, { ..., LR}.
The question immediately arises - finding the line is not a problem, but how to convert the offset in the file to a virtual address? Do not parse the file manually. Almighty Google comes to the rescue: there is a pyelftools package. So "pip install pyelftools", and we smoke the attached documentation. There is something there is nothing useful. OK, we stupidly go into the elffily.py file and see what is delicious there. We find there a function that does the inverse task - it finds the offset at the virtual address:
def address_offsets(self, start, size=1):
""" Yield a file offset for each ELF segment containing a memory region.
A memory region is defined by the range [start...start+size). The
offset of the region is yielded.
"""
end = start + size
for seg in self.iter_segments():
if (start >= seg['p_vaddr'] and
end <= seg['p_vaddr'] + seg['p_filesz']):
yield start - seg['p_vaddr'] + seg['p_offset']
Everything is clear, with a flick of the wrist these trousers turn into elegant shorts:
# удаляем f.close(), и на её месте пишем
from elftools.elf.elffile import ELFFile
f.seek(0, 0)
Elf = ELFFile(f)
def offToVA(offset):
for k in Elf.iter_segments():
if offset >= k['p_offset'] and offset <= k['p_offset']+k['p_filesz']:
return k['p_vaddr']+(offset-k['p_offset'])
Now we can find the line, and the place of its use:
s = "sendPacket"
## Find string itself
offStr = re.findall(s+"\x00", fw)
if len(offStr)==1:
offStr = re.search(s+"\x00", fw)
offStrVA = offToVA(offStr.start(0))
print "offStr["+s+"] =", hex(offStrVA)
elif len(offStr)==0:
print s, "string marker not found"
else:
print "Too many", s, "string markers found"
Then I lazily used another method of checking expectations - I just try to find everything first, and I think that everything is ok if it is one. Otherwise, I report an error. The method is also not convenient, + unnecessary searches, but it’s more clear to me personally, so I won’t repeat experiments with finditer.
Well, now find the offset where the string is used:
## Find offset to string
reStrLink = "\\x%02X\\x%02X\\x%02X\\x%02X" % (
(offStrVA)%256,
(offStrVA/256)%256,
(offStrVA/256/256)%256,
(offStrVA/256/256/256)%256 )
offLink = re.findall(reStrLink, fw)
if len(offLink)==1:
offLink = re.search(reStrLink, fw)
offLink = offLink.start(0)
if DEBUG: print "offLink["+s+"] = ", hex(offToVA(offLink))
return offLink
else:
print "Can't find usage of", s
As you might have guessed, instead of
s = "sendPacket"actually written def findStringLink(s):, because we need to find the function more than once. Well, now we need to find another beginning of the function - we go back 4 bytes in search of STMFD SP !, {..., LR}. Due to laziness, I limited myself to searching for STMFD SP !, {...} (so as not to analyze the bits). The reason is if you do not find the beginning of the function, then the search still breaks down, which I will learn about, and then I can already decide how to fix it better.
# Find previous function begin offset (nearest STMFD SP!, {...} instruction)
def findFuncBegin(offset, maxLen = 0x1000):
maxStart = max(0, offset-maxLen)
offset -= 4
while offset > maxStart:
if fw[offset+2:offset+4]=="\x2D\xE9":
return offset
offset -= 4
return None
So, we have everything we need, finally find the sendPacket function:
### 2. Find sendPacket function
sendPacketEnd = findStringLink("sendPacket")
sendPacketStart = findFuncBegin(sendPacketEnd)
if sendPacketStart is not None:
print "sendPacketStart = ", hex(offToVA(sendPacketStart))
else:
print "Can't find start of sendPacket"
Now we are close to the most interesting - we need to find a loop inside sendPacket, and make sure that this is it. We make up the mask as we learned earlier:
sendLoopMask = """
?? 00 00 EA ; B loopBody
; ---------------------------------------------------------------------------
mm ;loopNext ; CODE XREF: SendPacket+74j
?? ?? ?? E5 ; LDR R4, [R4,#4] (or R5)
00 00 ?? E3 ; CMP R4, #0 (or R5)
?? 00 00 0A ; BEQ loc_6CA74
;loopBody ; CODE XREF: SendPacket+4Cj
mm ; ; SendPacket+D0nj
?? ?? A0 E1 ; MOV R3, R4 (or R5)
?? ?? A0 E1 ; MOV R1, R5
?? ?? A0 E1 ; MOV R2, R7
?? ?? A0 E1 ; MOV R0, R6
mm
?? ?? ?? EB ; BL SendRTPOverTCP
00 00 50 E3 ; CMP R0, #0
F5 FF FF AA ; BGE loopNext
"""
But we need to check that the BL inside the loop is to SendRTPOverTCP, otherwise it either changed the function strongly, or the loop was already patched by us, so we also find SendRTPOverTCP:
### 3. Find sendRTPOverTCP function
sendRTPOverTCPStart = findFuncBegin(findStringLink("sendRTPOverTCP"))
if sendRTPOverTCPStart is not None:
print "sendRTPOverTCPStart = ", hex(offToVA(sendRTPOverTCPStart))
else:
print "Can't find start of sendPacket"
So, but we must be able to check whether the link is there. All transitions in ARM are not absolute, but relative, plus addresses are specified by calculating +2 instructions from the current, and even given in instruction quanta (that is, divided by 4). In general, the opcode is drawn something like this :
TargetAddr = Opcode*(4 bytes/word) + CurAddr + 8 As a result, we get such functions to calculate the address to which a certain instruction refers, and to calculate which operand of the instruction will be to a certain address to go to:
def BinArg(off):
return ord(fw[off])+ord(fw[off+1])*256+ord(fw[off+2])*256*256
def ArgToBin(arg):
return chr(arg%256)+chr(arg/256%256)+chr(arg/256/256%256)
def cmdTargetOffset(cmdoff):
d1 = BinArg(cmdoff)
if d1 >= 0x800000: d1 -= 0x1000000
return (cmdoff+(d1+1)*4+4)
def cmdTargetArg(cmdoff, target):
d1 = (target - (cmdoff+4))/4 - 1
if d1 < 0: d1 += 0x1000000
return d1
Now that the bricks are laid, we’ll build another bulkhead:
### 4. find loop in sendPacket
sendPacketLoopRx = maskToRegex(sendLoopMask)
sendPacketLoop = re.findall(sendPacketLoopRx, fw, re.DOTALL)
if len(sendPacketLoop)==1:
sendPacketLoop = re.search(sendPacketLoopRx, fw, re.DOTALL)
sendPacketLoopNext = sendPacketLoop.start(1)
sendPacketLoopBL = sendPacketLoop.start(3)
sendPacketLoop = sendPacketLoop.start(2)
if DEBUG: print "sendPacket loop at ", hex(offToVA(sendPacketLoop))
elif len(sendPacketLoop)==0:
print "Loop inside sendPacket not found"
else:
print "Non-unqiue loops masks for sendPacket found"
## 4.1. check that loop link is really to sendRTPOverTCP
if cmdTargetOffset(sendPacketLoopBL) != sendRTPOverTCPStart:
print "Loop's first call is not sendRTPOverTCP"
if cmdTargetArg(sendPacketLoopBL, sendRTPOverTCPStart)!=BinArg(sendPacketLoopBL):
print "BUG! cmdTargetArg inconsistent with cmdTargetOffset!"
if ArgToBin(cmdTargetArg(sendPacketLoopBL, sendRTPOverTCPStart)) != fw[sendPacketLoopBL:sendPacketLoopBL+3]:
print "BUG! ArgToBin inconsistent with cmdTargetOffset!"
So, we found a loop, checked that BL in it refers specifically to sendRTPOverTCP, and at the same time checked the address functions that they are consistent. This all allows you to protect yourself from typos and get rid of unnecessary coverage by function tests - we are writing a script, not a software product, therefore, only the necessary maximum body movements.
But in order to patch, we need to add 6 new instructions, and for this we need a place. Since we have two extra prints in the cycle, as we already know, we use them. The first printf of 5 instructions, so you have to overwrite both. To do this, they will have to find and make sure that they are:
### 5. Find next two printfs
printf1 = sendPacketLoopBL+4
while printf1 < sendPacketEnd:
if fw[printf1+3]=="\xEB":
printfStart = cmdTargetOffset(printf1)
break
printf1 += 4
printf1 += 4
printf2 = printf1 + 4
while printf2 < sendPacketEnd:
if fw[printf2+3]=="\xEB":
if cmdTargetOffset(printf2) != printfStart:
print "ERROR! After loop not two printfs!"
break
printf2 += 4
printf2 += 4
if (printf1-sendPacketLoop)/4-7 != 5:
print "WARN! First printf not 5 instructions"
if (printf1-sendPacketLoop)/4-7 > 5:
printf2 = printf1 # no need to cleanup 2nd printf
Again, we go the simplest way - take the two closest BLs after BL sendRTPOverTCP, check that they both refer to the same function, and check their sizes. If something goes wrong - immediately curse. If everything meets expectations, then everything is fine.
Well, now put it all together, add three buckets of dope, and generate a patch:
### 6. Generate new loop body
PatchSendPacket = ""
## 6.1. LDR R0, Socket(Rx#8)
ldrSock = "\x08\x00"+fw[sendPacketLoopNext+2]+"\xE5"
PatchSendPacket += ldrSock
## 6.2. BL makeSocketBlocking
tgtSocketBlock = cmdTargetArg(sendPacketLoop+len(PatchSendPacket), makeBlock)
PatchSendPacket += ArgToBin(tgtSocketBlock)+"\xEB"
## 6.3. Copy 4 MOVs
PatchSendPacket += fw[sendPacketLoop:sendPacketLoopBL]
## 6.4. BL sendRTPOverTCP
tgtSendRTPOverTCP = cmdTargetArg(sendPacketLoop+len(PatchSendPacket), sendRTPOverTCPStart)
PatchSendPacket += ArgToBin(tgtSendRTPOverTCP)+"\xEB"
## 6.5. STMFD SP!, {R0}
PatchSendPacket += "\x01\x00\x2D\xE9"
## 6.6. LDR R0, Socket
PatchSendPacket += ldrSock
## 6.7. BL makeSocketNonBlocking
tgtSocketNonBlock = cmdTargetArg(sendPacketLoop+len(PatchSendPacket), makeNonBlock)
PatchSendPacket += ArgToBin(tgtSocketNonBlock)+"\xEB"
## 6.8. LDMFD SP!, {R0}
PatchSendPacket += "\x01\x00\xBD\xE8"
## 6.9. CMP R0, #0
PatchSendPacket += "\x00\x00\x50\xE3"
## 6.A. BGE loopNext
tgtLoopNext = cmdTargetArg(sendPacketLoop+len(PatchSendPacket), sendPacketLoopNext)
PatchSendPacket += ArgToBin(tgtLoopNext)+"\xAA"
## 6.B. Fill up to printf2 with NOPs
Nops = (printf2 - (sendPacketLoop + len(PatchSendPacket))) / 4
PatchSendPacket += "\x00\x00\xA0\xE1" * Nops
## 6.C. Save generated patch
patches = []
patches.append( (sendPacketLoop, PatchSendPacket) )
print "Successfully patched"
Now that we have all the patches (for now there is one whole, but everything is forward), we will generate a new file:
### FIN: save patched file
if True:
f = open(fname+".fixed", "w+b")
patches.sort()
last = 0
for p in patches:
f.write( fw[last:p[0]] )
f.write( p[1] )
last = p[0]+len(p[1])
f.write(fw[last:])
f.close()
So, with the help of a stick, a rope, a python and a gram of the brain, we easily and easily patched all 11 streamers I needed immediately, and it took one evening, which is approximately equal to the time it would take to manually patch them all. But only with the next update, you don’t need to waste time at all!
for k in `(cd todo; ls -1)`; do g=$(echo $k | perl -pe 's/.*?([TV][^-]+).*?-V(2[^_]+).*/$1_$2/'); mv todo/$k .; ../repack/unpack.sh $k; cp $k.unpack/root/opt/topsee/rtsp_streamer rtsp_streamer_$g; done
for k in rtsp_streamer_*.[0-9]; do python tcpfix.py $k; done
...
ps: everyone who needs something else from these firmware has posted the scripts in a more convenient place - on the github .