Networking Challenge [the problem]
Translation of the original in Spanish. Read it in: CatalàCastellano
Part 1 of 3Networking challenge
- Networking Challenge [the problem]
- podcast 1x11: networking challenge [the problem]
- podcast 1×13: networking challenge [the solution]
Scenario
We have a hardware device with an embedded linux that can't be accessed because the manufacturer doesn't provide access, we've tried to get in but we haven't been able to, neither with our own means nor with those of an important electronics company. We believe the kernel is a 2.4.X from what we observe. The device has a screen and its purpose is to download a playlist through a proprietary protocol and a file transfer protocol (FTP) to play it on the screen.
Over the network it connects to the manufacturer's server from where it collects a playlist and some multimedia content. Essentially it uses the ports:
-
TCP/8889 (manufacturer control protocol) goes in 'plain text' we've already reverse engineered it.
-
FTP in passive mode
With the manufacturer's server it downloads correctly 99% of the time and without problems. In the attached file 'korea-ok.cap' if you examine it you'll see that everything works perfectly. The private IP you can see is the device's on the LAN and through an LMDS it connects to Korea at IP: 210.114.220.180, if you look closely the manufacturer uses a 'Serv-U' for Windows as FTP server.
Problem
The manufacturer's proprietary protocol has been implemented in Python and works perfectly, the problem comes when downloading over FTP. The device connects to our FTP server and completely randomly closes the transmission for no apparent reason and the device reboots. This always happens while an FTP-control (21/tcp) or FTP-data (random port of passive ftp/tcp) connection is taking place.
In the file 'cwd-fail.cap' you can see one of the typical failures the device has when trying to download from a local FTP server (in this case a proftpd on a Gentoo). Specifically, when sending the FTP command 'CWD' the device decides to close the connection and not continue with the download. After several attempts it reboots. To avoid getting you lost among the packets I've left only the socket with the problem in the capture.
So you don't get confused, remember that this doesn't necessarily happen on the FTP-control channel, it can also happen on an FTP-data one, it can happen at 1% of the file download, at 30%, 50%, 98%... it's completely random.
To have more examples of errors the file 'local-fail.cap' is attached, where you can see how an error identical to the previous one occurs but now on the FTP data channel. Specifically you can see how the socket is closed in packet 916 for no apparent reason. I should mention that the TCP checksum errors aren't relevant since they're due to the network card not being configured as follows:
# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off
In other tests they've been done with the parameters mentioned above and the error is the same. Even though the 'checksum error' alerts no longer appear. If you think it's vital to have a capture without the checksum errors there's no problem in making it. But after months of testing it's clear that this isn't relevant.
To prove that the behaviour is random, a file 'local-ok.cap' is attached below with the capture of a session on the local FTP server that worked perfectly. You can see in packet: 18975 that the content of a file is downloaded fine, specifically an .mpg file, with this file the whole data transfer finishes correctly.
Hardware and software tested
Countless tests and combinations have been done:
5 different servers:
-
1 debian with kernel 2.6.24.5-grsec-xxxx-grs-ipv4-64 at the ovh.es provider (100Mpbs symmetric without restrictions), proftpd FTP server port 2090/tcp
-
1 debian with kernel 2.6.18-5-686 jazztel provider without fw, nor any kind of restriction 100Mbps symmetric against the internet and FTP servers on port 21 pure-ftpd and on port 2090 proftpd.
-
1 ubuntu with kernel 2.6.24-16-server jazztel provider with fortinet as fw in transparent mode, proftpd on port 2090.
-
1 gentoo with kernel 2.6.25-gentoo-r5 on the local network with proftpd on port 2090.
-
1 virtualized windows xp with vmware workstation 6 over gentoo with a Serv-U FTP server, the virtual network card in bridget mode with the host operating system's one.
the behaviour is the same with all the combinations
Idea for a solution
Playing with the TCP kernel parameters (/proc/sys/net/ipv4) you can get behaviour changes in the error. For example, tests have been done with:
tcp_adv_win_sacle = 0,1,2
tcp_allowed_congestion_control = cubic, bic, reno, etc.
tcp_base_mss = 512, 1024, 1500
tcp_dsack = 0,1
tcp_ecn = 0,1
tcp_fack = 0,1
tcp_sack = 0,1
tcp_syncookies = 0,1
tcp_timestamps = 0,1
tcp_workaround_signed_windows = 0,1
tcp_window_scaling = 0,1
countless values and combinations tested in:
tcp_mem
tcp_moderate_rcvbuf
tcp_mtu_probing
tcp_rmem
tcp_tso_win_divisor
tcp_wmem
Tests have also been done limiting the bandwidth, and with some other kernel parameters that I don't remember now. We've tried to optimize the servers to perform at maximum, at minimum, by default... etc.
What all this achieves is that sometimes the download does happen, but we can't confidently establish patterns, that is, the TCP connection often keeps getting closed.
FAQ of the obvious
-
It's obvious that the device firmware is badly made, but that can't be touched.
-
The device can be configured via a compact flash, where a configuration file is stored, a binary file compiled for ARM and the multimedia content being downloaded.
-
We've tried to talk to the manufacturer a thousand times but it's like talking to a wall, besides the technicians don't speak English, only Korean. Incredible, isn't it?
-
The problem is with the device's TCP stack or some other similar element. That's the one that closes the connection!!! you can't keep the connection open from the server when the device has closed it, the only thing you can do is prevent the client from closing it. The problem is how.
-
Yes! the manufacturer's server downloads fine via FTP in 99% of the cases. That's the big problem, the question may sound silly but I've been asked it about 20 times, so it deserved to be here.
Files
The files talked about in this email can be downloaded from:
Challenge
I have the lab environment set up at the office, it's open 24x7 to meet with me and do tests in-situ. Obviously I'll answer everything that's online and I'll guide you with whatever I can and more.
I've been dragging this problem around for about 4 months and I've gone almost 4 days without a life, only obsessed looking for the solution but there's no way to see the light.
I don't have many resources, since I've already lost a great deal of money looking for the solution during all this time, but depending on results the privileged mind that manages to come up with the solution will be rewarded as much as possible.
I already have a solution
After turning the matter over for now I won't explain the various solutions I've already found to date for the problemón. So I'll take advantage of the problem to launch podcast 1x11 describing the problem in full detail and giving some hints on how to reach the solution. Maybe nobody's curiosity will be piqued enough to follow the solution in such detail but this way I keep a bit of the intrigue and if someone really needs to know the solution without following the culebrón they only have to send me a private email and I'll tell them. As a starter let me tell you that the intuition I had about the solution wasn't bad.
2 comments
Comments from the old blog, as they were written. Comments are closed.
rafalito el melocoton
quin trasto es aquest? simple curiositat.
Oriol Rius
Si alguien quiere información especifica del fabricante, modelo del dispositivo, etc. que me avise y se lo daré en privado.
Gracias.