A War Story
I once worked for a company that provided mobile data services to public safety organizations: we put PCs in police cars. Our value proposition was that we could connect the PC to the base station using off-the-shelf packet radio on the same frequency as the voice radio. This was attractive to small town police departments that could not afford to install a separate data radio system.
The company was started by a few ham-radio enthusiasts who were trying to leverage their expertise in packet radio to provide a low-cost product to the public safety market. One of the guys had written network code at DEC and wrote the network stack. This was in the days before the web and before TCP/IP was the standard protocol, so he wrote an ad-hoc protocol for the system.
I was called to one on-site installation to solve a vexing problem. The system would work for a while, but then it would hang and need to be completely reset. It only happened when a certain file was being sent. There was nothing unusual about the file, it was a text file.
The way the system worked was, let us say, imaginative. The base station had a command line interface, and the PC in the car would send commands to the base station over packet radio. It would literally type the received command into the base station's prompt. This violated all the best practices for network protocols. It was vulnerable to dropped packets, injection attacks, replay attacks, etc. It had no privacy, no integrity, and no authentication. I was horrified. But it was a working system, so I had to work with it.
After a few hours of debugging, I found that the base station was getting stuck in the packet receive loop. The cause was amusing, if pathetic. The command protocol was sloppy. About half the time, a command would be sent with one or more trailing newlines. The base station dealt with this by stripping trailing newlines, if present from the command before executing it.
The file transfer command would include the length of file in blocks. The length was sent in binary over the ascii channel.
The problem was a file that was 13 blocks long. The command would
be sent as file \x0d
. But the base station would
recognize the \x0d
as a trailing newline and strip it.
The file command, expecting a length byte, would read off the end of
the end of the command and get the null byte. This started the file
receive loop which would pre-decrement the unsigned length (to
account that first packet had been sent) and ended up with a value
of 65535. It then would sit and wait for the next 65535 packets to
arrive.
So the system wasn't exactly hung, it was waiting for the rest of the file.
There was no way I was going to try to repair this monstrosity. I suggested that we use a proper network protocol, like TCP/IP, but I pushed this back to the guy that wrote this mess. As a founder of the company, he wasn't about to change the perfectly good protocol that he had authored, so he came up with a workaround.
I didn't last too long at that company. I saw the writing on the wall and moved on.