Trouble with Minix behind a Firewall

1 Sept 2003

In the spring and summer of 2002 I had some difficulties with access to and from my server running Minix 2.0.2 at Hampshire College, minix1.hampshire.edu. Security-related changes on the Hampshire College network turned out to be the cause. Eventually the problems were resolved with help from Kees Bot, Philip Homburg, and network administrators on the host networks. A description of my experiences may be helpful to others.

Suddenly one day minix1.hampshire.edu could not be contacted from outside Hampshire College. Fortunately I have a login account on a server at Hampshire that was not affected, and I was able to verify that the minix1 system had not crashed. But when I made a telnet connection to minix1 from the other local system I found that I could not make any connections from minix1 to the outside world. Something was blocking connections in both directions.

In correspondence with the Hampshire system administrators I learned that a new firewall had been installed. Checking with Kees I learned that a Minix system should have no unusual requirements with respect to network ports, and the sysadmin assured me that the ports I would need for ftp, http, telnet, and other network protocols were open. Specifically, incoming ports 20-23, 80. 513 and 514 were open and all ports were open for outgoing traffic. But minix1 remained isolated behind the firewall.

Finally one of the system administrators noticed that they had blocked all ICMP (Internet Control Message Protocol) packets. When ICMP packets were allowed through the firewall minix1 was accessible, and had access, again. I'm not a networking protocol expert, so I had to go to my copy of Douglas Comer's Internetworking with TCP/IP (Volume I) text to refresh my memory on ICMP.

ICMP packets provide a way for a gateway to report an error back to the originator of a packet. The traceroute program takes advantage of ICMP by generating packets addressed to a distant location, but with the ttl (time-to-live) field of the first packet set to a value of one, which is then incremented with succeeding packets. The ttl field is used to keep packets from being relayed forever when routing errors exist, and is decremented each time a packet is relayed. When the value reaches zero no further attempt is made to relay a packet, but an ICMP "time exceeded" packet is sent back to the original source.So the packets sent out by traceroute normally result in a series of such ICMP messages being received, each one identifying a gateway along the path to a destination.

After the ICMP was turned on again I had no problems for a while, but a few months later there were new security alerts. Minix1 again was incommunicado. Apparently someone decided to block ICMP at the firewall again. This time I had trouble getting the attention of the very busy Hampshire system administrators. Other systems on the Hampshire College network were apparently not affected by the blocking of ICMP, but my Minix system was. I asked Kees why this should be so, and if there was anything I could do about it without having to wait for the system people to deal with all their higher-priority concerns. Here is his answer:

Minix uses a low TTL on TCP packets initially and increases it on ICMP "time exceeded" packets. The good thing about this is that connections can be rapidly discarded after close if the other side is only a few hops away. This allows rapid buildup and takedown of TCP sessions. (Nice for HTTP servers.)

The downside is that you need those ICMP packets. The current inet [for Minix-vmd] has ways not even to need them, but standard Minix doesn't have that version and is not likely to get it. (Inet can't grow any bigger.)

A workaround is to use an artificially high metric of 32 in add_route:

        add_route -g gateway -m 32
This will force a TTL of 32.

Firewalls should be smarter with ICMPs, off course. A good stateful firewall should be able to figure out if an ICMP belongs to an existing TCP connection. Alas most times what is named a firewall is nothing more than a dumb packet filter. A real firewall keeps track of sessions, and knows what traffic to expect given previous traffic.

Apparently other operating systems starts with a higher default time-to-live in their packets than Minix does. If this is the case, then I suspect that eventually somebody else on the Hampshire network would have run into a problem with the blocking of ICMP packets. But it was my bad luck that Minix was acting the role of a canary in a coal mine at this time.

Unfortunately, Kee's advice to add -m 32 in invoking add_route was not the end of my problem. For a number of reasons I am still using Minix release 2.0.2 on my minix1 servers. But, the -m flag is not recognized by the version of add_route provided with Minix releases previous to 2.0.3. The first suggestion was to take the Minix 2.0.3 version of add_route and compile it on the Minix 2.0.2 system. However, the new version of this command uses an ioctrl that isn't available in Minix 2.0.2. However, it turns out that it isn't very hard to achieve the same effect by modifying the add_route source and recompiling. Rather than reproduce a complete diff here, I'll just say this: find the line in /usr/src/commands/simple/add_route.c that reads:

               metric= 1;
and edit it to
               metric= 25;
then recompile and install. (Why 25 and not 32? I think it turns out that 25 is the maximum ttl that Minix normally uses. In any case, by the time I got around to solving the problem for good Philip had recommended using a value of 25.)

Another occurrence: Recently (summer 2003) I had another brush with the same kind of problem, this time involving a "private" Minix server I run on the same as network as my minix1.bio.umass.edu system. The minix1.bio system runs Minix 2.0.2 and, even though it was not necessary at the time, I installed the modified add_route on it when I resolved the problem at Hampshire College, just for the sake of keeping the two machines configured as identically as possible. My server minix2.bio.umass (not its real name) runs Minix 2.0.3 and is used as a place to test new or modified web pages before I post them publically. I'm also using it as a place to test Minix 2.0.3 before upgrading the public system.

Shortly after the SoBig and Blaster viruses appeared the university network was hit by a lot of packets from infected computers, and the administrators of the main campus network took steps to limit ICMP packets. In this case ICMP packets were not totally blocked, but the throttling was so extensive that pings directed at the main campus name server indicated 90% or more packet loss. Minix 1 was unaffected, except, of course for pings,but minix2 was unable to respond to or initiate any connections outside its local network. Adding the - m argument to add_route, as described above, seems to have restored service to normal.

I'm not sure how important this story might be for other Minix users. After the first time I encountered the problem my thought was that it was probably a fluke, and others I asked said they thought it was a poor practice to block ICMP packets at a firewall. But, whether it is a good practice or not, I have seen administrators on two different networks decide to block ICMP. I think the first time it may have been just a reflexive move to block anything other than the most essential traffic, and the consequences may not have been thoroughly investigated by system administrators faced with serious problems. The second time this occurred the administrators seemed convinced that ICMP packets from virus-infected workstations were a serious denial-of-service problem. The response this time was to throttle them, rather than block completely, and the person I talked to indicated it was meant to be a temporary measure. Unfortunately, although for some purposes this may be a satisfactory expedient (for example, 10% of pings getting through is enough to indicate a host is alive), for Minix in its default configuration ICMP is vital to successful use of TCP protocols. The Minix design is perfectly reasonable and uses network protocols in the way they were intended to be used. But it is different from other systems and makes Minix liable to suffer when system managers are dealing with a serious threat.


All material on this site not otherwise attributed is copyright ©1994-2003 Albert S. Woodhull
Click here for information on copying and other use.
Mail comments on this page to: Al Woodhull <awoodhull@hampshire.edu>

Viewable With Any Browser [Valid HTML 4.0!]