User protocol layer

ATM is not a ``reliable'' protocol in the networking sense - cells may be dropped or lost in transit, and no attempt is made to verify delivery. In order to facilitate reliable data transfer, and to allow the use of the wealth of software tools already available, we are running the standard IP protocols over ATM. We are using a pseudo-standard implementation known as ``Classical IP'', which defines a relationship between standard IP ``dot'' addressing and ATM PVCs, as well as data packet segmentation and reassembly algorithms for converting between IP packets and ATM cells[6]. Since the ATM switches know nothing of the protocol levels above ATM, the choice of this protocol affects merely the two endpoint systems. Those workstations both employ FORE SBA-200 ATM interface cards which perform IP packet segmentation and reassembly in hardware, enabling net ATM speeds of 155 Mbit/sec (OC-3).

Over the Classical IP level, we run the same TCP/IP and UDP/IP protocols that are used over standard Ethernet networks. Tools such as ftp, telnet, and the X Window System are part of every observing run, as are additional tools, such as an audio conferencing tool (rat) and a shared whiteboard application (wb).

The most important impact of a satellite component on a high-speed network is the relatively large delay introduced by the round-trip signal travel time to the satellite. In our network, this travel time is approximately 0.55 seconds, which corresponds to over 3 Mbytes of data at OC-1 speeds (51 Mbit/sec). The problem has to do with the connection-oriented nature of TCP/IP: TCP sends a very specific amount of data, known as a ``window'', after which time it expects an acknowledgement from the other end of the connection. This is the manner in which TCP/IP is able to implement a ``reliable'' connection. However, this window size is often very small; the default value for workstations running the SunOS 4.1.4 operating system is 4 Kbytes. If one were to use this system on our satellite network in its default configuration, a window of data would be sent in 0.0007 seconds, after which the sending system would be forced to wait 0.549 seconds for an acknowledgement. In other words, the system would be running at 0.1% efficiency, and the net throughput would reflect this: initial tests of our system under such conditions showed bandwidths of 0.1-0.2 Mbit/sec.

Fortunately, this problem is well-known in the high-speed networking community. Networks such as ours are known as ``long fat networks'' (LFN). The figure of merit for such networks is the window size, or the bandwidth-delay product:

$$\hbox{TCP window size} = \hbox{bandwidth} * \hbox{delay time}$$ (1)

There is an Internet RFC (Request For Comment) document on this subject, RFC 1323[7], and the problem has been discussed extensively. Many current operating systems support the RFC 1323 extensions, and provide options to increase the TCP window size to values in excess of 10 Mbytes. In the case of the SunOS operating system (to which we are constrained by legacy control software at Keck), we obtained the TCP-LFN package from Sun Consulting, which also purports to support the RFC 1323 extensions for long fat networks. In reality, a number of limitations of SunOS conspire to prohibit one from obtaining extremely large window sizes. In our case, the kernel limit of 2 Mbytes of Mbuf memory (IP packet wrappers) turned out to be the major constraint, limiting our window size to no more than 1 Mbyte. This is approximately one-third of the optimal value derived above, and, as such, we obtain maximum TCP/IP throughputs of approximately 10-15 Mbit/sec, about one-third of the 51 Mbit/sec OC-1 network operating speed. Although perhaps disappointing in a relative sense, this bandwidth is far in excess of T1 Ethernet speed (1.44 Mbit/sec) and allows an 8 Mbyte image to be transferred in approximately 5 seconds (not including protocol overhead). As a further comparison, this bandwidth exceeds that which is available on the local area Ethernet network at the Keck Telescope itself.