Internet Draft P. Cameron J. Bates Xylogics, International Ltd. April 1993 Connection Multiplexing Protocol (CMP) Status of this Memo This document is an Internet Draft. Internet Drafts are working documents of the Internet Engineering Task Force (IETF), its Areas, and its Working Groups. Note that other groups may also distribute working documents as Internet Drafts. Internet Drafts are draft documents valid for a maximum of six months. Internet Drafts may be updated, replaced, or obsoleted by other documents at any time. It is not appropriate to use Internet Drafts as reference material or to cite them other than as a "working draft" or "work in progress." Please check the I-D abstract listing contained in each Internet Draft directory to learn the current status of this or any other Internet Draft. It is intended that this document will be submitted to the IESG for consideration as a standards document. Distribution of this document is unlimited. Abstract One of the problems with terminal servers attached to a LAN is the large number of small packets they can generate. Frequently, most of these packets are destined for only one or two hosts. CMP is a protocol which allows multiple Telnet and Rlogin connections, between a server and a host, to share a single TCP connection. With simple extensions this can be expanded to include other TCP protocols. Acknowledgments This document is the work of many more contributors than just the listed authors both here at Xylogics International and at Xylogics Inc. Specifically Jim Barnes, Gary Malkin, James Carlson, Dave Hirst and Miguel Sasson. In addition, we would like to thank ICL Plc. in the U.K. and Sweden for reviewing early versions of this proposal, and for their assistance on early prototypes. Cameron, Bates Expires: 20 Oct 93 [Page 1] Internet Draft CMP April 1993 Table of Contents 1. Justification . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 3. Protocol Design . . . . . . . . . . . . . . . . . . . . . . . 3 3.1 Header Format . . . . . . . . . . . . . . . . . . . . . . . 3 3.2 Opening a Multiplexed Connection . . . . . . . . . . . . . . 4 3.3 Opening a New Subconnection . . . . . . . . . . . . . . . . 6 3.4 Sending Data . . . . . . . . . . . . . . . . . . . . . . . . 6 3.5 Urgent Data . . . . . . . . . . . . . . . . . . . . . . . . 6 3.6 Subconnection flow control . . . . . . . . . . . . . . . . . 7 3.7 Closing a Subconnection . . . . . . . . . . . . . . . . . . 8 3.8 Closing a Multiplexed Connection . . . . . . . . . . . . . . 9 3.9 Standard Error Codes . . . . . . . . . . . . . . . . . . . . 9 4. Multiplexed Messages . . . . . . . . . . . . . . . . . . . . . 10 4.1 Construction . . . . . . . . . . . . . . . . . . . . . . . . 10 4.2 Disassembly . . . . . . . . . . . . . . . . . . . . . . . . 10 5. Security Considerations . . . . . . . . . . . . . . . . . . . 10 6. Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . 11 1. Justification When network designers consider which protocols generate the most load, they naturally tend to consider protocols which transfer large blocks of data (e.g. FTP, NFS). What is often not considered is the load generated by Telnet and Rlogin because of the assumption that users type slowly and the packets are very small. This is a grave underestimation on networks which have many Telnet and Rlogin ports on multiple terminal servers. The problem stems from the fact that the work a host must do to process a 1-byte packet is very nearly as much as the work it must do to process a 1500-byte packet. That is, it is the overhead of processing a packet which consumes a host's resources, not the processing of the data. If one assumes that most users connected to a terminal server will be connecting to only a few hosts, then it should be obvious that the network and host load could be greatly reduced if traffic from multiple users, destined for the same host, could be sent in the same packet. 2. Overview CMP is designed to improve network utilization and reduce the load on the hosts by multiplexing TCP connections, which would otherwise generate many small packets, onto a single TCP connection which generates fewer, larger packets. Cameron, Bates Expires: 20 Oct 93 [Page 2] Internet Draft CMP April 1993 The following terminology is used throughout this document. Multiplexed Connection - The actual TCP connection which exists between the server and the host over which the subconnections exist. Subconnection - The "virtual TCP" connection between processes on the server and host which exists over the Multiplexed Connection. Message - A single packet of data for a single subconnection plus its header which is passed over the Multiplexed Connection. Endpoint - One of the two ends of a subconnection. 3. Protocol Design CMP operates by prepending a 4 octet header onto each subconnection's data, then appending that onto the end of the packet which will be sent over the multiplexed connection. Upon receiving a message, the individual subconnection datastreams are demultiplexed and the information is handed up to the application. 3.1 Header Format The 4 octet header has the following general format: | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 | +-----------+-------------------+ | TYPE | SIZE high | +-----------+-------------------+ | SIZE low | +-------------------------------+ | Dest ID (DID) high | +-------------------------------+ | Dest ID (DID) low | +-------------------------------+ The TYPE field has the following values: 0 - data (DATA) 1 - urgent data (URG_DATA) 2 - open subconnection (OPEN) 3 - reply to OPEN message (OPEN_RPLY) 4 - close subconnection (CLOSE) 5 - reply to CLOSE message (CLOSE_RPLY) 6 - credit update (CREDIT) Value 7 is reserved for future use. For most commands, the SIZE field contains the number of octets of 'data' following the header. For messages of type DATA and URG_DATA this will be the number of data octets, for other messages, this will Cameron, Bates Expires: 20 Oct 93 [Page 3] Internet Draft CMP April 1993 be the number of octets contained in subsidiary header fields. For the CREDIT message, this field contains the number of bytes of new credit available for the receiving end to use. The DID is an arbitrary 16-bit number assigned by the receiving end of a subconnection to identify the subconnection. The receiver of a message uses the DID to uniquely identify a subconnection. Note, this field is not used by the OPEN message as no destination has been allocated. 3.2 Opening a Multiplexed Connection The first time an endpoint tries to open a TCP connection to a host, it first attempts to create a Multiplexed Connection by opening to the CMP port, TBA. If that attempt succeeds (i.e. a TCP connection is established), the server sends an OPEN message. If the Multiplexed Connection cannot be established, the server should open a standard TCP connection directly to the specified remote port. The format of an OPEN message is: | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 | +-----------+-------------------+ | 2 | 0 | +-----------+-------------------+ | 6 | +-------------------------------+ | 0 | +-------------------------------+ | 0 | +-------------------------------+ | Source ID (SID) high | +-------------------------------+ | Source ID (SID) low | +-------------------------------+ | Dest port high | +-------------------------------+ | Dest port low | +-------------------------------+ | Initial Credit high | +-------------------------------+ | Initial Credit low | +-------------------------------+ The SID is an arbitrary number, for example, it could be the index into an internal table of connections. See discussion below on the use of this field. The TYPE is 2, indicating the opening of a new, in this case first, subconnection. The DID field is not used as no destination has yet been allocated. The Dest Port is the well known TCP port number for the application (e.g. 23 for Telnet). The initial credit is the number of octets of data that the endpoint opening the connection can initially receive from the other endpoint. The receiver of the open message will respond with an OPEN_RPLY Cameron, Bates Expires: 20 Oct 93 [Page 4] Internet Draft CMP April 1993 message. The format of an OPEN_RPLY message is: | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 | +-----------+-------------------+ | 3 | 0 | +-----------+-------------------+ | 6 | +-------------------------------+ | Dest ID (DID) high | +-------------------------------+ | Dest ID (DID) low | +-------------------------------+ | Source ID (SID) high | +-------------------------------+ | Source ID (SID) low | +-------------------------------+ | Initial Credit high | +-------------------------------+ | Initial Credit low | +-------------------------------+ | Error code (ERR) high | +-------------------------------+ | Error code (ERR) low | +-------------------------------+ If the receiver cannot open the subconnection for any reason, it will respond with an OPEN_RPLY message with a non-zero ERR field (and a SID field of 0). If the receiver succeeds in opening the subconnection, the ERR field of the OPEN_RPLY will contain zero. See section 3.9 for a list of standard error codes. The initial credit is the number of octets of data that the endpoint can initially receive from the other endpoint. It is important to note the use of the SID fields in both the OPEN and OPEN_RPLY messages. This field is used to pass to the other endpoint the value to be used in the DID field of all subsequent messages sent back on this Subchannel. In particular, the DID field of the OPEN_RPLY message should be set from the SID field of the OPEN message. Optionally having different DID fields in the two directions allows the implementor of each end much greater freedom in the use of the DID field, hence the opportunity to make the implementation more efficient. For example, if the endpoint has a table of information about each open subconnection, then the DID can be the index into this table. This is a much more efficient implementation that scanning the table and much easier to implement than a hash table lookup. Cameron, Bates Expires: 20 Oct 93 [Page 5] Internet Draft CMP April 1993 Thus, the sequence of messages and their SIDs and DIDs could be: Endpoint 1 Endpoint 2 ---------- ---------- TYPE=OPEN ---------------> DID=0 SID=D1 TYPE=OPEN_RPLY <---------------- DID=D1 SID=D2 TYPE=DATA ---------------> DID=D2 TYPE=DATA <---------------- DID=D1 3.3 Opening a New Subconnection When a user on a server attempts to open a TCP connection to a host for which a Multiplexed Connection exists, the server simply adds that TCP connection as a new subconnection by sending an OPEN message and receiving back an OPEN_RPLY message (see section 3.2). Once a Multiplexed Connection has been established, either side may initiate an OPEN request. 3.4 Sending Data Subconnection data is preceded by the following header: | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 | +-----------+-------------------+ | 0 | Length high | +-----------+-------------------+ | Length low | +-------------------------------+ | Dest ID (DID) high | +-------------------------------+ | Dest ID (DID) low | +-------------------------------+ The DID is the identification for the subconnection. The TYPE is 0. The Data SIZE field specifies the length of the data immediately following the header; it does not include the length of the header. Cameron, Bates Expires: 20 Oct 93 [Page 6] Internet Draft CMP April 1993 3.5 Urgent Data TCP requires the ability to send individual octets marked as "urgent" data. When an application requests this, it expects that the remote end of the communications stream is notified as soon as possible that urgent data are pending, and it expects that the remote end will be able to mark this octet and deliver it either in-line or as out-of- band data. This feature is implemented over a Multiplexed Connection using an URG_DATA message. The format of the urgent data message is: | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 | +-----------+-------------------+ | 1 | Length high | +-----------+-------------------+ | Length low | +-------------------------------+ | Dest ID (DID) high | +-------------------------------+ | Dest ID (DID) low | +-------------------------------+ The DID is the identification for the subconnection. The TYPE is 1. The SIZE field indicates the number of bytes of urgent data being sent. Some implementations may choose to expedite the transmission of data preceding and including the urgent data message. 3.6 Subconnection Flow Control CMP supports flow control on a subconnection basis to ensure that a single subconnection can not tie up vast amounts of resources to the detriment of the other subconnections. Flow control in CMP is accomplished by use of a credit/debit system. Each endpoint knows the maximum number of octets of data (contained in messages of type DATA) that the other endpoint is able to receive, and may not send more than this number, it can of course send less. After sending a DATA message, the endpoint should decrease its outstanding credit by the number of octets just sent. Once the other endpoint has processed the data, it will send back CREDIT messages to tell the sending endpoint how many more octets of data it can now receive. Note, an endpoint can send CREDIT messages when it wishes to, there are no constraints on the timing, and it may not grant back to the other endpoint all the credit it initially had (or it may grant more). The format of the CREDIT message is: Cameron, Bates Expires: 20 Oct 93 [Page 7] Internet Draft CMP April 1993 | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 | +-----------+-------------------+ | 6 | Extra credit high | +-----------+-------------------+ | Extra credit low | +-------------------------------+ | Dest ID (DID) high | +-------------------------------+ | Dest ID (DID) low | +-------------------------------+ The DID is the identification for the subconnection. The TYPE is 6. The extra credit field is the number of octets of data space that have become available at the endpoint sending the message. This value should be added to the credit value currently held. After an OPEN message, the credit available is given in the initial credit field of the OPEN or OPEN_RPLY message. Only messages of type DATA are controlled by the credit/debit system, all other messages (including URG_DATA) can be sent at any time. 3.7 Closing a Subconnection When one side of the subconnection wishes to close, it sends a CLOSE message. This indicates that there is no more data to be sent. After sending this message an endpoint MUST NOT send any more data. The side which receives the Close Request continues to send data until there is no more data to be sent, at which time it sends a CLOSE_RPLY message. A CLOSE message has the following format: | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 | +-----------+-------------------+ | 5 | 0 | +-----------+-------------------+ | 1 | +-------------------------------+ | Dest ID (DID) high | +-------------------------------+ | Dest ID (DID) low | +-------------------------------+ | Close type | +-------------------------------+ The DID is the identification for the subconnection. The TYPE is 5. The close type field contains the type of close to be done: 0x00 Standard close 0x01 Reset. A standard CLOSE will allow all data currently in transit to be sent, Cameron, Bates Expires: 20 Oct 93 [Page 8] Internet Draft CMP April 1993 but a reset CLOSE will cause this data to be flushed. The CLOSE_RPLY message has the following format: | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 | +-----------+-------------------+ | 6 | 0 | +-----------+-------------------+ | 2 | +-------------------------------+ | Dest ID (DID) high | +-------------------------------+ | Dest ID (DID) low | +-------------------------------+ | Error code (ERR) high | +-------------------------------+ | Error code (ERR) low | +-------------------------------+ The DID is the identification for the subconnection. The TYPE is 6. The ERR field is used to return an error code. A value of 0 is used where the Close was successful, a non-zero value indicates an error. See section 3.9 for a list of error codes. A subconnection is not closed until one endpoint issues a CLOSE Request, and the other endpoint responds with a CLOSE_RPLY message. Note, the subconnection is closed on receipt of the CLOSE_RPLY message, regardless of the value of the ERR field. Where the close type is 'reset' the receiver should send the CLOSE_RPLY message immediately, it should not wait for data to flush. If an endpoint sends a normal CLOSE message, and does not receive a CLOSE_RPLY message it is permissible to send a second CLOSE message with a 'close type' of 'reset'. In this situation it must be possible to cope with receiving either 1 or 2 CLOSE_RPLY messages. 3.8 Closing a Multiplexed Connection When the last subconnection on a Multiplexed Connection has closed, the Multiplexed Connection should also be closed, as any TCP connection would be. 3.9 Standard Error Codes The standard error codes to be used in the ERR field of the OPEN_RPLY and CLOSE_RPLY messages are: ESUCCESS 0 Command completed successfully ENXIO 5 No such device or address ENOMEM 8 Not enough core EACCES 9 Permission denied EBUSY 10 Device busy ENODEV 11 No such device Cameron, Bates Expires: 20 Oct 93 [Page 9] Internet Draft CMP April 1993 ESECURITY 54 Security fault EMJOB 57 Job limit exceeded ESECURITYERR 59 Security server is unreachable Any other non-zero values may also be used to signify an error condition. 4. Multiplexed Messages A Multiplexed Message is the packet which is sent over the Multiplexed Connection. It is constructed from multiple data packets and their associated CMP headers. 4.1 Construction When an application to a subconnection sends a packet, the CMP prepends that packet with a header to create a CMP message, then appends the message to the Multiplexed Message under construction. When the first message to be transmitted is placed into the Multiplexed Packet under construction, a timer is started. When the timer expires, all outstanding message are placed into the Multiplexed Message and it is transmitted. This ensures that all messages constructed before the timer expires are sent in a single Multiplexed Message. If during construction of the Multiplexed Message the buffer holding the packet fills, the Multiplexed Message is transmitted immediately. The delay time should be user configurable; a reasonable time is 20 to 30 milliseconds. Additionally, configuration options may limit the number of included data packets or the maximum size of the Multiplexed Message before it is transmitted. If urgent data arrives for a subconnection, this is added to the Multiplexed Message, which is transmitted immediately (see section 3.5). 4.2 Disassembly When a Multiplexed Message is received, the CMP immediately accepts it from TCP. It is important that the TCP connection be kept drained so that the window remains open. Each message in the Multiplexed Message (i.e. data message or control message) is processed in turn. The data contained in DATA and URG_DATA messages is sent to the applications as though they were directly interfaced to TCP. All other message types are dealt with by the CMP interface layer. Cameron, Bates Expires: 20 Oct 93 [Page 10] Internet Draft CMP April 1993 5. Security Considerations If a site's security is instituted on an host-to-host basis, the CMP does not introduce a security problem. If security exists on a per connection basis, the CMP would need to provide a mechanism for allowing or disallowing certain users access to a multiplexed connection. Such a mechanism could be added, but is beyond the scope of this document. 6. Authors' Addresses Peter Cameron and Julian Bates Xylogics International Ltd. Featherstone Road Wolverton Mill Milton Keynes MK12 5RD UK Phone: +44 908 222112 Fax: +44 908 222115 Email: cmp-id@xylint.co.uk Cameron, Bates Expires: 20 Oct 93 [Page 11]