The Need For Networking Extensions

While HyperTransport was initially developed to address bandwidth and scalability problems associated with moving data through the I/O subsystems of desktops and servers, the networking extensions bring a number of enhancements which permit the advantages of HyperTransport technology to be extended to communications processing applications. There are some major differences in the requirements of host-centric systems such as desktops and servers and communications processing systems.

Communications Processing Is Often Less Vertical

In communications applications, there may be a number of processors or coprocessors located in various corners of the topology. The host processor may assume responsibility for configuration and control of coprocessors and interface devices, while the coprocessors perform specialized data processing tasks. Because of the distributed responsibility for control and data handling tasks, these systems tend to be much less host processor-centric.

As a result of decentralizing data processing in communications systems, information flow may be omni-directional as coprocessors initiate transactions targeting devices under their control. When switch components are added to the topology, elaborate multi-port configurations are possible.

Summary Of Anticipated Networking Extension Features

Network Extensions Adds Message Semantics

In handling the special problems of communications processing, the HyperTransport networking extensions add message semantics to the storage semantics used in the 1.04 revision of the HyperTransport I/O Link Specification. Storage semantics were described in the last section. Message semantics are more efficient in handling variable length transfers, broadcasting messages, etc. The 64-byte HyperTransport packets are concatenated to form longer messages, and additions to request packet fields identify the start of a message, end of a message, or may even be used to signal the abort of a scheduled transaction. Unlike storage semantics, in which the payload is data targeting an address, messages can also be sent which convey interrupts and other housekeeping events.

Another difference between message semantics and storage semantics is the concept of addressing. In storage semantics, addresses are managed by the source device, and each byte of data transferred is associated with a particular address in the system memory map. This makes sense because the locations are within (and owned by) the device being targeted. In message semantics, the message is tagged as to which stream it belongs, and the destination determines where it goes. The ultimate destination is often external to the system, where the system memory map has no meaning.

16 New Posted Write Virtual Channels

Release 1.1 adds 16 new optional Posted Write Virtual Channels to the hardware of each node (above the three already required). Each of these new virtual channels may be given a dedicated bandwidth allocation, and an arbitration mechanism is defined for managing them.

An End-To-End flow control mechanism has also been added to allow devices to put millions of user streams into these 16 additional virtual channels. In this way, very large numbers of independent real-time streams (e.g. audio or video) make be handled.

Direct Peer-to-Peer Transfers Added

HyperTransport supports the full producer-consumer ordering model of PCI. In cases where this strict global ordering is needed, transactions from one HyperTransport I/O device to another (called peer-to-peer transfers) must first move upstream to the host bridge where they are then reissued downstream to the target device (a process HyperTransport calls reflection). Release 1.1 adds the option of sending send some traffic directly from peer-to-peer when the application does not require strict global ordering (it often isn't a concern in communications processing).

Link-Level Error Detection And Handling

With the addition of direct peer-to-peer transfers, Release 1.1 permits coprocessors and other devices to communicate directly without involvement of the host bridge. Along with this capability, network extensions provide for error detection and correction on the individual link level. In the event of an error, the receiver sends information back to the transmitter which causes a re-transmission of the packet. Obviously, the packet can't be consumed or forwarded until its validity is checked.

64 Bit Addressing Option

In keeping with the very large address space of many newer systems, Release 1.05 allows the optional extension of the normal 40-bit HyperTransport request address field to 64 bits.

Increased Number Of Host Transactions

Release 1.05 increases the number of outstanding transactions that a host bridge may have in progress from 32 to 128.

End-To-End Flow Control

In communication systems, there are occasions when devices are transferring packets to distant targets (not immediate neighbors) which may go "not ready" (or to another state which makes them unable to accept traffic) for extended periods. Prior to Release 1.1, HyperTransport devices only have flow control information for their immediate neighbors. Release 1.1 adds new end-to-end flow control packets which distant devices may send to each other to indicate their ability to participate in transfers. If a device is not ready, the source device does not start sending (or continue sending) packets; this helps eliminate bottlenecks which otherwise occur when the flow control buffers of devices in the path between source and target become full of packets which cannot be forwarded.

Switch Devices Formally Defined

Finally, Release 1.05 formally defines the switch device type which may be used to help implement the complex topologies required in communications systems. A switch behaves much like a two-level HyperTransport-HyperTransport bridge with multiple secondary interfaces. The basic characteristics of a switch include:

1. A switch consumes one or more UnitIDs on its host interface. The port attached to the host is the default upstream port.

2. The switch acts as host bridge for each of its other interfaces. Each interface has its own bus number.

3. Switches, like bridges, are allowed to reassign UnitID, Sequence ID, and SrcTag for transactions passed to other busses. The switch maintains a table of outstanding (non-posted) requests in order to handle returning responses.

4. Switches may be programmed to perform address translation.

5. Switches must maintain full producer-consumer ordering for all combinations of transaction paths.

6. Switches must provide a method for configuration of downstream devices on all ports.