Microsoft Word - EFR61504-000.doc

TCP/IP Router for Space Applications

Jim Joseph and Jennifer Lazbin

Spectrum Astro, Inc.

1440 N. Fiesta Blvd.

Gilbert, AZ 85233

Document # 0000 EF-R61504-000

Abstract- Advanced science missions are acquiring and

processing greater quantities of data from multiple instruments.

Data rates from instruments are increasing. Custom interfaces

have been developed in the past to address these needs. These

interfaces require custom test equipment to verify their

operation and add cost and time to spacecraft development

schedules. Once the architecture is designed for a spacecraft, it

is not easily reconfigured. These are issues that have been

resolved in the terrestrial environment by the use of standard

data protocols, standard communications interfaces, and the use

of data networks. The router is a network device that allows the

interface of devices with different data rates and different

electrical standards as well as data flow reconfiguration to meet

changing needs. Spectrum Astro was awarded a contract by the

Earth Science Technology Office to study the requirements for a

router qualified for space application and develop a board based

on those requirements.

This router development considers three network

configurations: onboard the spacecraft, spacecraft to ground,

and spacecraft to spacecraft possibly in a constellation. The

requirements consider router protocols, an embedded versus an

external routing processor, console port implementation, and

appropriate protocols that interface to a router to perform

router status and management. Because this is a study and not

directed toward a specific mission, derived requirements are

being established to guide the design of the router. The concepts

are implemented and tested on an existing development board

using Ethernet ports that Spectrum Astro developed to evaluate

design issues for flight applications. After proving the concepts,

Spectrum Astro will build an optimized version of the router

and perform thermal and vibration testing to verify the design.

This study is planned to close about September of 2005 at a

Technology Readiness Level (TRL) of 6~7.

I. INTRODUCTION

There is growing interest in the space community in

applying the use of open-standard protocols that are used for

terrestrial network communications, both Local Area

Network (LAN) and Wide Area Network (WAN), for

spacecraft architectures. Use of open-standard protocols as

opposed to custom protocols for interconnect and interfaces

within communication architecture for spacecraft reduces the

time to design, test, and integrate a spacecraft and reduces

costs over the spacecraft life cycle.

Transport Control Protocol/Internet Protocol (TCP/IP) has

long been in use for terrestrial communications. TCP/IP is

the key protocol used for Internet communications. Interest

in the use of TCP/IP (or IP, for short) for the links from the

ground to space has been a concept of research and

development for several years, but the idea of using TCP/IP

onboard a spacecraft is a newer concept.

Spectrum Astro has developed devices for space that are

implemented with open-standard network protocols. As part

of the Space Network Devices (SND) Program for the

National Aeronautics and Space Administration (NASA)

Computing, Information, and Communications Technology

(CICT) Program, Spectrum Astro developed two Ethernet

devices for space. The hardware is a 10/100 Mbps Network

Interface Controller (NIC) and 100 Mbps repeater-based hub

interconnect board. These devices were designed for

spaceflight from their inception. Spectrum Astro’s core area

of expertise is in spacecraft manufacturing, including the

development of radiation-hardened, flight electronics.

Typical mission requirements such as temperature, total dose

and Single Event Effects (SEE) radiation, shock, and

vibration were taken into consideration in the design of this

spacecraft network hardware. The NIC is an enabling

technology for support of onboard architectures that utilize

TCP/IP.

Spectrum Astro is continuing the work initiated on the

SND Program on the Space Network Router (SNR) Program,

launched in January 2004, for the NASA Earth Science

Technology Office (ESTO), Advanced Information Systems

Technology (AIST) Program. As part of the SNR Program,

Spectrum Astro is developing a single-board TCP/IP router

for space. The SNR router is based on a development board

built for the SND Program known as the Ethernet

Multipurpose Board (EMB), shown in Fig. 1.

A router is a network device used to interconnect networks

of disparate types. Different networks use various media and

protocols, each with varying data rates. A router uses

specific routing algorithms to determine the best route to send

data through a set of interconnected networks. Routers also

provide isolation between networks when faults occur to

reduce the effects on the set of networks that are connected

together. Specific network types of relevance for a single

spacecraft include point-to-point links and LANs. WANs

have utility for spacecraft constellations.

The SNR space router has four Ethernet interfaces that

provide the ability to interconnect various networks that exist

on a spacecraft and lays the foundation for seamless

interconnectivity in spacecraft communications architectures

among spacecraft in a constellation as well as between

spacecraft and the ground.

II. WHY USE NETWORK PROTOCOLS IN SPACE?

There are several advantages to using open standards rather

than custom implementations in a spacecraft communications

architecture. Custom communications interfaces and

interconnects require time to develop and extensive test time

to verify operation. Significant reductions in cost and

schedule can be realized with the utilization of open-standard

protocols. Use of open standards facilitates rapid design and

development efforts. Testing and integration schedules can

be compressed. Plug and play operation with devices from

multiple manufacturers is achievable when component

interfaces are built to open-system standards.

The use of terrestrial network open standards for space

communications architectures also has many advantages.

Though rad-hard implementations of devices such as network

controllers and routers for space differ internally from their

Earth-based cousins, the black-box functionality can remain

the same. A huge investment has already been made in the

development and testing of hardware and software that is

deployed in LANs and WANs all over the world. Protocols

such as Ethernet and TCP/IP are well understood by many

people. Commercial off the Shelf (COTS) products are

readily available to test hardware implemented with these

standards.

Use of network standards in space facilitates higher data

rates than traditionally used in space. For example, data rates

are bounded to 1 Mbps on MIL-STD-1553B busses and

interfaces, whereas 10Base-T Institute of Electrical and

Electronic Engineers (IEEE) 802.3 Ethernet supports 10

Mbps (20 Mbps in full-duplex mode) and 100Base-T

Ethernet supports 100 Mbps (200 Mbps in full-duplex mode).

LAN standards can provide a robust, fault-tolerant

architecture that is ideal for spacecraft. Seamless

interconnectivity between space and ground can be realized

using terrestrial network standards such as TCP/IP.

The network standards were designed based on the

International Standards Organization (ISO) Open System

Interconnection (OSI) 7-Layer Model. In this model, specific

functions are assigned to each layer. The advantage to using

layered standards is that changing the standard used for one

layer does not affect the function of other layers. For

example, the IEEE 802.3 Ethernet LAN standard defines

functions for Layer 1 (Physical Layer) and Layer 2 (Data

Link Layer) of the OSI Model. TCP/IP is defined for Layer 3

(Network Layer) and Layer 4 (Transport Layer). LAN

standards other than Ethernet can be used for Layers 1 and 2

while retaining the use of TCP/IP in the architecture.

III. NETWORK-CENTRIC SPACE COMMUNICATIONS

ARCHITECTURES

Space communications is comprised of three broad areas.

Most mission architectures are a hybrid mix of each of these

classes. The first architecture class area is the simplest:

communication strictly onboard the spacecraft. The second

architecture class area is communication between the

spacecraft and the ground. The ground can include the

Internet or simply the ground station. The third architecture

class area is communication among spacecraft in a

constellation. Each class has different requirements. For

example, security levels required for each class are different.

A more secure network is required in a constellation than

would be in a network that is isolated onboard a spacecraft.

Typical spacecraft have many duties they must execute.

Some duties are directly correlated to performing their

intended functions, while others are required to maintain their

ability to function properly. For example, there are payloads

and instruments that collect data. The spacecraft must

perform the functions of navigating through space and

monitoring its condition. The spacecraft must respond to

commands that are sent to it and send data regarding the state

of the spacecraft in addition to telemetry from the experiment

or function that the spacecraft performs. All of this requires a

communications architecture that can reliably move data

from one point to another in real time. Most of today’s

spacecraft use a mix of standard protocols such as MIL-STD-

1553B, RS-422, and Consultative Committee for Space Data

System (CCSDS) as well as custom schemes to transfer data.

However, few view onboard communications as a set of

networks.

On a spacecraft, instruments and payloads must

communicate with the spacecraft electronics bus. These

devices are commanded from the ground or flight software

resident in the electronics bus. Likewise, telemetry is sent

from instruments and payloads to the bus potentially for

processing and eventually to the ground for further

processing and/or analysis. There may be multiple busses in

the spacecraft architecture, such as a Command and Data

Handling (C&DH) bus and a payload bus, or a primary bus

and a redundant bus. Point-to-point links can be

implemented to interconnect a payload or instrument to a bus.

Spacecraft busses can be implemented as LAN networks.

This is the first class area architecture. In a large spacecraft,

Fig. 1. Ethernet Multipurpose Board

a router can be used to interconnect the busses. Use of a

router provides flexibility in the spacecraft architecture, for

the second and third class area architectures as well. The

appropriate technology can be selected for each bus or link,

while the router can provide interface ports that support each

standard. This also provides scalability in the design, where

various data rates can be implemented based on requirements

for a given mission. Router interface ports can be developed

for High-Level Data Link Control (HDLC), SpaceWire,

FireWire, and MIL-STD-1553B to provide IP support over

each of these protocols. HDLC is used for serial links such

as crosslinks or uplink/downlinks.

Payloads can be interfaced to the bus with point-to-point

links that use network protocols such as full-duplex Ethernet

or HDLC for Layers 1 and 2. These LAN protocols enable

the use of TCP/IP at Layers 3 and 4, with IP at Layer 3 and

then TCP or User Datagram Protocol (UDP) at Layer 4. In

this manner, the payload can have its own IP address that

enables a Principal Investigator (PI) to access the experiment,

with proper security implemented, from the office.

Flexible architecture implementations can also cross the

various architecture class areas. For example, onboard

communications could be performed with IP while link

communications is performed with traditional space

protocols. Alternatively, the space-ground link or crosslink

to another spacecraft could use IP while the onboard

communications could be based on traditional space

protocols. In this architecture, the spacecraft is effectively an

IP node on the network and thus integrated with the ground

network with which it is connected. An architecture with

fully-realized TCP/IP could utilize TCP/IP both on the

spacecraft and on the links.

IV. ANALYSIS OF LAN PROTOCOLS

During the SND Program, Spectrum Astro evaluated

Ethernet, SpaceWire, and FireWire for use in space. Of these

three LAN protocols, Ethernet has a long history of use with

TCP/IP. After carefully considering each protocol, Spectrum

Astro recommended development of devices for 10/100

Base-T Ethernet for the SND Program for several reasons,

including: data rate support for most mission requirements in

the near-term future and growth path to 1 Gbps data rate,

support for cable lengths to 100 meters, transformer isolation

at cable, existence of a standard for Ethernet across compact

Peripheral Connect Interface (cPCI) backplane, its behavior is

well understood, and test equipment is readily available. This

recommendation is not meant to imply that Ethernet is the

ideal solution to meet requirements for every mission. Which

LAN protocol or protocols are best for a mission, depends

highly on the mission requirements and objectives. In

general, however, Ethernet seemed a more worthy candidate

for development.

A standard is currently being developed for the use of

TCP/IP over FireWire. FireWire was developed to access

peripherals to a computer over a serial bus at data rates of 100

Mbps, 200 Mbps, and 400 Mbps. SpaceWire started with a

protocol used for transputers and was adapted for space

applications at data rates of 100 Mbps, 200 Mbps, and

400 Mbps. SpaceWire uses Low-Voltage Differential

Signaling (LVDS) as the electrical interface whereas

FireWire uses a modified form of LVDS that requires analog

circuitry and Ethernet uses a multi-level signaling technique

also requiring analog circuitry. These interfaces limit the

length of a physical link for SpaceWire to 10 meters,

FireWire to 4.5 meters and Ethernet to 100 meters. Within a

spacecraft, this is not much of a restriction but within an

integration and test bay, this has a significant impact. As far

as use across a backplane, Ethernet is supported in standards

for data rates of 10 Mbps, 100 Mbps, and 1000 Mbps while

FireWire is supported only at rates of 50 Mbps or less and no

standard exists for SpaceWire across a backplane. Backplane

implementation has significant utility for space where

transitional architectures are likely to include heritage bus

standards such as cPCI.

Transformer coupling has benefits with regards to Electro-

Static Discharge (ESD) and System Generated

Electromagnetic Pulse (SGEMP). In the IEEE 802.3 Ethernet

standard, the transformer isolation is located between the

cable and the electronics on the board. In FireWire, either

transformer or capacitor isolation is provided between the

analog and digital portions of the Physical Layer interface but

for the purpose of Direct Current (DC) isolation and not for

ESD or SGEMP. SpaceWire has no isolation at all. An

alternative to transformer isolation is fiber optic coupling.

This is available for Ethernet and not for FireWire.

SpaceWire could be implemented with fiber optics but no

standard exists at this time. Other space programs are using

SpaceWire and FireWire.

MIL-STD-1553B is a master-slave architecture. For this

reason, it does not map well to a client-server type model. It

is not impossible to run a protocol such as TCP/IP over

master-slave architecture, although it is not straightforward.

Packet sizes are limited and slaves require constant polling to

determine when they are waiting for service. MIL-STD-

1553B requires an acknowledgment from the slave within

microseconds of the end of a transmission by the master so it

is only useful within the bounds of the spacecraft and requires

careful consideration to how responses to IP packets will be

processed.

ANALYSIS OF ROUTING PROTOCOLS

Routing protocols are used by routers to route data packets

from source nodes to destination nodes. They function at

Layer 3, the Network Layer. Fundamentally, their objective

is to build and maintain a table of router port addresses that

can be used to reach specific networks. Routing tables also

contain information about various paths. Routing protocols

are based on algorithms that determine optimal routing paths

by using one or more metrics. Several different routing

algorithms exist for IP. Routers communicate with other

routers to update and maintain their routing tables. In this

way, the routers establish a database of the network topology.

Routers can share their routing tables with other routers or

send messages to other routers regarding the state of the

sender’s links, known as Link-State Advertisement (LSA).

Routing algorithms are designed to achieve several goals

including selecting the best route based on the metrics and

weights assigned to those metrics, efficiency with low

software and hardware utilization, rapid convergence,

robustness and stability, and flexibility. Routing algorithms

must converge quickly. When a network event occurs,

routers must quickly distribute updated routing information to

all of the routers on the network and then quickly recalculate

optimal routes. A routing algorithm must perform correctly

when unusual events occur, such as hardware failures, high

traffic loads, or incorrect implementations.

Many dynamic routing protocols exist within the Internet

Request For Comments (RFC) standards. Commercial

routers typically route via dynamic routing protocols or static

routes. The following are dynamic IP routing protocols:

Open Shortest Path First (OSPF), Routing Information

Protocol (RIP), Exterior Gateway Protocol (EGP), and

Border Gateway Protocol (BGP).

Each routing protocol uses a different algorithm to make

routing decisions. OSPF is a link state, interior gateway

protocol. RIP is a distance-vector protocol, and BGP is a

distance-vector protocol used to interconnect autonomous

systems. RIP uses hop count as a metric. A custom

algorithm such as a predictive one that uses a priori

knowledge about link status can also be utilized. The

combination of OSPF with a custom algorithm that takes into

account scheduled link makes and breaks is very powerful for

space applications.

VI. USE OF ROUTERS IN SPACE

Typically, the interface chosen for a particular device

onboard a spacecraft has depended on the application and the

development of instruments takes place long before the

design of a spacecraft architecture is considered. As missions

tend to use many different interfaces within the same

spacecraft, the router becomes an ideal component to

interface multiple networks of different types onboard the

spacecraft. How routers are implemented on a particular

mission depends on the mission requirements.

The space router should support communication with other

nodes in the network using one or both protocols

implemented at the Transport Layer (Layer 4), including

connection-oriented TCP and connectionless UDP.

Space networks differ from ground networks in that many

entities typically share use of a ground network. In the case

of the Internet, for example, a particular user has little control

over how packets from his message are routed to their

destination. Links between routers on the ground usually fail

due to congestion and not errors. The algorithms used to

detect faults are based on congestion and dealing with

congestion issues. In space, links are usually controlled by a

single entity. Links in space fail due to poor signal quality

and thus errors on the link. Algorithms designed to handle

errors are thus more optimal for space links. The Layer 4

algorithms used for links must be selected based on the

expected nature of the link.

Security is important, especially when routers are used.

Spectrum Astro is looking at standards for security, including

Internet Protocol Security (IPsec) and High Assurance IP

Equipment (HAIPE) encryption. Different applications have

different needs for encryption depending on where the data is

to be delivered and for what purpose. Routers often require

some level of decryption to enable them to determine the

destination for these datagrams. Authentication is the process

of verifying that the source of a communication is the

legitimate sender they claim to be. If the network is closed

on the spacecraft, this is not a problem. When the network

extends beyond the bounds of the spacecraft, this is

important. Authentication can occur at different levels of the

network stack also. For example, the link over which the

communications is taking place may be an authenticated link

but the datagrams may require authentication to verify that

they came from a recognized sender before they arrived at the

link.

A requirement of many space missions is redundancy of

critical subsystems. In many cases, this requires the backup

component to be operating in the fully functional mode that

the online component is while staying in the background.

Several design approaches are possible. At the Physical

Layer, the electrical interface can be made redundant as is

commonly implemented with MIL-STD-1553B. If the

Physical and Data Link Layer are taken together, then

redundancy can be implemented similar to redundant

transponders for space-to-ground links. The next level of

redundancy is making the entire router redundant similar to

block redundancy that is often used for the C&DH subsystem

of a spacecraft. Some of these approaches require that

parameters be passed between the online component and the

backup such that the backup component is able to maintain its

parameter tables to match the online component.

OSPF, described earlier, has a facility to allow two routers

to determine which is the designated router and which is the

backup designated router using the Hello protocol. If the

designated router becomes disconnected, the backup router

assumes the routing duties. This supports the concept of two

fully functional routers operating in hot standby and without

the need for a separate device to determine which router

should be online.

Important for space routers is network management.

Network management is the idea that elements of the network

such as routers can be managed, typically from a network

control center or ground station. In the trade studies for the

SNR Program, we looked at approaches for implementing

out-of-band management using the router console port. We

also looked at in-band management implementation. These

concepts as applied to space vary greatly from their ground

cousins.

Terrestrial routers are assumed to be mostly up and

forward data within a reasonable amount of time from when

the data arrives at the router. This is not so in space. Routers

can only forward data to links that are up. Links to the

ground or other spacecraft may exist for only finite amounts

of time, such as during a pass over a particular ground

station. Therefore, onboard data storage becomes important.

Data is queued in storage for downlink to the ground station

when the ground station link becomes available. Data storage

can be implemented as a network file server. Instruments

store data on the network file server until ground links

become active. The ground station is then able to access the

network file server independent of the instruments and the

spacecraft handles issues associated with link delays and

retransmission of dropped packets.