Understanding Lustre Networking

C H A P T E R 2

This chapter describes Lustre Networking (LNET) and supported networks, and includes the following sections:

Introduction to LNET

Supported Network Types

Important Terms

2.1 Introduction to LNET

In a Lustre network, servers and clients communicate with one another using LNET, a custom networking API which abstracts away all transport-specific interaction. In turn, LNET operates with a variety of network transports through Lustre Network Drivers (LNDs).

In this design, where Lustre servers are written to communicate using an abstract network layer such as LNET, makes it easier to add support for new network types in Lustre. The knowledge of specific network types is contained in the LNDs, not spread throughout Lustre server or client code.

Key features of LNET include:

RDMA, when supported by underlying networks such as Elan, Myrinet, and InfiniBand

Support for many commonly-used network types such as InfiniBand and IP

High availability and recovery features enabling transparent recovery in conjunction with failover servers

Simultaneous availability of multiple network types with routing between them

LNET is designed for complex topologies, superior routing capabilities and simplified configuration.

2.2 Supported Network Types

Lustre supports the following network types:

openib (Mellanox-Gold InfiniBand)

iib (Infinicon InfiniBand)

vib (Voltaire InfiniBand)

o2ib (OFED - InfiniBand and iWARP)

ra (RapidArray)

Elan (Quadrics Elan)

gm and mx (Myrinet)

LNET

2.3 Important Terms

The following terms are important to understanding Lustre networking.

LND: Lustre network driver. A modular sub-component of LNET that implements one of the network types. LNDs are implemented as individual Linux modules and, typically, must be compiled against the network driver software.

Network: A group of nodes that communicate directly with each other. The network is how LNET represents a single cluster. Multiple networks can be used to connect clusters together. Each network has a unique type and number (for example, tcp0, tcp1, or elan0).

NID: Lustre network identifier. The NID uniquely identifies a Lustre network endpoint, including the node and the network type. There is an NID for every network which a node uses.