SGI 10-Gigabit Ethernet Network Adapter Uživatelský manuál stáhnout pdf (Strana 112)

CCGrid '11

112

Low-level Latency Measurements

VPI-IB

Native IB

VPI-Eth

RoCE

Small Messages

Latency (us)

Message Size (bytes)

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

Large Messages

Latency (us)

Message Size (bytes)

ConnectX-DDR: 2.4 GHz Quad-core (Nehalem) Intel with IB and 10GE switches

RoCE has a slight overhead compared to native IB because it operates at a slower clock rate (required to support

only a 10Gbps link for Ethernet, as compared to a 32Gbps link for IB)

1 2 ... 107 108 109 110 111 112 113 114 115 116 117 ... 149 150

A Tutorial at CCGrid ’11 1
Computing Systems 3
Cluster Computing Environment 4
(http://www.top500.org) 5
Grid Computing Environment 6
Compute cluster 8
Cloud Computing Environments 9
Hadoop Architecture 10
Memcached Architecture 11
• Hardware components 13
• Software components 13
• Ex: TCP/IP, UDP/IP 14
– E.g., PCI, PCI-X 15
– One bit per wire 15
– Not scalable: 15
Myrinet (1993 -) 1 Gbit/sec 16
IB Trade Association 19
• Network speed bottlenecks 21
• I/O interface bottlenecks 21
(not shown) 24
• Fast Messages (FM) 27
• Myricom GM 27
IB Hardware Acceleration 28
• Interrupt Coalescing 29
• Jumbo Frames 29
• Hardware Checksum Engines 29
TOE and iWARP Accelerators 30
2006 (v3.0), 2008 (v3.1) 34
PCI-Express (PCIe) 34
2003 (Gen1), 2007 (Gen2) 34
IB, HSE and their Convergence 36
Traditional Ethernet 37
IB Overview 38
Components: Channel Adapters 39
• Switches: intra-subnet 40
– Not directly addressable 41
IB Communication Model 43
Queue Pair Model 44
Memory Registration 45
Memory Protection 46
(Send/Receive Model) 47
InfiniBand Device 48
Hardware ACK 48
Hardware Protocol Offload 51
• Buffering and Flow Control 52
• Switching and Multicast 52
Buffering and Flow Control 53
Virtual Lanes 54
Service Levels and QoS 55
Traffic Segregation Benefits 56
Identifiers) 57
Switch Complex 58
– 3D Torus (Sandia Red Sky) 59
More on Multipathing 60
IB Multicast Example 61
IB Transport Services 63
Reliability 64
Transport Layer Capabilities 65
Data Segmentation 66
Transaction Ordering 67
Message-level Flow-Control 68
Concepts in IB Management 71
Subnet Manager 72
HSE Overview 74
Differences 75
– Multi Stream Semantics 76
Offloaded TCP 83
Myrinet Express (MX) 85
Datagram Bypass Layer (DBL) 86
• Solarflare approach: 87
Ethernet 90
IB Network 90
IB Verbs 90
Application 90
Hardware 90
IB Transport 91
IB iWARP/HSE RoE RoCE 92
IB Hardware Products 94
Tyan Thunder S2935 Board 95
IB Hardware Products (contd.) 96
– Nortel Networks 97
• Support for VPI and RoCE 98
– www.openfabrics.org 99
– OFED 1.6 is underway 99
Verbs Interface 100
(libibverbs) 100
• Within the hardware: 101
OpenFabrics Software Stack 102
InfiniBand in the Top500 103
Cray Interconnect 104
NUMAlink 104
SP Switch 104
CCGrid '11 106
• Other small clusters 107
• Integrated Systems 107
Other HSE Installations 108
Presentation Overview 109
1/10 GigE 110
InfiniBand 110
Case Studies 111
Latency (us) 112
Message Size (bytes) 112
Bandwidth (MBps) 113
MVAPICH/MVAPICH2 Software 115
One-way Latency: MPI over IB 116
Bandwidth: MPI over IB 117
Bandwidth: MPI over iWARP 119
Convergent Technologies: 121
Traditional Model 123
Possible SDP Model 123
InfiniBand CA 123
SDP vs. IPoIB (IB QDR) 124
IB on the WAN 126
Features 127
• Supports SONET OC-192 or 128
“appear” local 128
Sunnyvale 129
700 miles 130
3300 miles 4300 miles 130
Cluster B 131
Communication Options in Grid 132
Interface 133
Modern WAN 133
Performance of Memory Based 134
Data Transfer 134
ADTS-64MB 135
IPoIB-64MB 135
Application Level Performance 136
Memcached Design Using Verbs 139
Memcached Get Latency 140
Memcached Get TPS 141
Bandwidth with C version 142
Hadoop Sort Benchmark 145
Concluding Remarks 147
Funding Acknowledgments 148
Personnel Acknowledgments 149
Web Pointers 150

Komentáře k této Příručce

Žádné komentáře

SGI 10-Gigabit Ethernet Network Adapter Uživatelský manuál Strana 112

Komentáře k této Příručce

Související produkty a manuály pro Sítě SGI 10-Gigabit Ethernet Network Adapter