Download & Share Technology Presentations http://goo.gl/k80oY0 Student Guide & Best http://goo.gl/6OkI77

1. 1

2. Sun SPARC Enterprise Mx000 Servers -
A Deep Dive (Part 1)
Presenter’s Name
Presenter’s Title
8.4

3. 3
Sun SPARC Enterprise Midrange
and High-end Servers

4. 4
• Mainframe-class reliability
• Highly scalable performance
• Advanced virtualization
• Seamless scalability
• Easy upgradability
• Balanced SMP design
• Price/performance targeted
• Competitive RAS and S/W
• Dense rack mount package
• Highest availability
• Highest absolute performance
• Sophisticated resource control
• Highest scalability
• Entry level enterprise system
• Mainframe-class RAS
• Dense rack mount package
SPARC Enterprise Servers
M4000 and M5000
M8000 and M9000
M3000

5. 5
Term Chart
Term Definition
CPU memory board Unit (CMU) Unit equipped with a CPU module and memory
IO board unit (IOU) Unit equipped with multiple PCI cards and disk drives (M8000/M9000
only)
eXtended System Board (XSB) Either the entire CMU (Uni-XSB) or a quarter of the a CMU (Quad-
XSB). Uni-XSB sometimes referred to as just PSB. Quad-XSB
sometimes referred to as XSB.
Logical System Board (LSB) A number assigned to an XSB to identify it in a
domain
XBU (M9000 Only) Cross Bar Unit. Connects CMUs & IOUs
CLKU (M9000 Only) Clock Unit that provides Clock for ASICs.
BPA (M8000 Only) Backplane with integrated XB and CLK
XSCFU eXtended System Control Facility. Three types: XSCFU (M3000-
M5000), XSCFU_B (M8000, M9000-32, Base cab of M9000-64), and
XSCFU_C (Exp cab of M9000-64)

6. 6
<Insert Picture Here>
Entry Level Systems –
M3000

7. 7
Updated Entry Level Enterprise Server
• 2U Rackmount
• Single SPARC64 VII
quad-core processor
– 2.75GHz/5MB$, four core with two
SMT threads per core
– Mainframe-heritage RAS features
• Up to 64GB main memory,
with 8GB DIMM
• Four PCI-Express
(low-profile) slots
•Four disk bays (2.5” SAS)
•On-board four Gigabit Ethernet,
one SAS port (two wide)
•XSCF; eXtended System
Control Facility
– XCP1091
•Solaris 10 10/08 (U6) + MU8
Sun SPARC Enterprise M3000 Server

8. 8
Data
Centric
Network
Centric
Multi-Thread
Performance
Single-Thread
Performance
SPARC Enterprise
M3000 server
SPARC Enterprise
T5220 server
Sun Fire V490 server
M3000 Workloads
The SPARC
Enterprise M3000 is
suitable for a broad
application space;
batch/DB as well as
AP/network front-end
server
SPARC64TM
VII Provides Balanced, High-Performance

9. 9
Product Comparison
M3000 T5220 V490
Processor SPARC64 VII
2.75 GHz, 5 MB L2$
Max 1 (4 cores)
UltraSPARC T2
1.4/1.6 GHz, 4 MB L2$
Max 1 (4,6, or 8 cores)
UltraSPARC IV+
2.1 GHz, 2 MB L2$,
32 MB L3$
Max 4 (8 cores)
Memory Max 64 GB (8 DIMMs) Max 128 GB (16 DIMMs) Max 64 (16 DIMMs)
Internal Disks Max 4 (2.5” SAS) Max 16 (2.5” SAS) Max 2 (2x FC-AL)
Removable Media DVD DVD DVD
On-Board I/O 4 GbE, 2 10 GbE, 1 x2 SAS, 1
Serial, 1 USB, 2 UPC
4 GbE, 1 Serial, 4 USB 2 GbE, 1FC, 3 Serial, 2 USB
PCI Slots 4 PCIe 6 PCIe 6 PCI
Enclosure 2U 2U 5U
Redundant Parts Disk, PSU, Fan Disk, PSU, Fan Disk, PSU, Fan
Hot-Swappable Parts Disk, PSU, Fan Disk, PSU, Fan Disk, PSU, Fan
Power Options 1-phase, 1+1 cables 1-phase, 1+1 cables 1-phase, 1+1 cables
Power Consumption Max 400W Max 896W Max 1,750W
System Management XSCF ILOM RSC

10. 10
Hardware Specifications
Measurement North America Metric
2RU 2RU
Height 3.43 in 8.7 cm
Width 17.4 in 44 cm
Depth 25.91 in 65.7 cm
Weight (max.) 50.6 lb. 22 kg

11. 11
Block Diagram
Slot 0
Slot 1
Slot 2
Slot 3

12. 12
M3000 IOU Device Paths

13. 13
I/O Slot to Device Path Tables
IOU Slot Slot Type XSB Device Path
Slot 0 PCIe 00-0 /pci@0,600000/pci@0/pci@8
Slot 1 PCIe 00-0 /pci@1,700000/pci@0/pci@0
Slot 2 PCIe 00-0 /pci@1,700000/pci@0/pci@8
Slot 3 PCIe 00-0 /pci@1,700000/pci@0/pci@9
Accessible
Internal Devices
Physical
Location
XSB Device Path
Network Port 0 System 00-0 /pci@0,600000/pci@0/pci@1/pci@0/network@4
Network Port 1 System 00-0 /pci@0,600000/pci@0/pci@1/pci@0/network@4,1
Network Port 2 System 00-0 /pci@0,600000/pci@0/pci@2/pci@0/network@4
Network Port 4 System 00-0 /pci@0,600000/pci@0/pci@2/pci@0/network@4,1
HD0 System 00-0 /pci@0,600000/pci@0/pci@0/scsi@0/disk@0
HD1 System 00-0 /pci@0,600000/pci@0/pci@0/scsi@0/disk@1
HD2 System 00-0 /pci@0,600000/pci@0/pci@0/scsi@0/disk@2
HD3 System 00-0 /pci@0,600000/pci@0/pci@0/scsi@0/disk@3
DVD System 00-0 /pci@0,600000/pci@0/pci@0/scsi@0/disk@4
SAS Port System 00-0 /pci@0,600000/pci@0/pci@0/scsi@0/disk@5 or
/pci@0,600000/pci@0/pci@0/scsi@0/disk@6

14. 14
System Cabinet
Hard Disk DrivesPower Switch
DVD DriveStatus Indicators
(Power, XSCF ready, System check)
Key Switch
(Lock, Service)
PCI slots Gigabit Ethernet
UPC
RCI
SAS
Front View
Rear View
USB Serial LAN 0 & 1

15. 15
DIMMSystem controller
Processor ModuleMCBI-SCF
RAM for
firmware
XSCF
Processor
XSCF
DVD
connector
Switch FLP
I/O controller (Oberon)
DCDC converter
Temperature
sensor
Detailed View

16. 16
Key Features (cont.)
• Eco-friendly
• Small size; system size is almost same as T5220
• Low power; Max. Watts less than T5220/T5120
• Multi-speed Fans provide optimized noise according to
temperature; operating acoustic noise ≤ 50dBA(target)
• Improved System Reliability/Availability
• Data validation by ECC/parity mechanism btw major
components
• Dual Power Feed, as well as hot-swappable/redundant Fan,
PSU and Disk
• Online firmware upgrade
• Harmonized manageability with M-Series Servers
• XSCF provides same user interface and remote notification

17. 17
JSC: Jupiter System Controller
• JSC is a chip, which is combined functionalities of
SC and MAC chips used in M4000-M9000 systems.
• Control the mutual access among CPU, memory and
IO
• Features:
• Consists of 1 JSC chip, and has the interface to 1 CPU and
1 Oberon.
• Internal 1224MHz operation
• Connects with CPU at 1224MHz
• Contains 612 MHz DDR-II interface and supports up to 8GB
for DIMM capacity.

18. 18
PCI-Express Sub-System
•PCIe (Gen 1) x8 lane support
•Not hot-pluggable
•No PCI-X support
•Low profile only
4 x GbE
PCIe
Bridge
“Oberon”
PCIe
Switch
PCIexpress Slot
PCIexpress Slot
PCIexpress Slot
PCIexpress Slot
8
8
8
8
G-Ether
SAS
4 Disks
DVD
External SAS
Interface (2 wide)
PCIe
Switch

19. 19
Cooling
•2 cooling zones
•3 variable speed fans
Zone 1
Zone 2
= Sensor(intake)
= Sensor (CPU)
= Sensor (exhaust)

20. 20
Power Condition: Single Phase AC
• Rated voltage: AC100-120 / 200-240 V +/- 10%
• Frequency: 50/60Hz (+2% / -4%)
• Power consumption: 505W max
• AC plug: 2 plugs
• 100V & 200V
• IEC60309 16A 250V (RoW w/o Japan,Korea and Taiwan)
• IEC60320-C14/13 250V 15A(NA,Japan)
• 2 PSU standard

21. 21
A word on power consumption....
• M3000 is listed as 505W max
• Actual power draw based on max config is 400W
• Quote both actual lines draw and the power supply
rating

22. 22
Managed FAN Speed
Room temperature affects sound pressure level
Sound pressure level of server at Celsius 25 is the same as the quiet
office, 47dB *
Sun Fire V445 rates 77dB; SPARC Enterprise T5220 rates 74dB
Level1
(～
19.9)
Level2
(19.0～
21.9)
Level3
(21.0～
23.9)
Level4
(23.0～
25.9)
Level5
(25.0～
27.9)
Level6
(27.0～
29.9)
Level7
(29.0～
31.9)
Level8
(31.0～
33.9)Level9
(33.0～
)
47 dB
25 degrees C
Sound Pressure
[dB]
Rang of Temperature [Celsius]
42
46
50
Sound Pressure
[dB]

23. 23
MBU_B
• Main board, integrated MBU, CPUM, MEMB, IOU,
and DDCR which existed on M4000/M5000 system
• 4 core CPU is soldered. In case of broken, MBU_B
itself will be replaced
• Eight DDR2 (612 MHz) SDRAM slots
• 2-way interleave
• Four low profile PCIe slots

24. 24
MBCI-SCF
• MBCI - Maintenance Bus Controller Interface
• MBCI-SCF is a communication device between MBU
and XSCF
• Access to POST/OBP FMEM
• Emulation of serial port
• Communication interface to XSCF
• Provides maintenance bus
MBU
XSCF
CPU
JSC
FMEM
(POST/OBP)
MBCI-SCF
SRAM
PPC

25. 25
Operator Panel
• Same specifications as current M-Series
systems
• 3 LEDS, 1 push switch, 1 key switch
• System Power LED, XSCF ready LED and System
check LED
• System power on/off; Push switch
• Key switch; Lock or Service

26. 26
Memory Installation
• There are 8 DIMM slots on an MBU
Group-A: x4 slots Group-B: x4 slots
• Install Group-A, first
• Install same type of DIMMs to slots in a group
• Same type means same capacity and same rank
• Capacity of DIMM in Group-B must be smaller or
equal to DIMM in Group-A
Ex.1: Group-A: 2GB 1rank DIMM Group-B: 1GB 1rank DIMM
Ex.2: Group-A: 2GB 1rank DIMM Group-B: 2GB 1rank DIMM

27. 27
System Bandwidth
M3000 (2.52 GHz) M3000 (2.75 GHz)
Stream (Copy) 5.6 GB/s 6.4 GB/s
Stream (Triad) 4.5 GB/s 5.1 GB/s
Peak System Bandwidth a
17 GB/s 20 GB/s
Peak I/O Bandwidth b
4 GB/s 4 GB/s
a. Theoretical system bandwidth at peak is calculated by multiplying the bus width by the bus frequency between the SC and the MAC.
b. Theoretical I/O bandwidth at peak is calculated by multiplying the bus width by the bus frequency between the SC and the PCIe bridge.

28. 28
Degradation
(Page)
Dual
power
feed
Reliability at Component Level
Memory
CPU
Dynamic
Degradation
(Cache Way or
Core)
Fans
Redundancy
Hot-swap
Hardware
Power Supply
Units
Redundancy
Hot-swap
Disk Drives
Redundanc
y
Hot-swap
* Available with
Software
Minimizes Unplanned
Downtime
Minimizes Planned
Downtime

29. 29
M3000% prtdiag
System Configuration: Sun Microsystems sun4u Fujitsu SPARC Enterprise M3000 Server
System clock frequency: 1224 MHz
Memory size: 65536 Megabytes
==================================== CPUs ====================================
CPU CPU Run L2$ CPU CPU
LSB Chip ID MHz MB Impl. Mask
--- ---- -------------------------------------- ---- --- ----- ----
00 0 0, 1, 2, 3, 4, 5, 6, 7 2750 5.0 7 160
============================ Memory Configuration ============================
Memory Available Memory DIMM # of Mirror Interleave
LSB Group Size Status Size DIMMs Mode Factor
--- ------ ------------------ ------- ------ ----- ------- ----------
00 A 32768MB okay 8192MB 4 no 2-way
00 B 32768MB okay 8192MB 4 no 2-way
========================= IO Cards =========================
LSB Name Model
--- --------- ------------
00 scsi LSI,1068E
00 network N/A
00 network N/A
00 network N/A
00 network N/A
==================== Hardware Revisions ====================
System PROM revisions:
----------------------
OBP 4.24.11 2009/04/21 14:53
=================== Environmental Status ===================
Mode switch is in UNLOCK mode
=================== System Processor Mode ===================
SPARC64-VII mode
System Details

30. 30
Differences With Current M-Series
Servers
• Chips for on-board devices
• M4000-M9000: SAS1064 X 1, BCM5704C X 1, PEX8532 X 1
• M3000: SAS1068E X 1, BCM5715C X 2, PEX8533 X 2
• External SAS i/f: newly added
• New ASIC JSC: SC + MAC
• Single mother board structure: MBU
• M4000/M5000: MBU+CPUM+MEMB+IOU+DDCR
• Reduction in boards provides higher reliability
• Change of power/cooling subsystem
• Number of FAN/DDC are changed

31. 31
Differences With Current M-Series
Servers
• SC to MAC data path
• M4000-M9000 has ECC on data path
• JSC uses parity for SC-MAC communication
• No support for the I/O Expansion Unit
• Domaining (These details will be covered later)
• Single domain. ID is 0
• LSB=0
• SB0 is in uni-xsb mode
• No DR support
• No memory mirroring

32. 32
External SAS Device Support

33. 33
Introduction to SAS
• Serial Attached SCSI
• Point to point serial communication protocol to transfer data
to/from devices
• SAS specification defines 3 protocols
• SSP – Serial SCSI Protocol
• STP – SATA Tunneling Protocol
• SMP – SAS Management Protocol

34. 34
Storage Protocol Comparison

35. 35
Advantages of SAS
• Advantages over SATA:
• Multi host support
• More robust error recovery and reporting
• Encapsulates SATA protocol
• Advantages over parallel SCSI:
• Supports larger number of devices (over 16K)
• Limited bus contention and full speed realized from
Initiator/Target
• Does not require termination
• Advantages over FiberChannel:
• Cheaper
• Less complex
• Supports SATA drives

36. 36
SAS – Connector Icons
• SAS defines 3 receptacle icons
• Circle – enclosure in
• Diamond – enclosure out
• Circle and Diamond – enclosure universal (Sun default)
• SAS also defines connector keys that prevent
incorrect cabling

37. 37
Understanding SAS “Wide” Ports
• Scaleable connect
• “x4 wide” creates 4 lanes in a single
connector
• A single SAS lane is 3 Gb/s
• The M3000 uses 2 wide, or 2 lanes
• M3000 SAS port can drive 6Gb/s
• Can support 117 external devices using
built-in SAS connection

38. 38
<Insert Picture Here>
Mid-Range Systems –
M4000 and M5000

39. 39
M4000/M5000: Mid-Range Systems
Overview
• Application acceleration
• Uses dual core SPARC64 VI and quad
core SPARC64 VII processors
• Can mix processors in same system and
domains
• Less down time
• Greatly improved RAS functionality
• Economical, modular datacenter
design
• 6U and 10U, rack-optimized enclosures
• Extended domaining and dynamic
reconfiguration support
• Up to 4 fully usable domains
• Solaris 10 operating system

40. 40
Side note: Getting Our Terms Straight
Uni-XSB: When the system board is not logically
divided, it consists of a single XSB
Quad-XSB: When the system board is logically divided,
it consists of four XSBs

41. 41
Mid-range Product Data Sheet
M4000 M5000
Processor SPARC64 VI
2.15 GHz, 5 MB L2$
Max 4 (8 cores)
SPARC64 VII
2.4 GHz, 5 MB L2$
2.53 GHz, 5.5 MB L2$
Max 4 (16 cores)
SPARC64 VI
2.15 GHz, 5 MB L2$
Max 8 (16 cores)
SPARC64 VII
2.4 GHz, 5 MB L2$
2.53 GHz, 5.5 MB L2$
Max 8 (32 cores)
SPARC64 VII+
2.66 GHz, 11 MB L2$
Max 8 (32 cores)
SPARC64 VII+
2.66 GHz, 11 MB L2$
Max 4 (16 cores)
Memory Max 256 GB (32 DIMMs) Max 512 GB (64 DIMMs)
Internal Disks Max 2 (2.5” SAS) Max 4 (2.5” SAS)
Removable Media DVD, DAT
On-Board I/O 2 GbE, 2 10/100E,1 Serial, 1 USB, 2 UPC 4 GbE, 2 10/100E, 1 Serial, 1 USB, 2 UPC
PCI Slots 4 PCIe, 1 PCI-X 8 PCIe, 2 PCI-X
Enclosure 6U 10U
Redundant Parts Disk, PSU, Fan
Hot-Swappable Parts Disk, PSU, Fan
Power Options 1-phase, 1+1 cables 1-phase, 2+2 cables
Power Consumption Max 1390W Max 2677W
System Management XSCF (service processor)
Domains Max 2 Max 4

42. 42
Mid-range Product Comparison
M4000 V490
Processor SPARC64 VI
2.15 GHz, 5 MB L2$
Max 4 (8 cores)
SPARC64 VII
2.4 GHz, 5 MB L2$
2.53 GHz, 5.5 MB L2$
Max 4 (16 cores)
UltraSPARC IV+
2.1 GHz, 2 MB L2$,
32 MB L3$
Max 4 (8 cores)
SPARC64 VII+
2.66 GHz, 11 MB L2$
Max 4 (16 cores)
Memory Max 256 GB (32 DIMMs) Max 64 (16 DIMMs)
Internal Disks Max 2 (2.5” SAS) Max 2 (2x FC-AL)
Removable Media DVD, DAT DVD
On-Board I/O 2 GbE, 2 10/100E,1 Serial, 1 USB, 2 UPC 2 GbE, 1FC, 3 Serial, 2 USB
PCI Slots 4 PCIe, 1 PCI-X 6 PCI
Enclosure 6U 5U
Redundant Parts Disk, PSU, Fan Disk, PSU, Fan
Hot-Swappable Parts Disk, PSU, Fan Disk, PSU, Fan
Power Options 1-phase, 1+1 cables 1-phase, 1+1 cables
Power Consumption Max 1390W Max 1,750W
System Management XSCF (service processor) RSC
Domains Max 2 Max 1

43. 43
Mid-range Product Comparison
M5000 V890
Processor SPARC64 VI
2.15 GHz, 5 MB L2$
Max 8 (16 cores)
SPARC64 VII
2.4 GHz, 5 MB L2$
2.53 GHz, 5.5 MB L2$
Max 8 (32 cores)
UltraSPARC IV+
2.1 GHz, 2 MB L2$,
32 MB L3$
Max 8 (16 cores)
SPARC64 VII+
2.66 GHz, 11 MB L2$
Max 8 (32 cores)
Memory Max 512 GB (64 DIMMs) Max 128 (32 DIMMs)
Internal Disks Max 4 (2.5” SAS) Max 12
Removable Media DVD, DAT DVD, DDS-4
On-Board I/O 4 GbE, 2 10/100E, 1 Serial, 1 USB, 2 UPC 1 GbE
PCI Slots 8 PCIe, 2 PCI-X 9 PCI
Enclosure 10U 17.5U
Redundant Parts Disk, PSU, Fan Disk, PSU, Fan
Hot-Swappable Parts Disk, PSU, Fan Disk, PSU, Fan
Power Options 1-phase, 2+2 cables 1-phase, 2+1 cables
System Management XSCF (service processor) RSC
Domains Max 4 Max 1
Peak Bandwidth 64 GB/s 9.6 GB/s
I/O Bandwidth 16 GB/s 1.2 GB/s

44. 44
Methodology Theoretical Peak Data B/W
• The data B/W values come from the SC (System
Controller) to MAC (Memory Access Controller) bus-
width multiplied by the frequency, then multiplied by
the number of SB (System Boards).
M4000: 32GB/s = (16B(in)+16B(out)) x 0.506GHz x 2SC x 1SB
M5000: 64GB/s = (16B(in)+16B(out)) x 0.506GHz x 2SC x 2SB

45. 45
Methodology Theoretical Peak I/O B/W
• The I/O B/W values come from the SC (System
Controller) to Oberon (PCI Host Bridge) bus-width
multiplied by the frequency.
M4000: 8GB/s = (4B(in)+4B(out)) x 0.506GHz x 2SC x 1IOB
M5000: 16GB/s = (4B(in)+4B(out)) x 0.506GHz x 2SC x 2IOB

46. 46
Mid-range System Servers
M4000 System
M5000 System
Front
Front Back
Back
Disk
PSU
DVD
CPU Memory
DAT
Fan
PCI
XSCF
CPU
Fan
Disk
DAT
PSU
DVD
Memory XSCF
PCI

47. 47
M4000 Product Concept
Front Iso View Rear Iso View
2 x 172mm Fans
2 x PSU
2 x
HDD,
DVD,
DAT
4 x
MEM
Modules
1 x IO Tray
(1 PCI-X, 4 PCI-Express)
XSCFU
2 x CPU
Modules

48. 48
M5000 Product Concept
Front Iso View Rear Iso View
4 x 172mm Fans
4 x PSU
4 x HDD,
DVD, DAT
8 x MEM
Modules
4 x CPU
Modules
2 x IO Trays
(1 PCI-X, 4 PCI-Express per Tray)
XSCFU

49. 49
Mid-Range Systems Components
M4000 Motherboard
CPU Module
Connectors
Memory Board
Connectors
SC
ASICs

50. 50
Mid-Range Systems Components
CPU Module
•2.15GHz SPARC64 VI
•2.4GHz/2.53GHz
SPARC64 VII
•2.66GHz SPARC64 VII+
•2 CPUs per module
•Both CPUs the same type
and speed
•2 (M4000) or
4 (M5000) modules
per system

51. 51
Mid-Range Systems Components
Memory Board
•8 DIMM slots
•1, 2, 4, and 8GB
DIMMs
•ECC
•Extended-ECC
•Mirrored
•4 (M4000) or
8 (M5000) modules
per system
•Minimum is 4 x 1GB
DIMMs

52. 52
Mid-Range Servers I/O Tray
Components
•Controls DVD, DAT
•Controls 2 SAS disks
•4 x PCIe
•1 x PCI-X
•2 GbE ports
•1 tray in M4000
•2 trays in M5000
I/O Tray

53. 53
System Views: M4000

54. 54
System Views: M5000

55. 55
PCI EXPRESS
(switched point to point)
PCI vs PCIexpress I/O Technology
PCI
(shared bus)

56. 56
PCIe Bandwidth
Technology Bus Speed Bus Width Bandwidth Comments
PCI Shared Bus
33 MHz 32 bit 133 MB/s
33 MHz 64 bit 266 MB/s
66 MHz 32 bit 266 MB/s
66 MHz 64 bit 532 MB/s
PCI-X Shared Bus
66 MHz 64 bit 512 MB/s
100 MHz 64 bit 800 MB/s
133 MHz 64 bit 1 GB/s Current supported speed
266 MHz 64 bit 2 GB/s
PCI Express Switched Point to Point
x1 500 MB/s
x2 1 GB/s
x4 2 GB/s
x8 4 GB/s Current supported speed
x12 6 GB/s
x16 8 GB/s
x32 16 GB/s

57. 57
PCICS: PCI Cassette for M4000/M5000
IO Cage
In PCICS, PCIexpress Short dedicated cassette provided
Carrier main
body (metal)
Moveable carrier
plate (plastic)
Carrier slot
keyholes
Operating
handle
Card retainer
(5 total, 3 types)

58. 58
2 x GbE
4
PCIe to
PCI-X Bridge
PCI-X Slot IOU
PCIe
Bridge
PCIe
Switch
PCIe
Switch
PCIe
Switch
8PCIexpress Slot
PCIexpress Slot
PCIexpress Slot
PCIexpress Slot
8
8
8
8
CPU
Module
CPU
Module
Internal
Storage DVD SC1SC0
System
Control Facility
SATA x4
8
DIMMs
MAC1 8
DIMMs
MAC0 CPU0 CPU1
Memory
Board
Memory
Board
CPU2 CPU3 MAC2
Memory
Board
MAC3 8
DIMMs
8
DIMMs
Memory
Board
Midrange M4000 Block Diagram

59. 59
SC ASIC
• The SC (System Controller) ASIC is responsible for
setup and enforcement of domaining.
• The multiple paths are for performance, not
redundancy
• The physical address space on each SB or CMU is
divided equally across the SCs
• Each SC stores L2$ tags for the processors on the
same SB or CMU.

60. 60
Midrange M5000 Block Diagram
CPU
Module
CPU
Module
8
DIMMs
MAC1 8
DIMMs
MAC0 CPU0 CPU1
Memory
Board
Memory
Board
CPU2 CPU3 MAC2
Memory
Board
MAC3 8
DIMMs
8
DIMMs
Memory
Board
IOU #0 and
Disks 0,1 and DVD/DAT
CPU
Module
CPU
Module
8
DIMMs
MAC4
8
DIMMs
MAC5 CPU4 CPU5
Memory
Board
Memory
Board
CPU6 CPU7
MAC
6
Memory
Board
MAC7 8
DIMMs
8
DIMMs
Memory
Board
IOU #1 and
Disks 2,3
SC3SC2
SC1SC0

61. 61
CPU
Module
CPU
Module
MAC1 CPU0 CPU1
Memory
Board
Memory
Board
CPU2 CPU3
Memory
Board
Memory
Board
Midrange M4000 Mirrored Memory
= Group A = Group B
4
DIMMs
Mirro
r
4
DIMMs
MAC0 MAC2 MAC3
4
DIMMs
Mirro
r
4
DIMMs
4
DIMMs
Mirro
r
4
DIMMs
4
DIMMs
Mirro
r
4
DIMMs

62. 62
CPU
Module
CPU
Module
MAC1 CPU0 CPU1
Memory
Board
Memory
Board
CPU2 CPU3
Memory
Board
Memory
Board
Midrange M5000 Mirrored Memory
= Group A = Group B
4
DIMMs
Mirro
r
4
DIMMs
MAC0 MAC2 MAC3
4
DIMMs
Mirro
r
4
DIMMs
4
DIMMs
Mirro
r
4
DIMMs
4
DIMMs
Mirro
r
4
DIMMs
CPU
Module
CPU
Module
MAC5 CPU4 CPU5
Memory
Board
Memory
Board
CPU6 CPU7
Memory
Board
Memory
Board
4
DIMMs
Mirro
r
4
DIMMs
MAC4 MAC6 MAC7
4
DIMMs
Mirro
r
4
DIMMs
4
DIMMs
Mirro
r
4
DIMMs
4
DIMMs
Mirro
r
4
DIMMs

63. 63
Memory Mirroring
Systems with no
memory mirroring
Memory mirroring on Mx000
Error Error
Memory Memory 0 Memory 1 Memory 0 Memory 1
Memory
Controller
Memory
Controller
Memory
Controller
Data read from memory Data read from memoryData write to memory
 On write, same data is
written to both memory 0 & 1
 On reads, reads from
memory 0 & 1
Even with uncorrectable error in memory 0, correct data in memory 1 is used
and business continuity is secured.

64. 64
Memory
Memory Patrolling
Memory Patrol (other vendors) Mx000 Servers Memory Patrol
Detection of memory corruption now earlier
CPU
Memory
Controller
OS
Memory
Memory
Controller
CPU
OSSoftware Memory
Patrolling
HW (MAC) incorporates
memory patrolling function
Burden on CPU
Swift patrolling
not enabled
No burden on CPU
Swift patrolling enabled
Memory patrolling may affect application performance,
so the frequency of patrolling is limited
Less influence on applications,
timely patrolling enabled

65. 65
Mid-Range M5000 I/O Block Diagram
M5000
Main
Board
2 x GbE
4
PCIe to
PCI-X Bridge
PCI-X Slot
PCIe
Bridge
PCIe
Switch
PCIe
Switch
PCIe
Switch
8
PCIexpress Slot
PCIexpress Slot
PCIexpress Slot
PCIexpress Slot
8
8
8
8
2 x GbE
4
PCIe to
PCI-X Bridge
PCI-X Slot IOU #0
PCIe
Bridge
PCIe
Switch
PCIe
Switch
PCIe
Switch
8
PCIexpress Slot
PCIexpress Slot
PCIexpress Slot
PCIexpress Slot
8
8
8
8
IOU #1

66. 66
Detailed M5000 Block Diagram

67. 67
Extended System Board on M4000
M4000 IOUM4000 MBU
MEM
Board
MEM
Board
MEM
Board
MEM
Board
PCIe
Switch
CPU Module
Uni-XSB 00-0
DVD/D
AT
Disks #0
& #1CPU
CPU
CPU Module
CPU
CPU
PCI-X
Slot
PCIe Slot 0
PCIe Slot 1
PCIe Slot 2
PCIe Slot 3
PCIe
Switch

68. 68
Extended System Board on M4000
M4000 IOUM4000 MBU
MEM
Board
MEM
Board
MEM
Board
MEM
Board
PCIe
Switch
CPU Module
Quad-XSB
00-3
Quad-XSB
00-2
Quad-XSB
00-1
Quad-XSB
00-0
DVD/D
AT
Disks #0
& #1CPU
CPU
CPU Module
CPU
CPU
PCI-X
Slot
PCIe Slot 0
PCIe Slot 1
PCIe Slot 2
PCIe Slot 3
PCIe
Switch

69. 69
Extended System Board on M5000
M5000 IOU #1M5000 MBU
MEM
Board
MEM
Board
MEM
Board
MEM
Board
Uni-XSB 00-0
MEM
Board
MEM
Board
MEM
Board
MEM
Board
Uni-XSB 01-0
CPU Module
CPU
CPU
CPU Module
CPU
CPU
CPU Module
CPU
CPU
CPU Module
CPU
CPU
SB0
SB1
M5000 IOU
#0PCI-X
Slot
PCIe Slot 2
PCIe Slot 3
PCIe Slot 0
PCIe Slot 1
PCI-X
Slot
PCIe Slot 2
PCIe Slot 3
PCIe Slot 0
PCIe Slot 1
DVD/D
AT
Disk 0,1
Disk
2,3
IO
Ctl
r
IO
Ctl
r
IO
Ctl
r
IO
Ctl
r

70. 70
Extended System Board on M5000
M5000 MBU
MEM
Board
MEM
Board
MEM
Board
MEM
Board
Quad-XSB
00-3
Quad-XSB
00-2
Quad-XSB
00-1
Quad-XSB
00-0
MEM
Board
MEM
Board
MEM
Board
MEM
Board
Quad-XSB
01-3
Quad-XSB
01-2
Quad-XSB
01-1
Quad-XSB
01-0
CPU Module
CPU
CPU
CPU Module
CPU
CPU
CPU Module
CPU
CPU
CPU Module
CPU
CPU
SB0
SB1
M5000 IOU #1
M5000 IOU #0
PCI-X
Slot
PCIe Slot 2
PCIe Slot 3
PCIe Slot 0
PCIe Slot 1
PCI-X
Slot
PCIe Slot 2
PCIe Slot 3
PCIe Slot 0
PCIe Slot 1
IO
Ctl
r
IO
Ctl
r
IO
Ctl
r
IO
Ctl
r
DVD/D
AT
Disk 0,1
Disk
2,3

71. 71
Uni-XSB 01-0
Extended System Board on M5000
M5000 MBU
MEM
Board
MEM
Board
MEM
Board
MEM
Board
Quad-XSB
00-3
Quad-XSB
00-2
Quad-XSB
00-1
Quad-XSB
00-0
MEM
Board
MEM
Board
MEM
Board
MEM
Board
CPU Module
CPU
CPU
CPU Module
CPU
CPU
CPU Module
CPU
CPU
CPU Module
CPU
CPU
SB0
SB1
M5000 IOU #1
M5000 IOU #0
PCI-X
Slot
PCIe Slot 2
PCIe Slot 3
PCIe Slot 0
PCIe Slot 1
PCI-X
Slot
PCIe Slot 2
PCIe Slot 3
PCIe Slot 0
PCIe Slot 1
IO
Ctlr
IO
Ctlr
IO
Ctlr
IO
Ctlr
DVD/D
AT
Disk 0,1
Disk
2,3

72. 72
Valid Memory Board Configurations
Per System Board
Uni-XSB Mode Quad-XSB Mode
= Memory Module= CPU Board

73. 73
M4000/M5000 Power Cords
= C19 to C20 Jumper cable ships standard with
each M4000/M5000
•Recommended rack is Sun Rack 1000-42 with the Modular
Power System (MPS)
•With Sun Rack 1000-42, using MPS, the power strips with
C13 receptacles must be unplugged from the MPS
•Standard jumper is 1.5m. Order SELX9P31z for the 2.5m
cable.

74. 74
<Insert Picture Here>
High-End Systems –
M8000, M9000

75. 75
M8000/M9000: High-End Systems
Overview
• Application acceleration
• Uses dual core SPARC64 VI and quad core SPARC64 VII
processors
• High performance interconnect
• Industry standard PCIe I/O
• State-of-the-Art RAS capabilities
• Unmatched configuration flexibility
• Modular building block architecture
(CMU, IOU, expansion box, XSCF)
• Initial SMP architecture up to 256 cores
• Expanded dynamic system domains
• Up to 24 fully usable domains
• Fine granularity of unit of domain
1 CPU chip, 4 DIMMs, 2 PCIe slots

76. 76
High-end Product Data Sheet
M8000 M9000-32 M9000-64
Processor SPARC64 VI & VII
VI 2.28/2.4 GHz, 5 MB L2$
VII: 2.52/2.88 GHz, 6 MB L2$
Max 16 (64 cores)
SPARC64 VI & VII
VI 2.28/2.4 GHz, 5 MB L2$
VII: 2.52/2.88 GHz, 6 MB L2$
Max 32 (128 cores)
SPARC64 VI & VII
VI 2.28/2.4 GHz, 5 MB L2$
VII: 2.52/2.88 GHz, 6 MB L2$
Max 64 (256 cores)
SPARC64 VII+
3.0 GHz,12 MB L2$
Max 16 (64 cores)
SPARC64 VII+
3.0 GHz,12 MB L2$
Max 32 (128 cores)
SPARC64 VII+
3.0 GHz,12 MB L2$
Max 64 (256 cores)
Memory Max 1 TB (128 DIMMs) Max 2 TB (256 DIMMs) Max 4 TB (512 DIMMs)
Internal Disks Max 16 (2.5” SAS) Max 32 (2.5” SAS) Max 64 (2.5” SAS)
Removable Media DVD, DAT
CMU and IOU Max 4 each Max 8 each Max 16 each
PCI Slots Max 32 PCIe Max 64 PCIe Max 128 PCIe
Domains Max 16 Max 24
Redundant Parts CMU, IOU, XSCF, Disk, PSU,
Fans in Fan Tray, XBU
CMU, IOU, XSCF, Disk, PSU, Fans in Fan Tray, XBU, CLKU
Hot-Swappable Parts CMU, IOU, XSCF, Disk, PSU, Fan Tray
Power Options 1-phase/3-phase/Dual Grid
Power Consumption Max 7.5KW Max 14.6KW Max 29.9KW
System Management XSCF (service processor)

77. 77
High-end Product Comparison
M8000 E6900
Processor SPARC64 VI & VII
VI 2.28/2.4 GHz, 5 MB L2$
VII: 2.52/2.88 GHz, 6 MB L2$
Max 16 (64 cores)
UltraSPARC IV+
1.95 GHz, 2 MB L2$, 32 MB L3$
Max 24 (48 cores)
SPARC64 VII+
3.0 GHz,12 MB L2$
Max 16 (64 cores)
Memory Max 1 TB (128 DIMMs) Max 384 GB (192 DIMMs)
Internal Disks Max 16 (2.5” SAS) None
Removable Media DVD, DAT None
CMU and IOU Max 4 each 6 Uniboards, 4 I/O Boats
PCI Slots Max 32 PCIe Max 24 PCI-X/100, 8 PCI
Domains Max 16 Max 4
Redundant Parts CMU, IOU, XSCF, Disk, PSU, Fans in Fan
Tray, XBU
Uni-Bd, IO-Bd, SC, PSU, Fans,
FireplaneSwitch
Hot-Swappable Parts CMU, IOU, XSCF, Disk, PSU, Fan Tray Uni-Bd, IO-Bd, SC, PSU, Fans
Power Options 1-phase/3-phase/Dual Grid 1-phase/Dual Grid
System Management XSCF (service processor) SC (System Controller)
System Bandwidth 184 GB/s 9.6 GB/s
I/O Bandwidth 61 GB/s 8 GB/s

78. 78
High-end Product Comparison
M9000 E25K
Processor SPARC64 VI & VII
VI 2.28/2.4 GHz, 5 MB L2$
VII: 2.52/2.88 GHz, 6 MB L2$
Max 64 (256 cores)
UltraSPARC IV+
1.8 GHz, 2 MB L2$, 32 MB L3$
Max 72 (144 cores)
SPARC64 VII+
3.0 GHz,12 MB L2$
Max 64 (256 cores)
Memory Max 4 TB (512 DIMMs) Max 1 TB (576 DIMMs)
Internal Disks Max 64 (2.5” SAS) None
Removable Media DVD, DAT None
CMU and IOU Max 16 each Max 18 each
PCI Slots Max 128 PCIe Max 72 PCI-X
Domains Max 24 Max 18
Redundant Parts CMU, IOU, XSCF, Disk, PSU, Fans in Fan
Tray, XBU, CLKU
Uni-Bd, IO-Bd, SC, PSU, Fans, Backplane
Hot-Swappable Parts CMU, IOU, XSCF, Disk, PSU, Fan Tray Uni-Bd, IO-Bd, SC, PSU, Fans
Power Options 1-phase/3-phase/Dual Grid 1-phase/Dual Grid
System Management XSCF (service processor) SC (System Controller)
System Bandwidth 737 GB/s 43 GB/s
I/O Bandwidth 244 GB/s 40 GB/s

79. 79
Methodology Theoretical Peak Data B/W
• The data B/W values come from the SC (System
Controller) to MAC (Memory Access Controller) bus-
width multiplied by the frequency, then multiplied by the
number of SB (System Boards).
M4000: 32GB/s = (16B(in)+16B(out)) x 0.506GHz x 2SC x 1SB
M5000: 64GB/s = (16B(in)+16B(out)) x 0.506GHz x 2SC x 2SB
M8000: 184GB/s = (16B(in)+8B(out)) x 0.480GHz x 4SC x 4SB
M9000 (32 CPUs): 368GB/s = (16B(in)+8B(out)) x 0.480GHz x 4SC x 8SB
M9000 (64 CPUs): 737GB/s = (16B(in)+8B(out)) x 0.480GHz x 4SC x 16SB

80. 80
Methodology Theoretical Peak I/O B/W
• The I/O B/W values come from the SC (System Controller)
to Oberon (PCI Host Bridge) bus-width multiplied by the
frequency.
M4000: 8GB/s = (4B(in)+4B(out)) x 0.506GHz x 2SC x 1IOB
M5000: 16GB/s = (4B(in)+4B(out)) x 0.506GHz x 2SC x 2IOB
M8000: 61GB/s = (4B(in)+4B(out)) x 0.480GHz x 2SC x 2Oberon x 4IOU
M9000 (32 CPUs): 122GB/s = (4B(in)+4B(out)) x 0.480GHz x 2SC x 2Oberon x 8IOU
M9000 (64 CPUs): 244GB/s = (4B(in)+4B(out)) x 0.480GHz x 2SC x 2Oberon x 16IOU

81. 81
Bandwidth and Streams Benchmark
•The theoretical system bandwidth at peak time is calculated by multiplying the bus width
by the bus frequency between the system controller and memory access controller.
•The theoretical I/O bandwidth at peak time is calculated by multiplying the bus width by
the bus frequency between the system controller and PCI bridge.
In GB/s
Theoretical
System
Bandwidth
Snoop
Bandwidth
Streams
(Triad)
Streams
(Copy)
Theoretical
I/O
Bandwidth
M4000 32 129 12.7 12.6 8
M5000 64 129 25.3 24.8 16
M8000 184 245 69.6 60.3 61
M9000-32 368 245 134.3 114.9 122
M9000-64 737 245 227.1 224.4 244

82. 82
High-end System Cabinets
M8000 System
M9000 System
M9000 System
Front Rear
CMU
Fan
Expansion
Cabinet
XB
PSU
DVD
IOU
XSCF
AC input selector
Clock Board
IOU
Fan
Base
Cabinet
** Rack for
I/O Expansion
**Rack for I/O
Expansion
** Rack for I/O Expansion
is optional and applies
to M8000, M9000
720
1800
1200
1200
1800
810

83. 83
COMMONALITIES (M8000 vs. M9000)
• CPU module
• CPU memory unit (CMU)
• I/O unit (IOU)
• FAN trays ( Fan_A (3))
• Power Supply Unit (PSU)
• XSCF board
• Operator Panel board
• 2.5” SAS disk
• DVD, DAT
• PCI cassette
• Memory
DIFFERENCIES (M8000 vs. M9000)
• Max. number of CMU and IOU
> M8000: 4CMU/IOU -> Max. 16 domain
> M9000: 8CMU/IOU in one cabinet -> Max
24domain
• XB/CLK
> M8000: XB/CLK exists on BP_A, Two clock
source, but one clock ASIC
> M9000: XB is in XBU, clock is in CLKU (Two
clock source, and two clock ASICs)
• DDC for XB/CLK
> M8000: Exist as a separate unit, DDC_A
> M9000: Integrated to XBU/CLKU
• Expansion cabinet for M9000 only
• Internal rack space
> M8000: 12U available
> M9000: None
• Power subsystem/Cooling subsystem
> Different number of PSU/power cables
> Different kind/number of Fans
M8000 and M9000 Comparison

84. 84
M8000/M9000 System's RAS
Management
•Redundant interconnect
•Redundant Service Processor
•No Single Failure can Prohibit Reboot
•Most FRU's are Hot-Swappable

85. 85
M8000 System
1-4 IO Units (IOU)
(4 Disk Bays
Each)
1-4 CMU
Boards
19” Rack Space
Fan Trays
PSU
Fan Trays
XSCFU_B#0
XSCFU_B#1
Rear View Front View
Rack space
for Dual
Power Grid
Option

86. 86
M9000-32 System
IO Boards
(4 Disk Bays
Each)
Fans
PSU’s
Crossbar
Boards
Clock
Boards
XSCF
Boards
DAT
DVD
Front
View

87. 87
M9000-32 System
IO Boards
(4 Disk Bays Each)
Fans
CMU Boards
Back View

88. 88
M9000-64 System
IO Boards
(4 Disk Bays
Each)
Fans
PSU’s
Crossbar
Boards
Clock
Boards
XSCF Boards
DAT
DVD
Front View
Clock
Boards
XSCF
Boards
Crossbar
Boards
DAT
DVD
IO Boards

89. 89
M9000-64 System (no cables)
Crossbar
Boards
Clock
Boards
XSCF
Boards
Crossbar
Boards
Clock
Boards
XSCF Boards

90. 90
M9000-64 System
IO
Boards
FansFans
CMU
Boards
Back View
CMU
Boards
IO
Boards

91. 91
M9000 + External Power Cabinets
External
Power
Cabinet
External
Power
Cabinet

92. 92
M9000 + External Power Cabinets
External
Power
Cabinet
External
Power
Cabinet

93. 93
Backplane BP_A (for M8000)
Active Backplane

94. 94
Backplane BP_B (for M9000)
Passive Backplane

95. 95
M8000/M9000 High-End Building Blocks
• Max 4 CPU sockets /
8 cores
• Can mix (2) SPARC64
VI and (2) SPARC64
VII/VII+ on same CMU
• 2x 16 DIMM slots
(1, 2, 4 and 8 GB)
• 16, 32, 64, 128, and
256 GB memory
expansions
• Memory mirror option
only in Uni-XSB mode
2x 16
DIMM
4 CPU
Sockets
Red. DC-DC
Converter
CPU/Memory Unit (CMU)

96. 96
Side note: Getting Our Terms Straight
Uni-XSB: When the system board is not logically
divided, it consists of a single XSB
Quad-XSB: When the system board is logically divided,
it consists of four XSBs

97. 97
M8000/M9000 High-End Building Blocks
• 1x IOU per CMU or
1x IOU per multiple CMU
• 2x 4 PCIe slots
• Optional base IO card
(2x SAS, 2x GbE)
using PCIe slot
• PCIe slot assignment
to 4x XSB
• Link up to 4 full I/O
expansion boats
(link cards use PCIe slots)
I/O Unit (IOU)
2x2 Disk Bays 2x4 PCIe Slots

98. 98
PCICS: PCI Cassette for M8000/M9000 IOU
Carrier main
body (metal)
Moveable carrier
plate (plastic)
Operating
handle Card retainer
(5 total, 3 types)
Circular area for
rotating the retainer
In PCICS, PCIexpress Short dedicated cassette provided

99. 99
•
•
•
CPU0
CPU1
CPU
2
CPU3
MAC0
MAC1
MAC2
MAC3
4
DIMMs
4
DIMMs
4
DIMMs
4
DIMMs
M8000/M9000 System Board Block Diagram
All Buses are 4B Uni-Directional Paths with Multiplexed Address and
Data Plus Control and ECC in Each Direction
SC0
SC1
SC2
SC3
4B per path @1.07 GHz
2 paths each direction
~5 to 6 GB/sec/way peak
(~30-35% data 65-70% broadcast
address and control)
to/from XB way-0
to/from XB way-1
to/from XB way-2
4B per path @ 0.53GHz
1 path each direction
to/from XB way-3
4B per path @1.07GHz
1 path each direction
4
DIMMs
4
DIMMs
4
DIMMs
4
DIMMs

100. 100
35 GB/sec
peak
M8000/M9000 System Board Block Diagram
8.7 GB/S
~20-24
GB/sec
peak
~40 GB/sec •
~10 GB/sec peak •
All Buses are 4B Uni-Directional Paths with Multiplexed Address and
Data Plus Control and ECC in Each Direction
CPU0
CPU1
CPU
2
CPU3
MAC0
MAC1
MAC2
MAC3
4
DIMMs
4
DIMMs
4
DIMMs
4
DIMMs
SC0
SC1
SC2
SC3
4
DIMMs
4
DIMMs
4
DIMMs
4
DIMMs
5~6 GB/sec
peak

101. 101
PCIexpress Short Slot8
PCIexpress Short Slot8
PCIexpress Short Slot8
PCIexpress Short Slot8
PCIexpress Short Slot8
PCIexpress Short Slot8
PCIexpress Short Slot8
PCIexpress Short Slot8
M8000/M9000 I/O Board Block Diagram
Base
IO
Board
PCIe
bridge
PCIe
bridge
PCIe
Switch
Internal DVD Drive
Internal DAT Drive
Internal Drives
EXP
DVD
DA
TGBit
Ethernet
SAS
PCIe
to PCI-
X
bridge
IOU
to/from
one way – 2 XB’s
EXP
in cabinet
5-6 GB/sec
peak
•
PCIe
Switch
PCIe
Switch
PCIe
Switch
to/from
one way – 2 XB’s

102. 102
IOUA (Base IO Card)
> To IOU Internal HDD when installed in PCI#0 or #4
> To DVD/DAT drive Via SAS exp when installed in PCI 0,2,4 or 6
Base
IO
Board
Internal DVD Drive
Internal DAT Drive
Internal Drives
GBit
Ethernet
SAS
PCIe
to PCI-
X
bridge

103. 103
~43 GBps
M8000/M9000 Boards
0.53 Ghz
~40
GBps
System Board
I/O Board
Paths are 4 bytes each way
and carry both address & data
~1/3 of traffic is data
1.07 GHz
24 Cables to
other Xbars
0.5–1 GHz
~48 GBps
~22GBps
XB Board
X8
X8
X8
X8
X8
X8
X8
X8
PCIexpress slot
PCIexpress slot
PCIexpress slot
PCIexpress slot
PCIexpress slot
PCIexpress slot
PCIexpress slot
PCIexpress slot
Processor
Memory Controller
Processor
Memory Controller
Memory Controller
Processor
Processor
Memory Controller
~22
GBps
System
Controller
System
Controller
System
Controller
System
Controlle
r
~5.5
GBps
PCIe x8
bridge
PCIe x8
bridge
8x8 Crossbar
8x8 Crossbar
8x8 Crossbar
8x8 Crossbar
~49
GBps
8 DDR2
DIMMs
8 DDR2
DIMMs
8 DDR2
DIMMs
8 DDR2
DIMMs
PCIe Switch
PCIe Switch
PCIe Switch
PCIe Switch

104. 104
Detailed M9000 Block Diagram

105. 105
Quad-XSB Configuration
System Board
Processor
Memory
Controller
8 DDR2
DIMMs
Processor
Memory
Controller
8 DDR2
DIMMs
Processor
Memory
Controller
8 DDR2
DIMMs
Processor
Memory
Controller
8 DDR2
DIMMs
Quad-XSB
00-3
Quad-XSB
00-2
Quad-XSB
00-1
Quad-XSB
00-0
I/O Board
PCIe slot 0
PCIe slot 1
PCIe slot 2
PCIe slot 3
PCIe slot 4
PCIe slot 5
PCIe slot 6
PCIe slot 7
PCIe x8
bridge
PCIe
x8
bridge
PCIe
Switch
PCIe
Switch
PCIe
Switch
PCIe
Switch

106. 106
M8000/M9000 Backplane Interconnect
XB
2
XB/way
4 ways
I
O
U
I
O
U
I
O
U
IOU
M9000
C
M
C
M
CMU
XB
2
XB/way
4 ways
I
O
U
I
O
U
I
O
U
IOU
C
M
C
M
CMU
368 GB/sec
peak
M8000
C
M CMU
I
O
U
I
O
U
I
O
U
IOU
XB
2 XB/way
4 ways
184
GB/sec
peak

107. 107
Scaleable Interconnect
M8000
C
M
C
M
CMU
I
O
U
I
O
U
I
O
U
IOU
XB
2 XB/way
4 ways
XB
2
XB/way
4 ways
I
O
U
I
O
U
I
O
U
IOU
M9000
C
M
C
U
CMU
I
O
U
I
O
U
I
O
U
IOU
XB
2 XB/way
4 ways
C
M
C
M
CMU
XB
2
XB/way
4 ways
I
O
U
I
O
U
I
O
U
IOU
M9000
C
M
C
M
CMU
368 GB/sec
peak
737 GB/sec
peak
184
GB/sec
peak

108. 108
Operator Panel
LED, Button or
Switch Name
Icon Color State/Setting Description
Power Green On
Off
System Power is on
System power is off
Standby
XSCF
Green On
Blinking
Indicates the system can be powered on
XSCF on-Indicates that the system can be powered on.
System initialization/power on processes in progress
Off XSCF off-Indicates that the system cannot be powered on. (circuit breakers are off)
Check Amber On
Blinking
Indicates the operator panel is the maintenance target device.
Indicates these conditions;
● Error detected. System startup disabled
● Power failure occurred
Off Indicates the operating status of the system.
Indicates one of three conditions;
● Normal operating state
● Circuit breakers are switched off
● Power failure occurred
POWER Button Less than 4 secs. ● Regardless of the MODE switch state, powers on all domains
● Skips processing for waiting for facility and power-on warm up.
4 secs. Or longer
(Service)
Performs one of these functions;
● If power-on is being performed, the power-on processing is cancelled and the system is powered
off.
● If the system is being powered off, the operator of the power switch is ignored. The power-off
process continues until the system is powered off.

109. 109
Operator Panel
LED, Button or
Switch Name
Icon Color State/S
etting
Description
MODE Switch (1) Locked ● Normal operator made. (recommended setting for day-to-day operators)
● System can be powered on.
● System cannot be powered off.
● Key can be removed.
MODE Switch (2) Service ● Service should be provided. (switch must be set to service mode)
● Key cannot be removed.
● System can be powered on and off with the POWER button.
● Prevents unintentional power-on.

110. 110
M9000-64 Cabling
XB Data : 48 cables
XB CLK : 8 cables
CLK Data : 2 cables
CLK CLK : 4 cables
XSCF Data : 2 cables
Total : 64 cables

111. 111
M9000-64 Cabling

112. 112
XBU #8 to #15
M9000-64 Cabling: XB
M9000 Base Cabinet M9000 Ex. Cabinet
Yellow
Pink
Brown
Green
Orange
Blue
Colored Tie Wraps
XBU #0 to #7
1.3m
Data Cable
1.0m
CLK Cable
SPOF

113. 113
M9000-64 Cabling: XB
M9000 Base Cabinet M9000 Ex. Cabinet

114. 114
M9000-64 Cabling: CLKU
CLKU#0 CLKU#1
out out in out out in
M9000 Base Cabinet M9000 Ex. Cabinet
CLKU#2 CLKU#3
out out in out out in
Data Cable
CLK Cable
SPOF

115. 115
M9000-64 Cabling: XSCF
XSCFU_B#0 XSCFU_B#1
M9000 Base Cabinet M9000 Ex. Cabinet
XSCFU_C#0 XSCFU_C#1
Data Cable

116. 116
Airflow
Front Back
M8000 M9000

117. 117
Dual Power Feed, 3-Phase Power for M8000
Single phase input & Dual Power
Feed
3-phase input (Delta) & Dual Power
Feed
3-phase input (Star) & Dual Power
Feed
Single Phase Input
(standard) 3 Cables
Base Cabinet
12U DPF
Option
3 Phase Input (DPF)
Base Cabinet
12
U
Power
Cabinet
3
Phase
(Delta
)
DPF
3 Phase Input (DPF)
Base Cabinet
12
U
Power
Cabinet
3
Phase
(Star)
DPF
7
U
Single
Phase
Input
(DPF
Option)
* Customer must have their electrician make 3-phase cables
Descriptions Remarks
1 Single phase DPF unit 7U rack unit for M8000 base cabinet, field installable
2 M8000 base cabinet + power cabinet for 3-
Phase (delta) DPF
M8000 Base cabinet + Power cabinet for 3-Phas (delta)/ Dual
Power Feed Power, 2 lines, Delta (208V) is mainly used in
Japan and North America.
3 M8000 base cabinet + power cabinet for 3-
Phase (star) DPF
M8000 Base cabinet + Power cabinet for 3-Phase(star)/ Dual
Power Phase, 2 lines, Star(415V) is mainly used in EU region.

118. 118
Dual Power Feed, 3-Phase Power for
M9000, Single Cabinet
3-phase input & Dual Power Feed
3 Phase Input (DPF)
Base CabinetPower
Cabinet
3
Phase
(Delta
or
Star)
DPF
Single phase input & Dual Power Feed
Base CabinetPower
Cabinet
DPF
Option
Single Phase Input
(DPF Option)
Single Phase Input
(Standard) 5 Cables
* Customer must have their electrician make 3-phase cables
Descriptions Remarks
1 Power cabinet for single phase DPF Power cabinet, field installable
2 M9000 base cabinet + power cabinet for 3-
Phase (delta) DPF
M9000 base cabinet for 3-phase(delta)/ Dual Power Feed, 2
lines, Delta(208V) is mainly used in Japan and North America.
3 M9000 base cabinet + power cabinet for 3-
Phase (star) DPF
M9000 base cabinet for 3-phase(star)/ dual Power Feed, 2
lines, Star(415V) is mainly used in EU region.

119. 119
Dual Power Feed, 3-Phase Power for
M9000, Dual Cabinets
3 Phase
Input (DPF)
3 Phase
Input (DPF)
3-phase input, Dual Power Feed
Extension
Cabinet
Base CabinetPower
Cabinet
3 Phase
(Delta
or Star)
DPF
Power
Cabinet
3 Phase
(Delta
or Star)
DPF
Single Phase
Input
(DPF Option)
Single Phase
Input
(Standard)
5 Cables
Single phase input, Dual Power Feed
Base CabinetPower
Cabinet
DPF
Option
Extension
Cabinet
Power
Cabinet
DPF
Option
Single Phase
Input
(Standard)
Single Phase
Input
(DPF Option)
* Customer must have their electrician make 3-phase cables
Descriptions Remarks
1 Dual power cabinet for single phase DPF 2x Power cabinet, field installable
2 M9000 base cabinet + expansion cabinet for 3-
Phase (delta) DPF
M9000 Base cabinet + Power cabinet for 3-Phase(delta)/ Dual
Power Feed Power, 4 lines, Delta(208V) is mainly used in Japan
and North America.
3 M9000 base cabinet + expansion cabinet for 3-
Phase (star) DPF
M9000 Base cabinet + Power cabinet for 3-Phase(star)/ Dual
Power Phase, 4 lines, Star(415V) is mainly used in EU region.

120. 120
Service Area: M8000
800
(31.5)
800
(31.5)
1260
(49.6)
750
(29.5)
M8000
Front
M8000
Front
PowerCabinet
1054
(41.5)
Sun Rack
900
(35.4)
900
(35.4)
850
(33.5)
1260
(49.6)
Front
Units in mm
() in inches

121. 121
Service Area: M9000
M9000
900
(35.4)
900
(35.4)
850
(33.5)
1260
(49.6)
Front
M9000
Front
PowerCabinet
1154
(45.4) Units in mm
() in inches
M9000
Expansion
Cabinet
Front
PowerCabinet
PowerCabinet
2282
(90.0)
M9000
Front
Expansion
Cabinet
1674
(65.9)
Only the M9000-64 requires a raised floor with ventilation holes

122. 122
<Insert Picture Here>
I/O Expansion Unit

123. 123
External IO Expansion Unit
• Common unit between
mid-range and high-end servers
• Rack mountable (4U)
• All cabling rear accessible
• Redundant and hot-swappable
PSU’s and fans
• Serial link connection between host using link cards
• Copper cable (3m) and optical fibre cable (up to 25m)
• Two IO boats with hot-pluggable PCI slots
• 2 Boats with PCI-X slots
• 2 Boats with PCIe slots
• One boat with PCIe slots and one boat with PCI-X slots

124. 124
I/O Box: FRUs and PCI Data Path
32 Lane PCIe
Switch
PLX PEX8532
Switch
Bridge Bridge Bridge
NEC PCI-X to PCIe
Bridge
UPD720400
8 Lane
PCIe
Uplink
6 PCI-X
Slots
Link
0
Slot
1
Slot
2
Slot
4
Slot
5
Slot
6
Slot
3
Switch
Switch Switch
8 Lane
PCIe
Link
0
Slot
1
Slot
2
Slot
4
Slot
5
Slot
6
Slot
3
6 PCIe
Slots

125. 125
What do you mean “bottleneck”?
•8 lane(8x) PCIe is approximately 4 times faster than
what we support with 133Mhz 64 bit PCI-X cards (32 Gb
-vs- 8 Gb)
•Advantage to the PCIe architecture is that it is point to
point, so issues around bus length are greatly reduced
•Current cards are 2x and 4x
Adapter Expected (Gb/s) Theoretical (Gb/s)
Quad GbE 2.8 4
10 GbE JumboFrame 9 20
10 Gb iSCSI w/TOE 9 20
4/2/1 Gb FC Dual Port 1 1.6
8/4/2 Gb FC Dual Port 1.8 3.2
IB 4x Dual Port 16 20
IB 8x (DDR) 32 40
40 GbE 35 80

126. 126
I/O Boat

127. 127
PCI Cassette for Boat in I/O Box
In PCICS, PCIexpress Short dedicated cassette provided
Carrier main
body (metal)
Moveable carrier
plate (plastic)
Carrier slot
keyholes
Operating
handle
Card retainer
(5 total, 3 types)
Circular area for
rotating the retainer

128. 128
Inserting and Removing a Carrier from
the I/O Boat
a. Pulling the carrier handle
raises PCI card out of PCI slot
connector
b. Pushing the carrier handle
lowers PCI card into PCI slot
connector
c. Lever locks the handle in the
extended position. Push the lever to
the left to unlock the carrier handle.

129. 129
I/O Box: Link Card
• Add-in card form factor
• 8 lane PCIe path
• Supports cable with sideband serial link
• Does store and forward of I2C IO Box management
traffic over sideband serial link
• Bridge controller provides reliable transport protocol
over sideband serial link
• Supports an optional (non-standard) interrupt pin for
host SP notifications

130. 130
Host to I/O Expansion Unit Link
•Host to I/O Expansion Unit
connectivity is obtained
through either
– (1) low height copper link
card kit
– (2) full height fibre link card
kit
•Host can have management
control via side-band signals

131. 131
I/O Box: Interconnect Cable/Fibre
• Copper cable - follows PCIexpress standard with
sideband signals
• Maximum length 5 meters
• Optical fibre - follows PCIexpress standard with
sideband signals
• Maximum length 25 meters
• Removal of cable will result in domain panic

132. 132
Copper Link Kit
•Each half height PCIe based
copper link card has 1
interconnect (i.e. cable plug)
which carries data over a 5
meter distance
• 8 lane PCIe (data)
• Bidirectional
• serial (mgmt)

133. 133
Fibre Link Kit
•Each full height PCIe based
fibre link card has:
– 1 Transmit (TX) Port
– 1 Receive (RX) Port
•Transceivers are/use:
– 850nm wavelength
– 50/125 MM Fibre
– IEC 60825 & CDRH
Class 1M laser eye safe

134. 134
Front View - Power Supplies (A195)
•Two 600W 120-240V Hot Swap PS's
– Redundant power & cooling (can run w/ 1 PS)
•Power supply and chassis status LED's
PSU 1
600 Watts
120/240
VAC
Fault
Power
Overtemp
Locate
PSU 0
600 Watts
120/240
VAC
AC Power
Ready-to-remove
Fault /Locate
OK

135. 135
Boat Fault/Locate
Rear View – I/O Expansion Boats
•Two independent I/O “boats”
– Mix/match either PCIe or PCI-X boat
– Dual hosting is not supported (discussed later)
Chassis
Locator
Chassis Fault
Chassis Power
Chassis Overtemp
PCIe link To
Host
Ready to Remove
Boat Ok

136. 136
PCI cards for M8000 - M9000
•PCI Cards
– Base I/O card (IOUA) can be populated into #0, #2, #4, #6, depending on hardware
implementation.
– Link card can be populated into #1, #3, #5, #7, to avoid conflicting Base I/O card.
IOU PCIe
Slot #
Base I/O Card
Allowed?
Link Card
Allowed?
Available Resources
In IOU Via Link Card
#0 Yes Disk#0, #1,DVD/DAT,2xGbE
#1 Yes I/O Boat
#2 Yes DVD(*)/DAT(*), 2xGbE
#3 Yes I/O Boat
#4 Yes Disk#0, #1, DVD/DAT,
2xGbE
#5 Yes I/O Boat
#6 Yes DVD(*)/DAT(*),2xGbE
#7 Yes I/O Boat

137. 137
IOUA and available resources in IOU
• When populated into #0, #4: Disks, and DVD(*)/DAT(*) are available
• When populated into #2, #6: DVD(*)/DAT(*) are available
(*) One DVD and one DAT (option) are installed per base cabinet and expansion
cabinet(M9000). And, DVD or DAT drive are exclusively connected to one Base I/O card.

138. 138
IO Box Maximum Connections
Max I/O Box on M9000-64 restricted to same quantity as M9000-32 due to
limited testing.
I/O Box Configuration Rules
Mixing PCIe IO-Boat and PCI-X IO-Boat in an IO-Box is supported.
A Link option must be ordered per IO-Boat.
M4000 M5000 M8000 M9000-32 M9000-64
Max IOU 1 2 4 8 16
Max I/O Boxes 2 4 8 16
Max I/O Boats 4 8 16 32

139. 139
Maximum IO Expansion Units for
M4000/M5000
• Concept
– Up to 2x IO Boxes
per IOU for M4000 &
M5000
IO-Box
IOU
PCIe
Link CardPCIe
PCIe
PCIe
PCIe
PCI-X
Link cards can be populated
into any PCIe slot
IO-Box
The number of
PCI slots per
18 slots
IOU : 25 slots
IOU: 1 (PCI-X)
IOU-Box: 2x 12
(see note-1)
Notes-1: As for 12 slots per IO-Box, either
PCIe or PCI-X can be selected in 6 slots.
Link Card
Link Card
Link Card
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
Max I/O
Boxes
Max PCIe
Slots
Remarks
M4000 2 25 1 IOU x 25 slots
M5000 4 50 1 IOU x 25 slots

140. 140
IO Box Configuration for M4000/M5000
M4000: Maximum of 2x IO Boxes
(M4000: 6U, IO Box: 4U)
M4000
IO-Box
IO-Box
IOU
#0
x2Link
19” Rack
M5000
IO-Box
IO-Box
IO-Box
IO-Box
IOU
#1
IOU
#0
x2Link x2
19” Rack
M5000: Maximum of 4x IO Boxes
(M5000: 10U, IO Box: 4U)
Notes: The physical mounting location of IO-Box is not fixed.

141. 141
Number of IO Expansion Units for
M8000/M9000
• Concept
– Up to 2x IO
Boxes per IOU
IO-Box
IO
U
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
Link Card
PCIe
PCIe
IO-Box
PCIe
PCIe
PCIe
PCIe
PCIe
PCIe
#0
#1
#2
#3
#4
#5
#6
#7
Notes-1: As for 12 slots per IO-Box,
either PCIe or PCI-X can be selected
in 6 slots.
Link Card
Link Card
Link Card
Max I/O
Boxes
Max PCIe
Slots
Remarks
M8000 8 112 4 IOU x 28 slots
M9000-32
16
224 8 IOU x 28 slots
M9000-64 288 Limited to testing

142. 142
IO Box Configuration for M8000 & M9000
Notes: The physical mounting location of IO Box is not fixed.
19” Rack
IO-
Box
x
2
x2 x2 x2Link
Base Cabinet
Rear
View
M8000: Maximum of 8x IO Boxes
(IO Box: 4U)
IOU
#3 #
2
#1 #0
19” Rack Base Cabinet19” Rack
Rear
View
M9000: Maximum of 16x IO Boxes
(IO Box: 4U)
Link
IO
U
#6 #4
IO-
Box
IO-
Box
IO-
Box
IO-
Box
IO-
Box
IO-
Box
IO-
Box
IO-
Box
IO-
Box
IO-
Box
IO-
Box
IO-
Box
IO-
Box
IO-
Box
IO-
Box
IO-
Box
IO-
Box
IO-
Box
IO-
Box
IO-
Box
IO-
Box
IO-
Box
IO-
Box x
2
x2 x2 x2
IOU IOU IOU
IO
U
IO
U
IO
U
#2 #0
IO
U
#7 #5
IO
U
IO
U
IO
U
#3 #1

143. 143
IO Box Configuration for M9000
IO-Box
IO-Box
IO-Box
IO-Box
IO-Box
IO-Box
IO-Box
Expansion
Cabinet
19”
Rack
IO-Box
IO-Box
IO-Box
IO-Box
IO-Box
IO-Box
IO-Box
IO-Box
Rear View
x1 x1 x1 x1Link
Base Cabinet19”
Rack
IO-Box
IOU
#14
M9000: Maximum of 16x IO Boxes Configuration (IO-Box: 4U)
Notes: The physical mounting location of IO Box is not fixed.
IOU
#12
IOU
#10
IOU
#8
IOU
#15
IOU
#13
IOU
#11
IOU
#9
IOU
#6
IOU
#4
IOU
#2
IOU
#0
IOU
#7
IOU
#5
IOU
#3
IOU
#1
x1 x1 x1 x1Link

144. 144
I/O Box SCF Software
• IO Manager Software on SCF
• Largely platform-independent, could be used on OPL,
Supernova or otherplatforms.
• Includes all the functionality required for IO Box
management.
• Discovers IO Boxes and FRUs (part numbers, serial
numbers, etc) when PCIe slots are powered on.
• Can collect environmental, voltage, status information from
IO Box FRUs.
• Receives interrupts when the IO Box detects errors

145. 145
I/O Box Software
SCF Software
Hardware
IO Box
BUI
IO Box CLI
Commands
Error
Reporting
SunMC
Agent
Module Wrapper
libsmbus
IO Manager
(libcharon)

146. 146
I/O Box CLI
•The only IO Box CLI command on SCF is ioxadm
Usage: ioxadm [­h]
or: ioxadm [­fpv] list [TARGET]
or: ioxadm [­fpv] env [­elt] [TARGET]...
or: ioxadm [­fpv] reset TARGET
or: ioxadm [­fpv] resetpci TARGET
or: ioxadm [­fpv] poweroff TARGET
or: ioxadm [­fpv] poweron TARGET
or: ioxadm [­fpv] setled [­n|­f|­S|­F] TARGET LED
or: ioxadm [­fpv] locator [­n|­f] TARGET
•TARGET is either a host pci slot location or a box/fru
– PCI Slot: IOU#0-PCI#3
– FRU: iox@<serial id>[/fru]
• iox@3B5A – The IO Box with serial id ending in 3B5A
• iox@3B5A/ps1 – PSU #1 in IO Box
• iox@3B5A/iob0/link – Link card in IO Boat #0 in IO Box

147. 147
For More Information
search.oracle.com
or
oracle.com/us/products/servers-storage/servers/sparc-enterprise
Servers

148. 148