Slide 1: HiWire SWITCH AEC and SONiC Dual TOR Webinar
HiWire™ SWITCH AEC and SONiC® Dual TOR Webinar Presented by Credo with Microsoft Hosted by © 2021 Credo, Microsoft, SONiC, 650 Group. All Rights Reserved.
Slide 2: Today's Host
Today’s Host HiWire Switch AEC and SONiC Dual TOR Webinar Alan Weckel | 650 Group is a Technology Analyst/Co-Founder at 650 Group, where he is in charge of Networking and Cloud research. Alan’s expertise has been quoted in CIO Today, Wall Street Journal, and Fierce Telecom. He has presented at a wid
Slide 3: 650 Group Overview
650 Group Overview HiWire Switch AEC and SONiC Dual TOR Webinar What We Track Who Uses Our Research Cloud • IaaS, SaaS, Colo, Search and Social Buy Side / Sell Side • CAPEX, Equipment Trends Telecom Equipment Cloud Providers • Broadband Access, Telecom Core • NFV, Mobile RAN, SP Routing, Optical Tra
Slide 4: Ethernet Switch – Data Center: SONiC Projections
Ethernet Switch – Data Center: SONiC Projections HiWire Switch AEC and SONiC Dual TOR Webinar SONiC Capable Ports SONiC Revenue by Customer Type Percent of DC Switching 100% $ Revenue In Billions ($) 80% 1st Party Optics Enterprise Revenue OS 60% Hybrid Enterprise, Rest of Cloud, Telco* Software 40%
Slide 5: Ethernet Switch – Data Center: AEC Adoption at the ToR-Server
Ethernet Switch – Data Center HiWire Switch AEC and SONiC Dual TOR Webinar 50% AEC Adoption at the ToR-Server Percent of ToR Connections Active Electrical Cables Switch AEC 25% Shift AEC Secure AEC Multi-Rack ToR AEC 0% ©650 Group Confidential Information - Redistribution is strictly prohibited © 20
Slide 6: Today's Speakers
Today’s Speakers HiWire Switch AEC and SONiC Dual TOR Webinar Dr. Lihua Yuan | Microsoft Don Barnetson | Credo is currently a Partner Software Engineering is the VP of Product for HiWire AECs at Credo with a Manager in Microsoft Azure Networking. He leads focus on product definition and customer the
Slide 7: Dual ToR Technology and Challenges
HiWire Switch AEC and SONiC Dual TOR Webinar Dual ToR Technology and Challenges © 2021 Credo, Microsoft, SONiC, 650 Group. All Rights Reserved. 7
Slide 8: Data Center Network with CLOS
Data Center Network with CLOS HiWire Switch AEC and SONiC Dual TOR Webinar Tier 3 – Regional T3-1 T3-2 T3-3 T3-4 … … … Tier 2 – Data center T2-1-1 T2-1-2 T2-1-8 T2-4-1 T2-4-2 T2-4-4 … … T1-7 … T1-7 Tier 1 – Row Leaf T1-1 T1-2 T1-7 T1-8 T1-1 T1-2 T1-8 T1-1 T1-2 T1-8 … … … T0-1 T0-2 T0-20 T0-1 T0-2 T0
Slide 9: Top of Rack Switch: Single Point of Failure for a Rack
Top of Rack Switch : Single Point of Failure for a Rack HiWire Switch AEC and SONiC Dual TOR Webinar ToR is a Single-Point-of-Failure (SPOF) for full rack of servers And TORs do fail ~2% of switches fail in first 3 months 32% due to hardware failures 27% due to power failures © 2021 Credo,
Slide 10: Classic Solution: Multi-Chassis LAG with Dual Uplink
Classic Solution: Multi-Chassis LAG with Dual Uplink HiWire Switch AEC and SONiC Dual TOR Webinar ToR 1 Applications NIC1 Host Inter-Switch Link Teaming Applications NIC2 ToR 2 Applications Network Server © 2021 Credo, Microsoft, SONiC, 650 Group. All Rights Reserved. 10
Slide 11: Network: Inter-Switch Link
Network: Inter-Switch Link HiWire Switch AEC and SONiC Dual TOR Webinar ToR 1 Applications NIC1 Host Inter-Switch Link Teaming Applications NIC2 ToR 2 Applications Network Server Inter-Switch Link requires custom design for capacity planning Requires complex state sync between ToRs Creates spl
Slide 12: HyperVisor: Performance Limit
HyperVisor: Performance Limit HiWire Switch AEC and SONiC Dual TOR Webinar ToR 1 vNIC Inter-Switch Link NIC1 Host vSwitch VM Teaming NIC2 ToR 2 Network NIC Hypervisor Guest OS A few Gbps/core Unstable latency Won’t work for RDMA © 2021 Credo, Microsoft, SONiC, 650 Group. All Rights Reserved. 1
Slide 13: SR-IOV: VM Complexity
SR-IOV: VM Complexity HiWire Switch AEC and SONiC Dual TOR Webinar ToR 1 SR-IOV vNIC NIC1 Inter-Switch Link VM NIC2 vNIC ToR 2 SR-IOV Network NIC Guest OS VM sees failures and must handle Hardware-dependent vNIC driver © 2021 Credo, Microsoft, SONiC, 650 Group. All Rights Reserved. 13
Slide 14: vSwitch Offloading: Vendor Specific
vSwitch Offloading: Vendor Specific HiWire Switch AEC and SONiC Dual TOR Webinar ToR 1 vNIC Inter-Switch Link NIC1 Host vSwitch VM Teaming NIC2 ToR 2 Network NIC Hypervisor Guest OS Variety of solutions to offload Teaming and vSwitch to Smart NICs Capabilities and Implementations vary by NIC Ven
Slide 15: The Solution Space - Options to Manage Redundancy
The Solution Space - Options to Manage Redundancy HiWire Switch AEC and SONiC Dual TOR Webinar Host ToR 1 vSwitch vNIC Teaming VM ?? NIC1 Inter- SR-IOV Switch NIC2 Link vNIC SR-IOV VM vNIC ToR 2 Network NIC Host OS Guest OS © 2021 Credo, Microsoft, SONiC, 650 Group. All Rights Reserved. 15
Slide 16: The Solution Space – Introducing the Switch AEC
The Solution Space – Introducing the Switch AEC HiWire Switch AEC and SONiC Dual TOR Webinar Host ToR 1 vSwitch vNIC Teaming VM NIC1 SR-IOV vNIC VM ToR 2 Network NIC Host OS Guest OS © 2021 Credo, Microsoft, SONiC, 650 Group. All Rights Reserved. 16
Slide 17: A New Approach: HiWire SWITCH AEC + SONiC Dual TOR Management Container
HiWire Switch AEC and SONiC Dual TOR Webinar A New Approach: HiWire™ SWITCH AEC + SONiC Dual TOR Management Container © 2021 Credo, Microsoft, SONiC, 650 Group. All Rights Reserved. 17
Slide 18: What are HiWire Active Electrical Cables (AECs)
What are HiWire™ Active Electrical Cables (AECs) HiWire Switch AEC and SONiC Dual TOR Webinar AECs are industry-backed solutions to solve the connectivity bottleneck • Integrate active components : retimer, gearbox, L2 Switch into Copper Cables to facilitate secure, high-integrity plug and play conn
Slide 19: New Solution in Collaboration with Microsoft / Azure
New Solution in Collaboration with Microsoft / Azure HiWire Switch AEC and SONiC Dual TOR Webinar QSFP QSFP ToR A L2 PCS NIC PCS SWITCH Dual TOR Management PCS QSFP ToR B Container HiWire SWITCH AEC: 50G/100G/200G QSFP AEC Cable Dual TOR Management Container on SONiC • Active/Standby Layer 2 switch
Slide 20: HiWire SWITCH AEC
HiWire SWITCH AEC HiWire Switch AEC and SONiC Dual TOR Webinar Performance • 50G – 200G NRZ or PAM4 Switching • Automatic switching on loss-of-signal and Manually switching on I2C command Updates • Hitless Firmware Updates Functionality • Gearbox, FEC translation AEC Classification • Networking devi
Slide 21: The Solution Space – Simplicity & Performance
The Solution Space – Simplicity & Performance HiWire Switch AEC and SONiC Dual TOR Webinar ToR 1 Host vSwitch vNIC Teaming VM NIC1 SR-IOV vNIC VM ToR 2 HiWire Switch AEC • Simpler than Link Aggregation Interconnect • Faster than Host Teaming • Decouples Server & Network Link Management © 2021 Credo,
Slide 22: How do we deploy?
HiWire Switch AEC and SONiC Dual TOR Webinar How do we deploy? © 2021 Credo, Microsoft, SONiC, 650 Group. All Rights Reserved. 22
Slide 23: Dual ToR with Switch AEC
Dual ToR with Switch AEC HiWire Switch AEC and SONiC Dual TOR Webinar T0 UT0 LT0 SPOF …… …… Single TOR : Failure Domain: Rack (20 – 40 Servers) Dual TOR + Switch AEC : Failure Domain: Single Server © 2021 Credo, Microsoft, SONiC, 650 Group. All Rights Reserved. 23
Slide 24: Switch AEC Data Plane Behavior
Switch AEC Data Plane Behavior HiWire Switch AEC and SONiC Dual TOR Webinar Active UT0 MUX TX UT0 NIC TX is always broadcasted NIC RX TX NIC RX will take active side NIC and only Switch with Loss-of- LT0 signal or on-command over time < 1ms RX X TX LT0 RX Standby © 2021 Credo, Microsoft, SONiC, 650
Slide 25: Independent Active/Standby Design
Independent Active/Standby Design HiWire Switch AEC and SONiC Dual TOR Webinar Discovery via ICMP PING Active/Standby detection Ping with SrcMac = A Forward if DstMac = A ToR A ToR A SWITCH AEC SWITCH AEC NIC NICTX NIC NICTX T TX NIC NIC X ASIC NIC NIC ASIC T TX T RX NIC TX TX TX NICRX X X R RX X To
Slide 26: Routing Behavior – Both ToRs announce via BGP
Routing Behavior – Both ToRs announce via BGP HiWire Switch AEC and SONiC Dual TOR Webinar announce 10.0.0.0/26 Both ToRs see Server (10.0.0.2/26) Active Link == BGP announce 10.0.0.0/26 Leaf #1 and announce to Leaf Nodes 10.0.0.2/26 Server 1 Active ToR All Leaf nodes see ToR announcements and annou
Slide 27: Southbound Packet Path
Southbound Packet Path HiWire Switch AEC and SONiC Dual TOR Webinar Decap and • Southbound traffic arriving on active ToR ToR A forward A is forwarded to NIC SWITCH AEC NIC NICTX TX NIC TX TX NIC TX NICRX ASIC • Southbound traffic arriving on Standby NIC RX NIC RX ToR B ToR B RX RX X NICTX ASIC Enca
Slide 28: Failure Scenarios 1: Planned Maintenance
Failure Scenarios 1: Planned Maintenance HiWire Switch AEC and SONiC Dual TOR Webinar Scenarios: SWITCH NIC AEC Leaf #1 ToR OS upgrade, hardware replacement NIC 1. ToR B proactively issues command to switch TX ToR A …… NIC RX ToR B MUX to ToR B 2. ToR B ICMP Ping is now forwarded; ToR A is Leaf #n T
Slide 29: Failure Scenarios 2: Hardware Failure for ToR and Link
Failure Scenarios 2: Hardware Failure for ToR and Link HiWire Switch AEC and SONiC Dual TOR Webinar Deployment Scenario: SWITCH Leaf #1 Cable cut, ToR port failure, ToR power failure, ToR NIC AEC 1. Cable detect Loss-of-sigal to ToR A NIC TX ToR A …… NIC 2. Cable auto switches on link failure in les
Slide 30: Failure Scenarios 3: ToR Forwarding Failure
Failure Scenarios 3: ToR Forwarding Failure HiWire Switch AEC and SONiC Dual TOR Webinar SWITCH Leaf #1 NIC AEC Deployment Scenario: NIC TX ToR grey failure impacts forwarding, but link ToR A …… NIC remained up. Actually ~26 scenarios here – RX ToR B review SONiC Dual TOR Container docs Leaf #n TOR
Slide 31: Demo Video
HiWire Switch AEC and SONiC Dual TOR Webinar Demo Video Terminal screen recording – end to end setup – no packet lost in middle © 2021 Credo, Microsoft, SONiC, 650 Group. All Rights Reserved. 31
Slide 32: SONiC Support for HiWire SWITCH AEC
SONiC Support for HiWire SWITCH AEC HiWire Switch AEC and SONiC Dual TOR Webinar SWITCH AEC fully upstreamed and abstracted in SONiC Support for any SONiC hardware Hitless firmware updates from any cable end Advanced Cable telemetry, loopback, BER and debug 50G and 100G HiWireTM SWITCH AEC
Slide 33: Time for Q&A
HiWire Switch AEC and SONiC Dual TOR Webinar Time for Q&A Active Electrical Cables © 2021 Credo, Microsoft, SONiC, 650 Group. All Rights Reserved. 33
Slide 34: Thank You
HiWire Switch AEC and SONiC Dual TOR Webinar Thank You Active Electrical Cables © 2021 Credo, Microsoft, SONiC, 650 Group. All Rights Reserved. 34
Slide 35: Active Electrical Cables
HiWire Switch AEC and SONiC Dual TOR Webinar All presentations, content, videos, and presentation recordings are © 2021 Credo, Microsoft, SONiC and 650 Group. Credo, HiWire, and the Credo and HiWire logos are trademarks of Credo. Microsoft, Microsoft Azure, and SONiC and the Microsoft, Microsoft Azu