Home

57712-k Software User Manual

1. ES Adapter4 57712 A1 EF 9 Porti D ST Te e 0007 Broadcom BCIM57712 Port Ia Broadcom BCM57712 MetXtreme II 10 GigE 345 E 0012 Broadcom BCM57712 NetXtreme IT 10 G NDIS VED Client 46 An iSCSI Discovery and Management Wizard is also included with BACS4 This can be used to connect and manage the iSCSI hardware offload enabled initiator devices to various iSCSI targets as shown below Broadcom Advanced Control Suite 4 El 899 Hosts S B wi RA3QIRKO731 Microsoft Windows Network Connections These devices can be used by any application as if they were a separate adapter port They appear as separate Ethernet devices in Network Connections from the Windows Control Panel The following shows eight USC BACS4 enabled Ethernet Protocol partitions as eight separate Ethernet network connections and these are arranged in port order O and 1 and partition order 1 through 4 B Nehwork Connections ke W p en MIC Partitions SPFP NE 57712 HP 57712 MP 57T712 NP Port ort 0 Pod 0 Port Partition 1 artili Parlition 3 Fartition4 Ts g ger BTTTAZ HP 57T12 MP 57712 MP 57712 MP Por 1 Fort 1 Port 1 Port 1 Faribon 1 ParBtionZ Pattini Partition A MIC Partitions BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 43 57712 k 578xxS Softw
2. Se wm Acn nr Cortot Tec Ters SS ee En Dees prm semen sees E BE Hewes S H wmemmarx E EM Adapter 1 CMS AN C0 X EE Adapter BCMS709 CO T EEM Adapters CHEN CO 3B MEM Adapters BCMS709 CH X EN Adapters BCM57711 AD Adaptere BCM57712A1 EL Port m D I 0084 Sroadcom BCMS7712 NetXtreme ET 30 Gage Multi Function 254 QN 0027 Broadcom 5057712 NetOreme D 10 GE NDIS VEO Chen 254 e W 0089 Grosdcom BCMS7712 NetXtreme IE 30 Cf Multi Function 255 a v 0013 Breacene 802453712 Nettireme D 10 Gof Mult Functon GCSE Adapter 299 ei ag ign 2011 68 com broadonm Sgt E n re Un m AER ig 2011 49 com broadonm gt Ls res 148 Portal feliz 2 20 effec MI 0030 Broadenm BOM STL Metre D 10 G GES VED Clenz 239 E T Oto Brosdcom BOMSTTI2 Heiene IL 20 GagE Multi Function ad ai 00 X3 Broadcom BCMST7 LE e pes D 10 G GT VED Cent a6 E Ti 006 1 Brosdcom BCMST712 Netxtreme IT 30 Get Multi Puncher ani uu 0014 Broadcom BOM 7712 ppes D 10 Get Mule Function gi Adapter ini 7 Hz Broadcom BCHSTT LI ener D 10 Ge GET VED Chere s i Putz CE TI AE EE uu de es Gomes E P etm etr S insta Sage D Hosts B J WIN RJKLISAEHT IC EN Adanteri BOws205 00 EN Adanter2 Boss C Em Adapters BONA C e EZ Adantert 50957711 Am j I Adapter 28457712 AT e Portl El T 22 54 Broadcom BMS 12 best tege IE 10 xg Mul Funcson 54 rT 0027 Brosdcom BOMSTT LZ Nettneme HL 20 Gage MOS VED Chet ait se m 2255 Broadc
3. 20 000 dece kae N The following shows the per partition per FCoE target information p TIPP TII ad ontrol Suite 4 ires EE lees DIE cot Target tnmen piore View In rmason Eis Property Nie E H wmRnAXuREDTS Target Informabon EE Adapteri BCMS708 CH FCID KE GS Adapter BCMS705 CO WW Port Name ES Adapter3 BCMST711 AD WW Node Name EE Adapters 57712 A1 L5 E Porti m 8 0080 Broadenm BCMS7712 Met pene II 10 GE Mui Function 250 Is Broadcom BCMS7712 NetXtreme II 10 GigE Multi Function 51 Er E 0053 Broadcom BCM57712 Netitreme IT 10 GigE Mult Function 253 GE 0030 Braadcom BOMS7712 Netktreme I 10 GigE NDIS VEO Ghent 53 E m 0054 Broadenm BCMS7712 NetXtreme IT 10 GE Mult Function 254 E ET 0008 Broadcom BCM57712 NetXtreme H 10 GigE Mult Function FCoE Adapter Ex B Part 20 00 00 10 18 6F DEED BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 38 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning The following shows the per port DCB protocol information Fen nah n Bye Coe eee d pe ges Acton T Context Teds Troms SCH Mep 1 mp ze less en F ence E pn H GE nenas riesis Contgeetere z araa 9 rimis U wmneseeewee D E ace BOE OU DE Asoo BOO CO VER Aner BOT AD B SR aces Cat A amp UP poti Broadese 8O 577 2 metteg E 12 Gef 253 PILLS feret 5 e
4. ka 0016 Broadcom BCOM57712 MetXtreme IT 10 Got Multi F ELE D IT C B Port20 00 00 10 18 6F D 3 8D af 0031 Broadcom BOMS7712 Netttreme II 10 Ga NDIS VED Chen 54 D Pert2 BROADCOM June 07 2012 e 2CS57712 SWUM103 R 1714693711 174551519204 gee HEETE 5 Om i SOM OOO oOo Orr alejes Switch Independent Partioning Setup Guide Page 37 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning The following shows the per partition FCoE device connection information Broadcom Advanced Control Suite 4 ALL VIEW v nfermason Tel FP Fateic Informaten Y Fabri Irformabon Port Type Nort WW Port Name u 20 00 00 10 1 6 bdY bd WU Node Nare 10 00 00 10 8 6 3b Adapters 57712 A1 Petro GE Fasz Draut n H Berti rkKeen Ak we rier al SoOQU m 0050 Broadcom BCM577 12 NetXmreme H 10 GigE Mult Functon 50 e T 0051 Broadcom BCMS7712 Netxtreme I 10 GagE Mul Function 251 L Feh ieMame xk bkOkerekmeDl zi D I 0053 Broadcom BCM57712 Netitreme H 10 GigE Mult Function 53 Fabric WLAN En RN 0030 Broadcom 8CM57712 Netxtreme H 10 Gg NDES VEO Clent 253 O DRM ee Been SPIT 0054 Broadcom BCM57712 NetXtreme H 10 Gg Multi Functon 54 FCF MacAddress ODIDECEIAEO0 E EE 0016 Broadcom BOMS7722 Netxtreme D 10 GE Mult Function FCoE Adapter __ ae eee see ee FCF WW Port Hame
5. Port Group Policy eG Compliant IP Address 1 1 50 1 Subnet Mask 255 255 0 0 Physical Network Adapter Name vmnici1 Device Broadcom Corporation NetXtreme II 57712 10Gigabit Ethernet Link Status Connected Configured Speed 10000 Mbps Full Duplex 4 Goto the Dynamic Discovery or Static Discovery tab and enter the desired iSCSI target LUN information General Dynamic Discovery Static Discovery Send Targets Discover iSCSI targets dynamically from the following locations IPv4 host name iSCSI Server Location BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 71 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning After completing these Discovery steps you are typically asked to refresh rescan the host bus adapter You can now add the discovered iSCSI LUNs as Hard Disks to the desired VMs localhost localdomain VMware ESXi 5 0 0 Getting Started Summary Virtual Machines Resource Allocation Performance Configuration mele BET ez T Groups Events Permissions Add Remove Refresh Rescan All Health Status Processors Memory Storage Networking ymhba32 iSCSI ign 1998 01 com emware localhast 610019708 34 bnx2i 0010183e1340 wvmhba33 ISCH ign 1998 01 com vmware localhast 610019708 35 bnx2i 0010183e1342 wmhba34 i5C5 ign 1898 01 com vmware localhast 6510019708 36 bnx2i 0010183e1344 vmhba35 GC iqn 1998 01 com vmware localhast 610019
6. Table 5 DCB Equal Oversubscription Relative Bandwidth Maximum Port Partition Weight Bandwidth Protocol Plot Color Port O Partition 1 P1i N A 100 iSCSI Offload Green Port O Partition 1 P1e N A 100 Ethernet Brown Port O Partition 2 P2e N A 100 Ethernet with Purple TOE Port O Partition 3 P3e N A 100 Ethernet Yellow Port O Partition 4 P4e N A 100 Ethernet Blue Port O Partition 4 P4i N A 100 iSCSI Offload Blue The following plot shows how the two iSCSI traffic streams in PG2 act verses the L2 Ethernet traffic streams in PGO The traffic in the two PGs will almost act independently of each other when their aggregated traffic bandwidth demands exceed the available bandwidth each taking it s half of the ETS managed pie e Starting at tO only P1i iSCSI Offload is sending so it takes 100 or all of the 10 Gbps bandwidth e When P1e L2 Ethernet starts to send at t1 both flows stabilize to 5 Gbps each P1i in PG2 takes it s allocated 5096 bandwidth and P1e in PGO takes it s allocated bandwidth of 50 e When P2e starts to send at t2 the traffic in P1i is not affected it remains at 5 Gbps due to it being in a different Priority Group Both P1e and P2e will stabilize to 2 5 Gbps each P1e and P2e equally share PGO s allocated portion of the bandwidth e When P3e starts to send at t3 P1i is still unaffected and remains at 5 Gbps The three L2 Ethernet traffic types will split their 5096
7. localhost VMware ESXi 4 1 0 Getting Started Summary Virtual Machines ResourceAllocation Performance Configuration WT BEET CT ee TT oM I AME ST TESTE iade Storage Adapters Health Status Dewice WWN Processors Broadcom iSCSI Adapter vmhba32 ign 1898 01 caom vmware localhost 974021991 34 bnxZi Q00101888e 28 vmhba33 ign 1998 01 com vmware localhost 302103589 33 bnx2i 00101888e7aa vmhba34 i ign 1988 01 com vmware lacalhost 9740219981 37 bmdai 00101888e7b4 vmhba35 ign 1998 01 com vmware localhost 902103589 35 bnxzi 00101888e b6 Memory Storage Networking Storage Adapters To associate or connect these iSCSI offload vmhba s to an existing vSwitch vmk vmnic use the following ESX commands on the Host to list the current associated iSCSI initiators vmkiscsid x select from ifaces more This lists all currently active and previously assigned inactive associations If there are a lot of associations use the more to make the output easier to read Since this is before associating any adapters there is nothing listed in this example a 7 root ESX41iSCSI unkiscsid x select Send ing SUL Request select from If aces conpleted uith status H lu Use the following command to list which vmnic can be associated with a specific vmhba which in this example are the numbers 32 through 35 esxcli swiscsi vmnic list d vmhba32 BROADCOM Switch Independent Partioning Setup Guide June
8. 1 0 Active 1 0 Active 1 0 Active 1 0 Active 1 0 Active 1 0 Active 1 0 Active 1 0 Active 1 0 Active 1 0 Active 1 0 Active 1 0 Note VMWare ESX ESXi 5 0 supports eight 10 GbE ports see http www vmware com pdf vsphere5 r50 vsphere 50 configuration maximums pdf Using Switch Independent Partitioning mode allows you to expand the number of ports usable from eight physical ports to 32 virtual Switch Independent Partitioning ports simultaneously with up to two iSCSI Offload adapters per physical port for up to 16 virtual Switch Independent Partitioning iSCSI devices or one iSCSI Offload and one FCoE Offload adapter per physical port for up to 8 virtual Switch Independent Partitioning iSCSI and eight virtual Switch Independent Partitioning FCoE devices This allows better port flexibility traffic isolation service quality and bandwidth tuning for your management backup migration production networks Details on Configuring iSCSI Offload on ESXi 5 0 After creating the desired networking vSwitches you still need to associate the Broadcom 5709 5771x 578xx iSCSI hardware offload HBA vmhba devices to a previously created vSwitch vmk vmnic 1 For the desired vmhba in the Configuration Storage Adapters window click Properties in the Details window below it 2 Select the Networking Configuration tab and click the Add button BROADCOM June 07 2012 e 2CS57712 SWUM103 R Switch Independent Parti
9. Allocation Menu Flow Control Partition 1 Partition 2 Partition 3 Partition 4 T Configure Global Bandwidth Allocation parameters ys and Enter to select Esc to exit page Tab to change focus X DOLL SYSTEM SETUP Help About Exit Integrated NIC 1 Port 1 Broadcom NetXtreme Il 10 Gb Ethernet BCM5 800 78 2B CB 5B 99 28 8 Broadcom Main Configuration Page NIC Partition Configuration Broadcom Main Configuration Page gt NIC Partition Configuration Broadcom NetXtreme Il 10 Gb Ethernet BCM57800 78 2B CB 5B 99 28 Global Bandwidth Allocation Menu Flow Control Tx Rx Flow Control Partition 1 Partition 2 9 Configure Global Bandwidth Allocation parameters PowerEdge R20 Arrow keys and Enter to select Service Tag TYHNJ36 Esc to exit page Tab to change focus If the Global Bandwidth Allocation Menu option was selected in Step 7 then the four on the 57712 k 57810 s or two on the 57800 s Relative Bandwidth Weights and Maximum Bandwidth setting window is presented BROADCOM June 07 2012 e 2CS57712 SWUM103 R Switch Independent Partioning Setup Guide Page 20 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning DOLL SYSTEM SETUP Help About Exit Integrated NIC 1 Port 1 Broadcom NetXtreme Il 10 Gb Ethernet BCM57810 84 2B 2B E8 22 E3 Main Configuration Page Global Bandwidth Allocation Menu Main Configuration Page gt NIC Partitio
10. BOU Hosts SG H wman DS EE Adapter 1 BCMS709 CO GSS Adapter BCMS709 CO SS Adapters BCM57711 A0 BE Adapters 7712 AT E 4 Porti B 0050 Broadcom BCM57712 MetXtreme IL 10 GigE Multi Furcton 250 ES 0012 Broadcom BCMS7712 MetXtreme TT 10 G Multi Function GOSI A Sch Porta 1 1 1 4 amp Portal fe30 210 18febidacs Broadcom BCMS7712 NetXtreme D 10 Gage NDES VED Cent 50 0051 Broadcom BCMS57712 NetXtreme IL 10 GigE Multi Function 51 0053 Broadcom BCM57712 NetXtreme H 10 GigE Multi Function 253 C 0030 Braadeeem BCM 57712 NetXtreme I 10 Gage NDIS VED Clent 253 0054 Broadcom BCMS57712 Metxtreme II 10 GigE Multi Function 254 ET 0015 Broadcom BCMS7712 NetXtreme I 10 GigE Multi Function FCoE A C73 D031 Broadcom BCM57712 NetXtreme II 10 GigE NDIS VED Chert a54 In Windows each individual partition s Ethernet and iSCSI protocol enabled adapter MTU size setting can be different For example e Port O partition 1 Ethernet can be set to 9600 bytes e Port O partition 1 iSCSI Offload HBA can be set to 2500 bytes e Port O partition 2 Ethernet can be set to 1500 bytes e Port O partition 2 iSCSI Offload HBA can be set to 9600 bytes e Port O partition 3 Ethernet can be set to 3500 bytes e Port O partition 4 Ethernet can be set to 5500 bytes BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 84 57712 k 578xxS Software User Manual Configuring Switch Independ
11. CH wmhba37 iSCSI ign 1998 01 com vmware localhost 610019708 39 bnx2i 00 101868ebde CH wmhba38 iSCSI iqn 1998 01 com vmware localhost 610019708 40 bnx2i 00 101888e5ec CH wmhba38 i5C5I ign 1998 01 com vmware localhost 610018708 41 bnx2i 00101888eeee Broadcom Corporation FCoE Adapter 3 vmhba42 Fibre Channel over Ethernet 10 00 00 10 18 88 ee ed 20 00 00 10 18 88 ee ed Network Adapters G vmhba43 FibreChannel ower Ethernet 10 00 00 10 18 88 ee ef 20 00 00 10 18 88 ee ef Advanced Settings Memory Storage Networking Power Management Details Model Broadcom Corporation FCoE Adapter Licensed Features WWN 10 00 00 10 18 88 ee ef 20 00 00 10 18 88 ee ef Time Configuration Targets 8 Devices 8 Paths 8 DNS and Routing Authentication Services Virtual Machine Startup Shutdown Virtual Machine Swapfile Location Security Profile Host Cache Configuration Software View Devices Paths Runtime Name Target LUN Status vmhba43 C0 T3 LO 20 00 00 00 c 9 92 c7 d8 10 00 00 00 c9 92 c7 d8 Active 1 0 wmhba43 CO T2 L0 20 00 00 00 c09 92 87 c3 10 00 00 00 c9 92 87 c3 Active 1 0 vmhba43 CO T7 LO 20 00 00 00 c9 92 87 c1 10 00 00 00 c9 92 87 c1 Active 1 0 wmhba43 CO Ti LO 20 00 00 00 c09 92 87 c2 10 00 00 00 c9 92 87 c2 Active 1 0 Dem b gcn B Alocagog 5 Add the discovered iSCSI and or FCoE LUNs as Hard Disks to the desired VMs BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 S
12. Gbps when the test application starts to transmit traffic at tO to that port by itself not expanding into the remaining unused 2 Gbps bandwidth e When the second traffic flow on the first partition P1e starts to send at t1 the two active traffic flows on the same partition share its 8 Gbps bandwidth for 4 Gbps each e When the third traffic flow P2e starts sending at t2 it only takes its partitions maximum bandwidth allowed 1 Gbps Partition P1 s two traffic flows are unaffected e When the fourth traffic flow P3e starts sending at t3 it again only takes its partitions maximum bandwidth allowed 1 Gbps Partition P1 s two traffic flows and the traffic flow on partition P2 P2e are unaffected e But when P4e starts to send traffic at t4 the condition is now oversubscribed Since P2e and P3e uses only 1 Gbps of their allocated 2 Gbps 10 Gbps 5 equally weighted traffic flows that leaves 8 Gbps free for the other three traffic flows The remaining traffic flows P1i Ple and P4e would then be allocated 2 6 Gbps each 8 Gbps 3 equally weighted traffic flows e But when P4i starts to send traffic at t5 it shares the available bandwidth within it s maximums with the other traffic flows P2e and P3e are still using only 1 Gbps of their allocated 1 6 Gbps 10 Gbps 6 equally weighted traffic flows which again leaves 8 Gbps free for the other four traffic flows Therefore these four P1i P1e P4e and P4i are allocated 2 Gbp
13. GigE 46 Broadcom BCMS709C NetXtreme II GigE 247 Be Broadcom BOM 09C Netxtreme II maus ES a Broadcom BCMS7712 NetXtreme I 10 GGE Multi Function 51 E Broadcom BCM57712 NetXtreme II 10 GigE Multi Function 52 SE Broadcom BCM57712 Netxtreme If 10 GigE Multi Function 53 78 Broadcom BCM57712 NetXtreme II 10 GigE Multi Function 54 Es Broadcom BCM57712 Netxtreme TI 10 GigE Multi Function 55 SE Broadcom BCM57712 NetXtreme IL 10 GigE Multi Function 56 4 8 Broadcom BCM57712 NetXtreme II 10 GigE Multi Function 57 BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 31 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning Right click the specific device and select its Properties to access some of the advanced features of the device E Intel R 5520 5500 X58 1 0 Hub PCI Express Root Port 7 340 Ba Broadcom 804577 LP Netxtreme IE 10 Gof Mult Functan 241 amp Broadcom BCMS7712 NP Heiene I 10 GE NDIS VEO Ceni 241 Ba qe a A Broadcom BCMS77128P Netilreme 20 GE Md Funcim 256 ca Broadcom BCMS57712HP Netktreme IT 10 Got Multi Function SS S Broadcom BCMS7712NP Nettreme IT 10 Got Multi Function i58 a Broadcom BCM57712 NP MetXreme D 10 Got NDTS VBD Ghent 355 I Broadcom BCMS7712NP NetXtreme IT 10 Got Multi Function 253 a Droackoen BOMS77 LIF NetXtreme D 10 G NDIS VED Clent 259 Properties lii Broadcom BCMS77 LAP NetXtreme If 10 GigE Multi Func
14. Manual Configuring Switch Independent Partitioning The following shows the per port DCB protocol statistics Broadcom Advanced Control Suite 4 Pes au vew Elbsess lp oce statisties E custom information Configurations Statistics Diagnostics ERIT moe Taie B g WIN RA3QRKD731 BEL D ES Adapter 1 BCM5709 CO DCEX Frames Sent H GES Adapter BCM5708 CO DCEX Frames Received E REN Adapters BCMS7711 AD EE Adapters 57712 A1 3 Porti E MF 0050 Broadcom BCM57712 NetNtreme I 10 GigE Multi Function 250 we 0012 Broadcom BCM57712 NetXtreme II 10 GigE Multi Function iSCSI Adapter T 0027 Broadcom BCMS7712NetXtreme H 10 GigE NDIS VED Clent 50 MAC Ri wing Pause Command lO 7 posi Broadcom BCMS7712 NetXtreme II GE Multi Function 51 MAC Sent Xon S PR 0053 Broadcom BCMS7712 NetXtreme IT 10 Gig Multi Function 253 MAC Sent Xoff Af 0030 Broadcom BCM57712 Netittreme H 20 GigE NDIS VEO Cieni 253 Pause fames recereed E PF 0054 Broadcom BCMS7712 NetXtreme II 10 GigE Multi Function 54 Pause frames sent Ge 0016 Broadcom BCM57712 Nettreme D 10 GigE Multi Function FCoE Adapter Af D031 Broadcom BCM57712 Netitreme H 10 GigE NDIS VED Cieni 254 Lroadeom Advanced Control Suite 4 Fie vew Acton Filter Context oos ren PCS Help E Adapter BCM3709 CO Hi NEM Adacter BCMS 05 CO IS SE Adapter 3 BCME7711 A0 E SE Adaptera 57712 A1 EF Port SPR 0050 Broadcom BCM57712 NetXtr
15. Offload essssssssssesessreeresresereerrrerreresereerreeens 100 Table 7 Non DCB Partitioned Overs bscriptiON EE 102 Table 8 Non DCB Weighted Oversubscription cccccsssccccsececsececescceenceesescecesececeeceeeneeeeeeeseneeseneeeseeseneees 105 Table 9 Non DCB Oversubscription With One High Priority Barttion esee 107 Table 10 Non DCB Default Fixed Subscription ccccceecceeeceeeccenceceeccenceeeceeeceeeceeseeeueeensetsuseeeeseeeeeeeeeneetes 109 Table 11 Non DCB Mixed Fixed Subscription and Oversubscription 111 Table 12 DCB Mixed Fixed Subscription and Oversubscription with Lossless FCoE Offload 113 Table 13 Non DCB Mixed Fixed Subscription and Oversubscription 115 BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 5 57712 k 578xxS Software User Manual About This Document About This Document Purpose This document provides instructions on how to enable Switch Independent Partitioning also referred to as NPAR in other documentation on Dell PowerEdge Servers installed with the Broadcom 57712 k Blade Converged Network Daughter Card NDC Dual Port 10 GbE A Fabric option see Figure 1 and the 578xxS device family see Figure 2 57800S Rack Converged NDC on page 7 Figure 3 57810S k Blade Converged NDC on page 8 Figure 4 57810S k Blade Converged Mezzanine on page 9 Figure 5 57810S Converged NIC on page 10 Figu
16. Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 23 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning See Examples on page 94 for more details on both the Relative Bandwidth Weight and Maximum Bandwidth settings affect traffic flow in DCB and non DCB modes of operation 9 If one of the Partition options was selected in the NIC Partition Configuration window in Step 5 you are able to change the selected partition s protocol settings here R DCLL SYSTEM SETUP Help About Exit Integrated NIC 1 Port 1t Broadcom NetXtreme Il 10 Gb Ethernet BCM57810 84 2B 2B E8 22 E3 Main Configuration Page NIC Partition Configuration Partition 1 Main Configuration Page gt NIC Partition Configuration gt Partition 1 Ethernet Protocol Disabled amp Enabled iSCSI Offload Protocol amp Disabled Enabled FCoE Offload Protocol Disabled Enabled Network MAC Address 18 03 3 FD DB 04 Virtual Network MAC Address 842B2B ES22E3 ISCSI MAC Address 18 03 73 FD DB 05 Virtual iSCSI MAC Address 842B2B ES22E4 FIP MAC Address 18 03 723 FD DB 05 Virtual FIP MAC Address 84 2B 2B E8 22 E5 World Yide Node Name 20 00718 03 3 FD C 3 25 Virtual Wor Vide Node Name 20 00 642B2EXES2Z E5 WI Enable Disable Ethernet Protocol Arrow keys and Enter to select 10 This window selects which protocols it will support and also displays the partitions assigned Networking MAC address the iS
17. SIZES In WINGO OWS nrisreriesii ire inr eneee vie EE I E 82 Setme MITO SIZES 10M Eu EE 85 Setting MTU Sizes in VMWare ESX ESXi A1 89 Setting MTU Sizes in VMWare ESXiI 5 0 E 92 Pamp A EEA 94 EA OversubScHption Gel 95 Partitioned Overs ubscription En le scsiesisasseseeiersioavecespinesssisaesntas stancesesscassaneseeavbiaeeeavisagsinevaessdeanenees 102 Weignted Che EECHELEN 105 Oversubscription With One High Priority Partition Exvample mnn 107 Default Fixed Subscription Example sees nnne nnn nnne enn 109 BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 3 57712 k 578xxS Software User Manual Table of Contents Mixed Fixed Subscription and Oversubscription Example eese enn 111 Mixed Weights and Subscriptions EXam ple eese enne nnne nnns 115 BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 4 57712 k 578xxS Software User Manual List of Tables List of Tables Table 1 Protocols Available in Operating Systems Versus SF and Switch Independent Partitioning Modes 26 Table 2 Dell 12G Networking Device Release verses Dell Servers ccccceeccsececeeceeeeceeceeeneceeeseeeceeeseegeeeess 27 Table 3 Port Function MAC Address Example EE 47 Table 4 Non DCB Equal OversubscriptiOn sinscusedis sa cea a toca te nV addet YER EITHER Eeeeeee 95 Table 5 DCB e EUR Re Te de ge E 98 Table 6 DCB Equal Oversubscription with one Lossless FCoE
18. TGP 1 1 1 24 49311 OF fF loaded TGP 1 1 1 24 49312 DFF loaded TGP 1 1 1 24 49313 OF fF loaded TGP 1 1 1 24 49314 OF fF loaded TGP 1 1 1 24 49315 Of F loaded TGP 1 1 1 24 49316 Of F loaded 3 3 3 24 49289 3 3 3 24 2 49291 3 3 3 24 49293 3 3 3 24 49294 InHost BROADCOM June 07 2012 e 2CS57712 SWUM103 R Foreign Address 1 1 1 25 5H1H 1 1 1 25 5H12 1 1 1 25 5HH1 1 1 1 25 5H13 1 1 1 25 5HH8 1 1 1 25 5883 1 1 1 25 5H14 1 1 1 25 5HBH EK WK PERLE 1 1 1 25 5HB6 1 1 1 25 5HH9 1 1 1 25 5H15 1 1 1 25 5HHB2 1 1 1 25 5811 1 1 1 25 5HHB5 1 1 1 25 5HH 3 3 3 24 49293 3 3 3 24 49294 3 3 3 24 49289 3 3 3 24 4929i State ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED Switch Independent Partioning Setup Guide 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning Broadcom Advanced Control Suite 4 BACS4 The BACS4 utility provides useful information about each network adapter that is installed in your system including partitioned adapters BACS4 enables you to perform detailed tests diagnostics and analyses as well as allows you to view and modify various property values and view traffic statistics for each adapter including other vendor devices BACS4 allows the enabling and c
19. This following plot shows how the four partition s send traffic is independent of each other Unlike the previous examples none of the partitions and their associated traffic flows take more than their designated bandwidth portion the total bandwidth of all four partitions of the port is equal to or less than the total available bandwidth of the port In this example each partition takes only 2596 or 2 5 Gbps of the total available bandwidth when their test application starts to transmit traffic Furthermore if a partition has more than one active traffic flow these flows will share that partition s allowed bandwidth Unused port bandwidth is not reallocated to any partition above it s own maximum bandwidth setting e When P1i starts to send traffic at tO it only takes the subscribed 25 of the 10 Gbps bandwidth available which is 2 5 Gbps e when P1e starts to send at t1 it will share partition P1 s 2596 with P1i Each is allocated 1 25 Gbps and neither expands into the unused 7 5 Gbps remaining e When P2e starts to send at t2 it only takes it s partitions subscribed 2 5 Gbps and does not affect either of partition P1 s sending traffic flows e When P3e starts to send at t3 it again only takes it s partitions subscribed 2 5 Gbps and does not affect P2e or either of partition P1 s sending traffic flows e When P4e starts to send at t4 it also only takes it s partitions subscribed 2 5 Gbps and does not affect any of the other part
20. VM s virtual adapter setting as desired and to use ping s to ensure the desired MTU size without switch fragmentation is supported throughout your network Examples e Equal Oversubscription Example e Partitioned Oversubscription Example e Weighted Oversubscription Example e Oversubscription With One High Priority Partition Example e Default Fixed Subscription Example e Mixed Fixed Subscription and Oversubscription Example e Mixed Weights and Subscriptions Example BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 94 57712 k 578xxS Software User Manual Examples r Note All bandwidths given in these examples are approximations Protocol overhead application send rate variances and other system limitations may give different bandwidth values but the ratio relationship between the send bandwidths of the four partitions on the same port should be similar to the ones given Depending upon OS requirements on the 57712 k 57810S the traffic types for each partition could be L2 Ethernet or iSCSI Offload or FCoE Offload where L2 Ethernet can be on any of the four partitions AND up to two iSCSI Offload can be on any two of the four partitions OR one FCoE Offload can be on any one of the four partitions plus one iSCSI Offload can be on any one of the remaining three partitions The partitions data flows on Windows can be a combination of L2 Ethernet traffic with or without TOE and or HBA traffic
21. Weights are all zeros OR in DCB mode each traffic type will have an equal weight with respect to one another see Examples on page 94 r Note DCB mode is supported in Windows Linux RHEL v6 x and SLES11 SP1 Oracle Solaris 10 u9 u10 and VMWare ESXi 5 0 OS s on the 57712 k 578xxS devices VMWare ESX ESXi 4 1 does not support DCB which includes both FCoE and DCB Lossless iSCSI on the 57712 k 578xxS devices Each partition s Maximum Bandwidth settings can be changed in the same way and has a range of 1 to 10096 in increments of 1 of the port s current Link Speed at 10 Gbps this would be in 100 Mbps increments and at 1 Gbps this would be in 10 Mbps increments This setting limits the most send bandwidth this partition will use and will appear as its approximate link speed in various places in the respective operating system even though the four partition s are sharing the same overall connection i e the four partitions may advertise in the OS that their link speed is 10Gbps each but they all share the same single 10Gbps connection Displayed values may be rounded off by various applications The Maximum Bandwidth value is applicable to both DCB and non DCB modes of operation The Maximum Bandwidth value is applicable to the send TX direction only r Note A partition s send Maximum Bandwidth setting does not affect a partition s receive direction traffic bandwidth so the link speed displayed for the partition is for the send transmi
22. actively sending partitions on that port when there is more send traffic pending on the partitions than send bandwidth available on that port It is more than just a minimum bandwidth setting This setting follows these rules e The individual configurable value range is O to 100 The recommended Relative Bandwidth Weight value range is either all 096 OR no lower than 10 on a single partition e The SUM of a single port s partitions values MUST be either exactly 100 or exactly 0 which means all four BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 21 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning of the partitions are set to 0 e If one or more of a partition s weight is set to 0 but the sum is 100 i e not all of the partitions are set to zero then that partition s relative bandwidth weight value is effectively 1 with respect to allocation calculations e Setting all the partition s values to O will give every traffic flow on every partition equal access to the ports available bandwidth without regard to which partition they are on unless restricted by the partition s Maximum Bandwidth settings e fthe sum of the relative bandwidth weights is 100 and there is more than one type of traffic flow on a specific partition i e iSCSI and L2 Ethernet or FCoE and L2 Ethernet then the traffic on that specific partition will share the bandwidth being allocated as if
23. ai 0030 Broadcom BOMS77 12 Net reme I 10 GigE NDIS WED Client 253 Driver Date 4 20 2011 1 oi f 0054 Broadcom BCM57712 Netitneme D 10 GigE Multi Function 254 BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 36 57712 k 578xxS Software User Manual The following shows the configuration of FcoE device settings Om Jdwvarnoed Contmi Suite 4 E me Adapter 3 BCMS7711 AD CI EE Adapters 57712 A1 Se Porti amp PF 0080 Sroedcom BCMS7712 Netireme IE 10 Ggf Mult Funcbon 250 0051 Broaden BCM57712 NetXtreme IT 10 GE Mui Funcbon F51 S I 0053 broskos BCM57712 NetXtreme IL 10 Gef Mul Funcon 453 CH 0000 Broadcom BCMS7712 Netitireme IT 10 Gef DES VED Cieni ep Et PI 0054 Broadcom 8CMS7712 Nettreme I 10 GE Mult Funcion 54 Configuring Switch Independent Partitioning Broadcom Advanced Control Suite 4 Fle W Acton Filter Context Tonos Tears SCSI Hek rw fae Eee De General E PS Hosts WIN RA3QRKD731 E EE Adapter 1 BCM5709 CO MS Adapter2 CMS Na CO WS Adapter3 BCMS7711 AO S EE Adapters 57712 A1 EF Porti e T 0050 Broadcom BCMS7712 NetXtreme IT 10 GigE Multi Function 50 0051 Broadcom BCM57712 NetXtreme TI 10 GigE Multi Function 851 amp PE 0053 Broadcom BCM57712 Netxtreme II 10 GigE Multi Function 253 SA 0030 Broadcom BCMS7712 NetXtreme I 10 GigE NDIS VED Client 53 F 0054 Broadcom BCMS7712 Netxtreme II 10 GigE Multi Function 54
24. from ifaces Record B key bnx2i 88237de882588unkB iface iscsi ifacenane bnx2i 88237de882588unkB iface net ifacenane unkB iface ipaddress null iface huaddress default iMface transport nane bnx2i 88237de88258 iface phydey yvmnic2 iface vlan 8 iface portFirst 63488 iface portCount 2848 Request completed with status HR The following command disassociates or removes all iSCSI offload adapters from the ifaces list vmkiscsid x delete from ifaces The following command can be used to delete a specific vmhba association esxcli swiscsi nic remove n port name d vmhba This command is used if changing the iSCSI HBA devices on your host Return to the vSphere Storage Adapters window and proceed with Discovering the iSCSI target LUNs This is done by selecting the desired vmhba in the Configuration Storage Adapters window and then clicking Properties in the Details window below it BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 64 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning 77 172 168 0 73 vSphere Client File Edit View Inventory Administration Plug ins Help gt A Home b il Inventory b SH Inventory 8 e 172 168 0 73 ESX 4 1 iSCSI Host Broadcom com VMware ESX 4 1 0 260247 Evaluation 59 days remaining H VM 1 W2k8R2 201 ee Ee Ee dE BE EE tb VM 2 W
25. iSCSI OR FCoE For VMWare ESX ESXi 4 1 and Linux RHEL v5 x the L2 Ethernet protocol is always enabled and up to two iSCSI Offloads can be enabled but no FCoE Offload For Linux RHEL v6 x and SLES 11 SP1 the L2 Ethernet protocol is always enabled and up to two iSCSI Offloads or one FCoE Offload and one iSCSI Offload can be enabled For Citrix XenServer 6 0 only the L2 Ethernet protocol is available for the partitions For Oracle Solaris 10 the L2 Ethernet protocols and up to one FCoE Offload can be enabled For VMWare ESXi 5 0 the L2 Ethernet protocols are also always enabled and up to two iSCSI Offloads or one FCoE Offload and one iSCSI Offload can be enabled Equal Oversubscription Example The following is an example of oversubscribed bandwidth sharing in non DCB mode All traffic types over the four partitions of the port have an equal weight i e they are all set to 096 and can individually use the maximum bandwidth of the connection i e 10 Gbps in this case In addition to the Ethernet Protocol s being enabled on all four partitions the iSCSI Offload protocol is enabled on partition 1 and 4 The iSCSI Offload Protocol can be enabled in any two partitions When all of the partitions have zero relative bandwidth weights each traffic flow will act as if in it s own separate partition each taking an equal share of the available bandwidth up to that partitions maximum bandwidth which is 10096 in this example so does not further limit an
26. iSCSI Offload FCoE Offload Windows 2008 R2 Yes Yes Yes Yes Yes Yes Windows 2008 R2 Yes Yes Yes Yes Yes Yes Hyper V Oracle Solaris 10 u9 Yes No Yes Yes No Yes u10 RHEL5 5 Yes Yes No Yes Yes No RHEL 6 0 Yes Yes Yes Yes Yes Yes SLES10 Yes Yes No Yes Yes No SLES11 SP1 Yes Yes Yes Yes Yes Yes Citrix XenServer v6 0 Yes No No Yes No No VMWare ESX ESXi 4 1 Yes Yes No Yes Yes No VMWare ESXi 5 0 Yes Yes Yes Yes Yes Yes a DCB DCBX PFC ETS supported BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 26 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning Supported Server Platforms The 57712 k and the new 12G release networking devices 5719 5720 578xxS are available for a wide variety of Dell server platforms as shown in Table 2 Table 2 Dell 12G Networking Device Release verses Dell Servers 57712 k 57810S K 57810S K 57810S 57800S T 57810S T 5719 NIC 5720 NIC 5720rNDC 5719 bMezz bNDC 57800S rNDC bNDC bMezz NIC rNDC NIC Dell R720 R720 R720 M620 blade M710HD R720 M620 M620 R720 R720 R720 Servers R720XD R720XD R720XD and server and R720XD and blade M910 and R720XD R720XD and R720XD R620 T620 R620 T620 R620 rack M915 R620 rack server M915 R620 R620 rack R620 T620 and other andother servers blade servers blade T620and servers and other rack and rack and servers servers other rack rack and towe
27. s cause the send bandwidth to be shared between the active sending partitions first and then the active sending traffic type flows in a two step manner Setting them to all 0 s causes the logic to work similarly to the way it does in DCB mode when all traffic types are in the same PG e fthere was only one traffic flow in each partition then the results would be similar to setting each partition s relative bandwidth weight to 096 since the single traffic flow would not be sharing a partition s bandwidth with another traffic type BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 103 Examples 57712 k 578xxS Software User Manual E EE m s Zement 8 1 oe Da kend 9 2 8 115 M UM UM M em UM NM M M Page 104 Switch Independent Partioning Setup Guide NM dr i a i GPS zg June 07 2012 e 2CS57712 SWUM103 R BROADCOM 57712 k 578xxS Software User Manual Examples Weighted Oversubscription Example The following is an example of weighted oversubscribed bandwidth sharing with different weights assigned to each partition in non DCB mode This example has each partition taking the maximum bandwidth when no other partition is active plus as each partition starts and stops sending the amount of bandwidth is shared as an approximate ratio of the currently sending partitions weight values There would be no difference between the previous examples and
28. size of 9000 bytes minus the 28 bytes of automatically added OVERHEAD but the outputted displayed size is plus 8 bytes which are the included ICMP header data bytes The Don t Fragment a k a DF flag d must also be set else the packet could be fragmented somewhere along the way and you would not know it The optional count c is the number of times you will send this ping You don t have to use the optional source interface command I a b c d or I devName unless you have multiple adapters that lead to the same target IP address and you want to check a specific one You can also use the ESX vmkping command root sieoral network scripts ping d c 3 I eth1 s 8972 192 168 20 10 PING 192 168 20 10 192 168 20 10 from 192 168 20 101 eth1 8972 9000 bytes of data 8980 bytes from 192 168 20 10 icmp_seq 1 ttl 255 time 0 185 ms 8980 bytes from 192 168 20 10 icmp_seq 2 ttl 255 time 0 177 ms 8980 bytes from 192 168 20 10 icmp_seq 2 ttl 255 time 0 180 ms 192 168 20 10 ping statistics 3 packets transmitted 3 packets received 075 packet lost round trip min ave max 0 177 0 181 0 185ms If it does not work you will see a failure message root esx41 ping d c 3 I eth1 s 8972 192 168 20 10 PING 192 168 20 10 192 168 20 10 from 192 168 20 101 eth1 8972 9000 bytes of data sendto failed Message too long sendto failed Message too long sendto failed Message too long 192 168 20 10 ping statist
29. 0 which is equivalent to 1 Gbps or 10096 of the 1GbE link speed the last zero position is not used The following shows the RHEL Network Configuration page with the eight enabled Ethernet protocol partitions always four per port as the Broadcom 57712 k 578xxS NetXtreme II 10 GigE Ethernet devices Eth2 through Eth9 in this example Linux enumerates the partitions in order of the PCI function numbers which is slightly different from Windows where port 0 has functions partitions 0 2 4 6 which are eth 2 4 6 8 and port 1 has functions partitions 1 3 5 7 which are eth 3 5 7 9 BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 52 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning Network Configuration Profile Help f B Os vw z New Edit Copy Activate Deactivate Devices Hardware Hosts You may configure network devices associated with physical hardware here Multiple logical devices can be associated with a single piece of hardware Device Nickname z Ethernet Gi Active Ethernet N Active s eth 1 57712 NPO Ethernet 5 57712 NP1 Ethernet 2 57712 NP2 Ethernet 6 57712 NP3 Ethernet 3 57712 NP4 Ethernet J 57712 NPS Ethernet 4 57712 NP6 Ethernet 8 57712 NP7 Ethernet E tL zm Ww Ln qi LJ m A Lri E e m M T c e mm Ft imd I E EE EE BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWU
30. 07 2012 e 2CS57712 SWUM103 R Page 62 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning m p u i m dr ty rt to mt D This command is a good way to determine which vmnic s and their associated vSwitch vmk are assignable to each iSCSI Offload vmhba s The following command lists the information on previously associated vmhba devices esxcli swiscsi nic list d vmhba32 Lo iD D CF 4 Im The following associates the desired iSCSI offload adapter vmhba to the desired vSwitch vmk vmnic The vSwitches were all created earlier in the networking window in vSphere esxcli swiscsi nic add adapter vmhba nic vmk Or esxcli swiscsi nic add d vmhba t n vmk BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 63 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning If the wrong information is entered an error message is displayed Bund ech SAK RAEIlEN dt esxcli swiscsi nic add d vmhba34 n vmkl Errors Failed to add nic In this example the Broadcom iSCSI adapter vmhba32 on vmnic2 is associated with the vSwitch vmkO Rerunning the previous ifaces list command shows the added iSCSI Offload device as associated with vmk0 vmnic2 LrootBESX411SCS J es cli suiscsi nic add adapter vnhba3Z2 nic vnkH root ESX41iSCSI IR vmkiscsid x select from ifaces Sending SOL Request select
31. 10 GE Mult Function SCSI Adap SS Portal 11 1 4 i j san 2010 10 com broadcom tgt 15a ries 158 pas CARMEL mm Aeris eek 3 Ek rmm 1 Brecon Gear ced Co Cbr Br CE sermon sso s D I pos Broadcom BCH37712 Het reme D 10 GE Mule Function 54 Ei ian 2011 05 com brosdcomcigt Srl 0027 Broadcom BCMS7712 Nettreme II 10 Gage NDIS VBD Ghent a54 Segen Name 3 D I 00591 Broadcom BOMS77 12 NetXtreme D 10 Gef Muis Function 59 Session id ign 2011405 c0m broadcom tptita aml FERRARO 190 72008 TTT LK SCO H Portal 11 11 24 l pys rectas ZR ports be t Lii 1 fed eeed ET GE 0031 Broadcom BOMS7712 Netxtreme II 10 GigE NOES VED Clent 59 e P ment Broadcom BCM57712 Netumreme D 10 Gage Mul Function s60 Lu S 27 0033 Broadcom BCMS7712 NetXtreme IT 20 GagE NOIS WED Chant 60 Fomaterus e D I rei Breadenm 80957712 Netireme D 10 Gig Multi Function 261 Ebay 2011405 com brosket ww 00 4 Broadcom BCM57712 Netitreme I 20 Cart Mult Function GOSI Adapter 261 Season Name jin 202105 com broadcoesctgt 1a rami GH 0034 Broadcom BCM57712 Metitreme I 20 Gage NDES VED Chent 251 Port2 BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 41 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning The following shows the per partition iSCSI device per Portal traffic statistics Hoe Adhere Contre Sethe d
32. 100 percent BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 45 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning Device PCle Bus Location The PCle interface Location Bus and Device position numbers are the same for both ports and all eight of the partitions on those ports The only PCle interface location values that are different are the Function numbers In non partitioned Single Function SF mode you would only have functions 0 and 1 In partitioned Switch Independent Partitioning mode you have functions 0 through 7 with functions 0 2 4 6 existing on the first port and functions 1 3 5 7 existing on the second port The actual numbering position an adapter is assigned by Windows is not entirely related to the PCle interface numbering and is more related to what open location position numbers are available in the registry when the adapters get enumerated Therefore port O partition 1 may not always occupy the first position in the Windows Device Manager s Network Adapters or Storage Controllers or System Devices sections 57712 57810 DUAL 10GbE Port Device KS Location Bus Device Function Ethernet MAC Example det 10GbE port The first three are the same for both ports Address IP Address Port 0 Partition 1 x Y Z 0 00 10 18 88 E7 A8 LA N N N X Ge p o A A mi Partition 4 00 10 18 88 E7 B4 4 4 4 b DP XD CD OD CP CD CD C
33. 2 MP Po 57712 MP Po 57712 MP Po 57712 MP Po 57712 NP Po REM rtlPartton rtOPert rion2 rtlPar bon2 rtOParbbon3 rtliParbton3 ae BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 49 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning In the Properties window select the device Configure button In the NDIS client device properties you will find that connection s PCle bus device function location information For partitioned adapters locate the function number that provides the partition that this connection is connected The same can be done with Device Manager especially for iSCSI Storage devices All of the enabled devices on the same partition have identical PCle interface location information with only the function number varying The following shows the eight Ethernet partitioned adapter s PCle device location information Notice the Windows enumerated device numbering values do not follow the PCle bus function numbering nor the port to partition numbering BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 50 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning i 1 57712 HPO Port Partition 1 1 1 1 20 Status 2 57712 NP2 Port 0 Partition 2 3 3 3 20 Status Broadcom BCH57712 NetXtreme II 10 GigE NDIS VBD Client 76 x Broadcom BCM57712 Netxtreme II 10 GigE NDIS
34. 2 SWUM103 R Switch Independent Partioning Setup Guide Page 17 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning DOLL SYSTEM SETUP Help About Exit Integrated NIC 1 Port t Broadcom NetXtreme Il 10 Gb Ethernet BCM5 800 78 2B CB 5B 99 28 Broadcom Main Configuration Page Broadcom Main Configuration Page Firmware Image Menu Device Configuration Menu MBA Configuration Menu ISCSI Boot Configuration Menu FCoE Boot Configuration Menu NIC Partitioning Configuration Menu Virtualization Mode Forced SF NPAR Chip Type BCM57800 BO PCI Device ID Bus Dev Func 01 00 00 p Virtualization mode configuration System must be rebooted in order for changes to take effect PowerEdge R720 Arrow e s and Enter to selec Default Esc to exit page Tab to change focus 6 From the Device Configuration window select either Enabled or Disabled for the device s DCB Protocol mode BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 18 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning DOLL SYSTEM SETUP Help About Exit Integrated NIC 1 Port 1 Broadcom NetXtreme Il 10 Gb Ethernet BCM5 810 84 2B 2B E8 22 E3 7 Main Configuration Page Device Configuration Main Configuration gt Device Configuration Device MAC Addresses Broadcom NetXtreme Il 10 Gb Ethernet BCM57810 84 2B 2B E8 22 E3 Broadcom Ne
35. 2 k 578xxS Software User Manual About This Document Figure 6 57810S T 10GBASE T Converged NIC BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 11 57712 k 578xxS Software User Manual About This Document iT Gu Hass tt GM iDSTA ut 4 A Figure 7 57800S T 10GBASE T Rack Converged NDC Audience This document is written for the network administrator who wishes to partition the Broadcom network controller on a Microsoft Windows Server VMWare ESX ESXi Oracle Solaris Citrix Xen Server SUSE Linux Enterprise Server SLES and Red Hat Enterprise Linux RHEL system with e up to eight functions four per port Ethernet enabled in addition to e up to four functions two per port iSCSI HBA enabled in operating systems where the specific HBA can be enabled Or BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 12 57712 k 578xxS Software User Manual Technical Support e up to two functions one per port FCoE HBA enabled plus up to two functions one per port iSCSI HBA enabled in operating systems where the specific HBA can be enabled Acronyms and Abbreviations In most cases acronyms and abbreviations are defined on first use For a comprehensive list of acronyms and other terms used in Broadcom documents go to http www broadcom com press glossary php Technical Support Broadcom provides customer access to
36. 2 k 578xxS Software User Manual Configuring Switch Independent Partitioning VMWare ESXi 5 0 VMWare ESXi 5 0 shows the respective device protocols that were enabled in USC if the respective NetXtreme II device drivers are installed Go to the Dell Driver Download web site http support del com for the latest device Firmware if not already installed Go to the VMWare web site http downloads vmware com for the latest device drivers and ensure they are installed on your system In VMWare the Ethernet Protocol should be enabled on all eight partitions VMWare ESXi 5 0 supports the 57712 k DCB L2 Ethernet iSCSI Offload and FCoE Offload Protocols in both Switch Independent Partitioning and SF modes in the VMWare driver versions 1 70 34 v50 1 bnx2x 2 70 1k v50 2 bnx2i 1 0 1v v50 1 bnx2fc and Dell firmware version 7 0 47 or later see http partnerweb vmware com programs hcl ESX NPAR config pdf These same VMWare driver versions support the 578xx devices L2 Ethernet and iSCSI Offload Protocols in both Switch Independent Partitioning and SF modes The 578xx devices support DCB L2 Ethernet iSCSI Offload and FCoE Offload Protocols in both Switch Independent Partitioning and SF modes with the VMWare driver versions 1 72 18 v50 4 bnx2x 2 72 10 v50 2 bnx2i 1 72 11 v50 1 bnx2fc and Dell firmware version 7 2 x or later The vSphere display on ESXi 5 0 is similar to ESX ESXi 4 1 in that the Network Adapters window shows the same eigh
37. 2003 Broadcom BC 7722 eem D 10 Gef CH Adames i5 cus 10124387 x TZ pod Broadcom BC 712 etie eme Z 10 Gg pn Adae adl 5 CS gt Oe geg ZEB 9017 Braedon 8957712 eepese D 1 Gg b OD here ei HN Ei oc Ports Pte 124587 m T pes seeks SOME TTL enes D Gef ail Ce OCH Broadcom 8457777 wegen D VG PCoE bispi st Geen G Ti poH faias BOSD erties D i g D I t Che 97 GER Aert BOT A e SE aimi BOT ag 8 12 3 4 5 8 7 0 52 0 0 0 0 8 n T NR T 1 T Ek oma Je a i CHE Fem BACSA allows per port enabling and configuring of the DCB protocol default and initially advertised settings and port willingness to change to the received DCBx values as shown below ven Bb ud miru t cu Dage Ve D weeps zs ES kipi CO CD EE kiri Gei Ka Cen GE Aomme OCHS AD FE Acdasger d CMS AO hw a ME i okoa BOMGET712 weng D 25 Gg 51 e D DXX Bekia SOMES Merete E 10 Gg SCSI Adap m FAI x TE QOO broden Bt 12 eiee D 15 Gef EC of dudes Ft ai 10027 Brocken BOT ete E 10 Gef NOUS VID Clan 621 i Bert S DE era tenacious tt Net reme D 25 Gef sl Ce 0007 Broadcom OMT TES Mette E 10 Gag PCoE Adapter i52 Tee onc Beaia BOLT LS eirese E 15 Gef GCSE arte aris TRE DON frech BOTT ED eire E 15 Gef S IS VIC Cleri art ENS Acdger ose AO d EN caeci CES AD BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 39 57712 k 578xxS Software User
38. 2103589 33 bnx2i 00101888 amp 7aa torage vmhba34 ign 1998 01 com vmware localhost 974021981 37 bnx2i 00101888e7b4 Networki etworking vmhba35 ign 1998 01 com vmware localhost 902103589 35 bnx2i 0010 1888 amp 7b6 Storage Adapters Network Adapters Advanced Settings Power Management Broadcom iSCSI Adapter ign 1938 01 com vmware localhast 374021991 34 BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 58 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning These Ethernet Protocol adapters in each partition are configurable like a normal port adapter in vSphere s Networking Configuration pages as shown below Getting Started Summary Virtual Machines Resource Allocation Hardware l I I I I Virtual Machine Startup Shutdown Virtual Machine Swapfile Location Security Profile System Resource Allocaton Advanced Settings BROADCOM June 07 2012 e 2CS57712 SWUM103 R Virtual Switch vSwitch Virtual Machine Port Group VM Network Widicenne Port L2 Management Network vmk 1 10 41 2 Virtual Switch vSwitch1 Ven ai Machine Port Group L2 VMK_NIC_1 El 1 virtual machine s VM41_P1_1 VMkamel Port 2 VMK I 1 vmk1 1 1 41 1 Virtual Switch vSwitch2 Virtual Macrone Pert Group LJ VMK NIC 2 B 1 virtual machine s VM41 P2 1 VMkemel Port L2 VvMK I2 vmk2 2 2 41 1 Virtual Switch vSwitch3 Virtual Mac
39. 2k8R2 202 Getting Started Summary Virtual Machines Resource Allocation Performance Ie TER Local Users amp Groups Events Permissions CB VM 3 W2k8R2 203 E VM 4 W2k8R2 204 Hardware Storage Adapters Refresh Rescan All Health Status wi CH vmhba34 Block SCSI ICH10 4 port SATA IDE Controller Q vmhbai Block SCSI 3 wvmhba33 Block SCSI Processors Memory Storage Networking Storage Adapters Lee Adapters ign 1998 01 com vmware ESX 4 1 I5CSI Host 720 Network Adapters WE Advanced Settings Details Power Management vmhba32 Software Model Broadcom iSCSI Adapter iSCSI Mame ign 1998 01 com vmware ESX 4 1 iSCSI Host 720214776 32 Licensed Features iSCSI Alias bnx2i 001018442f3a Time Configuration Connected Targets 0 Devices 0 Paths 0 DNS and Routing Authentication Services Virtual Machine Startup Shutdown Name Runtime Name Transport Capacity Owner Hardware Acc View Devices Paths Virtual Machine Swapfile Location Security Profile If the non offload iSCSI pathway is needed it must be enabled before use This is done by clicking it s Properties in the Details window below it and enabling it by clicking the Configure button in the General tab In the iSCSI Initiator vmhba Properties window select the Dynamic Discovery or Static Discovery tab and enter the desired iSCSI target LUN information 27 iSCSI Initiator vmhba37 Properties General Dyn
40. 3 R Page 107 57712 k 578xxS Software User Manual Examples e Finally when P4e stops sending traffic at t10 P4i will take all of the available bandwidth for 10 Gbps e Whenever the fourth partition s bandwidth needs drop off the other actively sending partitions will equally increase their respective shares to automatically occupy all of the available bandwidth BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 108 57712 k 578xxS Software User Manual Examples Default Fixed Subscription Example This is an example of the default partition settings that has all of the relative bandwidth weights set to 0 and the maximum bandwidths set to 2 5 Gbps Since the total of the maximum bandwidth values are set to exactly 100 i e never can reach an oversubscription situation the traffic flows in each partition will share that partition s bandwidth allocation with respect to it s overall maximum bandwidth ceiling and the relative bandwidth weights are never used Table 10 Non DCB Default Fixed Subscription Relative Bandwidth Maximum Port Partition Weight Bandwidth Protocol Plot Color Port O Partition 1 P1i 0 25 iSCSI Offload Green Port O Partition 1 P1e 0 25 Ethernet Brown Port O Partition 2 P2e 0 25 Ethernet Purple Port O Partition 3 P3e O 25 Ethernet Yellow Port O Partition 4 P4e 0 25 Ethernet Blue Port O Partition A P4i 0 25 iSCSI Offload Red
41. 51 Broadcom BCM57712 Neti treme I 20 GgE Multi Function 251 E Io Broadcom BCMS7712 Nettreme IT 10 GigE Mult Function 253 0030 Broadcom BOMS7722 NetXtreme I 10 GE NDIS VED Gent 553 m oo 0054 Broadcom BCMS7712 NetXtreme I 10 GigE Muli Function 254 Fi i BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 83 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning The MTU size for each individual iSCSI Offload HBA protocol enabled partition can be independently set from Normal 1500 bytes up to Jumbo 9600 bytes in the BACS4 iSCSI Management Configuration page Broadcom Advanced Control Suite 4 Fle View Acton Hier Context Tools Teens GH Heb jete paven E roras E F strane E BH Hosts S H magnon E Adapteri Mach E E Adapter B0M5709 OH E B Adapter3 BOM57711 Al B Adapter 57712 Al Ee Pert 0050 Sroadeom BCM57712 Nettreme D 10 Gig Multi Function 50 EM 0012 Broadcom BCMS7712 Netitreme II 10 GE Multi Function SCSI A Porta 1 1 1 4 scm Portal fa30 210 lofi fesdach The FCoE device MTU frame size is fixed at 2500 bytes and is not adjustable but is viewable in the FCoE device Information page Broadcom Advanced Control Suite 4 Fle Ven Acton Filter Context Tools Trans GCM Help Fiter aun view Te ifinformation F vital signs E Ge information
42. 5204 197157 58 VMWare driver versions can be checked using the following commands esxcli software vib list vmkload_mod s bnx2x vmkload_mod s cnic vmkload_mod s bnx2i vmkload_mod s bnx2fc vm support ethtool i vmnicN HHH H H HH Details on Configuring iSCSI Offload on ESX ESXi 4 1 After creating a networking vSwitch but before attempting to discover any iSCSI targets you need to associate any Broadcom 5709 5771x 578xx iSCSI hardware offload HBA vmhba to a previously created vSwitch vmk vmnic Otherwise you will see a caution message when you attempt to discover any iSCSI targets via the Broadcom iSCSI Offloaded pathway This association step is not required if using the non offload software iSCSI pathway BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 61 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning iSCSI Initiator vmhba3 Properties General Dynamic Discovery Static Discovery Send Targets Discover i5051 targets dynamically From the Following locations IP4 host name iSCSI Server me The hast bus adapter is not associated with a vmknic Ta configure targets the adapter should be associated with a mknic Refer to the VMware documentation to associate the adapter with a vmknic Close Help The Broadcom iSCSI Adapter s vmhba are listed in the vSphere Configuration tab Hardware Storage Adapters window
43. 60331 4 6 MiB TX bytes 10027 9 7 KiB Interrupt 225 Memory d388080800 d3TTTTTT Link encap Ethernet HWaddr 00 10 18 88 E7 AE inet addr 18 1 1 283 JBcast 18 255 255 755 Mask 255 60 8 6 inet6 addr feB88 218 18ff Te88 e7ae 54 Scope Link UP BROADCAST RUNNING MULTICAST MTU 9688 Metric 1 RA packets 26782 errors O dropped overruns 8 frame TX packets 34 errors 0 dropped 0 overruns O carrier 6 collisions txqu evel en 1060 RX bytes 480 217 4 5 MIB TX bytes 8866 8 6 KiB Interrupt 264 Memory d48606000 d4fTTTTT Link encap Ethernet HWaddr 00 10 18 88 E7 BO8 inet addr 10 2 2 200 Bcast 10 255 255 255 Mask 255 0 0 0 inet6 addr feB88 218 18TT T1e88 e7b8 64 Scope Link UP BROADCAST RUNNING MULTICAST HMTU 9600 Metric 1 RX packets 2687 errors dropped overruns amp frame 6 TX packets 34 errors 8B dropped 8 overruns O carrier collisions txquevelen 1600 RX bytes 472737 461 6 KiB TX bytes 9139 8 9 KiB Interrupt 204 Memory d5880080 d5TTTTTT Link encap Ethernet HWaddr 080 18 18 88 E7 B2 inet addr 18 2 2 201 Bcast 18 255 255 255 Mask 255 6 6 6 inet6 addr feB0 210 18ff fe8B e7b2 64 Scope Link UP BROADCAST RUNNING MULTICAST HMTU 96580 HMetric l RX packets 2233 errors 8 dropped 8 overruns 8 frame TX packets 32 errors H dropped 6 overruns 6 carrier collisions txqueuvelen 16606 RX bytes 393371 384 1 KiB TX bytes 8381 8 1 KiB Interrupt 181 Memory d6800888 deTTTTTT BROADCOM Switch Independent Parti
44. 708 37 bnx2i 0010183e1346 Storage Adapters vmhba36 i5C5 iqn 1988 01 com vmware localhost 610019708 38 bnx2i 00101868ebdc vmhba37 CS iqn 1998 01 com vmware localhost 610019708 39 bmai 00101868ebde Advanced Settings dag iSCSI ign 1998 01 com vmware localhost 610019708 40 bnx2i 00101888e5ec Power Management vmhba39 iqn 1998 01 com vmware lacalhost 610019708 41 bnx2i 00 101888eeec Broadcom Corporation FCoE Adapter CH vmhba42 Fibre Channel ower Ethernet 10 00 00 10 18 88 ee ed 20 00 00 10 18 88 ee ed Licensed Features CH vmhba43 Fibre ChanneloverEthernet 10 00 00 10 18 88 ee ef 20 00 00 10 18 88 ee e Time Configuration Details DNS and Routing Authentication Services vmhba38 Virtual Machine Startup Shutdown Mode Broadcom iSCSI Adapter iSCSI Name ign 1998 01 com vmware localhost 6 100 19708 40 Virtual Machine Swapfile Location SCSI Alias KEE Security Profile OE SEET Ge A Host Cache Configuration System Resource Allocation View Devices Paths Agent VM Settings Runtime Name Target LUN Status Advanced Settings wmbbase CO Ti L0 ign 2011 08 com broadcom tgt12a rami12a 1 1 50 wmhba38 C0 TU0 LO iqn 2011 08 com broadcom tgti3b rami3b 1 150 0 Active 1 0 wmhba38 C10 T3 ign 2011 08 com broadcom tgtl14a ram14a 1 1 50 Active wmhba38 C8 T1 L0 iqn 2011 08 com broadcom tgti2a rami2a 1 150 0 Active umbh a2358 71n T3 ian 7017208 ram hreadcnmstatikl ramiGhet 3 En 1 Artia r Note VMWare s ESXi 5 x can experien
45. A adapter to MTU 9600 bytes r Note Each partition s MTU size setting can be different Using the above protocol enabled partition example e Port O Partition 1 Ethernet and iSCSI Offload HBA can be set to 9600 bytes e Port O Partition 2 Ethernet and iSCSI Offload HBA can be set to 5500 bytes e Port O Partition 3 Ethernet can be set to 1500 bytes e Port O Partition 4 Ethernet can be set to 9000 bytes In Linux use the ping command with the s size option to verify that Jumbo Frame support is configured throughout the desired network path i e ping s 8972 A B C D The Don t Fragment a k a DF flag is AUTOMATICALLY set in Linux The unfragmentable ping packet size is the desired MTU size to be checked 9000 bytes minus the automatically added overhead 28 bytes or 8972 bytes Note that the ping will return the send size plus the 8 bytes of the ICMP header The c switch sets the number of times this ping command is sent You can also use the optional source interface Capital I command Ia b c d or I devName if you have multiple adapters that are connected to the same target IP address and you want to check a specific interface root server ping c 3 s 8972 192 168 20 10 PING 192 168 20 10 192 168 20 10 from 192 168 20 200 8972 9000 bytes of data 8980 bytes from 192 168 20 10 192 168 20 10 icmp seq 0 ttl 255 time 185 ms 8980 bytes from 192 168 20 10 192 168 20 10 icmp seq 1 ttl 255 time 0 177 ms BROADC
46. CB Priority Group 1 with an ETS 50 and the other traffic flows are Lossy and in Priority Group O with an ETS 50 BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 112 57712 k 578xxS Software User Manual Examples Table 12 DCB Mixed Fixed Subscription and Oversubscription with Lossless FCoE Offload Relative Bandwidth Maximum Port Partition Weight Bandwidth Protocol Plot Color Port O Partition 1 P1f N A 80 FCoE Offload Green Port O Partition 1 Ple N A 80 Ethernet Orange Port O Partition 2 P2e N A 10 Ethernet Purple Port O Partition 3 P3e N A 10 Ethernet Yellow Port O Partition 4 P4e N A 80 Ethernet Blue Port O Partition A P4i N A 80 iSCSI Offload Red This is a similar combination example of a fixed subscription three of the partitions sum to 100 but all four sum to 180 When all four or at least the last two partitions are running traffic they share the space with each other up to their partition maximum bandwidth values and their PG s ETS settings otherwise they act as if they are independent connections e The first partition s traffic flow P1f initially takes its maximum bandwidth designated 8 Gbps when the test application starts to transmit traffic at tO to that port by itself not expanding into the remaining unused 2 Gbps bandwidth e When the second traffic flow on the first partition P1e starts to send at t1 the two activ
47. CSI Offload MAC address the FCoE Offload FIP MAC address the FCoE Offload Node WWN and FCoE Offload Port WWN values BACSA can also be used to control and view these settings Protocol selection follows these rules e Amaximum of two iSCSI or one FCoE and one iSCSI Offload Protocols HBA can be enabled over any two of the available partitions of a single port e The FCoE Offload Protocol is only available if DCB is also enabled and active on that port i e the 57712 k 578xxS device port is connected to a DCB compliant and enabled link partner e The iSCSI Offload Protocol can function without DCB but if DCB Lossless iSCSI Offload TLV is required then DCB must be enabled and active on that port i e the 57712 k 578xxS device port is connected to a DCB compliant and enabled link partner e Only one Offload Protocol either iSCSI or FCoE can be enabled per single partition in Switch Independent Partitioning mode e For simplicity using the first partition of a port for FCoE offload protocol is recommended since the FCoE port WWN will be the same for both SF and Switch Independent Partitioning mode on the same port This will make your Fiber Channel Forwarder FCF switch configuration much simpler e For Windows operating systems you can have the Ethernet Protocol enabled on all some or none of the BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 24 57712 k 578xxS Software User Manual Con
48. D CD gp CD GP CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CP CD CD CD CD gp CD CD CD CD CD CD CD CD CD CD CD CD CD CD GP 2nd 10GbE port ep d ep CD ep ep ep CD CD CD CD CD CD ep CD ep CD CD CD CD ep CD ep CD CD CD CD ep GD CD CD CD CD CD ep ep CD CD CD CD CD CD ep CD CD CD CD CD CD a Port 1 Partition 1 9 9 9 1 eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee ee ee ee ee BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 46 57712 k 578xxS Software User Manual The partition s MAC addresses interleave the two ports as do the function numbers see Table 3 Configuring Switch Independent Partitioning Table 3 Port Function MAC Address Example Port O Partition 1 Function O MAC address AS Port O Partition 2 Function 2 MAC address AC Port O Partition 3 Function 4 MAC address BO Port O Partition 4 Function 6 MAC address B4 Port 1 Partition 1 Function 1 MAC address AA Port 1 Partition 2 Function 3 MAC address AE Port 1 Partition 3 Function 5 MAC address B2 Port 1 Partition 4 Function 7 MAC address B6 The quad port 57800S which combines two 10GbE with two 1GbE ports has slightly different PCI device numbering The 57800S can have up to two partitions per 10GbE port where there can be up to two L2 Ethernet protocols plus up to two iSCSI Offload protocols OR one FCoE Offload plus one iSCSI Offload protocols enabled pe
49. F Porti EP 0045 Broadcom BCM57712 Netxtreme II 10 GigE 45 L D007 Broadcom BCM57712 Netxtreme II 10 GigE NDIS VBD Client 45 E Port 0046 Broadcom BCM57712 NetXtreme II 10 GigE 46 Po gm 0017 Broadcom BCM57712 Netxtreme II 10 GigE NDIS VBD Client 46 Figure 9 Broadcom Advanced Control Suite 4 If you are in Switch Independent Partitioning mode go to the 57712 k 578xxS Adapter Configuration Property window and click the next to the NIC Partition to expand the menu In the expanded menu if the NIC Partition setting is unchecked Disabled change it to checked Enabled and then set the desired partition s Ethernet NDIS iSCSI and FCoE protocols Also set the Relative Bandwidth Weights Maximum Bandwidth settings and then click Apply You must reboot the system for Windows to discover and install the device drivers BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 29 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning Broadcom Advanced Control Suite 4 File View Acton Filter Context Tools Teams iSCSI Help mer Len El onion DE nc rrr Explorer View E Information Configuration HM Hosts pee ve E B WIN RA3QIRKO731 artition H REN Adapteri e co d EE Adapter4 57712 Al El 9 Porti E TR 0050 Broadcom SCH 0012 Broadc ew 0027 Broadc 0051 Broadcom I d 0053 Broadcom GO 0030 Broad
50. GbE 2x1GbE Rack Converged Network Daughter Card 57810S T dual port 10GBASE T Converged Network Adapter and 57800S T quad port 10GBASE T RJ45 2x10GbE 2x1GbE Rack Converged Network Daughter Card with Switch Independent Partitioning TOE CEE DCB iSCSI Offload iSCSI Offload TLV and FCoE Offload technology Dell specifically disclaims knowledge of the accuracy completeness or substantiation for all statements and claims made in this document regarding the properties speeds or qualifications of the adapter Broadcom Corporation reserves the right to make changes without further notice to any products or data herein to improve reliability function or design j ee eve rythin g BROADCOM E mm E J Wf m BROADCOM CORPORATION Phone 949 926 5000 5300 California Avenue Fax 949 926 5203 Irvine CA 92617 E mail info broadcom com 2012 by BROADCOM CORPORATION All rights reserved Web www broadcom com 2CS57712 SWUM103 R_ June 07 2012
51. M103 R Page 53 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning Linux s ifconfig command shows the partition s eight Ethernet protocol devices and various statistics Link encap Ethernet HWaddr 00 10 18 88 E7 AB inet addr 18 1 1 208 Bcast 16 255 255 255 Mask 255 6 0 6 inet amp addr fe80 210 18Tf Te88 e7a8 654 Scope Link UP BROADCAST RUNNING MULTICAST MTU 9688 Metric 1 RX packets 279037 errors 8B dropped 8 overruns 8B frame 8 TX packets 48 errors 8 dropped 6 overruns carrier 6 collisions O txqueuelen 1800 RX bytes 5038387 4 7 MiB TX bytes 9923 9 6 KiB Interrupt 169 Memory dl1880000 dlffffff Link encap Ethernet HWaddr 06 16 18 88 E7 AA inet addr 18 1 1 201 Bcast 18 255 255 255 M Mask 255 8 8 8 inet6 addr fe80 210 181T Te88 e7aa 64 Scope Link UP BROADCAST RUNNING MULTICAST MTU 9588 Metri c 1 RX packets 275080 errors 8B dropped 6 overruns frame TX packets 36 errors 8 dropped 8 overruns 8 carrier 6 collisions txqueuelen 1888 RX bytes 4946656 4 7 MiB TX bytes 93456 9 2 KiB Interrupt 225 Memory d2888888 d2TTfTTTT Link encap Ethernet HWaddr 66 16 18 88 E7 Ac inet addr 18 1 1 202 Beast 16 255 255 255 Mask 255 0 0 0 inetb addr Te88 2180 1BTT Te88 e7ac 54 Scope Link UP BROADCAST RUNNING MULTICAST MTU 9606 MMetric l RX packets 27034 errors 8 dropped overruns 8 frame 8 TX packets 42 errors 6 dropped 8O overruns 8 carrier collisions 8 txqueuelen 1000 RX bytes 48
52. MK 1 to 9000 bytes esxcfg vswitch m 9000 vSwitch1 The following command will list all of the current VMKernel NIC settings esxcfg vmknic 1 BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 89 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning Modify the VMKernel NIC s MTU size using the following command This modifies vmnic1 which is in port group VMK 1 to 9000 bytes esxcfg vmknic m 9000 VMK I 1 Note VMWare ESX uses the term vmnic instead of the typical Linux term eth in the command line E Finally don t forget to set the associated VM s virtual adapter setting as desired In a Windows Server 2008 R2 VM the Network Connection Advanced Properties Jumbo Packet setting would be either 9014 Bytes or DISABLED for standard size 1500 Bytes if the Virtual Adapter is the default E1000 Intel R PRO 1000 MT Network Connection pm General Advanced Driver Details Power Management The following properties ane avadatte for this network adapter Cick the property you want to change on the left and then select ts value on the ight Property Adaptive inter frame Spacing sables Row Control 8 Dues B en ag Moderation 16128 Bytes 4028 Bytes intemupt Moderation Fate 30 Bytes Tz 4 EOM d Locally Zant Addam Disabled 7 1500 bytes Humber of Caalesce Butters Priority amp VLAN Receive Butters 8014 Bytes Jumbo TCP Che
53. OM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 88 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning 8980 bytes from 192 168 20 10 192 168 20 10 icmp seq 2 ttl 255 time 0 180 ms 192 168 20 10 ping statistics 3 packets transmitted 3 packets received 07 packet lost time 3043ms rtt min ave max mdev 0 177 0 181 0 185 0 005 ms If it does not work you will see root server ping c 3 s 8972 192 168 20 10 PING 192 168 20 10 192 168 20 10 from 192 168 20 200 8972 9000 bytes of data 192 168 20 10 ping statistics 3 packets transmitted packets received 100 packet lost time 3010ms Setting MTU Sizes in VMWare ESX ESXi 4 1 In VMWare ESX ESXi 4 1 the MTU size for each individual Ethernet protocol enabled partition can be independently set from Normal 1500 bytes up to Jumbo 9600 bytes This MTU size change will affect both regular L2 Ethernet and iSCSI software non offload pathway generated traffic Unlike other Linux OS s you can not directly adjust the MTU size using the ifconfig command Instead you must use various ESX 4 1 esxcfg commands The Ethernet protocol enabled partition s adapter MTU size is adjusted using SSH command line for example List all current vSwitch s information using the esxcfg vswitch 1command Modify a specific vSwitch s MTU size with the following command this modifies vSwitch1 which is in port group V
54. Options Graph refreshes every 20 seconds 2000000 This shows two Ethernet Protocols on two seperate partitions 57712 NPAR running and then stopping and then running again You can t see both Networking L2 Ethernet and Storage mm FCoE ISCSI traffic at the same time on vSphere s Performance monitor page They are sharing the PGO s ETS bandwidth with iSCSI and also with PG1 s ETS ed FCoE traffic flow i i i I i i i i I 4 00 AM 4 05 AM 4 10 AM 4 15 AM 4 20 AM 4 25 AM 4 30 AM 4 35 AM 4 40 AM 4 45 AM 4 50 AM Time Performance Chart Legend Minimum Average Maximum Rollup Object Key lt Latest Units 0 15881133 601577 Average VM50 W7 o 396789 KBps 0 160875 36 609415 Average VM50_W6 oO 420249 KBps 0 0 0 Average VMS WS oO 0 KBps 0 0 0 Average VM50 W8 E 0 KBps BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 81 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning Setting MTU Sizes e Setting MTU Sizes in Windows e Setting MTU Sizes in Linux e Setting MTU Sizes in VMWare ESX ESXi 4 1 e Setting MTU Sizes in VMWare ESXi 5 0 r Note In all cases the connecting switch port that a 57712 k Switch Independent Partitioning port is connected to must have the switch s MTU size set to the largest MTU size of those four partitions of the port if the user wants to support all four partitions MTU size settings Additionally the remaining networ
55. S Software User Manual Examples bd1 11 swdvt lab iry broadcom com Mware ESXi 5 0 0 469512 Getting Started Summary Virtual Machines Resource Allocation Performance STET TET Tasks amp Events Alarms Permissions Maps Storage Views Hardware Status e we Mk EI ti SSES view sst tiers Diirib ked Suh Processors Networking eres Settings Security Traffic Shaping NIC Teaming Memory m Port Properties AER Standard Switch vSwitch Remove Properties Network Label VMkernel 2 Switch3 Properties D x VLAN ID Optional fal 4095 Ports Network Adapters Storage vMotion Enabled r Port Properties Network Label vMkernel 2 Fault Tolerance Logging Enabled VLAN ID All 4095 Management Traffic Enabled iSCSI Port Binding V Enabled vMotion Disabled Fault Tolerance Logging Disabled NIC Settings Management Traffic Disabled MTU iSCSI Port Binding Enabled MAC Address 00 50 56 7F 18 08 MTU 1500 IP Settings IP Address 192 168 140 111 Subnet Mask 255 255 255 0 View Routing Table m Effective Policies Security Promiscuous Mode Reject MAC Address Changes Accept Lm Remove Forged Transmits Accept Cancel Help The Ethernet protocol enabled partition s adapter MTU size can also be adjusted using the same commands previously described for ESX ESXi 4 1 Be sure to set the associated
56. Setting MTU Sizes in Linux In Linux the MTU size for each individual Ethernet protocol enabled partition can be independently set from Normal 1500 bytes up to Jumbo 9600 bytes Both the Ethernet protocol and iSCSI Offload HBA enabled partition s adapter MTU size is adjusted at the same time using the ifconfig command ifconfig eth3 mtu NNNN up From above eth3 is the port identification of the specific 57712 k 578xxS partition to adjust the MTU size The NNNN is the new size of the MTU for that partition and can be set from 1500 to 9600 bytes The following shows all eight partitions being set to different MTU values BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 85 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning root localhost root localhost ifconfig root localhost ifconfig roote localhost ifconfig root localhost 4 ifconfig root lLocalhost ifconfig root localhost ifconfig rootglocalhost ifconfig roota localhost 4 amp ifconfig root localhost Link encap Ethernet HwWaddr 60 16 18 88 E7 A8 inet addr 16 1 1 260 Bcast 18 255 255 255 Mask 255 6 6 6 inet addr feB88 218 18TT Te88 ha Scope Link UP BROADCAST RUNNING MULTICAST TERM Metric l RAX packets Aal errors 8 dropped 0 overruns Trame 8 TX packets errors dropped B overruns Cartier collisions 8 txqueuelen 1888 RX bytes 470808 45 9 KiB TX bytes 6 8 b In
57. Software User Manual srol com 57712 k 578xxS INS N S E Third party information brought to you courtesy of Dell Switch Independent Partitioning Setup Guide 2CS57712 SWUM103 R 5300 California Avenue Irvine CA 92617 e Phone 949 926 5000 e Fax 949 926 5203 June 07 2012 Revision History Revision Date Change Description 2CS57712 SWUM103 R 06 07 12 Updated e 7 2 software release by adding the 578xxS device family and VMWare ESXi 5 0 information 2CS57712 SWUM102 R 12 08 11 Updated e 16 4 software release by adding FCoE DCB VMWare ESX ESXi 4 1 and BACS4 information 2CS57712 SWUM101 R 3 31 11 Updated e Switch Independent Partitioning in title and footer e NIC Partition NIC Partitioned and NIC Partitioning to Switch Independent Partitioning throughout the document e Single Function to SF throughout the document 2CS57712 SWUM100 R 03 15 11 Initial release Broadcom Corporation 5300 California Avenue Irvine CA 92617 2012 by Broadcom Corporation All rights reserved Printed in the U S A Broadcom the pulse logo Connecting everything and the Connecting everything logo are among the trademarks of Broadcom Corporation and or its affiliates in the United States certain other countries and or the EU Any other trademarks or trade names mentioned are the property of their respective owners 57712 k 578xxS S
58. Users amp Groups Events Permissions M 8 23 2011 3 50 15 AM 8 23 2011 4 50 15 AM Chart Options switht HAH amp Q H Co SCSI Storage Access iSCSI Storage Access not running with the other with the other Ethernet Ethernet Partitions means this iSCSI storage access Partitions running means performance takes all of PGO s ETS ed BW portions this iSCSI storage access but shares it with the FCoE PG1 s ETS portion PGO s zs performance is sharing ETS is 50 and PG1 s ETS is 50 the ETS ed BW portions with the other partition s Ethernet streams AND the FCoE storage Access 1500000 FCoE Storage Access Only gets 100 of the two port s traffic since there is no other traffic _ _ simultaneously inning FCoE Storage Access shares 50 PG1 s ETS of the two port s traffic with the other traffic streams i i i i i l i i i i i i 3 55 AM 4 00 AM 4 05 AM 4 10 AM 4 15 AM 4 20 AM E 4 30 AM 4 35 AM 4 40 AM 4 45 AM 4 50 AM Performance Chart Legend Minimum Average Maxi Rollup Object 749044 MEER s AMEN Average vue W6 3768 Average VM50 W7 BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 80 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning localhost localdomain VMware ESXi 5 0 0 Se i eT mt Ric Metis ie nee Performance Geile Mele D EI acl Mat E us Network Real time 8 23 2011 3 54 03 AM 8 23 2011 4 54 03 AM Chart
59. VBO Client TI General Advanced Driver Detais Power Management General Advanced Driver Detais Power Management Mi Broadcom BCM57712 NetXtreme Il 10 GigE NDIS Ms Broadcom BCM57712 NetXtreme Il 10 GigE NDIS VED Clent 76 ae VBD Chert 78 Device type Network adapters Device type Network adapters Location 4 PCI bus 6 device 0 function 2 Devicestaus B 4 57712 MP6 Port 0 Partition 4 7 7 7 20 Status ye EE UU TEE Bla Broadcom BCMS7712 NeXtreme Il 10 GigE NDIS Broadcom BCMS7712 Weiter ll 10 GigE NDIS Location Location 4 PCI bus 6 device D function 4 PCIE D 5 NL T MPI Port 1 Partition 1 K d Bi t 20 Status Sl 6 57 712 NP3 Port 1 Partition 27 4 4 4 20 Status BCOMS7712 NetXtreme IT 10 GigE NDIS VED Client 75 ES Broadcom BCMS7712 Netkireme O 10 GigE NDIS VED Chent KC Broadcom BCM57712 NetXtreme H 10 GigE NDIS ia Broadcom BCM57712 NetXtreme II 10 GigE NDIS A VED Gert 75 A VED Cert 74 Location Location 4 PCI bus 6 device 0 function 1 Location Location 4 PCI bus 6 device 0 function 3 Broadcom BCMS7712 NetXtreme D 10 GigE NDIS VBD Client 77 Ea Broadcom BCMS7712 NetXtreme D 10 GigE NDIS VBD Client 73 E a Broadcom BCM57712 NetXtreme H 10 GigE NDIS La Broadcom BCM57712 Netitreme Il 10 GigE NDIS k Go A Ces Location Location 4 PCI bus 6 device 0 function 5 BROADCOM Switch Indep
60. Views Hardware Status Hardware View vSphere Standard Switch vSphere Distributed Switch Processors Networkin 2 Security Traffic Shaping NIC Teaming Memory Storage m vSphere Standard Switch Properties SE Standard Switch vSwitch Remove Properties GE Number of Ports vSwitch3 Properties Oo x Changes will not take effect until the system is restarted Ports Network Adapters 120 X Configuration f vSwitch 120 Ports Number of Ports MTU vike rne vMotion and IP Virtual Machine Advanced Properties MTU r vSphere Standard Switch Properties Advanced Properties A m Default Policies Security Promiscuous Mode Reject MAC Address Changes Accept Forged Transmits Accept Traffic Shaping Average Bandwidth Peak Bandwidth Burst Size Failover and Load Balancing Load Balancing Port ID Network Failure Detection Link status only Notify Switches Yes Failback Yes Active Adapters vmnic6 cancel Hep 2 In the same vSwitch s Properties widow select VMKernel in the Ports Tab This displays the current NIC MTU size setting for that VMKernel of that vSwitch To change the size click the same Edit button under the Ports tab In the new window s General tab adjust the NIC Settings MTU value as desired BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 93 57712 k 578xx
61. WUM103 R Page 77 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning Add Hardware Select and Configure a Raw LUN Which LUN would you like to use for this raw disk TRT Name Identifier Path ID LUN or Capacity contains Clear elect d WIS Select Target LUN Name Path ID LUN Capacity Hart Select Datastore 3RDIO Fibre Channel OverEthernet vmhba43 CO TO lO O 2 00 TB Not Compatibility Mode 3RDIO Fibre Channel OverEthernet vmhba43 C0 T5 10 0 2 00TB Not necp eee 3RDIO Fibre Channel Over Ethernet vmhba43 C0 T4 L0 0 2 00TB Not DS 3RDIO Fibre Channel Over Ethernet vmhba43 CD T3 LD 0 2 00TB Not 3RDIO Fibre Channel Over Ethernet wmbhba43 C0 T2 L0 O 2 00 TB Not 3RDIO Fibre Channel Over Ethernet vwmhbad3 CO T7 L0 0 2 00 TB Not 3RDIO Fibre Channel Over Ethernet wmbhba43 CO T1i L0 O 2 00 TB Not 3RDIO Fibre Channel OverEthernet vmhba43 CU T8 LU 2 00 TB Not IET iSCSI Disk t10 9454450000000 vmhba38 CO TO lO 16 00MB Unk Details on Viewing Networking and Storage Performance on ESX 1 To view a port or partitions performance go to ESX ESXi 4 1 or ESXi 5 0 vSphere s Performance tab and open the Chart Options dhcp 10 13 126 210 broadcom com VMware ESXi 5 0 0 469512 Getting Started Summary Virtual Machines ResourceAllocation Performance Me Tee SEL Local Users amp Groups CPU Real time 4 5 2012 5 12 07 AM 4 5 2012 6 42 07 AM chart Options J Graph
62. a wide range of information including technical documentation schematic diagrams product bill of materials PCB layout information and software updates through its customer support portal https support broadcom com For a CSP account contact your Sales or Engineering support representative In addition Broadcom provides other product support through its Downloads amp Support site http www broadcom com support BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 13 57712 k 578xxS Software User Manual Technical Support BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 14 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning Configuring Switch Independent Partitioning Using the Unified Server Configurator Use Dell s Unified Server Configurator USC to configure Broadcom s device Switch Independent Partitioning parameters To configure Switch Independent Partitioning with the USC 1 Enter USC during system boot up by selecting the UEFI boot option See the Dell website for more information on USC Y DOLL LIFECYCLE CONTROLLER UNFIED SERVER CONFIGURATOR Help About Exit Lifecycle Log Platform Update Hardware Configuration OS Deployment Platform Restore Hardware Diagnostics LC Settings LIFECYCLE CONTROLLER 2 System Setup RELEASE 1 0 Warning A system update is recommended since so
63. addr feB8 218 1BTfT TeBB e7b5 64 Scope Link UP BROADCAST RUNNING MULTICAST MTu 9688 Metric 1 RX packets l181 errors B dropped overruns O frame TX packets errors O dropped O overruns carrier collisions 8 txquevelen 1600 RX bytes 33762 32 9 KiB TX bytes 80 8 b Interrupt 169 Memory d8500000 dB8TTTTTf Additionally the Ethernet and iSCSI protocol enabled adapter MTU sizes can be simultaneously adjusted in the Ethernet Devices window from the Network Configuration GUI if available in the desired version of Linux BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 87 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning Ethernet Device General Route Hardware Device Activate device when computer starts J AEW Dyers Toa enable ad disable tr device Enable Pvg configuration for this interface Automatically obtain P address settings with 2 Ska av set IP addresses Manual IP Address Settings lhddress 10 1 1 200 subnet mask Jas UAD Default gateway address 2 SetmTute se00 Set MAU to 0 r Note In Linux both the Ethernet and iSCSI Offload HBA MTU sizes are changed simultaneously and will have the same value In other words setting eth2 to MTU size of 9600 bytes using the ifconfig eth2 mtu 9600 up sets this example s Port O Partition 1 Ethernet adapter to MTU 9600 bytes and the Port O Partition 1 iSCSI Offload HB
64. al active weights it is not halved since it is the only traffic flow on partition P2 e When partition P3e starts sending Ethernet traffic at t3 with a relative weight of 30 it takes 5 Gbps 30 60 of 10 Gbps P2e drops to 3 3 Gbps 20 60 and partition P1 s total drops to 1 65 Gbps 10 60 so that means P1i and P1e each get half of that or 70 825 Gbps each e When P4e starts 40 relative weight at t4 it takes 4 Gbps 40 100 and the three other partition s send traffic drop partition P1 is reduced to 1 Gbps 10 100 so that means P1i and P1e split that for 0 5 Gbps partition P2 drops to 2 Gbps 20 100 and since there is only one send traffic flow P2e it takes all of that assigned bandwidth and finally partition P3 with it s single traffic flow P3e drops to 3 Gbps 30 100 e When the second traffic flow on partition P4 P4i starts at t5 the two flows P4e and P4i on the same partition P4 split the partition s assigned bandwidth of 4 Gbps so each gets 2 Gbps The other send traffic on the other three partitions remains the same e f P1i stops at t6 the remaining traffic flow on partition P1 P1e absorbs that partition s share of the send bandwidth to 1 Gbps The remaining traffic flows on the other three partitions are unaffected BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 105 57712 k 578xxS Software User Manual Examples e If Pie stops at t7 the others adjust sli
65. amic Discovery Static Discovery Send Targets Discover iSCSI targets dynamically from the following locations IPv4 host name iSCSI Server Location After completing these Discovery steps you are typically asked to refresh rescan the host bus adapter BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 65 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning e 10 13 126 48 vSphere Client File Edit View Inventory Administration Plug ins Help A Home gt gh Inventory p sg Inventory EJ e Tio 13 126 48 ESX iSCSLhost com VMware ESX 4 1 0 260247 Evaluation 58 days remaining o Getting Started Summary Virtual Machines ResourceAllocation Performance Configuration Local Ee 1 Permissions Hardware Storage Adapters Health Status Device Broadcom iSCSI Adapter CH ymhba32 i iqn 1998 01 com vmware iSCSI ESX 1237061279 33 bn CH vmhba34 i ign 1998 01 com vmware iSCSI ESX 1237061279 34 bn Q vmhba35 i iqn 1998 01 com vmware iSCSI ESX 1237061279 35 bn Processors Memory Storage Networking Storage Adapters Network Adapters Advanced Settings vmhba32 Power Management Model Broadcom iSCSI Adapter iSCSI Mame iqn 1998 01 com vmware ESX 4 1 iSCSI Host 720214776 32 Software iSCSI Alias bnx2i 001018442f3a Connected Targets 5 Devices 1 Paths 5 Licensed Features Time Configuration View Device
66. are User Manual Configuring Switch Independent Partitioning Each of these network connection devices can be accessed individually as if they where separate adapters The connection status shows the USC BACS4 Maximum Bandwidth setting as the Speed of the connection BI 1 57712 NPO Port 0 Partition 1 1 1 1 20 Status Fa 2 57712 nP2 Port 0 Partition 2 3 3 3 20 Status X EH m Duration 17 02 12 Se Activity o Bytes 91 756 308 698 7 455 871 680 584 H 3 57712 NP4 Port 0 Partition 3 5 5 5 20 Status fi 4 57712 NP6 Port 0 Partition 4 7 7 7 20 Status x General General BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 44 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning The previous Link Speeds are the result of the following USC Maximum Bandwidth settings and show its 1 of line rate configurable granularity L UNIFIED SERVER CONFIGURATOR LIFECYCLE CONTROLLER ENABLED Broadcom NetXtreme II 10 Gigabit Ethernet 00 10 18 85F D2 84 Global Bandwidth Allocation Menu Partition 1 Relative Bandwidth Weight 0 Partition 1 Maximum Bandwidth Ep Partition 2 Relative Bandwidth Height Partition 2 Maximum Bandwidth 1 Partition 3 Relative Bandwidth Height Partition 3 Maximum Bandwidth ER Partition 4 Relative Bandwidth Height 0 Partition 4 Maximum Bandwidth 46 Configure maximum bandwidth Valid range 1
67. aths 15 Runtime Name Target 20 03 00 11 0d 13 51 06 20 03 00 11 0d 13 51 06 Virtual Machine Startup Shutdown Virtual Machine Swapfile Location Security Profile Host Cache Configuration System Resource Allocation Agent VM Settings Advanced Settings vmhba35 C0 T5 LU vmhba35 C0 T14 L0 vmhba35 CU TU LU vmhba35 CU T5 L0 vmhba35 C0 T13 L vmhba35 CU T4 LO vmhba35 C0 T9 LOU vmhba35 C0 T12 L vmhba35 CU T3 LU vmhba35 C0 T8 L0 wmhba35 C0 T11 L0 vmhba35 C0 T2 LO vmhba35 C0 T7 LU vmhba35 C0 T10 L vmhba35 C0 T15 L 20 03 00 11 0d 13 51 01 20 03 00 11 0d 13 51 01 20 01 00 11 0d 11 2d 01 20 01 00 11 0d 11 2d 01 20 03 00 11 0d 13 51 07 20 03 00 11 0d 13 51 07 20 03 00 11 0d 13 51 02 20 03 00 11 0d 13 51 02 20 01 00 11 0d 11 2d 02 20 01 00 11 0d 11 2d 02 20 03 00 11 0d 13 51 03 20 03 00 11 0d 13 51 03 20 01 00 11 0d 11 2d 06 20 01 00 11 0d 11 2d 06 20 01 00 11 0d 11 2d 03 20 01 00 11 0d 11 2d 03 20 03 00 11 0d 13 51 04 20 03 00 11 0d 13 51 04 20 01 00 11 0d 11 2d 07 20 01 00 11 0d 11 2d 07 20 01 00 11 0d 11 2d 04 20 01 00 11 0d 11 2d 04 20 03 00 11 0d 13 51 05 20 03 00 11 0d 13 51 05 20 03 00 11 0d 13 51 00 20 03 00 11 0d 13 51 00 20 01 00 11 0d 11 2d 05 20 01 00 11 0d 11 2d 05 20 01 00 11 0d 11 2d 00 20 01 00 11 0d 11 2d 00 EN E EN EN E EN EN e EN EN e E EN EN ON 6 6 Active 1 0 Active 1 0 Active 1 0 Active
68. b Interrupt 264 Memory d4860000 d4TTtttt Link encap Ethernet HWaddr 06 10 18 88 E7 66 inet addr 18 2 2 200 Bcast 16 255 255 255 Mask 255 6 6 6 inet addr fTe80 210 18T7TT Te88 e7b0 64 Scope Link UP BROADCAST RUNNING MULTICAST J 5588g Metric Ax packets 371 errors dropped overruns 8 frame 6 TX packets 6 errors O dropped overruns G carrier collisions 8 txqueuelen l1888 RX bytes 7213B 70 4 KiB TX bytes B 8 8 b Interrupt 2804 Memory d58068000 d5fTTTTTT Link encap Ethernet HWaddr 88 10 IB BB E7 B2 inet addr 10 2 7 201 8cast 16 255 255 255 Mask 255 6 6 6 inete addr Te80 210 18T77T T6e88 e7b2 64 Scope Link UP BROADCAST RUNNING MULTICAST gMIU 7 d80 gy Metric l RX packets 336 errors O dropped overruns frame 6 TA packets 0 errors O dropped 0 overruns O carrier 8 collisions B txquevelen 1000 RX bytes 63944 62 4 KiB TX bytes O 6 6 b Interrupt 181 Memory d6800000 dG6TTTTTT Link encap Ethernet HWaddr 08 18 18 88 E7 B4 inet addr 18 7 2 24787 Beast 168 755 255 255 Mask 255 68 68 8 inet6 addr fe860 210 1811 Te88 e7b4 cope Link UP BROADCAST RUNNING MULTICAST Metric l RX packets 316 errors 8 dropped 8 overruns B Trame 6 TX packets errors B dropped overruns B carrier collisions txquevelen 1666 RX bytes 60760 59 3 KiB TX bytes 6 0 0 b Interrupt 181 Memory d7800000 d7TTTTTT Link encap Ethernet HwWaddr 060 180 18 88 E7 B6 inet addr 10 2 2 203 Beast 16 2535 255 255 M Mask 255 0 80 0 inet6
69. bps which leaves 9 Gbps free for the other three traffic flows The two other partition s traffic flows are allocated 3 Gbps each 9 Gbps 3 equally weighted traffic flows the total bandwidth for P1i and P1e is 6 Gbps which is less than partition P1 s maximum of 80 of 10 Gbps e When P4i starts to send traffic at t5 the bandwidth is again redistributed P2e and P3e are still using only 0 5 Gbps 5 100 This again leaves 9 Gbps free for the remaining four equally weighted traffic flows therefore these four P1i P1e P4e and P4i all are allocated 2 25 Gbps each 9 Gbps 4 flows where P1i BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 115 57712 k 578xxS Software User Manual Examples plus P1e and P4e plus P4i totals are 4 5 Gbps each which is less than their respective partition s maximum bandwidth settings e When P1i stops at t6 it releases it s bandwidth to the available pool and since bie and P3e are capped by their relative bandwidth weight values to 0 5 Gbps the three other traffic flows P1e P4e and P4i automatically take equal shares of the remaining bandwidth and bump up their portion to 3 Gbps each where P4e plus P4i total is 6 Gbps which is still less than their respective partition s maximum bandwidth value e When Pie subsequently stops sending at t7 P4e and P4i grab up some of the extra available bandwidth and go to 4 Gbps each where they reach their partition s maxim
70. c d 0054 Broadcom HU n 0016 Broadc 0 0031 Broadc Figure 10 Broadcom Advanced Control Suite Adapter Settings After the devices are installed the enabled devices L2 Ethernet NDIS FCoE and iSCSI will be visible in the Windows Device Manager and BACSA The following is the 57712 k s Device Manager s display in SF mode see Figure 11 The 578xx devices are displayed similarly Device Hanager S S Intel R 82801 PCI Bridge 244E E Intel R processor DMI D130 Ba Intel R processor Miscellaneous Registers D158 EH n Intel R processor PCI Express Root Port 1 D138 i B BR Broadcom BCM57712 NetXtreme II 10 GigE 41 KP Broadcom BCM57712 Netxtreme II 10 GigE NDIS VBD Client 41 be EG Broadcom BCM57712 Netxtreme II 10 GigE FCoE Adapter 41 es Broadcom BCM57712 NetXtreme II 10 GigE iSCSI Adapter 41 E P Broadcom BCM57712 Netxtreme II 10 GigE 42 SP Broadcom BCM57712 Netxtreme II 10 GigE NDIS VBD Client 42 e Broadcom BCM57712 NetXtreme II 10 GigE FCoE Adapter 42 lt gt Broadcom BCM57712 Netxtreme II 10 GigE iSCSI Adapter 42 Figure 11 Windows Device Manager BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 30 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning Viewing the Enabled Devices in Device Manager Windows shows all of the enabled devices in Device Manager with the respective USC BACS4 enabled Switch Independent Par
71. ce boot up delays when there are remote iSCSI target connections present over either offloaded hardware or non offload software pathways See http kb vmware com selfservice microsites search do language en_US amp cmd displayKC amp externalld 2007108 for a description of the issue and workarounds Details on Configuring FCoE Offload on ESXi 5 0 After creating the desired networking vSwitches you need to associate the Broadcom 57712 578xx FCoE hardware offload HBA vmhba devices to a previously created vSwitch vmk vmnic This can be done two ways via the command line in SSH or vSphere To use vSphere 1 In ESXi 5 0 vSphere s Storage Adapters Configuration page click Add in the upper right corner of the window BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 2 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning dhcp 10 13 126 210 broadcom com VMware ESXi 5 0 0 469512 Getting Started Summary Virtual Machines ResourceAllocation Performance Configuration Local Users amp Groups Events Permissions onware Storage Adapters Remove Refresh Rescan All Health Status Processors Device Broadcom iSCSI Adapter ign 1998 01 com vmware dhcp 10 13 126 210 broadcom com 2145997461 33 bnx2i 00101888e5ee CH vmhba33 i iqn 1998 01 com vmware dhcp 10 13 126 210 broadcom com 2145997461 34 bnx2i 00101888e5f0 Q vmhba34 i iqn 1998 01 com vmwa
72. cksum Offload IPv4 ranami Bulters If you are using the VMXNET3 Virtual Adapter then change it s Advanced Properties Jumbo Packets setting to Jumbo 9000 BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 90 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning vmxnet3 Ethernet Adapter Properties General Advanced Driver Details Power Management a aaa In a RHEL VM the virtual adapter MTU sizes can be adjusted in the Ethernet Devices window from the Network Configuration GUI or command line using the ifconfig ethX mtu NNNN up command Ethernet Device General Route Hardware Device Nickname 7712 ethz2 Activate device when computer starts gt Allow all ysers to enable and disable the device Enable Pv configuration for this interface Automatically obtain IP address settings with Statically set IP addresses Manual IP Address Settings Address 111 12 subnet mask 255 255 0 0 Default gateway address 7 Set MTU t eon Set MRU t 0 In Linux use the following ping command to verify that Jumbo Frame support is configured throughout the desired network path BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 91 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning r Note The odd numbering for the ping packet size s NNNN which is the desired MTU
73. dth demands exceed the available bandwidth e Starting at tO only P1f FCoE Offload is sending so it takes 100 or all of the 10 Gbps bandwidth e When Pie L2 Ethernet starts to send at t1 both flows stabilize to 5 Gbps each P1f in PG1 takes 5096 and P1e in PGO takes the other 50 e When P2e starts to send at t2 the traffic in P1f is not affected it remains at 5 Gbps due to it being in a different PG Both P1e and P2e will stabilize to 2 5 Gbps each P1e and P2e equally share PGO s portion of the bandwidth so they each get 2 5 Gbps ETS of 50 total 10G BW 1 2 e When P3e starts to send at t3 P1f is still unaffected and remains at 5 Gbps The three L2 Ethernet traffic types will split their 5096 of PGO s share between themselves which is 1 65 Gbps each takes 1 3rd of 5G e When P4e starts to send at t4 the four Ethernet traffic flows take 1 Ath of PGO s bandwidth 71 25 Gbps each while P1f is still unaffected and remains at 5 Gbps e When P4i starts to send at t5 the four Ethernet traffic flows plus the new iSCSI traffic flow take 1 5th of PGO s bandwidth 1 Gbps each while P1f is still unaffected and remains at 5 Gbps e Then when P1f stops sending at t6 the five traffic flows in PGO now take all of the ports bandwidth so now their 1 5th of PGO s bandwidth doubles to 2 Gbps each the available bandwidth went from 5 Gbps to 10 Gbps e Aseach traffic flow stops sending in PGO the remaining member traffic
74. e traffic flows on the same partition share its 8 Gbps bandwidth for 4 Gbps each ETS does not take effect since the traffic in PGO and PG1 are still less than the amount prescribed by their respective ETS values e When the third traffic flow P2e starts sending at t2 it only takes its partitions maximum bandwidth allowed which is 1 Gbps Partition P1 s two traffic flows are unaffected and the unassigned 1 Gbps bandwidth remains free e When the fourth traffic flow P3e starts sending at t3 it only takes its partitions maximum bandwidth allowed which is 1 Gbps Now the first partition s two traffic flows readjust so that PG O does not get more than 50 of the overall bandwidth i e PGO s Ple P2e P3e 4096109641096 which is greater than 5096 The P1e traffic flow is reduced to 30 or 3 Gbps and the P1f traffic flow in PG1 is adjusted upwards to 50 or 5 Gbps e When P4e starts to send traffic at t4 it equally shares PGO s ETS assigned bandwidth with P1e Pie and P3e but since P2e and P3e use only 1 Gbps of their allocated 1 25 Gbps 5 Gbps 4 equally weighted traffic flows this leaves 3 Gbps free 5 Gbps available 2 GBps assigned to P2e and P3e for the other two traffic flows P1e and P4e and they are both allocated 1 5 Gbps each 3 Gbps 2 equally weighted traffic flows P1f is in PG1 so it is unaffected and keeps sending at 5 Gbps e When P4i starts to send traffic at t5 it also equally shares PGO s bandwidth 5 Gbps 5
75. eme TI 10 GigE Mul Function sm ES 0012 Broadcom BCMS7712 NetXtreme II 10 GigE Multi Function SCSI Adap Porti L114 3 ign 20 D0 DU com breavdcom bot 15a ram 15a J an 2010 10 com braadoom tgt Stra 15b ep 2011 05 com broadoom tgt 1a rae 14a Clem z011 05 com broadoomctgt L4b ram 14b ez Portal fe0 210 18 ff fee f d3cs OM 0027 Broadcom BCMS7712 Netxtreme II 10 GigE NDIS VED Chert 50 7 0051 Broadcom BCM57712 NetX trerme IT 10 Gage Multi Function 51 E PE 053 Broadcnm BCM57712 Netitreme TI 10 GigE Multi Function 53 af 0030 Broadcom BCMS7712 Netktreme IT 10 GE NDIS VED Cent 53 e D 0054 Broadcom BCIAS7712 NetXtreme D 10 Gg Mul Function 254 Sa 90 26 Broadcom BCMS7712 Nettineme IE 10 GigE Mule Function Poot Adap Ge Bi Port 20 00 00 10 18 6F D3 8D 20 H E H BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 40 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning The per partition iSCSI Offload device s VLAN MTU and IP address IPv4 and IPv6 settings are configured in BACS4 as shown below Broadcos m Adv ET ced Control Suite A E E Adapter wens ca S ES Adapter BCMS205 CH Si ES Adapter3 BCM57711 A9 EF EE Adapters 57712 A1 i E 4 Port res DA mune 255 255 0 0 o FIT 0050 Broadcom BCM57712 Netxtreme D 20 GE Mult Function 250 aa E 0022 Broadcom BCMS7712 NetXtreme TI
76. en thought they are in the same partition they share the total available bandwidth When P2e starts to send at t2 all three traffic flows P1i Ple and P2e will stabilize to 1 3rd of the bandwidth or 3 3 Gbps each they all equally share the available bandwidth When P3e starts to send at t3 all four traffic flows P1i P1e P2e and P3e will stabilize to 1 Ath of the bandwidth or 2 5 Gbps each effectively sharing the available bandwidth When P4e starts to send at t4 all five traffic flows P1i P1e P2e P3e and P4e will stabilize to 1 5th of the bandwidth or 2 Gbps each again equally sharing the available bandwidth When P4i starts to send at t5 all six traffic flows P1i P1e P2e P3e P4e and P4i will stabilize to 1 6th of the bandwidth or 1 65 Gbps each all sharing the available bandwidth When P1i stops sending at t6 the five currently active traffic flows P1e bie Pie P4e and P4i will readjust to 2 Gbps each equally absorbing the freed up bandwidth As a previously sending traffic flow stops sending t7 t8 t9 and t10 the remaining active flows will readjust to equally fill any available bandwidth Notice the symmetry of the BW allocation No matter which traffic type is currently running each will get an equal share with respect to the other currently transmitting traffic type flows This assumes the application creating the transmitted traffic type flow can fill the allocated amount of BW it is given if not the
77. endent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 51 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning Red Hat Enterprise Linux v6 x Linux shows the respective device protocols that were enabled in USC if the NetXtreme II device drivers are installed Go to the Dell Driver Download web site for the latest Linux device drivers software version 17 2 x or later and firmware version 7 2 x or later and ensure they are installed on your system In Linux the Ethernet Protocol is always enabled on all eight partitions and FCoE Offload HBA Protocol can be enabled on any single partition of each port and the iSCSI Offload HBA Protocol can be enabled on any remaining single partition of that port OR if the FCoE protocol is not enabled then on any two partitions of each port In newer versions of Linux the ethtool s ethX speed NNNNN command supports setting the 57712 k 578xxS partition s Maximum Bandwidth value without a system reboot where ethX is the partition and NNNNN is the speed which is in 1 Mbps increments i e itis NOT a percentage e Fora 10GbE link connection the range is from 100 which is equivalent to 100 Mbps or 196 of the 10GbE link speed to 10000 which is equivalent to 10 Gbps or 100 of the 10GbE link speed the last two zero positions are not used e Fora 1GbeE link connection the range is from 10 which is equivalent to 10 Mbps or 196 of the 1GbE link speed to 100
78. ending so at this point P4i gets 100 of all the bandwidth or 10 Gbps BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 99 57712 k 578xxS Software User Manual Examples This second DCB example of oversubscribed bandwidth sharing replaces one of the iSCSI Offloads with an FCoE Offload protocol This gives a total of three distinct traffic types in six flows with similar partition settings The other difference is that the FCoE traffic type is assigned to DCB Priority Group 1 and is Lossless with an ETS setting of 50 while the L2 Ethernet and iSCSI Hardware Offload traffic types are both now in Priority Group O and are both Lossy with an ETS setting of 50 Table 6 DCB Equal Oversubscription with one Lossless FCoE Offload Relative Bandwidth Maximum Port Partition Weight Bandwidth Protocol Plot Color Port O Partition 1 P1f N A 100 FCoE Offload Green Port O Partition 1 P1e N A 100 Ethernet with Brown TOE Port O Partition 2 P2e N A 100 Ethernet Purple Port O Partition 3 P3e N A 100 Ethernet Yellow Port O Partition 4 P4e N A 100 Ethernet with Blue TOE Port O Partition 4 P4i N A 100 iSCSI Offload Red The following plot shows how the first partition s FCoE traffic PG1 acts verses the other traffic types in PGO Just like the previous example the traffic in the two PGs will almost act independently of each other when their aggregated traffic bandwi
79. ent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 110 57712 k 578xxS Software User Manual Examples Mixed Fixed Subscription and Oversubscription Example This example shows partitions with all of the relative bandwidth weights set the same but with the partitions partially oversubscribing the available bandwidth unequally Two of the partitions are set to use 10 or 1 Gbps each of bandwidth and the other two the ones with the hardware offload protocol s enabled are set to use 80 or 8 Gbps of the connection s bandwidth thus oversubscribing the connection by 80 Table 11 Non DCB Mixed Fixed Subscription and Oversubscription Relative Bandwidth Maximum Port Partition Weight Bandwidth Protocol Plot Color Port O Partition 1 P1i 0 80 iSCSI Offload Green Port O Partition 1 P1e 0 80 Ethernet Orange Port O Partition 2 P2e 0 10 Ethernet Purple Port O Partition 3 P3e O 10 Ethernet Yellow Port O Partition 4 P4e 0 80 Ethernet Blue Port O Partition 4 P4i 0 80 iSCSI Offload Red This is a combination example of a fixed subscription three of the partitions sum to 100 but all four sum to 180 When all four or at least the two larger partitions are running traffic they share the space with each other up to their partition maximum bandwidth values otherwise they act as if they are independent connections e The first partition s traffic flow P1i initially takes its designated 8
80. ent Partitioning In Windows use the ping command with the f option to set the Don t Fragment DF flag AND the I size option small to verify that Jumbo Frame support is configured throughout the desired network path i e ping f 8972 A B C D The unfragmentable ping packet size is the desired MTU size to be checked 9000 bytes minus the automatically added overhead 28 bytes or 8972 bytes C gt ping f 1 8972 192 168 20 10 Pinging 192 168 20 10 from 192 168 20 50 with 8972 bytes of data Reply from 192 168 20 10 bytes 8972 time lt ims TTL 64 Reply from 192 168 20 10 bytes 8972 time 1ms TTL 64 Reply from 192 168 20 10 bytes 8972 time lt ims TTL 64 Reply from 192 168 20 10 bytes 8972 time lt ims TTL 64 Ping statistics for 192 168 20 10 Packets Sent 4 Received 4 Lost 0 0 loss Approximate round trip times in milli seconds Minimum Oms Maximum Oms Average Oms If it does not work you might see the following reply if there is connectivity try 1472 byte standard frames to see if the non jumbo frame size is passing through C gt ping f 1 8972 192 168 20 10 Pinging 192 168 20 10 from 192 168 20 50 with 8972 bytes of data Packet needs to be fragmented but DF set Packet needs to be fragmented but DF set Packet needs to be fragmented but DF set Packet needs to be fragmented but DF set Ping statistics for 192 168 20 10 Packets Sent 4 Received 0 Lost 4 100 loss
81. equally weighted traffic flows which means P1e Pie Pie P4e and P4i all send at 1 Gbps P1f in PG1 is still unaffected and keeps sending at 5 Gbps e When P1f stops at t6 it releases all of PG1 s bandwidth to the available pool and since P2e and P3e are capped by their maximum bandwidth value to 1 Gbps the three other traffic flows P1e P4e and P4i automatically take equal shares of 8 Gbps and bump up their bandwidth used to 2 6 Gbps each e When Pie subsequently stops sending at t7 P4e and P4i grab up the extra available bandwidth and go to 4 Gbps each Both P2e and P3e are unaffected and continue sending at 1 Gbps each e When P2e stops sending at t8 P4e and P4i are not able to make use of the freed up bandwidth since they are both in partition P4 which has a maximum bandwidth ceiling of 8 Gbps Therefore none of the traffic BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 113 57712 k 578xxS Software User Manual Examples flows increase their sending rates and the unused bandwidth is ignored e When P3e stops sending at t9 the same condition is still in effect Therefore none of the remaining active traffic flows increase their sending rates to use this extra bandwidth e Finally P4e stops at t10 and this allows it s companion traffic flow P4i to increase to 8 Gbps which is partition PA s maximum top end The remaining 2 Gbps is unassigned BROADCOM Switch Independent Part
82. er gema Q vmhbas7 Fibre Channel over Ethernet 10 00 00 10 18 88 ee ed 20 00 00 10 18 88 ee ed SF Storage Broadcom iSCSI Adapter Networking Q vmhba32 iSCSI iqn 1998 01 com vmware dhcp 10 13 126 210 broadcom com 2145997461 33 bnx2i 00101888e5ee Q vmhba33 iSCSI ign 1998 01 com vmware dhcp 10 13 126 210 broadcom com 2145997461 34 bnx2i 00101888e5f0 Q vmhba34 iSCSI ign 1998 01 com vmware dhcp 10 13 126 210 broadcom com 2145997461 35 bnx2i 00101888e5f2 Health Status Storage Adapters Network Adapters Advanced Settings Details Power Management vmhba37 Properties Kaes een Model Broadcom Corporation FCoE Adapter WWN 10 00 00 10 18 88 ee ed 20 00 00 10 18 88 ee ed You can also use the command line command and an SSH connection to the ESXi 5 0 host to list the current FCoE offload ports the ports may be inactive Active false at first 1 Enter the following command esxcli fcoe nic list vmnic User Priority 3 Source MAC XX XX XX XX XX XX Active false Priority Settable false Source MAC Settable false VLAN Range Settable false 2 Usethe following command to enable discovery activate on the selected FCoE ports BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 74 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning H esxcli fcoe nic discover n vmnic 3 Rerun the previous FCoE list command from Step 1 and yo
83. fff kernel bnx2fc 06 FCOE INIT passed kernel bnx2fc bnx2fc started For iSCSI also check the iscsi host folder to see if your iSCSI devices are present root localhost cd sys class iscsi host root localhost iscsi _host pwd sys class iscsi host root localhost iscsi host 11 total lrwxrwxrwx 1 root root 0 Jun 29 11 15 host3 gt devices pci0000 00 0000 00 09 0 0000 07 00 1 host3 iscsi host host3 lrwxrwxrwx 1 root root 0 Jun 29 11 15 host4 gt devices pci0000 00 0000 00 09 0 0000 07 00 0 host4 iscsi host host4 Each installed iSCSI device will appear here The iSCSI devices uses the bnx2i driver which can be checked for with the 1smod grep bnx2i command Additionally check the fc host folder to see if your FCoE devices are present root localhost cd sys class fc host root localhost fc_host pwd sys class fc_host root localhost fc_host 11 total 0 lrwxrwxrwx 1 root root 0 Jun 29 11 11 host8 devices pci0000 00 0000 00 07 0 0000 05 00 0 host5 fc host host5 lrwxrwxrwx 1 root root 0 Jun 29 11 11 host8 gt devices pci0000 00 0000 00 07 0 0000 06 00 0 host6 fc_host host6 Each installed FCoE device will appear here FCoE uses the bnx2fc driver which can be checked for with the lsmod grep bnx2fc command Another useful command is sg map i x which will show all SCSI LUN devices visible to the host For Fiber Channel another useful application is FCInfo utilit
84. figuring Switch Independent Partitioning four partitions on an individual port simultaneously with any enabled offload protocols e For Linux OSs the Ethernet protocol will always be enabled even if disabled in USC e For simplicity we recommend always using the first two partitions of a port for any iSCSI offload protocols e For Windows OSs the Ethernet protocol does not have to be enabled for the iSCSI or FCoE offload protocol to be enabled and used on a specific partition e For VMWare ESX ESXi 4 1 in Switch Independent Partitioning mode the host and hosted Virtual Machines VMs can only connect to enabled Ethernet and iSCSI Offload protocol partitions e For VMWare ESXi 5 0 in both SF and Switch Independent Partitioning modes the host and hosted VMs can connect to the enabled Ethernet and iSCSI Offload and FCoE Offload protocol enabled partitions BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 25 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning Supported Operating Systems The 57712 k 578xxS device SF and Switch Independent Partitioning mode supported operating systems are shown in Table 1 r Note The drivers may not be in the box Table 1 Protocols Available in Operating Systems Versus SF and Switch Independent Partitioning Modes SF Mode Switch Independent Partitioning Mode Operating System Ethernet iSCSI Offload FCoE Offload Ethernet
85. flow for determining the amount of bandwidth allocated to them e Starting at tO the first partition s iSCSI Offload traffic flow P1i initially takes 10096 of the available port s TX send bandwidth when an iSCSI test application is flooding that port by itself e When P1 s L2 Ethernet P1e starts to send at t1 both stabilize to 5 Gbps each even thought they are in the same partition they share the total available bandwidth This is because no other partition s traffic flow is sending e When P2e starts to send at t2 the traffic flows in P1 P1i and P1e will reduce to 2 5 Gbps each while the P2 traffic flow P2e will take 5 Gbps This is because the bandwidth is initially split by partition and then traffic flows within each individual partition e When P3e starts to send at t3 the two traffic flows in P1 P1i and P1e are further reduced to 1 65 Gbps half of the partition s 1 3rd allocation going to the three active partitions while P2e and P3e each stabilize at 3 3 Gbps e When P4e starts to send at t4 the three single partition traffic flows P2e P3e and P4e will stabilize to 1 4th of 10 Gbps or 2 5 Gbps each while the P1 partition shares it allocated bandwidth between it s two users P1i and P1e so each gets half of the allocated 1 4th of 10 Gbps or 1 8th which is 1 25 Gbps e When P4i starts to send at t5 the two single traffic flows P2e and P3e will remain at 2 5 Gbps each as well as the P1 partition s traffic f
86. flows equally increase their respective shares to automatically occupy all of the available bandwidth At t7 there are four active PGO flows so each gets 1Ath or 2 5 Gbps At t8 there are three active PGO flows so each gets 1 3rd or 3 3 Gbps At t9 there are two active PGO flows so each gets half or 5 Gbps Finally at t10 there is only one active PGO flow P4i so it gets 10096 or 10 Gbps e Any ofthe traffic flows will take 10096 of the available bandwidth if it is the only sending traffic flow BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 100 Examples 57712 k 578xxS Software User Manual Page 101 SS mem 12 UM 0M z em m E ee Se a m o Aem o 8 83 7 NN D Switch Independent Partioning Setup Guide ees LO 0 0 5 3 0 Mo o E E aM BM E BM s ju m MI M I D i m LSU E E E E E F WR WER me fe ne ee D AE NM mm BH o ee se ee ee S 2 02 l1 L RN S AE NS ms y Emm c A B B B ll S E E MN M o SE ten ees emm eg M DM aM M NM M NM BM M June 07 2012 e 2CS57712 SWUM103 R BROADCOM 57712 k 578xxS Software User Manual Examples Partitioned Oversubscription Example The following is an example of oversubscribed bandwidth sharing in non DCB mode where all four partitions of the port have their weight set to 25 and can individually use the maximum bandwidth of the connection e 10 Gbps in this case In additio
87. ghtly upwards to fill that newly available bandwidth partition P2 and it s traffic flow P2e increases to 2 2 Gbps 20 90 partition P3 and it s traffic flow P3e raises to 3 3 Gbps 30 90 and partition P4 raises to 4 5 Gbps 40 90 which means P4e and P4i split that for 2 25 Gbps each e f P2e stops at t8 the others again adjust upwards to fill that newly available bandwidth partition P3 and it s traffic flow P3e raises to 4 3 Gbps 30 70 and partition P4 raises to 5 7 Gbps 40 70 which means P4e and P4i split that for 2 85 Gbps each e f P3e stops at t9 partition P4 s share raises to 10096 of the bandwidth 40 40 so it s two traffic flows P4e and P4i split this for 5 Gbps each e When P4e stops at t10 the only remaining traffic flow P4i takes all of the bandwidth at 10 Gbps BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 106 57712 k 578xxS Software User Manual Examples Oversubscription With One High Priority Partition Example Next is an example of a single high priority partition with almost all the relative bandwidth weight but all four of the partitions are still oversubscribing the available bandwidth The first three partitions of the port have 1 weight and the last partition P4 has almost all the weight 97 All four partitions are set to use the maximum amount of the connection s bandwidth i e 100 which is 10 Gbps in this case Again since the
88. he dynamic weight ratio as a variable sized funnel that could be further restricted by the Maximum Bandwidth fixed sized funnel with the OS determining how the sharing traffic types are pouring into the combined funnels n non DCB mode where the sum of the partition s Relative Bandwidth Weights equals zeros i e each partition s Relative Bandwidth Weight is set to zero each individual traffic flow i e Ethernet or iSCSI Offload or FCoE Offload is equally scheduled to transmit within the limitations of the partition s Maximum Bandwidth and the overall connection s link speed This means if the Maximum Bandwidth of a specific partition is set to less than 10096 then the traffic flows sharing that partition will be further restricted to where their combined traffic flow bandwidth will be capped by that per partition setting If all four partition s individual Maximum Bandwidths are set to 100 i e they are unrestricted then each actively sending traffic flow without regard to which partition they are on will equally share the transmit directions total bandwidth i e TX link speed The actual inter partition ratio of the two sharing traffic flows is controlled by the host OS Think of the Maximum Bandwidth as a fixed sized funnel with the OS determining how the two sharing traffic types are pouring into that funnel n DCB mode all of the partition s Relative Bandwidth Weights are disregarded and the individual traffic flows are schedu
89. hine Port Group LJ VMK NIC 3 B 1 virtual machine s VM41_P1_2 lira De Sue Configuration mere Ber ET Ve en E Remove Properbes Physical Adapters EH vmnic 1000 Full Li Remove Properties Biysca Adapers sa vmnice 10000 Full L2 Remove Properties Pineal Adames E MB vmnic 10000 Full Remove Properties Phy scil Adapers Ng vmnic8 10000 Full LJ Switch Independent Partioning Setup Guide Page 59 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning VMWare s vSphere 4 1 or vCenter can be used to view a selected VM s Networking performance in the specific Host s Performance subtab selecting In the first example the first port s four partitions on VMNICO 8 10 12 are set to 096 Relative Bandwidth Weight and 10096 Maximum Bandwidth each and the second port s partitions on VMNIC7 9 11 13 are similarly set to 096 Relative Bandwidth Weight but the Maximum Bandwidth values are set to 1096 2096 3096 4096 respectively which results in VMNIC7 s link speed for the transmit direction only indicating 1000 Mbps VMNIC9 s link speed indicating 2000 Mbps VMNIC11 s link speed indicating 3000 Mbps and finally VMNIC13 s link speed indicating 4000 Mbps This is indicated in the vSphere Host s Configuration Network Adapter page The first port s network performance indicated in the vSphere Host s Performance page shows each individual VM s send traffic rate when no
90. hree of the partitions sum to 100 and oversubscription all four sum to 180 with different weights and maximum bandwidths When all four or at least the two larger partitions are running traffic they share the space with each other with respect to their partition s weight and maximum bandwidth values otherwise the partition s continue to act as if they are independent connections e The first partition s traffic flow P1i initially takes its designated 8 Gbps when the test application starts to transmit traffic at tO to that port by itself not expanding into the remaining unused 2 Gbps bandwidth e When the second traffic flow on the first partition P1e starts to send at t1 the two active traffic flows on the same partition share its 8 Gbps bandwidth for 4 Gbps each e When the third traffic flow P2e starts sending at t2 it only takes its partitions maximum bandwidth allowed 1 Gbps Partition P1 s two traffic flows are unaffected e When the fourth traffic flow P3e starts sending at t3 it again only takes its partitions maximum bandwidth allowed 1 Gbps Partition P1 s two traffic flows P1i and P1e and the traffic flow on partition P2 P2e are unaffected e But when P4e starts to send traffic at t4 the traffic needs are oversubscribed so the available bandwidth is redistributed based on each partition s individual weights and maximums settings P2e and P3e use 596 each 5 100 so their traffic flows are reduced to 0 5 G
91. ics 3 packets transmitted packets received 100 packet lost You should also check if the desired VMs can send Jumbo Frames all the way to their various end points using the applicable OS ping commands described earlier in this document Setting MTU Sizes in VMWare ESXi 5 0 In VMWare ESXi 5 0 the MTU size for each individual Ethernet protocol enabled partition can be independently set from Normal 1500 bytes up to Jumbo 9600 bytes This MTU size change affects both regular L2 Ethernet and iSCSI software non offload pathway generated traffic Unlike other Linux OS s you can not directly adjust the MTU size using the ifconfig command Instead you can use various ESX 4 1 esxcfg commands or change them via ESXi 5 0 s vSphere 1 In ESXi 5 0 s vSphere enter the desired vSwitch s Properties and select vSwitch in the Ports Tab This displays the current MTU size setting for that vSwitch To change the size click the Edit button under the Ports tab In the new window s General tab adjust the Advanced Properties MTU value as desired and click OK BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 92 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning bd1 11 swdvt lab irv broadcom com VMware ESXi 5 0 0 469512 Getting Started Summary Virtual Machines Resource Allocation Performance Configuration Tasks amp Events Alarms Permissions Maps Storage
92. ioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 114 57712 k 578xxS Software User Manual Examples Mixed Weights and Subscriptions Example This example shows partitions with different relative bandwidth weights and maximum bandwidths but with the same partitions partially oversubscribing of the available bandwidth as the previous example The first pair of partitions are set to use 10 or 1 Gbps each of bandwidth and both of their weights are set to 5 while the second pair of partitions are set to use 80 or 8 Gbps of the connection s bandwidth each with both of their relative bandwidth weights set to 45 The total is still oversubscribing the connection by 80 In DCB mode there would be no difference between the previous DCB mode example and this one since the Relative Bandwidth Weights are not applicable in DCB mode and also if all of the traffic types are in the same Priority Group the results would be similar Table 13 Non DCB Mixed Fixed Subscription and Oversubscription Relative Bandwidth Maximum Port Partition Weight Bandwidth Protocol Plot Color Port O Partition 1 P1i 45 80 iSCSI Offload Green Port O Partition 1 P1e 45 80 Ethernet Orange Port O Partition 2 P2e 5 10 Ethernet Purple Port O Partition 3 P3e 5 10 Ethernet Yellow Port O Partition 4 P4e 45 80 Ethernet Blue Port O Partition 4 P4i 45 80 iSCSI Offload Red This is a combination example of a fixed subscription t
93. ition s sending traffic flows e When P4i starts to send at t5 it will share partition P4 s 25 with P4e Each is allocated 1 25 Gbps and the other partitions are unaffected e When P1i stops sending at t6 it will release it s 12 5 share of the bandwidth and the other remaining partition P1 traffic flow P1e will increase to 2 5 Gbps while the other traffic flows are unaffected e When P1e stops sending at t7 there will only be three partitions but each is still assigned only 2596 of the overall bandwidth The other traffic flows P2e bie P4e and P4i will not change e When P2e stops sending at t8 again there will be no change to the other traffic flows e When P3e stops sending at t9 there will still be no change to the other traffic flows BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 109 57712 k 578xxS Software User Manual Examples e When P4e stops sending at t10 the remaining traffic flow on P4 P4i will absorb the freed 12 5 of the partition P4 s allocated bandwidth and will increase to 2 5 Gbps e Each partition s flows on the same port are logically isolated from the others as if they were on separate ports and a partition s send flows stopping or restarting will not affect its fellow partition s send traffic flows except where the flows are on the same partition and then they will only take the freed bandwidth for their respective partition BROADCOM Switch Independ
94. k that the traffic flows though must also support the desired MTU sizes for that sized frames to be used without being dropped truncated fragmented by the network Setting MTU Sizes in Windows The MTU size for each individual Ethernet protocol enabled partition can be independently set from Normal 1500 bytes up to Jumbo 9600 bytes in several places in Windows One place to set the Ethernet protocol enabled partition s adapter MTU size is in the Window s Networking Adapter Advanced Properties Jumbo Packet properties adapter Click the property you want to change on the waste left and then select Ex value on the mght Humber af ASS Gumus Pause On Exhausted Host Aing Pretty amp VLAN BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 82 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning Another place to set the MTU size is in the BACS4 NDIS device Configurations page l Broadcom Advanced Control Suite 4 i ET tes a vew z eontoreten HIE Advanced E ES Monte CNS C0 ES Adapter BOMS709 C0 EE Adapter3 BCMS7711 AD SS Adapters 57712 A1 EF e Pott E Isi Broadcom BCMS7712 Hetkene I 20 GigE Multi Function 50 SS 0012 Broadcom BCM57712 Netittreme H 10 GigE Multi Function iSCSI Ar f ES Portal LLL4 ace Portal fe80 210 T8 fe6 d xcs 7 ppoz7 Broadcom BCMS7712 Netktreme II 10 GigE NDIS VED Cheni 250 00
95. led to transmit within the limitations of the Priority Group s ETS value determined by it s Traffic Type and each partition s Maximum Bandwidth setting and the overall connections link speed For example the FCoE traffic type could be assigned to Priority Group 1 PG1 and all of the other traffic types iSCSI and Ethernet could be assigned to another Priority Group such as PGO Each Priority Group has it s own ETS value which works similarly to a minimum bandwidth BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 22 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning setting DCB Lossless iSCSI iSCSI TLV could be used simultaneously in another Priority Group or in place of FCoE for a similar effect where the Lossless iSCSI Offloaded traffic would go through it s assigned Priority Group while the Lossy Ethernet traffic would go through another Similarly to the other two rate controlling modes the host OS determines the actual inter partition traffic ratio for the cases where two traffic types share the same partition r Note A traffic type s send flow rate will be approximately the ratio of its individual partition s relative bandwidth weight setting divided by the sum of the relative bandwidth weights of all the partitions currently actively sending on that port or that partition s maximum bandwidth setting whichever is lower In the case where the Relative Bandwidth
96. llaneous Registers D158 E 4 Intel R processor PCI Express Root Port 1 0138 zi E Broadcom BCM57712 NetXtreme II 10 GigE 41 POR EF Broadcom BCM57712 NetXtreme II 10 GigE NDIS VED Client 41 B E Broadcom BCM57712 NetXtreme II 10 GigE 42 S d Broadcom BCM57712 MetXtreme II 10 GigE NDIS VBD Client 42 Figure 8 Windows Device Manager To enable the applicable offload devices to be detected by the operating system start BACS4 and while in 57712 k 578xxS SF mode select the port System Device gt Configuration gt Resource Reservations check the boxes to enable the applicable iSCSI and FCoE Offload Engines protocols and then click Apply Click Yes when the temporary network connection interruption warning displays and wait for the discovered devices to be installed by Windows no reboot is necessary while in SF mode This must be done for each applicable device port BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 28 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning 5 Broadcom Advanced Control Suite 4 File View Acton Filter Context Tools Teams iSCSI Help Fitter keen E Configurations J M Resource Reservations Licenses MN Advanced Explorer View Information Configurations Statistics EI 2 7 Hosts E WIN RA3QJRK0731 died LT Adapteri BCM5709 C0 Resource Reservations L7 Adapter BCM5709 CO LT Adapter3 BCM57711 AQ LC Adapter4 57712 A1 E
97. lows P1i and P1e each still getting 1 25 Gbps while P4 s allocated BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 102 57712 k 578xxS Software User Manual Examples bandwidth will now be split into two traffic flows P4e and P4i which means each get 1 25 Gbps e When P1i stops sending at t6 the only partition P1 traffic flow P1e will readjust to 2 5 Gbps and all of the others will remain the same e When P1e stops sending at t7 the other traffic flows will readjust to 3 3 Gbps P2e and P3e and 1 65 Gbps each for partition P4 s shared P4e and P4i traffic flows e When P2e stops sending at t8 partition P3 s single traffic flow will readjust to 5 Gbps P3e and partition P4 s shared P4e and P4i traffic flows will increase to half that or 2 5 Gbps e When P3e stops sending at t9 the remaining partition P4 will now receive all of the available bandwidth so it s two traffic flows P4e and P4i will equally share it for 5 Gbps each e Finally when P4e stops sending at t10 the remaining traffic flow P4i will now receive all of the available bandwidth or 10 Gbps e fthere is only one flow in a partition it would take all of the bandwidth allocated for that partition e The main difference between setting all four partition s relative bandwidth weight to 096 and setting them to all 2596 is that O96 s causes the send bandwidth to be shared between all active traffic flows while 25
98. maximum bandwidths are set to 100 for all four partitions there is no difference between the earlier DCB mode example and this one in DCB mode Table 9 Non DCB Oversubscription With One High Priority Partition Relative Bandwidth Maximum Port Partition Weight Bandwidth Protocol Plot Color Port O Partition 1 P1i 1 100 iSCSI Offload Green Port O Partition 1 P1e 1 100 Ethernet Brown Port O Partition 2 P2e 1 100 Ethernet Purple Port O Partition 3 P3e 1 100 Ethernet Yellow Port O Partition 4 P4e 97 100 Ethernet with Blue TOE Port O Partition 4 P4i 97 100 iSCSI Offload Red The following plot shows a similar effect to some of the previous examples except that the fourth partition takes as much of the bandwidth as it needs up to 100 when it starts to transmit In this example the three 0 Relative Bandwidth Weight partitions have an effective Relative Bandwidth Weight of 1 instead of 0 e The first partition s traffic flow P1i initially takes 100 of the available bandwidth when the test application starts to transmit traffic on that port by itself at tO e When P1e starts to send Ethernet traffic at t1 both will stabilize to 5 Gbps each e When partition P2 traffic flow P2e starts to send traffic at t2 since only partition s P1 and P2 are now sending each partition gets half of the send bandwidth so partition P2 P2e gets all of the allocated 5 Gbps and partition P1 s two traffic flo
99. me components are potentially out of date Please go to Platform Update to view and run available updates 2 From the USC select the System Setup Advanced Hardware Configuration option BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 15 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning DOLL LIFECYCLE CONTROLLER UNFIED SERVER CONFIGURATOR Help About Exit oe System Setup Lifecycle Log Advanced Hardware Configuration configures Human Interface Infrastructure HI enabled devices EX BIOS NICs Platform Update Hardware Configuration OS Deployment Advanced Hardware Configuration Platform Restore Hardware Diagnostics LC Settings System Setup NISY 3 Select the System Setup s Device Settings option DOLL SYSTEM SETUP Help About Exit System Setup System Setup Main Menu System BIOS IDRAC Settings Device Settings p Select to configure system BIOS settings PowerEdge R720 Arrow keys S Service Tag TYHNJ36 SC 4 Allofthe Broadcom Ethernet Controller devices should be displayed on this page Select the desired device port from the displayed list BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 16 57712 k 578xxS Software User Manual DOLL SYSTEM SETUP System Setup Device Settings Integrated RAID Controller 1 Dell PERC PERC H310 Mi
100. n Configuration gt Global Bandwidth Allocation Menu Partition 1 Relative Bandwidth VVeight Partition 1 Maximum Bandwidth Partition 2 Relative Bandwidth VVeight Partition 2 Maximum Bandwidth Partition 3 Relative Bandwidth VVeight Partition 3 Maximum Bandwidth Partition 4 Relative Bandwidth VYeight Partition 4 Maximum Bandwidth Configure relative bandwidth weight Valid range 0 100 percent The sum total of all partitions must be equal 0 or 100 focus DOLL STS TENISE Help About Exit Integrated NIC 1 Port 1 Broadcom NetXtreme Il 10 Gb Ethernet BCM57800 78 2B CB 5B 99 28 Broadcom Main Configuration Page Global Bandwidth Allocation Menu Broadcom Main Configuration Page gt NIC Partition Configuration gt Global Bandwidth Allocation Men Partition 1 Relative Bandwidth VVeight Partition 1 Maximum Bandwidth Partition 2 Relative Bandwidth VVeight Partition 2 Maximum Bandwidth Configure relative bandwidth weight Valid range 0 100 percent The sum total of all four partitions must be equal 0 or 100 dge RT20 Service Tag TYHNJ36 sc to exit page Tab to change focus See Broadcom Advanced Control Suite 4 BACS4 on page 35 for more information on how BACS4 can also be used to control these settings The Relative Bandwidth Weight is the value the port gives to that single partition s send or outgoing traffic with respect to any other
101. n to the Ethernet Protocol s being enabled on all four partitions the iSCSI Offload protocol is enabled on Partition 1 and 4 By setting the partition s relative bandwidth weights to 25 each partition s traffic flows i e P1 s iSCSI P1i L2 Ethernet P1e and P4 s iSCSI P4i L2 Ethernet P4e will be contained in their respective partition while each partition over all takes an equal share of the available bandwidth The traffic flows within that partition can only expand into that partition s allocated by weight portion There would be no difference between the previous examples and this example with DCB enabled since it essentially sets the Relative Bandwidth Weight values to all ZEROs and the PG s ETS would come into play Table 7 Non DCB Partitioned Oversubscription Relative Bandwidth Maximum Port Partition Weight Bandwidth Protocol Plot Color Port O Partition 1 P1i 25 100 iSCSI Offload Green Port O Partition 1 P1e 25 100 Ethernet Brown Port O Partition 2 P2e 25 100 Ethernet with Purple TOE Port O Partition 3 P3e 25 100 Ethernet Yellow Port O Partition 3 P4e 25 100 Ethernet with Blue TOE Port O Partition 4 P4i 25 100 iSCSI Offload Red The following plot shows how each traffic type flow must remain within a partition s share of a ports available send bandwidth i e if there are two different traffic type flows such as P1i and P1e in a single partition they are combined as if one
102. ne to all four are sending sharing the available bandwidth between each other File Edit View Inventory Administration Plug ins Help E 4 Home p el Inventory p sp Inventory g g H 1013 126 7 ESE OEE TERI ET EE ET TTT EI Started Summam Virtual Machine Resource Allocation Performance Confouapon Local Users 8 Groups Euentz Permtssiong Network Real time 1 12 2011 4 32 27 PM 1 12 2011 5 32 27 PM Chart Options Switch to Default Graph refreshes every 20 seconds 2000000 2 running shared BW 1 2 each 1500000 All 4 running 3 running 2 running 3 running All 4 running shared BW sharing BW sharing BW shared BW shared BW 1 running 1 running 1 3 of total each 1 4 of total each 1 4 of total each 1 3 ot total each 1 2 each all BW all BW 1000000 b i i 0 i i i i i i i i i 4 35 PM 4 40 PM 4 45 PM 4 50 PM 4 55 PM 5 00 PM 5 05 PM 5 10 PM 5 15 PM Time 0 running Performance Chart Legend Key Object Measurement Rollup Units Latest B Maximum Minimum Average RW VM 1 Data transmit rate Average KBps 291507 1158844 0 259082 90 ei VM 2 Data transmit rate Average KBps 291524 580845 229489 34 Oo 291624 O 291412 0 VM 3 Data transmit rate Average KBps 581939 0 250998 64 1159897 0 VM 4 Data transmit rate Average KBps 362926 90 The second port s network performance indicates each VM is limited to it s specific top end setting Maximum Bandwidth and does n
103. ni Configuration Utility Integrated NIC 1 Port 1 Broadcom NetXtreme Il 10 Gb Ethernet BCM57800 78 2B CB 5B 99 28 Integrated NIC 1Port 2 Broadcom NetXtreme Il 10 Gb Ethernet BCM57800 78 2B CB 5B 99 2A Integrated NIC 1Port 3 Broadcom Net Xtreme Il 1 Gb Ethernet BCM57800 78 2B CB 5B 99 2C Integrated NIC 1 Port 4 Broadcom Net Xtreme Il 1 Gb Ethernet BCM57800 78 2B CB 5B 99 2E NIC in Slot 6 Port tt Broadcom Net Xtreme Il 10 Gb Ethernet BCMS 7810 00 10 18 96 DC 6C NIC in Slot 6 Port 2 Broadcom NetXtreme Il 10 Gb Ethernet BCM57810 00 10 18 96 DC 6E Please note Only devices which conform to the Human Interface Infrastructure HI in the UEFI Specification are displayed in this menu T Configure Multiple Boot Agent MBA and iSCSI Boot Parameters r keys and Enter to selec o exit page Tab to ch focus Service Tag TYHNJ36 Configuring Switch Independent Partitioning Help About Exit 5 From the Broadcom Main Configuration Page select the Virtualization Mode Control to turn on NPAR or off Forced SF the Switch Independent Partitioning mode of operation for the selected devices Select the Device Configuration Menu option to enable or disable DCB mode Select the NIC Partitioning Configuration Menu option to change the device s Switch Independent Partitioning settings the mode must be enabled in this window before the option will appear also in this same window BROADCOM June 07 2012 e 2CS5771
104. of PGO s share between themselves which is 1 65 Gbps each takes 1 3rd of 5 Gbps e When P4e starts to send at t4 the four Ethernet traffic flows take 1 Ath of PGO s bandwidth 1 25 Gbps each while P1i is still unaffected and remains at 5 Gbps e When P4i starts to send at t5 the four Ethernet traffic flows remain the same but the two iSCSI traffic flows split PG2 s allocated bandwidth 2 5 Gbps each e Then when P1i stops sending at t6 the traffic flows in PGO are unaffected while P4i s share increases to all of PG2 s allocated bandwidth of 5 Gbps e Aseach of the traffic flows stops sending in PGO the traffic flows of the remaining member s of PGO equally increase their respective shares to automatically occupy all of the available bandwidth remain in PGO until all of PGO s Ethernet flows stop At t7 there are three active PGO flows P2e P3e and P4e so each gets 1 3rd of PGO s 5 Gbps or 1 65 Gbps At t8 there are two active PGO flows P3e and P4e so each gets 7 of PGO s 5 Gbps or 2 5 Gbps At t9 there is only one active PGO flow P4e so it gets all of PGO s bandwidth or 5 Gbps Through all of this the Lossless iSCSI flow of P4i remains at 5 Gbps since it takes all BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 98 57712 k 578xxS Software User Manual Examples of PG2 s portion of the overall bandwidth ETS of 50 Finally at t10 there is only one active flow after P4e stops s
105. oftware User Manual Table of Contents Table of Contents About This Document 6 feier 6 AUAN NETTE aes nase nu ares ooh T ncaa ea ean D DEL 12 Acronyms ANG ADDF VIE e ln EE 13 Technical SUDBOTE E 13 Configuring Switch Independent Partitioning cccccccesccccsecceseceescecesececsneceeeeceseseesaueeeeeuecensaeeeneess 15 sing Klees 15 Supported Operating Systems wisisscccesisisccseneasesevessvadernitevsvecers psssuna NENNEN ER E FR YER a ETE Sr EET LE EUR 26 Supported server Plat e le EE 27 Viewing and Configuring the Partitions ccccccsscccssecccssecceeceaescecaeececeeceseeceseneeseeeeseaceeeneesseneeeeeens 28 Windows Server 2005 28 Installing the Latest Dell RE 28 Viewing the Enabled Devices in Device Manager c sseccuseeceseeccnscecaucecaececauececsuecensneeeneess 31 Broadcom Advanced Control Suite 4 BACS4 essen nnne nnns 35 Microsoft Windows Network Connections eese eene nennen nnne nnn nnn 43 Device PCle Bus Eege NEE 46 Red Hat Enterprise LINUX VO EEN 52 UNIW I A E ES EE 57 Details on Configuring iSCSI Offload on ESX ESXi A7 61 VIV E ESAS EE 67 Details on Configuring iSCSI Offload on ESXi 5 0 ccccceccccssccccesccceseccceneceeeeeeeeneessseesaecensneeeneess 68 Details on Configuring FCoE Offload on ESXi bO nemen 72 Details on Viewing Networking and Storage Performance on SN 78 eM N O aaah a E E E E EE EE A E EEE E I 82 Setting MTU
106. om B9 57712 He Prime TE 10 tage Mul Funcson i55 E 20 13 Broadcom BOMSTT LZ Net peee ET 20 Gage Mul Function ESI Adapter a5 Er amp Porti 1 11 24 E ER in 2011 05 com broadenm tgtida ram 14a ail 0031 Broadcom BCMS 7712 NetCreme I 30 Gage DCS VED Cher 59 cn ei Breadcom BCM57712 NertOrenme TI 10 Gage Mus Function 260 E 0033 Broadcom ECH 7712 NetXreme I 30 Gage DCS VED Clenit 460 B E 006 1 Broadcom BCM57712 MeD reme D 10 Gage Mule Function S i Les 00H Broadcom BOMS77 12 NetXtreme I 30 GigE Multi Function GOSI Adapter 251 GE 0034 Broadcom BCMS77 12 NetXtreme ET 30 GigE NOES VED Chenit 61 Port BROADCOM June 07 2012 e 2CS57712 SWUM103 R gen TTT Tee Target Inforemalen Target Type Portal List __ Target IGN Nene Persstency Flag Yes Lu LUNG Terion nimaa Ej PPFPPABO 19072005 FFFFFASO1 Target ign 201 1 05 com brosdcomctgt a ram 13a Target Portal unin L11124 Member of Decnvery Group LLL1L222 11 agn 2011405 com broaden tgt 13a ram 1358 Switch Independent Partioning Setup Guide Page 42 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning The following shows the per device partition s current offload licenses Erosion Adanim Ga eee rine ur vew Te Configurators DIE Resource Reservatons E Licenses E Advanced Explorer View TS REN Adapter BCM5709 Co GS Adapter3 BCMS7711 AD
107. onfiguring Switch Independent Partitioning BACS4 displays the per partition Virtual Bus Device VBD information Broadcom Advanced Control Suite 4 la OO a pes P mer mtn rectum E BU Hosts Eg vweesaspmkon B ES Adapteri BCMS709 C0 D ES Adapter BCM5709 CU E ES Adapter3 BCMS7711 A0 B ES Adapters 57712 A1 3 0027 Broadcom BCMS7712 Netitreme H 10 GigE NDIS VED Client 50 DOSi Broadcom BCMS7712 Netxtreme II 10 Ge Multi Function 51 o D I 0053 Broadcom BCMS7712 NetXtreme II 10 Gig Multi Function 253 SA 0030 Broadcom BCM57712 Nettresme H 10 GigE NDIS VED Chent 53 SF posa Broadcom BCMS7712 Netxtreme II 10 GigE Multi Function 254 P 0016 Broadcom BCM57712 Netitreme II 10 GigE Multi Function FCoE Adapter SE 0031 Broadcom BCMS7712 NetXtreme IT 10 GigE NOIS VED Client 254 Broadcom Advanced Control Suite 4 Fle Ven Acton Fiter Context Took Team BCSI Hep Fiter AU ew Im information 7 vital Signs H Driver Information E 9 Hosts pert S H wma GE el S E Adapters CMS 709 00 MAC Address 0010156F0380 B Adapter2 0570 CQ A o EN SE hemm nonet unu S L EON WWPN Beeren d ET 0050 Broadcom BCMS7712 Net reme TT 10 Gef Multi Function 250 Lo me amp E oso is Gen a bute ys 0051 Broadcom BOMS77 12 Netitreme H 10 GigE Multi Function 51 Driver Status ILeaded CE 0053 Broadcom BCMS7712 NetXtreme D 10 Gg Mult Function 253 O Driver Version mg
108. onfiguring of both ports Switch Independent Partitioning flow control protocols Relative Bandwidth Weights Maximum Bandwidth settings The following figure shows the per partition Switch Independent Partitioning settings see Figure 12 This is where BACS4 can enable or disable Switch Independent Partitioning mode This is also where BACS4 controls the Switch Independent Partitioning per port IEEE 802 3x Link Level Flow Control settings used when DCB s PFC is disabled enabled protocols Ethernet or iSCSI or FCoE the Relative Bandwidth Weight values and the Maximum Bandwidth values per partition Broadcom Advanced Control Suite 4 File View Acton Filter Context Tools Teams iSCSI Help Fiter au view m Configuration 7 Iv nic Partition Morer View z El 888 Hosts E fj WIN RA3QJRKO731 9 CR Adapteri BCM5709 C0 E VER Adapter2 BCM5709 C0 E CR Adapter3 BCM57711 AD CEN Adapters 57712 A1 El 4 Porti ETT 0050 Broadcom d SCH 0012 Broadc FCoE Sr 0027 Broadc 1 h Relative Bandwidth Wei 5 0051 Broadcom z PP 0053 Broadcom 38 0030 Broadc EI 0054 Broadcom A 0016 Broadc b 28 0031 Broadc Port2 Enable Enable E Disable 45 80 v Enable m Disable I Disable 45 a Figure 12 BACS4 Switch Independent Partitioning Settings BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 35 57712 k 578xxS Software User Manual C
109. oning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 54 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning Link ncap Ethernet HWaddr 800 18 18 88 E7 B4 inet addr 18 2 2 282 Bcast 18 255 255 255 Mask 255 6 6 6 inet6 addr fe80 210 18ff fe88 e7b4 64 Scope Link UP BROADCAST RUNNING MULTICAST MTU 9600 Metric 1 RX packets 1781 errors 8 dropped 8 overruns 8O frame 8 TX packets 31 errors amp dropped 8O overruns 8O carrier collisions O txquevelen 1600 RX bytes 290494 283 6 KiB TX bytes 8110 7 9 KiB Interrupt 181 Memory d78866080 d77TTTTTT Link encap Ethernet HWaddr 880 18 18 88 E7 B65 inet addr 18 2 2 283 Bcast 10 255 755 255 Mask 255 6 60 6 inet addr fe808 218 18T1T Te88 e7b6 64 Scope Link UP BROADCAST RUNNING MULTICAST MTU 9600 HMetric l RX packets 1738 errors 8 dropped 8 overruns 8 frame TX packets 33 errors po dropped H overruns 8 carrier collisions B txquevelen 1666 RX bytes 299886 292 8 KiB TX bytes 8458 8 2 KiB Interrupt 169 Memory d88o0066 dsfttttt Check the Linux system message logs to see if the iSCSI HBAs in this example on the first two partitions of each port Port 0 Partition 1 eth2 Port O Partition 2 eth4 Port 1 Partition 1 eth3 and Port 1 Partition 2 eth5 have iSCSI Offload enabled fvar log messages monitored System Log Viewer Die Edit View Help e ivarAog localhost bnx2i eth5 free hba done after retries eron localhost bnx2i e
110. oning Setup Guide Page 68 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning 7 iSCSI Initiator vmhba38 Properties Port Group 5 VMkernel Adapter Port Group Palicy Path Status 3 Select the VMKernel vmk adapter that is compatible with this iSCSI Offload device for binding and click OK BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 69 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning Bind with VMkernel Network Adapter SE Ss A Port Group VMkernel Adapter Physical Adapter 9 VMk vmnici vSwitch1 vmk1 ER vmnici0 10000 Full Virtual Network Adapter WMkernel vmk1 Switch v5witch1 Port Group VMk vmnici IP Address 1 1 50 1 Subnet Mask 255 255 0 0 Physical Network Adapter Name vmnicil Device Broadcom Corporation NetXtreme II 57712 10Gigabit Ethernet Link Status Connected Configured Speed 10000 Mbps Full Duplex If successful the vmk displays that it is Compliant BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 0 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning iSCSI Initiator vmhba38 Properties Port Group f VvMkernel Adapter Port Group Policy Path Status 49 vMk vmnici vSwitchl vmki Q9 Compliant Not Used Virtual Network Adapter VMkernel vmk1i Switch wSwitch1 Port Group VMk vmnici
111. ot expand into the unused area BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 60 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning E 10 13 126 47 vSphere Client File Edit View Inventory Administration Plug ins Help E Home b gh Inventory gt EA Inventory a e D 10 13 126 47 ESX 1SCSI Host example com VMware ESX 4 1 0 260247 VM 5 j H 2 VM 6 Getting Started Summary Virtual Machines ResourceAllocation Performance Configuration Local Users amp Groups Events Permissions eb VM 7 Network Real time 1 13 2011 6 59 23 AM 1 13 2011 7 59 23 AM Chart Options Switch to NIC Partition Demo EI E D iJ cy e VM 8 Graph refreshes every 20 seconds 2000000 1500000 i i000 VM 6 NP 3 2G hell _VM 6 NP 3 2G vms NP1 1G VMS NP 1 1G VM 6 NP 3 2G _VM 5 NP 1 1G i i i 1 T T 7 05 AM 7 10 AM 7 15 AM 7 20 AM 7 25 AM 7 30 AM 7 35 AM Time Performance Chart Legend Object Measurement Rollup Units Latest Maximum Minimum Average VM 5 Data transmit rate Average KBps 6410 116517 104064 21 0 VM 6 Data transmit rate Average KBps 231296 232548 o 17341357 VM 5 NP 1 0 10 1G VM 6 NP 3 0 20 2G VM 7 Data transmit rate Average KBps 348975 349189 o 21630931 VWM 7 NP 5 0 30 3G VM 8 NP 7 0 40 4G 0 VM 8 Data transmit rate Average KBps 464236 46
112. other traffic flows will equally absorb the unused BW BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 96 Examples 57712 k 578xxS Software User Manual Page 97 Ea mmm 00 120 Ip a mmi ee Emm C EEN E 3 a LE oe z SH nmm x Lord R ER Switch Independent Partioning Setup Guide pM eO E E E M M M mm EE i WR 0M BP LLL LLLLILLLL LLL i e ER T Hii M E E E E S sl LL 00M 15 June 07 2012 e 2CS57712 SWUM103 R BROADCOM 57712 k 578xxS Software User Manual Examples The following two examples are of oversubscribed bandwidth sharing with DCB enabled The first example is similar to the above non DCB example with four Ethernet and two iSCSI Offload protocols enabled with the same two traffic types in six distinct flows with the partitions similarly configured The main differences here is the iSCSI Offloaded traffic type is assigned to DCB Priority Group 2 and is Lossless i e iSCSI TLV with an ETS setting of 50 while the L2 Ethernet traffic type is still assigned to Priority Group O and is Lossy with an ETS setting of 50 If the iSCSI Offload protocol traffic flows had been assigned to the same PG as the Ethernet protocol traffic flows then the traffic BW would have looked very similar to the previous Non DCB example since ETS would never be activated for traffic flows belonging to the same PG
113. r port The two dedicated 1GbE ports will remain in single function mode and do not support FCoE or iSCSI Offload modes The following figure illustrates the 57800S PCI bus numbering BROADCOM June 07 2012 e 2CS57712 SWUM103 R Switch Independent Partioning Setup Guide Page 47 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning 57800 DUAL 10GbE and DUAL1GbE Device PCle Location Bus Device Function Ethernet MAC Example 1st 10GbE port The first three are the same for both ports Address IP Address Port 0 Partition 1 Partition 2 v gp CO CO OP OP CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD 2nd 10GbE port e v OP UP Port 1 Partition 1 00 10 18 88 E7 AA Go WW e Partition 2 00 10 18 88 E7 B2 1st 1GbE port S pee HUETR Te On VU Ur t E Port 2 XIUIZIA 00 10 18 88 E7 AC 55 5 1 o H H LL oeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee ee ee ee 2nd 1GbE port o c wc wu Port 3 X Y Z 3 00 10 18 88 E7 AE 6 6 6 1 One way to locate PCle information is to open the individual Network Connection s Properties BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 48 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning A a Bo E Sg E T i WH a a a 57 71
114. r tower and tower tower servers servers servers servers BladelO NA NA NA 1GbE PTM k NA PTM k PTM k NA NA NA Modules Switches M8024 k M8024 k M8024 k and PTMs and and and M8428 k M8428 k M8428 k Ports 4 2 4 4 2 4 2 2 2 4 2 Interface 1000BASE T 1000BASE T 1000BASE T 1000BASE T KR 2x10G SFP KR KR 10G 2x10GBASE 10GBASE T 2x1000BASE SFP T T 2x1000BASE T BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 27 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning Viewing and Configuring the Partitions e Windows Server 2008 R2 e Red Hat Enterprise Linux v6 x e VMWare ESX ESXi 4 1 e VMWare ESXi 5 0 Windows Server 2008 R2 Installing the Latest Dell Drivers When the 57712 k 578xxS device is first installed the iSCSI and FCoE devices may not appear If the latest Dell driver is already present on the system the 57712 k 578xxS device will be identified and the NDIS personality protocol will be installed If the latest NetXtreme II drivers are not present on the system go to the Dell driver download web site http support dell com support downloads under the specific Dell blade server platform and download install the latest NetXtreme Il network drivers software version 17 2 x or later and firmware version 7 2 x or later for your specific server system BR Intel R 82801 PCI Bridge 244 a Intel R processor DMI D130 i A lt Intel R processor Misce
115. re 6 57810S T 10GBASE T Converged NIC on page 11 and Figure 7 57800S T 10GBASE T Rack Converged NDC on page 12 Figure 1 Broadcom 57712 k Blade Converged NDC BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 6 57712 k 578xxS Software User Manual About This Document mrt te zeen TUDCOEERY Y Y Y GN Se UGO ET ies 2M d e 4 o ooo o do odoo doo duai a dh a Figure 2 57800S Rack Converged NDC BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 7 57712 k 578xxS Software User Manual BROADCOM June 07 2012 e 2CS57712 SWUM103 R AMT AT m diii o 4 COPYRIGHT md K RIGHTS RESERVED 1 AT RS 1 Ze 8 DCOM PED BROA am ole eg E NES TENE A e weren Figure 3 57810S k Blade Converged NDC Beer CIT EE ET en About This Document Switch Independent Partioning Setup Guide Page 8 57712 k 578xxS Software User Manual About This Document Figure 4 57810S k Blade Converged Mezzanine BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 9 About This Document 57712 k 578xxS Software User Manual 7 3 Ki QS W SCH A W oS ond CH Ve n P H i 57810S Converged NIC 5 Figure Q O D ei D 85 O g Qa 3 DI Lo O S c 2 Switch Independent Part June 07 2012 e 2CS57712 SWUM103 R BROADCOM 5771
116. re dhcp 10 13 126 210 broadcom com 2145997461 35 bnx2i 00101888e5f2 Memory Storage Networking Storage Adapters Details Network Adapters Advanced Settings e Model Broadcom iSCSI Adapter Power Management iSCSI Name ign 1998 0 1 com vmware dhep 10 13 126 210 broadcom com 2145997461 33 iSCSI Alias bnx2i 00 10 1888e5ee a This is where you add either the iSCSI non offload software initiator pathway or in this case the Broadcom FCoE Adapter ESXi 5 0 lists all FCoE Adapters as Software even if Hardware Offloaded 2 Select Add Software FCoE Adapter and click OK Add Storage Adapter C Add Software iSC51I Adapter f Add Software FCoE Adapter e oe 3 In the next window select the desired 57712 k 578xx FCoE Hardware Offload adapter port and click OK C Add Software FCoE Adapter Physical Network Adapter VLAN ID Priority Class FCoE Controller MAC Address Cancel Help The new Broadcom FCoE Hardware Offloaded Adapter now appears in the Storage Adapter s list and is ready for use BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 73 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning dhcp 10 13 126 210 broadcom com VMware ESXi 5 0 0 469512 EE lee ee lee es Configuration WETTER Hardware Storage Adapters Add Remove Refresh Rescan All Device Type WWN e Broadcom CorporationFCoE Adapt
117. re4 r41 vsp 41 config max pdf Using the 57712 k 57810S Switch Independent Partitioning mode allows you to expand the number of ports usable from 4 physical ports to 16 virtual ports This allows better port flexibility traffic isolation service quality and bandwidth tuning for your management backup migration production networks The following shows the vSphere Network Adapters Configuration page with the eight enabled Ethernet protocol partitions always four per port as the Broadcom Corporation NetXtreme Il BCM57712 10 Gigabit Ethernet MultiFunction devices vmnic6 through vmnic13 in this example VMWare enumerates the partitions in the order of the PCI function numbers where port 0 has device functions 0 2 4 6 which are vmnics 6 8 10 12 and port 1 has device functions 1 3 5 7 which are vmnics 7 9 11 13 localhost VMware ESXi 4 1 0 260247 Getting Started Summary Virtual Machines Resource Allocation Performance Configuration Ew BRETT Rer TI Ee Permissions Hardware Device Speed Health Status Processors Configured Switch MAC Address Broadcom Corporation NetXtreme II BCM57712 10 Gigabit Ethernet Multi Function Observed IP ranc Memory Storage Networking Storage Adapters Network Adapters Advanced Settings Power Management vmnic8 vmnica vmnic vmnich vmnic13 vmniciz vmnic11 vmnic1 10000 Full 10000 Full 10000 Full 10000 Full 10000 Full 10000 Full 10000 Full 10000 F
118. refreshes every 20 seconds 100 r 2 In the next window select either Networking for L2 Ethernet OR Storage Adapter for iSCSI FCoE and e Select either line or graph e Select which ports partitions to view e Select what counters to display BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 78 57712 k 578xxS Software User Manual e Customize Performance Chart Saved Chart Settings Default Chart Options CPU Datastore Gei vSphere Replication Configuring Switch Independent Partitioning Description vmhba32 vmhba33 vmhba34 x Counters Descriptian Average commands issued per second Write rate LI Average write requests per second C Read rate C Average read requests per second Stacked graph g d gt _ All None Average Average Average Average Average Number KBps Number Kbps Number The following screen captures show the 57712 k Converged NDC in DCB mode with mixed L2 Ethernet iSCSI and FCoE traffic running at various points in both the ESX Performance Networking and Storage Adapter displays BROADCOM June 07 2012 e 2CS57712 SWUM103 R Switch Independent Partioning Setup Guide Page 79 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning localhost localdomain VMware ESXi 5 0 0 Getting Started Summary Virtual Machines ResourceAllocation Performance Configuration Local
119. s Bau DNS and Routing Runtime Name Target LUN Status Authentication Services vmhba32 C0 T3 L0 iqn 2010 10 com broadcom tgt20 ram0 172 168 1 0 Q Active 1 0 vmhba32 C0 T2 L0 iqn 2010 10 com broadcom tgti9 ram0 172 168 1 0 Active V vmhba32 C0 Ti LO iqn 2010 10 com broadcom tgti8 ram0 172 168 1 0 Q Active Security Profile vmhba32 C0 T0 LO iqn 2010 10 com broadcom tgti7 ram0 172 168 1 0 Active 0 o System Resource Allocation vmhba32 C0 T4 LO iqn 2010 10 com broadcom tgt21 ram0 172 168 1 Active Advanced Settinas Virtual Machine Startup Shutdown Virtual Machine Swapfile Location Now you can add the discovered LUNs as Hard Disks to the desired VMs i VM 2 W2k8R 20 Virtual Machine Properties Hardware Options Resources Virtual Machine version 7 P Physical LUN and Datastore Mapping File Show Al Devices Adde Remove Ivmfsjdevices disks t 10 94544500000000009637363796431 Summary 4096 ME 1 Video card Video card VWMICI device Restricted SCSI controller 0 LSI Logic 545 Hard disk 1 Virtual Disk CDIDVD Drive 1 Client Device Network adapter 1 Vu Network Floppy drive 1 Client Device Mew Hard Disk adding Mapped Raw LUN ES New Hard Disk adding MappedRawLUN uid fa NN Q e H3 Gi BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 66 5771
120. s each 8 Gbps 4 equally weighted traffic flows e When P1i stops at t6 it releases it s bandwidth to the available pool and since bie and P3e are capped by their maximum bandwidth value to 1 Gbps the three other traffic flows P1e P4e and P4i automatically take equal shares and increase their bandwidth used to 2 6 Gbps each e When Pie subsequently stops sending at t7 P4e and P4i grab up the extra available bandwidth and go to 4 Gbps each Both P2e and P3e are unaffected and continue sending at 1 Gbps each e When P2e stops sending at t8 P4e and P4i are not able to make use of the freed up bandwidth since they BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 111 57712 k 578xxS Software User Manual Examples are both in partition P4 which as a maximum bandwidth ceiling of 8 Gbps Therefore none of the traffic flows increase their sending rates and this unused bandwidth is ignored e When P3e stops sending at t9 the same condition is still in effect Therefore none of the remaining active traffic flows increase their sending rates to use this extra bandwidth e Finally P4e stops at t10 and this allows it s companion traffic flow P4i to increase to 8 Gbps which is partition P4 s maximum top end The remaining 2 Gbps is unassigned The following example is the same as the previous example but with FCoE in the first partition Additionally the FCoE traffic flow is Lossless and in D
121. t Broadcom Corporation NetXtreme II BCM57712 10 Gigabit Ethernet MultiFunction devices and the Storage Adapters window will show the same enabled iSCSI Offload devices The newest feature of ESXi 5 0 is that the enabled FCoE Offload protocol devices are now visible The following shows two FCoE Offload and two iSCSI Offload Adapters in the ESXi 5 0 vSphere Storage Adapters window BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 67 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning localhost localdomain VMware ESXi 5 0 0 Getting Started Summary Virtual Machines Resource Allocation Performance Configuration W I Users amp Groups Events Permissions Device Broadcom Corporation FCoE Adapter CH vmhba35 Fibre Channel o 10 00 00 10 18 88 ee ed 20 00 00 10 18 88 ee ed G vmhba365 Fibre Channel o 310 00 00 10 18 88 2e ef 20 00 00 10 18 88 e amp e amp f Broadcom iSCSI Adapter vmhba32 iSCSI C vmhba33 iSCSI Details Health Status Processors Memory Storage Networking Network Adapters Advanced Settings Power Management ign 1988 01 com vmware localhost 1850531662 3 ign 1998 01 com vmware localhost 185053 1662 3 vmhba35 Model Broadcom Corporation FCoE Adapter WWN 10 00 00 10 18 88 ee ed 20 00 00 10 18 88 ee ed Licensed Features Targets 16 Devices 16 Paths i6 Time Configuration DNS and Routing Authentication Services View Devices P
122. t outgoing direction only All partitions receive direction maximum bandwidth is always the ports current Link Speed and is regulated by the attached switch port just as it isin SF mode when multiple L2 Ethernet and iSCSI Hardware Offload and FCoE Hardware Offload traffic protocol types are enabled The Maximum Bandwidth settings can be used to oversubscribe a port This is done by setting the four partitions of that single port to having a total Maximum Bandwidth setting SUM of more than 100 i e 10000 Mbps or 10 Gbps That just means the various partitions will attempt to take as much bandwidth as allowed by their specific setting maximum limits and weights as their individual traffic flow needs change In an oversubscription situation the 57712 k 578xxS device will ration out free bandwidth based on the weights sum is O verses sum is 100 and maximum settings and the mode DCB verses non DCB it is in The above example shows the first port s four partitions being set to 10410480480 180 which means the port is 180 subscribed 18 Gbps or 80 oversubscribed i e 18 Gbps subscribed 10 Gbps line rate available 8 Gbps oversubscribed The Maximum Bandwidth setting applies to all protocols enabled on that partition Note When Switch Independent Partitioning mode is first enabled or after a reset the default values for all four partitions is Relative Bandwidth Weight 0 and Maximum Bandwidth 25 BROADCOM Switch Independent
123. tXtreme Il 10 Gb Ethernet BCM57810 84 2B 2B E8 22 E3 DCB Protocol Disabled amp Enabled T Enable Disable DCB Protocol PowerEdge M620 Arrow keys and Enter to select Esc to exit page Tab to change focus If the NIC Partition Configuration Menu is selected instead of the Device Configuration Menu in Step 5 the device port s Global Bandwidth Allocation Menu option Flow Control and individual Partition options are presented The Global Bandwidth Allocation Menu option will open the control window for the Relative Bandwidth Weight similar to a minimum bandwidth and Maximum Bandwidth settings for that port s partitions The Flow Control options sets this port s flow control settings The four on the 57712 k and 57810 s and two on the 57800 s Partition options open a window that allows the enabling of available protocols for the selected partition and also displays the various MAC address information of that partition BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 19 57712 k 578xxS Software User Manual DOLL SYSTEM SETUP Configuring Switch Independent Partitioning Help About Exit Integrated NIC 1 Port 1 Broadcom NetXtreme Il 10 Gb Ethernet BCM57810 84 2B 2B E8 22 E3 Main Configuration Page NIC Partition Configuration Main Configuration Page gt NIC Partition Configuration Broadcom NetXtreme Il 10 Gb Ethernet BCM57810 84 2B 2B E8 22 E3 Global Bandwidth
124. tan 260 JE Broadcom BCMS7712HP Nettreme IL 10 Gig Mult Function 262 This brings up that device s properties window The following is the BCM57712 iSCSI device showing the HBA and connected target session information send receive statistics and ping test results Broadcom BCM57712 NetXtreme II 10 GigE Multi Function iSCSI Ad General Advanced Driver Details ISCSIMAC 00 10 18 6F D3 BF HBA Portals 1 11 7 FESU DOOD DODO OOOU 02 10 1 8FFFEGF D3BF ign com broadcom tot 15a ram 15a ign 2010 10 com broadcom tgt 15b ram 15b ign 2011 05 com broadcom tgt 1 4a ram 14a iqn 401 1 15 com broadcom tagt 4b ram 14b Bytes Received 872225 MB BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 32 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning The following shows the properties window for the FCoE device showing World Wide IDs connected Fabric and Target information Broadcom BCM57 712 Netktreme II 10 GigE FCoE Adapter 79 Prop General Advanced Driver Details Hot WAN RR RESO Mode WAN 10 00 00 10 1 FCoE MAC 00 10 18 38 E8 C5 Fabrics ZCCD2UUID EL CA AS No 620056 2009 Targets CBO500 Ce1100 Ce4500 FU 06 07 60 44 60 00 CE FU 06 01 69 44 60 00 CE 2D OB ODEDE 11 1 2 7042 OU UE OT BU D BOE CE 20 07 OOEDE 11 12 70 42 The following properties ane awaldable for this network adapter Ckck the property you
125. terrupt 169 Memory dl8686080 dlffffft Link encap Ethernet HWaddr 800 10 18 88 E7 AA inet addr 18 1 1 281 BHBcast 10 255 255 255 M Mask 255 8 0 8 inete addr feB8 2180 1BTT TesSB e73a b4 Scope Link UP BROADCAST RUNNING MULTICAST Metric 1 RAX packets 418 errors dropped 8 overruns frame 6 TX packets errors B dropped 0 overruns carrier 6 collisions 8 txquevelen 1000 RX bytes 80164 78 2 KiB TX bytes 8 0 b Interrupt 225 Memory d2B8B8BB d2TTTTTT Link encap Ethernet Hwaddr 680 10 18 88 E7 AC inet addr 18 1 1 207 Bcast 10 255 255 255 Mask 255 6 0 6 inete addr TeB80 210 18TT Te88 e7ac 64 Scope Link UP BROADCAST RUNNING MULTICAST BATU 4400 Metric l AX packets 376 errors 8 dropped amp overruns 8 frame 8 TX packets errors dropped 8 overruns carrier 6 collisions txquevelen 1666 RA bytes 73164 71 4 KiB TX bytes 8 8 b Interrupt 225 Menmory d38008000 d3TTTTTT BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 86 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning Link encap Ethernet HWaddr O08 10 18 88 67 AE inet addr 180 1 1 203 Bcasti1l6 255 255 255 Mask 255 80 0 0 inet addr fe88 210 18TT Tebl ze Las be UP BROADCAST RUNNING MULTICAST AX packets 373 errors dropped 8 overruns 8 frame 8 TX packets 6 errors O dropped overruns G carrier 6 collisions 8 txquevelen 1660 Ra bytes 722606 768 5 KiB TX bytes B 8 8 D
126. th4 free hba done after retries maillag localhost bnx2i eth3 free hba done after retries messages localhost bnx2i eth2 free hba done after retries secure localhost Broadcom NetXxtreme II iSCSI Driver bnx2i v2 6 4 1b Xorg 0 lag localhost iscsi registered transport bnx2i localhost bnx2i 01 00 03 ISCSI INIT passed localhost bnxzi 01 00 02 ISCSI INIT passed localhost bnx2i 01 00 01 ISCSI INIT passed d A Applications Places System localhost bnx21 01 00 00 ISCSI INIT passed 4 February 4 2011 Sun Mon Tue Wed Thu Pn Sat 1 Z 3 a 13 14 15 16 17 18 19 20 78 22 23 24 25 26 Ber bnxz Sear 27 28 Version Current A e 3320 lines 294 7 KB last update Sat Feb 26 16 10 44 2011 ei C File Manager 5 Bl root localhost Aib modul vartogimessages moni en 1 G BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 55 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning The Linux System Log Viewer message for the FCoE initiator being detected on scsi5 might look like this kernel scsi5 Broadcom Offload FCoE Initiator The Linux System Log Viewer message for the FCoE driver bnx2fc being loaded on scsi6 might look like this kernel bnx2fc Broadcom NetXtreme II FCoE Driver bnx2fc v2 2 9 Feb 28 2012 kernel bnx2fc Entered bnx2fc ulp init kernel bnx2fc min xid 0x0 max xid Ox
127. there was only one traffic flow on that partition e The weight applies to all enabled protocols on that partition e The Relative Bandwidth Weight is not applicable when in Data Center Bridging DCB mode In DCB mode all traffic flows act as if their Relative Bandwidth Weights are set to all Os The DCB s Enhanced Transmission Selection ETS values are used instead e The Switch Independent Partitioning transmit direction traffic flow rates are affected by the three main modes in the following ways n non DCB mode where the sum of the partition s Relative Bandwidth Weights equal 100 each partition s combined traffic flow is equally scheduled to transmit within the limitations of the partition s Relative Bandwidth Weight and Maximum Bandwidth settings and the overall connection s link speed This means a specific partition s Relative Bandwidth Weight value will restrict the traffic flows sharing that partition s bandwidth allocation as if one combined traffic flow with respect to the other actively sending partitions The partition s send flow rate is based on the ratio of that partition s individual weight verses the aggregated weights of all the other actively sending partitions Furthermore each partition s combined traffic flow will be capped by that partition s Maximum Weight setting See the User Guide s examples for more details The actual inter partition ratio of the two sharing traffic flows is controlled by the host OS Think of t
128. this example with DCB enabled since the weights are ignored and if all traffic types are in the same single PG Table 8 Non DCB Weighted Oversubscription Relative Bandwidth Maximum Port Partition Weight Bandwidth Protocol Plot Color Port O Partition 1 P1i 10 100 iSCSI Offload Green Port O Partition 1 P1e 10 100 Ethernet Orange Port O Partition 2 P2e 20 100 Ethernet Purple Port O Partition 3 P3e 30 100 Ethernet with Yellow TOE Port O Partition 4 P4e 40 100 Ethernet with Blue TOE Port O Partition 4 P4i 40 100 iSCSI Offload Red The following plot shows e The first partition s traffic flow P1i initially takes 100 of the available bandwidth at tO when an iSCSI test application is sending traffic out that port by itself e When P1e starts to send Ethernet traffic at t1 the two active traffic flows have equal weights with respect to each other so they are allocate half of the total bandwidth available 10 Gbps to partition P1 which equates to 5 Gbps each for P1i and P1e e When P2e starts sending at t2 the partition s relative bandwidth weights come into effect Partition P1 has a weight of 1096 while P2 has twice as much at 2096 so P1 s two sending traffic flows are reduced to half of the partition s assigned 1 3rd derived from P1 s weight of 1096 P1 s weight of 10 P2 s weight of 20 or 1 65 Gbps each for P1i and P1e P2e starts at 6 7 Gbps it s relative weight is 20 3096 tot
129. titioning protocols The following example shows a 57712 k but the 578xx devices would be similar e Eight enabled Ethernet protocol partitions four possible per port as the Broadcom BCM57712 NetXtreme II 10 GigE NDIS VBD Client 50 through 57 in the Network Adapters section e Two enabled iSCSI protocol partitions up to two are possible per port if no FCoE is enabled as the Broadcom BCM57712 NetXtreme II 10 GigE Multifunction iSCSI Adapters 51 and 55 AND two enabled FCoE protocol partitions one possible per port as the Broadcom BCM57712 NetXtreme II 10 GigE Multifunction FCoE Adapters 452 and 456 in the Storage Controllers section e Eight Broadcom BCM57712 NetXtreme Il 10 GigE Multifunction virtual bus devices 50 through 57 in the System Devices section These eight virtual bus system devices are always present and are not controlled by what protocol is enabled in USC sid 2 E Server Manager server Manager SES SCS Device Manager amp E E Des adapters Broadcom BCM5709C NetXtreme II GigE NDIS VED Client 46 KE Broadcom BCM5705C NetXtreme IT GigE NDIS VEO Client 47 E Broadcom BCM5709C NetXtreme II GigE bris VBD Client 48 ae Broadcom BCM57712 Netxtreme II 10 GigE NOIS VBD Client sis f Broadcom 8CM57712 NetXtreme II 10 GigE NDIS VBD Client 54 ZE Broadcom BCM57712 NetXtreme II 10 GigE NDIS VBD Client 55 oe Broadcom BCM57712 NetXtreme GigE Be ven Heo a 56 1M Broadcom BCM5709C NetXtreme II
130. u should see the ports now active esxcli fcoe nic list vmnic User Priority 3 Source MAC XX XX XX XX XX XX Active true Priority Settable false Source MAC Settable false VLAN Range Settable false BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 75 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning 4 Use the following command to see the host s iSCSI and FCoE devices Hf esxcfg scsidevs a Vmhba bnx2i unbound iscsi vmhba Broadcom iSCSI Adapter Vmhba bnx2i unbound iscsi vmhba Broadcom iSCSI Adapter Vmhba bnx2fc link up fcoe xxx xxx Software FCoE Vmhba bnx2fc link up fcoe xxx xxx Software FCoE Vmhba iscsi vmk online iscsi vmhba iSCSI Software Adapter BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 6 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning Note that the FCoE Offload device is listed as Software FCoE by the command s output At this point the FCoE LUNs will show up IF the connected FCF switch has been setup for this initiator to connect to the desired FC or FCoE target LUNs localhost localdomain VMware ESXi 5 0 0 Getting Starte Summary Virtual Machines ResourceAllocation Performance Hardware Se F Rescan All Health Status pu pe a G ymhbas6 iSCSI iqn 1998 01 com vmware localhost 610019708 38 bnx2i 00 101868ebdc
131. ull 10000 Full 10000 Full 10000 Full 10000 Full 10000 Full 10000 Full 10000 Full 10000 Full Mone Mone Mane None None None Mone Mone 00 10 18 88 e7 ae 00 10 18 88 e7 ac 00 10 18 88 e7 aa 00 10 18 88 287 38 00 10 18 88 27 b6 00 10 18 88 27 b48 00 10 18 88 e7 b2 00 10 18 88 e7 b0 0 0 0 1 7 255 255 0 0 0 1 7 255 255 0 0 0 1 7 255 255 0 0 0 1 7 255 255 0 0 0 1 7 255 255 0 0 0 1 7 255 255 0 0 0 1 7 255 255 0 0 0 1 7 255 255 The following shows the vSphere Storage Adapters Configuration page with four enabled iSCSI Offload protocol partitions Broadcom iSCSI Adapter vmhba32 through vmhba35 In this example which is similar to the setup used for the user examples later in this document Port O s first partition device function O is vmhba 32 Port O s fourth partition device function 6 is vmhba 34 Port 1 s first partition device function 1 is vmhba 33 and Port 1 s fourth partition device function 7 is vmhba 35 BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 57 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning localhost VMware ESXi 4 1 0 260247 Getting Started Summary Virtual Machines Resource Allocation Performance Configuration Local Users amp Groups Events Permissions Health Status Processors Memory wmhba32 imware localhost 974021991 34 bm2i 0010 18882728 e vmhba33 ign 1998 01 com vmware localhost 80
132. um bandwidth value of 80 or 8 Gbps The remaining bandwidth is shared equally by P2e and P3e at 1 Gbps each e When P2e stops sending at t8 P4e and P4i are not able to make use of the freed up bandwidth since they are both in partition P4 which as a maximum bandwidth ceiling of 8 Gbps The same is true for P3e which is also at it s bandwidth maximum Therefore none of the remaining traffic flows increase their sending rates and this unused bandwidth is ignored e When P3e stops sending at t9 the same maximum ceiling condition is still in effect Therefore neither P4e or P4i increase their sending rates to use this extra bandwidth e Finally P4e stops at t10 and this allows it s companion traffic flow P4i to increase to 8 Gbps which is partition PA s maximum top end The remaining 2 Gbps is unassigned BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 116 57712 k 578xxS Software User Manual Examples BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 117 57712 k 578xxS Software User Manual Dell has tested and certified the Broadcom 57712 k dual port 10 GbE KR Blade Converged Network Daughter Card Broadcom 57810S dual port 10GbE SFP Converged Network Adapter Broadcom 57810S k dual port 10GbE KR Blade Converged mezzanine card Broadcom 57810S k dual port 10GbE KR Blade Converged Network Daughter card and Broadcom 57800S quad port SFP RJ45 2x10
133. want to change on the Seite et ard then select ta value on the night BROADCOM June 07 2012 e 2CS57712 SWUM103 R OU UE 1 50 C4 BOE CE ZU UU OC T Uc 8 8 200 2L UL 2L UE Switch Independent Partioning Setup Guide Page 33 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning f Note In Switch Independent Partitioning mode MS Window s TCP Chimney Offload or TOE functionality can be enabled or disabled on a per partition granularity in this Advanced Properties control window and in BACSA s NDIS Advanced Properties control window The number of iSCSI connections that are being offloaded by the hardware can be seen by using the iscsicli sessionlist DOS command which lists the Initiator Name with an iSCSI offloaded connection entry beginning with BO6BDRV and a non offloaded connection beginning with Root The number of currently active Windows TOE connections can be viewed by using the netstat not command in a DOS window gt Users Administrator netstat not Active Connections Proto Local Address OFF load State TCP i i i 24 49 38i OF JS loaded TCP 1 1 1 24 49382 OFF loaded TCP NK ee KE EE KIK DFF loaded TCP i i i 24 49304 Of F loaded TCP 1 1 1 24 493685 DFF loaded TCP 1 1 1 24 49366 Of d loaded TCP i i i 24 49307 Of F loaded TCP ER WK WE EE KRISE OFF loaded TGP 1 1 1 24 49389 Of F loaded TGP 1 1 1 24 49318 OF fF loaded
134. ws P1i and P1e will share it s allocated 5 Gbps or 2 5 Gbps each e When partition P3 P3e starts to send at t3 all three partitions will be allocated 1rd of the available bandwidth P3e will received 3 3 Gbps P2e will receive the same allocation of 3 3 Gbps and P1i an P1e will approximately split it s partition s bandwidth for 1 65 Gbps each e But when P4e starts to send Ethernet traffic at t4 it will take almost all of the 10 Gbps bandwidth regardless of the bandwidth needs of the other three partitions four traffic flows P1i and P1e will each get approximately half of 1 97 of the available bandwidth or 0 05 Gbps while P2e and P3e will receive 0 1 Gbps and P4e will take 9 7 Gbps e When P4i starts to send Ethernet traffic at t5 it will take half of the allocated bandwidth for partition P4 Therefore P4e will drop to 4 75 Gbps and P4i will start at 4 75 Gbps The other three partitions four traffic flows will be unaffected P1i and P1e will each get approximately half of 1 103 of the available bandwidth or 0 05 Gbps while P2e and P3e will receive 0 1 Gbps and P4e will take 9 7 Gbps e When P1i stops sending traffic at t6 it s freed up 0 05 Gbps bandwidth will be reallocated to the other traffic flows according to their relative bandwidth weight settings The same is true for when P1e at t7 P2e at t8 and P3e at t9 stop sending traffic BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM10
135. y of the traffic flows FCoE Offload is not available in non DCB mode so two iSCSI Offload protocols are used here Table 4 Non DCB Equal Oversubscription Relative Bandwidth Maximum Port Partition Weight Bandwidth Protocol Plot Color Port O Partition 1 P1i 0 100 iSCSI Offload Green Port O Partition 1 P1e 0 100 Ethernet Brown Port O Partition 2 P2e 100 Ethernet with Purple TOE Port O Partition 3 P3e O 100 Ethernet Yellow Port O Partition 3 P4e 0 100 Ethernet with Blue TOE Port O Partition A P4i 0 100 iSCSI Offload Red BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 95 57712 k 578xxS Software User Manual Examples The following plot shows how all of the partitions would share a ports available send bandwidth Traffic flows on the same partition such as P1i and P1e are expanded in to their own bandwidth trace for ease of understanding The send traffic flows are independent in each partition and the individual traffic type flow rate is balanced with each of the other traffic type flow rates when traffic demands exceed the available bandwidth Starting at tO the first partition s iSCSI Offload traffic flow P1i initially takes 10096 of the available port s TX send bandwidth when an iSCSI test application is flooding that port by itself When P1 s L2 Ethernet P1e starts to send at t1 both stabilize to half of the bandwidth or 5 Gbps each ev
136. y that displays the FCoE HBA port information FCInfo is also available on Solaris and Windows OS s Another useful tool for FCoE is the FCoE Administration Utility fcoeadm In order to use this tool the user must install on the host the FCoE Utils User Space Package from the Open FCoE Project website http www open fcoe org git p fcoe fcoe utils git a summary BROADCOM Switch Independent Partioning Setup Guide June 07 2012 e 2CS57712 SWUM103 R Page 56 57712 k 578xxS Software User Manual Configuring Switch Independent Partitioning VMWare ESX ESXi 4 1 VMWare ESX ESXi 4 1 shows the respective device protocols that were enabled in USC if the NetXtreme II device drivers are installed Go to the Dell Driver Download web site http support del com for the latest device Firmware if not already installed Go to the VMWare web site http downloads vmware com for the latest device drivers and insure they are installed on your system In VMWare the Ethernet Protocol is always enabled on all eight partitions VMWare ESX ESXi 4 1 does support the iSCSI Offload Protocol in both SF and Switch Independent Partitioning mode but does not support the FCoE Offload Protocol in SF or Switch Independent Partitioning modes in VMWare ESX ESXi 4 1 drivers 1 70 34 v41 1 bnx2x 2 70 1k v41 2 bnx2i and Dell firmware version 7 0 47 or later r Note VMWare ESX ESXi 4 1 only supports a maximum of four 10GbE ports see http www vmware com pdf vsphe

57712-k Software User Manual

Contents

Download Pdf Manuals

Related Search

Related Contents