Home

57712-k Software User Manual

1. The following properties are awaldable for this network adapter Ckck the property you want to change on the eiii lt and then select ts value on the night BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 21 57712 k Software User Manual Configuring NPAR Note In NPAR mode MS Window s TCP Chimney Offload or TOE functionality can be enabled or disabled on a per partition granularity in this Advanced Properties control window and in BACS4 s NDIS Advanced Properties control window The number of currently active Windows TOE connections can be viewed by using the netstat not command ina DOS window ETA AA rl not Active Connections Proto Local Address OFf load State TCP 1 1 1 24 49361 DEF loaded TCP 1 1 1 24 493B2 OFF loaded TCP 1 11 1 24 49383 OF F loaded TCP 1 1 1 24 49364 OF F loaded TCP 1 1 1 24 49365 OFF loaded TCP 1 1 1 24 49366 Of F loaded TCP DK WK WE EE KIC e DFF loaded TCP 1 1 1 24 49358 DFF loaded TCP 1 1 1 24 49389 Of T loaded TGP 1 1 1 24 49318 OF T loaded TCP 1 1 1 24 49311 OFF loaded TCP 1 1 1 24 49312 OF T loaded TCP 1 1 1 24 49313 OF F loaded TCP 1 1 1 24 49314 OF loaded TCP 1 1 1 24 49315 OF loaded TCP 1 1 1 24 49316 DFF loaded 3 3 3 24 49289 3 3 3 24 49291 3 3 3 24 49293 3 3 3 24 49294 InHost BROADCOM December 8 2011 e 2CS57712 SWUM102 R Foreign ddress 1 1 1 25 5H1H 1 1 1 25 5H12 1 1 1 25 5HH1 1 1 1 25 5H13
2. e When P3e starts to send at t3 P1i is still unaffected and remains at 5 Gbps The three L2 Ethernet traffic types will split their 5096 of PGO s share between themselves which is 1 65 Gbps each takes 1 3rd of 5 Gbps e When P4e starts to send at t4 the four Ethernet traffic flows take 1 4th of PGO s bandwidth 1 25 Gbps each while P1i is still unaffected and remains at 5 Gbps e When P4i starts to send at t5 the four Ethernet traffic flows remain the same but the two iSCSI traffic flows split PG2 s allocated bandwidth 2 5 Gbps each e Then when P1i stops sending at t6 the traffic flows in PGO are unaffected while P4i s share increases to all of PG2 s allocated bandwidth of 5 Gbps e Aseach of the traffic flows stops sending in PGO the traffic flows of the remaining member s of PGO equally increase their respective shares to automatically occupy all of the available bandwidth remain in PGO until all of PGO s Ethernet flows stop At t7 there are three active PGO flows P2e P3e and P4e so each gets 1 3rd of PGO s 5 Gbps or 1 65 Gbps At t8 there are two active PGO flows P3e and P4e so each gets Y of PGO s 5 Gbps or 2 5 Gbps At t9 there is only one active PGO flow P4e so it gets all of PGO s bandwidth or 5 Gbps Through all of this the Lossless iSCSI flow of P4i remains at 5 Gbps since it takes all BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 60 57712 k Software User Manual Exampl
3. 7 455 871 680 584 8 3 57712 NP4 Port 0 Partition 3 5 5 5 20 Status B 4 57712 NP6 Port 0 Partition 4 7 7 7 20 Status x BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 32 57712 k Software User Manual Configuring NPAR The previous Link Speeds are the result of the following USC Maximum Bandwidth settings and show its 100 Mbps 1 configurable granularity L UNIFIED SERVER CONFIGURATOR LIFECYCLE CONTROLLER ENABLED Broadcom NetXtreme II 10 Gigabit Ethernet 00 10 18 6F D2 A4 Global Bandwidth Allocation Menu Partition 1 Relative Bandwidth Weight U Partition 1 Maximum Bandwidth gt Partition 2 Relative Bandwidth Height U Partition 2 Maximum Bandwidth 1 Partition 3 Relative Banduidth Height U Partition 3 Maximum Bandwidth 33 Partition 4 Relative Banduidth Height U Partition 4 Maximum Bandwidth 46 Configure maximum bandwidth Valid range 1 100 percent BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 33 57712 k Software User Manual Configuring NPAR Device PCle Bus Location The PCle interface Location Bus and Device position numbers are the same for both ports and all eight of the partitions on those ports The only PCle interface location values that are different are the Function numbers In non partitioned Single Function SF mode you would only have functions 0 and 1 In partitioned NPAR mode you have functions O through 7 wit
4. Key Object Measurement Rollup Units Latest Maximum Minimum average VM 5 Data transmit rate Average KBps 64 116517 0 10406421 VM 6 Data transmit rate Average KBps 232548 o 13435 VM 5 NP 1 0 10 1G VM 6 NP 3 0 20 2G VM 7 Datatransmit rate Average KBps 349189 o 21630931 VM 7 N P 5 0 30 3G VM 8 N P 7 0 40 4G 0 VM 8 Datatransmitrate Average KBps 465204 197157 58 BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 45 57712 k Software User Manual Configuring NPAR Setting MTU Sizes e Setting MTU Sizes in Windows e Setting MTU Sizes in Linux e Setting MTU Sizes in VMWare ESX ESXi 4 1 Note In all cases the connecting switch port tnat a 57712 k NPAR port is connected to must have the switch s MTU size set to the largest MTU size of those four partitions of the port if the user wants to support all four partitions MTU size settings Additionally the remaining network that the traffic flows though must also support the desired MTU sizes for that sized frames to be used without being dropped truncated fragmented by the network Setting MTU Sizes in Windows The MTU size for each individual Ethernet protocol enabled partition can be independently set from Normal 1500 bytes up to Jumbo 9600 bytes in several places in Windows One place to set the Ethernet protocol enabled partitions adapter MTU size is in the Window s Networking Adapter Advanced Properties Jumbo Packet properties
5. 18TT Te88 6e7a8 54 Scope Link UP BROADCAST RUNNING MULTICAST MTU 9608 Metric 1 RX packets 27937 errors 8B dropped B overruns B frame 8 TX packets 48 errors dropped 80 overruns Carrier 0 collisions O txquevelen 1660 RX bytes 5030387 4 7 MiB TX bytes 9923 9 6 KiB Interrupt 169 Memory d 1899090 01TTTTTT Link encap Ethernet Hwaddr 00 10 18 88 E7 AA inet addr 18 1 1 2801 Beast 16 255 255 255 Mask 255 0 0 0 inet6 addr fe80 210 18ff Te88 e7aa 54 Scope Link UP BROADCAST RUNNING MULTICAST MTU 965080 HMetric l RX packets 275080 errors dropped 6 overruns 6 frame 6 TX packets 36 errors 0 dropped overruns 0 Carrier 0 collisions B txquevelen 1000 RX bytes 4946656 4 7 MiB TX bytes 9456 9 2 KiB Interrupt 225 Memory d28888880 d2TTfTTTT Link encap Ethernet HWaddr 808 18 18 88 E7 AC inet addr 18 1 1 202 Bcast 10 255 255 2955 Mask 255 0 0 0 inete addr Te88 210 IBTT TeBB e ac 654 Scope Link UP BROADCAST RUNNING MULTICAST MTU 9600 Metric 1 RX packets 27034 errors 8 dropped overruns 8 frame 8 TX packets 42 errors 6 dropped overruns 8 carrier 0 collisions 0 txqueuelen 1000 RX bytes 4860331 4 6 MiB TX bytes 10027 9 7 KiB Interrupt 225 Memory d38000800 d3TTTTTT Link encap Ethernet HwWaddr 00 10 18 88 E7 AE inet addr 18 1 1 203 Bcast 18 255 255 255 Mask 255 0 0 0 inet6 addr feB88 218 18ff fe8B8 e7ae 54 Scope Link UP BROADCAST RUNNING MULTICAST MTU 9660 Metric 1 RA packets 26782 errors O dropped over
6. Broadcom BCMH57712 Netttreme II 10 GigE Dual Port 5 T7 General Advanced Dever Detads Power Management adapter Click the property you want to change on the roca iit and then select ts value on the mght BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 46 57712 k Software User Manual Configuring NPAR Another place to set the MTU size is in the BACS4 NDIS device Configurations page Broadcom Advanced Control Suite 4 m m De Mew Acton He Context Tools Tears iSCSI Help fav lls 7 I Advanced ps E ES Adapter 1 BCM5709 CO ES Adapter B0M5709 C0 RN Adapter3 BCMS7711 AQ Adapters 57712 A1 EF 4 Pott E 0090 Broadcom BCM57712 Hez K ene I 10 GigE Multi Function 250 SS 0012 Broadcom BCM57712 NetXireme H 10 GigE Multi Function iSCSI x f SS Portal 1 1 1 4 SS Portal fe80 210 18fF8 F6F d3e5 E 0027 Broadcom E0M57712 Netxtreme H 10 Gg NDIS VED Ghent 550 7 10051 Broadcom BCM57712 Netklreme I 10 GgE Mult Function 251 E 0053 Broadcom BCM57712 NetXireme IT 10 GigE Mult Function 253 2 Bal Broadcom BCMS7712 NetXtreme H 10 GigE NDIS VED Cent 253 B 0054 Broadcom BCM57712 NetXtreme I 10 GigE Muli Function 254 MP L s BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 47 57712 k Software User Manual Configuring NPAR The MTU size for each individual iSCSI Offload HBA protocol enabled p
7. Intel R PRO 1000 MT Network Connection Pro The following properties are available for this network adapter Chck the property you want to change on the left and then select ts value on the ight l Bante inter Frame Spacng Row Control Keen et Moderatio 16123 Bytes EN Moderation 4088 Bytes Intemupt Moderation Rate 50 Bytes Tz 4 EOM d Locally Zant kinsan Disabled 1500 bytes Humber of Coalesca Butters Priority amp VLAN Receive Butters 8014 Bytes Jumbo TCP Checksum Offload IPv4 ranami Bulters If you are using the VMXNET3 Virtual Adapter then change it s Advanced Properties Jumbo Packets setting to Jumbo 9000 BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 54 57712 k Software User Manual Configuring NPAR vmxnet3 Ethernet Adapter Properties Genera Advanced Driver Details Power Management A a Offload tagged traffic Offload TCP Options Priorty VLAN tag In a RHELVM the virtual adapter MTU sizes can be adjusted in the Ethernet Devices window from the Network Configuration GUI or command line using the ifconfig ethX mtu NNNN up command Ethernet Device General Route Hardware Device Nickname 57712 eth2 Activate device when computer starts gt Allow all ysers to enable and disable the device Enable IPy amp configuration Tor this interface Autor ac ally obtain IP address settings with 2 Statically set IP addresses Manu
8. P4e starts to send at t4 it also only takes it s partitions subscribed 2 5 Gbps and does not affect any of the other partition s sending traffic flows e When P4i starts to send at t5 it will share partition P4 s 25 with P4e Each is allocated 1 25 Gbps and the other partitions are unaffected e When P1i stops sending at t6 it will release it s 12 5 share of the bandwidth and the other remaining partition P1 traffic flow P1e will increase to 2 5 Gbps while the other traffic flows are unaffected e When P1e stops sending at t7 there will only be three partitions but each is still assigned only 2596 of the overall bandwidth The other traffic flows P2e P3e P4e and P4i will not change e When P2e stops sending at t8 again there will be no change to the other traffic flows e When P3e stops sending at t9 there will still be no change to the other traffic flows BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 71 57712 k Software User Manual Examples e When P4e stops sending at t10 the remaining traffic flow on P4 P4i will absorb the freed 12 5 of the partition P4 s allocated bandwidth and will increase to 2 5 Gbps e Each partition s flows on the same port are logically isolated from the others as if they were on separate ports and a partition s send flows stopping or restarting will not affect its fellow partition s send traffic flows except where the flows are on the same partition and then they
9. SERVER CONFIGURATOR LIFECYCLE CONTROLLER ENABLED Hardware Configuration advanced Configuration Broadcom Netxtreme II 10 Gigabit Ethernet 00 10 18 6F D2 A4 Broadcom Netxtreme II 10 Gigabit Ethernet 00 10 18 6F D 4 From the Broadcom Main Configuration Page select Device Configuration Menu to turn on or off the NPAR mode of operation for the selected devices k DELL UNIFIED SERVER CONFIGURATOR LIFECYCLE CONTROLLER ENABLED Broadcom Netxtreme II 10 Gigabit Ethernet 00 10 18 6F D2 A4 Broadcom Main Configuration Page MBA Configuration Menu iscsI Boot Configuration Menu NIC Partitioning Configuration Menu PCI Device ID 1663 Bus Dev Func 06 00 00 Link Status UP BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 8 57712 k Software User Manual Configuring NPAR 5 From the Device Configuration window select either Enabled NPAR mode where each port has four functions or Disabled Single Function SF mode where each port has one function for the device s NPAR mode d DELL UNIFIED SERVER CONFIGURATOR LIFECYCLE CONTROLLER ENABLED Broadcom Netxtreme II 10 Gigabit Ethernet 00 10 18 6F D2 84 Device Configuration Device MAC Addresses Broadcom Netxtreme II 10 Gigabit Ethernet 00 10 18 6F D2 A4 Broadcom Netxtreme II 10 Gigabit Ethernet 00 10 18 6F D2 A6 NIC Partition Enabled Disabled Enabled Enable Disable HIC Partition Function This setting enables disables HIC partitio
10. User Manual Configuring NPAR These Ethernet Protocol adapters in each partition are configurable like a normal port adapter in vSphere s Networking Configuration pages as shown below e EN A ETC a Configuration ML e MAA E E lt Hardware Health Status Virtual Switch vSwitch Virtual Machine Port Group I VM Network VMkemel Port L2 Management Network vmk 1 10 41 2 Virtual Switch vSwitch1 Ve ual Machine Dez Group Licensed Features LI VMK NIC 1 Time Configuration E 1 virtual machine s DNS and Routing VM41_P1_1 Authentication Services VMkemel Port Virtual Machine Startup Shutdown LJ VMK I 1 Virtual Machine Swapfile Location vmki 1141 1 Security Prafile System Resource Allocabon esse A Virtual Machine Pert Group LJ VMK NIC 2 1 virtual machine s VM41 P2 1 VK ee Port LJ VMK I 2 vmk2 2 2 41 1 Virtual Switch vSwitch3 Virtual Machine Port Group LI VMK NIC 3 E 1 virtual machine s VM41 P1 3 MEy az De BROADCOM December 8 2011 e 2CS57712 SWUM102 R Remove Properties Physical Acanters EH vmnic 1000 Full E Remove Properties Ng vmnice 10000 Full LJ Pineal AS EE mni 10000 Full L2 Pv Sa Adapers Ng vmnica 10000 Full LJ NPAR Setup Guide Page 43 57712 k Software User Manual Configuring NPAR VMWare s vSphere 4 1 or vCenter can be used to view a selected VM s Networking performance in the specific Host s Performanc
11. active on that port i e the 57712 k port is connected to a DCB compliant and enabled link partner e Only one Offload Protocol either iSCSI or FCoE can be enabled per single partition in NPAR mode e For simplicity using the first partition of a port for FCoE offload protocol is recommended since the FCoE port WWN will be the same for both SF and NPAR mode on the same port This will make your Fiber Channel Forwarder FCF switch configuration much simpler e For Windows operating systems you can have the Ethernet Protocol enabled on all some or none of the four partitions on an individual port simultaneously with any enabled offload protocols e For Linux OSs the Ethernet protocol will always be enabled even if disabled in USC e For simplicity we recommend always using the first two partitions of a port for any iSCSI offload protocols e For Windows OSs the Ethernet protocol does not have to be enabled for the iSCSI or FCoE offload protocol to be enabled and used on a specific partition e For VMWare ESX ESXi 4 1 in NPAR mode the host and hosted Virtual Machines VMs should only BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 13 57712 k Software User Manual Configuring NPAR connect to enabled Ethernet protocol adapters DELL UNIFIED SERVER CONFIGURATOR LIFECYCLE CONTROLLER ENABLED Broadcom Netxtreme II 10 Gigabit Ethernet o0 10 18 6F Dzz h 4 Partition 1 Ethernet Protocol Enabled
12. in the two PGs will almost act independently of each other when their aggregated traffic bandwidth demands exceed the available bandwidth e Starting at tO only P1f FCoE Offload is sending so it takes 100 or all of the 10 Gbps bandwidth e When Ple L2 Ethernet starts to send at t1 both flows stabilize to 5 Gbps each P1f in PG1 takes 5096 and P1e in PGO takes the other 50 e When P2e starts to send at t2 the traffic in P1f is not affected it remains at 5 Gbps due to it being in a different PG Both P1e and P2e will stabilize to 2 5 Gbps each P1e and P2e equally share PGO s portion of the bandwidth so they each get 2 5 Gbps ETS of 50 total 10G BW 1 2 e When P3e starts to send at t3 P1f is still unaffected and remains at 5 Gbps The three L2 Ethernet traffic types will split their 50 of PGO s share between themselves which is 1 65 Gbps each takes 1 3rd of 5G e When P4e starts to send at t4 the four Ethernet traffic flows take 1 4th of PGO s bandwidth 1 25 Gbps each while P1f is still unaffected and remains at 5 Gbps e When P4i starts to send at t5 the four Ethernet traffic flows plus the new SCSI traffic flow take 1 5th of PGO s bandwidth 1 Gbps each while P1f is still unaffected and remains at 5 Gbps e Then when P1f stops sending at t6 the five traffic flows in PGO now take all of the ports bandwidth so now their 1 5th of PGO s bandwidth doubles to 2 Gbps each the available bandw
13. oo Co Co 0 Partition 4 BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 34 57712 k Software User Manual Configuring NPAR The partition s MAC addresses interleave the two ports as do the function numbers see Table 2 Table 2 Port Function MAC Address Example Port 0 Partition 1 Function O MAC address A8 Port 0 Partition 2 Function 2 MAC address AC Port 0 Partition 3 Function 4 MAC address BO Port 0 Partition 4 Function 6 MAC address B4 Port 1 Partition 1 Function 1 MAC address AA Port 1 Partition 2 Function 3 MAC address AE Port 1 Partition 3 Function 5 MAC address B2 Port 1 Partition 4 Function 7 MAC address B6 One way to locate PCle information is to open the individual Network Connection s Properties Ez Network Con nections qe s 57712 4 be LE am mE SE rd 57712 NP Po rtlPartiton1 rtQPartrion rt lPartbonzg rt ParBbon2 rtlParbton3 Wi ln Pr oper Des BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 35 57712 k Software User Manual Configuring NPAR In the Properties window select the device Configure button In the NDIS client device properties you will find that connection s PCle bus device function location information For partitioned adapters locate the function number that provides the partition that this connection is connected The same can be done with Device Manag
14. set from Normal 1500 bytes up to Jumbo 9600 bytes This MTU size change will affect both regular L2 Ethernet and iSCSI software non offload pathway generated traffic Unlike other Linux OS s you can not directly adjust the MTU size using the ifconfig command Instead you must use various ESX 4 1 esxcfg commands The Ethernet protocol enabled partition s adapter MTU size is adjusted using SSH command line for example List all current vSwitch s information using the esxcfg vswitch 1command Modify a specific vSwitch s MTU size with the following command this modifies vSwitch1 which is in port group VMK 1 to 9000 bytes esxcfg vswitch m 9000 vSwitch1 The following command will list all of the current VMKernel NIC settings esxcfg vmknic 1 BROADCOM NPAR Setup Guide December 8 2011 s 2CS57712 SWUM102 R Page 53 57712 k Software User Manual Configuring NPAR Modify the VMKernel NIC s MTU size using the following command This modifies vmnic1 which is in port group VMK 1 to 9000 bytes esxcfg vmknic m 9000 VMK I 1 Note VMWare ESX uses the term vmnic instead of the typical Linux term eth in the command line ES Finally don t forget to set the associated VM s virtual adapter setting as desired In a Windows Server 2008 R2 VM the Network Connection Advanced Properties Jumbo Packet setting would be either 9014 Bytes or DISABLED for standard size 1500 Bytes if the Virtual Adapter is the default E1000
15. to send at t1 the two active traffic flows on the same partition share its 8 Gbps bandwidth for 4 Gbps each ETS does not take effect since the traffic in PGO and PG1 are still less than the amount prescribed by their respective ETS values e When the third traffic flow P2e starts sending at t2 it only takes its partitions maximum bandwidth allowed which is 1 Gbps Partition P1 s two traffic flows are unaffected and the unassigned 1 Gbps bandwidth remains free e When the fourth traffic flow P3e starts sending at t3 it only takes its partitions maximum bandwidth allowed which is 1 Gbps Now the first partition s two traffic flows readjust so that PG O does not get more than 50 of the overall bandwidth i e PGO s P1e P2e P3e 4096109641096 which is greater than 50 The Ple traffic flow is reduced to 30 or 3 Gbps and the P1f traffic flow in PG1 is adjusted upwards to 50 or 5 Gbps e When P4e starts to send traffic at t4 it equally shares PGO s ETS assigned bandwidth with P1e P2e and P3e but since P2e and P3e use only 1 Gbps of their allocated 1 25 Gbps 5 Gbps 4 equally weighted traffic flows this leaves 3 Gbps free 5 Gbps available 2 GBps assigned to P2e and P3e for the other two traffic flows P1e and P4e and they are both allocated 1 5 Gbps each 3 Gbps 2 equally weighted traffic flows P1f is in PG1 so it is unaffected and keeps sending at 5 Gbps e When P4i starts to send traffic at t5 it also equally s
16. will only take the freed bandwidth for their respective partition BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 72 57712 k Software User Manual Examples Mixed Fixed Subscription and Oversubscription Example This example shows partitions with all of the relative bandwidth weights set the same but with the partitions partially oversubscribing the available bandwidth unequally Two of the partitions are set to use 10 or 1 Gbps each of bandwidth and the other two the ones with the hardware offload protocol s enabled are set to use 80 or 8 Gbps of the connection s bandwidth thus oversubscribing the connection by 80 Table 10 Non DCB Mixed Fixed Subscription and Oversubscription Relative Bandwidth Maximum Port Partition Weight Bandwidth Protocol Plot Color Port O Partition 1 P1i 0 80 iSCSI Offload Green Port O Partition 1 Ple 0 80 Ethernet Orange Port 0 Partition 2 P2e 0 10 Ethernet Purple Port O Partition 3 P3e O 10 Ethernet Yellow Port O Partition 4 P4e 0 80 Ethernet Blue Port 0 Partition 4 P4i 0 80 SCSI Offload Red This is a combination example of a fixed subscription three of the partitions sum to 100 but all four sum to 180 When all four or at least the two larger partitions are running traffic they share the space with each other up to their partition maximum bandwidth values otherwise they act as if they are independent connections e
17. 0 ping statistics 3 packets transmitted 3 packets received 0 packet lost round trip min ave max 0 177 0 181 0 185ms If it does not work you will see a failure message root esx41 ping d c 3 I eth1 s 8972 192 168 20 10 PING 192 168 20 10 192 168 20 10 from 192 168 20 101 eth1 8972 9000 bytes of data sendto failed Message too long sendto failed Message too long sendto failed Message too long 192 168 20 10 ping statistics 3 packets transmitted packets received 100 packet lost You should also check if the desired VMs can send Jumbo Frames all the way to their various end points using the applicable OS ping commands described earlier in this document Examples e Equal Oversubscription Example e Partitioned Oversubscription Example e Weighted Oversubscription Example e Oversubscription With One High Priority Partition Example e Default Fixed Subscription Example e Mixed Fixed Subscription and Oversubscription Example e Mixed Weights and Subscriptions Example BROADCOM NPAR Setup Guide December 8 2011 s 2CS57712 SWUM102 R Page 56 57712 k Software User Manual Examples Note All bandwidths given in these examples are approximations Protocol overhead application send rate variances and other system limitations may give different bandwidth values but the ratio relationship between the send bandwidths of the four partitions on the same port should be similar to the ones gi
18. 1 1 1 25 5HH8 1 1 1 25 5BB3 1 1 1 25 5H14 1 1 1 25 5HBHBH 1 1 1 25 548B84 1 1 1 25 5HHB6 1 1 1 25 5HH9 1 1 1 25 5H15 1 1 1 25 5HHB2 1 1 1 25 5B11 1 1 1 25 5HB5 1 1 1 25 5HH7 3 3 3 24 49293 3 3 3 24 49294 3 3 3 24 4928 3 3 3 24 49291 State ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED ESTABLISHED NPAR Setup Guide 57712 k Software User Manual Configuring NPAR Broadcom Advanced Control Suite 4 BACS4 The BACS4 utility provides useful information about each network adapter that is installed in your system including partitioned adapters BACS4 enables you to perform detailed tests diagnostics and analyses as well as allows you to view and modify various property values and view traffic statistics for each adapter including other vendor devices BACS4 allows the enabling and configuring of both ports NPAR flow control protocols Relative Bandwidth Weights Maximum Bandwidth settings The following figure shows the per partition NPAR settings see Figure 6 This is where BACS4 can enable or disable NPAR mode This is also where BACS4 controls the NPAR per port IEEE 802 3x Link Level Flow Control settings used when DCB s PFC is disabled enabled protocols Ethernet or iSCSI or FCoE the Relative Bandwidth Weight values and the M
19. 5 255 0 0 im NET Contro nee Sir amp TT posa Broadcom BCM577212 etr ere D 10 Gig Mult Funcion 154 Er kn 2011 05 com broadcoesci2 ZI 0027 Broadcom BCM5S7712 Netxtreme H 10 Gag INDIS VBD Chert 54 a Mare O O ign 2011405 c0m broadcom tptita aml UD Sa wos arbre BCMS7712 Metureme D NOS 255 id FEFFFADO 19072008 FEFPFASO 144BC 500 Yan 7372080416256 SS Portal 1 1 1124 Portal Me 210 taeda ened SPE 0031 Broadcom BCM57712 Netxtreme IT 10 GoE NOES VED Clent 259 e T eent Broadcom BCM5S7712 HetUrese TI 10 GE Mus Funcion 260 H 0033 Broadcom BCM57712 Netxtreme TL 10 GagE NOIS VED Ghent 250 amp f 0051 Broadcom BCM57712 MetUreme D 10 GE Mult Funcion 261 E 0014 Broadcom BCM57717 NetXtreme TL 10 GigE Multi Funcion 505 Adapter 251 CH 0034 Broadcom BCM57712 Netxtreme I 10 Gage NDIS WED Cent 51 I Pert2 372215682580 POLI sent 5245555 Deet errors IL BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 29 57712 k Software User Manual Configuring NPAR The following shows the per partition iSCSI device per Portal traffic statistics E ES Adapter 1 CHE CD REM Adapter BOM5709 CD X OE Adapters BCM5709 CD E ES Adapters BOM57711 40 MES Adapters BCM57712 A1 Port m W 0054 Broadcom BCM57712 Netxtreme EL 30 GigE Multi Function 254 27 pity Broadcom 50457712 NetUreme D 10 Gage DIS VEO Clent 254 m PF poss Broadcom BCM57712 NetXtreme EL 10 GE Multi Funch
20. 712 NetXtreme I 10 GigE Multi Function SCSI Adap Pots LLL qm jee 2010 20 com broadocm tgt 15b ram 15b a ari 2011 05 com brasdcom bgt L n ram las LJ 2011 05 c0m braadoomas bat L rg 16 SU Portal Fe60 210 wh feshidacs E 0027 Broadcom BCMS7712 HetX reme IE 10 Gage NDIS VED Chert 550 0051 Braadenm BCM57712 Het rere 10 GE Mulsi Funcica SEI E PE 0053 Broadcom BCM57712 NetXtreme IL 10 GigE Muls Function 1 53 af 0030 Broadcom BCMS7712 Netktreme IT 10 GE NDIS VED Cent 53 e D 0054 Broadcom BCIAS7712 NetXtreme D 10 GgE Mul Function 2 54 amp ES pos Broadcom BCMS7712 Netitreme If 10 GE Mult Function FOE Adap E LO dT 203 8 H E H m BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 28 57712 k Software User Manual Configuring NPAR The per partition SCSI Offload device s VLAN MTU and IP address IPv4 and IPv6 settings are configured in BACS4 as shown below Broadcom Ach ap ed Control Site 4 IE Ej Fie Wew Acton Piter Context Toole Tears SCSI Heb m ES Adapter GOM5709 CU E ES Adapter 8045709 00 E ES Adapters BCMS7711 AQ Er E Adapters 57712 A1 Portl 2 PF 0050 Broadcom ECM57712 NetXireme D 10 GgE Mus Funcion 750 EE 0012 Broadcom BCMS7712 hettreme TI 10 GigE Mul Function iSCSI Adap S Portal 1 1 1 4 i in 2020 10 com brnadeom tgt 15 78 15a ma CARMEL sam kee nam em hb 1 Ek rmm 1 LLL 25
21. 9 rades BOM 77 Metteg E 10 Gef PE arte e Lex 00 Breles ACME PLE Meme E 15 Gl CES datar ar ee DCN Brogadope BACHES Mr TFS E E OS B Chana si Y E dape ose AO gt ES lie ee AO BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 27 57712 k Software User Manual Configuring NPAR The following shows the per port DCB protocol statistics Broadcom Advanced Control Suite 4 Eeer tirer ere Statistics Diagnostics UCET oe Ma Bg wre T REN Adapter 1 BCMS709 CO DCEX Frames Sent FL GES Adapter B0M5709 CO DCEX Frames Received E REM Adapter3 BCMS7711 A0 E ES Adapters 57712 A1 Porti E FF 0050 amp osdcom BCM57712 NetXtreme 11 10 Gage Multi Function 250 0012 Broadcom BCM57712 NetXtreme H 10 GigE Multi Function iSCSI Adapter 21 0027 Broadcom BCM57712 NetXtreme TT 10 GigE NDIS VBD Clent 550 T Ted Broadcom BCM57712 NetXtreme Il 10 GigE Multi Function 251 FF 0053 Broadcom BCM57712 NetXtreme ll 10 GigE Multi Function 253 S 0030 Broadcom BCM57712 Nettreme H 10 Gig NDIS VEO Client 553 PF 0054 Broadcom BCMS7712 NetXtreme TT 10 GigE Multi Function 254 2 0016 Broadcom BCM57712 NetXtreme I 10 GigE Multi Function FCoE Adapter PO 0031 Broadcom BCM57712 Netxtreme H 10 GigE NDIS VED Cieni 254 Port E Z Adapters BCME7711 AG E ES Adaptera 57712 A1 EF Port e D I 0050 Broadcom 8CM57712 NetXtreme TI 10 GE Mus Funcion 250 ES 0012 Broadcom BCMS7
22. ADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 12 57712 k Software User Manual Configuring NPAR F f Note When NPAR mode is first enabled or after a reset the default values for all four partitions is Relative Bandwidth Weight 0 and Maximum Bandwidth 25 See Examples on page 56 for more details on both the Relative Bandwidth Weight and Maximum Bandwidth settings affect traffic flow in DCB and non DCB modes of operation 1 Return to the previous NIC Partition Configuration window to change any of the four partitions protocol settings by selecting the specific partition here UNIFIED SERVER CONFRSURATOR LIFECYCLE CONTROLLER 2 Inthe specific Partition window select which protocols it will support and also view the partitions assigned Networking MAC address the Windows used iSCSI MAC address the FCoE FIP MAC address the FCoE Node WWN and FCoE Port WWN values BACS4 can also be used to control these settings Protocol selection follows these rules e A maximum of two iSCSI or one FCoE and one iSCSI Offload Protocols HBA can be enabled over any two of the four available partitions of a single port e The FCoE Offload Protocol is only available if DCB is also enabled and active on that port i e the 57712 k port is connected to a DCB compliant and enabled link partner e The iSCSI Offload Protocol can function without DCB but if DCB Lossless iSCSI iSCSI TLV is required then DCB must be enabled and
23. AX packets 376 errors 8 dropped overruns Trame 8 TX packets errors dropped 8 overruns carrier 6 collisions txquevelen 1006 RA bytes 73184 71 4 KiB TX bytes 6 6 b Interrupt 225 Memory di800008 d3TTTTTT BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 50 57712 k Software User Manual Configuring NPAR Link encap Ethernet HWaddr 00 10 18 88 E7 AE inet addr 18 1 1 203 BHcast 18 255 255 255 Mask 255 0 0 0 inet addr fe80 210 18TT TeB8 e UP BROADCAST RUNNING MULTICAST RS packets 373 errors dropped 8 overruns 8 frame 8 TX packets errors dropped overruns G carrier collisions 8 txquevelen 1000 RA bytes 72266 70 5 KiB TX bytes B 8 8 b Interrupt 264 Memory 0450060060 04T TT TTT Link encap Ethernet HWaddr QD 30 15 69 EI 66 inet addr 18 2 2 200 Bcast 106 255 255 255 Mask 255 6 6 6 inet addr fTe80 210 18T7TT Te88 e7b0 64 Scope Link UP BROADCAST RUNNING MULTICAST J DI Metric Ax packets 371 errors dropped overruns 8 frame 6 TX packets 6 errors dropped overruns G carrier collisions 8 txquevelen 1000 RA bytes 7213B 6 4 KiB TX bytes B 8 8 b Interrupt 2804 Memory d58068000 d5fTTTTTT Link encap Ethernet HWaddr 88 10 IB BB E7 B2 inet addr 10 2 7 201 Bcast 10 255 255 255 Mask 255 6 6 6 inete addr 7e30 210 1011 1089 07b2 64 5cope Link UP BROADCAST RUNNING MULTICAST MII 700 T Metric l RA packets 336 errors O dropped overruns frame
24. BCM57712 NetXtreme 1I 10 GigE NDIS VBO Client 55 E de Broadcom BCM57712 NetXtreme II 10 GigE Nols veo cient a 56 an Broadcom BCM5709C NetXtreme IT GigE 46 K Broadcom BCMS5709C NetXtreme IT GigE 347 Be Broadcom BCM5709C Netxtreme inus 48 a Broadcom BCM57712 Netxtreme II 10 GigE Mult Function 51 BE Broadcom BCM57712 NetXtreme II 10 GigE Multi Function 52 BE Broadcom BCM57712 Netxtreme II 10 GigE Multi Function 53 BB Broadcom BCM57712 NetXtreme II 10 GigE Multi Function 54 BE Broadcom BCM57712 NetXtreme TI 10 GigE Multi Function 255 SC Broadcom BCM57712 Netxtreme II 10 GigE Multi Function 56 LE Broadcom BCM57712 NetXtreme II 10 GigE Multi Function 57 BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 19 57712 k Software User Manual Configuring NPAR Right click the specific device and select its Properties to access some of the advanced features of the device A PCT Pus ll Intel R 5520 5500 x 58 1 0 Hub PCT Express Root Port 7 340 Ba Broadcom BCMS57712NP Netxtreme I 10 Gg Multi Funcion 41 GP Broadcom BCMS7712 NP NetXtreme IT 10 Gg NOES VEO Cent 241 A Broadcom BCMS7712NP Netitreme D 10 GigE Mult Function 242 M Broadcom BCMS57712NP Netktreme IT 10 GgE Multi Function 256 ca Broadcom BCM57712NP Netktreme IT 10 GgE Multi Funci n 257 Be Broadcom BCMS7712NP Netktreme IT 10 Gpe Multi Function 558 K Broadcom BCM57712 NP MetXreme D 10 Got OS VED Ceni asa I Broadcom BCMS7712NP NetX
25. Context Tools Teams iSCSI Help Fiter ALL view _ Ei NEM Hosts GO wIN RA3QUJRKO731 Microsoft Windows Network Connections These devices can be used by any application as ifthey were a separate adapter port They appear as separate Ethernet devices in Network Connections from the Windows Control Panel The following shows eight USC enabled Ethernet Protocol partitions as eight separate Ethernet network connections and these are arranged in port order 0 and 1 and partition order 1 through 4 Ey Network Connections R A 4 qt j e NIC Partitions sma s7742 NP eru SA Port 0 Port 0 Port O Partition 1 artili Partition3 Par ition 4 Port 1 K Me Ki Ke NIC Partitions lt e a 577 12 NP BTT12 MP 57712 MP 5771 2 NP Port 1 Part 1 Port 1 Port 1 Faribon 1 ParBtionZ Pattini Partition A BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 31 57712 k Software User Manual Configuring NPAR Each of these network connection devices can be accessed individually as if they where separate adapters The connection status shows the USC Maximum Bandwidth setting as the Speed of the connection B 1 57712 NPO Port 0 Partition 1 1 1 1 20 Status DIB 2 57712 NP2 Port 0 Partition 2 3 3 3 20 Status x General No network access Enabled 17 02 12 KZ Bytes 91 756 308 698
26. Dell Drivers When the 57712 k cNDC is first installed the SCSI and FCoE devices may not appear If the latest Dell driver is already present on the system the 57712 k will be identified and the NDIS personality protocol will be installed If the latest NetXtreme ll drivers are not present on the system go to the Dell driver download web site http support dell com support downloads under the specific Dell blade server platform and install the latest NetXtreme ll network drivers for your specific installation system 9 e Intel R 82801 PCI Bridge 244 ME Intel R processor DMI D130 Ma Intel R processor Miscellaneous Registers D158 E Me Intel R processor PCI Express Root Port 1 0138 zi iL Broadcom BCM57712 NetXtreme II 10 GigE 41 E d i Broadcom BCM57712 Nex treme II 10 GigE NDIS VBD Client 41 El A Broadcom BCM57712 MetXtreme II 10 GigE 42 d Broadcom BCM 5 7712 NetxXtreme II 10 GigE NDIS VBD Client 42 Figure 2 Windows Device Manager To enable the devices to be detected by the operating system start BACS4 and while in 57712 k SF mode select the port System Device Configuration Resource Reservations check the boxes to enable the applicable iSCSI and FCoE Offload Engines protocols and then click Apply Click Yes when the temporary network connection interruption warning displays and wait for the discovered devices to be installed by Windows no reboot is necessary while in SF mode BROADCOM NPAR Setup Gui
27. I 10 Gef Mula Funcion 54 The following shows the per partition FCoE device statistics Broadcom Advanced Control Suite 4 Fie W Acton Filter Context Tonos Tears SCSI Hek poeem o eco General e e Hosts E WIN RA3QRKD731 E ES Adapter1 BCM5709 CU F EE Adapter BCM5709 00 m ES Adapter3 BCM57711 AQ EE Adaptera 57712 Al E Parti PF 0050 Broadcom BCM57712 NetXtreme II 10 GigE Multi Function 250 0951 Broadcom BCM57712 Netxtreme 11 10 GigE Multi Function 551 a FT 10053 Broadcom BCMS7712 Netktreme 11 10 GigE Multi Function 553 3 0030 Broadcom BCMS7712 NetXtreme I 10 Gig NDIS VED Client 253 F 0054 Broadcom BCM57712 Netxtreme II 10 Gef Multi Function 254 ei 0016 Broadcom BOMS7712 Nebitreme TI 10 GE Mult Funcion FCoE Adapter B Port 20 00 00 10 18 6F D3 8D 1714693711 174551519204 af 0031 Broadcom BCM57712 NetXtreme II 10 Ga NDIS VED Chent 254 Pert2 BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 25 57712 k Software User Manual The following shows the per partition FCoE device connection information Broadcom Advanced Control Suite 4 Fabric m itt BCM5709 CO Adapter BCM5709 CO Adapter 3 BCM57711 AD EE Adsptera 57712 A1 O Fabric Priority UrkKeenaleeIniterval B Porti t 0050 Broadcom BCM 57712 Nettireme I 10 QigE Mult Funcion 250 DOSI Broadcom BCMS7712 NetXtrem
28. Il 10 GigE Ethernet devices Eth2 through Eth9 in this example Linux enumerates the partitions in order of the PCI function numbers which is slightly different from Windows where port 0 has functions partitions 0 2 4 6 which are eth 2 4 6 8 and port 1 has functions partitions 1 3 5 7 which are eth 3 5 7 9 Network Configuration Profile Help ig 4 B 2 3 New Edit Copy Activate Deactivate Devices Hardware IPsec ONS Hosts a4 E You may configure network devices associated with a g Physical hardware here Multiple logical devices can i be associated with a single piece of hardware Profile Status Device Nickname Type Active E eth1 Ethernet Active eth0 Ethernet NS Active 157712 NPO Ethernet Active 23 577 12 NP1 Ethernet M Active 2 3 712 NP2 Ethernet Active 6 57712 NP3 Ethernet Gi Active 3 57712 NP4 Ethernet Gi Active sil J 57712 NPS Ethernet M Active uji 4 57712 NP6 Ethernet Active g 3 712 NP Ethernet E U de gt un Lh i L LI ao ur ro Gi EL lt t F daa KJ KI I KI KJ T IK BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 38 57712 k Software User Manual Configuring NPAR Linux s ifconfig command shows the partition s eight Ethernet protocol devices and various statistics Link encap Ethernet HWaddr 00 10 18 88 E7 AB inet addr 18 1 1 208 Bcast 16 255 255 255 Mask 255 0 0 0 inetb addr TeS80 210
29. Software User Manual BROADCOM 57712 k y W se Third party information brought to you courtesy of Dell NIC Partitioning NPAR Setup Guide 2CS57712 SWUM102 R 5300 California Avenue e Irvine CA 92617 Phone 949 926 5000 e Fax 949 926 5203 December 8 2011 Revision History Revision Date Change Description 2CS57712 SWUM102 R 12 08 11 Updated e 716 4 software release by adding FCoE DCB VMWare ESX ESXi 4 1 and BACS4 information 2CS57712 SWUM101 R 3 31 11 Updated e NPAR in title and footer e NICPartition NIC Partitioned and NIC Partitioning to NPAR throughout the document e Single Function to SF throughout the document 2CS57712 SWUM100 R 03 15 11 Initial release Broadcom Corporation 5300 California Avenue Irvine CA 92617 2011 by Broadcom Corporation All rights reserved Printed in the U S A Broadcom the pulse logo Connecting everything and the Connecting everything logo are among the trademarks of Broadcom Corporation and or its affiliates in the United States certain other countries and or the EU Any other trademarks or trade names mentioned are the property of their respective owners 57712 k Software User Manual Table of Contents Table of Contents About This Document 5 feier 5 IOT o ECO O S Ds 5 Acronyms and ADD ata indi 6 TECNICA SUD DO uuu uu td 6 Configuring NPAR VERREM 7 Using the United Serve
30. T passed localhost bnx21 01 00 00 ISCSI_INIT passed BD Applications Places System 4 February Y 4 2011 Sun Mon Tue Wed Thu Fri Sat 1 E 3 a 5 amp 7 8 9 10 11 12 13 14 15 16 17 18 19 Ea 23 24 25 ze Bier pnx2 Loes 27 28 Version Current e 3320 lines 294 7 KB last update Sat Feb 26 16 10 44 2011 e D Fie manager 6 8 rootelocamostbmodu jvartogmessages moni O NNN O BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 40 57712 k Software User Manual Configuring NPAR For iSCSI also check the iscsi host folder to see if your iSCSI devices are present root localhost cd sys class iscsi host root localhost iscsi host pwd sys class iscsi host root localhost iscsi host 11 total 0 lrwxrwxrwx 1 root root 0 Jun 29 11 15 host3 gt devices pci0000 00 0000 00 09 0 0000 07 00 1 host3 iscsi host host3 lrwxrwxrwx 1 root root 0 Jun 29 11 15 host4 gt devices pci0000 00 0000 00 09 0 0000 07 00 0 host4 iscsi host host4 Each installed SCSI device will appear here The SCSI devices uses the bnx2i driver which can be checked for with the 1smod grep bnx2 command Additionally check the fc host folder to see if your FCoE devices are present root localhost cd sys class fc host root localhost fc host lt pwd sys class fc host root localhost fc_host 11 total 0 lrwxrwxrwx 1 root root 0 Jun 29 11 11 host8 gt d
31. TA packets 0 errors 0 dropped overruns 0 carrier 8 collisions B txquevelen 1000 RX bytes 63944 62 4 KiB TX bytes 6 6 b Interrupt 181 Memory d6800000 dG6TTTTTT Link encap Ethernet HWaddr 08 18 18 88 E7 B4 inet addr 10 2 2 2 62 Bcast lE 255 255 255 Mask 255 0 0 0 inet6 addr fe860 210 1811 Te88 e7b4 cope Link UP BROADCAST RUNNING MULTICAST Metric l RX packets 316 errors 8 dropped 8 overruns 0 Trame 6 TX packets errors dropped overruns B Carrier 6 collisions txquevelen 1666 RX bytes 60760 59 3 KiB TX bytes 6 6 0 b Interrupt 181 Memory d7B8B0B8 d7TTTTTT Link encap Ethernet HwWaddr 06 16 18 88 E7 B6 inet addr 10 2 2 203 Bcast 10 255 255 255 Mask 255 0 0 0 inet addr 1e80 210 1811 10688 e7b6 64 Scope Link UP BROADCAST RUNNING MULTICAST MTu 9688 Metric 1 RX packets 181 errors dropped overruns O frame TX packets 0 errors dropped O overruns carrier collisions 8 txquevelen 1000 RX bytes 33702 32 9 KiB TX bytes 6 6 b Interrupt 169 Memory d8500000 dB8TTTTTf Additionally the Ethernet and iSCSI protocol enabled adapter MTU sizes can be simultaneously adjusted in the Ethernet Devices window from the Network Configuration GUI if available in the desired version of Linux BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 51 57712 k Software User Manual Configuring NPAR Ethernet Device General Route Hardware Device Activate device when computer start
32. TTL 64 Reply from 192 168 20 10 bytes 8972 time lt ims TTL 64 Reply from 192 168 20 10 bytes 8972 time lt ims TTL 64 Ping statistics for 192 168 20 10 Packets Sent 4 Received 4 Lost 0 0 loss Approximate round trip times in milli seconds Minimum Oms Maximum Oms Average Oms If it does not work you might see the following reply if there is connectivity try 1472 byte standard frames to see if the non jumbo frame size is passing through C gt ping f 1 8972 192 168 20 10 Pinging 192 168 20 10 from 192 168 20 50 with 8972 bytes of data Packet needs to be fragmented but DF set Packet needs to be fragmented but DF set Packet needs to be fragmented but DF set Packet needs to be fragmented but DF set Ping statistics for 192 168 20 10 Packets Sent 4 Received 0 Lost 4 100 loss Setting MTU Sizes in Linux In Linux the MTU size for each individual Ethernet protocol enabled partition can be independently set from Normal 1500 bytes up to Jumbo 9600 bytes Both the Ethernet protocol and iSCSI Offload HBA enabled partition s adapter MTU size is adjusted at the same time using the ifconfig command ifconfig eth3 mtu NNNN up From above eth3 is the port identification of the specific 57712 k partition to adjust the MTU size The NNNN is the new size of the MTU for that partition and can be set from 1500 to 9600 bytes The following shows all eight partitions being set to different MTU v
33. The first partition s traffic flow P1i initially takes its designated 8 Gbps when the test application starts to transmit traffic at tO to that port by itself not expanding into the remaining unused 2 Gbps bandwidth e When the second traffic flow on the first partition P1e starts to send at t1 the two active traffic flows on the same partition share its 8 Gbps bandwidth for 4 Gbps each e When the third traffic flow P2e starts sending at t2 it only takes its partitions maximum bandwidth allowed 1 Gbps Partition P1 s two traffic flows are unaffected e When the fourth traffic flow P3e starts sending at t3 it again only takes its partitions maximum bandwidth allowed 1 Gbps Partition P1 s two traffic flows and the traffic flow on partition P2 P2e are unaffected e But when P4e starts to send traffic at t4 the condition is now oversubscribed Since P2e and P3e uses only 1 Gbps of their allocated 2 Gbps 10 Gbps 5 equally weighted traffic flows that leaves 8 Gbps free for the other three traffic flows The remaining traffic flows P1i Ple and P4e would then be allocated 2 6 Gbps each 8 Gbps 3 equally weighted traffic flows e But when P4i starts to send traffic at t5 it shares the available bandwidth within it s maximums with the other traffic flows P2e and P3e are still using only 1 Gbps of their allocated 1 6 Gbps 10 Gbps 6 equally weighted traffic flows which again leaves 8 Gbps free for the other four t
34. Xtreme I 10 Gage NDIS VED Clent 253 0054 Broadcom ECMS7712 Netxrmreme I 10 GE Mult Function 54 ES 0016 Broadcom BCMS7712 NetXtreme I 10 GigE Multi Function FCoE A C73 D031 Broadcom BCM57712 NetXtreme II 10 GigE NDIS VED Chert a54 In Windows each individual partition s Ethernet and iSCSI protocol enabled adapter MTU size setting can be different For example e Port O partition 1 Ethernet can be set to 9600 bytes e Port O partition 1 iSCSI Offload HBA can be set to 2500 bytes e Port O partition 2 Ethernet can be set to 1500 bytes e Port O partition 2 iSCSI Offload HBA can be set to 9600 bytes e Port O partition 3 Ethernet can be set to 3500 bytes e Port O partition 4 Ethernet can be set to 5500 bytes BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 48 57712 k Software User Manual Configuring NPAR In Windows use the ping command with the f option to set the Don t Fragment DF flag AND the l size option small to verify that Jumbo Frame support is configured throughout the desired network path i e ping f 1 8972 A B C D The unfragmentable ping packet size is the desired MTU size to be checked 9000 bytes minus the automatically added overhead 28 bytes or 8972 bytes C N gt ping f 1 8972 192 168 20 10 Pinging 192 168 20 10 from 192 168 20 50 with 8972 bytes of data Reply from 192 168 20 10 bytes 8972 time lt ims TTL 64 Reply from 192 168 20 10 bytes 8972 time 1ms
35. al 11 11 34 E in 2011405 com broadeom tgt 14a rm 146 21 0031 Broadcom BCMS7712 NeDibreme ET 30 GigE NOS VED Clenit 754 B on 0060 Broadcom 50457712 NetUreme TI 10 GigE Mul Funcion 750 0033 Broadcom BCMS7712 Nettineme ET 30 GE NOS VED Cherry 750 B E 006 1 Breadcom 50457712 Dar pS ee I 10 GigE Muls Funcion 51 00 14 Broadcom BCM57712 NetXtreme II 30 Gage Multi Funchon 505 Adapter 561 ll 0034 Ernacicom BCM57712 HeTXireme I 30 Gag NDES WED Cent 551 Port BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 30 57712 k Software User Manual Configuring NPAR The following shows the per device partition s current offload licenses Broadcom Advanced Control Suite 4 ae aur vew TE contratos DIE Resource Reservations E Licenses E Advanced ES Adapter B0M5709 C0 G3 Adapter3 BOM57711 AD SS Adapter 57712 Al EF 4 Peti 0007 Broadcom BCM57712 Netxtreme II 10 Gage NDIS VED Cent 45 Port2 0046 Broadcom BCM57712 NetXtreme T 10 GigE 246 EN 0012 Broadcom BCM57712 NetXtreme II 10 GE NDIS VED Client 246 An SCSI Discovery and Management Wizard is also included with BACSA This can be used to connect and manage the iSCSI hardware offload enabled initiator devices to various iSCSI targets as shown below Broadcom Advanced Control Suite 4 File View Action Filter
36. al IP Address Settings Address 11 112 subnet mask 255 255 0 0 Default gateway address Z Set MTU to EE Set MRU t In Linux use the following ping command to verify that Jumbo Frame support is configured throughout the desired network path BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 55 57712 k Software User Manual Examples Note The odd numbering for the pink packet size s NNNN which is the desired MTU size of 9000 bytes minus the 28 bytes of automatically added OVERHEAD but the outputted displayed size is plus 8 bytes which are the included ICMP header data bytes The Don t Fragment a k a DF flag d must also be set else the packet could be fragmented somewhere along the way and you would not know it The optional count c is the number of times you will send this ping You don t have to use the optional source interface command l a b c d or I devName unless you have multiple adapters that lead to the same target IP address and you want to check a specific one You can also use the ESX vmkping command root sieoral network scripts ping d c 3 I eth1 s 8972 192 168 20 10 PING 192 168 20 10 192 168 20 10 from 192 168 20 101 eth1 8972 9000 bytes of data 8980 bytes from 192 168 20 10 icmp seq 1 ttl 255 time 0 185 ms 8980 bytes from 192 168 20 10 icmp seq 2 ttl 255 time 0 177 ms 8980 bytes from 192 168 20 10 icmp seq 2 ttl 255 time 0 180 ms 192 168 20 1
37. alues BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 49 57712 k Software User Manual Configuring NPAR rootelocalhost root localhost ifconfig rootialocalhost amp ifconfig root localhost ifconfig rootelocalhost ifconfig root elocalhost ifconfig rootBlocalhost ifconfig root localhost ifconfig root localhost ifconfig rootelocalhost Link encap Ethernet HWaddr 00 10 18 88 E7 48 inet addr 16 1 1 266 Bcast 18 255 255 255 Mask 255 0 0 0 inet addr feB88 218 18TT Te88 ha Scope Link UP BROADCAST RUNNING MULTICAST TERM Metric l RX packets Aal errors 8 dropped 0 overruns Trame 8 TX packets errors dropped B overruns B carrier collisions 8 txquevelen 10060 RX bytes 47008 45 9 KiB TX bytes 6 8 b Interrupt 16 9 Hemnory 0d166560006 d1ff111fT Link encap Ethernet HWaddr 800 10 18 88 E7 AA inet addr 18 1 1 281 BHBcast 10 255 255 255 M Mask 255 8 0 8 inete addr feB8 2180 1BTT TesSB e73a b4 Scope Link UP BROADCAST RUNNING MULTICAST Metric 1 RX packets 418 errors dropped 8 overruns frame 6 TX packets errors dropped overruns carrier collisions 8 txquevelen 1000 RX bytes 8B164 78 2 KiB TX bytes 0 B B b Interrupt 275 Memory d2B8B8BB d2TTTTTT Link encap Ethernet Hwaddr 680 10 18 88 E7 AC inet addr 18 1 1 207 Bcast 10 255 255 255 Mask 255 0 0 0 inetb addr feB0 210 18TT Te88 e7ac 64 Scope Link UP BROADCAST RUNNING MULTICAST PATU J400 Metric l
38. ame is true for when P1e at t7 P2e at t8 and P3e at t9 stop sending traffic BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 69 57712 k Software User Manual Examples e Finally when P4e stops sending traffic at t10 P4i will take all of the available bandwidth for 10 Gbps e Whenever the fourth partition s bandwidth needs drop off the other actively sending partitions will equally increase their respective shares to automatically occupy all of the available bandwidth BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 70 57712 k Software User Manual Examples Default Fixed Subscription Example This is an example of the default partition settings that has all of the relative bandwidth weights set to 0 and the maximum bandwidths set to 2 5 Gbps Since the total of the maximum bandwidth values are set to exactly 100 i e never can reach an oversubscription situation the traffic flows in each partition will share that partition s bandwidth allocation with respect to it s overall maximum bandwidth ceiling and the relative bandwidth weights are never used Table 9 Non DCB Default Fixed Subscription Relative Bandwidth Maximum Port Partition Weight Bandwidth Protocol Plot Color Port O Partition 1 P1i 0 25 SCSI Offload Green Port O Partition 1 P1e 0 25 Ethernet Brown Port 0 Partition 2 P2e 0 25 Ethernet Purple Port O Partition 3 P3e O 25 Et
39. andi En E eco ae 3f Informason Property EA Target Information NPAR Setup Guide Page 26 57712 k Software User Manual Configuring NPAR The following shows the per port DCB protocol information Fos nash E IDE ae d Ple ges Acton Pier Const Took o ECH Me Him istrmases IER vea Sera F mars B Grp B E Brent Des B irene confgratore states emeng _ ER oia smeiemeee ccm EE Mane L NOMEA OU POR S ES Acte B CO X EL aate Y Brien TIS Z E Anon BOTAS HF G Broadenes eee E Gef rij m 0005 Broadcze BOS 7252 Mette E HD ap delete SEL TE pE mado BOM 7323 einen Z 15 Gl paf Adapter PiN pra 2 Brad Bee perpe E AS VERD Denm arii E de Ports SP oix Eras BOM wiae E 15 Gef 683 CG D Bree 245775 Pere 2 E DIOE kasuna mil EGS8 Broad BCS 7323 Mb Z LO GE CS Ada ES X ENS Aoeiet CMA e EH tipmi BOI ag 61234547 TTT n is E sa jk ka i T mg g 9 s BACS4 allows per port enabling and configuring of the DCB protocol default and initially advertised settings and port willingness to change to the received DCBx values as shown below E dde MAT AL SE darer d CMS AO tl DSL redas ROT 12 rech geng IL E G if mi We gi bikia BOM 7710 entree E 10 Cf SOL Adapter arit ba D mos Brida AN Mere EZ 12 E PE dabas X gt PE Breles BOL I Micra E 1 0 GgE OS BD Clap d S de ertr Ba m Door Be paires LTT oe IE TZ E Sl E 0
40. any products or data herein to improve reliability function or design Information furnished by Broadcom Corporation is believed to be accurate and reliable However Broadcom Corporation does not assume any liability arising out of the application or use of this information nor the application or use of any product or circuit described herein neither does it convey any license under its patent rights nor the rights of others connecting a BROADCOM sasIrrI mSsC s Trr P Q gt BROADCOM CORPORATION Phone 949 926 5000 5300 California Avenue Fax 949 926 5203 Irvine CA 92617 E mail info broadcom com 2011 by BROADCOM CORPORATION All rights reserved Web www broadcom com 2CS57712 SWUM102 R December 8 2011
41. are User Manual Examples TX send bandwidth when an iSCSI test application is flooding that port by itself e When P1 s L2 Ethernet P1e starts to send at t1 both stabilize to half of the bandwidth or 5 Gbps each even thought they are in the same partition they share the total available bandwidth e When P2e starts to send at t2 all three traffic flows P1i P1e and P2e will stabilize to 1 3rd of the bandwidth or 3 3 Gbps each they all equally share the available bandwidth e When P3e starts to send at t3 all four traffic flows P1i P1e P2e and P3e will stabilize to 1 4th of the bandwidth or 2 5 Gbps each effectively sharing the available bandwidth e When P4e starts to send at t4 all five traffic flows P1i P1e Pie P3e and P4e will stabilize to 15th of the bandwidth or 2 Gbps each again equally sharing the available bandwidth e When P4i starts to send at t5 all six traffic flows P1i P1e P2e P3e P4e and P4i will stabilize to 1 6th of the bandwidth or 1 65 Gbps each all sharing the available bandwidth e When P1i stops sending at t6 the five currently active traffic flows P1e bie Pie P4e and P4i will readjust to 2 Gbps each equally absorbing the freed up bandwidth e Asa previously sending traffic flow stops sending t7 t8 t9 and t10 the remaining active flows will readjust to equally fill any available bandwidth e Notice the symmetry of the BW allocation No matter which traffic type is currently running
42. are now sending each partition gets half of the send bandwidth so partition P2 P2e gets all of the allocated 5 Gbps and partition P1 s two traffic flows P1i and P1e will share it s allocated 5 Gbps or 2 5 Gbps each e When partition P3 P3e starts to send at t3 all three partitions will be allocated 1 3rd of the available bandwidth P3e will received 3 3 Gbps P2e will receive the same allocation of 3 3 Gbps and P1i an Ple will approximately split it s partition s bandwidth for 1 65 Gbps each e But when P4e starts to send Ethernet traffic at t4 it will take almost all of the 10 Gbps bandwidth regardless of the bandwidth needs of the other three partitions four traffic flows P1i and P1e will each get approximately half of 1 103 of the available bandwidth or 0 05 Gbps while bie and P3e will receive 0 1 Gbps and P4e will take 9 7 Gbps e When P4i starts to send Ethernet traffic at t5 it will take half of the allocated bandwidth for partition P4 Therefore P4e will drop to 4 75 Gbps and P4i will start at 4 75 Gbps The other three partitions four traffic flows will be unaffected P1i and P1e will each get approximately half of 1 103 of the available bandwidth or 0 05 Gbps while P2e and P3e will receive 0 1 Gbps and P4e will take 9 7 Gbps e When P1i stops sending traffic at t6 it s freed up 0 05 Gbps bandwidth will be reallocated to the other traffic flows according to their relative bandwidth weight settings The s
43. artition can be independently set from Normal 1500 bytes up to Jumbo 9600 bytes in the BACS4 SCSI Management Configuration page Broadcom Advanced Control Suite 4 Fle View Acton Filter Context Tools Teens GH Heb rte paven Is EF strane E BM Hosts S wears E ES Adapter 1 SOR ER CU T ES Adapter 0M5709 00 E ES Adapter3 80457711 40 SE Adapters 57712 A1 E 4 Porti 0050 Broadcom BCM57712 Nettreme D 10 GE Multi Function 50 ME 0012 Broadcom BCMST712 Nettreme II 10 GE Multi Function BCSI A Z Portal 1 1 1 4 scm Portal fe80 210 I8fF faef d3cs The FCoE device MTU frame size is fixed at 2500 bytes and is not adjustable but is viewable in the FCoE device Information page Broadcom Advanced Control suite 4 Ele View Acton Filter Context Tools Tears GOS Heb her aun view E lemas CIE vital signs E Gre information G0 Hosts DS EE Adapter BCM5709 CO ES Adapter BCMS709 CO E ES Adapters BCM57711 A0 BE Adapters 7712 AT Er i Porti B 0050 Broadcom BCM57712 MetXtreme I 10 GigE Multi Funchan 250 D 0012 Broadcom BCMS7712 Netiireme D 10 GigE Multi Function iSCSI A Portal 1 1 1 4 Portal fed0 210 Loh RET 03 5 C 0027 Broadcom BCMS7712 NetXtreme II 10 Cf NDIS VED Chert ss 0051 Broadcom BCM57712 NetXtreme IT 10 GOE Multi Function 251 0053 Broadcom ECM57712 Netktreme H 10 GigE Multi Function 253 C 0030 Braadeeem BCM 57712 Net
44. aximum Bandwidth values per partition Broadcom Advanced Control Suite 4 File View Acton Filter Context Tools Teams iSCSI Help Eeer 998 Hosts E fj wIN RA3QJRK0731 H CR Adapteri BCM5709 C0 E LR Adapter2 ECM5709 CO H MES Adapter3 BCM57711 AO L Z Adapters 57712 A1 Fl 4 Porti Ez IZ Enable a 0012 Broade L i FO v Enable AN 0027 Broadc 0051 Broadcom I 2 0053 Broadcom E 0030 Broade Ee T pss Broadcom 0016 Broadc E 0031 Broad Port2 IZ Enable I Enable Disable ith Wei 145 30 Jh Enable E Disable Disable i 45 Maximum Bandwidth 80 Figure 6 BACS4 NPAR Settings BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 23 57712 k Software User Manual Configuring NPAR BACS4 displays the per partition Virtual Bus Device VBD information Broadcom Advanced Control Suite 4 recla ale Pete o T et E BS Hosts S D wrana B ES Adapter 1 BCM5709 CU E Adapter BCM5709 CU T ES Adapter 3 BCM57711 AN E Adapters 57712 A1 Sai 10827 EroedeombCM57712 Hef reme I 10 Ge QOIS WO Cien 850 D051 Brosdcom BCM57712 Netxtreme I 10 GigE Multi Function 251 S F 0053 Broadcom BCMS7712 Net treme IT 10 GigE Multi Function 253 SA 0030 Broadcom BCM57712 Nebireme I 10 GigE MOIS VED Client 253 S PF 0054 Broadcom BCMS7712 Netxtreme II 10 GE Multi Function 254 ZZ 0016 Broadcom BCM57712 NetXtrem
45. d on the ratio of that partition s individual weight verses the aggregated weights of all the other actively sending partitions Furthermore each partition s combined traffic flow will be capped by that partition s Maximum Weight setting See the User Guide s examples for more details The actual inter partition ratio of the two sharing traffic flows is controlled by the host OS Think of the dynamic weight ratio as a variable sized funnel that could be further restricted by the Maximum Bandwidth fixed sized funnel with the OS determining how the sharing traffic types are pouring into the combined funnels n non DCB mode where the sum of the partition s Relative Bandwidth Weights equals zeros i e each partition s Relative Bandwidth Weight is set to zero each individual traffic flow i e Ethernet or iSCSI Offload or FCoE Offload is equally scheduled to transmit within the limitations of the partition s Maximum Bandwidth and the overall connection s link speed This means if the Maximum Bandwidth of a specific partition is set to less than 10096 then the traffic flows sharing that partition will be further restricted to where their combined traffic flow bandwidth will be capped by that per partition setting If all four partition s individual Maximum Bandwidths are set to 100 i e they are unrestricted then each actively sending traffic flow without regard to which partition they are on will equally share the transmit directions total bandwi
46. de December 8 2011 e 2CS57712 SWUM102 R Page 16 57712 k Software User Manual Configuring NPAR a Broadcom Advanced Control Suite 4 File View Action Filter Context Tools Teams iSCSI Help Fiter lar VIEW liconfigurations 7 Resource Reservations Licenses MN Advanced den Information Configurations Statistics El 200 Hosts S WIN RA3QJRK0731 D LS Adapter 1 BCM5709 C0 H E Adapter2 BCM5709 C0 E gt Pre Alocated Resources D 23 Adapter3 BCM57711 AD HS E mg Adapter4 57712 A1 EF Porti DOE 0045 Broadcom BCM57712 NetXtreme II 10 GigE 45 EET TUIS 9007 Broadcom BCM57712 NetXtreme H 10 GigE NDIS VED Client 45 id 8 Port2 0046 Broadcom BCM57712 NetXtreme II 10 GigE 46 7 0012 Broadcom BCM57712 Netxtreme II 10 GigE NDIS VBD Client 46 Figure 3 Broadcom Advanced Control Suite 4 If you are in NPAR mode go to the 57712 k Adapter Configuration Property window and click the next to the NIC Partition to expand the menu In the expanded menu if the NIC Partition setting is unchecked Disabled change it to checked Enabled and then set the desired partition s Ethernet NDIS iSCSI and FCoE protocols Also set the Relative Bandwidth Weights Maximum Bandwidth settings and then click Apply You must reboot the system for Windows to discover and install the device drivers BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 17 57712 k Software User Manual Configuri
47. de December 8 2011 s 2CS57712 SWUM102 R Page 6 57712 k Software User Manual Configuring NPAR Configuring NPAR Using the Unified Server Configurator Use Dell s Unified Server Configurator USC to configure Broadcom s 57712 k NPAR parameters To configure NPAR with the USC 1 Enter USC during system boot up by selecting the UEFI boot option See the Dell website http www dell com content topics global aspx power en simplify_management c us amp l en amp cs 555 for more information on USC k DLL UNIFIED SERVER CONFIGURATOR LIFECYCLE CONTROLLER ENABLED UNIFIED SERVER CONFIGURATOR LIFECYCLE CONTROLLER ENABLED E Hardware Configuration Contiguration Wizards guide system device set up Ex RAID ibRmC Encryption HII Advanced Configuration configures Human Interface Infrastructure HII enabled devices Ex BIOS NICS Hardware Inventory is used to view or export the server s current hardware inventory or the factory shipped hardwsre inventory Delete Configuration and Reset Defaults deletes the Lifecycle Controller configuration and restores factory defaults BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 7 57712 k Software User Manual Configuring NPAR 3 Allofthe Broadcom Ethernet Controller devices should be displayed on this page Select the desired 57712 k device port from the displayed list Kee DLL UNIFIED
48. ditionally the FCOE traffic flow is Lossless and in DCB Priority Group 1 with an ETS 50 and the other traffic flows are Lossy and in Priority Group O with an ETS 50 BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 74 57712 k Software User Manual Examples Table 11 DCB Mixed Fixed Subscription and Oversubscription with Lossless FCoE Offload Relative Bandwidth Maximum Port Partition Weight Bandwidth Protocol Plot Color Port 0 Partition 1 P1f N A 80 FCOE Offload Green Port O Partition 1 P1e N A 80 Ethernet Orange Port 0 Partition 2 P2e N A 10 Ethernet Purple Port O Partition 3 P3e N A 10 Ethernet Yellow Port 0 Partition 4 P4e N A 80 Ethernet Blue Port 0 Partition 4 P4i N A 80 SCSI Offload Red This is a similar combination example of a fixed subscription three of the partitions sum to 100 but all four sum to 180 When all four or at least the last two partitions are running traffic they share the space with each other up to their partition maximum bandwidth values and their PG s ETS settings otherwise they act as if they are independent connections e The first partition s traffic flow P1f initially takes its maximum bandwidth designated 8 Gbps when the test application starts to transmit traffic at t0 to that port by itself not expanding into the remaining unused 2 Gbps bandwidth e When the second traffic flow on the first partition P1e starts
49. dth i e TX link speed The actual inter partition ratio of the two sharing traffic flows is controlled by the host OS Think of the Maximum Bandwidth as a fixed sized funnel with the OS determining how the two sharing traffic types are pouring into that funnel n DCB mode all of the Partition s Relative Bandwidth Weights are disregarded and the individual traffic flows are scheduled to transmit within the limitations of the Priority Group s ETS value BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 11 57712 k Software User Manual Configuring NPAR determined by it s Traffic Type and each partitions Maximum Bandwidth setting and the overall connections link speed For example the FCOE traffic type could be assigned to Priority Group 1 PG1 and all of the other traffic types iSCSI and Ethernet could be assigned to another Priority Group such as PGO Each Priority Group has it s own ETS value which works similarly to a minimum bandwidth setting DCB Lossless iSCSI iSCSI TLV could be used in place of FCoE for a similar effect where the Lossless iSCSI Offloaded traffic would go through it s assigned Priority Group while the Lossy Ethernet traffic would go through another Similarly to the other two rate controlling modes the host OS determines the actual inter partition traffic ratio for the cases where two traffic types share the same partition Note A traffic type s send flow rate will be approximately th
50. e December 8 2011 e 2CS57712 SWUM102 R Page 76 57712 k Software User Manual Examples Mixed Weights and Subscriptions Example This example shows partitions with different relative bandwidth weights and maximum bandwidths but with the same partitions partially oversubscribing of the available bandwidth as the previous example The first pair of partitions are set to use 10 or 1 Gbps each of bandwidth and both of their weights are set to 5 while the second pair of partitions are set to use 80 or 8 Gbps of the connection s bandwidth each with both of their relative bandwidth weights set to 45 The total is still oversubscribing the connection by 80 In DCB mode there would be no difference between the previous DCB mode example and this one since the Relative Bandwidth Weights are not applicable in DCB mode and also if all of the traffic types are in the same Priority Group the results would be similar Table 12 Non DCB Mixed Fixed Subscription and Oversubscription Relative Bandwidth Maximum Port Partition Weight Bandwidth Protocol Plot Color Port 0 Partition 1 P1i 45 80 SCSI Offload Green Port O Partition 1 Ple 45 80 Ethernet Orange Port 0 Partition 2 P2e 5 10 Ethernet Purple Port O Partition 3 P3e 5 10 Ethernet Yellow Port 0 Partition 4 P4e 45 80 Ethernet Blue Port 0 Partition 4 P4i 45 80 iSCSI Offload Red This is a combination example of a fixed subscription three of the partiti
51. e if there are two different traffic type flows such as P1i and P1e in a single partition they are combined as if one flow for determining the amount of bandwidth allocated to them e Starting at tO the first partition s iSCSI Offload traffic flow P1i initially takes 100 of the available port s TX send bandwidth when an iSCSI test application is flooding that port by itself e When P1 s L2 Ethernet P1e starts to send at t1 both stabilize to 5 Gbps each even thought they are in the same partition they share the total available bandwidth This is because no other partition s traffic flow is sending e When P2e starts to send at t2 the traffic flows in P1 P1i and P1e will reduce to 2 5 Gbps each while the P2 traffic flow P2e will take 5 Gbps This is because the bandwidth is initially split by partition and then traffic flows within each individual partition e When P3e starts to send at t3 the two traffic flows in P1 P1i and P1e are further reduced to 1 65 Gbps half of the partition s 1 3rd allocation going to the three active partitions while P2e and P3e each stabilize at 3 3 Gbps e When P4e starts to send at t4 the three single partition traffic flows P2e P3e and P4e will stabilize to 1 4th of 10 Gbps or 2 5 Gbps each while the P1 partition shares it allocated bandwidth between it s two users P1i and P1e so each gets half of the allocated 1 4th of 10 Gbps or 1 8th which is 1 25 Gbps e When P4i
52. e H 10 GgE Multi Funcion 251 11 0053 Broadcom BCMS7712 Net meme IT 10 Gage Mul Furx eon 253 SC 0030 Broadcom 80957712 NetXtreme D 10 Gage 045 VEO Chent 53 T Sd Broadcom BCM57712 Netxtreme EI 10 Gage Mult Funcion 554 amp ES 0016 Broadcom BCM57712 Metitreme H 10 Gig Multi Function FCoE Adapter The following shows the per partition per FCoE target information Broda aan Bh ds d Con trol Suite 4 Fle View Acton Filter i Context Too Teams GC Help es lass DE Fcog Target information E ses S wmmRANXR80731 EM Adapter 1 BCM5709 CO H N Adapter BOMS709 CO EM Adapter3 BCMS7711 ADJ ES Adapters 57712 AT EF Porti S E 0050 Broadcom BCMS7712 Hent Keme II 10 GigE Muls Funcion 250 0051 Broadcom BCMS7712 Mettreme II 10 Gef Muls Function 251 S W 0053 Broadcom BCM57712 NetXtreme II 10 GigE Mult Function 253 GU D030 Broadcom BCM 57712 Netktreme H 10 Gef NDIS VEO Clent 253 amp W 0054 Broadcom BCMS7712 Netktreme I 10 GigE Mult Funcion 254 a E 0016 Broadcom BCM57712 Nettreme H 10 GigE Mull Function FCoE Adapter E Ei Port 20 00 00 10 18 6F D3 8D BROADCOM December 8 2011 e 2CS57712 SWUM102 R Port Type WW Port Name WWW Node Nae Port FO Fo FOF Macadoresa FOP WW Hode Mame PE na Port Mame ac Configuring NPAR NPort 20 00 00 10 18 6F d3 bd 10 00 00 10 18 6633 54 EFC FOE 20 01 00 0d ec e3 a 01 1001 Bee ideco k
53. e IT 10 GE Multi Function FCoE Adapter GE 0031 Broadcom BCM57712 NetXtrese I 10 GigE NOIS VED Client 254 The following shows the per partition FCoE device information Broadcom Advanced Control Suite 4 Fie View riie Tools Teams Hep j Fiver a view information IER vial signa F Driver Information E 9 Hosts S H BBA RED Bi MEM Adapter B0M5709 Co EEN Adapter B045709 CU F EN adapter3 9057711 40 E I Adapters 57712 A1 Eb Parti 1 Grower InformaSon a PR 0053 Broadcom B0M57712 Menee TT 10 GigE Multi Function 2 50 0051 Brosdeom BCM57712 ey nene I 10 GigE Multi Function 251 Driver Status S 4 0053 Broadcom BCMS77 12 beige T 10 G Mult Funcion 253 m eise Jam 23 0030 Broadcom B0M57712 NetXtreme 11 10 GigE NDIS VED Clent 253 r 1 f 0054 Broadcom BCM57712 Netxtreme D 10 GigE Multi Function 254 BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 24 57712 k Software User Manual Configuring NPAR The following shows the configuration of FcoE device settings om Jdwvarnoed C a Suite 3 E me Adapter 3 BCMS7711 AG Ci E Adapters 57712 A1 Se Porti amp E 10050 Roads BCMS7712 Netireme IL 10 Ggf Mult Funcion 250 0051 Broaden BCM57712 NetXtreme IT 10 Gage MIS Funcbon F51 gt PF 0053 broskos BCM57712 NetXtreme IL 10 GigE Mus Funcion 153 CH 10000 Broadcom BCMS7712 Netxtreme IT 10 Gef NOIS VED Cieni ep e T 0054 Broadcom BCM57712 Netxtreme
54. e ratio of its individual partition s relative bandwidth weight setting divided by the sum of the relative bandwidth weights of all the partitions currently actively sending on that port or that partition s maximum bandwidth setting whichever is lower In the case where the Relative Bandwidth Weights are all zeros OR in DCB mode each traffic type will have an equal weight with respect to one another see Examples on page 56 r Note DCB mode is supported in Windows and some Linux RHEL v6 x and SLES11 SP1 OS s on the 57712 k VMWare ESX ESXi 4 1 does not support DCB which includes both FCoE and DCB Lossless iSCSI on the 57712 k Each partition s Maximum Bandwidth settings can be changed in the same way and has a range of 1 to 10096 in increments of 196 of the port s current Link Speed at 10 Gbps this would be in 100 Mbps increments and at 1 Gbps this would be in 10 Mbps increments This setting limits the most send bandwidth this partition will use and will appear as its approximate link speed in various places in the respective operating system even though the four partition s are sharing the same overall connection i e the four partitions may advertise in the OS that their link speed is 10Gbps each but they all share the same single 10Gbps connection Displayed values may be rounded off by various applications The Maximum Bandwidth value is applicable to both DCB and non DCB modes of operation The Maximum Bandwidth value is applicabl
55. e sub tab selecting In the first example the first port s four partitions on VMNIC6 8 10 12 are set to 0 Relative Bandwidth Weight and 100 Maximum Bandwidth each and the second port s partitions on VMNIC7 9 11 13 are similarly set to 0 Relative Bandwidth Weight but the Maximum Bandwidth values are set to 1096 2096 3096 4096 respectively which results in VMNIC7 s link speed for the transmit direction only indicating 1000 Mbps VMNIC9 s link speed indicating 2000 Mbps VMNIC11 s link speed indicating 3000 Mbps and finally VMNIC13 s link speed indicating 4000 Mbps This is indicated in the vSphere Host s Configuration Network Adapter page The first port s network performance indicated in the vSphere Host s Performance page shows each individual VM s send traffic rate when none to all four are sending sharing the available bandwidth between each other e Edit View Inventory Administration Plug ins Help LA e Home b gfj Inventory p s Inventory H 1013 126 77 ER TIAS BRETT ARCET RIETI Getting Started Summary Virtual Machines ResourceAllocation Performance Configuration Local Users amp Groups Events Permissions Network Real time 1 12 2011 4 32 27 PM 1 12 2011 5 32 27 PM Chart Options Switch to Default Graph refreshes every 20 seconds 2000000 2 running shared BW 1 2 each e 1500000 All 4 running 3 running 2 running k 3 running All 4 running shared BW sharing BW 1 3 of total each 1 4 of total each s
56. e to the send TX direction only Note A partitions send Maximum Bandwidth setting does not affect a partition s receive direction traffic bandwidth so the link speed displayed for the partition is for the send transmit outgoing direction only All partitions receive direction maximum bandwidth is always the ports current Link Speed and is regulated by the attached switch port just as it is in SF mode when multiple L2 Ethernet and iSCSI Hardware Offload and FCoE Hardware Offload traffic protocol types are enabled The Maximum Bandwidth settings can be used to oversubscribe a port This is done by setting the four partitions of that single port to having a total Maximum Bandwidth setting SUM of more than 100 i e 10000 Mbps or 10 Gbps That just means the various partitions will attempt to take as much bandwidth as allowed by their specific setting maximum limits and weights as their individual traffic flow needs change In an oversubscription situation the 57712 k will ration out free bandwidth based on the weights sum is 0 verses sum is 100 and maximum settings and the mode DCB verses non DCB it is in The above example shows the first port s four partitions being set to 10 10 80 80 180 which means the port is 180 subscribed 18 Gbps or 80 oversubscribed i e 18 Gbps subscribed 10 Gbps line rate available 8 Gbps oversubscribed The Maximum Bandwidth setting applies to all protocols enabled on that partition BRO
57. each will get an equal share with respect to the other currently transmitting traffic type flows This assumes the application creating the transmitted traffic type flow can fill the allocated amount of BW it is given if not the other traffic flows will equally absorb the unused BW BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 58 Examples 57712 k Software User Manual Page 59 NPAR Setup Guide al om mmm U 120 Ip a w emm TYP FR mmm eg lt SANA Lu ER A a 7 Rp z Un E TS mm EE O 0M BP b s beem LLL e i ER Hii M A E E RK sl CM WW December 8 2011 e 2CS57712 SWUM102 R BROADCOM 57712 k Software User Manual Examples The following two examples are of oversubscribed bandwidth sharing with DCB enabled The first example is similar to the above non DCB example with four Ethernet and two iSCSI Offload protocols enabled with the same two traffic types in six distinct flows with the partitions similarly configured The main differences here is the iSCSI traffic type is assigned to DCB Priority Group 2 and is Lossless i e iSCSI TLV with an ETS setting of 5096 while the L2 Ethernet traffic type is still assigned to Priority Group 0 and is Lossy with an ETS setting of 50 If the iSCSI Offload protocol traffic flows had been assigned to the same PG as the Ethernet protocol traffic flows then the traffic BW would have
58. er especially for SCSI Storage devices All of the enabled devices on the same partition have identical PCle interface location information with only the function number varying The following shows the eight Ethernet partitioned adapter s PCle device location information Notice the Windows enumerated device numbering values do not follow the PCle bus function numbering nor the port to partition numbering BROADCOM NPAR Setup Guide December 8 2011 s 2CS57712 SWUM102 R Page 36 57712 k Software User Manual Configuring NPAR al 1 57712 HPO Port Partition 1 1 1 1 20 Status 2 57712 HP2 Port 0 Partition 2 3 3 3 20 Statut Broadcom BCH57712 NetXtreme II 10 GigE NDIS VBD Client 76 x Broadcom BCM57712 NetXtreme TT 10 GigE NDIS VBD Client 78 F Ca Broadcom BCM57712 Net reene Il 10 GigE NDIS La Broadcom BCM57712 Nei renge Il 10 GigE NDIS B 3 57712 NP4 Port 0 Partition 3 5 5 5 20 Status B 4 57712 NP6 Port 0 Partition 4 7 7 7 20 Status Broadcom BCM57712 Netxtreme II 10 GigE NDIS VED Client 80 ES f Broadcom BCH57712 NetXtreme I 10 GigE NDIS VED Client 79 General Advanced Dever Detais Power Management General Advanced Dever Details Power Management Ms Broadcom BCM57712 Nettreme II 10 GigE NDIS Mus Broadcom BCM57712 Net rene Il 10 GigE NDIS Location 4 PCI bus 6 device function 4 5 57712 HP1 Port 1 Partition 1 2 2 2 70 Status x 6 57712 NP3 Port 1 Pa
59. es of PG2 s portion of the overall bandwidth ETS of 50 Finally at t10 there is only one active flow after P4e stops sending so at this point P4i gets 100 of all the bandwidth or 10 Gbps BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 61 57712 k Software User Manual Examples This second DCB example of oversubscribed bandwidth sharing replaces one of the SCSI Offloads with an FCoE Offload protocol This gives a total of three distinct traffic types in six flows with similar partition settings The other difference is that the FCOE traffic type is assigned to DCB Priority Group 1 and is Lossless with an ETS setting of 5096 while the L2 Ethernet and iSCSI Hardware Offload traffic types are both now in Priority Group O and are both Lossy with an ETS setting of 50 Table 5 DCB Equal Oversubscription with one Lossless FCoE Offload Relative Bandwidth Maximum Port Partition Weight Bandwidth Protocol Plot Color Port O Partition 1 P1f N A 100 FCoE Offload Green Port 0 Partition 1 P1e N A 100 Ethernet with Brown TOE Port 0 Partition 2 P2e N A 100 Ethernet Purple Port O Partition 3 P3e N A 100 Ethernet Yellow Port 0 Partition 4 P4e N A 100 Ethernet with Blue TOE Port 0 Partition 4 P4i N A 100 SCSI Offload Red The following plot shows how the first partition s FCoE traffic PG1 acts verses the other traffic types in PGO Just like the previous example the traffic
60. es Note that the ping will return the send size plus the 8 bytes of the ICMP header The c switch sets the number of times this ping command is sent You can also use the optional source interface Capital command l a b c d or l devName if you have multiple adapters that are connected to the same target IP address and you want to check a specific interface root Aserver ping c 3 s 8972 192 168 20 10 PING 192 168 20 10 192 168 20 10 from 192 168 20 200 8972 9000 bytes of data 8980 bytes from 192 168 20 10 192 168 20 10 icmp seq 0 ttl 255 time 0 185 ms 8980 bytes from 192 168 20 10 192 168 20 10 icmp seq 1 ttl 255 time 0 177 ms BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 52 57712 k Software User Manual Configuring NPAR 8980 bytes from 192 168 20 10 192 168 20 10 icmp seq 2 ttl 255 time 0 180 ms 192 168 20 10 ping statistics 3 packets transmitted 3 packets received 0 packet lost time 3043ms rtt min ave max mdev 0 177 0 181 0 185 0 005 ms If it does not work you will see root server ping c 3 s 8972 192 168 20 10 PING 192 168 20 10 192 168 20 10 from 192 168 20 200 8972 9000 bytes of data 192 168 20 10 ping statistics 3 packets transmitted packets received 100 packet lost time 3010ms Setting MTU Sizes in VMWare ESX ESXi 4 1 In VMWare ESX ESXi 4 1 the MTU size for each individual Ethernet protocol enabled partition can be independently
61. evices pci0000 00 0000 00 07 0 0000 05 00 0 host5 fc host host5 lrwxrwxrwx 1 root root 0 Jun 29 11 11 host8 gt devices pci0000 00 0000 00 07 0 0000 06 00 0 host6 fc_host host6 Each installed FCoE device will appear here FCoE uses the bnx2fc driver which can be checked for with the lsmod grep bnx2 command Another useful command is sg map i x which will show all SCSI LUN devices visible to the host For Fiber Channel another useful application is FCInfo that is part of the Broadcom Linux driver release utilities and displays the FCoE HBA port information BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 41 57712 k Software User Manual Configuring NPAR VMWare ESX ESXi 4 1 VMWare ESX ESXi 4 1 shows the respective device protocols that were enabled in USC if the NetXtreme ll device drivers are installed Go to the Dell Driver Download web site http support del com for the latest device Firmware if not already installed Go to the VMWare web site http downloads vmware com for the latest device drivers and insure they are installed on your system In VMWare the Ethernet Protocol is always enabled on all eight partitions VMWare ESX ESXi 4 1 does not support the iSCSI Offload Protocol in NPAR mode VMWare ESX ESXi 4 1 does not support the FCoE Offload Protocol in SF or NPAR modes r Note VMWare ESX ESXi 4 1 only supports four 10 GbE ports Using NPAR mode allows you to expand the number of ports
62. ffic flows on the other three partitions are unaffected e fP1e stops at t7 the others adjust slightly upwards to fill that newly available bandwidth partition P2 and it s traffic flow P2e increases to 2 2 Gbps 20 90 partition P3 and it s traffic flow P3e raises to 3 3 Gbps 30 90 and partition P4 raises to 4 5 Gbps 40 90 which means P4e and P4i split that for 2 25 Gbps each e f P2e stops at t8 the others again adjust upwards to fill that newly available bandwidth partition P3 and it s traffic flow P3e raises to 4 3 Gbps 30 70 and partition P4 raises to 5 7 Gbps 40 70 which means P4e and P4i split that for 2 85 Gbps each e f P3e stops at t9 partition P4 s share raises to 100 of the bandwidth 40 40 so it s two traffic flows P4e and P4i split this for 5 Gbps each e When P4e stops at t10 the only remaining traffic flow P4i takes all of the bandwidth at 10 Gbps BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 68 57712 k Software User Manual Examples Oversubscription With One High Priority Partition Example Next is an example of a single high priority partition with all of the relative bandwidth weight but all four of the partitions are still oversubscribing the available bandwidth The first three partitions of the port have 0 weight and the last partition P4 has all of the weight 100 All four partitions are set to use the maximum amount of the connecti
63. h functions 0 2 4 6 existing on the first port and functions 1 3 5 7 existing on the second port The actual numbering position an adapter is assigned by Windows is not entirely related to the PCle interface numbering and is more related to what open location position numbers are available in the registry when the adapters get enumerated Therefore port O partition 1 may not always occupy the first position in the Windows Device Manager s Network Adapters or Storage Controllers or System Devices sections Dell PCle Unified Server Location Bus Device Function Ethernet MAC Example Configurator The first three are the same for both ports Address IP Address e wc c c c CP CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD Port 0 Partition 1 X Y Z 0 00 10 18 88 E7 A8 1 1 1 1 0 0 e N WW N 00 00 00 BD 00 00 00 M 00 00 00 CC gt x Partition 2 09 09 09 00 wm M e 09 00 09 09 02 09 09 09 09 09 09 02 c 90 09 09 0 09 09 09 aa 09 aa 60 o e 00 09 00 Qo Qo m N 0 e Ge To Ge N gt gt eee R YX Y Y XK Y Y Y E X KX X gt x e A e 4 4 4 1 N lt O DI I 0 Partition 4 X Y 00 10 18 88 E7 Port 1 Partition 1 X Y Z 1 00 10 18 88 E7 AA 5 5 5 1 ER gv 06 gv 06 o e 90 go ER tn O gt ER g g8 Qe K ge Partition 3 gt
64. hares PGO s bandwidth 5 Gbps 5 equally weighted traffic flows which means P1e Pie Pie P4e and P4i all send at 1 Gbps P1f in PG1 is still unaffected and keeps sending at 5 Gbps e When P1f stops at t6 it releases all of PG1 s bandwidth to the available pool and since bie and P3e are capped by their maximum bandwidth value to 1 Gbps the three other traffic flows P1e P4e and P4i automatically take equal shares of 8 Gbps and bump up their bandwidth used to 2 6 Gbps each e When Pie subsequently stops sending at t7 P4e and P4i grab up the extra available bandwidth and go to 4 Gbps each Both P2e and P3e are unaffected and continue sending at 1 Gbps each e When P2e stops sending at t8 P4e and P4i are not able to make use of the freed up bandwidth since they are both in partition P4 which has a maximum bandwidth ceiling of 8 Gbps Therefore none of the traffic BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 75 57712 k Software User Manual Examples flows increase their sending rates and the unused bandwidth is ignored e When P3e stops sending at t9 the same condition is still in effect Therefore none of the remaining active traffic flows increase their sending rates to use this extra bandwidth e Finally P4e stops at t10 and this allows it s companion traffic flow P4i to increase to 8 Gbps which is partition PA s maximum top end The remaining 2 Gbps is unassigned BROADCOM NPAR Setup Guid
65. haring BW shared BW shared BW 1 running 1 running 1 4 of total each 1 3 of total each 1 2 each all BW all BW 1000000 N 500000 i e D i I i i p I I 4 35 PM 4 40 PM 4 45 PM 4 50 PM 4 55 PM 5 00 PM 5 05 PM 5 10 PM 5 15 PM Time 0 running Performance Chart Legend Key Object Measurement Rollup Units Latest Maximum Minimum Average a VM 1 Datatransmitrate Average KBps 291507 1158844 0 259082 90 VM 2 Datatransmitrate Average 291524 580845 D 229489 34 m 291624 O 291412 VM 3 Data transmit rate Average 581939 0 250998 64 1159897 0 362926 90 VM 4 Datatransmitrate Average The second port s network performance indicates each VM is limited to it s specific top end setting Maximum Bandwidth and does not expand into the unused area BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 44 57712 k Software User Manual Configuring NPAR 40 1 e A File Edit View Inventory Administration Plug ins Help E Home b gj Inventory b Fj Inventory 113 126 447 Er AS ei Bb vs Ep vue Getting Started Summary Virtual Machines Resource Allocation Performance Configuration LocalUsers amp Groups Events Permissions 2 VM 7 pass pea 6 59 23 AM 1 13 2011 7 59 23 AM Chart Options Switch to NIC PartitionDemo amp H 2 VM 8 3 refres every Seco 2000000 1500000 1000000 Performance Chart Legend
66. have zero relative bandwidth weights each traffic flow will act as if in it s own separate partition each taking an equal share of the available bandwidth up to that partitions maximum bandwidth which is 10096 in this example so does not further limit any of the traffic flows FCoE Offload is not available in non DCB mode so two iSCSI Offload protocols are used here Table 3 Non DCB Equal Oversubscription Relative Bandwidth Maximum Port Partition Weight Bandwidth Protocol Plot Color Port O Partition 1 P1i 0 100 SCSI Offload Green Port O Partition 1 Ple 0 100 Ethernet Brown Port 0 Partition 2 P2e 0 100 Ethernet with Purple TOE Port O Partition 3 P3e 0 100 Ethernet Yellow Port O Partition 3 P4e 0 100 Ethernet with Blue TOE Port 0 Partition 4 P4i 0 100 SCSI Offload Red The following plot shows how all of the partitions would share a ports available send bandwidth Each traffic type flow such as P1i and P1e is expanded in to its own bandwidth trace for ease of understanding The send traffic flows are independent in each partition and the individual traffic type flow rate is balanced with each of the other traffic type flow rates when traffic demands exceed the available bandwidth e Starting at tO the first partition s iSCSI Offload traffic flow P1i initially takes 100 of the available port s BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 57 57712 k Softw
67. he remaining bandwidth is shared equally by P2e and P3e at 1 Gbps each e When P2e stops sending at t8 P4e and P4i are not able to make use of the freed up bandwidth since they are both in partition P4 which as a maximum bandwidth ceiling of 8 Gbps The same is true for P3e which is also at it s bandwidth maximum Therefore none of the remaining traffic flows increase their sending rates and this unused bandwidth is ignored e When P3e stops sending at t9 the same maximum ceiling condition is still in effect Therefore neither P4e or P4i increase their sending rates to use this extra bandwidth e Finally P4e stops at t10 and this allows it s companion traffic flow P4i to increase to 8 Gbps which is partition PA s maximum top end The remaining 2 Gbps is unassigned BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 78 Examples 57712 k Software User Manual NPAR Setup Guide BROADCOM Page 79 December 8 2011 e 2CS57712 SWUM102 R 57712 k Software User Manual Dell has tested and certified the Broadcom 57712 k Dual port 10 GbE Converged Network Daughter Card with NIC Partitioning TOE iSCSI and FCoE ready technology Dell specifically disclaims knowledge of the accuracy completeness or substantiation for all statements and claims made in this document regarding the properties speeds or qualifications of the adapter Broadcom Corporation reserves the right to make changes without further notice to
68. hernet Yellow Port O Partition 4 P4e O 25 Ethernet Blue Port 0 Partition 4 P4i 0 25 SCSI Offload Red This following plot shows how the four partition s send traffic is independent of each other Unlike the previous examples none of the partitions and their associated traffic flows take more than their designated bandwidth portion the total bandwidth of all four partitions of the port is equal to or less than the total available bandwidth of the port In this example each partition takes only 25 or 2 5 Gbps of the total available bandwidth when their test application starts to transmit traffic Furthermore if a partition has more than one active traffic flow these flows will share that partition s allowed bandwidth Unused port bandwidth is not re allocated to any partition above it s own maximum bandwidth setting e When P1i starts to send traffic at tO it only takes the subscribed 25 of the 10 Gbps bandwidth available which is 2 5 Gbps e when P1e starts to send at t1 it will share partition P1 s 2596 with P1i Each is allocated 1 25 Gbps and neither expands into the unused 7 5 Gbps remaining e When P2e starts to send at t2 it only takes it s partitions subscribed 2 5 Gbps and does not affect either of partition P1 s sending traffic flows e When P3e starts to send at t3 it again only takes it s partitions subscribed 2 5 Gbps and does not affect P2e or either of partition P1 s sending traffic flows e When
69. iSCSI Offload Protocal Disabled FCoE Offload Protocol Enabled Enable Cisable 9 FCoE Offload Protocol This option is disabled if this port has a partition that already have FCoE Off load Protocol enabled BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 14 57712 k Software User Manual Configuring NPAR Supported Operating Systems The 57712 k SF and NPAR mode supported operating systems are shown in Table 1 Note The drivers may not be in the box Table 1 Protocols Available in Operation Systems Versus SF and NPAR Mode SF Mode NPAR Mode Operating System Ethernet iSCSI Offload FCoE Offload Ethernet SCSI Offload FCoE Offload Windows 2008 Yes Yes Yes Yes Yes Yes Windows 2008 R2 Yes Yes Yes Yes Yes Yes Windows 2008 R2 Yes Yes Yes Yes Yes Yes Hyper V Oracle Solaris 10u9 Yes No No Yes No No Linux Yes Yes Yes Yes Yes Yes VMWare ESX ESXi 4 0 Yes No No Yes No No VMWare ESX ESXi 4 1 Yes Yes NO Yes Not certified No a DCB DCBX PFC ETS supported b DCB DCBX PFC ETS supported in RHEL v6 x and SLES11 SP1 only c FCOE offload supported in RHEL v6 x and SLES11 ASP1 only BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 15 57712 k Software User Manual Configuring NPAR Viewing and Configuring the Partitions e Windows Server 2008 R2 e Red Hat Enterprise Linux e VMWare ESX ESXi 4 1 Windows Server 2008 R2 Installing the Latest
70. ide December 8 2011 e 2CS57712 SWUM102 R Page 18 57712 k Software User Manual Configuring NPAR Viewing the Enabled Devices in Device Manager Windows shows all of the enabled devices in Device Manager with the respective USC enabled NPAR protocols The following example shows e Eight enabled Ethernet protocol partitions four possible per port as the Broadcom BCM57712 NetXtreme Il 10 GigE NDIS VBD Client 50 through 57 in the Network Adapters section e Two enabled iSCSI protocol partitions up to two are possible per port if no FCoE is enabled as the Broadcom BCM57712 NetXtreme Il 10 GigE Multifunction iSCSI Adapters 51 and 55 AND two enabled FCOE protocol partitions one possible per port as the Broadcom BCM57712 NetXtreme Il 10 GigE Multifunction FCoE Adapters 452 and 456 in the Storage Controllers section e Eight Broadcom BCM57712 NetXtreme Il 10 GigE Multifunction virtual bus devices 50 through 57 in the System Devices section These eight virtual bus system devices are always present and are not controlled by what protocol is enabled in USC EN Server Manager t RE SCS Device Manager E gt Gs ES D Network adapters Si Broadcom BCM5709C NetXtreme II Get NDIS VBD Client 546 is Broadcom BCM5709C NetXtreme II GigE NDIS VBD Client 247 Do de Broadcom BMC MetXteme II susya is VBD leni z48 E Broadcom BCM57712 NetXtreme II 10 GigE NDIS VED Client 5 ZE Broadcom BCM57712 Netxtreme H 10 GigE NDIS VED Client 54 E Broadcom
71. idth went from 5 Gbps to 10 Gbps e Aseach traffic flow stops sending in PGO the remaining member traffic flows equally increase their respective shares to automatically occupy all of the available bandwidth At t7 there are four active PGO flows so each gets 1 4th or 2 5 Gbps At t8 there are three active PGO flows so each gets 1 3rd or 3 3 Gbps At t9 there are two active PGO flows so each gets half or 5 Gbps Finally at t10 there is only one active PGO flow P4i so it gets 100 or 10 Gbps e Any ofthe traffic flows will take 100 of the available bandwidth if it is the only sending traffic flow BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 62 Examples 57712 k Software User Manual sx E E E E E E a alll M o ees LO 0 0 AA E E aM BM E BM s ju m MI M I D mun mu 80 M WW Li mm 08 A 00 _ R H HP ER EE j ias iam wem E I p eee bees a R NA M M aM E re SB 8 a Sg a ee ndl I L0 DL M M M E o E E E E EO E d c emm mmm 1 BM S Ca M NN NM M M BM DM s m M OMM OM u AS 0 3 ip Ka J A M n en Ss eem 8 NPAR Setup Guide BROADCOM Page 63 December 8 2011 e 2CS57712 SWUM102 R 57712 k Software User Manual Examples Partitioned Oversubscription Example The following is an example of oversubscribed bandwidth sharing in non DCB mode where all four partitions of the port have their weigh
72. ight of 20 or 1 65 Gbps each for P1i and P1e P2e starts at 6 7 Gbps it s relative weight is 20 3096 total active weights it is not halved since it is the only traffic flow on partition P2 e When partition P3e starts sending Ethernet traffic at t3 with a relative weight of 30 it takes 5 Gbps 30 60 of 10 Gbps P2e drops to 73 3 Gbps 20 60 and partition P1 s total drops to 1 65 Gbps 10 60 so that means P1i and P1e each get half of that or 0 825 Gbps each e When P4e starts 40 relative weight at t4 it takes 4 Gbps 40 100 and the three other partition s send traffic drop partition P1 is reduced to 1 Gbps 10 100 so that means P1i and P1e split that for 0 5 Gbps partition P2 drops to 2 Gbps 20 100 and since there is only one send traffic flow P2e it takes all of that assigned bandwidth and finally partition P3 with it s single traffic flow P3e drops to 3 Gbps 30 100 e When the second traffic flow on partition P4 P4i starts at t5 the two flows P4e and P4i on the same partition P4 split the partition s assigned bandwidth of 4 Gbps so each gets 2 Gbps The other send traffic on the other three partitions remains the same BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 67 57712 k Software User Manual Examples e f P1i stops at t6 the remaining traffic flow on partition P1 P1e absorbs that partition s share of the send bandwidth to 1 Gbps The remaining tra
73. looked very similar to the previous Non DCB example since ETS would never be activated for traffic flows belonging to the same PG Table 4 DCB Equal Oversubscription Relative Bandwidth Maximum Port Partition Weight Bandwidth Protocol Plot Color Port O Partition 1 P1i N A 100 SCSI Offload Green Port 0 Partition 1 P1e N A 100 Ethernet Brown Port 0 Partition 2 P2e N A 100 Ethernet with Purple TOE Port O Partition 3 P3e N A 100 Ethernet Yellow Port 0 Partition 4 P4e N A 100 Ethernet Blue Port 0 Partition 4 P4i N A 100 SCSI Offload Blue The following plot shows how the two SCSI traffic streams in PG2 act verses the L2 Ethernet traffic streams in PGO The traffic in the two PGs will almost act independently of each other when their aggregated traffic bandwidth demands exceed the available bandwidth each taking it s half of the ETS managed pie e Starting at tO only P1i iSCSI Offload is sending so it takes 100 or all of the 10 Gbps bandwidth e When Ple L2 Ethernet starts to send at t1 both flows stabilize to 5 Gbps each P1i in PG2 takes it s allocated 5096 bandwidth and P1e in PGO takes it s allocated bandwidth of 50 e When P2e starts to send at t2 the traffic in P1i is not affected it remains at 5 Gbps due to it being in a different Priority Group Both P1e and P2e will stabilize to 2 5 Gbps each P1e and P2e equally share PGO s allocated portion of the bandwidth
74. ng NPAR i Broadcom Advanced Control Suite 4 File View Acton Filter Context Tools Teams iSCSI Help 99H Hosts Ek 2 WIN RA3QJRK0731 I EE Adapter 1 BCMS709 C0 T ES Adapter2 BCM5709 CO x H E Adapter3 BCM57711 A0 D Porti E 9 0050 Broadcom ME SCH 0012 Broadc ew 0027 Broadc 0051 Broadcom I i E 0053 Broadcom af 0030 Broadc SR 0054 Broadcom 0016 Broadc 9 0031 Broadc Maximum Bandwidth 91 EE Ei mehan 4 Figure 4 Broadcom Advanced Control Suite Adapter Settings After the devices are installed the enabled devices L2 Ethernet NDIS FCoE and iSCSI will be visible in the Windows Device Manager and BACSA The following is the 57712 k s Device Manager s display in SF mode see Figure 5 Device Manager Ba Intel R 82801 PCI Bridge 244 1M Intel R processor DMI D130 G J Intel R processor PCI Express Root Port 1 0138 ER Broadcom BCM57712 NetXtreme II 10 GigE 41 Er Broadcom BCM57712 NetXtreme II 10 GigE NDIS VED Client 41 5 Broadcom BCM57712 Netxtreme II 10 GigE FCoE Adapter 41 E Broadcom BCM57712 NetXtreme II 10 GigE iSCSI Adapter 41 C S Broadcom BCM57712 Netxtreme II 10 GigE 42 EP Broadcom BCM57712 NetXtreme II 10 GigE NDIS VED Client 42 gt Broadcom BCM57712 NetXtreme II 10 GigE FCoE Adapter 42 Broadcom BCM57712 NetXtreme II 10 GigE iSCSI Adapter 42 Figure 5 Windows Device Manager BROADCOM NPAR Setup Gu
75. ning for the entire device 6 Return to the Broadcom Main Configuration Page to edit the four partitions attributes by selecting the NIC Partitioning Configuration Menu DELL UNIFIED SERVER CONFIGURATOR LIFECYCLE CONTROLLER ENABLED Broadcom NetXtreme II 10 Gigabit Ethernet 00 10 18 6F 02 A4 Broadcom Main Configuration Page Firmware Image Menu Device Configuration Henu MBA Configuration Menu iSCSI Boot Configuration Menu NIC Partitioning Configuration Menu BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 9 57712 k Software User Manual Configuring NPAR 7 This window gives access to the Global Bandwidth Allocation Menu the Flow Control settings and each of the four partition s protocol settings First select the Global Bandwidth Allocation Menu option De LL UNIFIED SERVER CONFIGURATOR LIFECYCLE CONTROLLER Broadcom Metxtreme II 10 Gigabit Ethernet 00 10 18 6F 02 44 NIC Partition Configuration Broadcom Netxtreme II 10 Gigabit Ethernet 00 10 18 6F D2 A4 Partition z Partition 3 Partition 4 8 The Global Bandwidth Allocation Menu window controls the Relative Bandwidth Weight and Maximum Bandwidth parameters for all four partitions See Broadcom Advanced Control Suite 4 BACSA on page 23 for more information on how BACS4 can also be used to control these settings L UNIFED SERVER CONFIGURATOR LIFECYCLE CONTROLLER ENABL Broadcom Netxtreme II 10 Gigabit Etherne
76. ns 8 frame 8 TX packets 31 errors amp dropped overruns 8O carrier collisions O txqueuelen 1000 RX bytes 290494 283 6 KiB TX bytes 8110 7 9 KiB Interrupt 181 Memory d7886680 d77TTTTTT Link encap Ethernet HWaddr 00 10 18 88 E7 B6 inet addr 18 2 2 283 Bcast 10 255 255 255 Mask 255 6 6 6 inet addr fe88 218 18T1T T16e88 e7b6 64 Scope Link UP BROADCAST RUNNING MULTICAST MTU 9600 Metric 1 RX packets 1738 errors 8 dropped 8 overruns 8 Tr ane 9 TX packets 33 errors dropped overruns carrier 0 collisions 0 txqueuelen 1000 RX bytes 299886 292 8 KiB TX bytes 8450 8 2 KiB Interrupt 169 Memory d88800800 dBfTfTTfTT Check the Linux system message logs to see if the iSCSI HBAs on the first two partitions of each port Port O Partition 1 eth2 Port O Partition 2 eth4 Port 1 Partition 1 eth3 and Port 1 Partition 2 eth5 are available fvar log messages monitored System Log Viewer Ale Edit View Help ivarAog localhost bnx2i eth5 free hpa done after 0 retries cron localhost bnx2i eth4 Tree hpa done after O retries maillog localhost bnx2i eth3 free hba done after 0 retries messages localhost bnx2i eth2 free hba done after retries secure localhost Broadcom NetXtreme II iSCSI Driver bnx2i v2 6 4 1b Xorg 0 loq localhost iscsi registered transport bnx2i localhost bnx2i 01 00 03 ISCSI_INIT passed localhost bnxzi 01 00 02 ISCSI INIT passed localhost bnx2i 01 80 01 ISCSI_INI
77. on 559 E er 0013 Broadcom 50457712 iep Rere D 10 Gage Mult Funcion GC Adapter 255 a AER ign 2011 68 com broadonm Sgt E 35 r ars ls m kp 2011 08 com broadcnm tgt IA ram 148 gt Portal Bn 2 20 184 beb ezed ey 0031 Brenn BC 7712 Hetcreme D 10 XE DIS VED Clenz 235 T 0060 Broadcom BCOMST712 Nettreme EL 20 Cf Multi Function 660 E 0033 Broadcom BCE 7712 aer ere D 10 GF DIS VEDO Cheng 60 Ti 006 1 Broadcom BCIMS7712 Netktreme TIT 10 GE Multi Function aa 0014 Broadcom BCMS 7712 HetOneme D 10 E Mult Funcion CS Adapter 161 7 0034 Broadcom BCD 7712 Metres D 10 Gage MOIS VEO Clenz 161 Beer Fle View Acton Pie Context Tools Teams 505 Help j Fee IMA vew Il lienas gt FF scsi Target Information E SCSI Session Indeematon Egiari Y B information sossa a P EC E B unama Target ferma ber E NEM Adapter 5045709 C Target Type EN Adapter OM 5705 00 r Portal List LLL1L222 11 e EM Adapters CMS Ci O Tat GA OOo com broadeoectgtile mlia ES Adapters 8CM57711 40 RZ Adapters 80457712 AT EF Porti B El T 10054 Broadcom BCMS7712 MetXireme I 10 E Mus Funcion i54 ign 2011 05 com broadcomcigt13e rama DH 0027 Broadcom BCMST712 Metteg H 20 GigE NOES VIO Chest 54 RI Pa E OI ii Ia nm gt T 100991 Broadcom BOMS7712 Metureme I 10 Gef Mul Funcion 255 eget Ports TET MEE 0013 Broadcom BCMST712 Netitreme H 10 GaE Mult Funcion GESI Adapter 159 e 11 11 24 Er amp Port
78. on s bandwidth i e 100 which is 10 Gbps in this case Again since the maximum bandwidths are set to 100 for all four partitions there is no difference between the earlier DCB mode example and this one in DCB mode Table 8 Non DCB Oversubscription With One High Priority Partition Relative Bandwidth Maximum Port Partition Weight Bandwidth Protocol Plot Color Port 0 Partition 1 P1i O effectively 1 100 SCSI Offload Green Port 0 Partition 1 P1e 0 effectively 1 100 Ethernet Brown Port 0 Partition 2 P2e 0 effectively 1 100 Ethernet Purple Port 0 Partition 3 P3e 0 effectively 1 100 Ethernet with Yellow TOE Port 0 Partition 4 P4e 100 100 Ethernet Blue Port 0 Partition 4 P4i 100 100 SCSI Offload Red The following plot shows a similar effect to some of the previous examples except that the fourth partition takes as much of the bandwidth as it needs up to 100 when it starts to transmit In this example the three 0 Relative Bandwidth Weight partitions have an effective Relative Bandwidth Weight of 1 instead of 0 e The first partition s traffic flow P1i initially takes 100 of the available bandwidth when the test application starts to transmit traffic on that port by itself at tO e When P1e starts to send Ethernet traffic at t1 both will stabilize to 5 Gbps each e When partition P2 traffic flow P2e starts to send traffic at t2 since only partition s P1 and P2
79. ons sum to 100 and oversubscription all four sum to 180 with different weights and maximum bandwidths When all four or at least the two larger partitions are running traffic they share the space with each other with respect to their partition s weight and maximum bandwidth values otherwise the partition s continue to act as if they are independent connections e The first partition s traffic flow P1i initially takes its designated 8 Gbps when the test application starts to transmit traffic at tO to that port by itself not expanding into the remaining unused 2 Gbps bandwidth e When the second traffic flow on the first partition P1e starts to send at t1 the two active traffic flows on the same partition share its 8 Gbps bandwidth for 4 Gbps each e When the third traffic flow P2e starts sending at t2 it only takes its partitions maximum bandwidth allowed 1 Gbps Partition P1 s two traffic flows are unaffected e When the fourth traffic flow P3e starts sending at t3 it again only takes its partitions maximum bandwidth allowed 1 Gbps Partition P1 s two traffic flows P1i and P1e and the traffic flow on partition P2 P2e are unaffected e But when P4e starts to send traffic at t4 the traffic needs are oversubscribed so the available bandwidth is redistributed based on each partition s individual weights and maximums settings P2e and P3e use 596 each 5 100 so their traffic flows are reduced to 0 5 Gbps which leave
80. ple sss sss eee 35 Table 3 Non DCB Equal Oversubscrtption esee enne nennen nnn nnns 57 Table DCB Equal OversubSeripHOllsssosesensceunietsurti u uuu Order ea EbeRaS ada 60 Table 5 DCB Equal Oversubscription with one Lossless FCoE Offload 62 Table 6 Non DCB Partitioned Oversubscription sees eee eee ee eee 64 Table 7 Non DCB Weighted Oversubscription ccccccssccccsececsecccenceeeneecaesceeeececeuececeneeeeneceseneeeeneeeeeeteneeeges 67 Table 8 Non DCB Oversubscription With One High Priority Barttion 69 Table 9 Non DCB Default Fixed Subscription esse sees eee eee eee 71 Table 10 Non DCB Mixed Fixed Subscription and ODversubscription sss sese eee eee eee 73 Table 11 DCB Mixed Fixed Subscription and Oversubscription with Lossless FCoE Offload 75 Table 12 Non DCB Mixed Fixed Subscription and ODversubscription sss sese eee eee eee 77 BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 4 57712 k Software User Manual About This Document About This Document Purpose This document provides instructions on how to enable NIC Partitioning NPAR on a Dell PowerEdge M710HD and M915 Blade Servers installed with the Broadcom 57712 k Converged Network Daughter Card NDC Dual Port 10 GbE A Fabric option see Figure 1 Figure 1 Broadcom 57712 k Converged NDC Audience This document is written for the network administra
81. r Rel T ce resista 7 Supported Operating Systems 15 Viewing and Configuring the Partitions ccccccsseccssecccssecceecceascecsececeececeeceeeneeseeeeseeeseusesseeegeetens 16 WV IDG OWS Se Ver 2005 TTT 16 Instamng THE Latest Dell Diye soii iii 16 Viewing the Enabled Devices in Device Manager r 19 Broadcom Advanced Control Suite 4 BACGAl a 23 Microsoft Windows Network Connections sss ss see xe x e ee eee eee 31 Device PCIe BUS COCO IO Na 34 Red Hat Enteronse LINUX E 38 UNITE ES decr ABRE 42 satine MITU KT 46 Setting MTU Sizes KA lut e EE 46 Setting MTU Sizes in LINUX esse eee eee ee ee eee 49 Setting MTU Sizes in VMWare ESX ESXi 4 1 nnne nnn nennen nnns 53 ailin iaa 56 Equal OVersuDSerpion ll 57 Partitioned Oversabscription EXamMple u L u L u SERE u anam i bae 64 Weaeighted Ovyersubseripti6n EXAM DIO tarro 67 Oversubscription With One High Priority Partition Example esses 69 Default Fixed Subscription Example sse eee eee eee eee eee 71 Mixed Fixed Subscription and Oversubscription Example sss sees eee eee eee 73 Mixed Weights and Subscriptions Example nes 77 BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 3 57712 k Software User Manual List of Tables List of Tables Table 1 Protocols Available in Operation Systems Versus SF and NPAR Mode 15 Table 2 Port Function MAC Address Exam
82. raffic flows Therefore these four BIL P1e P4e and P4i are allocated 2 Gbps each 8 Gbps 4 equally weighted traffic flows e When P1i stops at t6 it releases it s bandwidth to the available pool and since P2e and P3e are capped by their maximum bandwidth value to 1 Gbps the three other traffic flows P1e P4e and P4i automatically take equal shares and increase their bandwidth used to 2 6 Gbps each e When Ple subsequently stops sending at t7 P4e and P4i grab up the extra available bandwidth and go to 4 Gbps each Both P2e and P3e are unaffected and continue sending at 1 Gbps each e When P2e stops sending at t8 P4e and P4i are not able to make use of the freed up bandwidth since they BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 73 57712 k Software User Manual Examples are both in partition P4 which as a maximum bandwidth ceiling of 8 Gbps Therefore none of the traffic flows increase their sending rates and this unused bandwidth is ignored e When P3e stops sending at t9 the same condition is still in effect Therefore none of the remaining active traffic flows increase their sending rates to use this extra bandwidth e Finally P4e stops at t10 and this allows it s companion traffic flow P4i to increase to 8 Gbps which is partition PA s maximum top end The remaining 2 Gbps is unassigned The following example is the same as the previous example but with FCOE in the first partition Ad
83. rtition 2 4 4 4 20 Status Broadcom BCMS7712 NetXtreme LI 10 GigE NDIS VED Client 475 Ed Broadoom BCM57712 NetXtreme 11 10 GigE NDIS VBD Chent 274 E General Advanced Driver Details Power Management General Advanced Driver Detata Power Management a Broadcom BCM57712 NetXtreme H 10 GigE NDIS w madcom BCM57712 NetXtreme Il 10 GigE NDI VED Cieni 75 La VBD Clent 74 Mit Location Location 4 PCI bus 6 device 0 funcion 1 Location Location 4 PCI bus 6 device 0 function 3 e Broadcom BCM57712 NetXtreme Il 10 GgE NDIS La Ereadeom BCM57712 NetXtreme II 10 GigE NDIS k Ge Ra ee SE Location Location 4 PCI bus 6 device function 5 qeu E device is working property BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 37 57712 k Software User Manual Configuring NPAR Red Hat Enterprise Linux Linux shows the respective device protocols that were enabled in USC if the NetXtreme ll device drivers are installed Go to the Dell Driver Download web site for the latest Linux device drivers and insure they are installed on your system In Linux the Ethernet Protocol is always enabled on all eight partitions and SCSI Offload HBA Protocol was enabled on the first two partitions of each port The following shows the RHEL Network Configuration page with the eight enabled Ethernet protocol partitions always four per port as the Broadcom 57712 k NetXtreme
84. runs 0 frame TX packets 34 errors dropped O overruns O Carrier 0 collisions txqueuelen 1880 RX bytes 4807217 4 5 MiB TX bytes 8B8b6 8 6 KiB Interrupt 204 Memory d4800000 d4tftttt Link encap Ethernet Hwaddr 00 10 18 88 E7 B0 inet addr 10 2 2 200 Bcast 10 255 255 255 Mask 255 0 0 0 inet6 addr TeB8 210 18TT Te8B8 e7b0 b54 Scope Link UP BROADCAST RUNNING MULTICAST HMTU 9600 Ret ric 1 AX packets 2682 errors dropped overruns amp frame 6 TX packets 34 errors 0 dropped 8 overruns O Carrier 0 collisions txqueuelen 1000 RX bytes 472737 461 6 KiB TX bytes 9139 8 9 KiB Imt errupt 204 Memory d5800000 dSTT TT TT Link encap Ethernet Hwaddr 00 10 18 88 E7 B2 inet addr 18 2 2 201 Bcast 10 255 255 255 Mask 255 0 0 0 inet6 addr feB0 210 18TT Te88 e7b2 64 Scope Link UP BROADCAST RUNNING MULTICAST MTU 9680 Ret ric RX packets 2233 errors 0 dropped 8 overruns 8 T rane 0 TX packets 32 errors 8B dropped 6 overruns O Carrier 0 collisions txqueuelen 1606 RX bytes 393371 384 1 KiB TX bytes 8381 8 1 KiB Interrupt 181 Memory d6800888 deTTTTTT BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 39 57712 k Software User Manual Configuring NPAR Link encap Ethernet HWaddr 800 18 18 88 E7 B4 inet addr 18 2 2 282 Bcast 10 255 255 255 Mask 255 0 0 0 inet6 addr fe80 210 18f1 fe288 e7b4 64 Scope Link UP BROADCAST RUNNING MULTICAST MTU 9680 Metric 1 RX packets 1701 errors 8 dropped 8 overru
85. s J Ala MI yiera to enable ord disable ie device Enable IPu6 configuration for this interface Automatically obtain PP address settings with 2 Ska av set IP addresses Manual IP Address Settings Agdross 10 1 1 200 subnet mask 22 00 0 Default gateway address 7 Set MTU to 9600 Set MAU to Note In Linux both the Ethernet and SCSI Offload HBA MTU sizes are changed simultaneously and will have the same value In other words setting eth2 to MTU size of 9600 bytes using the ifconfig eth2 mtu 9608 up sets this examples Port O Partition 1 Ethernet adapter to NITU 9600 bytes and the Port 0 Partition 1 iSCSI Offload HBA adapter to MTU 9600 bytes Note Each partition s MTU size setting can be different Using the above protocol enabled partition example e Port 0 Partition 1 Ethernet and iSCSI Offload HBA can be set to 9600 bytes e Port 0 Partition 2 Ethernet and iSCSI Offload HBA can be set to 5500 bytes e Port 0 Partition 3 Ethernet can be set to 1500 bytes e Port O Partition 4 Ethernet can be set to 9000 bytes In Linux use the ping command with the s size option to verify that Jumbo Frame support is configured throughout the desired network path i e ping s 8972 A B C D The Don t Fragment a k a DF flag is AUTOMATICALLY set in Linux The unfragmentable ping packet size is the desired MTU size to be checked 9000 bytes minus the automatically added overhead 28 bytes or 8972 byt
86. s 9 Gbps free for the other three traffic flows The two other partition s traffic flows are allocated 3 Gbps each 9 Gbps 3 equally weighted traffic flows the total bandwidth for P1i and P1e is 6 Gbps which is less than partition P1 s maximum of 80 of 10 Gbps e When P4i starts to send traffic at t5 the bandwidth is again redistributed P2e and P3e are still using only 0 5 Gbps 5 100 This again leaves 9 Gbps free for the remaining four equally weighted traffic flows therefore these four P1i P1e P4e and P4i all are allocated 2 25 Gbps each 9 Gbps 4 flows where P1i BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 77 57712 k Software User Manual Examples plus P1e and P4e plus P4i totals are 4 5 Gbps each which is less than their respective partition s maximum bandwidth settings e When P1i stops at t6 it releases it s bandwidth to the available pool and since bie and P3e are capped by their relative bandwidth weight values to 0 5 Gbps the three other traffic flows P1e P4e and P4i automatically take equal shares of the remaining bandwidth and bump up their portion to 3 Gbps each where P4e plus P4i total is 6 Gbps which is still less than their respective partition s maximum bandwidth value e When P1e subsequently stops sending at t7 P4e and P4i grab up some of the extra available bandwidth and go to 4 Gbps each where they reach their partition s maximum bandwidth value of 80 or 8 Gbps T
87. s weight values There would be no difference between the previous examples and this example with DCB enabled since the weights are ignored and if all traffic types are in the same single PG Table 7 Non DCB Weighted Oversubscription Relative Bandwidth Maximum Port Partition Weight Bandwidth Protocol Plot Color Port O Partition 1 P1i 10 100 iSCSI Offload Green Port O Partition 1 P1e 10 100 Ethernet with Orange TOE Port O Partition 2 P2e 20 100 Ethernet with Purple TOE Port O Partition 3 P3e 30 100 Ethernet with Yellow TOE Port 0 Partition 4 P4e 40 100 Ethernet with Blue TOE Port 0 Partition 4 P4i 40 100 SCSI Offload Red The following plot shows e The first partition s traffic flow P1i initially takes 100 of the available bandwidth at tO when an iSCSI test application is sending traffic out that port by itself e When P1e starts to send Ethernet traffic at t1 the two active traffic flows have equal weights with respect to each other so they are allocate half of the total bandwidth available 10 Gbps to partition P1 which equates to 5 Gbps each for P1i and P1e e When P2e starts sending at t2 the partition s relative bandwidth weights come into effect Partition P1 has a weight of 10 while P2 has twice as much at 20 so P1 s two sending traffic flows are reduced to half of the partition s assigned 1 3rd derived from P1 s weight of 10 P1 s weight of 10 P2 s we
88. setting them to all 25 is that 0 s causes the send bandwidth to be shared between all active traffic flows while 25 s cause the send bandwidth to be shared between the active sending partitions first and then the active sending traffic type flows in a two step manner Setting them to all 0 s causes the logic to work similarly to the way it does in DCB mode when all traffic types are in the same PG e fthere was only one traffic flow in each partition then the results would be similar to setting each partition s relative bandwidth weight to 096 since the single traffic flow would not be sharing a partition s bandwidth with another traffic type BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 65 Examples 57712 k Software User Manual s ms es m s L tar vm mr Da LL CW A kend mg EH EE mm mg Fee Feelen M em UM NM M i E fen LEE REM em eem es EJ uar rs py nn Page 66 NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R BROADCOM 57712 k Software User Manual Examples Weighted Oversubscription Example The following is an example of weighted oversubscribed bandwidth sharing with different weights assigned to each partition in non DCB mode This example has each partition taking the maximum bandwidth when no other partition is active plus as each partition starts and stops sending the amount of bandwidth is shared as an approximate ratio of the currently sending partition
89. starts to send at t5 the two single traffic flows P2e and P3e will remain at 2 5 Gbps each as well as the P1 partition s traffic flows P1i and P1e each still getting 1 25 Gbps while P4 s allocated BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 64 57712 k Software User Manual Examples bandwidth will now be split into two traffic flows P4e and P4i which means each get 1 25 Gbps e When Pli stops sending at t6 the only partition P1 traffic flow P1e will readjust to 2 5 Gbps and all of the others will remain the same e WhenPle stops sending at t7 the other traffic flows will readjust to 3 3 Gbps P2e and P3e and 1 65 Gbps each for partition P4 s shared P4e and P4i traffic flows e When P2e stops sending at t8 partition P3 s single traffic flow will readjust to 5 Gbps P3e and partition P4 s shared P4e and P4i traffic flows will increase to half that or 2 5 Gbps e When P3e stops sending at t9 the remaining partition P4 will now receive all of the available bandwidth so it s two traffic flows P4e and P4i will equally share it for 5 Gbps each e Finally when P4e stops sending at t10 the remaining traffic flow P4i will now receive all of the available bandwidth or 10 Gbps e fthere is only one flow in a partition it would take all of the bandwidth allocated for that partition e The main difference between setting all four partition s relative bandwidth weight to 096 and
90. t 00 10 18 6F D02 A4 Global Bandwidth Allocation Menu Partition 1 Relative Bandwidth Height d Partition 1 Maximum Bandwidth 10 Partition 2 Relative Bandwidth Height d Partition Z Maximum Bandwidth Partition 3 Relative Bandwidth Height Partition 3 Maximum Bandwidth B Partition d Relative Bandwidth Weight Q Partition 4 Maximum Banduidth BO Cont igure relative bandwidth weight Valid range 0 100 percent The sum total of all four partitions must be equal 0 or 100 BROADCOM December 8 2011 e 2CS57712 SWUM102 R NPAR Setup Guide Page 10 57712 k Software User Manual Configuring NPAR The Relative Bandwidth Weight is the value the port gives to that single partition s send or outgoing traffic with respect to any other actively sending partitions on that port when there is more send traffic pending on the four partitions than send bandwidth available on that port It is more than just a minimum bandwidth setting This setting follows these rules e The individual configurable value range is O to 100 e The SUM of a single port s four partitions values MUST be either exactly 100 or exactly 0 which means all four of the partitions are set to O e f one or more of a partition s weight is set to 0 but the sum is 100 i e not all of the partitions are set to zero then that partition s relative bandwidth weight value is effectively 1 with respect to allocation calculations e Setting all four par
91. t set to 25 and can individually use the maximum bandwidth of the connection i e 10 Gbps in this case In addition to the Ethernet Protocol s being enabled on all four partitions the iSCSI Offload protocol is enabled on Partition 1 and 4 By setting the partition s relative bandwidth weights to 2596 each partition s traffic flows i e P1 s iSCSI P1i L2 Ethernet P1e and P4 s iSCSI P4i L2 Ethernet P4e will be contained in their respective partition while each partition over all takes an equal share of the available bandwidth The traffic flows within that partition can only expand into that partition s allocated by weight portion There would be no difference between the previous examples and this example with DCB enabled since it essentially sets the Relative Bandwidth Weight values to all ZEROs and the PG s ETS would come into play Table 6 Non DCB Partitioned Oversubscription Relative Bandwidth Maximum Port Partition Weight Bandwidth Protocol Plot Color Port 0 Partition 1 P1i 25 100 SCSI Offload Green Port 0 Partition 1 P1e 25 100 Ethernet Brown Port O Partition 2 P2e 25 100 Ethernet with Purple TOE Port 0 Partition 3 P3e 25 100 Ethernet Yellow Port 0 Partition 3 P4e 25 100 Ethernet with Blue TOE Port 0 Partition 4 P4i 25 100 SCSI Offload Red The following plot shows how each traffic type flow must remain within a partition s share of a ports available send bandwidth i
92. tition s values to 0 will give every traffic flow on every partition equal access to the ports available bandwidth without regard to which partition they are on unless restricted by the partition s Maximum Bandwidth settings e Ifthe sum of the relative bandwidth weights is 100 and there is more than one type of traffic flow on a specific partition i e iSCSI and L2 Ethernet or FCoE and L2 Ethernet then the traffic on that specific partition will share the bandwidth being allocated as if there was only one traffic flow on that partition e The weight applies to all enabled protocols on that partition e The Relative Bandwidth Weight is not applicable when in Data Center Bridging DCB mode In DCB mode all traffic flows act as if their Relative Bandwidth Weight is set to all Os e The NPAR transmit direction traffic flow rates are affected by the three main modes in the following ways n non DCB mode where the sum of the partition s Relative Bandwidth Weights equal 100 each Partition s combined traffic flow is equally scheduled to transmit within the limitations of the partition s Relative Bandwidth Weight and Maximum Bandwidth settings and the overall connection s link speed This means a specific partition s Relative Bandwidth Weight value will restrict the traffic flows sharing that partition s bandwidth allocation as if one combined traffic flow with respect to the other actively sending partitions The partition s send flow rate is base
93. tor who wishes to partition the Broadcom network controller on a Microsoft Windows Server 2008 R2 VMWare ESX ESXi 4 1 Oracle Solaris SUSE Linux Enterprise Server SLES and Red Hat Enterprise Linux RHEL system with e upto eight functions four per port Ethernet enabled in addition to e upto four functions two per port SCSI HBA enabled in operating systems where the specific HBA can be enabled Or e uptotwo functions one per port FCoE HBA enabled plus up to two functions one per port iSCSI HBA enabled in operating systems where the specific HBA can be enabled BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 5 57712 k Software User Manual Technical Support Acronyms and Abbreviations In most cases acronyms and abbreviations are defined on first use For a comprehensive list of acronyms and other terms used in Broadcom documents go to http www broadcom com press glossary php Technical Support Broadcom provides customer access to a wide range of information including technical documentation schematic diagrams product bill of materials PCB layout information and software updates through its customer support portal https support broadcom com For a CSP account contact your Sales or Engineering support representative In addition Broadcom provides other product support through its Downloads amp Support site http www broadcom com support BROADCOM NPAR Setup Gui
94. treme IT 10 GE Multi Funcion 253 a Broadcom BOMS7712HP Hetem ll 10 CE OG VED Ghent DEE Properties zlii Broadcom BCM S771276 NetXtreme D 10 Gage Multi Funcion 250 Broadcom BCM S 77126 Netxtreme II 10 GigE Multi Funcion 62 This brings up that device s property window The following is the BCM57712 iSCSI device showing the HBA and connected target session information send receive statistics and ping test results Broadcom BCM57712 NetXtreme II 10 GigE Multi Function 5051 Ada iSCSI MAC 00 10 18 6F D3 BF HBA Fortals 1 1 1 7 FE80 0000 0000 0000 0210 18FF FE6F D3BF TUE iqn 4010 10 com broadcom igt 15a ram 15a iqn 4010 10 com broadcom Tat 15b ram 15b ign 2011 05 com broadcom Tat 1 4a ram 14a iqn 401 1 15 com broadcom Tat 4b ram 14b BROADCOM NPAR Setup Guide December 8 2011 e 2CS57712 SWUM102 R Page 20 57712 k Software User Manual Configuring NPAR The following shows the properties window for the FCoE device showing World Wide IDs connected Fabric and Target information Broadcom BCM57712 Netxtreme II 10 GigE FCoE Adapter 79 Prop General Advanced Driver Details Port WAN IA st Node AN 10 00 00 70 1 FCoE MAL III BEL Fabrics 0300 00 EC L 0481 Np 620056 2003 0 00 00 10 7 8 00 Targets CBO500 50 06 01 50 44 60 0C CE 50 06 01 60 C4 60 0C CE 20 01 61100 U UR UT R2 44 60 UL LE 2U UR UT BULA BU UL LE ZULU Ch250i 20 Up UU UE TTA 70 62 20 0700 0E 11 124 70 62 2D D
95. usable from 4 physical ports to 16 virtual ports This allows better port flexibility traffic isolation service quality and bandwidth tuning for your management backup migration production networks The following shows the vSphere Network Adapters Configuration page with the eight enabled Ethernet protocol partitions always four per port as the Broadcom Corporation NetXtreme II BCM57712 10 Gigabit Ethernet MultiFunction devices vmnic6 through vmnic13 in this example VMWare enumerates the partitions in the order of the PCI function numbers where port 0 has functions 0 2 4 6 which are vmnics 6 8 10 12 and port 1 has functions 1 3 5 7 which are vmnics 7 9 11 13 ESX NPAR example com VMware ESX 4 1 0 228591 Network Adapters Device MAC Address Speed Configured Health Status Broadcom Corporation Broadcom 577125 Processors Memory storage Networking Storage Adapters Metwork Adapters Advanced Settings Power Management BROADCOM December 8 2011 e 2CS57712 SWUM102 R vmnica vmnica vmnic vmnicb vmnic13 wmnicl2 vmnici1 vmnicil 00 10 18 6f d3 be 00 10 18 6f d3 bc 00 10 18 6f d3 ba 00 10 18 6f d3 b8 00 10 18 6f d3 0 00 10 18 6f d3 04 00 10 18 6f d3 2 00 10 18 6f d3 0 2000 Full 10000 Full 1000 Full 10000 Full 4000 Full 10000 Full 3000 Full 10000 Full Negotiate 10000 Full 1000 Full 10000 Full Negotiate 10000 Full Negotiate 10000 Full NPAR Setup Guide 57712 k Software
96. ven Depending upon OS requirements the traffic types for each partition could be L2 Ethernet or SCSI Offload or FCoE Offload where L2 Ethernet can be on any of the four partitions AND up to two SCSI Offload can be on any two of the four partitions OR one FCoE Offload can be on any one of the four partitions plus one SCSI Offload can be on any one of the remaining three partitions The partitions data flows on Windows can be a combination of L2 Ethernet traffic with or without TOE and or HBA traffic SCSI OR FCOE For Linux RHEL v5 x the L2 Ethernet protocol is always enabled and up to two SCSI Offloads can be enabled but no FCoE Offload For Linux RHEL v6 x and SLES 11 SP1 the L2 Ethernet protocol is always enabled and up to two iSCSI Offloads or one FCoE Offload and one SCSI Offload can be enabled For Solaris 10u9 and VMWare ESX ESXi 4 1 only the L2 Ethernet protocol is available for the partitions Equal Oversubscription Example The following is an example of oversubscribed bandwidth sharing in non DCB mode All traffic types over the four partitions of the port have an equal weight i e they are all set to 0 and can individually use the maximum bandwidth of the connection i e 10 Gbps in this case In addition to the Ethernet Protocol s being enabled on all four partitions the iSCSI Offload protocol is enabled on partition 1 and 4 The iSCSI Offload Protocol can be enabled in any two partitions When all of the partitions

57712-k Software User Manual

Contents

Download Pdf Manuals

Related Search

Related Contents