Home

Sun Microsystems 10GBE XFP Network Card User Manual

1. Accounting Adapter Server Gigabit Tagged VLAN 3 All VLANs VLAN 1 VLAN 2 VLAN 3 Shared Media Segment e Software Software Engineering Accounting Engineering PC 1 PC2 PC3 PC 4 Software PC 5 VLAN 2 VLAN 2 VLAN 1 VLAN 3 Adapter Gigabit Tagged VLAN 1 amp 2 FIGURE 8 1 Example of Servers Supporting Multiple VLANs With Tagging Adapters The example network has the following features The physical LAN network consists of a switch two servers and five clients The LAN is logically organized into three different VLANs each representing a different IP subnet m VLAN 1 is an IP subnet consisting of the Main Server Client 3 and Client 5 This represents an engineering group m VLAN 2 includes the Main Server Clients 1 and 2 by means of a shared media segment and Client 5 This is a software development group m VLAN 3 includes the Main Server the Accounting Server and Client 4 This is an accounting group 68 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 The Main Server is a heavily used server that needs to be accessed from all VLANs and IP subnets The server has a Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter installed All three IP subnets are accessed by means of the single physical adapter interface The server is attached to one of the switch s Gigabit Ethernet ports which is configured for VLANs 1 2 and 3
2. q 6 Select the packages you want to install m Press Return or type all to accept the default and install all packages m Type the specific numbers separated by a space if you prefer not to install any optional packages Chapter 2 Installing and Setting Up the Driver 7 7 Verify that the nxge driver is installed on the system Sun PCI E 10G 1G Ethernet Adapter Driver i386 1 0 REV 2006 12 05 10 0 Copyright 2006 Sun Microsystems Inc All rights reserved Use is subject to license terms Executing checkinstall script Using as the package base directory Processing package information Processing system information 4 package pathnames are already properly installed Verifying package dependencies Verifying disk space requirements Checking for conflicts with packages already installed Checking for setuid setgid programs This package contains scripts which will be executed with super user permission during the process of installing this package Do you want to continue with the installation of y n y Installing Sun PCI E 10G 1G Ethernet Adapter Driver as Installing part 1 of 1 kernel drv amd64 nxge verifying class Executing postinstall script Installation of was successful v To Remove the Driver from a Solaris Platform 1 To discover the driver packages run the pkginfo command pkginfo grep SUNWnxge SUNWnxge Sun PCI E 10G 1G Ethernet Adapt
3. Both the adapter and the connected switch port have tagging turned on The tagging VLAN capabilities of both devices enable the sever to communicate on all three IP subnets in this network yet continue to maintain broadcast separation among the three subnets The following list describes the components of this network m The Accounting Server is available to VLAN 3 only It is isolated from all traffic on VLANs 1 and 2 The switch port connected to the server has tagging turned off m Clients 1 and 2 are attached to a shared media hub that is then connected to the switch They belong to VLAN 2 only and are logically in the same IP subnet as the Main Server and Client 5 The switch port connected to this segment has tagging turned off m Client 3 is a member of VLAN 1 and can communicate only with the Main Server and Client 5 Tagging is not enabled on Client 3 s switch port m Client 4 is a member of VLAN 3 and can communicate only with the servers Tagging is not enabled on Client 4 s switch port m Client 5 is a member of both VLANs 1 and 2 and has a Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter installed It is connected to switch port 10 Both the adapter and the switch port are configured for VLANs 1 and 2 and have tagging enabled VLAN tagging is only required to be enabled on switch ports that create trunk links to other VLAN aware Ethernet switches or on ports connected to tag capable end stations
4. DR is not supported Installing the Adapter If you are installing the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter into a machine running Solaris 10 you must install the software before you install the hardware The following instructions describe the basic tasks required to install the adapter Refer to your system installation or service manual for detailed PCI Express adapter installation instructions specifically you must check which bracket your server requires low profile or standard height and install the standard height bracket if required Caution Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter mechanical fit limitations Do not install the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter in slot 2 of a Sun Fire T2000 system The SATA cables from the adjacent card slot prevent the proper installation of an Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter in this slot Note that this issue does not exist for Sun Fire T2000 that is RoHS compliant Caution Do not install the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter in slot 5 of a Sun Fire X4600 system because the heatsink on the adapter will sit against the daughter card of the Sun Fire X4600 system Note When you install the adaper ensure that the optical transceiver has not been installed The optical transceiver must be installed in
5. a Check the etc driver_aliases file to identify the name associated with a particular device grep nxge etc driver aliases nxge pciex108e abcd b Locate the path names and the associated instance numbers in the etc path to inst file grep nxge etc path to inst pci 780 pci 0 pci 8 network 0 0 nxge pci 780 pci 0 pci 8 network 0 1 1 nxge m In this example The first part within the double quotes specifies the hardware node name in the device tree The number not enclosed in quotes is the instance number shown in bold italics for emphasis The last part in double quotes is the driver name To identify a PCI E device unambiguously in the nxge conf file use the name parent name and the unit address for the device Refer to the pci 4 man page for more information about the PCI E device specification In this example m name pciex108e abcd m parent pci 780 pci 0 pci 8 network 0 mg unit address 0 2 Set the parameters for the nxge devices in the platform sun4u kernel drv nxge conf file a The following parameters can be set using the platform sun4u kernel drv nxge conf file 3t Gk PRESS rene Link Configuration The link parameters depend on the type of the card and the port 10 gigabit related parameters i e adv 10gfdx cap Chapter 5 Configuring the nxge Device Driver Parameters 39 HE OH db db db db db db OH
6. le des exportations et la liste de ressortissants sp cifiquement d sign s sont rigoureusement interdites L utilisation de pi ces d tach es ou d unit s centrales de remplacement est limit e aux r parations ou l change standard d unit s centrales pour les produits export s conform ment la l gislation am ricaine en mati re d exportation Sauf autorisation par les autorit s des Etats Unis l utilisation d unit s centrales pour proc der des mises jour de produits est rigoureusement interdite LA DOCUMENTATION EST FOURNIE EN L ETAT ET TOUTES AUTRES CONDITIONS DECLARATIONS ET GARANTIES EXPRESSES OU TACITES SONT FORMELLEMENT EXCLUES DANS LA MESURE AUTORISEE PARLA LOI APPLICABLE Y COMPRIS NOTAMMENT TOUTE GARANTIE IMPLICITE RELATIVE A LA QUALITE MARCHANDE A L APTITUDE A UNE UTILISATION PARTICULIERE OU A L ABSENCE DE CONTREFACON 5 cx CA Adobe PostScript Contents Preface xiii Product Overview 1 Components 1 Product Description 1 Hardware and Software Requirements 2 Features 3 Installing and Setting Up the Driver 5 Downloading and Installing the Driver on a Solaris SPARC or x86 Platform 5 v To Download the Driver on a Solaris Platform 5 v To Remove the Driver from a Solaris Platform 8 Downloading and Installing the Driver on a Linux Platform 9 Y To Remove the Driver From a Linux Platform 10 Installing the Adapter 11 Installing the Adapter 11 v Tolnstallthe Adapter 12 Installing an Optical Tran
7. you must specify the device instance for the utility Check the etc path_to_inst file to identify the instance associated with a particular device grep nxge etc path_to_inst pci 7c0 pci 0 pci 9 network 0 0 nxge pci 7c0 pci 0 pci 9 network 0 1 1 nxge To Specify Parameter Values Using the ndd Utility This section describes how to modify and display parameter values To modify a parameter value use the set option If you invoke the ndd utility with the set option the utility passes value which must be specified down to the named dev nxgedriver instance and assigns the value to the parameter ndd set dev nxgeX parameter value Where X is the driver instance for example dev nxge0 dev nxgel To display the value of a parameter specify the parameter name and omit the value When you omit the set option the utility queries the named driver instance retrieves the value associated with the specified parameter and prints it ndd dev nxgeX parameter To Use the ndd Utility in Interactive Mode To modify a parameter value in interactive mode specify ndd dev nxgeX ndd dev nxge0 name to get set Enter the parameter name or to view all parameters Chapter 5 Configuring the nxge Device Driver Parameters 37 After you enter the parameter name the ndd utility prompts you for the parameter value 2 To list all the parameters supported by the nxge
8. 3 netmask ffffff00 broadcast 199 199 123 255 ether 8 0 20 a4 4f b8 nxge224000 flags 1000843 lt UP BROADCAST RUNNING MULTICAST IPv4 gt mtu 1500 index 4 inet 199 199 224 3 netmask ffffff00 broadcast 199 199 224 225 ether 8 0 20 a4 4f b8 3 On the switch set VLAN tagging and set VLAN ports to coincide with the VLANs you ve set up on the server Using the examples in Step 2 you would set up VLAN ports 123 and 224 on the switch Refer to the documentation that came with your switch for specific instructions for setting VLAN tagging and ports Configuring VLANs in a Linux Environment v To Configure VLANs in a Linux Environment 1 Ensure that the nxge module is loaded modprobe nxge Chapter 8 Configuring VLANs 71 72 Plumb the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter interface ifconfig eth2 xxx xxx xx xxx up where xxx xxx xx xxx the IP address of the interface Insert the VLAN module sbin modprobe 8021q Add the VLAN instance VID sbin vconfig add eth2 5 Configure the nxge VLAN eth2 in this example ifconfig eth2 5 xxx xxx xx xxx up where xxx xxx xx xxx the IP address of the interface Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 APPENDIX A Specifications This appendix lists the specifications for the Sun x8 Express Dual 10 Gigabit Eth
9. 42 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 d The following example shows ports on two different cards being set Only one node needs to be specified name pciex108e abcd parent pci 780 pci 0 pci 8 unit address 0 class opt ipv4 tcp 0x100 name pciex108e abcd parent pci 7c0 pci 0 pci 9 unit address 0 class opt ipv4 tcp 0x40 3 Save the nxge conf file Tuning for Maximum Performance on a Solaris Platform Tuning for maximum performance in a Solaris platform depends on whether you are using an UltraSPARC CPU based platform or an AMD CPU based platform v To Improve Performance on an UltraSPARC CPU Based Sun Platform 1 Improve performance by adding the following etc systen file set ddi msix alloc limit 4 Increasing the MSI improves the Rx performance The default value for MSI is 2 but changing it to 4 improves performance 8 can be used for UltraSparc T1 based systems 2 Reboot the system reboot r Chapter 5 Configuring the nxge Device Driver Parameters 43 3 Add the following to a startup script or use ndd before plumbing the interface ndd set dev ip ip soft rings cnt 8 Utilizing more soft rings provided by the Solaris TCP IP stack significantly improves bulk throughput for Rx The default number of soft rings is 2 but changing it to 8 improves performance You can increase the number to 16 in UltraSparc T1
10. Overview Configuring Jumbo Frames enables the Ethernet interfaces to send and receive packets larger than the standard 1500 bytes However the actual transfer size depends on the switch capability and the ethernet adapter driver capability Note Refer to the documentation that came with your switch for exact commands to configure Jumbo Frames support Checking Jumbo Frames Configurations The jumbo frame configuration checking occurs at Layer 2 or Layer 3 depending on the configuration method 53 v To Show the Driver Statistics in a Solaris Environment 1 Use the kstat command to display driver statistics for example kstat nxge 1 grep rdc_packets rdc_packets 798982054 rdc_packets 792546171 rdc_packets 803941759 rdc_packets 805674872 rdc_packets 798714912 rdc_packets 799293256 rdc_packets 806470537 rdc_packets 805413540 The previous example diplays the receive packet counts on all of the eight Receive DMA channels on interface 1 Using the kstat nxge 1 shows all the statistics that the driver supports for that interface 54 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 2 Use the kstat command to display driver statistics of a VLAN interface for example kstat nxge 38001 module nxge name nxge38001 brdcstrcv brdcstxmt collisions crtime ierrors ifspeed ipackets ipackets64 multircv multixmt norcvbuf noxmtbuf obytes obytes64 oerrors opackets opacket
11. based systems v To Improve Performance on an AMD CPU Based Sun Platform 1 Enable soft rings and change to a higher value than the default of 2 by adding the following to the etc systen file set ip ip squeue fanout 1 set ip squeue soft ring 1 Bulk throughput for Rx can be significantly improved by utilizing more soft rings provided by the Solaris TCP IP stack Soft rings are disabled by default in previous and current releases of the Solaris x86 Operating System 2 Reboot the system reboot r 3 Set the MSI to 1 on AMD platforms by adding following to the etc systen file set ddi msix alloc limit 1 4 Reboot the system reboot r 44 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 v To Obtain Higher Throughput Using the Generic Tunables for the Solaris TCP IP Stack To obtain higher throughput add the following to a startup script ndd ndd ndd ndd ndd ndd set set set set set set dev tcp dev tcp dev tcp dev tcp dev tcp dev tcp tcp conn req max q 8192 tcp conn req max q0 8192 tcp max buf 4194304 tcp cwnd max 2097152 tcp recv hiwat 400000 tcp xmit hiwat 400000 Chapter 5 Configuring the nxge Device Driver Parameters 45 46 Setting Parameters on a Linux Platform v To Set Parameters Using the ethtool Utility 1 Determine which parameters are available using the ethtool utility ethtoo
12. by that adapter must be in VLAN tagged format v To Configure Static VLANs 1 Create one hostname nxgenumber file for each VLAN that will be configured for each adapter on the server Use the following naming format which includes both the VID and the physical point of attachment PPA VLAN logical PPA 1000 VID Device PPA nxge123000 1000 123 nxge This format limits the maximum number of PPAs instances you can configure to 1000 in the etc path to inst file For example on a server with the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter having an instance of 0 belonging to a member of two VLANs with VID 123 and 224 you would use nxge123000 and nxge224000 respectively as the two VLAN PPAs 70 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 2 Use the ifconfig 1M to configure a VLAN virtual device for example ifconfig nxge123000 plumb up ifconfig nxge224000 plumb up The output of ifconfig a on a system having VLAN devices nxge123000 and nxge224000 ifconfig a 100 flags 1000849 lt UP LOOPBACK RUNNING MULTICAST IPv4 gt mtu 8232 index 1 inet 127 0 0 1 netmask ff000000 hme0 flags 1000843 lt UP BROADCAST RUNNING MULTICAST IPv4 gt mtu 1500 index 2 inet 129 144 131 91 netmask ffffff00 broadcast 129 144 131 255 ether 8 0 20 a4 4f b8 nxge123000 flags 1000843 lt UP BROADCAST RUNNING MULTICAST IPv4 gt mtu 1500 index 3 inet 199 199 123
13. db db db db db db db db Hb OH HO Hb Hb Hb HB HB HB db db db db db db dB db db db db db o dt He S Hb OH OH db db db od apply only to 10gigabit ports Half duplex is not supported on any NIU card adv autoneg cap Advertise auto negotiation capability default is 1 adv autoneg cap 1 adv 10gfdx cap Advertise 10gbps Full duplex capability default is 1 adv 10gfdx cap 1 adv 1000fdx cap Advertise 1gbps Full duplex capability default is 1 adv 1000fdx cap 1 adv 100fdx cap Advertise 100mbps Full duplex capability default is 1 adv 100fdx cap 1 adv 10fdx cap Advertise 10mbps Full duplex capability default is 1 adv 10fdx cap 1 adv asmpause cap Advertise Asymmetric pause capability default is 0 adv asmpause cap 0 adv pause cap Advertise pause capability default is 1 adv pause cap 1 ceci Jumbo frame support To enable jumbo support for all nxge interfaces accept jumbo 1 To disable jumbo support for all nxge interfaces accept jumbo 0 Default is 0 See the example at the end of this file for enabling or disabling jumbo for a particular nxge interface 40 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 HE OH OH OH HB HB db db db db db db db db db OH OH zt dE o db Hb HB HB db db db db db Hb db db db db db db db HH db db db db db dt ut dt rxdma intr time rxdma Defaul rxdma rxdma class cla
14. install server s hard drive The package is a folder SUNWnxge v or SUNWnxge u which you can download from the following web site http www sun com download products xml id 44eblefd 4 On the install server install the Sun x8 Express Dual 10 Gigabit Ethernet software to the client s root directory as determined in Step 2 Replace root directory with the location of the client s root directory cd location where you downloaded the packages 1s SUNWnxge pkgadd R root directory Solaris_10 Tools Boot d SUNWnxge v Note If the commands above do not work correctly refer to the documentation for your version of the Solaris Operating System Note Perform the following steps on the client system 28 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 5 Shut down and halt the client system Use the shutdown command to go to the OpenBoot ok prompt shutdown i0 g0 y shutdown command messages omitted ok 6 At the ok prompt use the show nets command to find the device path of the 10 Gigabit Ethernet device The show nets command lists the system devices You should see the full paths and names of the network devices similar to the example below ok show nets pci 7c0 pci 0 pci 8 network 0 1 pci 7c0 pci 0 pci 8 network 0O pci 7c0 pci 0 pci 2 network 0O 1 pci 7c0 pci 0 pci 2 network 0 pci 780 pci 0 pci l network 0 1 pci 780 pci 0 pci l network 0 NO S
15. network 0 pci 7c0 pci 0 pci 2 network 0 1 pci 7c0 pci 0 pci 2 network 0 pci 780 pci 0 pci l network 0O 1 pci 780 pci 0 pci l network 0 NO SELECTION from the above list Hh D A wo ia Checking the properties output for each device is the surest way to identify the device Usually pci 7c0 pci 0 pci 8 or pci 7c0 pci 0 pci 9 correspond to PCIe slots so look at those devices first a pci 7c0 pci 0 pci 8 network 0O 1 b pci 7c0 pci 0 pci 8 network O Note If you do not see the device listed check that the adapter is properly seated If necessary reinstall the adapter 20 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 3 View the device that you installed Using the previous example type ok cd pci 7c0 pci 0 pci 8 network 0O 1 4 Use the properties command to display a list of device properties The properties command displays the specific information about the device If you are using the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter your output will be similar to the following ok properties assigned addresses 82080010 00000000 04000000 00000000 01000000 82080018 00000000 03500000 00000000 00008000 82080020 00000000 03508000 00000000 00008000 82080030 00000000 03600000 00000000 00100000 local mac address 00 14 4f 6b ce 88 phy type xgf reg 00080000 00000000 00000000 00000000 00000000 03080010
16. no no no no nxgel passive short yes no no no no no For more information refer to the man pages for dladm man dladm Configuring Link Aggregation in a Linux Environment This section explains how to configure link aggregation in a Linux environment v To Configure Bonding for Multiple nxge Interfaces 1 Modify the etc modprobe conf file for 2 6 kernels file by adding these lines alias bond0 bonding options bonding max bonds 2 mode 4 miimon 1000 where bond0 is the bonding device max bonds is the number of bond interfaces to be created mode specifies the bonding policies miimon is the frequency in milliseconds that MII link monitoring will occur Refer to Linux documentation for more information 2 Load the bonding driver modprobe bonding 64 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 3 Configure bondo interface In this example bondo is the master of the two interfaces eth4 and eth5 ip addr add 192 12 38 64 24 brd dev bond0 ip link set dev bond0 up ifenslave bond0O eth4 eth5 Chapter 7 Configuring Link Aggregation 65 66 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 CHAPTER 8 Configuring VLANs This chapter explains Virtual Local Area Networks VLANs in detail and provides configuration instructions and examples It contains the following sections m Overview of VLANs on page 67 m Configu
17. nxge conf file How to set parameters for a particular interface The example below shows how to locate the device path and set a parameter for a particular nxge interface Using jumbo support as an example Use the following command to find out the device paths for nxge more etc path_to_inst grep nxge For example if you see pci 7c0 pci 0 pci 8 network 0 0 nxge pci 7c0 pci 0 pci 8 network 0 1 1 nxge pci 7c0 pci 0 pci 8 network 0 2 2 nxge pci 7c0 pci 0 pci 8 network 0 3 3 nxge then you can enable jumbo for ports 0 and 1 and disable jumbo for ports 2 and 3 as follows name pciex108e abcd parent pci 7c0 pci 0 pci 8 unit address Qn accept_jumbo 1 name pciex108e abcd parent pci 7c0 pci 0 pci 8 unit address 2m accept jumbo 1 name pciex108e abcd parent pci 7c0 pci 0 pci 8 unit address memor accept jumbo 0 name pciexi108e abcd parent pci 7c0 pci 0 pci 8 unit address Eno accept jumbo 0 c In the following example the ports of all the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter are being set for load balancing Rx traffic based on IP source address The default value is F80 indicating Rx load balancing based on IP 5 tuple Notice the semi colon at the end of the last parameter class opt ipv4 tcp class opt ipv4 udp
18. pos es de Sun Microsystems Inc aux Etats Unis et dans d autres pays Toutes les marques SPARC sont utilis es sous licence et sont des marques de fabrique ou des marques d pos es de SPARC International Inc aux Etats Unis et dans d autres pays Les produits portant les marques SPARC sont bas s sur une architecture d velopp e par Sun Microsystems Inc UNIX est une marque d pos e aux Etats Unis et dans d autres pays et licenci e exlusivement par X Open Company Ltd UNIX Te T Express Le logo Adobe et le logo PostScript sont des marques de fabrique ou des marques d pos es de Adobe Systems Incorporated Ce produit est soumis la l gislation am ricaine en mati re de contr le des exportations et peut tre soumis la r glementation en vigueur dans d autres pays dans le domaine des exportations et importations Les utilisations ou utilisateurs finaux pour des armes nucl aires des missiles des armes biologiques et chimiques ou du nucl aire maritime directement ou indirectement sont strictement interdites Les exportations ou reexportations vers les pays sous embargo am ricain ou vers des entit s figurant sur les listes d exclusion d exportation am ricaines y compris mais de mani re non exhaustive la liste de personnes qui font objet d un ordre de ne pas participer d une facon directe ou indirecte aux exportations des produits ou des services qui sont r gis par la l gislation am ricaine en mati re de contr
19. such as servers or workstations with VLAN aware adapters Configuring VLANs in a Solaris Environment VLANs can be created according to various criteria but each VLAN must be assigned a VLAN tag or VLAN ID VID The VID is a 12 bit identifier between 1 and 4094 that identifies a unique VLAN For each network interface nxge0 and nxge1 4094 possible VLAN IDs can be selected Only 512 unique IDs can be used simultaneously Because IP subnets are commonly used it is best to use IP subnets when setting up a VLAN network interface This means that each VID assigned to a VLAN interface of a physical network interface will belong to different subnets Chapter 8 Configuring VLANs 69 Tagging an Ethernet frame requires the addition of a tag header to the frame The header is inserted immediately following the destination MAC address and the Source MAC address The tag header consists of two bytes of Ethernet Tag Protocol identifier TPID 0x8100 and two bytes of Tag Control Information TCI FIGURE 8 2 shows the Ethernet Tag Header format Octet TPID 0x8100 3 bits 1 12 bytes bit User priority CFI VID FIGURE 8 2 Ethernet Tag Header Format By default a single VLAN is configured for every port This groups all ports into the same broadcast domain just as if there were no VLANs at all VLAN tagging for the switch port is turned off Note If you configure a VLAN virtual device for an adapter all traffic sent or received
20. the device driver installed a loopback connector in place and Intervention mode enabled before running netlbtest netlbtest cannot run if the network interface is connected to a live network and requires that the Ethernet device be configured offline before running the test Use the ifconfig 1M command to bring the Ethernet device down before running netlbtest To Use the netlbtest Ensure that the Sun VTS software and the nxge driver are installed on your system Plug in a loopback cable Unplumb the interface from the system using the ifconfig command ifconfig nxgeinstance down ifconfig nxgeinstance unplumb Where instance is the instance number of the interface Refer to SunVTS documentation for instructions on how to run netlbtest Appendix B Diagnostic Software 79 80 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 Index Symbols etc hostname nxgenumber file 24 A assigning an IP address 24 C configuring the network host files 23 configuring VLANS 69 D diskless clients 25 documentation SunVTS 78 E editing the network host files xiii 23 H hardware components 1 requirements 2 l ifconfig a 71 ifconfig command 24 installing the adapter without DR 13 installing the Solaris operating environment over the network 27 N ndd utility 37 P parameter values how to modify and display 37 S setting nxge driver parameters us
21. to form a link aggregation group This link aggregation group appears to MAC clients as a regular link Link aggregation is defined by IEEE 802 2ad and it provides the following benefits Increased bandwidth Linearly incremental bandwidth Load sharing Automatic configuration Rapid configuration and reconfiguration Deterministic behavior Low risk of duplication or mis ordering Support of existing IEEE 802 3 MAC clients Configuring Link Aggregation in a Solaris Environment This section explains how to configure link aggregation in a Solaris environment 61 v To Configure Link Aggregation in a Solaris Environment 1 Aggregate nxge0 and nxgel to form an aggregation and a random number 33 as key a Unplumb the interfaces to be aggregated ifconfig down unplumb nxge0 ifconfig down unplumb nxgel b Create a link aggregation group with key 33 without specifying mode dladm create aggr d nxge0 d nxgel 33 As the command returns one line appears in etc aggregation conf file and indicates that the default mode is off as shown in the following example tail 1 etc aggregation conf 33 L4 2 nxge0 0 nxgel1 0 auto off short dladm show aggr key 33 0x0021 policy L4 address 0 3 ba d8 9d e8 auto device address speed duplex link state nxge0 0 3 ba d8 9d e8 10000 Mbps full up standby nxgel 0 3 ba d8 9d e9 10000 Mbps full up standby dladm show link aggr33 aggr33 type non vlan
22. 00000000 00000000 00000000 01000000 03080018 00000000 00000000 00000000 00008000 03080020 00000000 00000000 00000000 00008000 03080030 00000000 00000000 00000000 00100000 version 2XGF 10G Ethernet Adapter FCode 1 26 06 11 14 board model 501 7283 04 model SUNW pcie 2xgf compatible pciex108e abcd 108e 0 0 pciex108e abcd 108e 0 pciex108e abcd 0 pciex108e abcd pciexclass 020000 pciexclass 0200 pcil08e abcd address bits 00000030 max frame size 00002400 network interface type ethernet device_type network name network fcode rom offset 00000000 interrupts 00000001 class code 00020000 subsystem vendor id 0000108e revision id 00000000 device id 0000abcd vendor id 0000108e Chapter 3 Installing the Adapter 21 5 Type the following when you finish looking at the properties values ok device end Rebooting the System After verifying the adapter installation use the boot r command to perform a reconfiguration boot on your system ok boot r 22 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 CHAPTER 4 Network Configuration This chapter describes how to edit the network host files after the adapter has been installed on your system This chapter contains the following sections m Configuring the Network Host Files on page 23 m Setting Up a 10 Gigabit Ethernet Network on a Diskless Client System on page 25 m Installing the Solaris Operating System Over a 10 Gigabit Et
23. 255 Scotland United Kingdom Fax 650 786 3723 Tel 44 1 506 672 539 Fax 44 1 506 670 011 xi xii Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 Preface This guide provides instructions for installing both the hardware and software for the Sun Dual 10GbE XFP PCI Express card This manual also describes how to configure the nxge driver which controls the card These instructions are designed for enterprise system administrators with experience installing network hardware and software Note In this document the term x86 refers to 64 bit and 32 bit systems manufactured using processors compatible with the AMD64 or Intel Xeon Pentium product families For supported systems see the Solaris Hardware Compatibility Guide How This Document Is Organized Chapter 1 provides an overview of the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter Chapter 2 explains how to install the nxge device driver software in a Solaris SPARC Solaris x86 or Linux environment Chapter 3 describes how to install the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter in your system and verify that it has been installed correctly Chapter 4 describes how to edit the network host files after the adapter has been installed on your system Chapter 5 explains how to set the nxge device driver parameters to customize each device in your system xiii Chapter 6 describes how to configur
24. CD Note Refer to the Solaris Advanced Installation Guide for more information about installing the Solaris Operating System over the network To Install the Solaris Operating System Over a 10 Gigabit Ethernet Network Prepare the install server and client system to install the Solaris Operating System over the network The Solaris Advanced Installation Guide describes how to create the install server and set up the client systems Note If you want to install the client system over a network that is not part of the same subnet you must also create a boot server The Solaris Advanced Installation Guide describes how to create a boot server Chapter 4 Network Configuration 27 2 Find the root directory of the client system The client system s root directory can be found in the install server s etc bootparams file Use the grep command to search this file for the root directory grep client name etc bootparams client name root server name netinstall Solaris 10 Tools Boot install server name netinstall boottype in rootopts rsize 32768 In this example the root directory for the Solaris 10 client is netinstall In Step 4 you would replace root directory with netinstall Note If the root directory is not found in the etc bootparams file refer to the Solaris Advanced Installation Guide for configuration instructions 3 Download the Sun x8 Express Dual 10 Gigabit Ethernet driver onto the
25. ELECTION from the above list Hh D Q Q Oo D iQ 7 At the ok prompt boot the client system using the full device path of the 10 Gigabit Ethernet device for example ok boot pci 7c0 pci 0 pci 8 network 0 8 Proceed with the Solaris Operating System installation Refer to the Solaris Advanced Installation Guide for more information about installing the Solaris Operating System over the network Chapter 4 Network Configuration 29 9 After installing the Solaris Operating System install the Sun x8 Dual 10 Gigabit Ethernet software on the client system The software installed in Step 4 is required to boot the client system over the 10 Gigabit Ethernet interface You now need to install the software in order for the operating system to use the client s 10 Gigabit Ethernet interfaces in normal operation Before installing the Sun 10 Gigabit Ethernet driver ensure that the client system does not already have the driver installed Use the pkginfo command to see if the Sun 10 Gigabit Ethernet software packages are installed on the client system pkginfo grep SUNWnxge m If the software is installed the previous command will return the package name you typed in In that case skip to Step 10 m If the software is not installed install the software from the download center See Chapter 2 for instructions on installing the required software packages 10 Confirm that the network h
26. GURE 1 1 shows the adapter FIGURE 1 1 Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter Hardware and Software Requirements Before installing the adapter make sure your system meets the hardware and software requirements TABLE 1 1 lists the supported hardware and software TABLE 1 1 Hardware and Software Requirements for Sun SPARC and X64 Servers Requirements Hardware or Software Hardware Sun Fire T1000 Sun Fire T2000 Sun Fire V445 Sun Fire V245 Sun Fire V215 Sun Fire U45 Sun Fire X4600 Sun Fire X4200 Sun Fire X4100 Sun Fire X2100 Operating System Solaris 10 01 07 Operating System SuSE Linux Enterprise Server 10 RedHat Enterprise Linux 4 0_u4 RedHat Enterprise Linux 4 0_u3 Optical transceivers X5558A SR Short Reach XFP Transceiver for the base board X5560A z LR Long Reach XFP Transceiver for the base board 2 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 Features The Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter provides the following features Two full duplex 10 gigabit Ethernet interface that use a 10 gigabit Ethernet small form factor pluggable XFP optical transceiver with duplex LC fiber connector IEEE 802 3ae 2002 compliant Uses Sun s own ASIC and software for innovative throughput networking design Networking I O virtualization supporting Solaris LDOM 1 0 VMware support planned for mid 07 Hardware based flow class
27. SS 4v SUN microsystems Sun Dual 10GbE XFP PCI Express Card User s Guide Sun Microsystems Inc www sun com Part No 820 0184 13 January 2009 Revision A Submit comments about this document at http www sun com hwdocs feedback Copyright 2009 Sun Microsystems Inc 4150 Network Circle Santa Clara California 95054 U S A All rights reserved Sun Microsystems Inc has intellectual property rights relating to technology embodied in the product that is described in this document In pu and without limitation these intellectual property rights may include one or more of the U S patents listed at ttp www sun com patents and one or more additional patents or pending patent applications in the U S and in other countries Sun Sun Microsystems the Sun logo SunVTS and Solaris are trademarks or registered trademarks of Sun Microsystems Inc in the U S and other countries UNIX is a registered trademark in the U S and other countries exclusively licensed through X Open Company Ltd UNIX PCI Express The Adobe logo and the PostScript logo are trademarks or registered trademarks of Adobe Systems Incorporated Products covered by and information contained in this service manual are controlled by U S Export Control laws and may be subject to the export or import laws in other countries Nuclear missile chemical biological weapons or nuclear maritime end uses or end users whether direct or indirect are strictly prohi
28. bited Export or reexport to countries subject to U S embargo or to entities identified on U S export exclusion lists including but not limited to the denied persons and specially designated nationals lists is strictly prohibited Use of any spare or replacement CPUs is limited to repair or one for one replacement of CPUs in products exported in compliance with U S export laws Use of CPUs as product upgrades unless authorized by the U S Government is strictly prohibited DOCUMENTATION IS PROVIDED AS IS AND ALL EXPRESS OR IMPLIED CONDITIONS REPRESENTATIONS AND WARRANTIES INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE OR NON INFRINGEMENT ARE DISCLAIMED EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID Copyright 2009 Sun Microsystems Inc 4150 Network Circle Santa Clara Californie 95054 Etats Unis Tous droits r serv s Sun Microsystems Inc d tient les droits de propri t intellectuels relatifs la technologie incorpor e dans le produit qui est d crit dans ce document En particulier et ce sans limitation ces droits de propri t intellectuelle peuvent inclure un ou plus des brevets am ricains list s l adresse http www sun com patents et un ou les brevets suppl mentaires ou les applications de brevet en attente aux Etats Unis et dans les autres pays Sun Sun Microsystems le logo Sun SunVTS et Solaris sont des marques de fabrique ou des marques d
29. driver type ndd dev nxgel name to get set read only function_number read only adv_autoneg_cap read and write adv_10gfdx_cap read and write adv_1000fdx_cap read and write adv_100fdx_cap read and write adv_10fdx_cap read and write adv_pause_cap read and write accept_jumbo read and write rxdma_intr_time read and write rxdma_intr_pkts read and write class_opt_ipv4_tcp read and write class_opt_ipv4_udp read and write class_opt_ipv4_ah read and write class_opt_ipv4_sctp read and write class_opt_ipv6_tcp read and write class_opt_ipv6_udp read and write class opt ipv6 ah read and write class opt ipv6 sctp read and write Setting Parameters Using the nxge conf File Specify the driver parameter properties for each device by creating a nxge conf file in the kernel drv directory Use a nxge conf file when you need to set a particular parameter for a device in the system The man pages for prtcon 1M and driver conf 4 include additional details The next procedure shows an example of setting parameters in a nxge conf file To access any man page type the man command plus the name of the man page For example to access man pages for prtconf 1M type man prtconf 38 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 v To Set Driver Parameters Using an nxge conf File 1 Obtain the hardware path names for the nxge devices in the device tree
30. e that the boot order of Hard Drive is higher than network devices Refer the boot device order image The reconfiguration boot attaches the driver to the adapter You can now configure the driver parameters for your Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter Chapter 4 Network Configuration 31 Boot He ak HE ak ak ak ak ak ak ak ak ak ak ak ak ak ak ak Me ak ak ak ak ak ak ak oie ak Me ak ak ak ak ak ak ak k ak Me k ak k ak ak ak ak ak ak ak Me me oe k ak ak Me ak k He k k Me k k k k k k Device Priority Specifies the boot Boot Device Removable Dey available devices Boot Device CD DVD 2s Boot Device Hard Drive device enclosed parenthesis has be Boot Device 2 NVIDIA Boot Agen disabled in the Boot Device Network IBA GE Slo corresponding type Boot Device Network IB GE Slo menu Boot Device Network IBA GE Slo Boot Device Network IB GE Slo Boot Device Network IB amp GE Slo Boot Device Network IB GE Slo Select Scree Boot Device Network IBA GE Slo Select Item 4 Change Optio F1 General Help F10 Save and Exi ESL Exit Wk k ak ak ak ak ak ak ak ak ak ak ak ak ak ak ak ak ak ak ak ak ak ak ak ak ak ak ak ak ak ak ak ak ak ak k ak ak ak ak ak ak ak ak ak ak k ak ak ak ak ak ak ak k k k k k k k k k k k k k 9 Press the F10 key to save the boot configuration changes and exit System should reboot after saving the boot configura
31. e the Jumbo Frame feature Chapter 7 describes how to configure link aggregation Chapter 8 explains Virtual Local Area Networks VLANS in detail and provides configuration instructions and examples Appendix A lists the specifications for the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter Appendix B provides an overview of the SunVTS diagnostic application and instructions for updating the SunVTS software to recognize the adapter Using UNIX Commands This document might not contain information about basic UNIX commands and procedures such as shutting down the system booting the system and configuring devices Refer to the following for this information m Software documentation that you received with your system m Solaris Operating System documentation which is at http docs sum com xiv Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 Shell Prompts Shell Prompt C shell C shell superuser Bourne shell and Korn shell Bourne shell and Korn shell superuser machine name amp machine name Typographic Conventions Typeface Meaning Examples AaBbCc123 The names of commands files Edit your Login file and directories on screen Use 1s a to list all files computer output You have mail AaBbCc123 What you type when contrasted su with on screen computer output password AaBbCc123 Book titles new words or terms Read Chapte
32. eases Driver packages nxge version 2 0 0 and later are not compatible with the earlier nxge 1 x x releases To upgrade from 1 x x to 2 x x rpm package it is necessary to remove nxge 1 x x package before installing the 2 x x package The RHEL5 nxge 2 x x kmod package contains two separate packages One for the driver and another one for applications like nxge_config You must install both of the following packages kmod nxge rhel 2 0 1 x86 64 rpm driver binary package nxge apps rhel 2 0 1 x86 64 rpm application package To install the complete package enter the following rpm ivh nxge apps rhel 2 0 1 x86 64 rpm kmod nxge rhel 2 0 1 x86 64 rpm To ensure that the driver is loaded after the rpm installation enter the following modprobe nxge 5 Verify the new network interface instances corresponding to the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter ifconfig a grep eth eth0 Link encap Ethernet HWaddr 00 14 4F 20 F1 DC eth1 Link encap Ethernet HWaddr 00 14 4F 20 F1 DD eth2 Link encap Ethernet HWaddr 00 14 4F 20 F1 DE eth3 Link encap Ethernet HWaddr 00 14 4F 20 F1 DF eth4 Link encap Ethernet HWaddr 00 14 4F 6C 78 E8 eth5 Link encap Ethernet HWaddr 00 14 4F 6C 78 E9 The Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter instances eth4 and eth5 are shown in bold italics 6 Add the nxge interfaces to the etc modules conf
33. er Driver 2 To remove the driver packages run the pkgrm command pkgrm SUNWnxge plus any other packages from the previous command Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 Downloading and Installing the Driver on a Linux Platform 1 Login to your system 2 Download the driver RPM for your operating system http www sun com download products xml id 44eblefd For example nxge 1 0 1 x86_64 rpm 3 Discover the network interfaces before adding the package by using the ifconfig a command ifconfig a grep eth eth0 Link encap Ethernet HWaddr 00 14 4F 20 F1 DC eth1 Link encap Ethernet HWaddr 00 14 4F 20 F1 DD eth2 Link encap Ethernet HWaddr 00 14 4F 20 F1 DE eth3 Link encap Ethernet HWaddr 00 14 4F 20 F1 DF 4 Use the rpm tool to install the driver on SuSe and RedHat Linux rpm ivh tmp RHELAU4 large RPMS x86 64 nxge 1 0 1 x86 64 rpm Preparing HEHE HE HE HE HE HE HE HE HE E HE HE TEE EHE HERE EE EE HE HE FE HE EHE HERE HE HE HE E EHE HE H EHE ERE 1 nxge HEHE HE HE HE HE HE HE HE HE E HE HE HE FE HE HE HE HE FE HE EE FE HE FE HE HEHE HE HE HE HE HE HE HE HE HE H H HE ERE Chapter 2 Installing and Setting Up the Driver 9 Note In RHEL5 0 and later releases the driver is packaged in the kmod driver binary package format This packaging allows forward and backward driver binary compatibility within the same flavors of RHEL5 rel
34. er file in the diskless client s root directory Create an export root client name etc hostname nxgenumtbber file for the 10 Gigabit Ethernet interface See Configuring the Network Host Files on page 23 for instructions 5 Edit the hosts file in the diskless client s root directory Edit the export root client name etc hosts file to include the IP address of the 10 Gigabit Ethernet interface See Configuring the Network Host Files on page 23 for instructions 6 Set the MAC address on the server side and rebuild the device tree if you want to boot from the 10 Gigabit Ethernet port 7 To boot the diskless client from the 10 Gigabit Ethernet port type the following boot command ok boot path to device link param 26 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 Installing the Solaris Operating System Over a 10 Gigabit Ethernet Network The Solaris Advanced Installation Guide describes the full procedure for installing the Solaris Operating System over the network The following procedure assumes that you have created an install server which contains the image of the Solaris CD and that you have set up the client system to be installed over the network Before you can install the Solaris Operating System on a client system with a 10 Gigabit Ethernet adapter you must first add the 10 Gigabit Ethernet software packages to the install server These software packages are on Sun 10 Gigabit Ethernet Driver
35. ernet Fiber XFP Low Profile Adapter It contains the following sections Connectors on page 73 Performance Specifications on page 74 Physical Characteristics on page 75 Power Requirements on page 75 Connectors FIGURE A 1 shows the connector for the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter FIGURE A 1 Sun x8 Express Dual 10 Gigabit Ethernet XFP Low Profile Adapter Connectors 73 TABLE A 1 lists the characteristics of the fiber connectors used by the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter TABLE A 1 XFP Connector Characteristics Parameter Description Connector type LC Distance 10GBase SR Depending on fiber quality up to 300 meters for high bandwidth multimode fiber 10GBase LR Depending on fiber quality 10km maximum Fiber type 10GBase SR multimode 10GBase LR single mode Performance Specifications TABLE A 2 Performance Specifications Feature Specification Bus type x8 lane PCI Express 1 1 Bus width x8 lane PCI Express Bus speed x8 encoded rate Maximum Ethernet transfer rate Optics 20 Gbit sec uni directional 40 Gbit sec bidirectional theoretical 10 Gbps full duplex 10GBase SR 10GBase LR IEEE 802 ae 2002 compliant 74 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 Physical Characteristics TABLE A 3 Physical Characteristics Dimension Measurement Lengt
36. etup that remains the same after you reboot create an etc hostname nxgenumber file where number is the instance number of the nxge interface you plan to use To use the adapter s nxge interface in the Step 1 example create an etc hostname nxgeO file where 0 is the number of the nxge interface If the instance number were 1 the filename would be etc hostname nxgel m Do not create an etc hostname nxgenumber file for a Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter interface you plan to leave unused m The etc hostname nxgenumber file must contain the host name for the appropriate nxge interface m The host name must have an IP address listed in the etc hosts file 24 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 m The host name must be different from any other host name of any other interface for example etc hostname nxge0 and etc hostname nxge1 cannot share the same host name The following example shows the etc hostname nxgenumber file required for a system called zardoz that has a Sun x4 PCI Express Gigabit Ethernet driver zardoz 11 cat etc hostname nxge0 zardoz cat etc hostname nxgel zardoz 11 Create an appropriate entry in the etc hosts file for each active nxge interface For example cat etc hosts Internet host table 127 0 0 1 localhost 129 144 10 57 zardoz loghost 129 144 11 83 zardoz 11 Setting Up a 10 Gigab
37. file to automatically load the driver after system reboot alias eth4 nxge alias eth5 nxge 10 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 7 Use the ethtool command to check the parameter configurations that apply to the nxge driver For 10G ethtool i eth4 driver nxge version 2 0 1 firmware version 2XGF PXE1 47 FCode 3 9 07 04 24 bus info 0000 84 00 0 Equivalent for 1G ethtool i eth4 driver nxge version 2 0 1 firmware version QGC PXE1 47 FCode 3 9 07 04 24 bus info 0000 02 00 2 v To Remove the Driver From a Linux Platform To remove the driver packages from a Linux Platform use the rpm e command rpm e nxge 1 0 1 Chapter 2 Installing and Setting Up the Driver 11 12 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 CHAPTER 3 Installing the Adapter This chapter describes how to install the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter in your system and verify that it has been installed correctly This chapter contains the following section m Installing the Adapter on page 13 m Installing an Optical Transceiver on page 17 m Verifying the Hardware Installation on page 20 Note If you are installing the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter in a machine running either the Solaris x86 Operating System or the Linux x86 Operating System Dynamic Reconfiguration
38. following requirements of the EMC Directive 89 336 EEC As Information Technology Equipment ITE Class A per as applicable EN 55022 1994 A1 1995 A2 1997 Class A EN 61000 3 2 2000 Pass EN 61000 3 3 1995 A1 2000 Pass EN 55024 1998 A1 2001 A2 2003 Required Limits IEC 61000 4 2 4 kV Direct 8kV Air IEC 61000 4 3 3 V m IEC 61000 4 4 1 kV AC Power Lines 0 5 kV Signal and DC Power Lines IEC 61000 4 5 1 kV AC Line Line and Outdoor Signal Lines 2 kV AC Line Gnd 0 5 kV DC Power Lines IEC 61000 4 6 3V IEC 61000 4 8 1 A m IEC 61000 4 11 Pass Safety This equipment complies with the following requirements of the Low Voltage Directive 73 23 EEC EC Type Examination Certificates EN 60950 1 2001 1st Edition IEC 60950 1 2001 1st Edition CB Scheme Certificate No Evaluated to all CB Countries UL and cUL CSA 60950 1 2001 CSA C22 2 No 60950 00 File E138989 A82 Vol 54 FDA DHHS Accession Number Monitors Only Supplementary Information This equipment was tested and complies with all the requirements for the CE Mark This equipment complies with the Restriction of Hazardous Substances RoHS directive 2002 95 EC S S Dennis P Symanski DATE Donald Cameron DATE Worldwide Compliance Engineering Program Manager Customer Qualit Sun Microsystems Inc Sun Microsystems Scotland Limite 4150 Network Circle MPK15 102 Blackness Road Phase I Main Bldg Santa Clara CA 95054 U S A Springfield EH49 7LR Tel 650 786 3
39. frame 0 TX packets 8645339 errors 0 dropped 0 overruns 0 carrier 0 collisions 0 txqueuelen 1000 RX bytes 44983920 42 9 MiB TX bytes 75352303126 70 1 GiB 2 Use the ethtool utility with S option to get more detailed information This option displays a large amount of statistics manintined by the driver ethtool S eth9 grep rx pac rx packets 748274 rx packets 828 rx packets 112 rx packets 189088 rx packets 134 rx packets 196085 rx packets 177884 rx packets 93 rx packets 184050 58 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 3 To get more specific statitics use the grep command on the output of ethtool S ethtool S eth9 grep tx_pac tx_packets 8645339 tx_packets 3418334 tx_packets 26 tx_packets 0 tx_packets 22 tx_packets 401791 tx_packets 1466540 tx_packets 0 tx_packets 21 tx_packets 725 tx_packets 3357880 tx_packets 0 tx_packets 0 Chapter 6 Configuring the Jumbo Frames Feature 59 60 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 CHAPTER 7 Configuring Link Aggregation This chapter describes how to configure link aggregation It contains the following sections m Overview of Link Aggregation on page 61 m Configuring Link Aggregation in a Linux Environment on page 64 Overview of Link Aggregation Link Aggregation allows one or more network links to be aggregated together
40. g SunVTS to Recognize the Adapter 76 v To Update SunVTS to Recognize the Adapter 76 Using theSunVTS netlbtest 77 v ToUsethenetlbtest 77 Index 79 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 Tables TABLE 1 1 TABLE A 1 TABLE A 2 TABLE A 3 TABLE A 4 TABLE B 1 Hardware and Software Requirements for Sun SPARC and X64 Servers 2 XFP Connector Characteristics 72 Performance Specifications 72 Physical Characteristics 73 Low Profile Power Requirements 73 SunVTS Documentation 76 vii viii Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 FIGURE 1 1 FIGURE 8 1 FIGURE 8 2 FIGURE A 1 Figures Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter 2 Example of Servers Supporting Multiple VLANs With Tagging Adapters 68 Ethernet Tag Header Format 70 Sun x8 Express Dual 10 Gigabit Ethernet XFP Low Profile Adapter Connectors 73 x Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 Declaration of Conformity Compliance Model Number ATLS2XGF Product Family Name Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter USA FCC Class A This equipment complies with Part 15 of the FCC Rules Operation is subject to the following two conditions 1 This equipment may not cause harmful interference 2 This equipment must accept any interference that may cause undesired operation European Union This equipment complies with the
41. h 167 65 mm 6 6 inches Width 68 90 mm 2 713 inches Height Component side 14 48 mm 0 57 inches maximum Solder side 2 67 mm 0 105 inches maximum Power Requirements TABLE A 4 Low Profile Power Requirements Specification Measurement Power consumption 20 74W typical 24 33W peak Voltage 12V and 3 3V Appendix A Specifications 75 76 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 APPENDIX B Diagnostic Software This appendix provides an overview of the SunVTS diagnostic application and instructions for updating the SunVTS software to recognize the adapter This appendix contains the following sections m SunVTS Diagnostic Software on page 77 Updating SunVTS to Recognize the Adapter on page 78 Using the SunVTS netlbtest on page 79 SunVTS Diagnostic Software The SunVTS software executes multiple diagnostic hardware tests from a single user interface and is used to verify the configuration and functionality of most hardware controllers and devices The SunVTS software operates primarily from a graphical user interface enabling test parameters to be set quickly and easily while a diagnostic test operation is being performed Note SunVTS diagnostic software is not currently available for Solaris x86 Operating Systems The SunVTS nettest diagnostic can be used to test all of the networking interfaces on the system including the interfaces on the adapter To
42. hernet Network on page 27 m Booting Over the 10 Gigabit Ethernet Network for Solaris x86 and Linux Systems on page 31 Configuring the Network Host Files After installing the driver software you must create a hostname nxgenumber file for the adapter s Ethernet interface You must also create both an IP address and a host name for its Ethernet interface in the etc hosts file 23 1 At the command line use the grep command to search the etc path_to_inst file for nxge interfaces grep nxge etc path_to_inst pci 7c0 pci 0 pci 9 network 0 0 nxge pci 7c0 pci 0 pci 9 network 0 1 1 nxge In this example the device instance is from a Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter installed in slot 1 Be sure to write down your device path and instance which in the example is pci 7c0 pci 0 pci 9 network 0 0 Your device path and instance will be similar You need this information to make changes to the nxge conf file See Setting Parameters Using the nxge conf File on page 38 2 Use the ifconfig command to set up the adapter s nxge interface Use the ifconfig command to assign an IP address to the network interface Type the following at the command line replacing ip address with the adapter s IP address ifconfig nxge0 plumb ip address up Refer to the ifconfig 1M man page and the Solaris documentation for more information m If you want a s
43. ification for extending parallelism and virtualization to networking Up to 16 Receive DMA channels and up to 24 Transmit DMA channels multiple receive and transmit Descriptor Rings and dedicated networking hardware resources DMA interrupts buffer and more for each thread or strand CPU thread affinity and CPU load balancing at L1 L2 L3 and L4 Dynamic Reconfiguration DR Jumbo frame support up to 9KBytes IPv4 IPv6 and IPMP support TCP UDP IP checksum and CRC32C support IEEE 802 1Q VLAN support Chapter 1 Product Overview 3 4 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 CHAPTER 2 Installing and Setting Up the Driver This chapter explains how to download and install the nxge driver The nxge gigabit Ethernet driver nxge 7D is a multi threaded loadable clonable GLD based STREAMS driver The nxge driver is managed by the dladm 1M command line utility which allows VLANs to be defined on top of nxge instances and for nxge instances to be aggregated See the dladm 1M man page for more details on configuring the data link interfaces and link aggregations This chapter contains the following sections m Downloading and Installing the Driver on a Solaris SPARC or x86 Platform on page 5 m Downloading and Installing the Driver on a Linux Platform on page 9 Downloading and Installing the Driver on a Solaris SPARC or x86 Platform If your system uses the Solaris SPARC or x86 operating system
44. ing ndd using nxge conf 36 setting nxge driver parameters xiii 35 software requirements 2 SunVTS documentation 77 SunVTS diagnostic application xiv 77 SUNW nxge devices 35 V verify the installation 20 virtual device 71 VLAN ID 69 VLAN naming format 70 VLANS xiv 67 81 82 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009
45. it Ethernet Network on a Diskless Client System Before you can boot and operate a diskless client system across a 10 Gigabit Ethernet network you must first install the 10 Gigabit Ethernet software packages into the root directory of the diskless client You can find the 10 Gigabit Ethernet software packages at the following web site http www sun com download index jsp cat Networking amp tab 3 amp subcat Network 20Connectivity Refer to the Solaris Advanced Installation Guide and the System Administration Guide for more information about installing and administering diskless client systems Note The Solaris x86 version of the operating system do not support diskless clients Chapter 4 Network Configuration 25 v To Set Up a 10 Gigabit Ethernet Port on a Diskless Client 1 Locate the root directory of the diskless client on the host server The root directory of diskless client system is commonly installed in the host server s export root client name directory where client_name is the diskless client s host name In this procedure the root directory is export root client name 2 Download the software for Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter onto the server s drive 3 Use the pkgadd R command to install the software packages to the diskless client s root directory on the server Install the software packages to the client s root directory 4 Create a hostname nxgenumb
46. kt rate low 0 pkt rate high 0 rx usecs 20 rx frames 512 rx usecs irq 0 rx frames irq 512 tx usecs 0 tx frames 0 tx usecs irq 0 tx frames irq 0 rx usecs low 0 rx frame low 0 tx usecs low 0 tx frame low 0 rx usecs high 0 rx frame high 0 tx usecs high 0 tx frame high 0 To get status of L4 HW checksumming use the ethtool k command ethtool k eth4 Offload parameters for eth4 Cannot get device tcp segmentation offload settings Operation not supported rx checksumming on tx checksumming on scatter gather off tcp segmentation offload off Chapter 5 Configuring the nxge Device Driver Parameters 49 v To Set Parameters Using the Bundled configtool Utility 1 To get a list of tunable parameters use the nxge_config if_name get command usr local bin nxge_config eth4 get The tunable parameters exported by this device are class_opt_ipv4_tcp Read Write class_opt_ipv4_udp Read Write class_opt_ipv4_ah Read Write class_opt_ipv4_sctp Read Write class_opt_ipv6_tcp Read Write class_opt_ipv6_udp Read Write class opt ipv6 ah Read Write class opt ipv6 sctp Read Write These classification variables define how each IP class is configured This parameter also controls how the flow template is constructed and how packets are distrubuted within RDC groups Configuration bits 0x0010 use MAC Port for flow key 0x0020 use L2DA for flow key 0x0040 use VLAN f
47. l help eth4 ethtool version 1 8 Usage ethtool ethtool ethtool DEVNAMI a DEVNAME A DEVNAME autoneg on off E rx on off tx on off c DEVNAME C DEVNAME adaptive rx on off ethtool ethtool adaptive tx on off ethtool rx usecs N rx EX frames N rx usecs irq N frames irq N tx usecs N tx frames N tx usecs irq N N tx frames irq pkt rate low N rx usecs low N rx frames low N tx usecs low N tx frames low N pkt rate high N rx usecs high N rx frames high tx usecs high N g DEVNAME stats block usecs N N tx frames high N sample interval N Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 ethtool ethtool ethtool ethtool ethtool ethtool ethtool ethtool ethtool ethtool ethtool ethtool G DEVNAME rx N rx mini N rx jumbo N tx N i DEVNAME d DEVNAME e DEVNAME raw on off offset N length N E DEVNAME magic N N offset N value N k DEVNAME K DEVNAME rx on of tx on of sg on of tso on o r DEVNAME p DEVNAME d t DEVNAME online offline s DEVNAME speed 10 100 1000 duplex half full N port tp aui bnc mii fibre autoneg on off phyad d xcvr internal external wol plu m blalg s d 1 so
48. le Adapter and slide it into the opening 3 Applying even pressure at both corners of the transceiver push the transceiver until it is firmly seated in the slot 18 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 1 Repeat Step 1 through Step 4 to install the second optical transceiver Detach the wrist strap Caution If you pull the locking handle down when the optical transceiver is installed remove the optical transceiver entirely and reinstall it The handle operates an internal lock Pulling the handle down can disconnect the optical transceiver even though it might appear to be connected Chapter 3 Installing the Adapter 19 7 Connect the Ethernet cables Verifying the Hardware Installation After you have installed the Sun 10 Gigabit Ethernet adapter but before you boot your system perform the following tasks to verify the installation Refer to the your Solaris documentation for the detailed instructions Note Verification is not required if your system supports dynamic reconfiguration DR Verification is not supported if your system is running Solaris x86 software v To Verify the Hardware Installation 1 Power on the system and when the banner appears press the Stop A key sequence to interrupt the boot process and display the OpenBoot ok prompt 2 List the network devices on your system ok show nets pci 7c0 pci 0 pci 8 network 0 1 pci 7c0 pci 0 pci 8
49. m 44 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 v To Set Parameters Using the ethtool Utility 44 v To Set Parameters Using the Bundled configtool Utility 48 Tuning for Maximum Performance on a Linux Platform 49 Configuring the Jumbo Frames Feature 51 Jumbo Frames Overview 51 Checking Jumbo Frames Configurations 51 v To Show the Driver Statistics in a Solaris Environment 52 Enabling Jumbo Frames in a Solaris Environment 53 v To Enable Jumbo Frames in a Solaris Environment Using nxge conf 54 v To Check Layer 2 Configuration 54 v To Check Layer 3 Configuration 55 Enabling Jumbo Frames in a Linux Environment 55 v To Enable Jumbo Frames in a Linux Environment 55 v To Show the Driver Statistics in a Linux Environment 56 Configuring Link Aggregation 59 Overview of Link Aggregation 59 Configuring Link Aggregation in a Solaris Environment 59 v To Configure Link Aggregation in a Solaris Environment 60 Configuring Link Aggregation in a Linux Environment 62 v To Configure Bonding for Multiple nxge Interfaces 62 Configuring VLANs 65 Overview of VLANs 65 Configuring VLANs in a Solaris Environment 67 v To Configure Static VLANs 69 Configuring VLANs in a Linux Environment 70 v To Configure VLANs ina Linux Environment 70 Contents v vi Specifications 71 Connectors 71 Performance Specifications 72 Physical Characteristics 73 Power Requirements 73 Diagnostic Software 75 SunVTS Diagnostic Software 75 Updatin
50. mes Your Comments Sun is interested in improving its documentation and welcomes your comments and suggestions You can submit your comments by going to http www sun com hwdocs feedback Please include the title and part number of your document with your feedback Sun Dual 10GbE XFP PCI Express Card User s Guide part number 820 0184 13 Preface xvii xviii Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 CHAPTER 1 Product Overview This chapter provides an overview of the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter including m Components on page 1 m Product Description on page 1 m Hardware and Software Requirements on page 2 m Features on page 3 Components The carton in which your Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter was shipped should contain the following items m Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter with a low profile bracket attached m Standard height bracket m Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter Getting Started Guide Product Description The Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter is a low profile x8 lane PCI Express 10 gigabit Ethernet card utilizing the Sun ASIC with XFP based 10 gigabit Ethernet optics The Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter requires an optical transceiver FI
51. mtu 1500 aggregation key 33 62 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 2 Plumb up the interface aggrkey which is aggr33 is this case ifconfig aggr33 plumb ifconfig aggr33 aggr33 flags 1000842 BROADCAST RUNNING MULTICAST IPv4 gt mtu 1500 index 8 inet 0 0 0 0 netmask 0 ether 0 3 ba d8 9d e8 ifconfig aggr33 192 168 1 1 24 broadcast up ifconfig aggr33 aggr33 flags 1000843 lt UP BROADCAST RUNNING MULTICAST IPv4 gt mtu 1500 index 8 inet 192 168 1 1 netmask ffffff00 broadcast 192 168 1 255 ether 0 3 ba d8 9d e8 3 Show link aggregation status again and now the state should become attached dladm show aggr key 33 0x0021 policy L4 address 0 3 ba d8 9d e8 auto device address speed duplex link state nxge0 0 3 ba d8 9d e8 10000 Mbps full up attached nxgel 0 3 ba d8 9d e9 10000 Mbps full up attached 4 Use the dladm show aggr s command to display statistics dladm show aggr s key 33 ipackets rbytes opackets obytes ipkts opkts Total 0 0 16 1182 nxge0 0 0 16 1182 100 0 nxgel 0 0 0 0 0 0 Chapter 7 Configuring Link Aggregation 63 5 Use the dladm show aggr L command to display LACP specific information dladm show aggr L key 33 0x0021 policy L4 address 0 3 ba d8 9d e8 auto LACP mode off LACP timer short device activity timeout aggregatable sync coll dist defaulted expired nxge0 passive short yes no
52. nk e1000g0 type e1000g1 type e1000g2 type e1000g3 type nxgeO type nxgel type nxge38001 type non vlan non vlan non vlan non vlan non vlan non vlan vlan 38 mtu mtu mtu mtu mtu mtu mtu 1500 1500 1500 1500 9194 9194 9194 device device device device device device device e1000g0 e1000g1 e1000g2 e1000g3 nxgeO nxgel nxgel Enabling Jumbo Frames in a Linux Environment This section describes how to enable jumbo frames in a Linux environment v To Enable Jumbo Frames in a Linux Environment 1 Ensure that the nxge software is installed modprobe nxge 2 Plumb the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter interface ifconfig eth2 xxx xxx xx xxx up where xxx xxx xx xxx is the IP address of the interface Chapter 6 Configuring the Jumbo Frames Feature 57 3 Set the MTU for maximum performance ifconfig ethx mtu 9194 Where x is the instance number of the interface you want to configure jumbo frames on v To Show the Driver Statistics in a Linux Environment 1 Use the ifconfig utility to display driver statistics for example ifconfig eth4 eth9 Link encap Ethernet HWaddr 00 14 4F 6C 78 E9 inet addr 38 38 9 42 Bcast 38 38 9 255 Mask 255 255 255 0 inet6 addr fe80 214 4fff fe6c 78e9 64 Scope Link UP BROADCAST RUNNING MULTICAST MTU 9194 Metric 1 RX packets 748274 errors 0 dropped 0 overruns 0
53. or flow key 0x0080 use proto for flow key 0x0100 use IP src addr for flow key 0x0200 use IP dest addr for flow key 0x0400 use Src Port for flow key 0x0800 use Dest Port for flow key Note The classification variables are modified on an adapter basis that is if any of these variables is modifiied for one port the change carries over to all other ports of that adapter 2 To get a particular variable use the nxge config if name get param name usr local bin nxge config eth4 get class opt ipv4 udp class opt ipv4 udp Oxfe3 50 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 3 To set a particular variable use the usr local bin nxge config if name set param_name value usr local bin nxge config eth4 set class opt ipv4 tcp Oxfe0 Tuning for Maximum Performance on a Linux Platform The following tunings will improve the performance of the Sun x8 Express Dual 10 Gigabit Ethernet device driver on a system running the Linux operating system 1 Create the conf file for example sysct1_nxge conf that will be called by the sysctl utility IPV4 specific settings turns TCP timestamp support off default 1 reduces CPU use net ipv4 tcp_timestamps 0 turn SACK support off default on systems with a VERY fast bus gt memory interface this is the big gainer net ipv4 tcp_sack 0 sets min default max TCP read buffer default 4096 87380 174760 net ipv4 tcp_
54. ost files have been configured correctly during the Solaris installation Although the Solaris software installation creates the client s network configuration files you may need to edit these files to match your specific networking environment See Configuring the Network Host Files on page 23 for more information about editing these files 11 Use the dladm show dev command to show configuration information for all data links or the specified data link By default the system is configured to have one data link for each known network device dladm show dev e1000g0 link up speed 1000 Mbps duplex full e1000g1 link down speed 0 Mbps duplex half e1000g2 link down Speed 0 Mbps duplex half e1000g3 link down speed 0 Mbps duplex half nxgeO link up speed 10000 Mbps duplex full nxgel link up speed 10000 Mbps duplex full 30 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 Booting Over the 10 Gigabit Ethernet Network for Solaris x86 and Linux Systems To Boot Over the Network on Solaris x86 and Linux Systems Obtain the MAC address from the target Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter Set up the PXE boot server with the MAC addresses Choose one of the adapter ports as the boot interface Plug the Ethernet cable to the adapter port Power on the system Press the F2 key or the Control E keys to go to the BIOS Check and make sur
55. pass oe Sx 9x 9x 9x 9x 9x N msglvl d S DEVNAME Chapter 5 Configuring the nxge Device Driver Parameters 47 Following are some common parameters that can be changed ethtool c eth8 Coalesce parameters for eth8 Adaptive RX off TX off stats block usecs 0 sample interval 0 pkt rate low 0 pkt rate high 0 rx usecs 8 rx frames 512 rx usecs irq 0 rx frames irq 512 tx usecs 0 tx frames 0 tx usecs irq 0 tx frames irq 0 rx usecs low rx frame low tx usecs low Qo oo uo tx frame low rx usecs high 0 rx frame high 0 tx usecs high 0 tx frame high 0 rx usecs and rx frames control the RX interrupt rate per RX DMA channel RX interrupt will be generated after rx frames have been received or after rx usecs time interval if fewer than rx frames have been received within the interval For low latency applications it is recommened to set rx usecs to smaller value For bulk traffic it is recommended to use larger values of rx usecs and control the rate with rx frames rx frames irq controls the maximum number of rx packets processed with a single RX interrupt 48 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 2 To change RX interrupt Coalesce parameters use the ethtool C command D ethtool C eth4 rx usecs 20 ethtool c eth4 Coalesce parameters for eth4 Adaptive RX off TX off stats block usecs 0 sample interval 0 p
56. r 6 in the User s Guide words to be emphasized Replace command line variables with real names or values These are called class options You must be superuser to do this To delete a file type rm filename The settings on your browser might differ from these settings Preface XV Related Documentation The documents listed as online are available at http www sun com documentation Application Title Part Number Format Location Release Notes Sun x8 Express Dual 10 Gigabit Ethernet XFP Low 820 0185 PDF Online Profile Adapter Release Notes HTML Safety and Safety and Compliance Manual 816 7190 PDF Online compliance HTML Documentation Support and Training Sun Function URL Documentation http www sun com documentation Support http www sun com support Training http www sun com training Third Party Web Sites Sun is not responsible for the availability of third party web sites mentioned in this document Sun does not endorse and is not responsible or liable for any content advertising products or other materials that are available on or through such sites or resources Sun will not be responsible or liable for any actual or alleged damage or loss caused by or in connection with the use of or reliance on any such content goods or services that are available on or through such sites or resources xvi Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 Sun Welco
57. ring VLANs in a Solaris Environment on page 69 m Configuring VLANs in a Linux Environment on page 71 With multiple VLANs on an adapter a server with a single adapter can have a logical presence on multiple IP subnets By default 128 VLANS can be defined for each VLAN aware adapter on your server However this number can be increased by changing the system parameters If your network does not require multiple VLANs you can use the default configuration in which case no further configuration is necessary Overview of VLANs VLANs enable you to split your physical LAN into logical subparts providing an essential tool for increasing the efficiency and flexibility of your network VLANs are commonly used to separate groups of network users into manageable broadcast domains to create logical segmentation of workgroups and to enforce security policies among each logical segment Each defined VLAN behaves as its own separate network with its traffic and broadcasts isolated from the others increasing the bandwidth efficiency within each logical group Although VLANs are commonly used to create individual broadcast domains and or separate IP subnets it can be useful for a server to have a presence on more than one VLAN simultaneously Several Sun products support multiple VLANs on a per port or per interface basis allowing very flexible network configurations FIGURE 8 1 shows an example network that uses VLANs 67 Main Server
58. rmem 10000000 10000000 10000000 sets min pressure max TCP write buffer default 4096 16384 131072 net ipv4 tcp_wmem 10000000 10000000 10000000 sets min pressure max TCP buffer space default 31744 32256 32768 net ipv4 tcp mem 10000000 10000000 10000000 CORE settings mostly for socket and UDP effect maximum receive socket buffer size default 131071 net core rmem max 524287 maximum send socket buffer size default 131071 net core wmem max 524287 default receive socket buffer size default 65535 net core rmem default 524287 default send socket buffer size default 65535 net core wmem default 524287 maximum amount of option memory buffers default 10240 net core optmem max 524287 number of unprocessed input packets before kernel starts dropping them default 300 net core netdev max backlog 300000 Chapter 5 Configuring the nxge Device Driver Parameters 51 2 Set up the sysctl utility sysctl p etc sysctl nxge conf 52 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 CHAPTER 6 Configuring the Jumbo Frames Feature This chapter describes how to configure the Jumbo Frames feature It contains the following sections m Jumbo Frames Overview on page 53 m Checking Jumbo Frames Configurations on page 53 m Enabling Jumbo Frames in a Solaris Environment on page 55 m Enabling Jumbo Frames in a Linux Environment on page 57 a Jumbo Frames
59. s64 rbytes rbytes64 instance 38001 class net 0 0 0 3842 493000352 0 10000000000 2116069805 6411037101 0 0 0 0 2757388874 23380264381002 0 37606022 4332573318 2937141290 47178857920554 Enabling Jumbo Frames in a Solaris Environment This section describes how to enable jumbo frames in both a SPARC and an x86 environment Chapter 6 Configuring the Jumbo Frames Feature 55 v To Enable Jumbo Frames in a Solaris Environment Using nxge conf 1 Enable Jumbo Frames for a port using the nxge conf file For example name pciex108e abcd parent pci 780 pci 0 pci 8 network 0 unit address 0 accept jumbo 1 2 Reboot the system boot r 3 Set the maximum MTU for maximum performance ifconfig nxge mtu 9194 v To Check Layer 2 Configuration View the maximum transmission unit MTU configuration of an nxge instance at any time with the kstat command kstat nxge 0 grep mac mtu The kstat mac mtu variable represents the complete size of the Ethernet frame which includes the Ethernet header maximum payload and crc This value should be equal to or less than the MTU configured on the switch 56 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 v To Check Layer 3 Configuration Check the Layer 3 configuration by using the dladm command with the show link option as shown in the following example dladm show li
60. sceiver 15 v To Install an Optical Transceiver 16 Verifying the Hardware Installation 18 v To Verify the Hardware Installation 18 Rebooting the System 20 Network Configuration 21 Configuring the Network Host Files 21 Setting Up a 10 Gigabit Ethernet Network on a Diskless Client System 23 v To Set Up a 10 Gigabit Ethernet Port on a Diskless Client 24 Installing the Solaris Operating System Over a 10 Gigabit Ethernet Network 25 v To Install the Solaris Operating System Over a 10 Gigabit Ethernet Network 25 Booting Over the 10 Gigabit Ethernet Network for Solaris x86 and Linux Systems 29 v To Boot Over the Network on Solaris x86 and Linux Systems 29 Configuring the nxge Device Driver Parameters 33 nxge Hardware and Software Overview 33 Setting nxge Driver Parameters on a Solaris Platform 34 Setting Parameters Using the ndd Utility 34 Noninteractive and Interactive Modes 34 v To Specify Device Instances for the ndd Utility 35 v To Specify Parameter Values Using the ndd Utility 35 v To Use the ndd Utility in Interactive Mode 35 Setting Parameters Using the nxge conf File 36 v To Set Driver Parameters Using an nxge conf File 37 Tuning for Maximum Performance ona Solaris Platform 41 v To Improve Performance on an UltraSPARC CPU Based Sun Platform 41 v To Improve Performance on an AMD CPU Based Sun Platform 42 v To Obtain Higher Throughput Using the Generic Tunables for the Solaris TCP IP Stack 43 Setting Parameters on a Linux Platfor
61. ss Interrupts after this number of NIU hardware ticks have elapsed since the last packet was received A value of zero means no time blanking Default 8 intr pkts Interrupt after this number of packets have arrived since the last packet was serviced A value of zero indicates no packet blanking Default 20 t Interrupt Blanking parameters intr time 8 intr pkts 20 Classification and Load Distribution Configuration Opt These variables define how each IP class is configured Configuration options range from whether TCAM lookup ie is enabled to flow hash generation This parameters also control how the flow template is constructed and how packet is distributed within RDC groups supported classes class opt ipv4 tcp class opt ipv4 udp class opt ipv4 sctp class opt ipv4 ah class opt ipv6 tcp class opt ipv6 udp class opt ipv6 sctp class opt ipv6 ah Configuration bits The following bits will be decoded by the driver as hex format 0010 use MAC Port for flow key 0020 use L2DA for flow key 0040 use VLAN for flow key 0080 use proto for flow key 0100 use IP src addr for flow key 0200 use IP dest addr for flow key 0400 use Src Port for flow key 0800 use Dest Port for flow key opt ipv4 tcp fe0 Chapter 5 Configuring the nxge Device Driver Parameters 41 b The following parameters operate on a per port basis and can be set using the platform sun4u kernel drv
62. ter s PCI connector If the adapter does not seat properly when you apply even pressure remove the adapter and carefully reinstall it 10 Detach the wrist strap and close the system unit 16 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 11 Install the optical transceivers Note Illustrations for installing the adapter and installing the optical transceiver show different systems This difference in systems provides the maximum clarity for installing the optical tranceiver and shows the most complicated adapter installation Installing an Optical Transceiver The Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter requires an optical transceiver in each port to create and Ethernet connection The short range optical transceiver Part Number 135 1172 01 and the long range optical transceiver Part Number 375 3495 01 are both available from Sun Microsystems Chapter 3 Installing the Adapter 17 v To Install an Optical Transceiver 1 Pull the white locking handle into the full horizontal position You will feel the handle click into position when it is fully opened Note If you are installing an optical transceiver in a SunFire T1000 ensure that the label and the edge connector is visible to you 2 Holding the optical transceiver by the edges align the transceiver with the slot in the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profi
63. these parameters m nxge Hardware and Software Overview on page 35 m Setting nxge Driver Parameters on a Solaris Platform on page 36 m Setting Parameters Using the ndd Utility on page 36 m Setting Parameters Using the nxge conf File on page 38 m Setting Parameters on a Linux Platform on page 46 nxge Hardware and Software Overview The Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter provides two 10 Gigabit Full Duplex networking interfaces The device driver automatically sets the link speed to 10000 Mbit sec and conforms to the IEEE 802 3 Ethernet standard Each interface has 8 Receive DMA Channels and 12 Transmit DMA Channels to allow for parallel processing of the packets The Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter extends CPU and OS parallelism to networking with its support for hardware based flow classification and multiple DMAs Using CPU thread affinity to bind a given flow to a specific CPU thread it enables a one to one correlation of Rx and Tx packets across the same TCP connection This can help avoid cross calls and context switching to deliver greater performance while reducing the need for CPU resources to support I O 35 processing The Sun 10 Gigabit Ethernet Adapter utilizes Sun s own innovative MAC Controller to map the 10 Gigabit XAUI interface onto the PCI Express form factor It supports 10 Gb sec bandwidth using eight transmit and eight recei
64. tion 32 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 10 Press the F12 key to install the OS from the network If the cable is connected to the right port you should see the MAC address that you assigned to your PXE server displayed by BIOS image pxe mac addr PXE MOF Exiting Intel Boot Agent NVIDIA Boot Agent 217 0513 PXE MOF Exiting Intel Boot Agent NVIDIA Boot Agent 217 0513 PXE MOF Exiting Intel Boot Agent Intel R Boot Agent GE v1 2 43 Beta 1 00144F26E0B7 PXE E61 Media test failure check cable Copyright C 2001 2005 NVIDIA Corporation Copyright C 1997 2000 NVIDIA Corporation PXE E61 Media test failure check cable Copyright C 2001 2005 NVIDIA Corporation Copyright C 1997 2000 NVIDIA Corporation PXE E61 Media test failure check cable Copyright C 1997 2006 Intel Corporation CLIENT MAC ADDR 00 15 17 13 90 00 GUID 00000000 0000 0000 0000 11 You can now install the nxge driver and configure the adapter Chapter 4 Network Configuration 33 34 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 CHAPTER 5 Configuring the nxge Device Driver Parameters The nxge device driver controls the Sun x8 Express Dual 10 Gigabit Ethernet interfaces You can manually set the nxge driver parameters to customize each device in your system This chapter lists the available device driver parameters and describes how you can set
65. to the adapter when it is in place This section contains the following topics m To Install the Adapter on page 14 m Installing an Optical Transceiver on page 17 m Verifying the Hardware Installation on page 20 v To Install the Adapter 1 Halt and power off your system 2 Power off all of the peripherals connected to your system 3 Open the system unit 4 Attach the adhesive copper strip of the antistatic wrist strap to the metal casing of the power supply Wrap the other end twice around your wrist with the adhesive side against your skin 5 Remove the filler panel from the PCI E opening 14 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 Note These illustrations show the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter being installed into a Sun Fire T2000 which requires a standard height bracket for the low profile adapter Open the retaining clip at the edge of the PCI E opening Holding the adapter by the edges align the adapter edge connector with the PCI E slot Slide the adapter face plate into the small slot at the end of the PCI E opening Applying even pressure at both corners of the adapter push the PCI Express adapter until it is firmly seated in the slot Chapter 3 Installing the Adapter 15 9 Secure the retaining clip Caution Do not use excessive force when installing the adapter into the PCI E slot You might damage the adap
66. use the nettest diagnostic you must have the SunVTS software installed on your system Refer to your Solaris documentation for installation instructions Refer to the SunVTS documentation listed in TABLE B 1 for instructions on how to run and monitor the nettest diagnostic These SunVTS documents are available online at the following URL 77 http www sun com products n solutions hardware docs Software system administration tools testing valid sunvts index html Select the document for the Solaris release on your system TABLEB 1 SunVTS Documentation Title Description SunVTS User s Guide Describes the SunVTS diagnostic environment SunVTS Test Reference Manual Describes each SunVTS test including the nettest and describes the various test options and command line arguments SunVTS Quick Reference Provides an overview of the user interface Updating SunVTS to Recognize the Adapter Use SunVTS 6 3 or later You will need to update the SunVTS configuration to recognize the adapter v To Update SunVTS to Recognize the Adapter 1 Plug in a loopback cable 2 Ensure that the SunVTS software and the nxge driver are installed on your system 3 Add the following lines to the opt SUNWvts lib conf netlbtest conf and opt SUNWvts lib conf nettest conf files nxge nxge 10gbaset 78 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 Using the Sun VTS netlbtest You must have the Ethernet card and
67. ve lanes Setting nxge Driver Parameters on a Solaris Platform You can set the nxge device driver parameters in two ways m Using the ndd utility m Using the nxge conf file If you use the ndd utility the parameters are valid only until you reboot the system This method is good for testing parameter settings To set parameters so they remain in effect after you reboot the system create a platform sun4u kernel drv nxge conf file and add parameter values to this file when you need to set a particular parameter for a device in the system Setting Parameters Using the ndd Utility Use the ndd utility to configure parameters that are valid until you reboot the system The following sections describe how you can use the nxge driver and the ndd utility to modify with the set option or display without the set option the parameters for each nxge device Noninteractive and Interactive Modes You can use the ndd utility in two modes m Noninteractive m Interactive In noninteractive mode you invoke the utility to execute a specific command Once the command is executed you exit the utility In interactive mode you can use the utility to get or set more than one parameter value Refer to the ndd 1M man page for more information 36 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 To Specify Device Instances for the ndd Utility Before you use the ndd utility to get or set a parameter for a nxge device
68. you will need to download and install the nxge device driver for Solaris platforms v To Download the Driver on a Solaris Platform 1 Locate and download the nxge device driver software at the following web site http www sun com products networking ethernet index html 2 Uncompress the gzipped tar file gunzip nxge tar gz 3 Unpack the tar file tar xvf nxge tar a For SPARC systems change to the following directory cd 10 GigabitEthernet Solaris 10 sparc Packages b For x86 systems cd 10 GigabitEthernet Solaris 10 i1386 Packages 4 For SPARC systems determine which architecture your system is running uname m a For sun4v systems install the software packages by typing the following at the command line usr sbin pkgadd d SUNWnxge v SUNWnxgem b For sun4u systems install the software packages by typing the following at the command line usr sbin pkgadd d SUNWnxge u SUNWnxgem 6 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 5 For x86 systems install the software packages by typing the following at the command line usr sbin pkgadd d A menu similar to the following displays The following packages are available 1 SUNWnxge Sun PCI E 10G 1G Ethernet Adapter Driver 1386 1 0 REV 2006 12 05 10 0 Select package s you wish to process or all to process all packages default all

Sun Microsystems 10GBE XFP Network Card User Manual

Contents

Download Pdf Manuals

Related Search

Related Contents