Home

"user manual"

1. Portgrid Ports Summary Switches Topology Device Ports 413104 4 15 53 PM IBM_2109_F32 10 64 210 150 IBM_3534_1RU 10 64 210 152 ics L BRD_12K_sw0_116 BRD_12K_sw0_116 10 64 210 100 brocadessm Principal switch IBM_2109_F16 10 64 210 105 H IBM_2109_F32 herbosas S Member switches gt SwitchGroups Switch status gt Marginal 3 60 Switch status Switch types gt gt 4 FabricOS versions gt 2 Port information 79 free 95 of 120 Device information 18 Active zones gt 3 IBM_SGU_CFG Launch switch brocadessm 10 64 210 25 Principal switch brocadessm 10 64 210 25 Member switch brocadessm 10 64 210 25 Switch status Healthy Switch type 2Ghit 16 port embedded fabric switch FabricOS version v4 24 Port information gt 87 free 14 of 16 Device information 2 Active zone Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 18 2 Figure 2 13 Fabric Manager Topology View 2 distinct fabrics shown The Brocade SAN Switch Module is the top right switch in the fabric on the left Fabric Manager By x File Edit View Actions Topology Tools Help Address fitp 10 64 210 257 yl ala a ete ome DPeveivals
2. cccecceeceeeeeeeeeeeeeeeeeeseeaeeeees 122 Brocade SAN Switch Module Management Tools 125 INI a ssceirs cet cin dasssagaecadstadcdecercs E A 126 RELIANT E Ke a E ARSS EEEE EE SEEE 135 Advanced Performance Monitoring ccceeceeeeeeeeeeeeeeeees 148 APPONI A runea teate E A AOS 159 Naming Conventions for the IBM TotalStorage SAN Switch family OPTIE E EE N EEEE T E E TS 159 Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 Vv About This Document This document is a user guide written for SAN administrators to help you learn about many of the Brocade licensed products and how they relate to the Brocade Enterprise SAN Switch Module for IBM eServer BladeCenter and the Brocade Entry SAN Switch Module for IBM eServer BladeCenter This guide covers both models About This Document contains the following sections e How This Document Is Organized e What s New in This Guide e Document Conventions e Additional Information e Getting Technical Help e Document Feedback How This Document Is Organized This document is organized into two sections to help you find the particular information that you want as quickly and easily as possible The first section Section I describes each of the specific Brocade SAN Switch Module licensed feature products This section provides only conce
3. Figure 12 1 Core Edge topologies showing a starter configuration with 16 port core switches and larger much more scalable configuration with 128 port core switches Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 62 12 A reasonable network progression might start with 16 port or 32 port core switches and migrate to 64 port or 128 port cores when the scalability limits of the smaller cores are reached See the Brocade SAN Migration Guide for detailed information on how such a migration would be performed As shown in Figure 12 2 when additional ports are required the lower port count switches in the core can be replaced with the higher density 64 port or 128 port switches The lower port count switches can then be redeployed at the edge Figure 12 2 Upgrading the core to higher port count switches and redeploying the former core switches to the edge Install larger core fabric switches Redeploy existing core switches to the edge In these diagrams the embedded Brocade SAN Switch Modules can be any of the Edge switches The Brocade SAN Switch Module should be included in larger network Note fabrics as Edge switches and not as Core switches simply because there are not enough empty ports for ISLs to connect other switches to The ultimate limitation in a fabric design today and as defined in the Fibre Channel standa
4. Management e Update Cancel Ping Switch Module Start TelnetWeb Session The IBM eServer BladeCenter Management Module should have the latest Note firmware version This firmware file is contained in the Management Module FW update package and can be found on the IBM Support website For more information refer to the Brocade Enterprise SAN Switch Module for IBM eServer BladeCenter and Brocade Entry SAN Switch Module for IBM eServer BladeCenter Installation Guide Fabric OS Command Line Interface The Fabric OS command line interface CLI accessed through telnet there is no direct Serial Console port on the Brocade SAN Switch Module provides the user with a full range of management capabilities The Fabric OS CLI enables an administrator to monitor and manage individual switches and ports from a standard workstation Many commands can monitor and manage certain aspects of a fabric as well Access is controlled by a switch level password for each user level root factory USERID user The commands available through the CLI are based on the user s login level and the license keys used to unlock certain features Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 11 2 All configuration and management tasks are available using the USERID or user level the root and factory levels should be used with caution The Brocade Fabric
5. Additional benefits of the Core Edge topology e Well tested and reliable e Widely deployed in production environments e Simple and easy to understand e Able to solve most design problems fits well with many SAN solutions and is an effective choice when design requirements are not well known e Easy to grow without downtime or disconnection of links and devices e Pay as you grow e Flexible e Capable of exhibiting stellar performance with full utilization of FSPF load sharing and resiliency features e Conducive to performance analysis Because the Core Edge topology is symmetrical it is a straightforward process to identify performance issues Every device has an equivalent path to any other device and the same available bandwidth between any two devices To identify a SAN performance issue it is only necessary to monitor the core switches With other topologies this is not the case e The potential to scale to thousands of ports using high port count switches The guidelines established in this chapter can also apply to other topologies such as a full mesh or ring It is up to the reader to interpret the guidelines in this section for topologies other than Core Edge Regardless of topology chosen a redundant fabric i e dual fabric SAN is recommended Device Attachment Strategies The Brocade SilkWorm Design Deployment and Management Guide SAN DDM Version 3 0 covers device attachment strategies in great depth The IBM eServ
6. The implementation of zoning is recommended for any SAN and especially Guideline critical for any large fabric since zoning is fundamental to the functioning of multi hundred port fabrics With Fabric OS v3 1 v4 1 and later zoning changes cause different RSCN Registered State Change Notification behavior In Fabric OS v3 1 v4 1 and later when zone changes are enabled or disabled fabric RSCNs are only sent to devices that completed an SCR State Change Registration and that are in the affected zones In all Fabric OS v2 x releases the locally connected devices that completed an SCR will receive these RSCNs regardless if the device is affected by a zone change With a mixed fabric the devices in the zones that are affected as well as all devices local to the Fabric OS v2 x switches receive an RSCN The RSCN filtering of a device is handled by the Name Server of the switch to which it is attached The Fabric OS of the switch that originates a zoning change is irrelevant Make certain devices that require RCSN suppression are directly attached to a Guideline switch running Fabric OS version 3 1 or 4 1 or higher Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 76 16 Zoning Database Size Zoning consumes a finite amount of processing and memory resources As the number of devices in a SAN grows so do the demands on these same resou
7. You must perform zone management operations from the primary FCS switch using a zone management interface such as telnet or Advanced Web Tools You can alter a zoning database provided you are connected to the primary FCS switch When two secure fabrics join the traditional zoning merge does not occur refer to the Secure Fabric OS User s Guide for information for more information Advanced Security Secure Fabric OS requires the activation of a Brocade Not aes we ai Security license and an Advanced Zoning license Zoning Architecture Zoning commands are executed under the transaction model A working copy of the defined configuration is created at the start of a transaction Each zoning command is executed from the working copy When a transaction is opened all new zoning information is placed in a transactional buffer The new changes are not applied to the fabric until the transaction is closed A transaction is aborted when another switch closes its transaction or by issuing the command cfgTransAbort When a transaction is closed all new and existing zoning information is applied to the fabric and saved to flash memory Zone configuration is managed on a fabric basis Zoning can be implemented and administered from any switch in the fabric that has a Zoning license enabled When a change in the configuration is saved enabled or disabled per the transactional model it is automatically by closing the transaction distributed to a
8. 1 and Option 2 So a message gets sent to the Error Event Log and to an SNMP Trap Figure 26 3 Example of the fwConfigure command with Threshold Alarm Levels at 3 brocadessm USERID gt fwconfigure Environment class SFP class Port class Fabric class E Port class F FL Port Copper class F FL Port Optical class Alpa Performance Monitor class EE Performance Monitor class 10 Filter Performance Monitor class 11 Security class 12 Switch Availability Monitor class 13 Quit Select a class gt 1 13 13 1 1 Temperature 2 Fan 3 Power Supply AYA PWNHE Oo 4 return to previous page Select an area gt 1 Index ThresholdName Status LastEvent LasteventTime LastState envTemp001 enabled inBetween i 738 2004 envTemp002 enabled inBetween i 2 49 38 2004 envTemp003 enabled inBetween i 2 49 38 2004 envTemp004 enabled inBetween i 238 2004 Normal Normal Normal OOOO OA ea Normal refresh disable a threshold enable a threshold advanced configuration Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 137 5 return to previous page Select choice gt 1 5 5 4 Index ThresholdName BehaviorType BehaviorInt 1 envTemp001 Triggered 1 2 envTemp002 Triggered I 3 envTemp003 Triggered 1 4 envTemp004 Triggered 1 Threshold boundary l
9. 20 00 00 00 c9 30 71 93 B OpenSystems_Zone_2 8 Aliases E E IBM_MIDDLE_Host_2 H E JBOD_2_Sdisks B E Opensystems_Zone_3 B E Aliases H E IBM_BOTTOM_Host_3 B E JBOD_3_Sdisks F SGU_ZONE1 Switch Commit Messages Zone Admin opened at Tue Apr 13 2004 04 54 52 PM Loading information from Fabric Done 2 4 E3 Applet com brocade web zoning ZoneAdminApplet_SS started D Internet For more detailed information on Advanced Zoning refer to the following publications and material e Brocade Fabric OS Features Guide Chapter 2 e Brocade Advanced Web Tools Administrator s Guide Chapter 6 entitled Zone Administration e Brocade Fabric OS Procedures Guide Chapter 10 entitled Zoning Procedures e Brocade Fabric Manager User s Guide Chapter 6 entitled Zoning e Fabric OS Reference Manual CLI commands used for managing zoning e Brocade SilkWorm Design Deployment and Management Guide SAN DDM Version3 0 Section 3 3 entitled Zoning Design Considerations amp Guidelines under SAN Design and Section 5 8 entitled Zoning Plan under SAN Deployment e Zoning Implementation Strategies For Brocade San Fabrics Whitepaper Note that in terms of Zoning behavior the Brocade SAN Switch Module behaves exactly like the SilkWorm 3850 3250 as may be described in the above publications Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management
10. Community ro public Trap Recipient s IP address in dot notation 004 192 168 2 10 Community ro common Trap Recipient s IP address in dot notation Community ro FibreChannel Trap Recipient s IP address in dot notation SNMP access list configuration Access host subnet area in dot notation Read Write true t false f true Access host subnet area in dot notation Read Write true t false f true Access host subnet area in dot notation Read Write true t false f true Access host subnet area in dot notation Read Write true t false f true Access host subnet area in dot notation Read Write true t false f true Access host subnet area in dot notation Read Write true t false f true sysDescr Brocade SAN Switch Module for IBM eServer BladeCenter sysLocation End User Premise sysContact Field Support swEventTrapLevel 5 authTraps 1 ON Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 128 25 continued SNMPvl community and trap recipient configuration Community 1 Secret COde rw No trap recipient configured yet Community 2 OrigEquipMfr rw No trap recipient configured yet Community 3 private rw No trap recipient configured yet Community 4 public ro Trap recipient 192 168 2 10 Community 5 common ro N
11. Disable Telnet Daemon on switches running Fabric OS 4 1or later Set the switch Recovery Password and Boot Password Fabric OS 4 1or later Rename the admin level ID USERID to the Brocade specific default of admin and the user level ID to the Brocade specific default of user before enabling security Otherwise the switch will not be allowed in the secure fabric This is covered in Chapter 8 of this manual Enable SFOS and verify its operation Backup Primary FCS switch configurations with configupload Each switch in the fabric needs to be rebooted to activate Advanced Security Secure Fabric OS Be aware that enabling Advanced Security Secure Warning Fabric OS will reboot BOTH CPs simultaneously on the SilkWorm 12000 and 24000 It is recommended to schedule downtime on single fabric SANs before enabling Advanced Security Secure Fabric OS The Primary FCS databases are not automatically backed up when secure mode is disabled Data will be lost if the FCS switch is not backed up before Warning the command secmodedisable is done on the Primary FCS switch To backup the Advanced Security Secure Fabric OS data use the command configupload on the Primary FCS Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 85 19 Consider a locked closet to physically secure the Primary and Backup FCS deli cuide lime s
12. Dual Connected Redundant Fabric SANs Resilient fabrics and the fault tolerant components that comprise them are very reliable However no single fabric can ever truly be a High Availability HA solution The fabric itself is still potentially subject to failures caused by things like operator error disaster or hardware software malfunctions To account for those categories of error another level of availability must be used The redundant fabric SAN This is sometimes known as a multi fabric or dual fabric SAN Redundancy in SAN design is the duplication of components up to and including the entire fabric to prevent the failure of the SAN solution Using a fully redundant fabric makes it possible to have an entire fabric fail as a unit or be taken offline for maintenance without causing downtime for the attached nodes When describing availability characteristics what we are concerned with is path availability If a particular link fails but the path to the data is still there no downtime is experienced by the users of the SAN It is possible that a performance impact may occur but this is a very small event compared to one or many crashed servers Two or more fabrics must be used in conjunction with multiple HBAs multiple RAID controllers and path switch over software to be effective for those SAN devices that require the highest availability Figure 11 3 illustrates the ability of redundant fabrics to withstand large scale failures Note tha
13. EE eeeeeeeee eee OK Cancel Java Applet Window Step 8 Repeat 7 by finding the Destination ID the Disk Drive or by placing the value of 0307ba into the DID box Figure 27 11 Figure 27 11 Place DID 0307ba into the DID box SID DID Performance Setup x Port or Sid Did Selection List Enter drag port number i Domain_1 brocadessm Domain_2 IBM_2109_F32 Domain_3 IBM_3534_1RU 0 Enter drag SID number Hex 1 La 2 Add Sid gt 010100 p 3 a Enter drag DID number Hex 5 6 o307ba 47 ee me O30 ba 1d 0307 be lt Search Domain_4 IBM_3534_F08 OK Cancel Java Applet Window Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 153 27 Step 9 Click the OK button and the new graph will get added to the existing canvas Figure 27 12 Figure 27 12 New Advanced Performance Monitoring graph is added alo xi Actions Performance Graphs Switch Throughput Utilization Ref rate 30 Secs SID O10 Performance Sid 010100 Did 0307ba Port 1 Ref rate 30 Transmitted Bytes sec Received Bytes sec el lava Applet Window Step 10 Repeat Steps 4 8 to add a second Advanced Performance Monitoring Filter but this time add it to Port 2 and use the SID of 010200 and DID of 0307bc Figure 27 13 Brocade Enterprise and Entry SAN Switch
14. Enabled gt Cancel Save a E Done Internet Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 10 2 Advanced Management functions of the IBM eServer BladeCenter Management Module for the Brocade SAN Switch Module continued See Figure 2 6 Figure 2 6 Advanced Management features for the Brocade SAN Switch Module E 10 64 210 20 BladeCenter Management Module Microsoft Internet Explorer E loj x File Edit View Favorites Tools Help Back gt G A A Aseach GaFavortes Efmeda 4 S Sp A l Address http 10 64 210 20 private main ssi Go Links BladeCenter Management Module Restore Factory Defaults This action will cause all module settings to be set to their factory defaults You will lose all the changes you made to the configuration of this module as a result In order to preserve the new IP configuration set the field labeled Preserve new IP configuration on all resets to enabled Clearing of the configuration will be followed by a restart of the module Click the Restore Defaults button if you want to proceed Cancel Restore Defaults Send Ping Requests You can test the internal path between the management module and the switch module by sending it ping requests In order to ping this switch module at its current IP adrress of 10 564 210 25 click the Ping Switch Module button
15. Front Panel policy Use to restrict which switches can be accessed by front panel e Options policy Use to restrict the types of WWNs that can be used for zoning e Device Connection Control DCC policies Use to restrict which fibre channel device ports can connect to which fibre channel switch ports e Switch Connection Control SCC policy Use to restrict which switches can join the fabric The Brocade SAN Switch Module switch has a different default username than admin which exists on all other SilkWorm switch products Due to this change a new command userrename must be used to rename the default USERID user account to admin before connecting the Brocade SAN Switch Module to a secure fabric made up of other Brocade SilkWorm switches When using Advanced Security Secure Fabric OS rename the admin level ID to the Brocade specific default of admin and the user level ID to the Brocade specific default of user before enabling security Otherwise the switch will not be allowed in the secure fabric The following error message will display Error from domain lt domain ID gt Switch does not have all default account names To rename the admin level ID from USERID to admin switch admin gt userRename USERID admin Note Managing Advanced Security Secure Fabric OS You can manage Advanced Security Secure Fabric OS using the following methods e Telnet command line interface Secur
16. While not covered it is important to validate the Secure FOS environment as well If at all possible it is a good idea to do these tests with generated I O preferably with the application up and running These tests are meant to be used as guidelines and a proof point that the SAN is operating properly before it is put into production This section will focus on validating a Core Edge SAN If Core Edge is not used all tests in this section can be tailored for other fabric topologies Separate tests will be required for the particular application in use in the SAN This chapter contains the following sections e Sample Script to Generate I O e Sample Validation Recommendations o Fabric Stability Validation o High Availability ISL Failure Simulation o High Availability Switch Failure Simulation Sample Script to Generate I O With no application it is possible to generate I O If using UNIX hosts the following sample script can be used For Windows hosts use an I O tool such as IOmeter This script creates and writes to a file and then does continuous reads of it The path and size in blocks need to be specified As an example sbtest f file01 1000 will create a file of size 1000 blocks in file01 and once created it will do successive reads until terminated More I O can be generated on a single host using multiple instances of this script running in the background Portperfshow is a helpful CLI command that can be used to quickly check exp
17. available SAN to ensure this continuous access is critical Resilient and redundant fabric designs are highly recommended especially in large storage consolidation solutions In a storage consolidation solution many devices contend for a shared storage port The performance limiting factor is often the over subscription or fan out ratio of that port and not the network Because of this it is possible to design SANs with a certain amount of over subscription without adversely affecting application performance Because the benefits of storage consolidation grow proportionally with the number of hosts and storage the capability for a SAN to scale is important It is possible to choose a SAN architecture that can grow from tens of ports to hundreds and in some cases thousands of ports while minimizing or eliminating downtime Topologies such as the core edge are optimal for enabling this type of scaling and availability Backup A SAN based backup is in some respects a form of storage consolidation in that an I O device the tape drive is available to be shared by many hosts The difference is that the shared device is tape rather than disk This distinction can affect SAN design in several ways e Currently tape libraries tend to be single attach so the multi pathing approaches used in storage consolidation will usually not work e Backup devices tend to be more sensitive to I O disruption than disk arrays Arrays can recover from small glit
18. ee ee Status Healthy Polled at 03 23 04 04 11 PM Hame brocadessm Fabric OS version v4 241_1c1 Domain ID 1 Ethernet IP 10 64 210 25 Ethernet Mask 255 255 240 0 FCnet IP 0 0 0 0 FCnet Mask 0 0 0 0 Gateway IP 10 64 208 1 WWH 10 00 00 05 16 34 00 c7 Current Zone Config None Status Legend E Heatthy Marginal E critical Unmonitored ES Applet com brocade web switchview SwitchApplet started FJ Internet The switch icon for the Brocade SAN Switch Module switch consists of the following e External ports and status LEDs e Internal ports and status indicators e Switch status LEDs For more information on the purpose of these LEDs and indicators refer to the Si kWorm 3016 Hardware Reference Manual and the Brocade Enterprise SAN Switch Module for IBM eServer BladeCenter and Brocade Entry SAN Switch Module for IBM eServer BladeCenter Installation Guide For more information on Web Tools itself refer to the Brocade Advanced Web Tools Administrator s Guide Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 14 Fabric Watch Brocade Fabric Watch is a licensed product that monitors the health and performance of Fibre Channel networks and SilkWorm switches alerting you when problems arise before they become costly failures SAN managers can configure Fabric Watch software to monitor any of the following e Environmental conditi
19. oj x SwitchName brocadessm DomainID 1 WAAN 10 00 00 05 1 34 00 c7 Tue Mar 23 2004 4 16 PM Upload Download SNMP License Admin Port Setting Routing Extended Fabric Configure Switch Information Network Config Name and ID Name brocadessm Manufacturer Serial Nvos0000069 Domain ID fi Supplier Serial Zocoewanes Switch Status _ gt Email Configuration Enable Disable DNS Server 1 DNS Server 2 OoOo O O Domain Name OoOO O View Report Remove All l Report Apply Close Reset Refresh Switch Adrinistration opened Tue Mar 23 2004 4 16 PM Change current switch settings ay Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 98 20 7 a amp b Optional Core PID and Extended Edge PID Formats If introducing the Brocade SAN Switch Module into an existing Fabric OS 2 x and 3 x environment consider changing the setting of the Core PID on all switches running Fabric OS 2 x and 3 x The Brocade SAN Switch Module has the Core PID format ON set to 1 by default Fabric OS 4 2 or greater only supports PID Formats 1 and 2 2 is the Extended Edge Format If these switches are introduced to a fabric that has switches running Fabric OS 2 x or 3 x with the default Core PID format setting of 0 the Brocade SAN Switch Module will segment When introducing the Brocade SAN Switch
20. racking and installing the Brocade switches and other components Staging also includes configuring the firmware and software of each component Guidelines on how to accomplish these tasks are presented in this chapter in the form of checklists This chapter assumes that a new Core Edge fabric with clean Brocade switches will be staged A clean switch has no defined or active zoning configuration and has all default settings It is assumed throughout this chapter that a Core Edge topology has been chosen as part of the design criteria The guidelines in this chapter may need to be modified for other topologies This chapter will not explicitly cover an existing SAN fabric migration to other Brocade switch platforms Please refer to the SAN Migration Guide for recommended guidelines and procedures Staging a new SAN Fabric requires essentially two tasks The first task is uncrating racking cabling and providing power to the Brocade switches The second task is configuring the Brocade Fabric Operating System firmware The focus of this chapter will be on the commands and Web Tools views used in staging Brocade SAN fabrics with Fabric OS 4 2 1 New high level troubleshooting functions are also introduced within these Fabric OS releases and will be discussed at a high level in this chapter This chapter will not provide recommendations on racking cabling or power installation for the Brocade SilkWorm Fabric Switch Family Guidelines for these
21. 0 0 2 P12 0 0 2 SID O10 Performance Sid 010100 Did 0307ba Port 1 Ref rate 30 Transmitted Bytes sec Received Bytes sec SSS SSS SID O1D Performance Sid 010200 Did 0307bc Port 2 Ref rate 30 Transmitted Bytes sec Received Bytes sec J ava Applet Window Step 12 To save the Graphs configurations for use at a later time Click on the Actions Menu and choose the Save Current Canvas Configuration from the drop down menu list Figure 27 15 Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 156 27 Figure 27 15 Saving the Performance Monitoring Canvas Actions Performance Graphs Save Current Canvas Configuration ut Utilization Ref rate 30 Secs Display Canvas Configurations Display Resource Usage Print All Graphs N eo P7 0 0 2 P8 0 0 2 P9 0 0 2 P10 0 0 2 P11 0 0 2 P12 0 0 2G P13 0 0 2 P14 0 0 2 P15 35 5M 2 SID D10 Performance Sid 010100 Did 0307ba Port 1 Ref rate 30 Transmitted Bytes sec Received Bytes sec SID O10 Performance Sid 010200 Did 0307bc Port 2 Ref rate 30 Transmitted Bytes sec Received Bytes sec lava Applet Window Step 13 The Save Canvas Configuration dialog box appears Choose an Appropriate Name and an appropriate description for this set of graphs Click on the Save Canvas button and then clic
22. 101 Figure 20 19 portCfgShow information from the CLI File Edit Terminal 10 11 12 13 14 15 cu cu cu cu cu 2G 2G 2G 2G 2G N2 No_Light No Light No Light No_Light No_Light No_Module brocadessm USERID gt brocadessm USERID gt portcefgshow Ports of Slot 0 omi r he a 6 9 10 11 12 13 14 15 Speed Trunk Port Long Distance VC Link Init Locked L_Port Locked G Port Disabled E Port ISL R_RD Mode Persistent Disable Locked Loop HD AN 2G 26 2G 2G 2G 2G 26 2G 2G 2G 2G 2G 2G 2G AN ON se os ee ee oe oe Gir cic cits oo oe es oe ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON where AN AutoNegotiate OFF INVALID brocadessm USERID gt brocadessm USERID gt Connected to 10 64 210 25 telnet 20 10 Optional Name Devices with PortName In Fabric OS 4 2 1 there is a command portname that lets the administrator label a port Use portname to label port 26 on a Brocade SAN Switch Module as shown In the Port Setting tab of the Web Tools Admin interface there is a column called Port Name The Port Name can be modified from this location If using the CLI portName Figure 20 20 Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 102 20 Figure 20 20 Port Name informat
23. 50 MB sec I Sarver mox 50 MB sec Single ISL Storage Storage Note that in the ISL in Figure 13 1 the total amount of traffic that is intended to cross between switches never exceeds the 200 MByte sec capacity of the link assuming a 2 Gbit sec ISL Even if the servers in Figure 13 1 are running at their theoretical maximum performance there still might be performance bottlenecks with the storage devices In Figure 13 2 the two servers are accessing a single storage port so the 2 1 fan out of the storage port becomes the limiting factor Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 65 13 Figure 13 2 High bandwidth consumption with 2 ISLs switches with 2 ISLs Hosts The key to managing bandwidth is capturing or estimating performance requirements and matching these requirements to an appropriately designed SAN If the servers are capable of generating 400 MB sec of traffic and there are two 2 Gbit sec ISLs see Figure 13 2 the network routes the traffic over both of them and congestion does not occur The storage port can operate at only 50 MB sec therefore each server can average only 25 MB sec This scenario is common in storage consolidation environments where many servers need to share a single storage port However the T O requirements for most servers can be surprisingly low 1 or 2 MB sec and a single
24. 7f 90 FCP 2 3 INS i Q SwitchGroups lS PortGroups 4 5 _ steftler2 localhost Step 4 From the WebTools Performance Monitor window locate the SID DID performance option in the Advanced Monitoring drop down list of the Performance Graphs Tab Figure 27 7 Figure 27 7 SID DID Performance Graphs in WebTools F Performance Monitor brocadessm OOOO Actions Performance Graphs Basic Monitoring Advanced Monitoring SID DID SCSI Commands gt SCSI vs IP Traffic ALPA Error Performance J ava Applet Window Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 151 27 Step 5 The following SID DID Performance Setup window appears Figure 27 8 Figure 27 8 SID DID Performance Graphs in WebTools x Port or Sid Did Selection List Enter drag port number Domain_1 brocadessm l Domain_2 IBM_2109_F32 Domain_3 IBM_3534_1RU Domain_4 IBM_3534_F08 Enter drag SID number Hex Add Sid gt Enter drag DID number Hex Aidd Did gt lt Search OK Cancel Java Applet Window Step 6 Maneuver through the Port Selection List on the left to Port 1 IBM eServer BladeCenter Server Blade in bay 1 on Domain 1 the Brocade SAN Switch Module and find the SID for that device Figure 27 9 Figure 27 9 SID 010100 in Port Selection List a x Port or Sid Did Selection
25. 8 2 The Firmware VPD tab in the IO Modules section for the Brocade SAN Switch Module contains the firmware revision See Figure 2 3 Figure 2 3 Brocade SAN Switch Module Firmware VPD 10 64 210 20 BladeCenter Management Module Microsoft Internet Explorer O xj File Edit View Favorites Tools Help Back gt E Bi tat B Search Favorites Meda Bh 3 a Eil Address http 4 10 64 210 20fprivate main ssi ZO BladeCenter Management Module lO Module Firmware VPD Bay Type Firmware Type Build ID Released Revision 1 Ethernet SM Boot ROM BRESMB4G 11 30 2002 04 Main Application BRESMR4G 07 15 2003 68 plication 1 V O4 A 421 E Firmware VPD all Internet In the IO Modules Task tab click on Management to manage the Brocade SAN Switch Module from the IBM eServer BladeCenter Management Module See Figure 2 4 Figure 2 4 IBM eServer BladeCenter Management IO Module Task E 10 64 210 20 BladeCenter Management Module Microsoft Internet Explorer loj x File Edit view Favorites Tools Help e Back gt O A A seach GyFavorites GPmedia C4 D 3 A E Address a http 10 64 210 20 private main ssi x Go Links gt BladeCenter Management Module server Bay 3 Fibre SM Current IP Configuration Configuration method Static IP address 192 168 196 121 Subnet mask 255 255 255 0 Gateway address 192 168 196 1 New Static IP Configuration Status Enabled To change the I
26. Apply Reset Even though E mail notification can be specified for any class element it is recommended that it should be set to receive notification only for critical alerts that require immediate attention For example instances like fabric re configurations fabric segmentations switch failures or security violations The remaining informational and warning messages can be left directed to the system Error log Guideline Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 143 26 Track Changes Setup As an added security monitoring feature Brocade switches have the ability to track the occurrence of an administrator logging into the switch or logging out of a switch and whether a switches configuration has changed The commands used to configure the Track Changes feature are listed under the Track Changes help command trackChangesHelp To determine if this feature is turned on use the trackChangesShow command The Track Changes feature can also send its event messages to SNMP To configure Track Changes and send SNMP Traps when events occur use the trackChangesSet command The sw7TrackChangesTrap in the file named SW_v5_1 mib described in the SNMP section above receives Track Change notifications of an event Figure 26 7 shows an example of turning on the Track Changes feature This feature can only b
27. BM switchGroups IBM_2109_F32 v4 2 0b 10 64 210 104 Subordinate 118 1080676 FAO3X9012E7 Online 12 BM S PortGroups IBM_2109_F32 v4 2 0b 1064 210 150 Subordinate 2 1080645 FA04x905009 Online 12 BM v3 1 2a 10 64 210 106 Subordinate 120 1309331 Online 16 3 1 2a 10 64 210 105 Subordinate 119 1093326 Online 9 v3 1 2a 1064 210 151 Subordinate 4 1040508 Online 16 v2 6 2a n 0 64 210 107 Principal 121 Online 2 v262a 1064210152 _Subordnate 5 Online 4 a 10 steffler2 localhost 4 Check Switch Status The recommended CLI command to check the Brocade SAN Switch Module status is switchStatusShow The Switch status can also be found in the Web Tools and Fabric Manager interfaces See Figure 20 10 for the Web Tools view Click on the Status button from the main Web Tools Window Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 96 20 Figure 20 10 Switch Status in Web Tools switch Status for brocadessm me IC x e overall switch status is HEALTHY OK JavaAppletWindow 5 Set the Date and Time To set the date and time use the date command Figures 20 11 and 20 12 have examples that set the date and time using the CLI and Fabric Manager Using date with no arguments provides the current day and time Alternatively Fabric Manager can be used to push the current time on the server where Fabric M
28. Fabric OS Procedures Guide for the specific version of Fabric OS used on the switch Be sure to upgrade to the firmware qualified or recommended by the switch provider Each Brocade based SAN is unique This is due to the wide variety of OS platforms HBAs and storage arrays that may be attached This uniqueness means that each upgrade needs to be carefully planned to minimize the risk of unscheduled downtime Scheduled downtime is normally required for single fabric SANs For dual fabric SANs upgrade one fabric at a time Fabric OS 4 2 1 only supports the FTP protocol for firmware upgrades If you wish to use the CLI the Brocade SAN Switch Module allows two admin telnet sessions Use one for irmwaredownload and the other for firmwaredownloadstatus Firmwaredownloadstatus is a handy command that shows a log of each upgrade phase When complete use firmwareshow to display the firmware version on each compact flash partition The Firmware download procedure can also be completed using the Web Tools and Fabric Manager GUI interfaces Fabric Manager has the added advantage of being able to download to multiple switches concurrently Fabric OS Upgrade Planning Checklist 1 Analyze the potential risks and impact to each device on the SAN 2 In order to maximize fall backwards capability preserve each fabric switch configuration with configupload 3 Use Fabric Manager for larger multi fabric SANs 4 Verify the upgrade version is supportabl
29. IBM eServer BladeCenter Design Deployment and Management Guide describes the following Brocade optionally licensed products Brocade Advanced Web Tools Brocade Advanced Zoning Brocade Fabric Watch Brocade ISL Trunking Brocade Advanced Performance Monitoring Brocade Extended Fabrics Brocade Remote Switch Brocade Secure Fabric OS Advanced Security Brocade Fabric Manager The information for Advanced Zoning ISL Trunking Advanced Performance Monitoring Extended Fabrics and Remote Switch has been consolidated into the Brocade Fabric OS Features Guide Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 vii The other optional products have their own documentation and can be found in the following additional user s manuals Advanced Web Tools Administrator s Guide for procedures using Advanced Web Tools Fabric Watch User s Guide Secure Fabric OS User s Guide Secure Fabric OS QuickStart Guide and the Fabric Manager User s Guide for procedures using Fabric Manager These products have separate books because the books are too large to be easily incorporated into the Brocade Fabric OS Features Guide Also because Brocade Advanced Web Tools and Brocade Fabric Manager are alternate ways of managing a fabric they are documented separately as the counterpart to the Brocade Fabric OS Procedures Guide There are several other
30. Initiate a fastboot on a switch module with no I O to simulate edge switch failure Repeat this test three times in succession The switch should fully recover after the fastboot and the fabric should rebuild Other devices not associated with the rebooted switch should continue to perform I O Verify this with the fabricshow command Case 2 Core Switch Failure Initiate a fastboot on a core switch Repeat this test three times in succession The switch should full recovery after the fastboot and the fabric should rebuild Verify this with the fabricshow command Case 3 CP Failover SilkWorm 12000 24000 Initiate an hafailover on an active CP with I O running in the fabric Full recovery without an I O pause should occur in all cases Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 110 21 Case 4 Non Disruptive Code Load Fabric OS 4 1 or higher only Initiate a firmwaredownload on a switch with Fabric OS 4 1 or higher Full recovery without an I O pause should occur in all cases Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 111 Chapter 22 Maintenance After the Brocade SAN fabrics have been staged and validated it is important to maintain them for proper and continued operation This chapter will provide guidelines on firmwar
31. Light 2G 2G Bay11 izi Apply Close Reset Refresh Switch Administration opened Fri Mar 26 2004 3 07 PM Configure Port Setting parameters Note Do not change the default port settings on the Brocade SAN Switch Module for any of the Internal ports 1 to 14 Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 27 Designing the Fabric for Optimize Use of ISL Trunking ISL Trunking can be used to simplify SAN storage area network design and improve performance When designing the SAN consider the following recommendations in addition to the standard guidelines for SAN design Note Evaluate the traffic patterns within the fabric Discussed in more detail in the Fabric OS Features Guide and the Brocade SilkWorm Design Deployment and Management Guide SAN DDM Version3 0 This allows your fabric to implement trunking groups that will help to optimize its performance Activate an ISL Trunking license on each switch that is expected to participate in a trunking group Verify that the lengths of the ISLs in the group do not differ by more than the recommended values The lengths of the ISLs in the group should differ by less than 30 meters recommended and must differ by less than 400 meters Large differences in latency decrease the efficiency of load sharing If cable lengths differ by 400 meters or m
32. Management Fabric Management Recommendations High Availability User Access Levels Switch Licensing IBM eServer BladeCenter Management Module Fabric OS Command Line Interface Advanced Web Tools Fabric Watch Fabric Manager SNMP Fabric OS Access Layer API Management Server Overview of Switch Management The Brocade SAN Switch Module can be managed using many local and remote access methods including access from the IBM eServer BladeCenter Management Module Web interface To manage a switch you must have access to one of the following available management methods IBM eServer BladeCenter s Management Module Web Interface Fabric OS Command Line Interface CLI through Telnet or Secure Shell ssh Advanced Web Tools through a supported Web Browser Brocade Fabric Manager a stand alone JAVA application SNMP through a Third Party SNMP management application Brocade Fabric Access Application Programming Interface API through a Third Party management product or a Scripting ToolKit Interface Management Server through a Third Party management product Telnet Advanced Web Tools Fabric Manager SNMP and the Fabric Access API require that the switch be accessible using a network connection The IBM eServer BladeCenter provides this network connection through the Management Module s Ethernet port out of band The switch module must be configured with an IP address to allow for the network connection Refer to the Brocade SilkWor
33. Module to an existing fabric of legacy switches with the Core PID format of 0 it is recommended to change the legacy switches Core PID parameter to 1 This is a fabric wide parameter Before changing the Core PID parameter please read the recommendations and details for setting the Core PID format in the Fabric OS Procedures Guide and consult the Brocade SilkWorm Design Deployment and Caution Management Guide SAN DDM Version 3 0 for more information on the deployment of these settings 7c Optional Principle switch The principal switch is responsible for handing out domain IDs to the rest of the fabric upon a fabric build In some cases it may be desirable to hard set a switch to always be the principal switch In Fabric OS 4 1 or greater only a preferred principal switch can be selected Please read the recommendations and details for setting the Core Principle switch in the Fabric OS Procedures Guide and consult the Brocade SilkWorm Design Deployment and Management Guide SAN DDM Version 3 0 for more information on the deployment of this feature 8 Add Fabric OS Licenses Follow the instructions given by your switch provider This normally entails using the paper pak instructions and going to a web site brocade com and entering a transaction key number 1 Inthe Info tab of the Web Tools main interface look in the LicenseID field If using the CLI licenseidshow Figure 20 15 2 Inthe License Admin tab o
34. SNMP Trap configuration allows the user to choose which MIB trap elements will be used to send information to the SNMP management station There are two main MIB Trap choices for the user to configure The user can choose to use the Brocade specific MIB Trap which is associated with the Brocade specific SilkWorm SW MIB or the user can choose to use the Fibre Alliance MIB Trap which is associated with the Fibre Alliance FA MIB The Brocade MIB was created specifically for monitoring SilkWorm switches The Fibre Alliance MIB was created by a consortium of companies to manage any type of SAN switch and SAN devices from any company If you wish to use both it should be noted that some of the information that is sent is the same for both MIBs If both Traps are chosen the user may receive multiples of the same traps duplicates of the same information The user can also turn off the FA MIB itself completely There is no choice to turn off the SW MIB it is always enabled The user also has the choice of enabling or disabling two additional MIBs and their associated TRAPs the FICON MIB HA MIB FICON TRAP and HA TRAP The FICON MIB and TRAP are only required for FICON environments and the HA MIB and TRAP are only necessary for the Enterprise Class Switches and Directors SilkWorm 12000 and SilkWorm 24000 The SW EXTTRAP is used to provide additional information to a Trap as it includes the swSsn Software Serial Number to be sent as a part of the Brocad
35. Z Fabric Watch for brocadessm Microsoft Internet Explorer Fabric Watch Alarm Notification Threshold Configuration Email Configuration 2 Fabric Switch Select Area E Ports Downs z Environment SFP E Parts Downs Security Fabric Reconfigure Switch Availability Area Configuration Element Configuration ConfifDomain ID Changes Ports Segmentation Boundary Port Zone Changes Fabric lt gt QL FIFL Copper Port Unit Down s Down s Fabric Logins ne none M So FIFL Optical Port fig oe SFP State Changes 0 fo i AL PA BufferSize 0 fo End to End Filter Based Select Boundary Level Default x E Port Boundary Settings Default Settings in Parent Alarm Setting Alarm Notification Mechanisms Default Mechanisms in Parentheses 7 ERRLOG 7 shim M PORTLOG 7 RAPT Jo EMAIL M ERRLOG J M PORTLOG M RAPI P EMAIL 7 ERRLOG J 7 PORTLOG hm EMAIL 7 ERRLOG J M PORTLOG M RAPI Jo EMAIL Inbetween 7 ERRLOG J PORTLOG 7 RAPI Jo EMAIL Select Alarm Level Default v I Save Configuration to Switch Apply Reset For more information refer to the Brocade Fabric Watch User s Guide Fabric Manager Brocade Fabric Manager is a java based management application that provides a central point of control to manage multiple fabrics Fabric Manager is tightly coupled with Brocade Web Tools Fabric Watch and Advanced Performa
36. address all of the design implications that must be addressed in an Advanced Security Secure Fabric OS environment For comprehensive information on how security works refer to the Brocade Secure Fabric OS User s Guide and the Brocade SilkWorm Design Deployment and Management Guide SAN DDM Version 3 0 these documents provide the necessary guidelines for implementing security Note Note Note A secured fabric must be entirely secured and all switches in a secured fabric must run a version of Fabric OS that supports security and these switches must be licensed to run security The following minimum version requirements apply for fabrics that need secure mode enabled e Fabric OS version 2 6 2 or higher with the Security license for secure fabrics containing a SilkWorm 2000 series switch and any of the following SilkWorm switches 3016 3200 3250 3800 3850 3900 12000 24000 e SilkWorm 3200 and 3800 switches Secure Fabric OS version 3 1 or higher with the Security license e SilkWorm 3900 and 12000 switches Secure Fabric OS version 4 1 or higher with the Security license e SilkWorm 3250 3850 and 24000 switches Secure Fabric OS version 4 2 or higher with the Security license e Brocade SAN Switch Module Advanced Security Secure Fabric OS version 4 2 1 or higher with the Security license The Brocade SAN Switch Module switch has a different default username than admin which exists on all other Silk Worm s
37. apply to the Fibre Channel IP address and the Fibre Channel subnet mask also referred to as the in band IP address and subnet mask These can still be configured using any of the standard switch management mechanisms In the IO Modules Task tab click on Management to manage the Brocade SAN Switch Module s IP Address from the IBM eServer BladeCenter Management Module In this example the Current IP Address is 192 168 196 121 This example shows how to change that to 10 64 210 25 Follow Figures 20 1 to 20 4 By default the IBM eServer BladeCenter Management Module assigns the Brocade SAN Switch Module in I O Bay 3 with 192 168 70 129 and the Switch Module in I O Bay 4 with 192 168 70 130 The default IP Address for the IBM eServer BladeCenter Management Module is 192 168 70 125 Note Figure 20 1 IP Address configuration using the IBM eServer BladeCenter Management Module Step 1 View current configuration 10 64 210 20 BladeCenter Management Module Microsoft Internet Explorer loj x File Edit View Favorites Tools Help Kal Back gt O A A Reach Favorites meda A BD 3 A Gi z 6o Links Address 0 64 210 20 private main ssi BladeCenter Management Module server Bay 3 Fibre sm Current IP Configuration Configuration method Static IP address 192 168 196 121 Subnet mask 255 255 255 0 Gateway address 192 168 196 1 New Static IP Configuration Status Enabled To change the IP config
38. are also seen automatically in the GUI so validation of connectivity can be done simultaneously If possible use persistent binding on the host This will provide consistent controller target and LUN numbers for each storage LUN Backup applications are especially sensitive as these numbers map directly to the backup application device identities Guideline If using World Wide Names WWN zone by port World Wide Name rather than node World Wide Name This is because a port World Wide Name Guideline uniquely identifies a port to which a target is attached Some Multi pathing software may get confused and not be able to discover targets properly This is especially true when using multi port HBAs Be aware of mixing different HBA vendors in a single zone Each vendor Guideline HBA responds differently to RSCNs a method to notify an HBA for device discovery and may cause one of the HBAs to lose the zoned device Guideline Use single initiator zones that have one HBA per zone Separate HBAs from each other for clustered hosts Allow each HBA to see Guideline the same storage but not each other Once again RSCNs may cause the clustered host HBA to lose the storage array Security Planning Due to the complex nature of Security this Guide cannot address all of the deployment implications that must be addressed in an Advanced Security Secure Fabric OS environment For comprehensive information on how security works refer to the Brocad
39. counts the frames that have an SID of 0x050c00 and a DID of 0x111eef For monitor 0 RX_COUNT the number of words from Host A to Dev B TX_COUNT the number of words from Dev B to Host A CRC_COUNT the number of frames in both directions with CRC errors Monitor 1 counts the frames that have an SID of 0x11 1leef and a DID of 0x050c00 For monitor 1 RX_COUNT the number of words from Dev B to Host A TX_COUNT the number of words from Host A to Dev B CRC_COUNT the number of frames in both directions with CRC errors End to end performance monitoring monitors traffic on the receiving port respective to the SID only In Setting End to End Monitors on a Port if you add a monitor to Port 30 specifying Device B as the SID and Host A as the DID no counters except CRC will be incremented Notes Filter Based Monitoring Filter based monitoring counts the number of times a frame with a particular pattern is transmitted by a port Filter based monitoring is achieved by configuring a filter for a particular purpose The filter can be a standard filter for example a read command filter that counts the number of read commands transmitted by the port or a user defined filter that you customize for your particular use The maximum number of filters is eight per port however depending on your combination of standard and user defined filters that number might be reduced Standard Filter Based Monitors Using the standard filters you can co
40. displays the number of CRC errors that have occurred on frames sent from each AL _PA ona specific port AL_PA based performance monitoring does not require explicit configuration The switch hardware and firmware automatically monitors CRC errors for all valid AL_PAs End to End Monitoring End to end monitoring provides information regarding performance between a source SID and a destination DID on a fabric or a loop Up to eight SID DID pairs can be specified per port For each of the SID DID pairs the following information is available e Number of fibre channel words received by the port for the SID DID pair RX_COUNT e Number of fibre channel words transmitted from the port for the SID DID pair TX_COUNT e Number of frames with CRC errors received at or transmitted from the port for the SID DID pair CRC_COUNT To enable end to end performance monitoring you must configure an end to end monitor on a port specifying the SID DID pair The monitor counts only those frames with matching SID and DID Each SID or DID has three fields listed in the following order e Domain ID DD e Area ID AA e AL PA PP For example the SID 0x118a0f has domain ID 0x11 area ID Ox8a and AL_PA Ox0f The prefix 0x denotes a hexadecimal number The monitor counts the number of words received number of words transmitted and number of CRC errors detected in frames qualified using either of the following two conditions e For frames receive
41. end visibility into the fabric e More accurate reporting for service level agreements and charged access applications e Shortened troubleshooting time e Better capacity planning e Increased productivity via preformatted and customized screens and reports e The Advanced Performance Monitoring product provides the following functionality e Measures the bandwidth consumed by individual routes host target pairs e Provides device performance measurements by port AL_PA and LUN e Reports CRC error measurement statistics e Compares IP versus SCSI traffic on each port e Includes a wide range of predefined reports e Allow you to create customized user defined reports Advanced Performance Monitoring Architecture Using Advanced Performance Monitoring you can track the following e Number of CRC errors for AL_PA devices e Number of words received and transmitted in Fibre Channel frames with a defined SID DID pair e Number of frames with CRC errors received at the port with a defined SID DID pair e Number of times a particular filter pattern in a frame is transmitted by a port Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 31 These functions are broken down into three configuration groups e AL PA monitoring e End to end monitoring e Filter based monitoring AL_PA Monitoring AL_PA performance monitoring tracks and
42. essential SAN deployment information to ease the transition from the planning stage to production All checklists can be used as is or modified to fit specific SAN environments Taken as a whole the guidelines and practices discussed within each checklist will allow for sound decisions optimizing the IT investment once in production The checklists are not meant to be all inclusive Other methods of performing the activities discussed are possible and encouraged It is crucial that the production SAN be supportable simple to maintain and easily scaled Proper planning and documentation is critical to make these goals attainable Effective up front planning simplifies the actual staging phase Node location will be known storage requirements met and cabling simplified Proper documentation functions much like a map allowing the quick identification of devices and configuration settings This reduces the potential downtime whether it is scheduled or not Since these benefits and others are so important there is emphasis on putting together an effective planning strategy Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 80 Chapter 19 Planning Having an effective plan prior to the staging of the equipment is critical for overall SAN deployment success This success is measured in many forms The greatest benefit of a good plan is that it ge
43. external failures The IBM eServer BladeCenter has the option to install 2 Brocade SAN Switch Modules into the Switch Module bays at the rear end of the chassis Each of the Internal Host connections if installed with the Fibre Channel Expansion Cards have redundant connections to the Brocade SAN Switch Modules in bay 3 and in bay 4 The IBM eServer BladeCenter architecture provides dual connected hosts to redundant fabrics right inside the BladeCenter Chassis The full benefit of this is to extend that redundant resilient internal architecture outside of the box when connecting the IBM eServer BladeCenter to an external SAN fabric It is recommended to connect the Brocade SAN Switch Module in Bay 3 to Guideline one fabric and the Brocade SAN Switch Module in Bay 4 to a separate redundant fabric This provides the maximum failure protection An example of the IBM eServer BladeCenter integrated into a resilient redundant Core Edge Topology can be seen in Figure 11 4 Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 60 11 Figure 11 4 Brocade SAN Switch Module s in a Dual Fabric Core Edge Topology Internal Brocade Switch Modules become Edge switches in the fabric IBM A Bay 3 BladeCenter gay 4 Brocade Brocade Switch Module Switch Module amp Dual Connected Failed Core Y Switch Resilient Redundant Fab
44. factory This is required for Advanced Security Secure Fabric OS implementation e pkishow Fabric OS 4 1 e configshow pki Fabric OS 3 1 2 6 1 4 If the PKI objects do not exist obtain the PKICert tool to setup the fabric for secure mode This is required for older switches This utility runs on Windows and Solaris only This can be obtained from the following IBM Web Site link the for Advanced Security Secure Fabric OS Upgrades http www 1 ibm com support docview wss rs 0 amp context HW200 amp context S WJ00 amp q 2109 2Bssel amp uid ssg 1 1001653 amp loc en_US amp cs utf 8 amp lang 5 Download and Install Brocade SecTelnet and Secure Shell client SSH Security Software Utilities 6 Verify Fabric OS Version Update as required 7 Install Security and Zoning Licenses on all switches in the SAN This is required for Advanced Security Secure Fabric OS 8 Schedule downtime when enabling secure mode A reboot of each fabric in the SAN is required as the firmware update is disruptive 9 Recommended If introducing the Brocade SAN Switch Module into an existing Fabric OS 2 x and 3 x environment consider changing the setting of the Core PID on all switches running Fabric OS 2 x and 3 x Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 84 19 Before changing the Core PID parameter please read the recom
45. gt 2 Figure 25 6 shows an example of turning the Brocade SW TRAP information on the FICON MIB TRAP information off the HA MIB TRAP information off and the Fibre Alliance FA TRAP information off Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 131 25 Figure 25 6 Brocade SilkWorm SW Trap configuration with snmpmibcapset brocadessm USERID gt snmpmibcapset The SNMP Mib Trap Capability has been set to support FE MIB SW MIB FA MIB FICON MIB HA MIB SW TRAP FA TRAP FICON TRAP HA TRAP FA MIB yes y no n yes n FICON MIB yes y no n yes n HA MIB yes y no n yes n SW TRAP yes y no n yes FA TRAP yes y no n yes n SW EXTTRAP yes y no n no FICON TRAP yes y no n yes n HA TRAP yes y no n yes n brocadessm USERID gt 3 Figure 25 7 shows an example of turning the Fibre Alliance specific Trap information on and the Brocade Trap information off Figure 25 7 Fibre Alliance FA Trap configuration with snmpmibcapset brocadessm USERID gt snmpmibcapset The SNMP Mib Trap Capability has been set to support FE MIB SW MIB SW TRAP FA MIB yes y no n no y FICON MIB yes y no n no HA MIB yes y no n no SW TRAP yes y no n yes n FA TRAP yes y no n no y SW EXTTRAP yes y no n no FICON TRAP yes y no n no HA TRA
46. is persistent across switch reboots and power cycles 11 Optional Set Telnet Session Timeout Value It is recommended to at least maintain the default admin telnet session timeout value which is set to 10 minutes This ensures security best practices and prevents a telnet session from locking up access to a switch The Brocade SAN Switch Module s telnet timeout default is 10 minutes If you wish to change this value use the timeout CLI command Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 103 20 12 Optional Customize Monitoring Features Although optional it is recommended to make you aware of the following features e SwitchStatusPolicySet which monitors the status of several key health parameters e TrackChangesSet which tracks the occurrence of an administrator logging into the switch or logging out of a switch and whether a switches configuration has changed e Configuration of the Switch Module s SNMP agent so that you can incorporate the switch into your existing SNMP management framework e Fabric Watch tuning for your particular environment For guidelines on setting these refer to SAN Management section later on in this Guide Additional information can be found in the Brocade SilkWorm Design Deployment and Management Guide SAN DDM Version 3 0 as well 13 Environmental Status There are several commands
47. is not measured so much by how many files are located on the storage but by the amount of space taken up by the files It is recommended to review the zoning configuration of a fabric periodically for unused alias zoning and configuration entries and to then delete these unnecessary entries Unnecessary alias zone and configuration entries frequently result from the merging of fabrics or the addition of a switch with predefined zones into an existing fabric The maximum zoning database size for Silk Worm 2000 series 3200 and 3800 switches is 96 KB and 128 KB for the Brocade SAN Switch Module switch A switch with a zoning database size limit of 96 KB limits the size of the zoning database for the whole fabric to 96 KB Note Limit the name of an alias zone or configuration to as few characters as Guideline possible while maintaining meaning of that name Target 16 characters or less for an alias zone or configuration name For SANs that exceed several hundred ports monitor the size of the fabric Guidelme zoning database with the CLI command cfgSize Routinely review a zoning configuration to identify unused aliases zones Guideline and configurations and then remove these unused entries Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 11 Chapter 17 Security Design Due to the complex nature of Security this Guide cannot
48. manuals that provide more detailed information on the CLI commands SNMP and the Diagnostics available in this Fabric Operating System The Brocade Fabric OS Reference Manual covers command line descriptions the MIB Reference Manual covers the SNMP agent supplied by the switch and the Diagnostic and System Error Messages Reference Manual covers the switch diagnostic commands and messages The Brocade Fabric OS Procedures Guide completes the set of manuals provided and offers more in depth descriptions of the commands that can be used and how to use them to manage Brocade Fabrics to help storage area network SAN administrators like you configure and manage your Brocade SilkWorm SAN In addition there are two hardware reference manuals that explain how to plug the Brocade SAN Switch Module into the IBM eServer BladeCenter chassis and get the unit up and running The Sil kWorm 3016 Hardware Reference Manual and the SilkWorm 3016 QuickStart Guide Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 viii Document Conventions This section describes text formatting conventions important notices formats and terms as they are used in this document Text Formatting The following table describes the narrative text formatting conventions that are used in this document Convention Purpose e Identifies command names e Identifies GUI elements bol
49. normal operation Some devices only support active passive dual pathing With active passive dual pathing the passive path is utilized only when the primary path fails Some devices such as tape drives are not currently capable of supporting multiple paths It is possible to address this issue by equally distributing tape devices between the redundant fabrics and configuring the backup applications to use an alternate tape drive should an outage on one of the fabrics occur However some elements of a tape backup solution such as Robot control within a single library enclosure do not currently map well into a redundant fabric environment Any single attached devices such as a tape drive non critical storage and hosts can be single attached by alternately assigning them between the fabrics When implementing a logical group of single attached devices such as a tape library with multiple tape drives and a robot ensure that these devices reside on the same fabric and if possible on the same switch When deploying redundant fabrics it is not always necessary to deploy symmetrical fabrics For example when using a dual fabric the first fabric could consist of several interconnected SAN islands while the second fabric consists of isolated islands The IBM eServer BladeCenter architecture provides all of the internal and external capabilities to connect to resilient redundant fabrics It was designed to provide the maximum resilience to internal and
50. not used in the calculation of the overall switch status Figure 26 8 Switch Status Policy settings brocadessm USERID gt switchstatuspolicyshow The current overall switch status policy parameters Marginal FaultyPorts MissingSFPs Temperatures PortStatus ISLStatus brocadessm USERID gt Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 145 26 Figure 26 9 shows the configuration of the Switch Status Policy feature Figure 26 9 Switch Status Policy configuration brocadessm USERID gt switchstatuspolicyset To change the overall switch status policy parameters The current overall switch status policy parameters Marginal FaultyPorts MissingSFPs Temperatures PortStatus ISLStatus Note that the value 0 for a parameter means that it is NOT used in the calculation In addition if the range of settable values in the prompt is 0 0 the policy parameter is NOT applicable to the switch Simply hit the Return key The minimum number of FaultyPorts contributing to DOWN status 0 FaultyPorts contributing to MARGINAL status MissingSFPs contributing to DOWN status 0 MissingSFPs contributing to MARGINAL status Bad Temperatures contributing to DOWN status 0 Bad Temperatures contributing to MARGINAL status Down PortStatus contributing to DOWN status 0 Down PortStatus contri
51. part of the HCA process This process completes in less than 60 seconds with the critical fabric services available in about 50 seconds Again although there is a window when fabric services are unavailable there is no disruption to end to end data flow between SAN devices Because it does take longer to do hot code activation on the Brocade SAN Switch Module as compared to the SilkWorm 24000 or SilkWorm 12000 which both have redundant CPs the switches directly linked to the Brocade SAN Switch Module need to be tolerant of the 60 seconds when no fabric services from the Brocade SAN Switch Module will be seen Brocade has made the necessary modifications to Fabric OS v3 1 0 and v2 6 1 and later versions to extend the time out values so that link and fabric re configuration is avoided For this reason it is strongly recommended that customers deploy neighboring switches i e immediately connected via an E port to the Brocade SAN Switch Module with Fabric OS v2 6 1 or later on the SilkWorm 2000s series Fabric OS v3 1 0 or later on the SilkWorm 3200 3800 or Fabric OS v4 1 0 or later on the SilkWorm 3250 3850 3900 12000 24000 During the Brocade SAN Switch Module hot code activation if earlier releases Fabric OS v2 6 0x Fabric OS v3 0 2x or v4 0 x are deployed on the neighboring switches to a Brocade SAN Switch Module a time out will occur on these neighboring switches resulting in a fabric re configuration The Brocade SAN Switch Modu
52. port persistently use portcfgpersistentenab le To temporarily enabled a port use portenable PortCfgDefault will turn off persistent disabling of a port Note PortCfgDefault will also set all other port settings to default values When a portcfgpersistentdisable is done on an enabled E port a fabric reconfiguration may occur This is the same behavior as the Warning portdisable command Persistently disabling groups of ports is not supported Each port must be persistently disabled individually Do not use the portCf gDefault command on the Brocade SAN Switch Module Use individual port commands to configure ports The Brocade SAN Guideline Switch Module is shipped with particular defaults to ensure it works properly as an embedded system where known deterministic devices will be attached to the Internal 14 ports Note The IBM eServer BladeCenter Management Module performs the portCfgPersistentDisable if the Disable External Ports has been configured in the Advanced Setup Menu of the IO Module Tasks for that IO bay Figure 22 4 Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 116 22 Figure 22 4 IBM eServer BladeCenter Management Module External Ports Setting iojxi File Edit View Favorites Tools Help EJ Back gt O A A Aseach Favorites media D 3 A l Address http 10 64 210 20 private main ssi v
53. storage port can sustain many hosts without overwhelming its I O capability O Profiles Soage Understanding an application s I O requirements is essential to the SAN design process An individual I O can be classified as either a read or a write operation Although I O is usually a mixture of reads and writes some applications are strongly biased For example video server I O activity is normally almost 100 percent reads while video editing cluster I O may be mostly writes Tape I O is primarily write oriented unless doing a restore T O can further be classified as random or sequential Examples of random I O include an e mail server or an OLTP server Sequential I O is characteristic of decision support such as data warehousing scientific modeling applications or backup applications The third characteristic of I O is size which typically ranges from 2 KB to over 1 MB Typically user file systems have smaller I O sizes whereas video servers or backups may have very large sizes The Brocade SilkWorm Design Deployment and Management Guide SAN DDM Version 3 0 provides more detail on the application I O profiles that establish the typical magnitude of application bandwidth consumption For SAN design performance purposes I O is classified by bandwidth utilization light medium and heavy It is very important to support test assumptions by gathering actual data when possible You can gauge the type of I O activity in your existi
54. the SAN any device can see any other device This is the default setting Once zoning is in place all devices must be members of a defined zone Those devices that are not will be blind to all others This section will provide some guidelines as to zoning plan definition For additional information reference the Brocade Zoning User s Guide More detail is provided in the Brocade SilkWorm Design Deployment and Management Guide SAN DDM Version 3 0 Zoning requires careful thought and planning Armed with the documentation created in the previous section and understanding the requirements allows the creation of a zoning plan Creativity is important here as there is no one correct zoning configuration for a given SAN fabric configuration In general follow any specific zoning recommendations provided by the switch vendor Zoning Plan Checklist 1 Gather the list of host and storage devices to be zoned from the device spreadsheet 2 Define the storage requirements for each host based upon software application requirements 3 Adhere to recommended storage device configurations such as LUN masking LUN security and other specific features supported by the vendor 4 Consider specific host requirements for storage value added feature sets such as Server Free backup LUN snapshots or LUN mirroring over distance Some key points when planning for zoning e Clearly understand the storage requirements for each host This means understandi
55. the Technical Committee T11 Web site This Web site provides interface standards for high performance and mass storage applications for fibre channel storage management as well as other applications http www tl1 org For information about the Fibre Channel industry visit the Fibre Channel Industry Association Web site http www fibrechannel org Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 xiii Getting Technical Help Contact your switch support supplier for hardware firmware and software support including product repairs and part ordering To expedite your call have the following information available 1 General Information Technical Support contract number if applicable Switch model Switch operating system version Error messages received supportshow command output Detailed description of the problem and specific questions Description of any troubleshooting steps already performed and results 2 Switch Serial Number The switch serial number and corresponding bar code are provided on the serial number label on the side of the unit The serial number is 12 characters long and looks something similar to this SN ZXXXXX3WA069 The serial number label is located as follows SilkWorm 3016 switch Side of switch module It can also be obtained o From a telnet session to the switch use the chassissh
56. the switch etc Customers planning to build very large fabrics approx more than 1000 ports should plan on implementing them solely with the Brocade 2 Gbit sec switch family as there are limitations in the memory and processor power of the legacy SilkWorm 1 Gbit sec switches Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 74 15 The Brocade SAN Switch Module utilizes a different and more powerful control processor than the legacy SilkWorm 2000 series 3200 and 3800 switches Additionally the Brocade SAN Switch Module is configured with more memory than the SilkWorm 2000 and 3200 3800 switches These additional resources position the Brocade SAN Switch Module to function well in larger fabrics Hot code activation HCA is one of the features in Fabric OS 4 1 and later versions including v4 2 1 The defining characteristic of hot code activation is that while a new firmware image is being activated on a switch there is no disruption of end to end data flow between the hosts and storage devices No disruption means no dropped frames no retries and no time outs The ASICs on the switch continue to process frames while the new firmware is being activated Hosts that are logged into their targets will never be aware that anything has happened The Brocade SAN Switch Module with only one Control Processor must shutdown and reboot Fabric OS as
57. to reset the Brocade SAN Switch Module back to factory defaults The Restore Factory Defaults option in the Advanced Management section of the I O Module Management Task will reset the passwords back to the factory default USERID PASSWORD setting and allow the user to change the other 3 on the first login Figure 20 6 Figure 20 6 Login change password Challenge lolx File Edit view Favorites Tools Help Ea Back gt gt A A GAsearch Favorites Media 3l BD 4 amp ol Address http 10 64 210 20 private main ssi v Go Links BladeCenter Management Module server Restore Factory Defaults This action will cause all module settings to be set to their factory defaults You will lose all the changes you made to the configuration of this module as a result In order to preserve the new IP configuration set the field labeled Preserve new IP configuration on all resets to enabled Clearing of the configuration will be followed by a restart of the module Click the Restore Defaults button if you want to proceed Cancel Restore Defaults Send Ping Requests You can test the internal path between the management module and the switch module by sending it ping requests In order to ping this switch module at its current IP adrress of 10 64 210 25 click the Ping Switch Module button Cancel Ping Switch Module Ei Internet h Brocade Enterprise and Entry SAN Switch Modules for IBM
58. 1 23 93 ess093 5 2G 500 2 6 32 ESS032 13 1G 3 13 32 ESS032 6 2G 500 4 5 93 ess093 23 1G 1000 5 9 31 ess031 E Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 118 22 If no parameters are given the PathInfo command works in interactive mode The menu is displayed much like the configure command Interactive mode allows more parameters to be specified The choices are shown below with default values e max hops default 25 e domain required e source port default embedded e destination port default embedded e enable basic statistics default no e enable extended statistics default no e trace reverse path default no e source route default no e strict source routing may be specified if source route enabled e timeout default 5 seconds The interactive mode menu is shown with various choices being made are shown in the following output ess03l admin gt pathinfo Max hops i227 25 Domain 1 239 1 32 Source port 0 15 1 Destination port 0 255 1 13 Basic stats yes y no n no y Extended stats yes y no n no y Trace reverse path yes y no n no y Source route yes y no n no n Timeout 1 30 5 Pathinfo can also be used in advanced mode to display basic statistics The following example shows partial output with extended statist
59. 2 1 change threshold alarm level change changed alarm change exceeded alarm change below alarm change above alarm change inBetween alarm apply threshold alarm changes cancel threshold alarm changes return to previous page 1 2 3 4 5 6 7 8 9 OMDAIHDNPWNHE ndex ThresholdName BehaviorType BehaviorInt 0 eportRXPerf000 Triggered I 15 eportRXPerf015 Triggered 1 Threshold boundary level is set at Default Default Custom Unit KB s KB s Time base Low 120000 120000 High 220000 220000 BufSize 0 0 Threshold alarm level is set at Custom Errlog 1 SnmpTrap 2 PortLoghock 4 RapiTrap 8 EmailAlert 16 Valid alarm matrix is 31 Default Changed Exceeded Below Above InBetween Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 141 change behavior type change behavior interval change threshold boundary level change custom unit change custom time base change custom low change custom high change custom buffer apply threshold boundary changes 10 cancel threshold boundary changes Select choice gt 1 19 19 17 change threshold alarm level change changed alarm change exceeded alarm change below alarm change above alarm change inBetween alarm apply threshold alarm changes cancel threshold alarm changes return to previous page AANHADUPWNEFE OAATNHDOFPWNE Oo I
60. 3 10 PM Switch Information Network Config Upload Download SNMP License Admin Port Setting l Routing Extended Fabric Configure Trunk Information Extended Fabric Mode Enable Disable Port Number VCXLT Link Init Enabled Port Speed Long Distance Setting 0 O N2 LO Normal 1 O 2G LO Normal 2 O 2G LO Normal 3 O 2G LO Normal 4 Oo 2G LO Normal 5 O 2G LO Normal 6 O 2G LO Normal 7 r 2G LO Normal 8 O 2G LO Normal Apply Close Reset Refresh Switch Administration opened Fri Mar 26 2004 3 07 PM Long Distance Port Configuration For more detailed information on Extended Fabrics refer to the following publications and material e Brocade Fabric OS Features Guide Chapter 6 e Brocade Fabric OS Procedures Guide Chapter 6 entitled Distributed Fabrics Procedures e Fabric OS Reference Manual CLI commands used for managing extended fabrics e Brocade SilkWorm Design Deployment and Management Guide SAN DDM Version3 0 Section 5 16 entitled Brocade Extended Fabrics Planning under SAN Deployment and especially Appendix B Long Distance Technologies for Storage Area Networks Note that in terms of Extended Fabrics behavior the Brocade SAN Switch Module behaves similarly to the SilkWorm 3850 3250 as may be described in the above publications except where noted in this section Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCente
61. 59
62. 7 1 Figure 27 1 Advanced Performance Monitoring License brocadessm USERID gt licenseshow SzdyQeSzRh0ezRj Web license Zoning license Fabric license Fabric Watch license Performance Monitor license Although there are CLI commands to configure Advanced Performance Monitoring the WebTools interface provides some easy to use pre defined and customizable graphs to get you up and running quickly Start by clicking on the Performance Monitoring icon Perf from the Main Switch Explorer window of Web Tools for the Brocade SAN Switch Module Figure 27 2 Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 148 27 Figure 27 2 Advanced Performance Monitoring icon on Switch Explorer EE 210 x File Edit Yiew Favorites Tools Help e Bak gt Q A A Reach Favorites Meda E FME A El 3 Switch Explorer Microsoft Internet Explorer Address http 10 64 210 25 switchExplorer html v Go Links View by Name be Fabric IBM_2109_F32 IBM_3534_1RU IBM_3534_F08 brocadessm 4 Segmented Switches x Status Healthy Polled at 04 2304 11 11 AM Hame brocadessm Fabric OS version v4 24 Domain ID 1 Ethernet IP 10 64 210 25 Ethernet Mask 255 255 240 0 FCnet IP 0 0 0 0 FCnet Mask 0 0 0 0 Gateway IP 10 64 208 1 WWH 10 00 00 05 1 34 00 c7 Current Zone Config None Status Legend E Health
63. 9 J20 00 00 20 37 c8 3d 09 040700 N 20 07 00 60 69 c0 07 19 10 00 00 00 c9 2b 7F 90 20 00 00 00 c9 2b 7F 90 Refresh Print Close Refreshing Name Server Information done Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 150 This same data can be obtained from the Device Ports table in Fabric Manager Figure 27 6 Figure 27 6 Device Ports Table in Fabric Manager i Fabric Manager File Edit View Actions Topology Tools Help Address pittp 10 64 210 25 x Qaae DRS DF olr o ID Name ocade Device Po SAN Elements 7 Fiter Detail Device Po Devices Event Portgrid Ports Summary Switches Topology Domain ID Port Port ID Port Type Fabric Port WWAN Device Port WWAN Device Node WWAN Device Name FC4 Type COS Capability 4 0x01 4 0x01 010100 N 21 00 00 09 6b 36 40 14 20 00 00 09 6b 36 40 14 none 3 NS 1 4 0x01 2 0x02 010200 IN 21 00 00 09 6b 36 01 10 20 00 00 09 6b 36 01 10 none 3 NS 1 3 0x03 7 0x07 O307bc NL 21 00 00 20 37 c8 3d d9 20 00 00 20 37 c8 3d d9 FCP 3 NS 1 3 0x03 7 0x07 0307ba NL 21 00 00 20 37 c8 40 b4 20 00 00 20 37 c8 40 b4 IFCP 3 INS 1 4 0x04 7 0x07 040700 IN 10 00 00 00 c9 2b 71 90 20 00 00 00 c9 2b
64. Brocade SilkWorm product family The main features are indicated below The Brocade SAN Switch Module is pictured in Figures 1 2 and 1 3 e 1 or 2 Brocade SAN Switch Modules per IBM eServer BladeCenter Chassis to be placed in Bays 3 or 4 e 14 Internal Ports o Ports 1 to 14 o Connect to IBM eServer BladeCenter Server Blades o Fixed at 2Gbit sec and Server Blades log into the switch as F ports requires optional Fibre Channel Expansion Card installed on Server Blade e 2 External Ports o Ports 0 and 15 o Connect to existing Fibre Channel SAN switches or directly to Fibre Channel Storage devices o Auto negotiate link speed 1Gbit sec or 2Gbit sec o U port initialization E port F port or FL port o Can form a single 4Gbit sec ISL Trunk requires optional license e Two internal full duplex 100Mbps Ethernet interfaces terminated at a single MAC e Hot Code Activation e Frame filtering technology that enables Advanced Zoning and Advanced Performance Monitoring capabilities e Redundant Power and Cooling provided by IBM eServer BladeCenter chassis The Brocade SAN Switch Module can only be used inside the IBM eServer BladeCenter chassis The Brocade SAN Switch Module can be directly connected to external Fibre Channel storage devices or to an existing or new Fibre Channel SAN fabric To extend the SAN outside of the IBM eServer BladeCenter chassis refer to the IBM TotalStorage SAN Switch family of products For reference the entire IBM TotalStorage SA
65. Center Management Module Microsoft Internet Explorer E 0 x File Edit view Favorites Tools Help Back gt amp A A Aseach GFavortes meda 3 D 3 A E EACE i EEP server LAA IO Modules Bay Status Type MAC Address IP Address Pwr POST Status 1 Ethernet SM 00 05 5D 89 9E BC 10 64 210 22 On POST results available FF Module completed POS 2 No module present a A EEAS E FOST results available FF Module completed POS 4 No module present SM Switch Module CM Concentrator Module PM Pass thru Module z x yT EA gt Internet The Hardware VPD tab in the IO Modules section for the Brocade SAN Switch Module contains the switches Serial Number information See Figure 2 2 Figure 2 2 Brocade SAN Switch Module Hardware VPD 2 10 64 210 20 BladeCenter Management Module Microsoft Internet Explorer File Edit view Favorites Tools Help Back gt gt O A A search Favorites meda CH B Sb A ol Address E http 10 64 210 20 private main ssi DLNK na n a 2 1603 o1Roso7 59P6620 J1P2G34E412 n a 4 4903 DA Z 69 r 1 Management module 1 is not installed 2 WMN315838924 SLRM n a nfa 4 2703 73P9270 73P9273 KIDVY37413R _ ee ee F r Internet E 7 Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01
66. Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 158 Appendix A The Brocade SAN Switch Module can only be used inside the IBM eServer BladeCenter chassis The Brocade SAN Switch Module can be directly connected to external Fibre Channel storage devices or to an existing or new Fibre Channel SAN fabric To extend the SAN outside of the IBM eServer BladeCenter chassis refer to the IBM TotalStorage SAN Switch family of products For reference the entire IBM TotalStorage SAN Switch family names and their associated Brocade names are listed below Naming Conventions for the IBM TotalStorage SAN Switch family IBM Name 1 Gbit Family 3534 1RU SAN Managed Hub 2109 S08 SAN Switch 2109 S16 SAN Switch 2 Gbit Family 3534 F08 SAN Switch 2109 F16 SAN Switch 2109 F32 SAN Switch 2109 M12 SAN Switch new 2 Gbit Family 2005 H08 SAN Switch 2005 H16 SAN Switch 2109 M14 SAN Switch Brocade SAN Switch Module for IBM eServer BladeCenter Brocade Name Brocade SilkWorm 2010 20X0 Brocade SilkWorm 2400 Brocade SilkWorm 2800 Brocade SilkWorm 3200 Brocade SilkWorm 3800 Brocade SilkWorm 3900 Brocade SilkWorm 12000 Brocade SilkWorm 3250 Brocade SilkWorm 3850 Brocade SilkWorm 24000 Brocade SilkWorm 3016 Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 1
67. Go Links BladeCenter Management Module Power Restart Advanced Setup Update Fast POST Enabled gt External ports Disabled fia External management over all ports Enabled gt Preserve new IP configuration on all resets Enabled Cancel Internet y iz Pathinfo Pathinfo is new command available in Fabric OS 2 6 1 3 1 2 and 4 2 4 2 1 The pathinfo command provides traceroute functionality for the SAN It allows the paths to be discovered and viewed in a usable form The previous method of tracing paths used urouteshow and was complicated to analyze The fabric topology information is actually known at every switch however FSPF routing is known only locally there is no global end to end routing table In this context end to end refers to a source switch with connected E Ports and a destination switch also with connected E Ports and connected devices In general each switch in the fabric contains a table of routes that enable frames to be forwarded to adjoining switches If desired an end to end routing table can be constructed by logging into every switch and reading the local routing tables Pathinfo simplifies this task by providing a flexible command line and an interactive menu This interactive menu is displayed by default if no arguments are supplied PathInfo provides additional routing information including e Destination port state e Link statistics for every hop from source to destinatio
68. Guide Publication Number 53 0000561 01 24 Chapter 4 ISL Trunking This chapter provides the following information e Overview of ISL Trunking e ISL Trunking Architecture e Trunking Ports e Designing the Fabric for Optimal Use of ISL Trunking e Managing ISL Trunking Overview of ISL Trunking ISL Trunking is an optionally licensed product available for the Brocade SAN Switch Module and is licensed on a per switch basis The ISL Trunking feature is provided with the Fabric OS and can be activated by entering a license key available from the switch supplier It optimizes network performance by forming trunking groups that can distribute traffic across the shared bandwidth of all the ISLs inter switch links in the trunking group It is compatible with both short wavelength SWL and long wavelength LWL fiber optic cables and transceivers The Brocade SAN Switch Module ISL Trunking feature allows up to two ISL connections between itself and any other Brocade switch that has an ISL Trunking license installed This allows the two ISLs to merge logically into a single 4 Gbit sec link between the two switches Figure 4 1 This enables traffic to be routed through any available ISL in the group rather than being restricted to a specific potentially congested ISL This feature has significant advantages for this architecture ISL Trunking can result in more optimal throughput by avoiding congestion ISL Trunking distributes traffic dynam
69. IP Address configuration using the IBM eServer BladeCenter Management Module Step 3 Wait for the Management module to update the new information File Edit View Favorites Tools Help lol x Back gt O A A Qsearch GFavortes media GB D SH a gt iol Address http 10 64 210 20 private main ssi BladeCenter Management Module Bay 3 Fibre Sm Current IP Configuration Configuration method Static IP address 10 64 210 25 Subnet mask 255 255 240 0 Gateway address 10 64 208 1 New Static IP Configuration Status Enabled To change the IP configuration for this switch module fill in the following fields and click Save This will save and enable the new IP configuration IP address 10 64 210 25 255 255 240 0 10 64 208 1 Subnet mask Gateway address Advanced Management Go Links Save Bay 4 no module installed Internet Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 92 20 Figure 20 4 IP Address configuration using the IBM eServer BladeCenter Management Module Step 4 If you wish to maintain this configuration across resets of the Switch Module then ensure the Preserve new IP configuration on all resets is Enabled for this Switch Module 10 64 210 20 BladeCenter Management Module Microsoft Internet Explorer 3 loj x Eile Edt
70. ISL Trunking describes Trunking Design concepts for the Brocade SAN Switch Module Chapter 15 Architecting SANs with SilkWorm Switches describes device attachment strategies and platform specific features Chapter 16 Zoning Design guidelines for Advanced Zoning Chapter 17 Security Design guidelines for Advanced Security Secure Fabric OS Section II SAN Deployment Chapter 18 SAN Deployment Overview covers overview of deployment strategies Chapter 19 Planning introduces deployment planning concepts Chapter 20 Staging introduces deployment checklists for a Brocade SAN Chapter 21 Validation introduces SAN validation concepts Chapter 22 Maintenance introduces SAN maintenance concepts Section II SAN Management Chapter 23 SAN Management Overview introduces management concepts for a Brocade SAN Chapter 24 Brocade SAN Switch Module Management Tools introduces an overview of the management tools that can be employed to manage a SAN Chapter 25 SNMP introduces SNMP management concepts and examples Chapter 26 Fabric Watch introduces Fabric Watch management concepts and examples Chapter 27 Advanced Performance Monitoring introduces Advanced Performance Monitoring management concepts and examples What s New in This Guide This is the first release of this guide The Brocade Enterprise and Entry SAN Switch Modules for
71. List Enter drag port number 4 Domain_1 brocadessm a eon 0 Ext0 B 1 Bay1 o KKLK Enter drag SID number Hex H 0 2 Bay2 3 Bay3 Add Sid gt 4 Bay4 5 ae Enter drag DID number Hex er sa TT 8 Bay8 9 Bay 10 Bayt 0 lt Search 11 Bay11 12 Bay12 13 Bay13 14 Bay14 15 Ext1 5 j A Maminin TINK AINN AN L OE E E E E eee OK Cancel Java Applet Window Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 152 27 Step 7 Enter or drag and drop Port Number 1 7 Bay1 into the Enter drag port number box The box should have a 1 in it Enter or drag and drop the SID number by clicking on the d 010100 bullet to highlight it and then click on the Add Sid gt button in the middle to place the value of 010100 into the SID box Figure 27 10 Figure 27 10 Place SID 010100 into the SID box SID DID Performance Setup x Port or Sid Did Selection List Enter drag port number Domain_1 brocadessm a h en 0 Ext0 B 1 Bay1 mee 010100 Enter drag SID number Hex IL 2 Bay2 3 Bay3 010100 4 Bay4 5 Bay5 Enter drag DID number Hex pot saws 8 Bay8 9 Bay 10 Bayt 0 lt Search 11 Bay11 12 Bay12 13 Bay13 14 Bay14 15 Ext15 i t Maminin AAMA AINN TAN
72. Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 154 27 Figure 27 13 Second Advanced Performance Monitoring Filter applied to Port 2 SID DID Performance Setup Na x Port or Sid Did Selection List Enter drag port number Domain_1 brocadessm A Domain_2 IBM_2109_F32 S Domain_3 IBM_3534_1RU 0 Enter drag SID number Hex Add Sid gt 010200 Enter drag DID number Hex os07be NOOO eS WN Lpeeeeee Id 0307ba i EEDE lt Search Domain_4 IBM_3534_F08 Java Applet Window Step 11 Finally start I O on the Server Blades and view the throughput of the devices In this example Ports 0 and 15 are ISL Trunked and sending a shared load seen in the top graph The two Internal Server Blades are in the middle and bottom graphs Figure 27 14 Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 155 27 Figure 27 14 View Switch Throughput on a per port basis and the two specific SID DID Advanced Performance Monitoring Filters for each of the 2 Server Blades Performance Monitor brocadessm a 5 xj Actions Performance Graphs Switch Throughput Utilization Ref rate 30 Secs PO 36 9M 2G P1 37 0M 2G P2 36 8M 2 P3 0 0 2 P4 0 0 2 PS 0 0 2 P6 0 0 2 P7 0 0 26G P8 0 0 2 P9 0 0 2 P10 0 0 2 P11
73. N Switch family names and their associated Brocade names are listed in Appendix A Brocade Fabric OS 4 2 1 Brocade Fabric OS provides the core infrastructure growing businesses need to deploy scalable and robust Storage Area Networks SANs Fabric OS runs on the SilkWorm family of Fibre Channel switches It supports scalable SAN fabrics that interconnect thousands of devices while ensuring high performance data transfer among connected resources and servers Fabric OS easily manages both large switch fabrics and Fibre Channel Arbitrated Loop FC AL configurations Moreover Fabric OS is highly flexible making it easy for network administrators to add functionality and scale their SANs at the speed of business Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 2 Figure 1 2 Brocade SAN Switch Module for IBM eServer BladeCenter Figure 1 3 Rear View of the IBM eServer BladeCenter chassis showing two Brocade SAN Switch Modules on the far left in switch module Bays 3 and 4 Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 3 Chapter 2 Switch and Fabric Management This chapter explains the different methods used to manage the Brocade SAN Switch Module and a Brocade SAN The following information is discussed Overview of Switch
74. OS Reference lists all the commands available to the user and USERID level users Fabric OS CLI is a complete switch management tool for Brocade SAN Switch Modules providing the following advantages e Access to the full range of Fabric OS features based on which license keys you purchase e A full set of tools for configuring monitoring dynamic provisioning and daily management of every aspect of storage area networks e The ability to configure and manage a Brocade switch on multiple efficient levels e Fine grain management of every aspect of the switch and its features e Extensive diagnostic capabilities The command line interface can be accessed from the Advanced Management link of the IO Module Management Task through a Telnet connection See Figure 2 7 Figure 2 7 Advanced Management link of IO Module Management Task 10 64 210 20 BladeCenter Management Module Microsoft Internet Explorer loj x File Edit view Favorites Tools Help Ea Back gt O A A Gsearch GyFavortes Meda C4 B 3 A fl Address E http 10 64 210 20 private main ssi x Go Links gt BladeCenter Management Module server configuration on all resets to enabled Clearing of the configuration will be followed by a restart of the module Click the Restore Defaults button if you want to proceed Cancel Restore Defaults Send Ping Requests e You can test the internal path between the management module and the switch module b
75. P yes y no n no brocadessm USERID gt Brocade SilkWorm SW Trap Detail The Brocade SW TRAP information can be obtained by opening up the SW MIB file SW_v5_1 mib Please refer to the Brocade MIB Reference Manual for more detail 1 swFault An swFault trap is generated whenever the diagnostics detects a fault with the switch Variables swDiagResult swSsn The SSN of the switch which sent this trap swGroupName Specific to the SilkWorm 6400 swGroupTyp e Specific to the SilkWorm 6400 swGroupMemPos Specific to the SilkWorm 6400 2 swSensorScn A swSensorScn trap is generated whenever an environment sensor changes its operational state Variables swSensorStatus The current status of the sensor swSensorIndex The index of the sensor in the sensorTable swSensorType The type temperature fan etc of the sensor swSensorValue The reading from the sensor swSensorInfo The type and number of the sensor in text format swSsn The SSN of the switch which sent this trap swGroupName swGroup Type swGroupMemPos Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 132 25 3 swFCPortScn A swFCPortScn trap is generated whenever an FC_Port changes its operational state Variables swFCPortOpStatus The operational status of the Port offline online etc swF CPortIndex The index o
76. P configuration for this switch module fill in the following fields and click Save This will save and enable the new IP configuration IP address 10 64 210 25 Subnet mask 255 255 240 0 Gateway address 10 64 208 1 Update Advanced Management Bay 4 no module installed j ry Internet Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 9 2 From here the Advanced Management functions of the IBM eServer BladeCenter Management Module for the Brocade SAN Switch Module can be performed See Figure 2 5 Figure 2 5 Advanced Management features for the Brocade SAN Switch Module 10 64 210 20 BladeCenter Management Module Microsoft Internet Explorer E lol x File Edit view Favorites Tools Help e Back O A A Aseach GaFavorites meda lt 4 A GH A Address http 10 64 210 20 private mainssi dG Links 2 BladeCenter Management Module server Advanced Management for I O Module 3 Use the following links to jump down to different sections on this page POST Results Advanced Setup Restore Factory Defaults Send Ping Requests Start Telnet VVeb Session POST Results POST results available FF Module completed POST successfully Advanced Setup Fast POST Enabled External ports Enabled External management over all ports Enabled Preserve new IP configuration on all resets
77. Refresh Switch Administration opened Tue Apr 13 2004 5 09 PM Switch Administration closed Tue Apr 13 2004 5 09 PM Switch Administration opened Tue Apr 13 2004 5 11 PM View Trunking Information Figure 4 5 Topology view showing Trunked ports in Fabric Manager Fabric Manager File Edit View ctions Topology Tools Help Address fittp 110 64 210 25 1 Qlaer aR SGI DFvlivoAlar a AROA ID Name X a opolo EI SAN Elements Fier Detail Device Ports Devices Portoria Summary Switches EUT 0 BRD_12K_sw0_116 0 brocadessm 0 brocadessm 0 IBM_2109_F32 0 IBM_3534_1RU 0 IBM_3534_F08 AOF SwitchGroups lS PortGroups Show Hide Links Show Hide Nodes F 1 Gbitseco u PA 7 Device Groups D 4 steffier2 localhast 0 Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 29 For more detailed information on ISL Trunking refer to the following publications and material Brocade Fabric OS Features Guide Chapter 5 Brocade Advanced Web Tools Administrator s Guide Chapter 3 entitled Managing your Fabrics Switches and Ports Brocade Fabric OS Procedures Guide Chapter 9 entitled ISL Trunking Procedures Brocade Fabric Manager User s Guide Chapter 11 entitled ISL Checking Fabric OS Reference Manual CLI commands used for ma
78. Secure Fabric OS Planning for SAN security and change management is important Organizations understand that the data managed in their SAN environment is often highly sensitive and must have controlled access properly to ensure confidentiality integrity and availability A compromise in any of these areas could have unintended consequences resulting in the loss of proprietary information capital or other core business resources Proper SAN Security planning with Advanced Security Secure Fabric OS mitigates these risks by ensuring proper SAN security access controls are in place and enforced Because SAN security is by its own right a separate subject a comprehensive treatment will not be discussed in this document In order to be effective in the implementation of Advanced Security Secure Fabric OS there are two assumptions that are made for the duration of the discussion 1 Significant non SFOS security measures are already in place and 2 proper security practices exist within the IT infrastructure Advanced Security Secure Fabric OS Preparation Checklist 1 Obtain and read the Advanced Security Secure Fabric OS documentation 2 Be safe Backup switch configurations with configupload Prior information such as zoning will be wiped out when a switch or fabric is allowed to join an Advanced Security Secure Fabric OS enabled fabric 3 Verify PKI Objects Exist All Brocade SAN Switch Modules are shipped with the PKI objects from the
79. Security A Best Practices Guide Note that in terms of Advanced Security Secure Fabric OS behavior the Brocade SAN Switch Module behaves similarly to the SilkWorm 3850 3250 as may be described in the above publications except where noted in this section Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 46 Chapter 9 Brocade Interoperability Mode This chapter provides information on setting up a heterogeneous fabric that is a fabric that includes both Brocade switches and other manufacturer s switches For more detailed information on Interoperability refer to the Brocade Fabric OS Procedures Guide Chapter 12 entitled Using Interoperability Mode e Interoperability e Brocade Switch Requirements e McDATA Firmware Requirements e Supported Brocade Features e Unsupported Brocade Features e Configuration Recommendations e Configuration Restrictions e Pre Configuration Planning e Enabling Interoperability Mode e Disabling Interoperability Mode Interoperability Interoperability mode enables Brocade switches and other manufacturer switch fabrics to exchange interoperability parameters in such a way that both fabrics merge and form one single fabric with one principal switch and all unique domain IDs In a heterogeneous fabric several features are not available in order to provide maximum compatibility between swi
80. a dual fabric architecture a single resilient fabric may provide sufficient availability High Availability Clustering High availability HA clusters are used to support critical business applications They provide a redundant fail safe installation that can tolerate equipment software and or network failures and continue running with as little impact upon business as possible HA clusters have been in use for some time now However until the advent of Fibre Channel they were very limited in size and reliability This is because clusters require shared storage and sharing SCSI storage subsystems is difficult and unreliable In fact sharing a SCSI device between more than two initiators can be difficult due to SCSI cabling limitations and the SCSI support level for multiple initiators Clustering technology has therefore been greatly enhanced by the network architecture of SANs SANs provide ease of connectivity and the ability to interconnect an arbitrarily large number of devices SANs can support as few as two hosts in a failover configuration and can be expanded to support many to one configurations The primary advantages that a SAN affords a cluster are connectivity scalability and reliability Extended Distance Solutions SANs enable the ability to replicate data over distances of hundreds of meters to thousands of kilometers The data is replicated via mirroring at the host level or at the storage level Data can also be b
81. a set to be backed up e The need for improved high availability and disaster tolerance e The need to enhance storage resource management e Decreased total cost of ownership for storage Four popular SAN solution categories are Storage Consolidation LAN Free Backup High Availability and Extended Distance Solutions Each of these SAN solutions is generically described in the following sections and key attributes of each are discussed in terms of their affect on SAN design This chapter contains the following sections e Storage Consolidation e Backup e High Availability Clustering e Extended Distance Solutions Storage Consolidation Storage consolidation is a way of optimizing storage resource utilization It is often the result of migrating directly attached storage DAS and hosts to a SAN environment In a SAN it is no longer necessary to have a one to one correspondence between a host port and a storage port Instead many hosts can share a single storage port and a single host port can access many storage devices This immediately reduces cost on hosts because fewer HBAs are needed and on storage because fewer controllers are needed In addition savings can be accrued by reducing storage management power cooling and floor space costs However the greatest savings comes from improved utilization of free space on enterprise storage subsystems With the lowering cost of Fibre Channel HBAs and switch infrastructure the storage conso
82. abric and fabric troubleshooting A rich set of management interfaces exists for the SilkWorm family of switches Effectively integrating a particular management interface such as Fabric Manager or Fabric Watch into the enterprise management system capacity planning and SAN management with SNMP are just a few examples of topics addressed in the section on SAN Management This guide is targeted for use by storage administrators SAN administrators system administrators SAN architects systems engineers and SAN operators that are involved with the design deployment and management of SANs Background information and supporting information for a particular topic are kept to a minimum and as appropriate the reader is referred to supporting documentation The reader is expected to have working experience with Brocade products General computer system level troubleshooting skills are always important when configuring sophisticated enterprise solutions System administration or storage administration experience is also helpful in comprehending this document Guidelines are provided throughout the document Guidelines are recommendations for consideration The adoption of these guidelines is a function of the user s ability to interpret and correlate relevant SAN information and make decisions based upon their operational policies and SAN requirements Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and M
83. acked up to a remote site Ina Business Continuance example disk mirroring can occur by day and tape vaulting can occur over IP by night Should Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 55 10 the primary site become disabled the data located at the remote site is accessible immediately so that critical applications can continue operation Some business continuance implementations deploy standby hosts that remain idle until needed Other sites utilize the idle equipment for testing and development purposes until that equipment is needed for production purposes A key element to this strategy is having the hosts boot from the SAN and the existence of a boot image of the primary site hosts located at the remote site Underlying connections to a remote site of up to 100 KM can be enabled by use of Fibre Channel technology such as long wave length lasers LWL extended long wave length lasers ELWL or wave division multiplexing WDM Use of wide area network technology such as IP internet protocol Fibre Channel bridges or SONET Fibre Channel bridges enables the remote site to be separated from the primary site by thousands of kilometers Performance over long distance links can vary for multiple reasons The number of buffer to buffer credits determines the number of Fibre Channel frames that a switch can transmit on a link at one ti
84. ade branded switches is for the IOD option to be off this enables fast rerouting after a fabric topology change Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 106 20 Follow the guidance of your SAN support vendor for the configuration of this t Note setting Viewing and changing this parameter 1 Enter the iodShow command to view the current IOD setting One of the following messages appears e IOD is set Enables the in order delivery IOD option Enforces in order delivery of frames during a fabric topology change e OD is not set Turns off the in order delivery IOD option This command may cause out of order delivery of frames during fabric topology changes 2 Enter the iodSet command to enable In Order Delivery 3 Enter the iodReset command to disable In Order Delivery Example switch admin gt iodshow IOD is set switch admin gt configshow route d route delayReroute 1 switch admin gt iodreset Committing configuration done switch admin gt iodshow IOD is not set switch admin gt configshow route d route delayReroute 0 Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 107 20 Profiling The SAN Once the SAN fabric has been built and zoning has been configured it is good prac
85. agement module an important part of the switch management and administration process e Provides the Hardware and Firmware Vital Product Data VPD of the switch module for asset management and support e Can power on power off or restart the switch module and determine if extended diagnostics are run on reboot e Allows the user to configure or manage the IP Address configuration e Displays Power on Self Test POST results e Can program whether Fast POST is enabled or disabled e Can program whether the External ports 0 and 15 are persistently enabled or disabled after a reboot or power cycle If the ports are disabled the Port diagnostics LEDs will flash slowly in amber e Can program whether External Management is enabled on the external ports e Can program whether the IP Address configuration changes are preserved on the switch module e Restore the switch configuration to factory defaults e Send Ping Requests to ensure Ethernet connectivity through the management module e Provide a launch location for Telnet and Web Tools sessions to the switch module Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 7 The status of the Brocade SAN Switch Module can be found in the IO Modules section of System Status See Figure 2 1 Figure 2 1 IBM eServer BladeCenter Management Module System Status of IO Modules E 10 64 210 20 Blade
86. ages 10 564 210 25 _ brocadessm v4 2 1 0 Fri Apr 16 16 44 10 2004 Firmwaredownload has started a B E Fabrics B Eg brocadessm Eji brocadessm 0 BM _2109_F32 0 IBM_3534_1RU 0 IBM_3534_FO8 B BRD_12K_sw0_116 SwitchGroups Help Using Fabric OS Troubleshooting Tools This section will just provide an overview of Brocade Fabric OS troubleshooting tools Some examples will be given to demonstrate the practical use For details and examples for setting up persistent error logs and the uses of portlogdump please see the Brocade Fabric OS Procedures Guide Support Show Command Groups Supportshow is an invaluable troubleshooting tool It essentially provides a one time event capture of the entire system at the point in time the command is run It includes many of the typical show commands that experienced Fabric OS users are familiar with and much more output so that the Support organization can resolve any issue without having to go back for more data Supportshow can be run with any combination of 11 command groups This makes Supportshow more flexible and easier to capture the desired information If required for support reasons supportshow can always be reconfigured to display more information As an example of a command group here are the commands for the FC Fabric command group number 4 fabricShow islShow trunkShow topologyShow faShow qlShow cfgShow fabStatsShow fabLogDump This section w
87. all of these hosts tried to simultaneously use the ISLs at full speed in a sustained manner even if the hosts were accessing different storage devices each would receive only about one seventh of the potential bandwidth available Figure 14 2 shows the ISL Over subscription ratio for the IBM eServer BladeCenter architecture Figure 14 2 IBM eServer BladeCenter over subscription ratio IBM 7 1 ISL Over subscription ratio BladeCenter embedded Brocade Switch Module Maximum 14 Host Blades 14 x 2 Gbit sec maximum 2 ISLs per switch module 2x 2Gbit sec Recommended ISL Over subscription Ratios ISL over subscription ratios apply in practice to Core Edge fabrics The calculations for ISL over subscription ratios for a Core Edge fabric are simple and straightforward while these same calculations become more complex and less pertinent for other topologies such as a ring topology ISL over subscription ratios principally apply to edge switches as the role of a core switch is to connect other switches Connecting devices to a core switch is supported and makes sense for particular scenarios such as when there are excess ports available on the core switch or for performance Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 68 14 purposes When devices are connected to the core switch the number of ISLs trunks is usually equal
88. anagement Guide Publication Number 53 0000561 01 52 SAN Design This section discusses fundamental SAN Design amp Architecture concepts and associated guidelines for developing an effective SAN design A background on SAN design solutions topologies availability scalability performance and Trunking is provided as a foundation for the more advanced topics discussed in Chapter 14 Architecting SANs With SilkWorm Switches This section contains the following chapters e SAN Solutions e SAN Availability e SAN Scalability e SAN Performance e ISL Trunking e Architecting SANs With SilkWorm Switches e Zoning Design e Security Design Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 53 Chapter 10 SAN Solutions While many SAN users begin their SAN experience with one particular SAN solution the SAN quickly becomes the basis for many other applications For example a company may start out with SAN based backup and quickly integrate storage consolidation and clustering into the existing SAN foundation In that respect a SAN decision is a strategic one and should receive an appropriate level of attention The adoption of SANs is being driven by a variety of objectives Some examples are e The need for more efficient usage of enterprise storage arrays e Decreasing size of backup restore windows e Increasing size of dat
89. anagement Guide Publication Number 53 0000561 01 iv Extended PADIS vicidcckccictiotvansciesvedivsarseeidd atten tuierein otis wenibe 39 Remote SWIC scree hase dea a a anna hace aga gatas 42 Brocade Advanced Security Secure Fabric OS 43 Overview of Advanced Security Secure Fabric OS 43 Advanced Security Secure Fabric OS Architecture 44 Managing Advanced Security Secure Fabric OS 0 45 Brocade Interoperability Mode c cccccceseeeeeeeneeneesenes 47 Section I antore en scale a A RRR I R 52 SAN DESIGN morien sites A e E EA st 53 SAN SOMINOMS rcn a aa aaa aE E Ea 54 SAN Ayaila bility n E AAA E 57 SAN Scalability Asrai aie EE EEEE AE ETES 62 SAN Performance isis Zee areds oes aN tasitedeuseevec da teneceesbe ees devsdudeentons 65 bs TRG a glare lene eremen ee E T arene 67 Architecting SANs With SilkWorm Switches cceeeeeeees 72 Device Attachment Strategies ccccccseeeeeeeeeeeseeeeeeeeeeeeeeeees 74 ZOMG ADIE T a seston craic siesta le sea detacea Nia 76 Sec rity DSSIOM nsa e a e sa un N a i 78 SAN Deployment mesian enaena AAEE E EERROR 79 SAN Deployment Overview ccccccccccsssseeeeeseesseeeesssesseeeeees 80 PUIG scecrriece e e ER E E 81 Ste a a Lo Ae E E EE E E E 89 Validato Ns eee A a crete cas ai a e ea 109 MaINON ANCE anenee a E aE Ea 112 SAN Managemen xsscc ceived ase adn tie eee 121 SAN Management Overview
90. anager is installed onto a group of switches in a fabric This can be found in the Actions Menu under Set Time Figure 20 11 Command Line Set Date and Time brocadessm USERID gt date 0331160204 Sunday March 22 16 02 00 UTC 2004 brocadessm USERID gt Figure 20 12 Fabric Manager Set Date and Time rme x Set time for fabric brocadessm Wednesday April 21 2004 4 01 21 PM PDT E OK Cancel 6 Set the Switch Name Use switchname to set the name of the switch Setting a name helps to quickly identify a switch and its role within the fabric Figure 20 13 below sets the name of the switch to be FabA C1 117 since it has been designated to be the first core switch in the fabric and the last octet of its IP Address is 117 The default switch name for all Brocade SAN Switch Modules is brocadessm Note the prompt changes to reflect the new switch name You can also perform this task from Fabric Manager by right clicking on the switch icon and choosing the rename option from the drop down list Figure 20 13 Set Switch Name brocadessm USERID gt switchname FabA C1 117 FabA C1 117 USERID gt Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 97 20 7 Set the Domain ID Number While it is not necessary it is highly recommended to set a domain ID If no domain is set the switch wil
91. and a Brocade switch is changed via fabric reconfiguration on the next and subsequent fabric reconfiguration the Brocade switch will try to use the new domain ID from the flash while McDATA will try to use it s old domain ID from the flash This situation may cause a domain ID overlap to occur during multiple fabric reconfigurations Domain ID overlap is not supported for Brocade McDATA interoperability Between Brocade switches you can connect more than one ISL when in Interoperability mode To determine whether or not a mixed vendor SAN is supported it will be necessary to work with your switch provider to determine if your SAN design is valid Important variables that determine the supportability of a particular mixed vendor SAN include the number of switches version of Fabric OS the topology number of ISLs number of connected devices and hop count Zoning Restrictions Zoning has the following restrictions in interoperable fabrics Zoning must be consistently and entirely managed from a single selected switch type for the lifetime of the Effective Configuration Attempting to manage from both Brocade and McDATA switches may result in the loss of zoning information The McDATA switch forwards only the Active Zone Set The Full Zone Set information is not forwarded to the neighboring switches Only zoning by port WWN is allowed Zone members specified by node WWN will be ignored Zone configurations that use either domain port
92. and the actual reboot or code activation operation occurs Control of the switch is passed seamlessly to the standby image As a result there is no effect whatsoever on the flow of data between logged in hosts and storage devices Read and write tasks proceed with no delay or interruption whatsoever No frames are dropped No devices that are currently logged in have to re login Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 5 User Access Levels There are four levels of user access to the Brocade SAN Switch Module e root e factory e USERID e user Not all commands are available to all levels Commands are assigned a minimum login level to execute Root level has access to all commands user level has limited access to commands USERID replaces the admin account that exists on all other Brocade SilkWorm switches and has the same level of administration Note Because the USERID account provides access to all the commands needed to manage and configure a switch or fabric it is the recommended login level Switch modules can be accessed simultaneously from different connections for example through the CLI and Advanced Web Tools If this happens changes from one connection might not be updated to the other and some changes might be lost When connecting with simultaneous multiple connections make sure that you do not overwrit
93. architecture see Figure 15 1 The logic behind this approach is that should a single Core fail the second fabric in the SAN can maintain operations In a resilient Core Edge fabric two or more switches reside in the center of the fabric the Core and interconnect a number of other switches the edge Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 72 15 Figure 15 1 Single Core Edge fabric vs Dual Core Edge fabric IBM BladeCenter A IBM TotalStorage M14 Director IBM BladeCenter IBM TotalStorage M14 Director Switches that reside in the middle of the fabric are referred to as core switches The switches that are interconnected by the core switches are referred to as edge switches The simple form of the core edge fabric has two or more core elements each of which consists of a single switch In a simple core the core switches do not connect with each other Edge switches in a core edge fabric also do not connect to each other They only connect to the core switch Devices such as hosts and storage are attached to free ports on the edge switches These ports are referred to as edge ports or user ports Free ports on the core switches should usually be reserved for additional edge switches when using 16 port switches and can connect SAN devices for higher port count switches The scalability of a core edge fabric is r
94. at are not possible through the Web Tools interface The Fabric OS Reference Manual has a comprehensive list In the Port Setting tab of the Web Tools Admin interface there are several columns which can be modified If using the CLI portCfgShow shows the current settings for the ports Figure 20 18 and 20 19 Figure 20 18 Port Name information is found in the WebTools Port Setting Tab Z Switch Admin Microsoft Internet Explorer L SwitchName brocadessm DomainID 1 WAN 10 00 00 05 1e 34 00 c7 Tue Mar 23 2004 4 20 PM Switch Information Network Config Upload Download SNMP License Admin Port Setting Routing l Extended Fabric Configure Persistent Current Change Disable Speed Speed 0 Vv O Vv No_Module N2 Negotiate Y JExt0 1 Cc Vv O Online 2G 2G jBay1 2 r Vv O Online 2G 2G Bay2 3 O Vv O No_Light 2G 2G Y Bay3 4 O Vv O No_Light 2G 2G Bay4 5 O Vv Cr No_Light 2G 2G Y Bay5 6 O Vv O No_Light 2G 2G Bay6 7 m Vv Cr No_Light 2G 2G Bay 8 cr Vv E No_Light 2G 2G jBays 9 O Vv O No_Light 2G 2G jBay9 10 O Vv O No_Light 2G 2G Bay10 11 r Vv O No_Light 2G 2G jBay11 j i Switch Adrinistration opened Tue Mar 23 2004 4 16 PM Configure Port Setting parameters Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01
95. aving an ISL over subscription ratio that lies in the mean of performance and available host connections In summary the ISL attachment strategies for the Brocade SAN Switch Module can be characterized into the following four choices e Ifyou are connecting to a single switch ISL Trunk for performance Figure 14 3 e If you are connecting to a single bladed Director you have two options Figure 14 4 o Split the ISLs across blades for availability o Trunk the ISLs for performance e If you are attaching to a fabric with a single core switch ISL Trunk for performance Figure 14 5 If you are attaching to a fabric with dual core switches you have two options Figure 14 6 o Split the ISLs for availability o Trunk the ISLs for performance Figure 14 3 Connecting to a single switch IBM BladeCenter Brocade Brocade Switch Module Switch Module Trunk the Trunk the ISLs for ISLs for Performance f gt F i l Performance Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 69 14 Figure 14 4 Connecting to a single Director IBM BladeCenter Brocade Brocade Switch Module Switch Module Option 2 Option 1 Trunk the Split the ISL s ISLs for across blades Performance for availability IBM TotalStorage IBM M12 or M14 TotalStorage Director M12 or M14 Director Figure 14 5 Connecting to a fabric with a si
96. be inspected for compatibility Zones should be inspected to ensure that they meet the zone criteria and restrictions Refer to Zoning Restrictions above Remove or disable any unsupported optional features Disable the Platform Management functions using the msplmgmtdeactivate command Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 50 Enabling Interoperability Mode To enable interoperability mode 1 Verify that you have implemented all the Brocade prerequisites necessary to enable interoperability mode on the fabric Refer to Configuration Restrictions and Pre Configuration Planning 2 Connect to the switch as the administrator Disable the first switch using the switchdisable command gt At the command line enter the interopmode 1 command to enable interoperability This command resets a number of parameters and enables interactive mode Reboot the switch after changing the interoperability mode Repeat this procedure on all Brocade switches in the fabric Other manufacturers switches may require the execution of a similar command to enable interoperability onan Once you have enabled inter operability mode on the Brocade switches and other manufacturer s switches you can cable the other manufacturers switches into the Brocade fabric one at a time Example switch admin gt switchdisable switch admin gt
97. buting to MARGINAL status Down ISLStatus contributing to DOWN status 0 Down ISLStatus contributing to MARGINAL status No change brocadessm USERID gt Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 146 26 Figure 26 10 shows two sample outputs of switchstatusshow The first shows a Healthy condition the second reveals error conditions have triggered for several contributors Figure 26 10 Switch Status Policy messages brocadessm USERID gt switchstatusshow The overall switch status is HEALTHY OK brocadessm USERID gt brocadessm USERID gt switchstatusshow The overall switch status is Marginal Warning Contributing factors Switch Offline triggered the Marginal Warning status brocadessm USERID gt Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 147 Chapter 27 Advanced Performance Monitoring Brocade Advanced Performance Monitoring is a tool which helps SAN managers measure the efficiency of their SAN resources Trends and patterns can be read using charts and graphs The overall throughput performance of an application is entirely dependent on the configuration of the host and storage hardware being utilized A SAN configuration providing connectivity to host and storag
98. ches tape solutions sometimes do not recover as easily This is a known issue in the industry and something being addressed with the emergence and adoption of the FC TAPE standard e The availability of tape drives is usually not as critical as that of disk arrays e The performance requirements of tape drives typically involve the streaming of large blocks of I O with bandwidth requirements of 30 60 Mbyte sec Non SAN based backups take the form of direct attach tape drives or backup over IP networks IP backups contend with the normal day to day traffic already on the Local Area Network LAN Using direct attach tape on each host is costly because of the number of tape devices tape management and increased infrastructure cost for floor space power cooling etc High speed SAN enabled backups reduce backup and restore windows and can enable disaster tolerance by locating libraries at remote sites SAN based backup improves on traditional backup by enabling the sharing of fewer larger tape libraries and by minimizing or eliminating the performance issues associated with traditional backup architectures A disruption in a backup SAN is usually not as critical as a disruption in a storage SAN Mission critical applications require continuous access to storage while a tape backup normally can be restarted without end users seeing the effect Therefore a SAN architecture solely used for backups may not require the highest availability enabled by
99. ction is to provide guidance on preparing Brocade Fabric OS for production in a multi switch SAN fabric There are two major steps First prepare each switch for attaching to the corporate LAN infrastructure and joining a fabric The second step is to do the fabric wide configuration this being primarily zoning Each step will have a separate checklist As with the rest of the document this section first provides bare bones guidelines Examples will be used throughout Some guidelines will be considered optional and will be noted as such All commands discussed in this section are available to the admin user Brocade Switch Preparation Checklist Gather Planning Information Switch Spreadsheet Templates are provided in the Brocade SilkWorm Design Deployment and Management Guide SAN DDM Version 3 0 1 Set IP Address 2 Login and Change Passwords 3 Check Fabric OS version 4 Check Switch Status 5 Set the Date and Time a Optional Set the Time Zone Set the Switch Name 7 Set the Domain ID Number a Optional Set the Core PID parameter on Fabric OS 2 x or 3 x switches if required b Optional Set the Extended Edge PID Format if required c Optional Set up a preferred principal switch in the fabric Fabric OS 4 1 and above only 8 Add Fabric OS licenses 9 Optional Modify Port Configurations if required a Optional Setup ports for Extended Fabrics if required 10 Optional Name devices with portName 11 Set Telnet Session Time
100. ctivity The Fibre Channel Expansion Card option for the IBM eServer BladeCenter Server Blades include Two Fibre Channel ports One port connects to a switch module located in bay three The second port connects to a switch module located in bay four Some of the other Server Blade features for the IBM eServer BladeCenter include 1 4 Processors depending on Server Blade Up to 16GB Memory depending on Server Blade Optional Hot swap SCSI or Fixed IDE Drives Up to 4 Integrated Gigabit Ethernet controllers depending on Server Blade Up to 2 Expansion Card connections depending on Server Blade Figure 1 1 Front view of the IBM eServer BladeCenter showing the Standard chassis with 14 Server Blades Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 1 Brocade SAN Switch Module SilkWorm 3016 Throughout this document the term Brocade SAN Switch Module Brocade Switch Module SilkWorm 3016 or switch module refers to the Brocade Enterprise SAN Switch Module for IBM eServer BladeCenter or the Brocade Entry SAN Switch Module for IBM eServer BladeCenter The Brocade SAN Switch Module is a 16 port embedded switch It supports link speeds up to 2 Gbit sec The Brocade SAN Switch Module is based on the Brocade Fabric Operating System Fabric OS version 4 x and is compatible with the entire
101. cumentation Library gt Technical How To Guides gt There are several How To Guides there The latest DDM Guide is the document entitled Brocade SilkWorm Design Deployment and Management Guide SAN DDM Version 3 0 The Guides can also be obtained through the Brocade Connect Website http www brocadeconnect com Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 X The following related Whitepapers are provided on the Brocade web site and are an excellent resource for additional information http www brocade com san white_papers jsp e Building Enterprise SANs Through Intelligent Networking e Zoning Implementation Strategies For Brocade San Fabrics The following related publications are provided on the Brocade Partner Web Site or the IBM TotalStorage SAN Switch Support Web Site and are an excellent resource for additional information e SAN Migration Guide e LAN Guidelines For Brocade SilkWorm Switches e SAN Security A Best Practices Guide For practical discussions about SAN design implementation and maintenance you can obtain Building SANs with Brocade Fabric Switches through http Avww amazon com For additional Brocade documentation visit the Brocade SAN Info Center and click the Resource Library location http www brocade com Release notes are bundled with the Fabric OS Brocade Enterp
102. d swF wLabel The label of the threshold swFwLastEventVal The last event value of the threshold swF wLastEventTime The last event time of the threshold swF wLastEvent The last event type of the threshold swF wLastState The last event state of the threshold swSsn The SSN of the switch that sent the trap swGroupName swGroupType swGroupMemPos 6 swTrackChangesTrap This trap gets sent when track changes is set to ON and is configured to send SNMP traps A trap is sent whenever somebody logs in to the switch or logs out of the switch or when there are switch configuration changes Variables swTrackChangesInfo The description of the track changes event config change login logout etc swSsn The SSN of the switch that sent the trap swGroupName swGroupType swGroupMemPos Refer to the following table for the six traps defined in the SW MIB when they occur and how to configure the trap if possible SN Name Specific When Configure swFault 1 During boot if diagnostics fail Always on swSensorScn 2 Sensor state change Always on swPortSsn 3 Port changes state Always on swEventTrap 4 Switch event Command agtCfgSet Variable swEventLevel swFabricWatch 5 Threshold reached Command fwConfigure swTrackChanges 6 Login Logout or Config change Command trackChangesSet Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publicat
103. d at the port with end to end monitor installed the frame SID is the same as SourceID and the frame DID is the same as DestID Both RX_COUNT and CRC_COUNT are updated accordingly e For frames transmitted from the port with end to end monitor installed the frame DID is the same as SourceID and the frame SID is the same as DestID TX_COUNT and CRC_COUNT are updated accordingly Where to Add End to End Monitors Depending on the application any port along the routing path can be selected for such monitoring Figure 5 1 Setting End to End Monitors on a Port shows two devices e Host A connected to domain 0x05 switch area port 12 area ID 0x0c AL_PA 0x00 on Switch X e Device B connected to domain 0x11 switch area port 30 area ID Oxle AL_PA Oxef on Switch Y Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 32 Figure 5 1 Setting End to End Monitors on a Port Host A Switch X Switch Y Device B gt gt gt SID lam Fi DID 0x050c00 0x111eef Monitor 0 Monitor 1 Domain 5 switch area Port 12 Domain 11 switch area Port 30 AL_PA 0x00 AL_PA Oxef To monitor the traffic from Host A to Device B add a monitor to port 12 specifying 0x050c00 as the SID and 0x11leefas the DID To monitor the traffic from Device B to Host A add a monitor to port 30 specifying 0x11 leef as the SID and 0x050c00 as the DID Monitor 0
104. d boundary changes 10 cancel threshold boundary changes Select choice gt 1 19 19 15 Errlog 1 SnmpTrap 2 PortLoghock 4 RapiTrap 8 EmailAlert 16 change threshold alarm level change changed alarm change exceeded alarm change below alarm change above alarm change inBetween alarm apply threshold alarm changes cancel threshold alarm changes return to previous page AAYNAUPWNHE OMNIA NPWNHEHE wo Valid alarm matrix is 31 Enter above alarm matrix gt 0 31 0 3 Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 140 Index ThresholdName BehaviorType BehaviorInt 0 eportRXPerf000 Triggered 1 15 eportRxXPerf015 Triggered i Threshold boundary level is set at Default Default Custom Unit KB s KB s Time base Low 120000 120000 High 220000 220000 BufSize 0 0 Threshold alarm level is set at Default Errlog 1 SnmpTrap 2 PortLoghock 4 RapiTrap 8 EmailAlert 16 Valid alarm matrix is 31 Default Changed Exceeded Below Above InBetween change behavior type change behavior interval change threshold boundary level change custom unit change custom time base change custom low change custom high change custom buffer apply threshold boundary changes 0 cancel threshold boundary changes Select choice gt 1 19 19 11 Default 2 custom Enter alarm level type gt 1
105. d text e Identifies keywords operands e Identifies text to enter at the GUI or CLI e Provides emphasis e Identifies variables italic text e Identifies paths and internet addresses e Identifies document titles e Identifies CLI output code text i e Identifies syntax examples Notes Cautions and Warnings The following notices appear in this document Note A note provides a tip emphasizes important information or provides a reference to related information o Guidelines are recommendations for consideration The adoption of these guidelines is a Guideline function of the user s ability to interpret and correlate relevant SAN information and make decisions based upon their organization and SAN requirements A caution alerts you to potential damage to hardware firmware software or data Cautions Caution indicate that a particular action or type of connection is not recommended as it may cause failure of the switch or fabric Warning A warning alerts you to potential danger to personnel Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 ix Additional Information This section lists additional Brocade IBM and industry specific documentation that you might find helpful Brocade Resources The following related documentation is provided on the Brocade SAN Switch Module for IBM eServer BladeCenter Documentat
106. devices in an enterprise and have extended their support to manage Brocade SANs These SNMP Management stations can be run separately by the SAN administrator so that they can get alerts trend performance and capture details of error status on switches separately Within the SNMP model a manageable network consists of one or more management stations and a collection of agent systems that are also known as network elements The manager communicates with the agent using SNMP Protocol Brocade currently supports SNMP version 1 SNMPv1 The fabric switches must be properly configured either from the command line interface or using Brocade Web Tools Web Tool can also be accessed from the Fabric Manager Admin menu A switch configuration setup consists of e Brocade SNMP Configuration e Brocade SNMP Agent Setup e Brocade SNMP Trap Setup e Brocade Track Changes Setup Brocade SNMP Configuration To configure the SNMP agent software on a Brocade switch no special license key is needed It is included in the base functionality of the switch The commands used to configure the SNMP Agent are agtcfgShow agtcfgSet and agtcfgDefault Once the SNMP agent software is configured SNMP information can be automatically sent to an SNMP management station The SNMP trap software on a Brocade switch also requires no special license The commands used to configure whether SNMP traps are sent out is snmpmibcapset This feature allows the us
107. e 5 Gather the documentation and readme notes for the firmware release 6 Schedule downtime for single fabric updates 7 For dual fabrics update one fabric at a time Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 87 19 LAN Planning There are a few guidelines to consider when attaching the fabric to the corporate LAN infrastructure In general it is highly recommended to configure a separate VLAN for each fabric If at all possible avoid the use of proxy servers from the SAN management stations outside the local subnet In fact it is recommended to use a management station on the same VLAN For detailed guidelines on how to connect Brocade switches to the corporate network please refer to the technical white paper titled LAN Guidelines for Brocade SilkWorm Switches When taking the LAN Guidelines for Brocade SilkWorm Switches into account it is important to note that all IP Ethernet communication to the Brocade SAN Switch Module is actually forwarded from the IBM eServer BladeCenter Management Module Note Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 88 Chapter 20 Staging Once the plan is complete and the resources readied the new SAN can be built and prepared for production Staging the SAN is more than uncrating
108. e the work of another connection In Fabric OS v4 x each user access level can have the number of simultaneous logins shown User Name Maximum Number of Simultaneous Sessions root 4 factory 4 USERID 2 user 4 Switch Licensing The Brocade SAN Switch Module will be shipped in two versions The first version will be called the Brocade Entry SAN Switch Module for IBM eServer BladeCenter This switch will be shipped with the following licenses e Advanced Web Tools Web e Advanced Zoning Zoning e Fabric Watch Fabric Watch e Two Domain Fabric License 2 Domain Fabric The Two Domain license limits the size of the fabric that this Brocade Entry SAN Switch Module can be a member of In this case the number is 2 The license can be upgraded using the Brocade Entry Switch Full SAN Upgrade license This license may be purchased from your switch supplier Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 6 The second version will be called the Brocade Enterprise SAN Switch Module for IBM eServer BladeCenter This switch will be shipped with the following licenses e Advanced Web Tools Web e Advanced Zoning Zoning e Fabric Watch Fabric Watch e Full Fabric License Fabric With the Full Fabric license the theoretical size limit of the fabric that this Brocade SAN Switch Module can be a member of is 239
109. e Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Document Title Publication Summary of Publication Number Changes Date Brocade Enterprise and Entry SAN Switch Modules for 53 0000561 01 Tirithe May 2004 IBM eServer BladeCenter Design Deployment and f this ouid Management Guide Version 1 0 release of this guide Contents About This Document pate Behe oe eh ee ae ti vi SECUN oere E E E E E E A 1 PIOQUGE INTOOQUGCHON feces es eien oe ee ee ee es 1 IBM eServer BladeCenter Overview cccccccceccsseseeeeeseesseeees 1 Brocade SAN Switch Module SilkWorm 3016 cceee 2 Brocade Fabric OS 4 2 1 isccsscanisnrrieressuecerbentadaantedusnncemueesnoeess 2 Switch and Fabric MAnaGe Ment ccccceccesesnseeeeeeseenenees 4 Overview of Switch Management ccccceseeeeeeeeeeeeeeeeeeeeeeees 4 Fabric Management Recommendations cc seeeeeeeeeeeeeeeees 5 PIG Ae AIRY raaa Ds sse ETA Oi 5 User ACCESS LGVElS fied aes lecaccdate ed terenn rt Nu eceredaesaa ears 6 SWATCH LICENSING ia Nh renee cecal cai du sas er O E A 6 IBM eServer BladeCenter Management Module 00 7 Fabric OS Command Line Interface ccceceeeeeeeeeeeeeeeeeeees 11 Advanced Web Tools wcscnscccenceotiasedacennercash iepeeea aca Testa ws 13 Fabre Watchers e a sshuateenaeee Musas 15 Fabric Manager geenenennereee ener ere reee aa eee ne
110. e Fabric Watch button from the Main WebTools switch view to enter the Fabric Watch configuration applet Figure 26 6 Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 142 26 Figure 26 6 Example using fwConfigure to change the Threshold Alarm Levels to 3 Z Fabric Watch for brocadessm Microsoft Internet Explorer E oj x Fabric Watch Alarm Notification Threshold Configuration Email Configuration E Fabric Fabric Switch Select Area Rx Performance Environment SFP Security Switch Availabilty Area Configuration Element Configuration Configuration Report Ports Port eo Han Boundary Settings Default Settings in Parentheses FFL Copper Port Unit KB s KB s TimeBase none none so FIFL Optical Port Performance High 220000 220000 Low 120000 120000 AL PA BufferSize 0 fo End to End Filter Based Select Boundary Level Default 7 Alarm Setting Boundary Alarm Notification Mechanisms Default Mechanisms in Parentheses P ERRLOG P SUMP F PORTLOG P EMAIL 7 ERRLOG P SNMP TZ PORTLOG 7 EMAIL JV ERRLOG JV SNMP J PORTLOG 7 EMAIL JV ERRLOG JV SNMP PORTLOG 7 EMAIL I Inbetween ERRLOG P shme JT PORTLOG P EMAIL Select Alarm Level custom v I Save Configuration to Switch
111. e SW Traps It is also used in conjunction with the SilkWorm 6400 integrated fabrics to provide the detailed group information for a particular trap The SilkWorm 6400 is no longer sold For a description of the Brocade MIB files naming conventions and loading instructions refer to the Brocade MIB Reference Manual The Brocade SilkWorm SW MIB for the Fabric OS 4 2 1 release is named SW_v5_1 mib and can be found in the un compressed firmware directory called mibs The Brocade SW TRAP information is located within this MIB The Fibre Alliance SNMP MIB and Traps can be found in the FA_v3_0 mib file in the same location All of the Brocade MIBs for this release are located in the mibs directory of the uncompressed v4 2 1 firmware files To configure the Trap options use the snmpmibcapset command This command is not available from the WebTools interface 1 Figure 25 5 shows the default values for the snmpmibcapset telnet command Figure 25 5 default snmpmibcapset brocadessm USERID gt snmpmibcapset The SNMP Mib Trap Capability has been set to support FE MIB SW MIB FA MIB FICON MIB HA MIB SW TRAP FA TRAP FICON TRAP HA TRAP FA MIB yes y no n yes FICON MIB yes y no n yes HA MIB yes y no n yes SW TRAP yes y no n yes FA TRAP yes y no n yes SW EXTTRAP yes y no n no FICON TRAP yes y no n yes HA TRAP yes y no n yes no change brocadessm USERID
112. e Secure Fabric OS User s Guide and the Brocade SilkWorm Design Deployment and Management Guide SAN DDM Version 3 0 these documents provide the necessary guidelines for implementing security A few details are covered here to help you get started e Security Measures e Advanced Security Secure Fabric OS Security Measures There are some SAN Security measures that should be in place before implementing Advanced Security Secure Fabric OS Here are some guidelines in the form of a checklist to assist with the planning process These steps can be taken to provide some initial restrictions on accessing the SAN and to provide some control over change management For Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 83 19 maximum SAN Security these measures should be used in conjunction with Advanced Security Secure Fabric OS Advanced Security Secure Fabric OS provides a single point of management and policies that allow complete control over what switches devices and management stations are allowed to access the SAN More detail is provided in the Brocade SilkWorm Design Deployment and Management Guide SAN DDM Version 3 0 Security Measures Checklist 1 Prevent Physical Access 2 Prevent Remote Access through IP security measures 3 Hard Zone the devices 4 Lock Down E port creation with portCf gEport Advanced Security
113. e Succeeded If using Advanced Security Secure Fabric OS do not disable the security command group as shown Finally verify the setting with supportshowcfgshow The output that should be seen is shown in the following example brocadessm root gt supportshowcfgshow os disabled exception enabled port disabled fabric enabled services enabled security disabled network disabled portlog disabled system enabled extend disabled filter disabled perfmon disabled brocadessm root gt For all supportShow output no matter how its configured there are certain commands that will always be executed These are in the order of execution date version and supportshowcfgshow Enabling and disabling is persistent except for the filter and extended groups This is not a concern as these two command groups are rarely used Two command groups have only one member The exception group has errdump The portlog group only has portlogdump The supportshowcfgshow commands supportShowCfgShow Displays list of command groups and whether they are enabled supportShowCfgEnable Allows root user to enable a single command group supportShowCfgDisable Allows root user to disable a single command group Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 115 22 Persistently Disabling a Switch or Port There are four commands to allow for per
114. e Telnet only e Secure Shell required in a Advanced Security Secure Fabric OS environment e Fabric Manager Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 45 e Fabric Access API via third party software The Brocade Advanced Security Secure Fabric OS license must be purchased from your switch supplier An example of the Advanced Security Secure Fabric OS graphical user interface launched from Fabric Manager is shown in Figure 8 1 Figure 8 1 Advanced Security Secure Fabric OS Interface launched from Fabric Manager ecurity Admin BRD_12K_sw0O_116 a xi efined Policy Active Policy scc FCS Policy FFCS Policy FCS 40 00 00 60 69 80 05 26 BRD_12K_sw0_116 10 00 00 60 69 80 05 26 BRD_12K_sw0_116 10 00 00 60 69 80 05 27 BRD_12K_sw1_117 10 00 00 60 69 80 05 27 BRD_12K_sw1_117 TELNET 10 00 00 60 69 90 12 e7 IBM_2109_F32 10 00 00 60 69 90 12 e7 IBM_2109_F32 RSNMP 10 00 00 60 69 51 96 27 IBM_2109_F16 10 00 00 60 69 51 96 27 IBM_2109_F16 7 WEN 10 00 00 60 69 c0 16 7d IBM_3534_F08 10 00 00 60 69 c0 16 7d IBM_3534_F08 HTTP SCC Policy SCC Policy API Policy does not exist Policy does not exist DCC Policy DCC Policy DCC Policy does not exist Policy does not exist z SES Policy SES Policy Policy does not exist Policy does not exist SERIAL ts Policy MS Policy EESE Policy does not e
115. e and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 104 20 Preparing the Fabric Configuration Once each switch has been prepared the SAN fabric can be built and configured A checklist provides some essential high level guidelines and task order Most of what needs to be done is the zoning of devices Once these steps are complete the staging is essentially complete This section only covers the high level outline of that checklist please consult the Brocade SilkWorm Design Deployment and Management Guide SAN DDM Version 3 0 for detailed step by step procedures SAN Fabric Configuration Checklist 1 Gather Planning Documentation ISL Map Device Spreadsheet Logical Design Diagram 2 Cable ISLs Check the fabric using the commands islshow trunkshow Cable Host and Storage Devices Check using the commands switchshow and nsshow 3 4 Use documentation to label all cables 5 Check consistency of Routing Settings for DLS and IOD on each switch in the fabric 6 Optional Create a Dummy Zone to prevent device access The default zoning configuration allows any device to see any other device If devices are plugged in and there is an initial desire to have all of them locked out until you can create proper zones define a dummy zoning configuration To do this create a zone configuration with an unused port This example shows the four steps that creat
116. e configured from the CLI Figure 26 7 Configuring Track Changes to send SNMP Traps brocadessm USE Track change Track change brocadessm U brocadessm USE Committing c RID gt trackchangesshow status OFF generate SNMP TRAP NO RID gt RID gt trackchangesset 1 1 figuration done RID gt RID gt trackchangesshow status ON generate SNMP TRAP YES RID gt brocadessm USE brocadessm U Track change Track change brocadessm USE NAHNANDNNWONNA NHN Switch Status Policy Setup The Brocade Switch Status Policy feature monitors the overall status of the switch based on several contributing parameters The policy parameter values determine how many failed or faulty units of each contributor are allowed before triggering a status change in the switch from HEALTHY to MARGINAL or DOWN These status events are integrated into Brocade Web Tools and Fabric Manager so that if the overall status of the switch is HEALTHY the switch color is Green If the overall switch status is MARGINAL the switch color is Yellow Finally if the overall switch status is DOWN the switch color is RED The overall status is calculated based on the most severe status of all contributors For the Brocade SAN Switch Module the following are monitored by this function e Faulty Ports triggers if the port goes faulty e Missing GBICs SFPs triggers if there are missing GBICs SFPs e Temperatures triggers if a temperature senso
117. e devices must not be a performance factor Ideally a storage port providing connectivity to one or more hosts must be able to satisfy the aggregated bandwidth requirement of all the hosts However if not carefully planned a congestion condition may exist when a port is over subscribed Advanced Performance Monitoring helps identify and correct this over provisioning condition Other SAN management areas which can benefit from the monitoring system e Capacity Planning e SAN Performance tuning e End to end visibility into fabric e Increasing productivity via Pre formatted Reports The following sections discuss e WebTools Performance Graphs e Configuring Advanced Performance Monitoring WebTools Performance Graphs WebTools Performance Monitoring allows you to set up a canvas of performance graphs from the Action menu e An existing graph can be selected from a list of graphs that are predefined or in some cases can be customized to monitor the specific objects of the fabric e The canvas window displaying the graph can hold up to a maximum of eight graphs simultaneously Any graph can be magnified added or removed from the main canvas e Up to 20 individual canvases can be saved for later retrieval e Graphs can be printed Configuring Advanced Performance Monitoring The Advanced Performance Monitoring license must be loaded on your Brocade SAN Switch Module This can be verified with the licenseshow command See Figure 2
118. e most recent Fabric Watch User s Manual e Environment class The environment class groups areas that deal with the physical environment inside a switch Specifically the environment class encompasses the ambient temperature of the switch the speed of the fans within the switch and the functionality and presence of power supplies An environment class alert will alert you to problems or potential problems with temperature fans and power e SFP class The SFP class groups areas that monitor the physical aspects of smart transceivers A SFP class alarm will alert you to faults that indicate that a transceiver may have deteriorated o SFP Small Form Factor Pluggable transceiver used in 2Gbit sec switches Note SFP class monitors smart SFP s only e Fabric class The fabric class groups areas that deal with potential problems that may arise between devices or other switches in a fabric The fabric class includes the monitoring of Inter Switch Links ISL fabric reconfigurations fabric segmentations domain ID conflicts zoning changes and other related changes to a fabric A fabric class alert will alert you to problems or potential problems with interconnectivity Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 135 26 e Port class The port class appears as the following three separate classes o E Port class o F FL Port Co
119. e upgrades troubleshooting commands for gathering important information about the SAN This chapter contains the following sections e Fabric OS Upgrades e Using Fabric OS Troubleshooting Tools e Support Show Command Groups e _ Persistently Disabling a Switch or Port e PathInfo e Support for Boot Over SAN Fabric OS Upgrades This section will provide some high level guidelines when upgrading firmware on the Brocade SAN Switch Module from Fabric 4 2 1 to a later release that contains new features or bug fixes This section will cover Web Tools firmwaredownloads on a single switch CLI commands will be mentioned For upgrading multiple switches Brocade Fabric Manager is suggested In Fabric OS 4 x firmwaredownload only allows the FTP protocol to be used for upgrades When upgrading from Fabric OS 4 2 1 to a newer release on the Brocade SAN Switch Module there is only one processor and the firmware must failover to itself In this case the Fabric OS Linux kernel and other processes that run in user space must be stopped and restarted through a fastboot By default a firmwarecommit is launched after the switch is rebooted This process runs in the background and copies the new firmware from the flash memory primary partition just upgraded to the backup partition Firmwaredownloadstatus can be used on another telnet session to check the upgrade progress This essentially plays back the firmwaredownload log The only message that wi
120. eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 94 20 Setting the Brocade SAN Switch Module to factory defaults should be done with caution If the Brocade SAN Switch Module is already part of an Caution existing fabric this action may cause fabric disruption Perform this during a maintenance window and ensure you have captured a configuration file using the configupload CLI command or from the Web Tools interface 3 Check Fabric OS version The recommended CLI commands to check the Fabric OS version for the Brocade SAN Switch Module are version and firmwareshow The firmwareshow command will provide additional information on the two banks of firmware stored in the compact flash for redundancy The Fabric OS version can also be found in the Web Tools and Fabric Manager interfaces See Figures 20 7 to 20 9 Figure 20 7 Version and firmwareshow output brocadessm USERID gt version Kernel 24509 Fabric OS v4 2 1 Made on Thu Apr 8 19 45 43 2004 Flash Tue Apr 13 19 07 48 2004 BootProm 4 1 0 brocadessm USERID gt brocadessm USERID gt firmwareshow Primary partition v4 2 1 Secondary Partition v4 2 1 brocadessm USERID gt Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 95 Figure 20 8 Fabric OS Version in Web Tools 4 Switch Explorer Microsoft Internet Explorer
121. ecipient configured yet Community 6 FibreChannel ro No trap recipient configured yet SNMP access list configuration Entry 0 No access host configured Entry 1 No access host configured Entry 2 No access host configured Entry 3 No access host configured Entry 4 No access host configured Entry 5 No access host configured brocadessml1 USERID gt Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 127 25 Figure 25 2 agtcfgset command usage brocadessm USERID gt agtcfgset Customizing MIB II system variables At each prompt do one of the following lt Return gt to accept current value enter the appropriate new value lt Control D gt to skip the rest of configuration or lt Control C gt to cancel any change To correct any input mistake lt Backspace gt erases the previous character lt Control U gt erases the whole line sysDescr Brocade SAN Switch Module for IBM eServer BladeCenter sysLocation End User Premise sysContact Field Support swEventTrapLevel 0 5 0 5 authTrapsEnabled true t false f false t SNMP community and trap recipient configuration Community rw Secret COde Trap Recipient s IP address in dot notation 0 Community rw OrigEquipMfr Trap Recipient s IP address in dot notation Community rw private Trap Recipient s IP address in dot notation
122. ect Proper planning allows for estimation of time and effort and provides justification for resources A good plan provides a means of measuring progress and greatly assists in avoiding potential pitfalls that prevent timely project completion 2 Staging After the planning phase the SAN needs to be put together Staging covers everything from uncrating and racking the switches to configuring Brocade Fabric OS and the applications that will run on the hosts storage and other devices that are attached to the SAN 3 Validation Once staged the entire SAN configuration needs to be tested and validated to confirm it is ready for production The tests should verify device connections check for the SAN robustness and most importantly test the application availability under varying failure sceneries 4 Maintenance Once the SAN transitions to an operational state changes are likely to occur such as the addition of hosts or storage This may require more switches if all user ports are allocated The Fabric OS or other firmware and software may need to be upgraded Maintaining the SAN is all about the day to day activities that keep the SAN running smoothly and efficiently Like any complex project there are many different ways of deploying a SAN Even though there is a level of complexity there are still general guidelines and practices that should be followed Many guidelines in this Section are in the form of checklists Checklists provide the
123. ected I O performance and functionality The default Performance Monitoring Graphs in Web Tools will work well too The sample script is shown below Figure 21 1 Figure 21 1 Sample I O script for UNIX validation bin sh PATH 1 SIZE 2 COUNT usr bin expr S SIZE 2 TMPFILE PATH sbtest RUN 0 echo Building test file STMPFILE usr bin dd if dev zero of TMPFILE bs 512k count COUNT gt dev null 2 gt amp 1 echo Done while 0 eq 0 J do DATE usr bin date echo Run SRUN Timestamp DATE usr bin dd if S TMPFILE of dev null bs 512k count SCOUNT gt dev null 2 gt amp 1 RUN usr bin expr RUN 1 Done Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 109 21 Sample Validation Recommendations SAN Fabric Validation Checklist 1 Fabric Stability Validation 2 High Availability ISL Failure Simulation 3 High Availability Switch Failure Simulation Fabric Stability Validation For validating the stability of the fabric run the host application software for a period of time If available use simulation tools provided by the software application vendor Run the sample script above for approximately72 hours to check for I O stability and observe for any problems Switch event logs should be setup to capture any problems in the fabric In addition a host syslogd can be setup
124. educed when a device is attached to a core A Core Edge fabric is typically built with two or more core switches but can be built with a single core switch if that fabric is used in a dual fabric SAN the redundant fabric maintains the SAN availability should the single core switch become unavailable Note A Core Edge topology can be built with a variety of switch platforms such as the SilkWorm 2000 series 3016 3200 3250 3800 3850 3900 12000 and 24000 The type of switch in a fabric does have some bearing on the practical as well as supported size of a fabric A SilkWorm 24000 is used as the core in Figure 15 1 as the 128 ports per domain support enables the size of the fabric to grow to several thousand ports by connecting edge switches The edge can be built with a variety of switch platforms The IBM eServer BladeCenter is a perfect fit for an Edge switch in this type of configuration A key benefit of the core edge topology is the use of FSPF which automatically distributes the load across all paths equally In fact all edge to edge paths are equal in a true core edge topology There are two or more paths between any two edge switches in a resilient core edge topology Because of this core edge fabrics have very good performance under varying to zero locality conditions Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 73 15
125. eferred for scalable available and high performance fabrics for a number of reasons With a Core Edge topology it is easy to satisfy SAN functional requirements Given a diverse set of requirements performance locality connectivity and scalability the Core Edge topology provides the most flexible architecture to address these overall requirements The Core Edge fabric is a variation on the well established star topology popular in Ethernet LANs There are a few differences Because Fibre Channel uses a routing protocol i e FSPF with load sharing Fibre Channel fabrics can take full advantage of multiple Core switches In an Ethernet network multiple switches at the center of a star would usually act in an active passive backup relationship using a Spanning Tree Protocol or some variation These differences make multi core fabrics very popular since it is possible to easily scale the fabric s bandwidth by adding Core elements In addition the requirements of a Core fabric switch are more stringent than those of the center switch in an Ethernet star Due to the properties of Fibre Channel the acceptable performance and reliability characteristics are very high The introduction of trunking further increases the effectiveness of a Core Edge fabric due to more efficient utilization of the ISLs and lessened management requirements Some Core Edge implementations opt for a single Core switch when deploying the fabric in a dual fabric SAN
126. er BladeCenter architecture simplifies the device connection strategy of the Host connections to the fabric How switches connect to other switches and how devices connect to those switches significantly influences the performance and availability of a SAN Easy to understand and consistent device attachment strategies also simplify the operation and maintenance of a SAN For more detail on other device connections strategies refer to the above mentioned DDM Guide These include e Trunk and ISL Connections e Edge Switch ISL Trunk Connections e Core Switch and Standalone ISL Trunk and Device Connections e Attaching SAN Devices for Availability e Connecting Devices to the Core e Low Locality Device Attachment e Platform Specific Design Considerations o Silk Worm 2000 Series 32x0 and 38x0 Switches o SilkWorm 24000 12000 and 3900 Brocade SAN Switch Module Design Considerations Since its inception Brocade has been the leader in developing technology and testing practices to expand the limits of Fibre Channel fabrics As fabrics increase in size the numbers of switches inter switch links ISLs and edge devices increase rapidly This in turn increases the demand on the fundamental computing resources of the Control Processor in each switch as it must rapidly complete tasks such as processing Zoning configuration updates distributed by other switches analyzing and distributing RSCNs responding to Name Server queries from hosts logging into
127. er to choose which MIB elements will be used to send SNMP trap information Brocade SNMP Agent Setup A default agtcf gShow output is included in Figure 25 1 The most notable aspects of configuring the Agent are included in the following steps 1 To begin the sysDescr sysLocation and sysContact information fields can be changed for the specific environment that the switches are installed in 2 swEventTrapLevel indicates which severity level messages will be sent to the SNMP Manager By default this value is set at 0 implying that no swEventTrap is sent Possible values are 0 none 1 critical 2 error 3 warning 4 informational 5 debug Note To configure the SNMP Manager to receive all messages this would be changed to a 5 Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 126 25 You might want to set the swEventTrapLevel to something in the middle say a 3 which will capture all the critical error and warning messages If you set it to 4 or 5 you will be inundated with messages which may be appropriate if you are troubleshooting an issue in the SAN Guideline 3 authTraps does some authentication checking if a request comes in from a community that is not known to the agent 4 The next section of the SNMP agent configuration is where the Trap Recipient IP addresses and community strings are ente
128. erver BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 16 2 e The ability to identify isolate and manage SAN events across multiple switches and fabrics e Drill down capability to individual SAN components through tightly coupled Advanced Web Tools Fabric Watch and Advanced Performance Monitoring integration e Discovery of all SAN components and ability to view the real time state of all Brocade fabrics e Moulti fabric administration of Advanced Security Secure Fabric OS through a single encrypted console e Call Home facility integrated into the IBM TotalStorage support system Brocade Fabric Manager 4 1 1 key features are briefly described below e SAN Discovery and Topology Display e SAN Logical Grouping Switch Groups and Port groups e Switch Port Administration e At A Glance views of switch and device information in the fabric e Fabric Change Management e ISL Checking e Fabric Merge Checking checks for Zoning and Switch Configuration parameter conflicts e Web Tools Device Management launcher e Telnet Sec Telnet launcher e Consolidated Event Monitoring e Switch Firmware Download e Switch Sequence Reboot e Switch Configuration Download Upload Base lining e Complete Fabric Backup and Compare e Zoning Management e Advanced Security Secure Fabric OS Policy Management e Set Time on all switches in a Fabric e Call Home support e Switch License Management e Firm
129. es a dummy zone for switch domain 101 and port 1 With this complete no devices are allowed to access any other device brocadessm USERID gt zonecreate dummyzone 101 1 brocadessm USERID gt cfgcreate dummycfg dummyzone brocadessm USERID gt cfgenable dummycfg brocadessm USERID gt cfgsave 7 Setup an NTP Time Service for the Fabric Use the Principal or Primary FCS Switch only to connect to the external NTP time server 8 Validate the fabric by comparing the entries in the Name Server with the devices attached to the fabric i e compare the number of entries in nsallshow with the number of devices attached to the switches 9 Pre Zoning Check Verify hosts and storage are in the fabric that are to be zoned On a core switch use the command nscamshow 10 Implement Zoning 11 Optional Implement Security 12 Profile each SAN fabric Routing Settings DLS and IOD Dynamic Load Sharing DLS Routing is generally based on the incoming port and the destination domain This means that all the traffic coming in from a port either an E_Port or an Fx_Port directed to the same remote domain is routed through the same output E Port If DLS is turned off using dlsReset load sharing is performed only at boot time or when an Fx_Port comes up This is the factory default for the Brocade SAN Switch Module and all IBM TotalStorage SAN switches By disabling DLS the possibility of dropped frames is eliminated every
130. es to Silk Worm 2000 series switches as well as SilkWorm 3000 series switches to SilkWorm 12000 switches Directly connecting a SilkWorm 2000 series switch to a SilkWorm 3000 12000 series switch is unsupported when using Extended Fabrics For a mixed SilkWorm 2000 and SilkWorm 3000 12000 fabric where the long distance ports are between SilkWorm 2000 series switches the fabric wide parameter fabric ops mode longDistance must be set to a value of 1 on all switches within the fabric For mixed fabric configurations where long distance ports are located between SilkWorm 3000 and or SilkWorm 12000 series switches the fabric wide long distance parameter is not required Brocade Security features are supported on Extended Fabric links This includes connections over dark fiber and DWDM networks Please refer to Chapter 6 for more specific information on the limitations of INe Bitended Pabricwith th Brocade SAN Switch Module Fabric OS Upgrade Planning This section will provide some high level guidelines when defining a strategy for performing a firmware upgrade of Fabric OS on an existing Brocade fabric This activity typically happens in the maintenance phase of a SAN deployment For the purposes of this discussion it is assumed that no new switches will be added or removed from the existing SAN infrastructure For that case and more details please see the Brocade SAN Migration Guide For detailed instructions on all upgrades refer to the Brocade
131. etailed how to information for the Brocade SAN Switch Module from a design deployment and management perspective This portion of the Design Deployment and Management guide is intended to be used in conjunction with existing Brocade manuals release notes and related Brocade publications especially the Brocade SilkWorm Design Deployment and Management Guide SAN DDM Version 3 0 The flow and organization of this section follows the process of first designing a Brocade SAN with focus on the Brocade SAN Switch Module for IBM eServer BladeCenter followed by the deployment and operation of that SAN It is important to understand how new features such as Advanced Security Secure Fabric OS non disruptive code activation and scalability impact a SAN and how these topics relate across the SilkWorm and Fabric OS family Many Brocade features span the disciplines of SAN design deployment and management For example ISL Trunking influences a SAN design has specific deployment tips and can be managed via various interfaces such as the CLI Web Tools and Fabric Manager Discussed in the SAN Design section are topics such as device attachment strategies switch placement in a fabric design related Security topics and Zoning guidelines Once the SAN design and other planning has taken place the switches and devices require deployment The SAN Deployment section covers subjects such as preparation planning usage of new features validating a f
132. ether activated or not Both policy Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 44 8 sets are distributed to all switches in the fabric by the primary FCS switch Advanced Security Secure Fabric OS recognizes each type of policy by a predetermined name Available Advanced Security Secure Fabric OS Policies Advanced Security Secure Fabric OS supports the following policies e FCS policy Use to specify the primary FCS and backup FCS switches This is the only required policy e Management Access Control MAC policies Use to restrict management access to switches The following specific MAC policies are provided Telnet policy Use to restrict which workstations can use sec telnet or Secure Shell to connect to the fabric telnet is not available when Advanced Security Secure Fabric OS is enabled HTTP policy Use to restrict which workstations can use HTTP to access the fabric Read and Write SNMP policies Used to restrict which SNMP hosts are allowed read and write access to the fabric API policy Use to restrict which workstations can use API to access the fabric SES policy Use to restrict which devices can be managed by SES Management Server policy Use to restrict which devices the management server can access Serial Port policy Use to restrict which switches can be accessed by serial port
133. evel is set at Default Default Custom Unit C Cc Time base Low 0 0 High gai 71 BufSize 10 10 Threshold alarm level is set at Default Errlog 1 SnmpTrap 2 RapiTrap 8 EmailAlert 16 Valid alarm matrix is 27 Default Changed Exceeded Below Above InBetween change behavior type change behavior interval change threshold boundary level change custom unit change custom time base change custom low change custom high change custom buffer apply threshold boundary changes 10 cancel threshold boundary changes Select choice gt 1 19 19 change threshold alarm level change changed alarm change exceeded alarm change below alarm change above alarm change inBetween alarm apply threshold alarm changes cancel threshold alarm changes return to previous page AANA BARUN OANA FWNH ER Koj 4 Figure 26 4 shows an example using the fwConfigure command to change the value of the Threshold Alarm Level to 3 for RX Performance in the E Port Class Note These examples can also be done through the Web Tools interface which simplifies the process The defaults are shown in orange and the values they are changed to are in violet Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 138 26 Figure 26 4 Example using fwConfigure to change the Threshold Alarm Levels to 3 br
134. f failures should occur An example of the use of Trunking in a dual redundant fabric configuration can be seen in Figure 14 1 This diagram expands upon the concepts learned in the previous discussions on fabric resilience and redundancy Figure 14 1 Utilizing ISL Trunking in Brocade SAN Switch Module in a Dual Fabric Core Edge Topology to increase performance Use of ISL Trunking to combine 2 ISLs in IBM each Brocade Switch Module into 1 logical BladeCenter ISL with a bandwidth of 4 Gbit sec Bay 3 Bay 4 Brocade Brocade Switch Module Switch Module ik OP a S Failed Core Resilient Switch Redundant Fabrics Failed Edge Switch Fabric B Vw Nv oua s MAN Nv Connected Fabric A Devices Dual Connected Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 67 14 ISL Over subscription Ratios When designing a SAN it is important to understand the performance boundaries such as storage fan out ratios and storage performance While any SAN device that connects to a SAN at 2 Gbit sec is theoretically capable of 2 Gbit sec in reality that device is most likely capable of a much lower performance If a device truly is capable of generating 2 Gbit sec of I O then the principles of locality should be applied or sufficient bandwidth should be provisioned for the ISLs A very popular SAN application is storage consol
135. f the Web Tools Admin interface copy the new license into the New License Key field and then click the Add button If using the CLI licenseadd i e brocadessm admin gt licenseadd SeQedReQRSbfRfeB Figure 20 16 3 Verify the new licenses are listed in the License Admin Tab You may need to click on the Refresh button If using the CLI licenseshow Figure 20 17 Figure 20 15 LicenselD information is found in the WebTools Info Tab E Switch Information for brocadessm loj x 10 00 00 05 1e 34 00 c7 EtherIP 10 654 210 25 Rade ID 1 Ether NM 255 255 240 0 Role Principal FC IP 0 0 0 0 State Online FO NM 0 0 0 0 Firmware w4 2 1_1e1 Gateway 10 64 208 1 Manufacturer Serial NYO40000069 LicenselD 10 00 00 05 1e 34 00 c7 Supplier Serial DOCCC2AWAIBS Java Applet Window Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 99 20 Figure 20 16 Adding a License using WebTools Switch Admin Microsoft Internet Explorer SwitchName brocadessm DomainID 1 WAAN 10 00 00 05 1e 34 00 c7 Tue Mar 23 2004 4 20 PM Switch Information Network Config Upload Download SNMP License Admin Port Setting Routing Extended Fabric Configure Feature s S9eyybz5QdTAfzzW Zoning Rz5eyQey9b50Tfzp Web SyRcRydee9Vz5d5x Fabric Watch bzze9QSb9zc00fA0 Fabric New Lice
136. f the port in the swF CPortTable where index port number 1 swF CPortName The port name identified by the portName command swSsn The SSN of the switch sending this trap swGroupName swGroup Type swGroupMemPos 4 swEventTrap This trap is generated when an event whose level is at or below swEventTrapLevel occurs Note This acts like a filter to control the traps that may be sent by the agent whenever an event is written to the Error Event Log If this is set to 0 then the agent will send no event traps If this is set to 5 then the agent will receive all levels of event traps Variables swEventIndex The index of the event in the swEventTable swEventTimelnfo The time at which event happened swEventLevel The severity level of the event swEventRepeatCount The number of times this event was repeated swEventDescr The description of the event swSsn The SSN of the switch that sent this trap swGroupName swGroupType swGroupMemPos 5 swFabricWatchTrap This trap is sent by Fabric Watch to notify of a Fabric Watch event Note Fabric Watch can further be configured to control the traps sent to an SNMP manager for each Class Area This feature acts as an additional filter for Fabric Watch traps sent out by the agent Variables swFwClassArealndex The Class Area of the threshold swF wThresholdIndex The index of the threshold in the swf wThresholdTable swFwName The name of the threshol
137. formation Network Config Upload Download SNMP SNMP Information Contact Name Field Support Description Brocade SAN Switch Modu Location End User Premise Trap Level fo None x J Enable Authentication Trap Community Trap Recipient Access Control List Community String Recipient Access Control Access Host Access Control List Secret Code 0 0 0 0 Read Write Read Write OrigEquip Mfr 0 0 0 0 Read Write Read Write private 0 0 0 0 Read Write Read Write public 0 0 0 0 Read Only Read Write common 0 0 0 0 Read only Read Write FibreChannel 0 0 0 0 Read Only Read Write Apply Close Reset Refresh Switch Administration opened Fri Mar 26 2004 3 07 PM Configure SNMP parameters For more information about using Brocade s SNMP agent refer to the Brocade MIB Reference Manual Brocade Fabric Access API The Brocade Fabric Access API is an application programming interface that enables any application to access critical information about a Brocade SAN With Fabric Access an application can query or control individual switches or the entire fabric The Fabric Access API has the following advantages e Itcan create SAN management applications specific to your needs using the tools available in the Fabric OS e It uses third party software to manage a Brocade fabric i e IBM Tivoli Storage Area Network Manager Wot The Brocade SAN Switch Module is su
138. gement Guide Publication Number 53 0000561 01 48 Configuration Restrictions In interoperable fabrics the following restrictions apply Note There is an architecture maximum of 31 switches However the actual configuration tested is much less Domain IDs must be in the 97 to 127 value range for successful connection to McDATA switches The firmware automatically assigns a valid domain ID if necessary when the interopmode command is enabled on the switch fabricShow only shows the WWN and Domain ID for McDATA It will not indicate the IP address or switch name Brocade switches will show all of the above When in Interoperability mode if managing zoning from the Brocade switches then all Brocade switches must have at least one direct connection to another Brocade switch For example you cannot have a McDATA switch in between two Brocade switches if you are managing zoning from the Brocade switches JDSU SFPs will come up as In_Sync and an ISL cannot be established if they are connected to a MCDATA ISL Note A work around is documented in the Brocade Fabric OS v4 2 1 Release Notes When a Brocade switch gets a new domain ID assigned through a fabric reconfiguration it will write the new domain ID to flash and the old domain ID value will be overwritten When a McDATA switch gets a new domain ID assigned through a fabric reconfiguration it will keep the original domain ID in flash So then when the domain ID of a McDATA switch
139. gement process by providing centralized management Brocade Fabric Manger is capable of managing complex tasks at the fabric switch and port levels In addition access to Brocade Web Tools Fabric Watch and Advanced Performance Monitoring is made available via the Fabric Manager interface This allows for complete SAN management from a single user interface Brocade SAN Switch Module management and monitoring products include e IBM eServer BladeCenter s Management Module e Command Line Telnet e Web Tools HTTP e Fabric Manager e SNMP e Fabric Watch e Advanced Performance Monitoring e Brocade Fabric Access API The Brocade CLI and WebTools interfaces have been featured throughout this Guide The following chapters will focus on SNMP Fabric Watch and Advanced Performance Monitoring Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 125 Chapter 25 SNMP One of the standard methods for monitoring and managing a network device is through Simple Network Management Protocol SNMP It is a universally accepted protocol that is portable lightweight and is widely deployed SNMP allows an administrator to monitor the health and performance of countless devices locally or remotely via an Ethernet port Enterprise management software like IBM Tivoli Netview and IBM Tivoli Storage Area Network Manager ITSANM can monitor thousands of
140. guration database depends on the switches in the fabric whether or not interopmode is enabled and the number of bytes required for each item The number of bytes required for an item depends on the specifics of the fabric but cannot exceed 64 bytes per item At 64 bytes per item you can have 498 entries for a fabric consisting solely of 4 x switches and interopmode enabled You can use the c gSize command to check both the maximum available size and the currently saved size If you believe you are approaching the maximum you can save a partially completed zoning configuration and use the cfgSize command to determine the remaining space Zone Name Restrictions The name field must contain the ASCII characters that actually specify the name not including any required fill bytes Names must adhere to the following rules Note A name must be between 1 and 64 characters in length All characters must be 7 bit ASCII characters The first character of a given name must be a letter A letter is defined as either an upper case A Z character or a lower case a z character Any character other than the first character must be a lower case character a z an upper case character A Z a number 0 9 or one of the following symbols _ When interopmode is in effect the space available for the zoning database is only half the normal size Pre Configuration Planning Before enabling interoperability mode the individual fabrics should
141. he first telnet login to the Brocade SAN Switch Module the user will be challenged to change passwords for three of the four accounts on the system Note root factory and user The four accounts are root factory USERID and user This step can be skipped by using the CONTROL C key combination however the challenge will continue at every new login until all of the accounts passwords have been changed Store the passwords in a safe location For more information about using the command line interface refer to the Brocade Fabric OS Procedures Guide and the Brocade Fabric OS Reference Manual Advanced Web Tools Brocade Advanced Web Tools is an excellent partner to the traditional Fabric OS CLI commands in many ways it provides faster and more effective results than can be achieved strictly through the CLI Advanced Web Tools provides a graphical interface that enables you to monitor and manage individual switches and ports from a standard workstation It is a licensed product that runs on Fabric OS All switches in the fabric are displayed in the main window of Advanced Web Tools including switches that do not have an Advanced Web Tools license however only those switches that have an Advanced Web Tools license installed can be managed through Advanced Web Tools other switches must be managed through telnet Following are some of the features that make Advanced Web Tools an important part of the switch management and administration proces
142. he internal ports 9 and 10 Certain long distance Caution configurations depending on the length and speed of the links involved might affect the performance of servers in bays 9 and 10 of the IBM eServer BladeCenter and in the most extreme cases can prevent these ports from coming up Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 39 6 For external ports operating at 2 Gbit sec the following restrictions should be observed for the Brocade SAN Switch Module Note External Port External Port 0 or 15 L2 60km L2 60km L2 60km L1 50km L1 50km L1 50km L0 5 25km 0 or 15 L2 60km L0 5 25km E port L1 50km L0 5 25km E Port L0 5 25km Notes Not Allowed Two internal ports disabled One internal port disabled Two internal ports disabled One internal port disabled One internal port disabled One internal port disabled When using the LD setting of external ports LD mode auto senses the actual cable length and depending on its distances internal ports 9 and 10 may be disabled For external ports operating at 2 Gbit sec the following are valid Extended Fabrics configurations for the Brocade SAN Switch Module The internal ports are not affected in the following configurations Oor15 E Port LE 10km L1 50km L0 5 25km L0 5 25km External Port External Port 0 or 15 E Po
143. he primary FCS switch it distributes the following information to the other switches in the fabric e Zoning configuration e Advanced Security Secure Fabric OS policies e Fabric password database e SNMP community strings e System date and time The role of the FCS switch is separate from the role of the principal switch Note which assigns Domain IDs The role of the principle switch is not affected by whether Secure Mode is enabled When Secure Mode is enabled only the primary FCS switch can propagate management changes to the fabric When a new switch joins the fabric the primary FCS switch verifies the digital certificate then it provides the current configuration overwriting the existing configuration of the new switch Since the primary FCS switch distributes the zoning configuration zoning databases do not merge when new switches join the fabric Instead the zoning information on the new switches is overwritten when the primary FCS switch downloads zoning to these switches if Secure Mode is enabled on all the switches For more information about zoning refer to the Fabric OS Features Guide For more information about merging fabrics see the Chapter 4 of the Secure Fabric OS User s Guide The remaining switches listed in the FCS policy act as backup FCS switches If the primary FCS switch becomes unavailable for any reason the next switch in the list becomes the primary FCS switch A minimum of one backup FCS switch is strongl
144. i Ea BIE Tenet a aje gE HE Info ig Status Healthy Polled at 04 23 04 11 24 AM Hame brocadessm Fabric OS version v4 24 Domain ID 1 Ethernet IP 10 64 210 25 Ethernet Mask 255 255 240 0 FCnet IP 0 0 0 0 FCnet Mask 0 0 0 0 Gateway IP 10 64 208 1 WWH 10 00 00 05 1 34 00 c7 z G0 Links gt Name Server f o B Current Zone Config None Status Legend E Heatthy _ Marginal Critical _ Unmonitored E Click here for Name Server I fa Internet Step 2 Sort the entries by Port and then Domain Then look to the Port ID column to obtain the Source ID s SID and Destination ID s DID of the devices you wish to monitor In this example the two Server Blades are SID_1 010100 Port 1 on Domain 1 and SID_2 010200 Port 2 on Domain 1 The Fibre Channel JBOD drives are located on the same port Domain 3 Port 7 DID_1 0307ba the Port ID of the disk drive that SID_1 is communicating with and DID_2 0307bc the Port ID of the disk drive that SID_2 is communicating with Figure 27 5 Figure 27 5 Name Server icon in WebTools Name Server Table Microsoft Internet Explorer 0 xj Name Server I Auto Refresh Auto Refr rval 115 seconds Number of Devices 5 Y Domain Port Port Name a ID Port Port WW Port WiWN Bay1 010100 N none Bay2 010200 N none 0307ba INL 169 20 00 00 40 b4 FCP 0307bc NL 20 07 00 60 69 30 11 4d 21 00 00 20 37 c8 3d d
145. ically across the merged ISLs at the fibre channel frame level while preserving in order delivery of the frames ISL Trunking Architecture The ISL Trunking software identifies and constructs trunking groups as soon as the ISL Trunking license is activated A license must be activated on each switch that will participate in trunking The ISLs and ports that participate in trunking groups are referred to as trunking ISLs and trunking ports ISL Trunking makes it possible to accomplish the same fabric performance with fewer ISLs resulting in simplified fabric design and management lowered cost of ownership increased fabric performance and increased data availability Figure 4 1 One Logical ISL made up of 2 physical ISL links With Trunking A A The link acts as a single 4 Gbit sec pipe Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 25 Trunking Ports The first ISL link that is connected to another Brocade switch that supports ISL Trunking will be the master ISL Trunk master The master ISL port works to direct traffic over the trunked ISLs If the second ISL is connected to the same switch and the same Trunk Group 4 port grouping also known as a quad on that switch then the second ISL link becomes the secondary E port of that Trunk Group If it is connected to different Trunk Group quad or a different switch then it wi
146. ics each fabric only containing a single core Multi fabric resilient The most common multi fabric SAN is the dual fabric SAN In a dual fabric resilient SAN half of the switches are connected to form one fabric and the other half form a separate fabric This model can be extended to more than two fabrics if desired No fabric has a single point of failure that could cause the fabric to segment This design can be used in combination with dual attached hosts and storage devices to keep an application running even if one entire fabric fails due to operator error catastrophe or hardware software issues This is the best design approach for high availability environments Another key benefit of this design is the ability to take part of the SAN offline for online upgrades or maintenance without affecting production operations on the remaining fabric s An example of this type of SAN is a dual fabric SAN built with core edge fabrics each fabric containing two or more core switches Figure 11 3 Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 58 11 Figure 11 3 In a Dual Fabric Core Edge Topology there is resilience to Core fabric switch failures and Edge fabric switch failures Hosts y Dual Connected Failed Core Switch Resilient Redundant N Fabrics Failed Edge Switch Fabric B Fabric A Devices
147. ics for one hop Hop In Port Domain ID Name Out Port BW Cost T 23 93 ess093 5 2G 500 Port 23 5 TX Rx Tx Rx F s 1s 0 0 0 0 F s 64s 0 0 0 0 Words 9270 8003 9296 9741 Frames 435 450 544 SER Errors 0 0 Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 119 22 Support for Boot Over SAN Contact IBM Support for supported configurations and instructions for configuring the IBM eServer BladeCenter Fibre Channel Expansion Card as the boot device For more information on the Fibre Channel Expansion Cards refer to the latest IBM eServer BladeCenter Fibre Channel Expansion Card BIOS driver Readme and the IBM eServer BladeCenter Fibre Channel Expansion Card Installation and User s Guide available at http www ibm com pc support In addition certain storage vendors may have statements of support for Remote Boot environments please contact them directly Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 120 SAN Management This section discusses fundamental SAN Management concepts and associated guidelines This section contains the following chapters e SAN Management Overview e Brocade SAN Switch Module Management Tools e SNMP e Fabric Watch e Advanced Performance Monitoring Brocade Enterprise and Entry SAN Swi
148. idation where many hosts share a storage device or port Several popular storage vendors target an average of a 6 1 fan out This means that on average six hosts are sharing a single storage port If there were 32 storage ports in a fabric then one would expect to find an average of 192 hosts Even if every host requires 1 Gbit sec or 2 Gbit sec of bandwidth the storage devices in the fabric are only capable of delivering 32 Gbit sec 1Gbit sec ports or 64 Gbit sec 2 Gbit sec ports This equates to 3 6 MB sec per host While some ports in the fabric may require maximal bandwidth not all ports require sustained maximal bandwidth and rarely if ever do these ports require maximal bandwidth simultaneously When all ports operate at the same speed ISL over subscription is the ratio of device or data input ports that might drive I O between switches to the number of ISLs over which the traffic could cross For the IBM eServer BladeCenter architecture there can be a maximum of 14 internal Server connections that can go over a maximum of 2 ISLs per Brocade SAN Switch Module The maximum over subscription ratio on the Brocade SAN Switch Module is then 14 host device ports to 2 ISLs Which can be abbreviated to14 2 This can further be reduced as a fraction to a 7 1 over subscription ratio The basic over subscription formula is ISL Over Subscription Number of Nodes Number of ISLs or lo Nn Ni This is reduced as a fraction so that Ni 1 If
149. ill provide some guidelines on setting up supportshow command groups and make a recommendation on which ones to use By default eight command groups are enabled for supportshow These are shown by supportshowcfgshow output below admin USERID level command However changes to the supportshow configuration can only be made as the root user Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 114 22 brocadessm USERID gt supportshowcfgshow os enabled exception enabled port enabled fabric enabled services nabled security enabled network enabled portlog enabled system enabled extend disabled filter disabled perfmon disabled brocadessm USERID gt If you have the root password consider configuring supportshow to disable the following groups For general information about the fabric and the switch supportshow is running on these groups are really all that is needed For most SAN administrative troubleshooting cases the data provided by the remaining groups will do brocadessm root gt supportshowcfgdisable os Config update Succeeded brocadessm root gt supportshowcfgdisable port Config update Succeeded brocadessm root gt supportshowcfgdisable security Config update Succeeded brocadessm root gt supportshowcfgdisable portlog Config update Succeeded brocadessm root gt supportshowcfgdisable network Config updat
150. ily available allows change management of all components In addition being organized with updated documentation saves time and effort when referencing equipment for service calls Even while not being serviced when information is needed it can be referenced quickly and easily Here is a checklist that provides a recommendation as to what documentation should be created before and during the staging of the equipment For templates to assist with your planning refer to the Appendices of the Brocade SilkWorm Design Deployment and Management Guide SAN DDM Version 3 0 Documentation Checklist 1 Get an Equipment Binder Logical Design Diagram Switch Spreadsheet ISL Port Map Device Spreadsheet Label All Cables SAN Verification Test Plan Requirements SAN Verification Test Plan oN A oa A U N Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 81 19 Set unique domain numbers for each switch in the SAN This allows for Gueine simpler merging of fabrics As a convention consider setting the domain ID of each switch to the last Guideline octet of its IP address Be aware that the highest allowed domain number is 239 Zone Planning Zoning allows the hosts to access specific storage devices on the SAN For those SANs with multiple OS platforms zoning allows for OS separation and co existence With no zoning defined on
151. interopmode 1 The switch effective configuration will be lost when the operating mode is changed do you want to continue yes y no n no y done Interopmode is enabled Note It is recommended that you reboot this switch for the new change to take effect switch admin gt Disabling Interoperability Mode To disable interoperability mode 1 Connect to the switch as the administrator 2 Enter the switchdisable command to disable the switch At the command line enter the interopmode 0 command to disable interoperability This command resets a number of parameters and disables interactive mode 4 Reboot the switch after changing the interoperability mode Wait for a fabric reconfiguration after adding each switch 6 Repeat this procedure on all Brocade switches in the fabric Example switch admin gt switchdisable switch admin gt interopmode 0 The switch effective configuration will be lost when the operating mode is changed do you want to continue yes y no n no y done Interopmode is disabled Note It is recommended that you reboot this switch for the new change to take effect switch admin gt Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 51 Section Il The second section of the Brocade SAN Switch Module for IBM eServer BladeCenter Design Deployment and Management Guide is focused on covering d
152. involve parameters that are applied to many switches at once Switch tasks generally involve environmental monitoring Port level tasks generally involve performance monitoring setting up port level definitions etc Of course there is overlap between these areas The following list itemizes SAN Management at the Fabric Switch and Port level and the various administrative tasks associated with them In each area a recommended Brocade management tool for these tasks is shown Fabric Administration Fabric Manager SAN Discovery and Topology Display SAN Logical Grouping Switch Groups and Port Groups Fabric Login Global access Switch Port Administration At A Glance views of switch and device information in the fabric Fabric Change Management ISL Checking Fabric Merge Checking checks for Zoning and Switch Configuration parameter conflicts Web Tools Device Management launcher Telnet Sec Telnet launcher Consolidated Event Monitoring Switch Firmware Download Switch Sequence Reboot Switch Configuration Download Upload Base lining Complete Fabric Backup and Compare Zoning Management through WebTools Name Server Displays through WebTools Advanced Security Secure Fabric OS Policy Management Set Time on all switches in a Fabric Call Home support Switch License Management Firmware Download to FDMI capable Host Bus Adapter Switch Administration Web Tools or CLI Switch information Network configuration Firmwared
153. ion The following information will be covered in this section e Fabric Watch Configuration e Fabric Watch Setup e Track Changes Setup e Switch Status Policy Setup Fabric Watch Configuration Brocade Fabric Watch proactively monitors and reports on the health of the switches and the SAN fabric Proactive monitoring and notification improves SAN availability The real time alerts from Fabric Watch software help SAN managers solve problems before they become costly failures With Fabric Watch software SAN managers can place limits or thresholds on the behavior of different switch and fabric elements A notable feature of Fabric Watch is that Fabric Watch messages can be configured by individual class elements to send events to an SNMP Manager If the Brocade SW SNMP traps are turned on as described in the SNMP chapter you will receive messages from Fabric Watch for the class elements that have their Threshold Alarm Level configured to send SNMP traps By default the Environmental classes within Fabric Watch are configured to send SNMP traps For other classes one must configure the appropriate Threshold Alarm Level Fabric Watch elements are any fabric or switch component that the software monitors To monitor elements Fabric Watch categorizes them into areas and further groups areas into classes The following are the list of classes that fabric watch currently monitors and reports on For more detail on any of these classes please read th
154. ion CD ROM the IBM eServer BladeCenter Web Site the IBM TotalStorage SAN Switch Web Site or on the Brocade Web Site through Brocade Connect Fabric OS e Brocade Fabric OS Procedures Guide e Brocade Fabric OS Reference Manual e Brocade Diagnostic and System Error Messages Reference Manual e Brocade MIB Reference Manual e Brocade Fabric OS v4 2 1 Release Notes Fabric OS Optional Features e Brocade Fabric OS Features Guide e Brocade Advanced Web Tools Administrator s Guide e Brocade Fabric Watch User s Guide e Brocade Secure Fabric OS User s Guide e Brocade Secure Fabric OS QuickStart Guide Fabric Management Software e Brocade Fabric Manager User s Guide Brocade SAN Switch Module for IBM eServer BladeCenter SilkWorm 3016 e SilkWorm 3016 Hardware Reference Manual e SilkWorm 3016 QuickStart Guide Additional Resource Information For information about how to use many of the features in this document in a SAN solution refer to the Brocade SilkWorm Design Deployment and Management Guide SAN DDM Version3 0 53 0000366 03 The latest DDM can be found through the Brocade portal link on the IBM TotalStorage SAN Switch Support Website Go to http www storage ibm com ibmsan products 2 109 library html support Click on any of the links at the bottom of the page for Software Product Manuals At the redirected site navigate to Software Product Manuals Technical How To Guides Home gt Technical Resource Center gt Do
155. ion Number 53 0000561 01 129 25 Figure 25 4 SNMP Agent Web Tools configuration Z Switch Admin Microsoft Internet Explorer 15 x SwitchName brocadessm Thu Apr 22 2004 3 22 PM DomainID 1 WAAN 10 00 00 05 1e 34 00 c7 License Admin Fort Setting Routing Extended Fabric Configure Trunk Information Switch Information Network Config Upload Download SNMP SNMP Information Contact Name Field Suppo Description Brocade SAN Switch Modu Location Ena User Premise Trap Level o None xl Bj rap rCommunity Trap Recipient Recipient Access Control Read Write Access Control List Access Control List Read Write 0 0 0 0 Read Write Read Write 0 0 0 0 Read Write Read Write public 192 168 2 10 Read Only 0 0 0 0 Read Write common 0 0 0 0 Read Only 0 0 0 0 Read Write FibreChannel 0 0 0 0 Read Only 0 0 0 0 Read Write Apply Close Reset Refresh Switch Adrninistration opened Thu Apr 22 2004 3 22 PM Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 130 25 Brocade SNMP Trap Setup An unsolicited message that goes to the SNMP management station from the SNMP agent on the switch is called a trap Brocade switches send traps out on UDP port 162 only The Brocade
156. ion Number 53 0000561 01 133 25 The swEventTrap can be configured using the agtC gSet command The other traps can not be configured independently and the SNMP management station will receive the other traps even if swEventTrap is turned off Note Fibre Alliance FA Trap Detail The Fibre Alliance SNMP Traps can be found in the FA_v3_0 mib file which provides Traps on the following information Please refer to the Brocade MIB Reference Manual for more detail 1 connUnitStatusChange The overall status of the connectivity unit has changed Variables connUnitStatus status of the connection unit connUnitState state of the connection unit 2 connUnitDeletedTrap A connUnit has been deleted from this agent Variables connUnitID id of the connection unit 3 connUnitEventTrap The connectivity unit has generated an event Variables connUnitEventld Internal event ID ranging between 0 and connUnitMaxEvents now obsolete connUnitEventType The type of this event connUnitEventObject This is used with the connUnitEventType to identify which object the event refers to connUnitEventDescr The description of the event 4 connUnitSensorStatusChange The status of the sensor associated with the connectivity unit has changed Variables connUnitSensorStatus The status indicated by the sensor 5 connUnitPortStatusChange The status of the sensor associated with the connectivity unit has changed Va
157. ion is found in the WebTools Port Setting Tab Switch Admin Microsoft Internet Explorer a loj x SwitchName brocadessm DomainID 14 WAAN 10 00 00 05 1e 34 00 c7 Tue Mar 23 2004 5 30 PM Switch Information Network Config Upload Download SNMP License Admin Port Setting Routing Extended Fabric Configure Persistent Enable Current Disable Trunking Speed o O Vv Vv No_Module N2 Negotiate fExt0 1 O IV CT Online 2G 26 Bay1 2 O Vv O Online 2G 2G Bay2 3 m 7 o Nolit 26 26 Ieay3 4 O Vv O No_Light 2G 2G Bay4 5 O Vv E No_Light 2G 2G v Bay5 6 O Vv m No_Light 2G 2G v Bay6 7 O Vv 0 No_Light 2G 2G Y Bay 8 T F D Nolit 26 2G Bays 9 O Vv O No_Light 2G 2G Bay 10 O Vv Oo No_Light 2G 2G Bay10 i Tr m r mout 2 2G gt Bay11 h i Apply Close Reset Refresh Switch Adrinistration opened Tue Mar 23 2004 4 39 PM Configure Port Setting parameters i The Brocade SAN Switch Module comes pre configured with PortNames that match the internal Server Blade Bays and External Port identifiers It is best Suey to append to the existing Port Names with additional information that might be pertinent for that port Port names can be cleared on port by port basis with portCfgDefault Note The portShow command displays port name in the first line of the output switchShow does NOT display the portName The port name label
158. ise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 136 26 2 The swFabricWatchTrap in the file named SW_v5_1 mib described in the SNMP section above receives Fabric Watch notifications of an event if the Threshold Alarm Level for a particular class includes the SNMP TRAP option The fwConfigure telnet command can be used to configure Fabric Watch to control the traps that Fabric Watch can send SNMP traps for each Class Area can be turned ON or OFF This effectively acts like a filter for Fabric Watch traps sent out by the agent 3 Figure 26 3 shows an example of the fwConfigure command This example shows that the Value of Threshold Alarm Level is 3 for Temperature in the Environmental Class This means that a Fabric Watch event is sent to the Error Event Log and to an SNMP management station The other possible options are indicated as well 1 Errlog sent to the Switch Error Event Log 2 SnmpTrap sent to an SNMP management station 4 PortLogLock locks the portlog if an event is triggered 8 RapiTrap sent to the Brocade API 16 EmailAlert sent to an Email recipient configured using fwMailCfg The Threshold Alarm Level is configured using a matrix value with a maximum value of 31 Combine any of the option numbers above to choose the value you wish to use For example the value of 3 is Option
159. ist 0 Fri Apr 16 16 30 44 2004 Firmwaredownload has started 1 Fn Apr 16 16 34 51 2004 Firmwaredownload has completed successfully 2 Fn Apr 16 16 36 02 2004 Firmwarecoramit has started 3 Fn Apr 16 16 39 43 2004 Firmwarecoramit has completed successfully 4 Fn Apr 16 16 39 43 2004 Firmwaredownload command has cormpleted successfully Firmware download completed Fn Apr 16 2004 04 39 55 PM xl Firmware download completed successfully Figure 22 2 WebTools Firmware Download Confirmation pop up Confirm Firmware Download x Firmware download may take a while A This operation will cause the switch to reset during the process This will cause temporary loss of network connection to the switch All other operations will be disabled during the download Do you want to proceed Java Applet Window Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 113 22 Figure 22 3 Firmwaredownload using Fabric Manager Firmware download to switches Ee 10l x Host IP address 10 64 210 23 Remote User Name fanonymous Firmware file 4 21 wN4 2 1 irelease plist Browse Select Protocol FTP Y Password Required for FTP fresexensenansenarens I Save settings to options C Name C P C WAN Selected Switches 1 E SAN Elements S7 Fiter IP Address Switch Name Firmware version Status Mess
160. ividual port Port Snapshot Error AL_PA Errors Displays CRC errors for a given port and a given AL_PA SCSI Commands by port and LUN __ Displays the total number of read write commands on a given port and R W R W read write commands to a specific LUN User Defined Graphs You can modify the predefined graphs based on parameter fields such as SID DID LUN AL_PA and port to create your own customized graphs These user defined graphs can be added and saved to canvas configurations Using Advanced Web Tools Advanced Performance Monitoring allows you to set up a canvas of performance graphs The canvas can hold up to eight graphs per window Multiple canvases can be set up for different users or different scenarios Up to 20 individual canvases each with up to eight graphs can be saved Each canvas is saved with a name and an optional brief description In addition to the graphs the Performance Monitoring Resource Usage Display shows which filter slots have been used for each port and which are available Advanced Performance Monitoring also has an extensive set of CLI commands for additional information refer to the Fabric OS Reference Manual and the Fabric OS Procedures Guide The Brocade Advanced Performance Monitoring license must be purchased from your switch supplier An example of the Basic and Advanced Performance Monitoring graphical user interfaces choices launched from Advanced Web Tools is shown in Figures 5 2 a
161. k on OK on the confirmation message to save the canvas Figure 27 16 Figure 27 16 Save Canvas Configuration Save Canvas Configuration 3 xj Name IEM BladeCente Description Server Performance of Servers in bav 1 and 2l Save Canvas Cancel Java Applet Window Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 157 27 Step 14 To display or load the Saved Canvas Configurations at a later time i e when starting a new session choose the Display Canvas Configurations from the Actions menu A list of the Saved Canvas Configurations will appear You can Load Edit Copy or Remove any of the Canvas Configurations from the list Figure 27 17 Figure 27 17 Save Canvas Configuration Canvas Configuration List for brocadessm i 4 lol x anvas Name Description IBMBladeCenterAPM All port and single HBA to single Di IBM BladeCenter Perf Server Performance of Servers in ba Load Edit Copy Remove Close Java Applet Window This is a simplified example to show how easy it is to create an Advanced Performance Monitoring Filter In more complex environments when many Server devices are talking to many storage ports the ability to analyze the Hot Spots of data throughput in the SAN will assist in being able to scale the SAN infrastructure as your bandwidth needs increase Brocade Enterprise and
162. l automatically derive a domain ID as part of a fabric rebuild To automatically obtain a domain from the fabric it is necessary that the switch connect to the fabric in a disabled state and then be enabled once the connection is complete By setting the domain ID the domain will become deterministic on subsequent power cycles or reboots Setting the domain ID number of the switch applies to all versions of Fabric OS The domain ID number is the first 8 bits of the 24 bit port ID PID The default domain number is 1 for all switches The following steps provide a high level outline of the procedure for more in depth information please refer to the Fabric OS Procedures Guide 1 Inthe Switch Information tab of the Web Tools Admin interface click on the Disable button in the Switch Status field and then click the Apply button If using the CLI switchdisable 2 Choose a new Domain ID from 1 to 239 and place it in the Domain ID field of the Name and ID area and then click the Apply button If using the CLI you will need to use the configure menu driven command 3 Inthe Switch Information tab of the Web Tools Admin interface click on the Enable button in the Switch Status field and then click the Apply button If using the CLI switch enable The Web Tools interface is shown in Figure 20 14 Figure 20 14 Domain ID configuration in Web Tools E Switch Admin Microsoft Internet Explorer
163. l time critical fabric event detection and notification e Fault isolation and corrective action e Analyze long term behavioral trends SAN security is a crucial issue that must be addressed by ensuring that only a select group of authorized users are given access to make SAN configuration changes in a controlled manner SAN access can be secured by implementing secure policies at the system as well as the Fabric OS level Graphical display of a SAN provides a quick view of its components and their relationship to one another The device type and in some cases the status of a device should be determined by the device icon coloring scheme It also simplifies the SAN device access process utilizing the point and click method Maintaining a consistent configuration within the fabric and across fabrics requires performing a variety of tasks on more than one switch in a fabric For example upgrading firmware can be a time consuming task if a large number of switches are involved An efficient method of upgrading firmware is to discover the switches by logging into a single switch and then initiating the firmware download procedure on a selected group of switches A real time self monitoring status and advanced warning can be very helpful to maintain the health of the SAN by avoiding costly down time in an enterprise environment In a redundant system the failure of a FRU may not be necessarily a disastrous event however the failure does increase the vu
164. le HCA procedure should always be performed when the fabric is stable However recovery actions will be taken upon completion of the HCA to ensure that no RSCNs or fabric configuration changes are missed If any new hosts or targets were added to the Brocade SAN Switch Module switch during the HCA reboot time then the initial FLOGI will time out After the reboot the switch will reset that port to cause the FLOGI to happen again If any device is added elsewhere in the fabric the RSCN will be delivered after the reboot completes Finally if any E port cables are pulled or the fabric rebuilds for any reason then after the HCA reboot completes the Brocade SAN Switch Module will cause the fabric to rebuild again so that it may participate in the fabric To prevent unnecessary disruptions in the fabric when firmware is activated on a Brocade SAN Switch Module it is recommended that any switches directly connected to the Brocade SAN Switch Module use Fabric OS v2 6 1 v3 1 0 or v4 1 0 or subsequent versions Guideline Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 75 Chapter 16 Zoning Design This chapter contains the following sections e Zoning Design Considerations and Guidelines e Zoning and Scalability e Zoning Database Size Zoning Design Considerations amp Guidelines Zoning is an important element of a secure and hea
165. lidation value proposition has never been better Assume that 20 hosts each connect to 100 GB of storage in a direct attach environment requiring a total 2000 GB of storage Some space on each system is free This is known as white space or headroom The average utilization of this directly attached storage DAS is 50 leaving 50 white space The total storage utilized is 1000 GB which leaves 1000 GB of white space With the use ofa SAN it is possible to achieve much higher utilization since every host has access to all storage in the SAN In this example a modest 10 20 improvement in storage utilization could result in a savings of several hundred GB of storage In addition a reduction in associated ownership costs of that surplus storage would occur In the storage consolidation model if a host is not using all of its storage it is possible to rapidly reallocate this extra storage to a different host It is also possible to add additional storage for all servers to access rather than having to purchase storage for specific hosts In a direct attach environment it is more difficult to do so forcing the need to have very high white space overhead to allow growth Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 54 10 Since many hosts depend upon continuous access to their storage in a storage consolidation solution designing a highly
166. ll be received during the upgrade is Firmwaredownload has started After the fastboot which happens automatically run firmwaredownloadstatus The total upgrade time is about 10 minutes including the required firmwarecommit This process backs up the primary flash partition to a secondary one Although it takes about 13 minutes to complete the upgrade on a Brocade SAN Switch Module it will be non disruptive Non disruptive means that data traffic continues to flow during the failover or firmware upgrade The fabric state will be saved in non volatile memory and after the reboot all of the fabric services start up first The characteristics of firmware activation on the Brocade SAN Switch Module are that data flow Reads Writes is not delayed at all The Fibre Channel ports and ASICs remain programmed and data continues to flow The Fabric Services Name Server FLOGI etc are paused between 30 50 seconds while the switch goes through its fastboot process and the processor comes back with a new image 1 In the Upload Download tab of the Web Tools Admin interface there is a function button called Firmware Download This button must be chosen The protocol drop down menu only allows the ftp service to be chosen for this switch type The FTP server information must be placed into the following fields User Name Host IP Password and File Name The Fabric OS 4 2 1 code is typically bundled into a zip file for Windows se
167. ll become its own master link and no Trunk Group will form on the Brocade SAN Switch Module For more information about Trunk Groups on other Brocade switch models refer to the Brocade Fabric OS Features Guide On the Brocade SAN Switch Module all trunking ports must meet the following criteria e There can only be one Trunk Group on the Brocade SAN Switch Module consisting of two ports e The port speed must be set to auto negotiate or to 2 Gbit sec the default speed is auto negotiate e If Extended Fabrics is in use the ports must be in LO mode Trunking is supported for normal E_Ports referred to as LO in the portcfglongdistance command with LWL media up to 5km at the full speed permitted by the link With LWL media the throughput begins to fall off beyond 5km due to normal latency effects ISL Trunking does not support the LE L0 5 L1 or L2 portcfglongdistance modes at this time For information about these modes and Extended Fabrics in general refer to Fabric OS Features Guide and the Fabric OS Procedure s Guide On the Brocade SAN Switch Module external ports 0 and 15 are in the same four port grouping or quad as internal ports 9 and 10 The two internal ports 9 and 10 do not have the Trunking parameter enabled and so will not be affected if Trunking is configured on the external ports 0 and 15 Note The diagram in Figure 4 2 shows the possible Trunk Groups on the Brocade 3016 Figure 4 2 Trunk Port Groupings on a Br
168. ll switches in the fabric preventing a single point of failure for zone information Managing Zoning You can manage Zoning using the following methods e Telnet command line interface e Advanced Web Tools e Fabric Manager e Secure Shell e Fabric Access API via third party software All Brocade SAN Switch Modules for IBM eServer BladeCenter have the Advanced Zoning license installed from the factory An example of the Zoning graphical user interface launched from Advanced Web Tools is shown in Figure 3 1 Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 23 Figure 3 1 Advanced Zoning Interface launched from Web Tools File Edit View Favorites Tools Help Ea back gt A search Favorites media lt 4 D 3 A Gi Address amp http 10 64 210 100 ZoneAdmin html Authorization Custom_RS4 GnjAgQ2dkiweoZh3annWD7TP37TbS7FUBE TuouQqtHSP 2F al8PahH2CACegvYoy3HP JFtHcdp47Odme 6o Links gt File Edit View Actions Mixed Zoning Enabled Config IBM_SGU_CFG Create Delete Rename Config Members Alias Zone QuickLoop Fabric Assist Config 5 Zones r A OpenSystems_Zone_1 B fa OpenSystems_Zone_1 Add Member gt h A OpenSystems_Zone_2 B E Aliases A OpenSystems_Zone_3 B E JBOD_1_Sdisks Remove Member Analyze Config H E 20 00 00 04 cf 3b 1c 51 B E IBM_TOP_Host_1 Ej
169. llect the following fibre channel frame statistics e Number of SCSI read write or read write commands e Number of SCSI traffic frames e Number of IP traffic frames Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 33 5 User Defined Filter Based Monitors In addition to the standard filters read write read write frame count and IP count you can create custom filters to qualify frames to gather statistics to fit your needs To define a custom filter use the perfaddusermonitor telnet command Managing Advanced Performance Monitoring You can manage Advanced Performance Monitoring using the following methods e Telnet command line interface e Advanced Web Tools e Fabric Manager e Fabric Watch e SNMP to view but not set e Use Advanced Web Tools to view customize or monitor performance by e Viewing predefined reports for AL_PA end to end and filter based performance monitoring e Creating user definable reports e Viewing performance canvas for application level or fabric level views e Accessing configuration editor save copy edit and remove multiple configurations e Saving persistent graphs across reboots saves parameter data across reboots Predefined Performance Graphs Advanced Web Tools provides graphs to simplify performance monitoring A wide range of end to end fabric LUN device and por
170. lnerability of the overall system An advanced warning helps the administrator plan corrective action in time to restore redundancy A fabric event critical in nature can disrupt the fabric operation For example a failure of an Inter Switch Link ISL between two switches may result in fabric segmentation When a faulty condition is self detected and immediate notification is sent by an appropriate method s corrective action may be initiated before the event escalates Thus a self monitoring system can assist a SAN administrator minimize disruption time Diagnostic utilities are desirable to identify a failing component and possible root cause of the failure A diagnostic utility should be able to run in the background without disturbing the fabric operation When implemented it can minimize the maintenance time significantly Analyzing behavior patterns on a switch port for an extended period can expose a deficiency and or anomaly affecting the overall throughput performance of the SAN Detailed information must be gathered over a period of time using an application which can probe the switch on a periodic basis Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 122 23 SAN Management Scope The SAN Management administrative scope can be broken down into three distinct areas Fabric Switch and Port Fabric administration tasks typically
171. loj x File Edit view Favorites Tools Help Kal lt Back gt amp A seach GaFavortes meda lt 4 D G A Gl Address http 10 64 210 25 switchExplorer html v Go Links A View by Name x Fabric E m 2109 F32 S IBM_3534_1RU IBM_3534 _F03 i Segmented Switches KETTE MMEM M Fee Status Healthy Polled at 04 21 04 03 53 PM Hame brocadessm Fabric OS version v4 24 Domain ID 1 Ethernet IP 10 64 210 25 Ethernet Mask 255 255 240 0 FCnet IP 0 0 0 0 FCnet Mask 0 0 0 0 Gateway IP 10 64 2081 WAH 10 00 00 05 1e 34 00 c7 p A o g Current Zone Config None Status Legend a Healthy _ Marginal i Critical _ Unmonitored E Applet com brocade web switchview SwitchApplet started Internet Figure 20 9 Fabric OS Version in Fabric Manager H Fabric Manager loj xj File Edit View Actions Topology Tools Help Address Fite in 064 210 1007 bad Q e OF Sl F Yv A ID Name y ee i EI E SAN Elements 7 Fiter Switches fi Topology 5 TASA Version Domain ID Supplier Serial Number Serial Number Switch Type Manu 1 5 Fabrics 10 64 210 Principal Online 22 at BRD_12K_sw0_116 BRD_12K_sw0_116 v4 2 0b 10 64 210 100 Subordinate S 1234567 FTO1X800526 Online 10 IBM J brocadesstn BRD_12K_sw1_117 v4 2 0b 10 64 210 101 Subordinate 1171234567 ___FT01x600526 Online 10
172. lthy SAN Zoning does have an impact on SAN designs For an overview of how zoning works refer to the Brocade Zoning User s Guide and the Brocade SilkWorm Design Deployment and Management Guide SAN DDM Version 3 0 these documents also provide guidelines for implementing zoning This section highlights key elements of zoning that relate to a SAN design Zoning and Scalability Zoning optimizes fabric services such as RSCN distribution and name server response and limits unnecessary device discovery With the new zoning and related name server changes in Fabric OS 3 1 and 4 1 and subsequent releases zoning becomes necessary for the proper functioning of large fabrics For instance the distribution of RSCNs registered state change notifications is reduced to only devices affected by a zone change In prior releases of Fabric OS 3 x 4 x and all versions of Fabric OS 2 x a zone activation for example executing the command cf gEnable resulted in an RSCN being distributed to all devices regardless of whether these devices were affected by a zone change Additionally not using zoning results in unnecessary delays during device discovery for some hosts especially when a host pointlessly authenticates with hundreds of devices These delays can last minutes pause ongoing I O and cause unpredictable behavior on a host Use of zoning on the switches limits the number of devices visible to a host and eliminates this host based scalability problem
173. m 3016 Hardware Reference Manual or to the Brocade Enterprise SAN Switch Module for IBM eServer BladeCenter and Brocade Entry SAN Switch Module for IBM eServer BladeCenter Installation Guide for specific information on physically connecting to the switch Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 4 Fabric Management Recommendations Listed next are some recommendations for managing multiple Brocade switches and fabrics e Mixed fabrics In a mixed fabric containing Fabric OS v4 x v3 x and v2 x switches manage the fabric via the switch with the latest Fabric OS version as the first criteria then the most advanced as the second criteria e Multiple connections Switches can be accessed simultaneously from different connections for example Advanced Web Tools CLI API and the IBM eServer BladeCenter Management Module If this happens changes from one connection might not be updated to the other and some modifications might be lost When connecting with simultaneous multiple connections make sure that you do not overwrite the work of another connection e Fabric level tasks A number of management tasks whether executed from the CLI Advanced Web Tools or other management interfaces are designed to make fabric level changes for example the zoning commands When executing fabric level configuration tasks make sure to allow ti
174. me before requiring an acknowledgement back from the receiver If there are not enough frames to fill the pipe then performance degradation may result Another performance factor on long distance links is the response time for SCSI transactions Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 56 Chapter 11 SAN Availability A computer system or application is only as available as the weakest link To build a highly available computer system it is not sufficient to only have a highly available SAN It is necessary to account for availability throughout the entire computer system dual HBAs multi pathing software highly available and multi ported storage subsystems and clustering software are some of the components that may make up such a system This chapter contains the following sections e Fabric Resiliency e SAN Availability Classifications e Redundant Fabric SANs e SAN Topologies Fabric Resiliency Many fabric topologies are available that provide at least two internal fabric routes between all switches that comprise that fabric These topologies are considered resilient because each topology can withstand a switch or ISL failure while the remaining switches and overall fabric remain operational This self healing capability is enabled by the Brocade authored Fabric Shortest Path First FSPF protocol While originally a Brocade only
175. me for the changes to propagate across the fabric before executing any sequential tasks For a large fabric this might be up to a few minutes e Command letter casing In this guide all commands are shown as lowercase Fabric OS v4 x unlike previous versions of the Fabric OS is case sensitive For backward compatibility you can enter the commands using the legacy mixed case notation for example portCf gShow Lowercase is recommended and always works on both v3 x and v4 x High Availability This section provides information on the Brocade SAN Switch Module High Availability HA features Brocade SAN Switch Module HA Features The Brocade SAN Switch Module switches deliver a number of HA features including e Non disruptive firmware download external server to flash e Non disruptive code activation The same mechanisms used to provide non disruptive features on the Silk Worm 3850 3250 are used to perform non disruptive reboots and code activation on the Brocade SAN Switch Module When a reboot or code activation process is initiated a reboot management utility is launched This program creates a standby image of the current switch state The reboot management program then initiates synchronization causing all components of the active image to replicate to the standby image As the state updates are synchronized messages are saved in Flash When all components have finished the synchronization the active image is disengaged
176. mendations and details for setting the Core PID format in the Fabric OS Procedures Caution Guide and consult the Brocade SilkWorm Design Deployment and Management Guide SAN DDM Version 3 0 for more information on the deployment of these settings Setting the Core PID at the initial staging phase on Fabric OS 2 x or 3 x Guideline based switches will allow for a seamless introduction of a switch running Fabric OS 4 x into the SAN fabric Now that the SAN is prepared for Advanced Security Secure Fabric OS SFOS here are some guidelines for planning the implementation Proper planning is essential to create the most secure environment possible It is highly recommended that the checklist of activities in used as a basis for any specific plan which should be followed rigorously Advanced Security Secure Fabric OS Implementation Checklist 1 Ce ND Aw RF wD 10 11 Create a SFOS switch and device list for SFOS policies This list should contain hostnames switch names IP addresses and WWNs Plan the FCS placements for each switch Select a Primary and Backup FCS switch Covertly mark FCS Switches Use a small physical mark so that FCS switches are easily located Determine policy requirements for each device and host Select the SFOS management hosts Brocade SecTelnet and a Secure Shell SSH client may be required Perform a final review of all configuration selections Verify all changes have been included Optional
177. ministrator s Guide Chapter 5 entitled Performance Monitoring Administration e Brocade Fabric OS Procedures Guide Chapter 8 entitled Performance Monitoring Procedures e Brocade Fabric Watch User s Guide information on Performance Monitor Class e Fabric OS Reference Manual CLI commands used for managing performance monitoring e Brocade SilkWorm Design Deployment and Management Guide SAN DDM Version3 0 Chapter 14 entitled Advanced Performance Monitor Overview under SAN Management Note that in terms of Advanced Performance Monitoring behavior the Brocade SAN Switch Module behaves exactly like the SilkWorm 3850 3250 as may be described in the above publications Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 38 Chapter 6 Extended Fabrics Brocade Extended Fabrics is an optionally licensed product that enables inter switch links ISLs to extend up to 100 km This is achieved by optimizing the internal buffering algorithm used by Brocade switches It provides maximum buffering between E_Ports connected over an extended distance The buffer reconfiguration results in line speed performance of 2 Gbit sec for switches interconnected at up to 60 km at the moment and of 1 Gbit sec for switches interconnected at up to 100km at the moment The Extended Fabrics feature is provided with the Fabric OS and can be activa
178. n e Link utilization for each hop from source to destination PathInfo Examples As designed pathinfo is intended to gather information on a specific data stream not the entire fabric The simplest example uses a destination domain as an argument When used this will provide routing information from the embedded port on the local switch to the embedded port on the remote switch domain The example below shows the command and expected output for the destination domain of 32 Each hop is shown along the way to the destination ess03l admin gt pathinfo 32 1 13 Target port is Embedded Hop In Port Domain ID Name Out Port BW Cost 0 E 31 ess031 9 1G 1000 1 23 93 ess093 5 2G 500 32 ESS032 E Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 117 22 Another common usage is to gather the routing information from a source port on a local switch to a destination port on a remote switch This requires three arguments the source domain the source port and the destination port To use the embedded port use 1 The example below uses a source domain of 32 it s embedded port and a destination port of 13 To determine more complex pathing it is recommended to use a diagram of the fabric with labeled ISL connections and device attachments This allows arguments to be quickly determined If the appropriate information is not entered pr
179. n configured for a fixed 2Gbit speed for all of the internal ports as that is the only supported Fibre Channel Expansion Card configuration on the server blades This can be identified in the CLI portcfgshow command or by viewing the Port Setting pane of the Switch Admin window of Web Tools As you can see in Figure 4 3 Port Number 0 has a check mark on the Enable Trunking box and a value of Negotiate in the Change Speed box Port Number 1 does not have a check mark on the Enable Trunking box and has a value of 2G in the Change Speed box Figure 4 3 Default Port Settings on the Brocade SAN Switch Module yj Switch Admin Microsoft Internet Explorer wy i 5i x SwitchName brocadessm DomainiID 1 WAAN 10 00 00 05 1e 34 00 c7 Fri Mar 26 2004 3 08 PM Switch Information Network Config Upload Download SNMP License Admin Port Setting Routing Extended Fabric Configure Trunk Information Persistent Enable Current Change Disable Trunking Speed Speed 0 CT Iv Vv No_Light N2 Negotiate Y fExt0 1 O v E Online 2G 2G Bay1 2 E Iv r Online 2G 2G jBay2 3 O IV Cr No_Light 2G 2G jBay3 4 E M O No_Light 2G 26 Bay4 5 a V E No_Light 2G 2G Y jBay5 6 O IV r No_Light 2G 2G Y jBay6 7 E Iv O No_Light 2G 2G Y Bay 8 O IV O No_Light 2G 2G Bays 9 O v E No_Light 2G 2G fBay9 10 T Vv O No_Light 2G 2G Y Bay10 11 M z No
180. n to construct topology relationships In addition the basic configuration services provided by the management server allow certain attributes associated with switches to be obtained and in some cases modified For example logical names identifying switches can be registered with the Management Server For more information refer to the Fabric OS Procedures Guide Note The Management Server is disabled if the switch is in secure mode Refer to the Secure Fabric OS User s Guide for more information Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 21 Chapter 3 Advanced Zoning This chapter contains the following information e Overview of Advanced Zoning e Other Aspects of Advanced Zoning e Using Zoning to Administer Security e Zoning Architecture e Managing Zoning Overview of Advanced Zoning Advanced Zoning is a licensed Brocade product that allows partitioning of a storage area network SAN into logical groupings of devices that access each other These logical groupings are called zones You can use Advanced Zoning to customize environments and optimize resources e Customize Environments You can use zones to create logical subsets of the fabric to accommodate environments such as closed user groups or functional areas within the fabric For example you can identify selected devices within a zone for the exclusive u
181. naging trunking Brocade SilkWorm Design Deployment and Management Guide SAN DDM Version3 0 Section 2 3 entitled Trunking Section 2 4 entitled ISL Over subscription Ratios and Section 3 1 1 entitled Trunk and ISL Connections under SAN Design Note that in terms of Trunking behavior the Brocade SAN Switch Module behaves similarly to the SilkWorm 3850 3250 as may be described in the above publications except where noted in this section Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 30 Chapter 5 Advanced Performance Monitoring This chapter contains the following sections e Overview of Advanced Performance Monitoring e Advanced Performance Monitoring Architecture e Managing Advanced Performance Monitoring Overview of Advanced Performance Monitoring Advanced Performance Monitoring is an optionally licensed product used for monitoring the performance of networked storage resources This tool helps reduce over provisioning while enabling SAN performance tuning and increasing administrator productivity The Advanced Performance Monitoring feature is provided with the Fabric OS and can be activated by entering a license key available from the switch supplier Advanced Performance Monitoring provides SAN performance monitoring through an end to end monitoring system with the following benefits e Increased end to
182. nce Monitor Fabric Manager provides a graphical interface for monitoring and managing multiple IBM eServer BladeCenters in multiple fabrics comprised of Brocade switches from a standard workstation The GUI simplifies task administration at the fabric switch and port levels in a medium to large size Brocade SAN environment Fabric Manager can be used to manage multiple switch fabrics in addition to individual Brocade switches It provides consolidated high level information about all the switches in the fabric launching the Advanced Web Tools application when more detailed information is required for a particular switch The launching of Advanced Web Tools is transparent providing a seamless user interface and experience Fabric Manager is installed on a workstation All switches in the fabric are represented in the main window of Fabric Manager but only those with an Advanced Web Tools license can be managed through Fabric Manager Fabric Manager is the complete SAN management tool for Brocade SANs providing the following advantages to administrators e A highly scalable Java based application that manages multiple switches and multiple fabrics in real time e Time savings by enabling global integration and execution of processes across multiple fabrics through a single point SAN management platform e More effective management by providing rapid access to critical SAN information Brocade Enterprise and Entry SAN Switch Modules for IBM eS
183. nd 5 3 Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 35 Figure 5 2 Basic Monitoring Graphs provided by Advanced Web Tools A z Eas Performance Monitor brocadessm Dien Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 36 Figure 5 3 Advanced Performance Monitoring Graphs provided by Advanced Web Tools g Performance Monitor brocadessm a Oj x Actions Performance Graphs Basic Monitoring eas Utilization Ref rate 30 Secs Advanced Monitoring SID DID Performance SCSI Commands SCSI Read Write on a Lun per port SCSI vs IP Traffic SCSI Read on a Lun per port ALPA Error SCSI Write on a Lun per port os SCSI Read Write per port SCSI Read per port SCSI Write per port Java Applet Window A real life example of using Port Throughput from the Basic menu option and SID DID Performance from the Advanced Performance Monitoring menu is shown in Figure 5 4 In this example Ports 0 and 15 are Trunked and each sending a shared load of approximately 60 MB s each Trunk sending 30MB seen in the top graph There are two Internal Server Blades each talking to their own Disk in a Fibre Channel JBOD that exists on another switch in the SAN The SID of the Server in Bay 1 P
184. ndex ThresholdName BehaviorType BehaviorInt 0 eportRXPerf000 Triggered 1 15 eportRXPerf015 Triggered 1 Threshold boundary level is set at Default Default Custom Unit KB s KB s Time base Low 120000 120000 High 220000 220000 BufSize 0 0 Threshold alarm level is set at Custom Errlog 1 SnmpTrap 2 PortLogLock 4 RapiTrap 8 EmailAlert 16 Valid alarm matrix is 31 Default Changed Exceeded Below Above InBetween change behavior type change behavior interval change threshold boundary level change custom unit change custom time base change custom low change custom high change custom buffer apply threshold boundary changes 10 cancel threshold boundary changes Select choice gt 1 19 19 change threshold alarm level change changed alarm change exceeded alarm change below alarm change above alarm change inBetween alarm apply threshold alarm changes cancel threshold alarm changes return to previous page AYA FPWNHE OMDANAOFPWNHE oO These configuration settings are now saved to the switch and will be visible in the configshow output under the thresh variables as shown in Figure 26 5 Figure 26 5 Fabric Watch parameters in configshow output brocadessm USERID gt configshow thresh thresh cust eport RXPerf above 3 thresh cust eport RXPerf below 3 thresh eportRXPerf actLevel 2 The WebTools Fabric Watch interface can also accomplish the same set of tasks Use th
185. ne or more zones Every zone must have at least one zone object Empty zones are not allowed Zone configurations are consistent across reboots and power cycles If two switches are connected in a fabric they can become isolated for example due to an ISL failure however when rejoined they maintain the same fabric configuration unless one of the switches has had a configuration change In a non Secure Fabric OS environment the Zoning database is distributed to every switch in the fabric by the Principle switch Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 22 Using Zoning to Administer Security Zones can provide controlled access to fabric segments and establish barriers between operating environments They isolate systems with different uses protecting individual systems in a heterogeneous environment for example when zoning is in secure mode no merge operations occur In an Advanced Security Secure Fabric OS environment Brocade Zoning is managed from the primary Fabric Configuration Server FCS The primary FCS switch makes zoning changes and other security related changes The primary FCS switch also distributes zoning to all other switches in the secure fabric All existing interfaces can be used to administer zoning depending on the policies refer to the Secure Fabric OS User s Guide for information about security policies
186. ng environment by using I O measurement tools such as iostat and sar UNIX or diskperf Microsoft Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 66 Chapter 14 ISL Trunking This chapter contains the following sections e ISL Trunking e ISL Over subscription ratios e Recommended ISL Over subscription ratios Brocade ISL Trunking is a feature that enables traffic to be optimally shared across available inter switch links ISLs while preserving in order delivery A trunk group logically joins two three or four ISLs into one logical ISL on the Brocade SAN Switch Module the limit is two ISLs in one trunk group as there are only two external ports on a Brocade SAN Switch Module Use of trunking can minimize or eliminate congestion in the SAN because trunking optimizes ISL utilization The use of trunking minimizes the effort of managing a SAN since ISLs are now managed as a group instead of individually and optimizes FSPF performance as FSPF does not have to compute as many routes The ISLs of the Brocade SAN Switch Module in the IBM eServer BladeCenter can be trunked and will provide additional performance for the 14 internal Server Blades of the IBM eServer BladeCenter It is recommended when using trunking to incorporate the use of this feature into a dual redundant fabric configuration This provides the best availability of the fabric i
187. ng specifically what storage is needed for the software application that will be in production To understand this the number and size of LUN presentations on each storage array Fibre Channel port must be clearly defined e Be sure to adhere to recommended storage configurations by the switch vendor for LUN masking and other storage specific features Some vendors may recommend using a separate HBA for tape devices For this case define the zoning configuration to fence off tape devices from any other HBAs which see the disk storage in the same host e Keep in mind the different OS platforms backup application requirements and the number of paths to each LUN which may drive the zoning plan There may be specific host requirements for storage value added feature sets such as Server Free backup LUN snapshots or LUN mirroring over distance e Asa general rule have overlapping zones in all cases An overlapping zone has the HBAs share one or more storage ports but with the HBAs separate from each other This is sometimes referred to as a single initiator zoning Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 82 19 For a large number of zone configurations generally over 15 use Fabric Manager or Web Tools These tools vastly simplify the zoning Guideline implementation WWNs of devices and the ports on the switch they are attached to
188. ngle core IBM BladeCenter ee is Trunk the Performance Brocade Brocade ISLs for Switch Module Switch Module Performance Vv YY La Ww y ZT YY Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 70 14 Figure 14 6 Connecting to a fabric with a dual core Option 2 ae ue IBM Trunk the plit the ISLs ISLs for for Availability BladeCenter an Brocade Brocade Sronnance Switch Module Switch Module Ife A AAA AZ w NY Oo VN yw Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 71 Chapter 15 Architecting SANs With SilkWorm Switches This chapter contains the following sections e Architecting SANs with SilkWorm Switches e Device Attachment Strategies e Brocade SAN Switch Module Design Considerations Many SAN related elements such as device attachment strategies platform specific topics switch location in the fabric Zoning Advanced Security Secure Fabric OS Extended Fabrics and supportability have some bearing on the design of a SAN It is these more in depth topics and the usage of SilkWorm switches in a SAN architecture that are discussed in this section The Core Edge topology is identified as a reference topology for the guidelines presented in this chapter The Core Edge topology is pr
189. nse Key Add Remove Chose Refresh Switch Administration opened Tue Mar 23 2004 4 16 PM Add or remove feature licenses Figure 20 17 New Licenses listed in WebTools A Switch Admin Microsoft Internet Explorer SwitchName brocadessm DomainID 1 WAAN 10 00 00 05 1e 34 00 c7 Fri Mar 26 2004 3 08 PM Switch Information Network Config Upload Download SNMP License Admin Port Setting Routing Extended Fabric Configure Trunk Information S9eyybz5QdTAfzzw Zoning ae Rz5SeyQey9b50Tfzp Web SyRcRydee9jz5d5B Security RQRzSScbeySRSOTo Remote Switch ReyyddceSRSdzzeS Extended Fabric SyRcRydee9xz5dSz Performance Monitor zl New License Key Add Remove close Refresh Switch Administration opened Fri Mar 26 2004 3 07 PM Add or remove feature licenses m Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 100 20 9 Optional Port Configurations It is recommended to maintain the existing Port Configurations settings on the Brocade SAN Switch Module The Internal Port Configuration Settings for Ports 1 14 should never be changed If for some reason some of the external Port Configuration settings need to be changed they can be changed from the WebTools interface However the CLI interface provides a more comprehensive set of port configuration commands th
190. nt access e Specify which switches and devices can join the fabric e View statistics related to attempted policy violations e Manage the fabric wide Advanced Security Secure Fabric OS parameters through a single switch e Create temporary passwords specific to a login account and switch e _ Enable and disable Advanced Security Secure Fabric OS as desired Advanced Security Secure Fabric OS uses digital certificates based on PKI Public Key Infrastructure to provide switch to switch authentication Advanced Security Secure Fabric OS can be used to increase the security of the local and remote management channels including Fabric Manager Web Tools standard SNMP applications Management Server and a supported command line interface CLI client such as sec telnet The access through a channel can be restricted by customizing the Advanced Security Secure Fabric OS policy for that channel Advanced Security Secure Fabric OS policies are available for telnet includes sec telnet and Secure Shell SNMP Management Server HTTP and API Fabric Manager Web Tools and API all use both HTTP and API to access the switch To use any of these management tools to access a fabric that has Secure Mode enabled ensure that the workstation computers can access the fabric by both API and HTTP If an API or HTTP policy has been created it must include the IP addresses of all the workstation computers After a digital certificate has been installed
191. nt and received These variables are also known as managed objects All managed objects are contained in the MIB When passive the SNMP management station receives an unsolicited message trap from the switch agent if an unusual event occurs The Brocade SNMP agent can receive queries from one or more SNMP management stations and can send traps to up to six SNMP management stations The Brocade SAN Switch Module has two customized SNMP parameter settings The first parameter that has been customized is the sysDescr OID of the MIB II RFC1213 MIB The typical value for all Brocade switch products is Fibre Channel Switch It has been customized to Brocade SAN Switch Module for IBM eServer BladeCenter This can be identified in the CLI agtcfgset command or by viewing the SNMP pane of the Switch Admin window of Web Tools See Figure 2 14 The second parameter that has been customized is the sysObjectID OID of the MIB II RFC1213 MIB Its customized value is 1588 2 1 1 22 and is required for IBM Director integration Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 19 Figure 2 14 SNMP configuration settings zax SwitchName brocadessm DomainiD 1 WWAN 10 00 00 05 1e 34 00 c7 Fri Mar 26 2004 3 08 PM License Admin Port Setting Routing Extended Fabric Configure Trunk Information Switch In
192. number or port IDs are not supported in interopmode When there is no zoning configuration in effect the default effective configuration is all ports are isolated and traffic is not permitted This is in contrast to the Brocade standard behavior when interoperability mode is off where all data traffic is enabled Web Tools can be used for zone configuration as long as Brocade switches are directly connected to each other If Web Tools is used to setup zoning then Web Tools must be used as the only zone management method Brocade switches behind a McDATA switch will only receive the effective configuration when a zone merge occurs This is because McDATA only has an effective configuration and will discard the defined configuration when it sends merge info to the Brocade switch However a zone update will send both defined and effective configurations to all switches Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 49 When a Brocade switch is reconfiguring do not perform any zoning commands that are supposed to propagate until the fabric routes are fully set up Use the fabricShow command to verify that all of the fabric routes are set up and all of the switches IP addresses and names are present This does not apply to McDATA as it will only show the WWN and domain ID The maximum number of items that can be stored in the zoning confi
193. o trap recipient configured yet Community 6 FibreChannel ro No trap recipient configured yet SNMP access list configuration Entry 0 No access host configured Entry 1 No access host configured Entry 2 No access host configured Entry 3 No access host configured Entry 4 No access host configured Entry 5 No access host configured Committing configuration done Brocadessm USERID gt Figure 25 3 modified agtcfgshow settings brocadessm USERID gt agtcfgshow sysDescr Brocade SAN Switch Module for IBM eServer BladeCenter sysLocation End User Premise sysContact Field Support swEventTrapLevel 5 authTraps 1 ON SNMPvl community and trap recipient configuration Community 1 Secret COde rw No trap recipient configured yet Community 2 OrigEquipMfr rw No trap recipient configured yet Community 3 private rw No trap recipient configured yet Community 4 public ro Trap recipient 192 168 2 10 Community 5 common ro No trap recipient configured yet Community 6 FibreChannel ro No trap recipient configured yet SNMP access list configuration Entry 0 No access host configured Entry 1 No access host configured Entry 2 No access host configured Entry 3 No access host configured Entry 4 No access host configured Entry 5 No access host configured brocadessm USERID gt Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publicat
194. ocade SAN Switch Module System status LEDs Port 0 LEDs Poto Ports 0 amp 15 Port 15 I Port 15 LEDs 000 ISL Trunking mode is enabled by default for ports 0 and 15 on the Brocade SAN Switch Module For instructions on disabling and enabling trunking capability for Note individual ports or all the ports on the switch refer to the Fabric OS Procedures Guide The Trunking mode is enabled by default but a Trunking license is still required Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 26 4 To facilitate ease of deployment and configuration the Brocade SAN Switch Module has customized port settings The External Ports 0 and 15 have been configured to auto negotiate their Link Speed and have their Trunking parameter enabled A Trunking license must be installed on the Brocade SAN Switch Module as well as a remote Brocade switch Provided the above guidelines have been followed the switches will automatically create the appropriate Trunking Groups when connected These settings are the same on all Brocade switch products For the internal 14 server connected ports the Trunking parameter has been disabled by default because these ports will never be connected to another switch they will only exist as server connected ports In addition the Link Speed has bee
195. ocadessm USERID gt fwconfigure Environment class SFP class Port class Fabric class E Port class F FL Port Copper class F FL Port Optical class Alpa Performance Monitor class EE Performance Monitor class Filter Performance Monitor class Security class Switch Availability Monitor class Quit lect a class gt 1 13 13 5 oma tA OfFWNE Link loss Sync loss Signal loss Protocol error Invalid words Invalid CRCS RXPerformance TXPerformance State Changes return to previous page Select an area gt 1 10 10 7 Index ThresholdName LastEvent 0 eportRXPerf000 LasteventTime Status CurVal LastVal enabled below Fri Apr 16 16 49 38 2004 15 eportRXPerf015 enabled LastState Informative Informative below Fri Apr 16 16 49 38 2004 1 refresh 2 disable a threshold 3 enable a threshold 4 advanced configuration 5 return to previous page S elect choice gt 1 5 5 4 Index ThresholdName 0 eportRXPerf000 15 eportRXPerf015 BehaviorType BehaviorInt Triggered 1 Triggered 1 Threshold boundary level is set at Default Default Custom Unit KB s KB s Time base LOW 120000 120000 High 220000 220000 BufSize 0 0 Threshold alarm level is set at Default Errlog 1 SnmpTrap 2 PortLoghock 4 RapiTrap 8 EmailAlert 16 Valid alarm matrix is 31 Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management G
196. odule admin user account login is Note USERID with a password of PASSWORD The 0 is the number zero and not the letter O Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 93 20 Figure 20 5 Login change password Challenge brocadessm login USERID Password Please change your passwords now Use Control C to exit or press Enter key to proceed Warning Access to the Root and Factory accounts may be required for proper support of the switch Please ensure the Root and Factory passwords are documented in a secure location Recovery of a lost Root or Factory password will result in fabric downtime for user root Changing password for root Enter new password Re type new password passwd all authentication tokens updated successfully Please change your passwords now for user factory Changing password for factory Enter new password Re type new password passwd all authentication tokens updated successfully Please change your passwords now for user user Changing password for user Enter new password Re type new password passwd all authentication tokens updated successfully Saving passwords to stable storage Passwords saved to stable storage successfully brocadessm USERID gt If you forget your passwords the IBM eServer BladeCenter Management Module has the capability
197. on I e Chapter 1 Product Introduction introduces the Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter also known as the Brocade SilkWorm 3016 e Chapter 2 Switch and Fabric Management explains the different methods used to manage a Brocade SilkWorm switch and a Brocade SAN e Chapter 3 Advanced Zoning describes the Advanced Zoning licensed feature e Chapter 4 ISL Trunking describes the ISL Trunking licensed feature e Chapter 5 Advanced Performance Monitoring describes the Advanced Performance Monitoring licensed feature e Chapter 6 Extended Fabrics describes the Extended Fabrics licensed feature Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 vi Chapter 7 Remote Switch describes the Remote Switch licensed feature Chapter 8 Brocade Advanced Security Secure Fabric OS describes the Brocade licensed feature for security Chapter 9 Brocade Interoperability Mode describes heterogeneous fabric connectivity Section II SAN Design Chapter 10 SAN Solutions introduces SAN solution concepts Chapter 11 SAN Availability introduces SAN Availability concepts Chapter 12 SAN Scalability introduces SAN Scalability concepts Chapter 13 SAN Performance introduces SAN Performance concepts Chapter 14
198. on the switch Fabric OS v2 6 2 v3 1 2 v4 2 0 and v4 2 1 or higher encrypt sec telnet passwords automatically regardless of whether Advanced Security Secure Fabric OS is enabled Advanced Security Secure Fabric OS uses digital certificates based on public key infrastructure PKI and switch WWNs to identify the authorized switches and prevent the addition of unauthorized switches to the fabric A PKI certificate installation utility PKICERT is provided for generating certificate signing requests CSRs and installing digital certificates on switches Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 43 Advanced Security Secure Fabric OS Architecture Fabric Configuration Server Switches Fabric Configuration Server FCS switches are one or more switches that are specified as trusted switches switches that are in a physically secure area for use in managing Advanced Security Secure Fabric OS These switches should be both electronically and physically secure At least one FCS switch must be specified to act as the primary FCS switch and one or more backup FCS switches are recommended to provide failover ability in case the primary FCS switch fails FCS switches are specified by listing their WWNs in a specific policy called the FCS policy The first switch that is listed in this policy and participating in the fabric acts as t
199. ons such as temperature on the Brocade SAN Switch Module e SFP behavior such as temperature current voltage and transmit and receive power serial diagnostic SFPs only e Port behavior such as state changes errors and performance e Fabric events such as fabric reconfigurations segmentations and zone changes e Advanced Performance Monitoring provided a license key has been purchased and installed e Security such as management violations security policy violations and login violations provided a license key has been purchased and installed e Switch Availability such as down time up time With Fabric Watch software you can place limits or thresholds on the behavior of different switch and fabric elements Fabric Watch then monitors these behavior variables or counters and issues an alarm to address problems when a counter exceeds a threshold An alarm can send a message to the Event Log Error Log SNMP trap Fabric Access API E mail or it can even lock the port log depending on how you configure the alarm All Brocade SAN Switch Modules for IBM eServer BladeCenter have the Fabric Watch license installed from the factory An example of the Fabric Watch management tab in the Advanced Web Tools Interface is shown in Figure 2 10 Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 15 Figure 2 10 Fabric Watch
200. operly pathinfo may provide error messages ess03l admin gt pathinfo 32 1 13 Target port is F Port Hop In Port Domain ID Name Out Port BW Cost 0 E 31 ess031 9 1G 1000 I 23 93 ess093 5 2G 500 32 ESS032 T3 1G An example of this is shown next If no devices are attached to the destination port being used as the argument the error message displayed is target port not active Normally this does not mean that no frames are being passed to the device or that it is not online it simply means no device is attached If a device is attached this may mean the device is not online ess031 admin gt pathinfo 32 1 14 Target port not active Hop In Port Domain ID Name Out Port BW Cost 0 E 31 ess031 9 1G 1000 I 23 93 ess093 5 2G 500 2 6 32 ESS032 14 1G S For determining complex path information quickly and easily create or use a diagram of the SAN fabric with labeled ISL connections and device locations This simplifies the process of gathering the appropriate pathinfo arguments Use 1 as the argument for an embedded switch port Guideline This next example shows how to display the return path Use the r option as shown Depending on the fabric topology it may not be the same as the outbound path The following example shows that the reverse path in this case is equivalent essQ3l admin gt pathinfo r 32 1 13 Target port is F Port Hop In Port Domain ID Name Out Port BW Cost 0 E 31 ess031 9 1G 1000
201. ore Trunking groups form only for the ISLs that have lengths that differ by less than 400 meters When connecting two switches with two or more ISLs ensure that all trunking requirements are met to allow a trunking group to form Trunking groups can be used to resolve ISL over subscription if the total capability of the trunking group is not exceeded SWL and LWL fiber optic cables and transceivers can be used in the same trunking group Managing ISL Trunking You can manage Trunking using the following methods Telnet command line interface Advanced Web Tools Fabric Manager SNMP to view but not set The Brocade ISL Trunking license must be purchased from your switch supplier An example of the Trunking interface launched from Advanced Web Tools is shown in Figure 4 4 Visibility to existing Trunk groups and connections can be seen in the Topology view of Fabric Manager Figure 4 5 Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 28 Figure 4 4 Trunking Interface launched from Web Tools Switch Admin Microsoft Internet Explorer SwitchName brocadessm DormainiD 4 WAAN 10 00 00 05 4 34 00 c7 Tue Apr 13 2004 5 17 PM Switch Information Network Config Upload Download SNMP License Admin Port Setting Routing ExtendedFabric Configure Trunk Information Trunk Group Close
202. ort 1 is 0x010100 and the Disk it is talking to on a remote switch has a DID of 0x0307ba This is shown in the middle graph This server is communicating with the Disk at about 35 MB s The SID of the Server in Bay 2 Port 2 is 0x010200 and the Disk it is talking to on a remote switch has a DID of 0x0307bc This is shown in the middle graph This server is communicating with the Disk at about 25 MB s Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 37 Figure 5 4 Advanced Performance Monitoring Graphs in Web Tools amp Performance Monitor brocadessm J lej x Actions Performance Graphs Switch Throughput Utilization Ref rate 30 Secs VUVUVUVVUVVUVUVVUUUD SSeS S SSO SNS AONSS MAONO gooooooooooooygg Repocoososcoone ENN RNANN ANANN S OS 25000000 00000222 N NNN O00 S S A a ol ka o Ki a A 1 6 18 2 06 SID D10 Performance Sid 010100 Did 0307ba Port 1 Ref rate 30 Transmitted Bytesisec Received Bytes sec 10G SID O10 Performance Sid 010200 Did 0307bc Port 2 Ref rate 30 Transmitted Bytes sec Received Bytes sec 10G Java Applet Window For more detailed information on Advanced Performance Monitoring refer to the following publications and material e Brocade Fabric OS Features Guide Chapter 3 e Brocade Advanced Web Tools Ad
203. ote switch Switch management is routed through the Fibre Channel connection e Support for inter switch links ISLs Sites requiring redundant configurations can connect multiple E_Ports to remote sites by using multiple gateways Standard Fabric OS routing facilities automatically maximize throughput and provide automatic failover during interruption on the WAN connection The Remote Switch feature operates in conjunction with a gateway The gateway provides an E_Port interface that links to the SilkWorm E_Port After the link between the two E_Ports has been negotiated the gateway E_Port moves to pass through mode and passes fibre channel traffic from the SilkWorm E_ Port to the WAN The gateway accepts fibre channel frames from one side of a Remote Switch fabric transfers them across a WAN and passes them to the other side of the Remote Switch fabric Remote Switch is automatically activated when you enable the license key The only required action is to connect the fabrics through the gateway device and make sure that the configure parameters are compatible with the gateway device The Brocade Remote Switch license must be purchased from your switch supplier For more detailed information on Remote Switch refer to the following publications and material e Brocade Fabric OS Features Guide Chapter 7 e Brocade Fabric OS Procedures Guide Chapter 6 entitled Distributed Fabrics Procedures e Fabric OS Reference Manual CLI commands
204. out Value a Optional Disabling the telnet daemon when secure mode is enabled Fabric OS 4 1 only 12 Optional Customize Monitoring Features a Switch Status Policy b Track Changes c SNMP Traps d Fabric Watch 13 Check Environmental Status 14 Baseline and backup the switch configuration Before getting started gather the switch spreadsheet put together during the planning phase This shows the planned IP addresses and domain names as well as the switch roles Core Switch Edge switch Management Switch etc Once the IP addresses and domain numbers are known it is just a matter of executing the appropriate commands to set these values on the associated switch Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 90 20 Switch Staging Steps 1 Set IP Address The Ethernet IP Address Ethernet Subnet Mask and Gateway IP address should not be configured using local mechanisms on the Brocade SAN Switch Module such as the ipaddrset CLI command or Advanced Web Tools The values must be configured using the IBM eServer BladeCenter Management Module because all IP Ethernet access to the switch module itself is forwarded through the Management Module If the switch module s IP address information is changed without changing the Management Module configuration then telnet access to the switch may be lost This restriction does not
205. ow command it is located in the Serial Num field o From a Web Tools session Click on the Info tab it is located in the Supplier Serial field o From Fabric Manager Click on the Switches tab it is located in a column titled Supplier Serial Number o From the IBM eServer BladeCenter Management Module Click on the Hardware VPD tab it is located in a column titled FRU Serial No 3 License ID Information Required to obtain optional licenses SilkWorm 3016 Provide the license ID It can be obtained o From a telnet session to the switch use the licenseidshow command to display the license ID o From a Web Tools session Click on the Info tab it is located in the LicenseID field Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 XIV Document Feedback Because quality is our first concern at Brocade we have made every effort to ensure the accuracy and completeness of this document However if you find an error or an omission or you think that a topic needs further development we want to hear from you Forward your feedback to documentation brocade com Provide the title and version number and as much detail as possible about your issue including the topic heading and page number and your suggestions for improvement Brocade Enterprise and Entry SAN Switch Module
206. ownload Configuration upload download SNMP setup License Administration Fabric parameter configuration Routing DLS and IOD setup Zoning Management Name Server Displays Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 123 23 Port Administration Web Tools or CLI e Port Configuration e Fxtended Fabric configuration e Trunking information Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 124 Chapter 24 Brocade SAN Switch Module Management Tools Brocade s approach to SAN Management is fabric centric The main emphasis is to simplify the entire SAN management process by simplifying the three levels of management for Brocade switches from a centralized management server Brocade has developed and refined a rich command line interface for advanced users and a simple to use Web Tools interface for new users and advanced users alike Licensed fabric health monitoring tools including Brocade Fabric Watch and Advanced Performance Monitoring assist SAN administrators with SAN monitoring and improving the operating environment As SAN configurations grow larger and more complex managing them using multiple management tools has become very challenging Brocade Fabric Manager was introduced to assist users by simplifying the entire mana
207. pper class o _F FL Port Optical class o Multiple port classes let you set thresholds for different types of ports Fabric Watch monitors the behavior of the port for state changes link issues quality of connection and optimal performance You can configure separate and unique thresholds for E Ports and for F FL Ports e Advanced Performance Monitor class The performance monitor class appears as the following three separate classes o ALPA Performance Monitor class o End to End Performance Monitor class o Filter Performance Monitor class O The advanced performance monitor classes serve as tuning tools Advanced Performance monitor classes group areas that track the source and destination of traffic You can use advanced performance monitor class thresholds and alarms to determine traffic load and flow and to reallocate resources appropriately e Security class The security class monitors all attempts to breach your SAN security helping you fine tune your security measures e Switch Availability Monitor class The switch availability monitor class monitors the efficiency of all active ports providing a measure of switch availability It provides statistics on switch downtime and uptime to help you identify problems with ports Check the synchronization status of a port by running portShow and viewing the portPhys content Fabric Watch Setup The Fabric Watch feature will already be loaded on your Brocade SAN Switch Module This can be
208. pported by Fabric Access API v3 0 2 or higher An example of a product that uses the Fabric Access API is IBM Tivoli Storage Area Network Manager For more information on the Brocade Fabric Access API refer to the following web site link http www brocade com products fabric_access_api jsp Management Server The Fabric OS includes a Distributed Management Server The Management Server MS allows a SAN management application to retrieve information and administer the fabric and interconnected elements such as switches servers and storage devices The Management Server is located at the Fibre Channel well known address FFF FF Ah The implementation of the Management Server provides the following management services Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 20 2 Fabric Configuration Service Provides basic configuration management for topology information referred to as Topology Discovery Unzoned Name Server access Provides a management view of the Name Server information for all devices in a fabric regardless of the active zone set Fabric Zone Service FDMI The services provided by the Management Server assist in the auto discovery of switch based fabrics and their associated topology A client of the Management Server can determine basic information regarding the switches in the fabric and use this informatio
209. protocol the standards bodies have accepted FSPF as the standard protocol for Fibre Channel fabric routing Figure 11 1 depicts the failure of a switch in a Cascade topology Switches A and B are unable to communicate with the remaining switches when the switch marked with the X fails resulting in the fabric segmenting into three separate fabrics However a switch failure in a Core Edge or other resilient topology fabric does not cause a loss of communication with the remaining switches as shown in Figure 11 2 If switch B fails switch A can still communicate with switch C through the alternate path indicated by the arrows The fail over to alternate paths is effectively transparent to the attached devices This fail over is performed by FSPF which automatically reroutes the data around the failure Figure 11 1 In a Cascaded Topology there is no resilience to Fabric failures Failed Switch Switch A N Nv Switch B Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 57 11 Figure 11 2 In a Core Edge Topology there is resilience to Fabric failures Switch A Ww Failed Switch resilient data path NY Switch B Ww N XY Switch C SAN Availability Classifications Devices attached to a fabric may require highly reliable access to support applications such as storage consolidation server clus
210. pts If you are already familiar with the licensed products described in this guide you might want to refer to the Brocade Fabric OS Features Guide the Brocade Fabric OS Procedures Guide or the Brocade Web Tools Administrator s Guide to learn how to use the products in more detail The second section Section II is focused on covering detailed how to information for the Brocade SAN Switch Module from a design deployment and management perspective This portion of the Design Deployment and Management guide is intended to be used in conjunction with existing Brocade manuals release notes and related Brocade publications especially the Brocade SilkWorm Design Deployment and Management Guide SAN DDM Version 3 0 The document contains the following components e The title page and Table of Contents provide the version number date and part number of this publication copyrighted information contained in the document Brocade Communications Inc locations around the world a document history table and the topics covered in this particular document e About This Document provides information specific to this document how it is organized what information has changed since its most recent publication the typographic conventions and particular terminology that it uses where to go for further information on the topic how to get technical assistance with your product and how to provide your feedback about this document Secti
211. r Design Deployment and Management Guide Publication Number 53 0000561 01 41 Chapter 7 Remote Switch Remote Switch is an optionally licensed product that enables you to connect two remote Brocade fabrics over an IP network enabling communication of IP or ATM protocols as well as fibre channel traffic The Remote Switch feature is provided with the Fabric OS and can be activated by entering a license key available from the switch supplier The Brocade Remote Switch feature functions with the aid of a bridging device or fibre channel gateway The gateway supports a fibre channel physical interface and a secondary non fibre channel physical interface such as IP SONET or ATM Remote Switch functions over E_Port connections With Remote Switch on both fabrics the gateway accepts fibre channel frames from one fabric tunnels them across the network and passes them to the other fabric From the viewpoint of the connected hosts and storage devices fabrics using Remote Switch interact the same as locally connected switches Remote Switch provides many of the same capabilities of normal ISL links including e Coordinated fabric services The Remote Switch fabric configuration fully supports all fabric services including Distributed Name Service Registered State Change Notification and Alias Service e Distributed management Management tools such as Advanced Web Tools Fabric OS and SNMP are available from both the local switch and the rem
212. r ereneenr cre tecnmnrerer rrr rea 16 SN T me P E E EE E TTA 19 Brocade Fabric Access API c sccccccsssseeeeeeceeeeeeeeeseenseeeeees 20 Management Server ccccccssscccccssssseceesseesseeeeeeseeeeeeeeessasseeeenes 20 ACVANCCO ZOMMG season ae a EA EEEE EEE 22 Overview of Advanced Zoning ccccceeeeeeeeeeaeeeeeeeeeeneeeeeeees 22 Other Aspects of Advanced ZONING cccceeseeeeeeeeeaeeeeeeeeees 22 Using Zoning to Administer Security ccceeeeeeeeeeeeeeeeeeeeeees 23 Zoning AN CHIC CUP Cece recs trankcouiedeveccuaieassreacadt eal add ceeds 23 Managing ZOMG sinter corsawascmccarsiivenoysaunweeealeomsomeersnavixeiesihwn 23 SET PONTING vesssetis oo eee taco cor renscn enced ce ameae dan taeeu tare anec tas 25 Overview Of ISL Trunking cccccccssscececeeeeseeeeeeeesseeeesseeaseeeeens 25 ISL Trunking Architecture cccccsssssscceecseesseeeeeeeeeeeesenseaseees 25 Designing the Fabric for Optimize Use of ISL Trunking 28 Managing ISL Trunking visuiecwvessecucsasate ied nerbnnsncesschasavectasdaywenent 28 Advanced Performance Monitoring 1 cccccccssseeseessenees 31 Overview of Advanced Performance Monitoring 00 31 Advanced Performance Monitoring Architecture 008 31 Managing Advanced Performance Monitoring 00000 34 Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and M
213. r goes out of its Fabric Watch range e Port Status triggers if a port changes its status i e online offline etc e ISL Status triggers if a certain number of ISLs go down e Internal Switch Status triggers if the switch is disabled or enabled The overall status can be one of the following e Healthy OK every contributor is healthy e Marginal Warning one or more components are causing a warning status e Down Failed one or more contributors have failed The commands used to configure the Switch Status Policy feature are switchStatusPolicyShow and switchStatusPolicySet The swEventTrap in the file named SW_v5_1 mib described in the SNMP Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 144 26 section above receives Switch Status Policy notifications of an event because they are logged to the Error Event Log This feature can only be configured from the CLI Figure 26 8 shows the default parameter settings for the Switch Status Policy feature for the Brocade SAN Switch Module other Brocade platforms may vary in function Marginal indicates the switch will turn yellow Down indicates the switch will turn red If the value of a policy parameter is set to 0 it means that Note this factor is not used to determine the status of the switch If both of the values are 0 that contributor is
214. rces The zoning implementation is optimized to minimize processing resources and leverage ASIC capabilities as much as possible The zoning database size for SilkWorm 2000 series 3200 and 3800 switches is 96 KB and 128 KB for SilkWorm 3016 3250 3850 3900 12000 and 24000 switches To check the size of a zone database use the command cfgSize A switch with a zoning database size limit of 96 KB limits the size of the zoning database for the whole fabric even if a SilkWorm 3016 3250 3850 3900 12000 or 24000 switch is present in the fabric As the size of a SAN grows it is important to monitor the zoning database Typically the zone database size needs to be of concern as the size of a SAN exceeds several hundred ports The size of an alias name zone name or configuration name is limited to 64 characters for Fabric OS versions 2 6 1 3 1 and Fabric OS 4 1 or later releases While it is possible to create 64 character zone alias or configuration names doing so consumes more memory than a shorter name Additionally shorter names are easier to remember and less prone to typing errors Be wary of sacrificing meaning for shortness See the Whitepaper Zoning Implementation Strategies For Brocade SAN Fabrics for effective guidance for naming aliases zones and configurations The variable size of zone objects makes it very difficult to state guidelines as a number of zone entries or alias A zone database size is similar to disk storage The usage
215. rds is a maximum of 239 physical switches be they 8 16 64 or 128 port versions As a practical matter no vendor has yet tested networks of this size due to the expense and complexity of implementing such a network The current practical switch count limit is fewer than 239 switches based upon empirical testing To determine whether or not a SAN is supported it will be necessary to work with your switch provider to determine if your SAN design is valid Important variables that determine the supportability of a particular SAN include the number of switches version of Fabric OS the topology number of ISLs trunks number of connected devices and hop count Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 63 12 For very large SAN Design configurations please supply your switch provider with the following information e The maximum number of switches e The total number of ports e The maximum number of SAN devices e The minimum version of Fabric OS implemented e The fabric hop count The Brocade SAN Switch Module counts as a switch when calculating Note i i maximum number of switches in a fabric Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 64 Chapter 13 SAN Performance This chapter contains the following section
216. red The user should configure the IP addresses and community strings that are on their SNMP management station 5 The final section is an SNMP Access Control List ACL This provides a way for the administrator to restrict SNMP get set operations to certain hosts IP addresses This would be used for enhanced management security in the Storage Area Network 6 Figure 25 2 shows an example of setting the swEventTrapLevel to 5 so that all Trap events are sent to the SNMP manager and configuring the authTraps parameter so that authentication warning messages get sent to the SNMP manager It also shows the user how to set the IP Address of the Trap recipient 7 Figure 25 3 shows the resultant configuration settings from command performed in step 2 These configurations can also be performed from Web Tools Interface through the Admin interface See Figure 25 4 Figure 25 1 default agtcfgshow for the Brocade SAN Switch Module brocadessm1 USERID gt agtcfgshow sysDescr Brocade SAN Switch Module for IBM eServer BladeCenter sysLocation End User Premise sysContact Field Support swEventTrapLevel 0 authTraps 0 OFF SNMPvl community and trap recipient configuration Community 1 Secret COde rw No trap recipient configured yet Community 2 OrigEquipMfr rw No trap recipient configured yet Community 3 private rw No trap recipient configured yet Community 4 public ro No trap recipient configured yet Community 5 common ro No trap r
217. rely using the requirements at the time of implementation as a target This will prevent the SAN from being painted into a corner and needing to be fundamentally restructured after entering production Investment protection is another area that relates to scalability If an existing switch is replaced with a newer or higher port count switch to increase scalability it is valuable to reuse the existing switch elsewhere in the fabric Proper initial planning facilitates this as well The core edge fabric topology is the most frequently deployed topology in cases where scalability needs are great It is derived from the star topology which is common in traditional data networks The core edge fabric topology see Figure 11 2 is a similar design except that the core is normally redundant and there is typically only one level of edge switches Some core edge implementations opt for a single core switch when deploying the fabric in dual fabric SAN architecture The logic behind this approach is that should a single core fail the remaining and redundant fabric can maintain SAN operations A core edge topology is scalable from many perspectives It is possible to use variable size switches in the cores and the edges The larger the core switch the larger the fabric can grow If large cores and edges are utilized it is possible to build very large fabrics This concept of scaling a fabric by using variable port count switches is shown in Figure 12 l
218. riables connUnitPortStatus An overall protocol status for the port connUnitPortState The user selected state of the port hardware Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 134 Chapter 26 Fabric Watch This section provides information to guide advance users to customize and administer Fabric Watch on the Brocade SAN Switch Module The primary purpose of implementing Fabric Watch is to monitor the health status of the fabric elements by continuously ensuring that they are operating within the specified threshold boundaries In the event a safe operating limit of an element is breached an appropriate message is forwarded to the user by one or more pre selected methods The severity state appears in the message to indicate the urgency of the event to assist the administrator to take appropriate action Fabric Watch accomplishes this in three steps by 1 Measuring values 2 Comparing against threshold boundary limits 3 Event generation and notification The Fabric Watch classes of elements are predefined for Brocade SilkWorm switches The threshold levels for these element classes are provided in a default configuration The default configuration saves valuable configuration setup time for a new user Advanced users have the capability to fine tune these thresholds to their unique fabric environment by choosing a customized configurat
219. rics Failed Edge Switch FaDicA Fabric B Devices Dual Connected SAN Topologies The Brocade SilkWorm Design Deployment and Management Guide SAN DDM Version 3 0 covers the aspects of fabric topology such as scalability performance and availability in great depth The flexible fabric architecture of Brocade switches allows arbitrarily complex fabrics to be built when it is necessary to solve complex problems but also allows simple solutions to be built to solve simple problems The following simple topologies are discussed in more detail in that guide e Cascade e Ring e Full Mesh e Partial Mesh e Core Edge e Composite Core Edge Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 61 Chapter 12 SAN Scalability The scalability of a SAN is the size to which that SAN could be expanded without fundamental restructuring Scalability is so important to SAN design that it is frequently the first criteria used in deciding how to approach the SAN architecture The designer starts by asking two questions 1 how many ports does the SAN need now and 2 how many ports will the SAN need in the near future The solution is then designed to meet the port count requirements SANs should be designed to scale to the largest size that they could reasonably be expected to need to achieve in a reasonable time frame rather than me
220. rise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 xi IBM Resources The following related documentation is provided on the Brocade SAN Switch Module for IBM eServer BladeCenter Documentation CD ROM on the IBM eServer BladeCenter Web Site or on the IBM TotalStorage SAN Switch Web Site IBM eServer BladeCenter Documentation e Brocade Enterprise SAN Switch Module for IBM eServer BladeCenter and Brocade Entry SAN Switch Module for IBM eServer BladeCenter Installation Guide e IBM eServer BladeCenter Management Module Installation Guide e IBM eServer BladeCenter Management Module User s Guide e IBM eServer BladeCenter Management Module Command Line Interface Reference Guide e IBM eServer BladeCenter Fibre Channel Expansion Card Installation and User s Guide e IBM eServer BladeCenter Type 8677 Installation and User s Guide e IBM eServer BladeCenter Type 8677 Hardware Maintenance Manual and Troubleshooting Guide e IBM eServer BladeCenter Type 8677 Rack Installation Instructions e IBM Configuration Options Guide IBM eServer xSeries BladeCenter Marketing Web Site For additional marketing resource information visit the IBM eServer BladeCenter Marketing Web site http www ibm com servers eserver bladecenter IBM eServer BladeCenter Support Websites For additional support resource information visit the IBM eServer BladeCenter Web site Thi
221. rt LE 10km LE 10km LE 10km E Port The long distance Extended Fabric configuration can be established among SilkWorm 3016 3200 3250 3800 3850 3900 12000 and 24000 switches Long distance ports consume more buffers than regular ISL ports which means that configuration of a long distance port could disable the port itself or other ports in the same quad due to lack of buffers Note Note Long distance among SW3016 SW3200 SW3250 SW3800 SW3850 SW3900 SW12000 and SW24000 ports is not supported when the long distance fabric wide parameter fabric ops mode longDistance is set ISL Trunking is not supported on an Extended distance ISL at this time Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 40 6 Due to the Extended Fabrics and ISL limitations only 2 external ISLs of the Brocade SAN Switch Module consider using separate switches to connect remote sites and then connect the Brocade SAN Switch Modules to these switches Guideline The Brocade Extended Fabrics license must be purchased from your switch supplier An example of the Extended Fabrics tab launched from Advanced Web Tools is shown in Figure 6 1 Figure 6 1 Extended Fabrics tab in Advanced Web Tools Z Switch Admin Microsoft Internet Explorer el ES SwitchName brocadessm DormainID 1 WAN 10 00 00 05 1 34 00 c7 Fri Mar 26 2004
222. rvers or a tar gz file for UNIX servers These compressed files will need to be uncompressed in an appropriate directory on the FTP server Once unbundled there are a number of files but the user must only indicate a hidden symbolic link file called release plist in the File Name for download If using the CLI firmwaredownload An example is provided in Figure 22 1 A confirmation pop up that comes up after initiating is shown in Figure 22 2 The Fabric Manager interface is shown in Figure 22 3 For more information on Firmware Downloads read the Brocade Fabric OS Procedures Guide and the Brocade Fabric OS Reference Manual Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 112 Figure 22 1 Firmwaredownload using WebTools Z Switch Admin Microsoft Internet Explorer SwitchName brocadessm DomainID 1 WAAN 10 00 00 05 1e 34 00 c7 Fri Apr 16 2004 4 42 PM Switch Information Network Config Upload Download SNMP f2 Function Firmware Download Config Upload to Host Contig Download to Switch Host Details Protocol ftp Sull Install JV zeboot after download fv SutoGommit Jw User Name anonymous Host IP fi 0 64 210 23 Password esansansnnsanseaansae File Name iv4 2 11v4 2 1irelease plist Firmware Download Status Apply Close Reset md From Host 10 64 210 23 File Path Ar4 2 1 44 2 1 release pl
223. ry change in the fabric e DLS is not set This message means that the DLS option is turned off Load sharing is only reconfigured when the switch is rebooted or an Fx_Port comes up 2 Enter the dlsset command to enable Dynamic Load Sharing 3 Enter the dlsReset command to disable Dynamic Load Sharing Example switch admin gt dlsShow DLS is not set switch admin gt configshow route s route stickyRoutes 1 switch admin gt dlsSet Committing configuration done switch admin gt dlsShow DLS is set switch admin gt configshow route s route stickyRoutes 0 In Order Delivery IOD The IOD parameter enforces in order delivery of frames during a fabric topology change In a stable fabric frames are always delivered in order even when the traffic between switches is shared among multiple paths However when topology changes occur in the fabric for instance a link goes down traffic is rerouted around the failure and some frames might be delivered out of order The setting of this parameter ensures that frames are not delivered out of order even during fabric topology changes If IOD is turned on using iodSet frames are always delivered in order even during fabric topology changes This is the factory default for the Brocade SAN Switch Module and all IBM TotalStorage SAN switches If IOD is turned off using dlsReset frames may be delivered out of order during fabric topology changes The default behavior for Broc
224. s Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 13 2 e Advanced Web Tools can be used simultaneously with Fabric OS CLI commands Simply open a second window and you can take advantage of the benefits of both interfaces at the same time e Advanced Web Tools can be used from a standard workstation allowing you to be virtually in front of any fabric switch or port e Advanced Web Tools allows you to zone your configuration by dragging addresses and port numbers rather than laboriously typing them out e Advanced Web Tools provides a Performance Monitor feature enabling you to view the status and traffic of a switch or port in seconds via a variety of effective graphs Refer to Chapter 5 Advanced Performance Monitoring for more information on this feature e Advanced Web Tools is easy and intuitive to use All Brocade SAN Switch Modules for IBM eServer BladeCenter have the Web Tools license installed from the factory An example of the Advanced Web Tools Graphical User Interface is shown in Figure 2 9 Figure 2 9 Advanced Web Tools GUI 2 Switch status LEDs 6 External Port LEDs 28 Internal Port status indicators 3 per port 2 per port E Switch Explorer Microsoft Internet Explorer E File Edit View Fav ites Tools Help Back Ei Fabric Al brocadessm YS Segmented Switches HHROBBABABRDADAAAADA M
225. s e SAN Performance e IJO Profiles An over subscribed link is one on which multiple devices might contend for bandwidth A congested link is one on which multiple devices actually are contending for bandwidth Traditional data networks have been built with very high levels of over subscription on links for years The Internet is probably the best known example of this While not capable of supporting Internet like over subscription ratios real world SANs can be expected to have several characteristics that enable them to function well even with over subscribed links These characteristics include bursty traffic shared resources low peak usage by devices good locality and devices that can generate only a small fraction of the I O as compared to the available bandwidth Most networks have all of these characteristics to some degree Moreover organizations can often realize substantial cost savings by deliberately designing a SAN with a certain amount of over subscription When performance service levels are critical and the bandwidth requirements are high lower over subscription levels or traffic localization should be targeted Today many devices attached to a SAN are not capable of generating traffic at the full Fibre Channel bandwidth of 100 MB sec or 200 MB sec Figure 13 1 and Figure 13 2 detail a simplified scenario using a handful of devices to explain SAN bandwidth consumption Figure 13 1 Low server utilization Server max
226. s Web site provides additional information on the IBM BladeCenter http www ibm com pc support http www 306 ibm com pc support site wss document do Indocid MIGR 54597 IBM RedBooks Websites For additional IBM eServer BladeCenter integration resource information visit the IBM RedBooks Website and search for BladeCenter http www redbooks ibm com IBM TotalStorage SAN Switch Support Websites For additional resource information on the complete range of Brocade switch products from IBM visit the IBM TotalStorage Web site This Web site provides additional information on the IBM TotalStorage SAN Switch product line that is fully compatible with the Brocade SAN Switch Module for IBM eServer BladeCenter TotalStorage SAN Switch Marketing Website http www storage ibm com ibmsan products 2109 san_switch_solu html TotalStorage SAN Switch Support Site http www storage ibm com ibmsan products 2109 library html support Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 xii IBM Support Web Site For additional general support information visit the IBM Support Web site http www ibm com pc support If you have any questions or problems go to the following Help Center World Telephone Numbers URL http www ibm com planetwide Other Industry Resources For additional resource information visit
227. s for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 XV Chapter 1 Section Product Introduction The Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide provides detailed information to help you better utilize the Brocade SAN Switch Module licensable features and software Brocade SilkWorm switches are available in models ranging from 8 port switches with a few available features to high port count Director class switches providing many licensable features to help you better utilize your fabric and investment The following topics are covered e IBM eServer BladeCenter Overview e Brocade SAN Switch Module SilkWorm 3016 e Brocade Fabric OS v4 2 1 IBM eServer BladeCenter Overview IBM eServer BladeCenter is an innovative and manageable modular computing platform that provides outstanding performance density and affordable availability features This highly integrated infrastructure building block is designed to reduce datacenter complexity and simplify deployment IBM eServer BladeCenter is a modular server design that gathers computing resources into a cost effective high density enclosure that supports hot swappable high performance 2 way and 4 way Intel processor based and new 2 way POWER processor based server blades Figure 1 1 A general overview of the features of the IBM eServer BladeCenter Chassis incl
228. se of zone members or you can define a zone to create separate test or maintenance areas within the fabric e Optimize Resources You can use zones to logically consolidate equipment for efficiency or to facilitate time sensitive functions for example to create a temporary zone to back up non member devices A zone is a specified group of fabric connected devices also called zone objects which have access to one another Zone objects are grouped into zones and zones are grouped into a zone configuration Any zone object connected to the fabric can be included in one or more zones Objects in a zone can communicate only with other objects in the same zone Zones can overlap that is a zone object can belong to more than one zone and a fabric can have multiple zones A switch can have any number of resident zone configurations however only one active configuration can be enabled at a time After zoning is enabled if a device is not explicitly defined in a zone that Note ene ae f device is isolated and inaccessible by other devices in the fabric Other Aspects of Advanced Zoning In addition to the above zones can be configured dynamically and can vary in size depending on the number of fabric connected devices Zoning can be disabled at any time When zoning is disabled the fabric is in a non zoning mode and devices can freely access other devices in the fabric All devices connected to a fabric can be configured into o
229. sistently disabling ports or switches When configured the state of the switch or port will remain disabled through power cycles or reboots Reasons why this may be done include e There may be a bad SFP or switch which causes fabric instability These may need to be brought down temporarily until a replacement is found e Unused ports may be persistently disabled for security concerns When in this state no device or switch will be allowed to join the fabric on that port e To prevent operator error when devices or switches are connected to the wrong port To disable a switch persistently use switchCf gPersistentDisable After a few moments the switch is disabled To verify use switchshow to display the current state Note that SwitchRole is now Disabled Persistent This indicates that the command has taken effect The portcfgshow still shows all ports as not disabled This is fine since the switch as a whole is disabled To re enable the switch persistently use switchCfgPersistentEnab le Note that the fabric must re configure after it is brought back online If the switch is re enabled with switchenab1 e it will be enabled temporarily The next power cycle reboot or fastboot will cause the switch to be disabled persistently This is because the state of the switch is now stored in the flash non volatile memory Note Use portCfgPersistentDisable to persistently disable a port Use portcfgshow to check the status To re enable a
230. slam Pom Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide DDM Version 1 0 Publication Number 53 0000561 01 Publication Date May 4 2004 Copyright 2004 Brocade Communications Systems Incorporated ALL RIGHTS RESERVED Publication Number 53 0000561 01 Brocade the Brocade B weave logo Secure Fabric OS and SilkWorm are registered trademarks of Brocade Communications Systems Inc in the United States and or in other countries The following terms are trademarks of the IBM Corporation in the United States or other countries or both IBM BladeCenter eServer Microsoft Windows and Windows NT are trademarks of Microsoft Corporation in the United States other countries or both Linux is a registered trademark of Linus Torvalds All other brands products or service names are or may be trademarks or service marks of and are used to identify products or services of their respective owners Notice The information in this document is provided AS IS without warranty of any kind including without limitation any implied warranty of merchantability non infringement or fitness for a particular purpose Disclosure of information in this material in no way grants a recipient any rights under Brocade s patents copyrights trade secrets or other intellectual property rights Brocade reserves the right to make changes to this document at any time witho
231. switches Refer to your switch supplier for maximum supported fabric size limits The following software features are optional e Brocade ISL Trunking for IBM eServer BladeCenter e Brocade Advanced Performance Monitoring for IBM eServer BladeCenter e Brocade Performance Bundle for IBM eServer BladeCenter includes ISL Trunking and Advanced Performance Monitoring e Brocade Extended Fabrics for IBM eServer BladeCenter e Brocade Remote Switch for IBM eServer BladeCenter e Brocade Advanced Security Secure OS for IBM eServer BladeCenter e Brocade Entry Switch Full SAN Upgrade for IBM eServer BladeCenter Upgrades from Two Domain to Full Fabric Entry version only e Brocade Fabric Manager v4 x for IBM eServer BladeCenter stand alone application must be purchased separately To activate and use these optional software features you must purchase the corresponding license keys Visit the IBM eServer BladeCenter Web Site for more details IBM eServer BladeCenter Management Module The IBM eServer BladeCenter Management Module provides a Web browser interface that enables you to monitor and manage several key aspects of the individual switch modules from a standard workstation Only the switch modules enclosed in one IBM eServer BladeCenter chassis are displayed in the Web interface Detail for the Brocade SAN Switch Modules can be found in the IO Modules sections of the individual panes Following are some of the features that make the man
232. t tape drives can be effectively utilized in a redundant fabric environment as well even if they are single attached In a redundant SAN architecture there must be at least two completely separate fabrics just as a high availability server solution requires at least two completely separate servers Duplicating components and providing switch over software is well established as the most effective way to build high availability systems Similarly multi fabric SAN architectures are the best way to achieve high availability in a SAN In addition to enhancing availability using redundant fabrics also enhances scalability Using dual fabrics essentially doubles the maximum size of a SAN Ifa fabric is limited by vendor support levels to 34 switches 1200 user ports and a single fabric solution with dual attach devices is utilized then the SAN is limited to 1200 ports Twelve hundred dual attach ports is equivalent to 600 Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 59 11 devices However if a dual fabric with dual attach device solution is utilized the SAN is capable of supporting 2400 user ports or 1200 devices Any devices that are dual attached and are capable of supporting an active active or active passive dual path essentially double the potential bandwidth An active active dual path means that I O is capable of using both paths in
233. t metrics graphs are included The table Predefined Performance Graphs lists the performance graphs available You can access the basic monitoring graphs on switches that do not have an Advanced Performance Monitoring license activated The advanced monitoring graphs give more detailed performance information to help you manage your fabric Predefined Performance Graphs Basic Monitoring Graphs Description Displays the performance of a port in bytes per second for frames received and transmitted Switch Aggregate Throughput Displays the aggregate performance of all ports on a switch Blade Aggregate Throughput Displays the aggregate performance of all ports on a blade Switch Throughput Utilization Displays the port throughput at the time the sample is taken Port Error Displays a line of CRC errors for a given port Displays the percentage of usage of a chosen switch at the time the sample is taken Port Throughput Switch Percent Utilization Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 34 Displays the CRC error count between sampling periods for all the ports on a switch Advanced Monitoring Description Graphs p SID DID Performance Displays the traffic between SID and DID pair on the switch being managed SCSI vs IP Traffic Displays percentage of SCSI versus IP frame traffic on each ind
234. tasks are provided in the Brocade SilkWorm Design Deployment and Management Guide SAN DDM Version 3 0 Nor will this chapter focus on the devices such as HBAs and storage targets attached to the Brocade SAN Fabric Please see the Brocade Fabric Manager User s Guide and the Brocade Advanced Web Tools Administrator s Guide for detailed instructions on usage For an online list of commands use the help command To view the online command reference use the command help lt command gt For example help Note switchshow will provide the online reference for the switchshow command For more information please refer to the Brocade Fabric OS Reference Guide This chapter contains the following sections e Preparing the Switches for the SAN Fabric e Switch Staging Steps e Preparing the Fabric Configuration e Profiling the SAN Preparing the Switches for the SAN Fabric There are many possible ways to configure the switches that make up a fabric Such as the configuration of a switch is dependant upon its role During the planning phase of deployment the following questions should be addressed e Will it be the principal switch e Isita core or edge switch e Is long distance required e Are the correct license keys installed Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 89 20 Once the answers are known the goal of this se
235. tch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 121 Chapter 23 SAN Management Overview The trend of fabrics increasing in switch count and geographically separate locations of fabrics and switches has created the need for centralized secure and cost effective SAN management A SAN consists of switches host systems and storage devices Switch vendors have provided several command and web based tools to manage switch configurations A few major storage providers have integrated a basic level of switch management utilities into their existing management software packages However their primary focus remains on managing storage not the fabric As SAN fabrics become larger managing a fabric by accessing each individual switch from a command line interface can be an inefficient and time consuming process SAN management software must be capable of performing complex tasks including configuring maintaining monitoring and troubleshooting a SAN in a simple and effective manner Although there are many SAN management software packages in the market place they are somewhat limited in providing SAN level management functionality Today s SAN Management software applications must include SAN Security e Graphical display of SAN devices e Easy access to all switches of the fabric to maintain consistent configurations e Real time self monitoring status and advance warning e Rea
236. tches Use the interopmode command to enable or disable interoperability mode for individual Brocade switches This feature enables other manufacturers switches to be used in a Brocade fabric This command must be executed on all Brocade switches in the fabric The switch must be rebooted after changing interoperability mode Other manufacturers switches may require the execution of one or more commands that select interoperability mode for their switches Brocade SAN Switch Module Requirements e The Brocade SAN Switch Module must be running 4 2 1 or later firmware e A Zoning license and a Fabric license must be installed on each Brocade SAN Switch Module Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 47 McDATA Firmware Requirements e McDATA ES 3016 or equivalent OEM versions that are plug compatible Contact your switch supplier for their currently supported vendors switches Note and firmware versions Supported Brocade Features The following features are supported on Brocade switches only e Brocade Advanced Web Tools e Brocade Fabric Watch e Brocade Fabric Access API functions can be accessed from Brocade switches only but other manufacturers switch information may be reported The object information and zoning actions are configurable from the API e _ Brocade s translative mode which registers private s
237. ted by entering a license key available from the switch supplier The Extended Fabric feature achieves long distance connections by allocating more frame buffers for fibre channel traffic Long distance connections require more frame buffers than regular ISL connections The greater the distance the more frame buffers required If the long distance port is part of a quad this limits the buffer space left over for the remaining ports in the quad which must therefore be configured appropriately On the Brocade SAN Switch Module external ports 0 and 15 share a buffer mele pool with internal ports 9 and 10 part of the same quad The user must be careful when using the Extended Fabric licensed feature with the Brocade SAN Switch Module The Extended Fabric feature allows the user to configure external ports 0 and 15 for long distance performance However certain long distance configurations can disable the other external and possibly some internal ports This could cause a disruption in traffic When considering configuring external ports for long distance both the port speed 1 or 2Gbit sec and the distance setting LE L0 5 L1 L2 and LD must be considered The two internal ports 9 and 10 may be disabled due to long distance configuration of the external ports The two external ports of the Brocade SAN Switch Module can be configured as long distance ports but the user must be aware that these ports 0 and 15 share buffers with t
238. tering high availability or business continuance operations Both resilient and non resilient dual fabrics can be referred to as redundant fabric SANs Redundant designs are always recommended for high availability systems and any large SAN deployment where downtime for the entire SAN could affect hundreds of servers There are four primary categories of availability in SAN architecture In order of increasing availability they are Single fabric non resilient All switches are connected to form a single fabric which contains at least one single point of failure The Cascade topology is an example of this category of SAN see Figure 11 1 Single fabric resilient All switches are connected to form a single fabric but there is no single point of failure that could cause the fabric to segment Topologies such as core edge ring and full mesh topologies are examples of single resilient fabrics see Figure 11 2 Multi fabric non resilient In a dual fabric non resilient SAN half of the switches are connected to form one fabric and the other half form a separate fabric This model can be extended to more than two fabrics if desired Within each fabric at least one single point of failure exists This design can be used in combination with dual attached hosts and storage devices to keep a solution running even if one fabric fails or if a rolling upgrade is needed An example of this type of SAN is a dual fabric SAN built with core edge fabr
239. that are recommended to check the switch environmental status The recommended commands on the Brocade SAN Switch Module are uptime tempshow sensorshow and switchstatushow Switchstatusshow displays any triggered messages set by switchstatuspolicyset Switchstatuspolicyset discussed in the SAN Management section allows the environmental settings that trigger warning messages to be customized 14 Baseline and Backup the Switch Once all the settings are complete use conf igupload to backup the switch configuration This uploads all of the configuration parameters that define the switch configuration This is highly recommended in the event the original settings need to be restored If minor changes to the configuration are required just edit the configuration file and re download the configuration to the switch One further benefit is that a standard switch configuration can be defined and uploaded to all of the remaining switches in the fabric This is known as baselining Define a golden switch For larger SAN fabrics or for the staging of many smaller fabrics use configupload to baseline the golden switch The golden switch configuration can be downloaded with Fabric Manager to all others in the fabric Do baselines by Fabric OS version and platform i e have a separate configdownload file for the Brocade SAN Switch Modules Doing this helps when it comes time for change management Guideline Brocade Enterpris
240. tialization to the portCfgLongDistance CLI command When set to 1 this parameter indicates that enhanced link reset protocol should be used on the port The default value for this parameter is 0 and is compatible with earlier Fabric OS v3 0 x implementations For optimal performance specify 1 when E Port links are between switches with Fabric OS v3 0 2 and greater or Fabric OS v4 0 2 and greater Specify 0 or nothing when connecting a switch with Fabric OS v3 0 2 or above switch to previous releases of Fabric OS If an Extended Fabrics port is to be configured on a SilkWorm 2000 series switch the fabric wide long distance parameter fabric ops mode longDistance must be set to 1 in the configure CLI menu Configuration of this parameter requires the switch to be disabled This parameter must be set on all switches within the fabric Also one Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 86 19 must set the appropriate Extended Fabrics mode for each long distance port using the portCf gLongDistance CLI command When configuring Extended Fabrics on Brocade SAN Switch Module series switches only port level configuration is necessary and the fabric ops mode longDistance must be set to 0 which is the default value Brocade supports Extended Fabrics ports between same series switches i e Silk Worm 2000 seri
241. tice to capture a profile of the fabric The show commands are ideal for this This list can be used for Brocade SAN Switch Module switches which run Fabric OS 4 2 lor greater These commands can be easily scripted using the Brocade API Scripting Toolkit As changes take place in the fabric consider periodically updating the profile information This will proactively simplify change management and provide a living documentation set that can be referenced when technical support is required Brocade SAN Switch Module Profiling Commands version firmwareshow chassisshow licensidshow licenseshow ipaddrshow switchshow portcfgshow configshow agtcfgshow snmpmibcapset cfgshow nsallshow nsshow nscamshow fabricshow islshow trunkshow tempshow sensorshow sfpshow errdump Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 108 Chapter 21 Validation Once the SAN is staged it is recommended to verify its functionality and robustness before going into production While less important for the entry level environment validation becomes critical for SANs with higher port counts While all pertinent tests for a particular implementation will not be discussed this section should provide a framework for a customized validation plan Sample procedures in this section will demonstrate how to check the SAN stability and High Availability HA
242. time a change in the fabric occurs A change in the fabric is defined as an E_Port going up or down or an Fx_Port going up or down Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 105 20 If DLS is turned on using dlsSet when there are multiple equivalent paths to a remote switch traffic is shared among all the paths Load sharing is recomputed when a switch is booted up or every time a change in the fabric occurs With DLS enabled traffic on existing ISL ports might be affected when one or more new ISLs are added between the same two switches Specifically adding the new ISL might result in dropped frames as routes are adjusted to take advantage of the bandwidth provided by the new ISL Enabling DLS optimizes fabric routing For example if an Fx_Port goes down another Fx_Port might be rerouted from one E Port to a different E_ Port The switch minimizes the number of routing changes but some are necessary to achieve optimal load sharing This is the factory default on all Brocade branded switches Follow the guidance of your SAN support vendor for the configuration of this Not s ay setting Viewing and changing this parameter 1 Enter the dlsshow command to view the current DLS setting One of the following messages appears e DLS is set This message means that the DLS option is turned on Load sharing is reconfigured with eve
243. to capture event data No problems should be observed If there are troubleshoot them at this time Check the switch event logs for any issues with the fabric while validating its Guideline stability Setup the syslogd system log on a host and configure to capture fabric event messages High Availability ISL Failure Simulation Take a case where all hosts are attached to one edge switch From this switch one trunk with two ISLs is attached to a core switch to form a trunk group with 4 Gbit sec of available bandwidth This trunk group contains an ISL trunk master and a second ISL as a trunk member A maximum of 400 MB sec should be generated across them to provide maximum ISL stressing These tests simulate failures on an edge and a core switch Hot code load should also be tested before going into production Case 1 Member ISL Cable Pull With I O remove the non trunk master ISL for ten seconds and then replace it Repeat this test three times I O should continue without any effect Some frames may be lost when the cable is pulled If this is the case the host will log messages indicating that the I O is being retried Case 2 Trunk Master ISL Cable Pull With I O remove the Trunk master ISL for ten seconds and then replace it Repeat this test three times I O traffic may be paused briefly on the second fabric however the I O should not timeout on the host High Availability Switch Failure Simulation Case 1 Edge Switch Failure
244. to or greater than the number of devices and the devices actually are under subscribed or at a 1 1 ISL over subscription ratio minimizing the value of this metric for core switches The ISL over subscription ratio does become more meaningful for a core switch when devices are connected to the core and there are more devices than ISLs trunks A 7 1 ISL over subscription ratio is aligned with an industry average of 6 1 fan out The trend in the storage industry is that the hosts to storage ratios are increasing as is the performance of storage devices A 7 1 ISL over subscription ratio should be targeted in SAN designs with the ISL over subscription ratio being adjusted higher or lower to meet particular performance requirements The higher the ISL over subscription ratio the lower the performance and conversely the lower the ISL over subscription ratio the higher the performance An ISL over subscription ratio of 3 1 results in high performance and fewer available ports while an ISL over subscription ratio of 15 1 results in lower potential performance and more available user ports The practical boundaries for ISL over subscription ratios is 3 1 for high performance SANs and 15 1 where lower performance is sufficient The factors that influence this position include resiliency number of ports available on a Brocade switch and industry host to storage fan out ratio The IBM eServer BladeCenter architecture approaches the best of both worlds by h
245. torage target devices into the fabric can be used in a heterogeneous fabric as long as the devices are directly connected to Brocade switches The devices will be accessible from any port on the fabric Unsupported Brocade Features In a heterogeneous fabric the following Brocade optional licensed features are not supported and cannot be installed on any switch in the Fabric e QuickLoop QuickLoop Fabric Assist or QuickLoop Zoning e ISL Trunking e Extended Fabrics e Remote Switch e Advanced Security Secure Fabric OS In a heterogeneous fabric the following Brocade Fabric OS services are not supported on any switch in the Fabric e Open E Port e Extended Edge PID format e Broadcast Zoning e Management Server Service e Alias Server e Platform Service e Virtual Channels e IP over FC Configuration Recommendations The following is recommended when configuring an interoperable fabric e Avoid Domain ID conflicts before fabric reconfiguration There should not be duplicate domain IDs for switches joining the fabric e Add switches to the fabric slowly You should wait for a fabric reconfiguration after adding each switch when adding multiple switches to a fabric e Remove switches from the fabric slowly You should wait for a fabric reconfiguration after removing each switch when removing multiple switches from the fabric Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Mana
246. ts the SAN deployed on time and within budget Doing this ensures the ROI to be realized in the shortest time possible Planning is all about understanding the requirements and allocating necessary resources Proper planning does take extra up front effort and cost For higher port count SANs it is critical that there be at least one person who is accountable for developing and driving it With a good plan progress towards completion can be measured the right persons identified the roles and responsibilities defined the site and SAN documented and SAN resources can be efficiently utilized With proper planning less effort is needed to maintain the infrastructure When the staging requirements are known obstacles that may impede progress can be avoided This chapter provides some guidelines on what essential information is needed for putting an effective plan together Checklists are used throughout this chapter to provide a framework for gathering requirements to meet specific site needs This chapter contains the following sections e Planning Documentation e Zone Planning e Security Planning e Extended Fabrics Planning e Fabric OS Upgrade Planning e LAN Planning Planning Documentation This section will provide some guidelines regarding documenting the SAN Knowing what documentation is needed allows for planning IP addresses switch domain numbers what ports should be used for ISLs etc Once created having the documentation read
247. ude e Space saving form factor 7U e High availability midplane e Server blade bays up to 14 2 way and up to 7 4 way e Standard media USB CD ROM and Floppy diskette drive accessible from each server blade e Switch modules up to 4 switch module bays e 2 Power supply modules are standard and there is an option for two more e 2 Cooling modules are standard e 1 KVM management module is standard and there is an option for a redundant KVM management module e Innovative design reduces cables by up to 83 saving installation time and money e Delivers pay as you grow scalability e Provides fault tolerant connection from the server blade to ALL modular components e Inventive design allows you to you to upgrade to new technologies and preserve your original investment Fibre Channel SAN Switch Module options for the IBM eServer BladeCenter e Up to 2 Fibre Channel switch modules are supported in switch module bays 3 or 4 e Each switch module has at least one connection to each server blade e 2 switch modules provide the ability for redundant fabric connections from the chassis e Switch modules work in active active redundant fabric configuration e Each module supports two external port connections Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 1 Each module supports 14 internally facing ports for server blade conne
248. uide Publication Number 53 0000561 01 139 26 Changed Exceeded Below Above InBetween change behavior type change behavior interval change threshold boundary level change custom unit change custom time base change custom low change custom high change custom buffer apply threshold boundary changes 10 cancel threshold boundary changes Select choice gt 1 19 19 14 change threshold alarm level change changed alarm change exceeded alarm change below alarm change above alarm change inBetween alarm apply threshold alarm changes cancel threshold alarm changes return to previous page MAAINADANFPWNEHE OAATAHADNSPWNHEHE wo Errlog 1 SnmpTrap 2 PortLoghock 4 RapiTrap 8 EmailAlert 16 Valid alarm matrix is 31 Enter below alarm matrix gt 0 31 0 3 Index ThresholdName BehaviorType BehaviorInt 0 eportRXPerf000 Triggered I 15 eportRXPerf015 Triggered 1 Threshold boundary level is set at Default Default Custom Unit KB s KB s Time base Low 120000 120000 High 220000 220000 BufSize 0 0 Threshold alarm level is set at Default Errlog 1 SnmpTrap 2 PortLogLock 4 RapiTrap 8 EmailAlert 16 Valid alarm matrix is 31 Default Changed Exceeded Below Above InBetween change behavior type change behavior interval change threshold boundary level change custom unit change custom time base change custom low change custom high change custom buffer apply threshol
249. uration for this switch module fill in the following fields and click Save This will save and enable the new IP configuration IP address 192 168 196 121 Subnet mask 255 255 255 0 Gateway address 192 168 196 1 Advanced Management Bay 4 no module installed Internet Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 91 20 Figure 20 2 IP Address configuration using the IBM eServer BladeCenter Management Module Step 2 Change the new static IP address configuration then Click the save bottom on the bottom right File Edit View Favorites Tools Help 0 x Heak gt gt OA Asearch Gyravortes Breda Eby G A Gi Address fa http 10 64 210 20 private main ssi BladeCenter Management Module Bay 3 Fibre Smy Current IP Configuration Configuration method Static IP address 192 168 196 121 Subnet mask 255 255 255 0 Gateway address 192 168 196 1 New Static IP Configuration Status Enabled To change the IP configuration for this switch module fill in the following fields and click Save This will save and enable the new IP configuration IP address 10 64 210 25 Subnet mask 255 255 240 0 Gateway address 10 64 208 1 Advanced Management G60 Links server Save Bay 4 no module installed Internet Figure 20 3
250. used for managing remote switch e Brocade SilkWorm Design Deployment and Management Guide SAN DDM Version3 0 Section 5 16 entitled Brocade Extended Fabrics Planning under SAN Deployment and Appendix B Long Distance Technologies for Storage Area Networks Note that in terms of Remote Switch behavior the Brocade SAN Switch Module behaves exactly like the SilkWorm 3850 3250 as may be described in the above publications Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 42 Chapter 8 Brocade Advanced Security Secure Fabric OS This chapter includes the following sections e Overview of Advanced Security Secure Fabric OS e Advanced Security Secure Fabric OS Architecture e Managing Advanced Security Secure Fabric OS Overview of Advanced Security Secure Fabric OS Advanced Security Secure Fabric OS is an optionally licensed product that provides customizable security restrictions through local and remote management channels on a Brocade fabric It is available for the Brocade SAN Switch Module and is licensed on a per switch basis The Advanced Security Secure Fabric OS feature is provided with the Fabric OS and can be activated by entering a license key available from the switch supplier e Advanced Security Secure Fabric OS provides the ability to do the following e Create policies to customize fabric manageme
251. ut notice and assumes no responsibility for its use The authors and Brocade Communications Systems Inc shall have no liability or responsibility to any person or entity with respect to any loss cost liability or damages arising from the information contained in this book or the computer programs that accompany it Notice The product described by this document may contain open source software covered by the GNU General Public License or other open source license agreements To find out which open source software is included in Brocade products view the licensing terms applicable to the open source software and obtain a copy of the programming source code please visit http www brocade com support oscd Export of technical data contained in this document may require an export license from the United States Government Brocade Communications Systems Incorporated Corporate Headquarters European and Latin America Headquarters 1745 Technology Drive 29 route de l Aeroport San Jose CA 95110 Case Postale 105 T 408 333 8000 CH 1211 Geneva 15 F 408 333 8101 Switzerland Email info brocade com T 41 22 799 56 40 F 41 22 799 56 41 Email europe info brocade com Asia Pacific Headquarters Shiroyama JT Trust Tower 36th Floor 4 3 1 Toranomon Minato ku Tokyo Japan 105 6036 T 81 35402 5300 F 81 35402 5399 Email apac info brocade com Document History The following table lists all versions of the Brocad
252. verified with the licenseshow command See Figure 26 1 Figure 26 1 Fabric Watch License brocadessm USERID gt licenseshow SzdyQeSzRh0ezRj Web license Zoning license Fabric license Fabric Watch license The commands used to configure Fabric Watch are listed under the Fabric Watch help command fwHelp To determine if all the Fabric Watch classes are turned on use the fwAlarmsFilterShow and fwAlarmsFilterSet commands By default the Fabric Watch alarms are enabled on the Brocade SAN Switch Module Fabric Watch has been pre programmed with defaults from Brocade If you know the specific traffic patterns of your SAN or have specific requirements for any of the class variables Fabric Watch allows you the flexibility to tune the class alarms to your unique Storage environment 1 To determine if all the Fabric Watch alarm classes are enabled use fwAlarmsFilterShow If the alarms are disabled enable them using the fwAlarmsFilterSet command as show in Figure 26 2 This should not be necessary on the Brocade SAN Switch Modules as this has already been pre programmed Figure 26 2 Enabling the Fabric Watch Classes switch admin gt fwalarmsfiltershow FW Alarms are disabled switch admin gt switch admin gt fwalarmsfilterset 1 Committing configuration done FW Alarms are enabled switch admin gt switch admin gt fwalarmsfiltershow FW Alarms are enabled switch admin gt Brocade Enterpr
253. view Favorites Tools Help Kal eux 2 OR A Aer Gimme Gun G B S oH Address http 10 64 210 20 private main ssi z Go Links BladeCenter Management Module server Advanced Setup Fast POST Disabled External ports Enabled gt External management over all ports Enabled gt Preserve new IP configuration on all resets HEREJE Cancel Restore Factory Defaults e This action will cause all module settings to be set to their factory defaults You will lose all the changes you made to the configuration of this module as a result In order to preserve the new IP configuration set the field labeled Preserve new IP configuration on all resets to enabled Clearing of the configuration will be followed by a restart of the module Click the Restore Defaults button if you want to proceed Cancel Restore Defaults Send Ping Requests Internet 2 Login and Change Passwords On the first telnet Login to the Brocade SAN Switch Module the user will be challenged to change passwords for three of the four accounts on the system root factory and user The four accounts are root factory USERID and user This step can be skipped by using the CONTROL C key combination however the challenge will continue at every new login until all of the accounts passwords have been changed Store the passwords in a safe location See Figure 20 5 for an example By default the Brocade SAN Switch M
254. ware Download to FDMI capable Host Bus Adapter Fabric Manager is a stand alone Software Program that must be purchased from your switch supplier The Brocade SAN Switch Module is supported by Fabric Manager version mite 4 1 1 or higher Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 17 An example of the Fabric Manager application is shown in Figure 2 11 Figure 2 12 and Figure 2 13 Figure 2 11 Fabric Manager At A Glance Switch view i Fabric Manager File Edit View Actions Topoloay Tools Help Address pitp 10 64 210 25 J Qla rxrloah ealDFvlvaA brocadessm Detail E SAN Elements Fiter Device Ports Devices Event Portgrid Ports Summary Switches Topology Fabrics s Healthy 2Ghit 16 port embedded fabric switch F v4 21_rc1 1 10 64 210 25 255 255 240 0 0 0 0 0 0 0 0 0 10 64 208 1 10 00 00 05 1 34 00 c7 Principal Online None 16 None Healthy i 2 IS SwitchGroups 2 Ghitisec IS PortGroups 2 Port information gt 87 free 14 of 16 Device information 2 steffler2 localhost Figure 2 12 Fabric Manager At A Glance Fabric view Probie anges T File Edit View Actions Topology Tools Help Address fittp 10 64 210 100 Q i erl AR BR lt vV A ID Name as D Devices E SAN Elements 7 Fiter 16
255. witch products Due to this change a new command userrename must be used to rename the default USERID user account to admin before connecting the Brocade SAN Switch Module to a secure fabric made up of other Brocade SilkWorm switches Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 78 SAN Deployment This section discusses fundamental SAN Deployment concepts and associated guidelines and checklists for deploying an effective SAN design This section contains the following chapters e SAN Deployment Overview e Planning e Staging e Validation e Maintenance Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 79 Chapter 18 SAN Deployment Overview Once the SAN is designed using sound principles as detailed in the SAN Design section it needs to be deployed The process of SAN deployment is more than plugging in cables turning on the power and setting IP addresses In fact there are four distinct phases all SAN deployments go through Briefly these four phases are planning staging validation and maintenance The definition and benefits of doing each of them properly are summarized below 1 Planning Preparing for the staging of the SAN and related equipment is important therefore a chapter is dedicated to this subj
256. witches and or the management station SecTelnet or SSH must be used to administer the Primary FCS switch when running Advanced Security Secure Fabric OS Brocade SecTelnet requires Note digital certificates are installed on each switch to be administered This utility only encrypts the passwords sent over the LAN all other commands etc are sent as clear text In a fabric that contains SilkWorm 2000 series switches the maximum security DB size is limited to 32 KB with 16 KB active In a fabric Note containing Brocade SAN Switch Module switches the security DB size maximum is 128 KB with 64 KB active For all fabrics the maximum number of DCC policies is limited to 620 at this time Note Secure Shell SSH is a standards based secure method for accessing SilkWorm switches running Fabric OS 4 1 or later Any SSH client that supports version 2 of the protocol can be used There are literally hundreds of freeware SSH clients available that have this capability On the switch side SSH is only supported on Fabric OS 4 1 or higher Two popular clients have been tested Putty and Fsecure Extended Fabrics Planning This section will provide the information for planning connections of Brocade SilkWorm fabrics over longer distances One common reason is for data replication which provides site redundancy In this way if one site goes down due to a disaster the data can be recovered and brought online in minutes rather than days Another t
257. x aan k aA J ID Name Ime a opolo E SAN Elements g Fiter Detail Device Ports Devices Portgrid Ports Summary Switches opolo S My SAN a BRD_12K_sw0_116 J brocadessm 0 brocadessm 0 IBM_2109_F32 0 IBM_3534_1RU 0 IBM_3534_F08 D Berma E DETRE G SwitchGroups lS PortGroups Show Hide Links Show Hide Nodes Z1 Ghitsec I Device Groups Bie o ae steftier2 iocathost o For more information refer to the Brocade Fabric Manager User s Guide and Fabric Manager online help SNMP A common method for monitoring and managing a network device is using the Simple Network Management Protocol SNMP Understanding the components of SNMP make it possible to use any tool to view browse and manipulate Brocade switch variables as well as set up an enterprise level management process Every Brocade switch supports SNMP and carries an agent and management information base MIB The agent accesses information about a device and makes it available to a network manager station When active the SNMP management station inspects get or alters set variables when it queries an agent The get getnext and set commands are sent from the SNMP management station and the agent replies once the value is obtained or altered Agents use variables to report such data as the number of bytes and packets in and out of the device or the number of broadcast messages se
258. xist Policy does not exist ions Serial Policy Serial Policy Password Policy does not exist Policy does not exist RSNMP Policy RSNMP Policy Policy does not exist Policy does not exist WSNMP Policy WSNMP Policy Policy does not exist Policy does not exist HTTP Policy HTTP Policy Policy does not exist Policy does not exist API Policy API Policy Policy does not exist Policy does not exist Telnet Policy Telnet Policy Policy does not exist Policy does not exist FrontPanel Policy FrontPanel Policy Policy does not exist Policy does not exist Activate Save Close Help For more detailed information on Advanced Security Secure Fabric OS refer to the following publications and material e Brocade Secure Fabric OS User s Guide e Brocade Secure Fabric OS QuickStart Guide e Brocade Fabric OS Procedures Guide Chapter 3 entitled Securing Fabric OS e Brocade Fabric Manager User s Guide Chapter 9 entitled Security Management e Fabric OS Reference Manual CLI commands used for managing security e Brocade SilkWorm Design Deployment and Management Guide SAN DDM Version3 0 Section 3 4 entitled Designing SANs With Secure Fabric OS under SAN Design and Section 5 10 entitled Secure Fabric OS Planning and Section 6 5 entitled Staging SAN Fabrics with Secure Fabric OS under SAN Deployment and Section 9 6 16 entitled Secure Fabric OS Policy Management under SAN Management e SAN
259. y Marginal Oo Critical m Unmonitored Ea Applet com brocade web switchexplorer applet FabricTreeApplet started Internet The following default switch performance Monitoring Graph appears Figure 27 3 Figure 27 3 Default Performance Monitoring Graph iol xi Actions Performance Graphs Switch Throughput Utilization Ref rate 30 Secs Java Applet Window The following example will add two new Advanced Performance Monitoring Graphs that will display the Transmit and Receive Performance from each internal Server Blade in a BladeCenter 2 HS20 Server Blades that are each communicating with one Fibre Channel Disk Drive from an FC JBOD Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 149 27 Step 1 Use the Name Server icon on the bottom left half of the Switch Explorer WebTools window to access a list of devices in the fabric Figure 27 4 Figure 27 4 Name Server icon in WebTools Switch Explorer Microsoft Internet Explorer Explorer Microsoft iain A E 2101 xJ File Edit View Favorites Tools Help Ea Back gt amp A A Gsearch GaFavortes meda D 3 A Gl Address a http 10 64 210 25 switchExplorer html View by Name T Fabric IBM_2109_F32 IBM_3534_1RU IBM_3534_F08 brocadessm i Segmented Switches 4 V m a o a a o a i o D i a D a a D a a I a a A a a D a a D a o D a o A i
260. y recommended to reduce the possibility of having no primary FCS switch available It is possible to designate as many backup FCS switches as desired however all FCS switches should be physically secure Any switches not listed in the FCS policy are defined as non FCS switches The root and factory accounts are disabled on non FCS switches For information about customizing the FCS policy see Chapter 4 of the Secure Fabric OS User s Guide Fabric Management Policy Set Advanced Security Secure Fabric OS supports the creation of several types of policies that can be used to customize various aspects of the fabric By default only the FCS policy exists when Secure Mode is first enabled Advanced Security Secure Fabric OS policies can be created and managed by the CLI or Fabric Manager Advanced Security Secure Fabric OS policies can be created displayed modified and deleted They can also be created and saved without being activated immediately to allow implementation at a future time Saved policies are persistent meaning that they are saved in flash memory and remain available after switch reboot or power cycle The group of existing policies is referred to as the fabric management policy set referred to as FMPS which contains an active policy set and a defined policy set The active policy set contains the policies that are activated and currently in effect The defined policy set contains all the policies that have been defined wh
261. y sending it ping requests In order to ping this switch module at its current IP adrress of 10 64 210 25 click the Ping Switch Module button Cancel Ping Switch Module Start TelnetWeb Session In order to start a telnet or web session to the management interface of this module click Start Web Session or Start Telnet Session Note that some modules only provide a telnet interface and dont support a web based interface The web session will Firr date connect to the standard HTTP port 80 and the telnet session will connect to the standard telnet port 23 Note The Java 1 4 Plug in is required for the telnet session You will be given the option to download and install the Plug in if necessary Cancel Start Telnet Session Start Web Session j Internet Brocade Enterprise and Entry SAN Switch Modules for IBM eServer BladeCenter Design Deployment and Management Guide Publication Number 53 0000561 01 12 The Telnet Session will look similar to that shown in Figure 2 8 Figure 2 8 Brocade SAN Switch Module Telnet Session ip x File Edit Terminal a Fabric OS brocadessm brocadessm login USERID Password Please change your passwords now Use Control to exit or press Enter key to proceed Password was not changed Will prompt again at next login until password is changed brocadessm USERID gt brocadessm USERID gt brocadessm USERID gt I Connected to 10 64 210 25 telnet On t
262. ypical use is consolidated remote data archival to tape This Guide cannot address all of the deployment implications that must be addressed for Extended Fabric environments For comprehensive information on how Extended Fabrics works refer to the Brocade Fabric OS Features Guide and the Brocade SilkWorm Design Deployment and Management Guide SAN DDM Version 3 0 these documents provide the necessary guidelines for implementing Extended Fabrics For additional information refer to Appendix B Long distance Technologies for Storage Area Networks in the DDM listed above A few details are covered here to help you get started SAN long distance connectivity may involve single mode fiber or through more sophisticated network equipment that allows for greater line availability For Metro Area Networks MAN those distances up to 100 Km DWDM equipment is generally used Longer distances that go up to thousands of kilometers generally require FC protocol conversion This means a Wide Area Network WAN transport method is required Different equipment such as an ATM long haul switch may be required These types of devices are out of scope for this document Configuring a port for Extended Fabrics can be performed using the portCfgLongDistance CLI command or through Web Tools Specify the port and Extended Fabrics level as arguments to the portCfgLongDistance command Fabric OS v4 2 1 contains an additional optional parameter VC Translation Link Ini

"user manual"

Contents

Download Pdf Manuals

Related Search

Related Contents

&quot;user manual&quot;

Contents

Download Pdf Manuals

Related Search

Related Contents

"user manual"