Configuring and Tuning HP ProLiant Servers for Low - Fusion-io
Contents
1. ing HP TimeTest with an explanation of each component of the Bind to core 3 and use local memory nice probably not necessary HP TimeTest7 2 executabl output in Comma Separated Variable csv format print SMI count at the beginning and end print a timestamp at the beginning and end latency is determined by cycles instead of ime threshold is 5 psec on 2 90 GHz processor run for 30 minutes on 2 90 GHz processor 44 is 4 of cycles per loop iteration I get Use FIFO scheduling at priority 80 use nice of 20 I suspect irrelevant for RT policies Generating the output in CSV format allows for easy import into a spreadsheet for plotting The HP Low Latency team is working on an updated version of HP TimeTest and expects to have it ready for the next update of this document The following changes are planned e Make a change to address the issue of out of order processors which can cause bleeding of instructions past the Read Cycle Counter instruction However this change will probably not appear in any of the latency plots e Check the SMI count when latency spikes are detected not just at the beginning and end of program execution e Add an option to keep the processor core as busy as possible in an attempt to consume maximum power e Add an option to specify a runtime instead of a loop count Such an option might not work when run at a sufficiently high real time priority To provide2. 2 SCore dc service irqbalance stop until irqbalance status irqbalance is stopped do sleep 1 done IROBALANCE ONESHOT 1 IRQBALANCE BANNED CPUS CoreMask irqbalance sleep 1 until service irqbalance status irqbalance is stopped do sleep 1 done Recommended operating system tuning 20 numactl physcpubind Core localalloc nice 20 HP timetest7 2 v f csv o smi count On SLES the name of the IRQ balancer service is irq balancer Consider the following Consider changing the smp_affinity for the IRQs For example on a 2p16c server on which you want to leave cores and 8 for the OS the following masks off the other processors for all IRQs for MF in find proc irq name smp affinity do awk F i 1 i lt NF it printf 00000000 printf 8 8x n and 0x00000101 strtonum 0x SNF SMF gt SMF done Consider using cset hitp code google com p cpuset to shield cores from the OS For example on a 2p16c server on which you want to keep the OS from all cores except O and 8 use the following command cset shield cpu 1 7 9 15 kthread on If running as root the following command can then be used to move the current PID to the user set of cores 4 cset proc mov pid ss threads toset user Windows HP BIOS low latency options are supported in Windows Server 2008 and 2012 environments To apply the low l
3. Other considerations for turbo mode As noted in Active cores page 14 C states must be disabled in the BIOS However some versions of Linux ignore the BIOS setting and must be configured to disable C states For more information see Recommended Linux boottime settings on page 20 Disabling Processor Power and Utilization Monitoring and Memory Pre Failure Notification SMIs Disabling System Management Interrupts to the processor provides one of the greatest benefits to low latency environments Disabling the Processor Power and Utilization Monitoring SMI has the greatest effect because it generates a processor interrupt eight times a second in G6 G7 and Gen8 servers Disabling the Memory Pre Failure Notification SMI has a much smaller effect because it generates an interrupt at a lower frequency once per hour on and G7 servers and once every five minutes on Gen8 servers Disabling each option causes some server features to become unavailable Before reconfiguring BIOS be sure that none of the features described below are required Disabling Processor Power and Utilization Monitoring disables the following features Recommended platform tuning 14 e Processor State Monitoring e Insight Power Manager CPU Utilization Reporting e Dynamic Power Savings Mode Disabling Memory Pre Failure Notification has the following effects e Disables Memory Pre Failure Warranty Support e Disables notification when corr
4. Recommended platform tuning 11 Intel Virtualization Disabled Allows Virtual Machine Managers to utilize Technology virtualization hardware capobilities Intel Hyperthreading Options Disabled Allows Hyperthreading which adds logical cores but increases computational jitter Intel Turbo Boost Technology Enabled This option allows the processor to make a transition to a frequency that is higher than its rated speed Intel VT d Disabled Enables virtualized Directed Thermal Configuration First try Optimal Cooling then This option enables you to step through the repeat with Increased Cooling different available cooling settings available in and then Max Cooling if RBSU Use the one that provides the preferred available performance for the lowest power consumption For more information see Thermal Considerations on page 14 HP Power Profile Maximum Performance Disables all power management options that may negatively affect performance HP Power Regulator HP Static High Performance Keeps processors in their maximum Mode power performance state Intel Link Power Disabled Precludes placing unutilized links into low Management power state Minimum Processor Idle No C states Precludes processor transitions into low power Power Core State core C States Minimum Processor Idle No Package State Precludes processor transitions into low power Power Package State package C States Energy Performance Bias Maximum
5. a e terre Ter 21 i 27 m T mM 22 Frequently asked D TITTEN NE 23 Supe itae cm te at 25 Resources and documentation ae esae na nin epe pa Rn tett ttti pae eXXPOMR 25 Before you Contact 25 HP contact information 26 9 NR RR T a7 Documentation TEA 29 Contents 3 Introduction Overview Low latency deterministic system performance is a required system characteristic in the financial services market where it enables high frequency trading market data distribution and exchange data processing It is also required in other industries such as real time signal and image processing These systems must respond rapidly to external events in a predictable manner They must do so under heavy workloads sometimes reaching millions of transactions per second To achieve this level of performance system designers must consider the following factors during system design and configuration e Hardware System design processor type and speed memory latency speed and capacity network components storage subsystem including SSDs e OS selection Operating system kernels specifically designed and tuned for minimum latency and in some cases real time preemption e BIOS configuration BIOS support configured for minimum latency and maxim
6. Consider a single processor configuration if your application needs approximately 6 cores only The benefits are as follows Automatic PCl to core affinity no application rewrite DDIO performs optimally Cache snooping is eliminated latency Operation at the maximum turbo mode frequency is more likely due to reduced thermal power load Even with one processor there are still two x8 and one x16 PCle slots for NICs timing cards Fusion io and so forth The ProLiant BL460c Server Blade has one x8 mezzanine slot that communicates with processor 1 and one x8 mezzanine that communicates with processor 2 plus a FlexibleLOM off processor 1 PCle NIC o Mellanox ConnectX 3 based adapters offer ultra low latency and are designed specifically for HP servers in three form factors PCle card FlexibleLOM and server blade mezzanine They are sold integrated and directly supported HP Mellanox ConnectX 3 is the only NIC offering native Gen3 x8 performance 40GbE and FDR InfiniBand Recommended hardware configurations 6 TheSolarflare SFC9020 10GbE controller is now supported PCI option for HP ProLiant DL servers For more information see the HP 570SFP QuickSpecs on the HP website http h18004 www 1 hp com products quickspecs 14544 div 14544 div pdf o Additional popular third party PCle Ethernet cards for ultra low latency are available from Myricom and can be installed in HP indu
7. D ca ca ebd ed dur Qd pU a D Ip DN ERE M EISE DINE g Taking inventories or eee ener 9 9 Upgrading 10 Obtaining the Scripting Toolkit 10 i eia 11 SYStEMFEQUIFEMENTS Lc 11 Tuning recommendations and explanations RU bU ER QNO nee enone CERE MAR nesana 11 Turbo mode information and eorsiderdtiGHs cione rr QR 13 Disabling Processor Power and Utilization Monitoring and Memory Pre Failure Notification SMls 14 Disabling Dynamic Power Capping Functionality NER RARE 15 Disabling Patrol nat x erai nnda cim d Rav DU Rap aM Ren Scri R bonae iQ adu 15 Setting the Memory Relreshi Mate iioii tp ape CHUHERH EAE LIE rPptER RU Prop ERE en LM ad 15 Setting Memory Power Savings Mode and ACPI SLIT Preferences 15 Tuning MC Peppe 16 Recommended operating system 19 19 19 ET Em 20 Recommended Linux boot time settings mee eme nennen ener 20 Verifying the configUraliona xke ias aree peu pute FREE HER 20
8. Low Latency Applications White Paper 581608 005 includes the following additions and updates e Recommended hardware configurations on page 6 o Added information on the E5 2687W processor for the HP ProLiant DL360p Gen8 SE Server Introduction 4 Added information on the Solarflare SFC9020 10 GbE controller Added HP ProLiant DL360p Gen8 SE Server on page 8 Tuning recommendations and explanations on page 1 1 o Updated the following information in the table Intel Turbo Boost Technology Thermal Configuration o Added the following information to the table Dynamic Power Capping Functionality Memory Patrol Scrubbing Memory Refresh Rate Memory Power Savings Mode ACPI SLIT Preferences Updated Turbo mode information and considerations on page 13 and added the following subsections o Power consumption on page 13 o Thermal considerations on page 14 o Active cores on page 14 o Other considerations for turbo mode on page 14 Added Disabling Dynamic Power Capping Functionality on page 15 Added Disabling Patrol Scrubbing on page 15 Added Setting the Memory Refresh Rate on page 15 Added Setting Memory Power Savings Mode and ACPI SLIT Preferences on page 15 Updated the following tuning procedures o Tuning with the ROM based Setup Utility on page 16 o Tuning with the HP ROM Configuration Utility Gen8 and later on page 16 o Tuning with conrep on page 16 Linux o Updat
9. Performance Configures processor subsystems for high performance low latency Collaborative Power Control Disabled Precludes the OS from changing clock frequency DIMM Voltage Preference Optimized for Performance Runs DIMMs at a higher voltage if it increases performance Dynamic Power Capping Disabled This option allows for disabling System ROM Functionality Power Calibration during the boot process Doing so accelerates boot times but precludes enabling of a Dynamic Power Cap Memory Power Savings Maximum Performance This option configures several memory Mode parameters to optimize the memory subsystems performance and is configured to Balanced by default ACPI SLIT Preferences Enabled This ACPI SLIT describes the relative access times between processors memory subsystems and subsystems Operating systems that support the SLIT can use this information to improve performance by allocating resources and workloads more efficiently This option is disabled by default on most ProLiant Gen8 servers Processor Power and Disabled Disables iLO Processor State Mode Switching Utilization Monitoring and Insight Power Manager Processor Utilization Monitoring and its associated SMI Recommended platform tuning 12 Memory Pre Failure Disabled Disables Memory Pre Failure Notification and Notification its associated SMI Memory Patrol Scrubbing Disabled The Memory Periodic Patrol Scrubber corrects DL580 G7 memory soft errors so that ov
10. Refresh setting on Gen8 servers gt lt helptext gt platforms lt platform gt Gen8 lt platform gt lt platforms gt lt nvram gt 0x261 lt nvram gt lt value id 0x01 gt 1x_Refresh lt value gt lt value id 0x00 gt 2x Refresh lt value gt lt value id 0x02 gt 3x Refresh lt value gt lt mask gt 0x03 lt mask gt lt Section gt Section name Memory Double Refresh DL580G7 gt lt helptext gt lt CDATA This setting allows the user to change the Memory Double Refresh setting on the DL580 G7 server gt lt helptext gt lt romfamilies gt lt romfamily gt P65 lt romfamily gt lt romfamilies gt lt nvram gt 0x5F lt nvram gt lt value id 0x10 gt Disabled lt value gt lt value id 0x00 gt Enabled lt value gt lt mask gt 0x10 lt mask gt lt Section gt Section name Memory Patrol Scrubbing DL580G7 Recommended platform tuning 17 Xhelptext CDATA This setting allows the user to change the Memory Patrol Scrubbing setting on the DL580 G7 server gt lt helptext gt lt romfamilies gt lt romfamily gt P65 lt romfamily gt lt romfamilies gt lt nvram gt 0x6F lt nvram gt lt value id 0x10 gt Disabled lt value gt lt value id 0x00 gt Enabled lt value gt lt mask gt 0x10 lt mask gt lt Section gt Capture a snapshot of your current settings conrep s x conrep xml f conrep settings xml Browse through the menus to change the parameters For more in
11. address and your HP contact s name Can interrogate or confirm the memory operating speed A To interrogate or confirm the memory operating speed ensure your SMBIOS is 2 7 or later and use dmidecode 2 11 or later with the following command dmidecode t 17 How do tune a network adapter for optimum low latency A This white paper does not address this topic Refer to the supplier of the network adapter s controller technology For example tuning advice for Mellanox ConnectX 3 adapters integrated and supported by HP is available on the Mellanox website http www mellanox com related docs prod software Performance Tuning Guide for Mellanox Net work Adapters pdf How does HP recommend disable cores in ProLiant Gen8 servers A Do the following Frequently asked questions 23 1 From the RBSU menu navigate to System Options gt Processor Options gt Processor Core Disable Intel Core Select 2 Enter the number of cores per processor that you want to enable For example if you have 8 core processors and want to disable 1 core enter 7 in this field 3 Boot the server Verify that the correct information appears during POST for example 2 Processor s detected 14 total cores enabled The number of enabled cores can also be modified with hprcu or conrep To modify the number of enabled cores with conrep use version 3 40 or later available from STK for Linux 9 20 or later Frequently asked qu
12. operating state with the default Optimal Cooling setting If the system requires more cooling the server will respond by increasing the fan speed to deliver the necessary cooling However some demanding environments may require a greater base level of cooling If testing shows that your server s turbo frequency varies in response to exceeding temperature limits due to varying system load evaluate the Increased Cooling option which carries a penalty of increased system power consumption acoustics and airflow demand The third setting for this parameter is Maximum Cooling which causes the fans to always operate at their highest speed Use this setting only if your environment requires it as it has significantly higher power consumption acoustic noise and facility airflow demand Keep in mind that different processors have different requirements The E5 2687W has a notably higher TDP than the E5 2690 but the T for the E5 2687W is 5 C 9 lower than for the E5 2690 making proper cooling especially important Active cores In addition to TDP and thermals the amount of frequency boost obtained is a function of the number of active cores which is never more than the number of operational cores as specified by a BIOS setting Active cores are cores in CO or state and HP recommends disabling C states in order to keep the number of active cores constant and avoid the attendant latency jitter of changing turbo frequencies
13. segment e Support for the 150W 3 1 GHz Intel Xeon E5 2687W processor With Turbo Boost this processor is capable of running at between 3 4 and 3 8 GHz depending on the number of active cores with turbo boost stepping of 3 3 3 4 4 5 5 7 Compare this to the Intel Xeon E5 2690 range of 3 3 to 3 8 GHz with turbo boost stepping of 4 4 4 5 5 7 7 9 e Enhanced thermal design accommodates the higher wattage processors with standard air cooling e A PCle slot connected directly to each processor This provides a latency benefit to configurations with two processors and two I O cards such as Mellanox ConnectX 3 NICs for environments that pay careful attention to process placement e Updated BIOS to support the new processors and configuration As a Special Edition product the HP ProLiant DL360p Gen SE is only available through special order For more information or to place an order contact your HP representative or channel partner Although it is an Americas product it can be shipped to other geographies Recommended hardware configurations 8 Preparing for low latency configuration Taking inventories or snapshots Before you configure servers for low latency applications HP recommends that you take an inventory or snapshot of the following items This will enable you to track changes during the optimization process dmidecode For RHEL before 6 2 obtain v 2 11 from the nongnu website http www nongnu org dmidecode ls
14. website http www8 hp com us en support drivers html If the problem cannot be resolved at the website call 1 800 633 3600 For more information about Care Packs see the HP website http pro ag sama houston hp com services cache 10950 0 0 225 121 html On a best effort basis only HP offers technical assistance on low latency tuning to customers who have followed this guide and still have questions For more information contact HP mailto low latency hp com Provide your name and your company s name as well as your mailing address and your HP contact s name Support and other resources 26 Acronyms and abbreviations ACPI Advanced Configuration and Power Interface AMP Advanced Memory Protection HPRCU HP ROM Configuration Utility iLO Integrated Lights Out LAN on Motherboard MRG Messaging Realtime and Grid POST Power On Self Test RBSU ROM Based Setup Utility SLERT SUSE Linux Enterprise Real Time Extension SLIT System Locality Information Table SMI System Management Interrupt STK Scripting Toolkit Acronyms and abbreviations 27 Thermal Design Power Acronyms and abbreviations 28 Documentation feedback HP is committed to providing documentation that meets your needs To help us improve the documentation send any errors suggestions or comments to Documentation Feedback mailto docsfeedbackGhp com Include the document title and part number version number
15. 0 W 3 1 GHz 68 3 4 GHz 3 6 GHz If the penalty of computational jitter is too severe and you are unable to control temperature and you should disable Turbo mode It is possible to maintain a constant number of active cores Power consumption Pushing the processor s TDP limit will result in the processor changing its turbo frequency if the processor consumes too much power Because of the risk of processor failure Intel offers no method to lock a processor into Turbo Mode Most applications will not consume enough power to exceed the processor s TDP If you are concerned that yours might then you can disable a core per processor from within the BIOS reducing power consumption and providing TDP headroom Recommended platform tuning 13 Tests have shown that the E5 2690 processor under heavy computational load is able to stay at the maximum Turbo frequency indefinitely when the system is properly configured as outlined in this document However this is not guaranteed behavior and you should verify this with your application Thermal considerations The processor s thermal limits are another consideration in maintaining consistent turbo operation Ensure that the server s inlet temperature meets the specification in the associated QuickSpec Beyond thot there is a BIOS parameter that can be used to regulate the amount of cooling delivered by the fans but before changing it note that most configurations will maintain the preferred
16. 6 zlib devel i686 zlib i686 libstdct i686 Recommended platform tuning 16 e compat libstdct 296 1686 e compat libstdct 33 1686 To configure BIOS low latency options using the conrep utility in STK 9 40 1 Change the current directory to the STK utilities directory cd STK utilities 2 Edit the conrep xm1 file to include the following stanzas before lt Conrep gt at the end of the file Section name PowerMonitoring lt helptext gt lt CDATA This setting determines if Pstate logging and utilization is supported lt helptext gt ev COHGV3 ev lt length gt 1 lt length gt lt value id 0x00 gt Enabled lt value gt lt value id 0x10 gt Disabled lt value gt lt mask gt 0x10 lt mask gt lt byte gt 0 lt byte gt lt Section gt Section name DisableMemoryPrefailureNotification gt lt helptext gt lt CDATA This setting allows the user to disable Memory Pre Failure Notification support which will remove the periodic SMI associated with this support Not recommended for anyone except for those who absolutely need every periodic SMI removed gt lt helptext gt lt ev gt CQHGV3 lt ev gt lt length gt 1 lt length gt lt value id 0x00 gt No lt value gt lt value id 0x20 gt Yes lt value gt lt mask gt 0x20 lt mask gt lt byte gt 0 lt byte gt lt Section gt Section name Memory Refresh Gen8 gt helptext CDATA This setting allows the user to change the Memory
17. Configuring and Tuning HP ProLiant Servers for Low Latency Applications White Paper Abstract This document is intended to assist HP customers in configuring tuning and optimizing HP ProLiant servers for ultra low latency applications Part Number 581608 005 April 2013 Edition 5 Copyright 2009 2013 Hewlett Packard Development Company L P The information contained herein is subject to change without notice The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services Nothing herein should be construed as constituting an additional warranty HP shall not be liable for technical or editorial errors or omissions contained herein AMD is a trademark of Advanced Micro Devices Inc Intel and Intel Xeon are trademarks of Intel Corporation in the U S and other countries Windows Server is a U S registered trademark of Microsoft Corporation ConnectX 3 is a trademark of Mellanox Technologies Ltd Solarflare is a trademark of Solarflare Communications Inc Contents ee Ee ARIS I S NRI TIAS PAS 4 OVERVIEW 4 H 4 Recommended hardware configurations s ccic sicccccseisentecedsondnnedvesdsendsevsventeacoverdanndseedventecsdenstencdensoeants 6 HP Proliant DES60p Gen8 SE Server 8 Prepare Joriowdalency Geo Gur Bl
18. additional suggestions contact the HP Low Latency team HP TimeTest 22 Frequently asked questions Q Does disabling Memory Pre Failure Notification disable memory error correction A Memory errors are still corrected but notification that the error rate has exceeded a pre set threshold is disabled The latency impact of this feature is very small HP recommends disabling Memory Pre Failure Notification only if absolutely necessary What memory features are lost if Memory Pre Failure Notification is disabled A If Memory Pre Failure Notification is disabled Online Spare and Mirroring memory modes become unavailable The system is forced to run in Advanced ECC mode regardless of the mode set in BIOS Memory Pre Failure Warranty Support also becomes unavailable because there is no notification of errors exceeding the programmed threshold Q How does disabling iLO Processor State Monitoring in the HP ProLiant c Class enclosure affect power management A Disabling state monitoring does not affect power management How can verify that a server has the low latency option set A Use one of the following options to verify that the low latency option is set e the information in Tuning recommendations and explanations on page 1 1 e HP TimeTest to see if you are getting spikes For more information contact HP mailto low latency hp com Provide your name and your company s name as well as your mailing
19. aries based on the amount of installed memory Low Latency installations can avoid this SMI by disabling Patrol Scrubbing which is an option in the Service Options menu On other platforms Patrol Scrubbing does not require SMI functionality and does not need to be disabled Setting the Memory Refresh Rate An extremely rare potential for memory errors is eliminated by the default memory refresh rate of 2x Decreasing the rate to 1x will improve memory performance but with a vanishingly small potential for memory errors This affects G6 G7 and Gen8 servers This option is available in the Service Options menu Setting Memory Power Savings Mode and ACPI SLIT Preferences A new BIOS for many Gen8 platforms dated 20 August 2012 along with previous BIOS releases provides enhancements that are of interest to Low Latency environments Later versions of the BIOS are available but this version is cited as the earliest version to support these settings Two new BIOS setting available with this release are Memory Power Savings Mode and ACPI SLIT Preferences For more information on these settings see Tuning recommendations and explanations on page 11 Recommended platform tuning 15 Tuning procedures Tuning with the ROM based Setup Utility To configure BIOS low latency options using RBSU Tuning Tuning 1 2 3 4 Boot the server When prompted during POST press F9 to enter RBSU When the RBSU menu appears press CTRL A to dis
20. atency options in a Microsoft Windows environment 1 2 3 Obtain the STK Obtaining the Scripting Toolkit on page 10 Run the SmartComponent for the most recent version of the STK note the directory it is in and then change to it in Windows Explorer or a command window Run conrep Tuning with conrep on page 16 For other low latency tuning recommendations in a Windows environment do the following Review the technical information for Windows 2012 on the Microsoft website http technet microsoft com en us library hh831415 aspx See the Windows Server 2012 Tuning Guide on the Microsoft website http msdn microsoft com library windows hardware jj2487 19 For more information or assistance contact Microsoft to be put in touch with one of their low latency experts Recommended operating system tuning 21 HP Timelest The original behavior of HP TimeTest has been maintained through its many edits but this behavior is not optimal For example it runs at real time priority 99 but should be run at no higher than 80 On an otherwise idle system a reaHime priority of 1 is adequate for HP TimeTest to run properly The following provides an example of runn command time numactl physcpubind 3 localalloc nice n 20 b HP TimeTest HP TimeTest7 2 f csv o smi o date m cycles iE t echo 000005 2900000000 0k 1 pt de 1 expr 2900000000 60 30 44 p FIFO 80 20
21. conman cpuspeed cron cups cupsrenice dnsmasg dund firstboot hidd N ip6tables ipmi irda kudzu libvirtd N lvm2 monitor mcstrans mdmonitor mdmpd messagebus N multipathd netconsole netfs netplugd nscd N oddjobd pand pcscd postfix powersaved N psacct rdisc readahead early readahead later restoresecond rhnsd rpcgssd rpcidmapd rpcsvgssd saslauthd N sendmail slpd smartd smbfs suseRegister N sysstat wpa_supplicant xfs vpbind yum updatesd novell zmd do chkconfig level 2345 SSERVICE off service SSERVICE stop done 4 Use the irqbalancer to preclude some cores from servicing software IRQs Recommended operating system tuning 19 a Enter the following command service irgbalance stop b Do a one time run of irq balancer IROBALANCE ONESHOT 1 IRQBALANCE BANNED CPUS CoreMask irqbalance Wait until the command service irqbalance status returns irgbalance is stopped d On SLES the name of the IRQ balancer service is irq balancer Red Hat MRG Realtime Red Hat recently resolved scaling issues for the MRG 2 3 operating system for ProLiant servers with large core counts such as the DL580 G7 server with four 10 core E7 4870 processors If you are using MRG 2 3 on servers with a large number of cores be sure to use a release with a kernel version equal to or greater than the following kernel rt 3 6 11 rt30 25 el ort In addition to having a large number of cores
22. ectable memory errors occur above a pre defined threshold Forces the system to run in Advanced ECC Mode regardless of the mode configured in RBSU 27 IMPORTANT Online Spare Mode Mirroring Mode and Lock step Mode are not supported when Memory Pre Failure Notification support is disabled Supported AMP modes depend on the generation and model of the ProLiant server Disabling Memory Pre Failure Notification does not disable the Advanced ECC mode or correction of errors Uncorrectable errors are still flagged logged and bring the system down The only difference when this SMI is disabled is that there is no early notification if the uncorrectable error threshold is exceeded Disabling Dynamic Power Capping Functionality Disabling Dynamic Power Capping Functionality prevents the ability to enable a Power Cap via iLO When this parameter is disabled the option to enable a Power Cap via iLO is no longer available Since low latency installations are unlikely to set power caps the Dynamic Power Capping Functionality option may be safely disabled in the BIOS This option accelerates the boot process but does not have any impact on latency when the platform is operating Disabling Patrol Scrubbing Patrol Scrubbing is a feature that scans memory to correct soft memory errors On the HP ProLiant DL580 G7 and HP ProLiant DL980 G7 Servers the Patrol Scrubber re arms itself through an SMI The frequency of this event is roughly once per day but v
23. ed RHEL and SLES on page 19 o Added Red Hat Realtime on page 20 o Added Recommended Linux boottime settings on page 20 Added HP TimeTest on page 22 Introduction 5 Recommended hardware configurations HP recommends the following ProLiant Gen8 hardware configuration when low latency is required This information is subject to change and is valid as of the date of publication For the latest information see the server QuickSpecs on the HP website http www hp com go support Processor o 5 2690 and E5 2643 in BL servers HP recommends these processors for server blades o 5 2690 8c 2 9GHz and E5 2643 4c 3 3GHz in DL servers o E5 2687W 3 1 GHz in the HP Proliant DL360p Gen8 SE Server For more information see HP ProLiant DL360p Gen8 SE Server on page 8 For the latest information on supported processors see the server QuickSpecs on the HP website http www hp com go support Memory o 80r 16 Dual Rank 1600MT s CAS 11 RDIMMs o If installing only one DIMM per channel consider using 8 Dual Rank 1600MT s UDIMMs for a 1 clock latency advantage o Each channel should be populated with at least one DIMM PCle Gen3 architecture o HP ProLiant DL380p Gen8 Server offers three x8 or higher slots that communicate with processor 1 and three x8 or higher slots that communicate with processor 2 o HP ProLiant DL360p Gen8 Server offers three x8 or higher slots that communicate with processor 1
24. er the length of the system runtime the risk of producing multi bit and uncorrectable errors is reduced The default value for this parameter is Enabled Memory Refresh Rate 1 Refresh This option controls the refresh rate of the memory controller The default value for this parameter is 2x Memory Double Refresh Disabled This option controls the refresh rate of the DL580 G7 DL580 G7 memory controller The default value for this parameter is Enabled for a 2x refresh rate Turbo mode is enabled then step through the available cooling settings described in Thermal considerations on page 14 Otherwise the default value is adequate Turbo mode information and considerations Intel Turbo Boost can be used to increase the processor s operating clock frequency but at the risk of computational jitter if the processor changes its turbo frequency When that happens processing stops for a small period of time introducing uncertainty in application processing time Turbo operation is a function of power consumption processor temperature and the number of active cores Carefully managing these factors however can result in consistent turbo operation without jitter The maximum turbo frequencies for various numbers of active cores for two selected processors are given in the following table Processor Base frequency Number of active Turbo enabled cores frequency 5 2690 135 2 9 GHz 68 3 3 GHz 5 2687 15
25. estions 24 Support and other resources Resources and documentation The following resources are available HP ROM Based Setup Utility User Guide on the HP website http www hp com support rbsu iLO documentation o HP ilO 4 User Guide for Gen8 servers on the HP website http bizsupport2 austin hp com bc docs support SupportManual c03334051 c03334051 pdf o HPiLO 4 Scripting and Command Line Guide for Gen8 servers on the HP website http bizsupport2 austin hp com bc docs support SupportManual c03334058 c03334058 pdf o HP Proliant Integrated Lights Out 3 v1 20 User Guide for G7 servers on the HP website http bizsupport2 austin hp com bc docs support SupportManual c02774507 c02774507 pdf o HP Proliant Integrated Lights Out 3 v1 20 Scripting and Command Line Guide for G7 servers on the HP website http bizsupport2 austin hp com bc docs support SupportManual c02774508 c02774508 pdf HP Scripting Toolkit for Linux User Guide on the HP website http h20000 www2 hp com bc docs support SupportManual c03297832 c03297832 pdf HP Scripting Toolkit for Windows User Guide on the HP website http h20000 www2 hp com bc docs support SupportManual c03297836 c03297836 pdf STK on the HP website http www hp com go support The hcrpu and hpdiscovery utilities are available in the STK For more information on downloading STK see Obtaining the Scripting Toolkit on page 10 HP TimeTest 7 2 utilit
26. formation see Tuning recommendations and explanations on page 1 1 Update the BIOS with the modified settings conrep 1 x conrep xml f conrep settings xml Reboot the server reboot Recommended platform tuning 18 Recommended operating system tuning linux RHEL and SLES Before configuring a Proliant Gen8 server for low latency do the following 1 Make the following edits o For RHEL systems Edit boot grub grub conf and add nosoftlockup intel idle max cstate 0 mce ignore ce fo the kernel line o For SLES systems Edit boot grub menu lst and add intel idle max cstate 0 mce ignore to the kernel line nosoftlockup prevents RHEL from logging an event when a high priority thread executes continuously on a core for longer than the soft lockup threshold intel idle max_cstate 0 prevents the kernel from overriding the BIOS C state setting mce ignore ce prevents Linux from initiating a poll every five minutes of the Machine Check Banks for correctable errors which can cause latency spikes For more information see the Linux Kernel Archives website http www kernel org doc Documentation x86 x86_64 boot options txt 2 Reboot the server 3 After reboot run the stop services sh script to stop extraneous services The following example stops the services shown and prevents them from starting on subsequent boots for SERVICE in acpid alsasound autofs avahi daemon bluetooth
27. if your server is running the MRG 2 3 or later Realtime kernel it is using the SLUB memory allocator The SLUB memory allocator requires additional tuning for realtime performance The SLUB allocator has pseudo files named partial inthe sys kernel slab file system To get the best realtime performance from the allocator these files should be set to 0 disabling the cpu_partial logic This can be done with the following command find sys kernel slab name cpu partial exec echo 0 gt Recommended Linux boot time settings The Linux boot parameter idle poll keeps the processing cores in CO state when used in conjunction with intel_idle max_cstate 0 Without it the processor will enter C1 state RHEL systems Edit boot grub grub conf and add idle poll to the kernel line This is in addition to the nosoftlockup intel idle max cstate20 mce ignore_mce parameters that should have been added previously e For SLES systems Edit boot grub menu 1st and add idle poll to the kernel line This is in addition to the nosoftlockup intel_idle max_cstate 0 mce ignore_mce parameters that should have been added previously Verifying the configuration To verify your ProLiant server is properly configured for low latency operation clear one core selected at random of the operating system IRQs and then run the HP TimeTest utility on the randomly selected core Core 5 CoreMask echo 16
28. ions tab and then follow the instructions on the Online ROM Flash Component page Upgrading firmware Before making changes for low latency operation be sure that all platform firmware is up to date For low latency it is especially important to upgrade the network card and iLO 4 firmware to the latest versions To obtain the latest network card firmware Go to the HP website http www hp com go support Select Drivers amp Software Enter the server model number and then click Search 1 2 3 4 Select the appropriate product link 5 Select your operating system 6 Select Firmware Network 7 Download the appropriate NIC firmware To obtain the latest iLO 4 firmware Go to the HP website http www hp com go support Select Drivers amp Software Enter the server model number and then click Search Select the appropriate product link Select your operating system Select Firmware Lights Out Management DC Gee Ge des EA db o Click Obtain software and then click the executable file to download it Obtaining the Scripting Toolkit The conrep and hprcu utilities can be used to configure the BIOS for minimum latency and are included in the STK Use STK 9 40 or later For Gen8 servers SSSTK is now called STK hprcu is available for Gen8 servers only conrep is available for Gen8 servers and earlier conrep is the only method available for configuring these options on HP ProLiant G5 servers and HP ProLiant G se
29. n page 9 e Upgrade the BIOS Upgrading BIOS on page 9 Upgrade the firmware Upgrading firmware on page 10 e If using a Linux based server prepare the server for low latency tuning e the recommended changes to the BIOS For tuning recommendations and instructions see the following sections Tuning with the ROM Based Setup Utility on page 16 o Tuning with the HP ROM Configuration Utility Gen8 and later on page 16 o Tuning with conrep on page 16 e Verify the server is configured for low latency operation Verifying the configuration on page 20 HP servers are configured by default to provide the best balance between performance and power consumption These default settings may not provide the lowest latency The first step in tuning for low latency is to examine these additional settings that may assist in obtaining optimal low latency performance These settings are accessible through RBSU and with the conrep and hprcu utilities configuration tools provided by HP All HP ProLiant and later Intel based servers regardless of ROM version support setting Intel Turbo Boost and C States For G7 and earlier servers HP ProLiant 100 Series servers do not support advanced features for iLO Performance Monitoring and Memory Pre Failure notification The following table provides descriptions of the recommended low latency settings for Linux environments For recommended Windows settings see Windows on page 21
30. or the URL when submitting your feedback Documentation feedback 29
31. pci vv conrep for ProLiant Gen8 and earlier servers or hprcu for ProLiant Gen8 servers HP recommends using hprcu because of the additional benefits it provides over conrep hpdiscovery To obtain the latest versions of hprcu or hpdiscovery see Obtaining the Scripting Toolkit on page 10 sysctl a HP TimeTest7 2 HP TimeTest is a utility distributed by HP that enables customers to test for jitter in a server To obtain the HP TimeTest utility contact HP mailto low latencyGhp com Provide your name and your company s name as well as your mailing address and your HP contact s name Capture kernel boot settings o cat boot grub grub conf for RHEL o cat boot grub menu lst for SLES Upgrading BIOS Before making BIOS changes for low latency operation upgrade the BIOS to the most recent version To obtain the most recent BIOS upgrade for HP Proliant servers Go to the HP website http www hp com go support Select Drivers amp Software Enter the server model number and then click Search Select the appropriate product link Select your operating system Select the BIOS System ROM category To obtain the BIOS upgrade do one of the following o Download the latest firmware and then upgrade the firmware using the instructions included with the ROMPaq Preparing for low latency configuration 9 Select Online ROM Flash Component click the Installation Instruct
32. play the option for the Service Options menu Browse through the menus to change the parameters For more information see Tuning recommendations and explanations on page 1 1 27 IMPORTANT Do not change the other options in the Services Options menu Verify that the parameters are set as indicated in Tuning recommendations and explanations on page 11 with the HP ROM Configuration Utility Gen8 and later To configure BIOS low latency options using the hprcu utility in STK 1 Change the current directory to the STK utilities directory STK utilities 2 Capture a snapshot of your current settings specifying to include the Service Options settings hprcu a s f hprcu_settings xml conrep requires editing the conrep xml file to access the Service Options settings With hprcu it is only necessary to specify the undocumented option Edit the file npzcu settings xml to obtain the preferred settings as listed above 4 Browse through the menus to change the parameters For more information see Tuning recommendations and explanations on page 1 1 5 Update the BIOS with the modified settings hprcu a 1 f hprcu settings xml 6 Rebootthe server with conrep conrep is a 32 bit executable and requires 32 bit libraries when run on a 64 bit operating system For example you may need to install the following list of packages glibc i686 nss softokn freebl i6086 libxml2 i686 libxml12 devel i68
33. rvers that utilize AMD Opteron processors The utility is one method available for configuring HP ProLiant and G7 servers that utilize Intel Xeon processors To install the STK 1 Go to the HP website http www hp com go ProLiant STK Select STK for Linux Select Download Create a new directory Unpack archive in new directory Preparing for low latency configuration 10 Recommended platform tuning System requirements The HP BIOS configuration options described in this document include options in HP ProLiant servers to disable the generation of periodic System Management Interrupts SMls used for Power Monitoring and for Memory PreFailure Notification with their attendant latency impact BIOS options are generally independent of the OS and a properly tuned low latency operating system is also required to achieve deterministic performance The tuning recommendations described in this document are based on testing and customer interactions But no single recipe can be prescribed Customers needing a low latency environment often perform exhaustive testing of the latency impact of various tuning parameters with their application and systems to determine the optimum settings for their environment Tuning recommendations and explanations Consider the following options as part of any deployment in low latency OS kernel environments e Take an inventory or snapshot Taking inventories or snapshots o
34. stry standard ProLiant DL ML and SL servers o The graph below shows the HP Mellanox FlexibleLOM Part 649282 B21 in a back to back configuration no switch running Netperf v2 5 0 Netperf TCP Latency on DL380p with ConnectX 3 14 1 2 RT 12 amp 10GbE 40GbE 10GbE VMA S 40GbE VMA gt gt L 6 4 2 0 1 4 16 64 256 1024 4096 Message Size bytes o half round trip with core 2 as an example uses the following command line netperf n 16 H peer ip c C 0 t RR 1 10 T 2 2 r lt message size gt e Storage o Compared with G7 storage controllers the HP SmartArray P420 storage controller for Gen8 servers has double the cache size 6 times the performance with SSD and double the number of supported drives o HP I O Accelerator now supports up to 1 2TB MLC in server blade mezzanine cards For more information see the HP IO Accelerator for HP BladeSystem c Class QuickSpecs on the HP website http h18004 www hp com products quickspecs 13220 div 13220 div pdf e Tuning See Tuning recommendations and explanations on page 1 1 Recommended hardware configurations 7 HP ProLiant DL360p Gen8 SE Server HP has made available a new variant of the DL360p Gen8 server designed specifically for low latency environments the HP ProLiant DL360p Gen8 SE Server This special edition server has multiple features in response to the requirements of this market
35. um performance e Networking fabric Network technology 1 10 40 Gigabit Ethernet InfiniBand Fibre Channel e Middleware Messaging and database services on the network designed for minimum latency and maximum throughput with reliability e End user applications Designed to perform multicast messaging accelerated via kernel bypass and RDMA techniques Physical distances Physical separation between the information sources and clients affects overall system performance This document presents suggestions and best practice recommendations on BIOS configuration and on OS tuning to obtain the lowest latency performance from HP ProLiant BL c Class server blades and HP ProLiant DL ML and SL servers While this document contains information pertaining to G7 and earlier Proliant servers the primary focus is Gen8 servers and later The recommendations to disable System Management Interrupts SMIs are intended only for extremely latency sensitive use cases Most customers benefit from the power savings monitoring and notifications that the SMIs enable These SMIs consume less than 0 1 of the server s processing capability and HP continues to reduce their impact with each new generation of ProLiant server Ur IMPORTANT The information in this document is accurate as of the document s release date but is subject to change based on updates made by HP What s new The current edition of the Configuring and Tuning HP Proliant Servers for
36. y To obtain the utility contact HP mailto low latency hp com Provide your name and your company s name as well as your mailing address and your HP contact s name Before you contact HP Be sure to have the following information available before you call HP Active Health System log HP ProLiant Gen8 or later products Download and have available an Active Health System log for 3 days before the failure was detected For more information see the HP iLO 4 User Guide or HP Intelligent Provisioning User Guide on the HP website http www hp com go ilo docs Onboard Administrator SHOW ALL report for HP BladeSystem products only Support and other resources 25 For more information on obtaining the Onboard Administrator SHOW report see the HP website http www hp com go OAlog e Technical support registration number if applicable e Product serial number e Product model name and number e Product identification number e Applicable error messages e Add on boards or hardware e Third party hardware or software Operating system type and revision level HP contact information For United States and worldwide contact information see the Contact HP website http www hp com go assistance In the United States e contact HP by phone call 1 800 334 5144 For continuous quality improvement calls may be recorded or monitored e Ifyou have purchased a Care Pack service upgrade see the Support amp Drivers
Download Pdf Manuals
Related Search