Home

RU Builder User Manual

1. To date the event driven threading model has been tested the most and was successfully used in the CMS Magnet Test and Cosmic Challenge MTCC of 2006 and in the global runs of 2007 The self driven threading model has been kept as a debugging aid as the occupancy of the message FIFOs in this threading model give a clear indication of whether or not an application is keeping up with its workload DAQ CERN PH CMD 2008 Page 27 of 54 6 Application state machines RU Builder User Manual 6 Application state machines 6 1 Commonalities of the application finite state machines The finite state machines of the BUs EVM and RUs have commonalities Figure 13 shows the finite state transition network FSTN which all three types of application follow There are four common behaviors Firstly all RU builder applications read and act upon configuration parameters when they receive a Configure SOAP message Secondly all RU builder applications only participate in event building when they are enabled Thirdly all RU builder applications throw away their internal data and any incoming I20 message frames when they are halted Fourthly all RU builder applications go to the Failed state when they encounter a fatal error A RU builder application cannot be recovered from the Failed state Configuration parameters are read and acted The application can only participate in event upon when the application is configured building when it is in the Enable
2. I20 EVM ALLOCATE CLEAR I20 BU_CACHE Figure 7 RU builder 12O0 messages when connected to the rest of the EVB DAQ CERN PH CMD 2008 Page 17 of 54 4 RU builder 120 interface RU Builder User Manual I20 EVMRU_DATA READY I20_TA_CREDIT I20 RU READOUT I20 EVM TRIGGER I20 RU_SEND I20 EVM ALLOCATE CLEAR I20_BU_CACHE Figure 8 RU builder 120 messages when connected within a local DAQ or test beam system DAQ CERN PH CMD 2008 Page 18 of 54 4 RU builder 120 interface RU Builder User Manual 4 2 The external 120 interfaces of the RU builder Figure 9 shows the external 120 interfaces of the RU builder the TA EVM interface the RUI RU interface and the BU FU interface This chapter is divided into three sub sections one for each interface RUI RU interface uae RU builder TA EVM interface BU FU interface ttf a ws Figure 9 External I2O interfaces of the RU builder DAQ CERN PH CMD 2008 Page 19 of 54 4 RU builder 120 interface RU Builder User Manual 4 2 1 TA EVM interface The TA EVM interface specifies how The EVM gives the TA trigger credits The TA gives the EVM trigger data Figure 10 is a sequence diagram describing the protocol between the EVM and the TA TA EVM Function code I20 TA CREDIT Step 1 C structure I20 TA CREDIT MESSAGE FRAME Function code I20 EVM TRIGGER C structure I20 EVENT DATA BLOCK MESSAGE FRAME ae ad Figure 10 TA EVM interface sequence d
3. then they are referred to its website http www doxygen org Doxygen documentation for this particular version of the RU builder can be browsed on line at the official RU builder web site http cms ru builder web cern ch cms ru builder rubuilder G V01 06 00 doxygen html index htm It must be emphasized that the RU builder is still under development and subject to change No description of the RU builder given in this document can be relied upon to be valid beyond this release 1 2 Intended readership This document has been written with two audiences in mind system integrators and DAQ operators A system integrator is someone that needs to integrate the RU builder into a data acquisition system DAQ It is assumed that the DAQ system is based on the XDAQ framework If the reader is not familiar with this framework then they are referred to the XDAQ website http xdag web cern ch xdag A DAQ operator should know how the RU builder works so that they can efficiently diagnose problems that may occur during a DAQ shift DAQ CERN PH CMD 2008 Page 7 of 54 1 Introduction RU Builder User Manual 1 3 References 1 The CMS collaboration The Trigger and Data Acquisition project Volume II Data Acquisition amp High Level Trigger CERN LHCC 2002 26 ISBN 92 9083 111 4 1 4 Definitions Acronyms and Abbreviations BU Builder Unit I20 Intelligent Input Output CVS Concurrent Versioning System RCMS CMS Run control system DAQ D
4. 5 BU Default web page Standard monitoring parameters cccsceceeeeeeeeeeeeees 39 Table 6 BU Debug web page Debug configuration parameters sssesseeeeeeeeeeees 40 Table 7 BU Debug web page Debug monitoring parameters seeeesssessssssssssseseerrreeeee 40 Table 8 EVM Default web page Data flow through node parameters eeeeeeees 4 Table 9 EVM Default web page Standard configuration parameters eeeeeeeeeees 41 Table 10 EVM Default web page Standard monitoring parameteTS cccccceeesseeeees 43 Table 11 EVM Debug web page Debug configuration parameters ccccccceeeeneeeees 44 Table 12 EVM Debug web page Debug monitoring parametefs scssseeeeeeeeeees 45 Table 13 RU Default web page Data flow through node parameters cccccceseeeees 46 Table 14 RU Default web page Standard configuration parameters cccccceeesseeeees 46 Table 15 RU Default web page Standard monitoring parameters ccccccceeeeeeeeeeeees 47 Table 16 RU Debug web page Debug configuration parameters ccceeeceseeeeeeeeeeeeeeees 48 Table 17 RU Debug web page Debug monitoring parameters essssssssoeeressssssseeeeee 49 1 Introduction RU Builder User Manual I Introduction The RU builder is a dist
5. Default web page Standard monitoring parameters DAQ CERN PH CMD 2008 Page 43 of 54 10 RU builder application web pages User Manual RU Builder rcmsStateListener selfDriven workLoopName ageMessages msgAgeLimitDtMSec monitoringSleepSec generateDummy Triggers fedPayloadSize dummyTriggerSourceld IZ20_TA_CREDIT_Packing IZO0_RU_READOUT_Packing rulnstances oldMessageSenderSleepUSec dumpTriggersToLogger The class name and instance number of the RCMS state listener The threading model used by the application A value of true indicates self driven whereas a value of false indicates event driven See chapter 6 RU builder threading models for more information The name of the worker thread when the application is using the self driven threading model See chapter 6 RU builder threading models for more information Specifies whether or not control messages should be aged i e whether or not a control message should be sent if it is too old as opposed to only when it is full How old a message can be in milliseconds before it will be sent if ageMessages Is set to true The number of seconds the monitoring thread should sleep between monitoring information updates Specifies whether or not the EVM should generate dummy triggers When generateDummyTriggers is set to true this parameter specifies the size of the FED payload of dummy triggers in bytes The FED payload must be a multiple of 8 b
6. Overview of the RU builder I2O interface 4 2 The external I2O interfaces of the RU builder 4 2 1 TA EVM interface 4 2 2 RU RUI interface 4 2 3 BU FU interface RU builder threading models Application state machines 6 1 Commonalities of the application finite state machines 6 2 BU EVM and RU finite state machines Starting the RU builder Stopping the RU builder Exported configuration parameters RU builder application web pages 10 1 BU web pages 10 2 EVM web pages 10 3 RU web pages How to install the RU builder Example configuration file COCO CONN N a Ca O WO 12 12 13 13 14 15 16 16 19 20 22 23 27 28 28 29 31 32 33 36 36 41 46 50 50 13 14 RU builder self test Configuration guidelines 51 54 List of Figures Figure 1 RU builder connected to the rest of the EVB cc ccccccccccceceeeneeeeesessssseseeeeeeeeeeees 9 Figure 2 RU builder connections within a local DAQ or test beam system ccceeeeeees 10 Peiro BO FIPO cous seers sce necele tele to E oe persed ae eect chit oamesace auas eens A E 12 Figure 4 EVM FIFOs when trigger sent via RUI EVM interface ec eeeeeeeeeeeseeeeeeeeeeeees 13 Fiure SEVM FIFO oerni en a rA ageless eaadehietece 14 Powe ERUPE O ceea eaa a a a cea eee ne 15 Figure 7 RU builder 120 messages when connected to the rest of the EVB eee 17 Figure 8 RU builder 1I120 messages when connected within a lo
7. What the EVB does For each event the EVB Reads out the trigger data This trigger data will become the first super fragment of the event Reads out the fragments of the event from the detector front end drivers FEDs Builds the fragments into RU super fragments using the FED builder Builds the whole event using the RU builder The whole event is the trigger super fragment plus the set of RU super fragments Decides whether or not the event is interesting for physics using the filter units FUs Sends the event to permanent storage if it is interesting for physics or discards it if it is not 2 3 The RU builder applications The RU builder consists of a single event manager EVM one or more readout units RUs and one or more builder units BUs The EVM is responsible for controlling the flow of data through the RU builder The RUs are responsible for buffering super fragments until they are requested by the BUs The BUs are responsible for building and buffering events until they are requested by the filter units FUs The trigger adapter TA readout unit inputs RUIs and filter units FUs are external to the RU builder The TA is used within local DAQ and test beam systems to interface the DAQ trigger to the EVM The RUIs are responsible for pushing super fragment data from the FED builder into the RUs The FUs are responsible for selecting interesting events for permanent storage DAQ CERN PH CMD 2008 Page 11 of 54 3 RU builder
8. application FIFOs RU Builder User Manual 3 RU builder application FIFOs The RU builder applications use FIFOs to keep track of requests trigger data and event data Knowledge of these FIFOs is required in order to correctly configure the RU builder This chapter is divided into three sections one for the BU one for the EVM and one for the RU Each section gives a brief description of the application s behavior and how its FIFOs are used 3 1 BU FIFOs A BU is responsible for building events An event is composed of one trigger super fragment and N RU super fragments where N is the number of RUs To understand the internal FIFOs of a BU it is necessary to know its dynamic behavior Figure 3 shows the internal FIFOs of a BU With free capacity available a BU requests the EVM to allocate it an event step 1 The EVM confirms the allocation by sending the BU the event ID and trigger data of an event step 2 This trigger data is the first super fragment of the event The BU now requests the RUs to send it the rest of the event s super fragments step 3 The BU builds the super fragments it receives from the RUs step 4 into a whole event within its resource table step 5 FUs can ask a BU to allocate them events step 6 A BU services a FU request by sending the FU a whole event step 7 When a FU has finished with an event it tells the BU to discard it step 8 The event IdFIFO blockFIFO requestFIFOs and discardFIFO store incoming
9. display something similar to mane Tubuilder tester Applicationd Enabled EVM 0 Cill mane AML Tit Page last updated Tue Aug 7 15 28 41 2007 UTC BU a HvperDAQ Control Messages Main EVMO Event Nb 0 Figure 16 rubuilder tester Application web page DAQ CERN PH CMD 2008 Page 52 of 54 13 RU builder self test RU Builder User Manual Step 6 Go to the control web page by clicking on the Ctrl icon You should now see something like TA rubuilder tester ApplicationO Enabled EVM 0 Cir Megs FANER XML RUI Test RUQ Page last updated Tue Aug 7 15 30 54 2007 UTC Bud FU HyperDAQ Control Messages Main stor Figure 17 rubuilder tester Application control web page Step 7 Start the self test by clicking the start button The button should now be labeled stop Step 8 Return to the main web page of the RU builder tester by clicking on the RUB Test icon Clicking the refresh button of your browser should now show events being built For example the eventNb variable of the EVMO should increment RUNE rubuilder tester Application0 Enabled AML Ciri lgs Seer Test R Page last updated Tue Aug 7 15 32 29 2007 UTC g FU AyperDAC Control Messages ai 5 EYMO Event Nb 101504 Figure 18 Running RU builder DAQ CERN PH CMD 2008 Page 53 of 54 14 Configuration guidelines RU Builder User Manual 14 Configuration guidelines This chapter summarizes and highlig
10. in bytes that passed mPa e OE CRS cL stateName od Te current state of the application o a er nbSuperFragmentsInRU The occupancy of the RU as the number of super fragments e e Ee a ve Numae romi VAE eere ee DEE E A one A A N a en Ae R _lastEventNumberloBUs The lastevent number senttothe BUS o i aseneseaseiaeni nbSuperFragments The number of super fragments that have passed through the nee eee vee E a nbEmptySuperFragments The number of empty super fragments generated by the RU a gn ea vv Te PPE E L E a IZO0_EVMRU_DATA_READY_Payload The payload in bytes transferred by IZO0_EVMRU_DATA_READY messages See eaaa lL OEVMRU_DATALREADY Logical Count I20_EVMRU_DATA_READY_LogicalCount The number of super fragments received from the RUI RUIs send RUs super fragment data using ee 12 O EVMRU_DATAREADY messages I2D_EVMRU_DATA_READY_120Count The 12O message frame count for IZQ0_EVMRU_DATA_READY messages See IZ0_EVMRU_DATA_READY_LogicalCount DAQ CERN PH CMD 2008 Page 46 of 54 10 RU builder application web pages RU Builder User Manual IZ0_RU_READOUT_LogicalCount The number of event number event ID pairs the RU has received from the EVM The EVM sends event number event ID pairs to the RUs using IZO_RU_READOUT a B SSS o a I2D0_RU_READOUT_120Count The 12O message frame count for 12O_RU_READOUT a messages See 120_RU_READOUT_LogicalCount I2O0_RU_SEND_Payload_BUn The payload in bytes transferred by I12O_RU_S
11. it is empty and the data rate is 0 The sample period of the rate calculation is defined by the value of monitoringSleepSec _nbTriggers The number of triggers that have entered the EVM nbevtsBuilt The number of events built by the RU builder lastEventNumberFromTrigger The last event number that the EVM received from the a trigger _lastEventNumberToRUs Te last event number sent to the RUS _lastEventNumberToBUs CT last event number sent to the BUs aa IZ0_TA_CREDIT_Payload The payload in bytes transferred by 120_TA_CREDIT a messages See 12O_TA_CREDIT_LogicalCount IZ20_TA_CREDIT_LogicalCount The number of trigger credits sent to the TA The EVM sends the TA trigger credits using I2O_TA_CREDIT a eal messages a I20_TA_CREDIT_I20Count The 12O message frame count for 12O_TA_CREDIT Fo ee ee ee el messages See 12O_TA_CREDIT_LogicalCount IZO0_EVM_TRIGGER_Payload The payload in bytes transferred by 12O0_EVM_TRIGGER pe ges ec es aca des coca eeaeerseeeraencellal messages See 12O_EVM_TRIGGER_LogicalCount IZO_EVM_TRIGGER_LogicalCount The number of triggers received from the TA The TA sends the EVM triggers using IZO_EVM_TRIGGER messages I2O_EVM_TRIGGER_I20Count The 120 message frame count for 12O_EVM_TRIGGER E messages See I2O_EVM_TRIGGER_LogicalCount IZO0_EVMRU_DATA_READY_Payload The payload in bytes transferred by IZO0_EVMRU_DATA_READY messages See aaa L2ZO_EVMRU_DATA_READY_LogicalCount IZO0_EVMRU_DATA_READY_LogicalCount Th
12. scenario the RU builder is connected to the rest of the EVB This is how the RU builder will be connected during official CMS data taking runs In the second scenario the RU builder is connected within a local DAQ or test beam system The difference between the two scenarios is how the triggers are sent to the EVM Triggers are sent to the EVM via the RUI EVM interface when the RU builder is connected to the rest of the EVB Triggers are sent to the EVM via the TA EVM interface when the RU builder is used in a local DAQ or test beam system Figure 1 depicts the first scenario and figure 2 the Trigger fragments EEE Fragments FED builder RUI 0 RUI 1 RUI n Trigger super fragments eesseee Super fragments second rem eo RU builder Permanent storage The external interfaces of the RU builder assume that triggers are given to the EVM in the same order as their corresponding event data is given to the RUs Figure 1 RU builder connected to the rest of the EVB DAQ CERN PH CMD 2008 Page 9 of 54 2 RU builder overview RU Builder User Manual Trigger credi ts anarai Super fragments Permanent storage Figure 2 RU builder connections within a local DAQ or test beam system The external interfaces of the RU builder assume that triggers are given to the EVM in the same order as their corresponding event data is given to the RUs DAQ CERN PH CMD 2008 Page 10 of 54 2 RU builder overview RU Builder User Manual 2 2
13. the RUI informs the RU of event data that is ready to be processed step 2 A RU places each super fragment for which it has received a pair into the fragment lookup table step 3 BUs ask RUs to send them the super fragments of the events they are building step 4 A RU services a BU request by retrieving the super fragment from its fragment lookup table and asking the BU to cache the super fragment step 5 All of the internal FIFOs of a RU that is to say the pairFIFO blockFIFO and requestFIFOs store incoming messages until they can be processed RUI blockFIFO EVM pairFIFO Readout L Hh i g aihh CequestCELEFOS BU Send Cache BU Figure 6 RU FIFOs The RU can be set into a fault tolerant mode in which it continues to operate if the RUI stops sending super fragments In that case the RU will go into the TimedOutTolerating state and serve empty super fragments for each BU request and ignore the EVM If data is again made available by the RUI the RU synchronizes the pairFIFO and blockFIFO by removing messages if earlier event numbers Once the FIFOs are aligned the RU goes back to the normal processing Enabled state DAQ CERN PH CMD 2008 Page 15 of 54 4 RU builder 120 interface RU Builder User Manual 4 RU builder 120 interface This chapter is divided into two sections The first gives an overview of the RU builder I2O interface which shows which I2O messages are sent between which applicatio
14. will ask the EVM to allocate it an event step 4 Within such a request a BU will normally give back the ID of an event to be cleared For each cleared event ID the EVM makes the ID a free event ID step 5 The EVM confirms the allocation of an event by sending the requesting BU the event ID and trigger data of the allocated event step 6 The triggerFIFO clearedEventIdFIFO and requestFIFO store incoming messages until they can be processed The pairFIFO keeps track of the event ID trigger data pairs that have yet to be sent to requesting BUs The freeEventIdFIFO stores the ids of free events RUI Trigger RU Readout pairFIFO clearedEventIdFIFO freeFventIdFIFO Ct ir Allocate and or clear Step 5 Step 4 recquestr ILKO gt BU LTT 1 Figure 4 EVM FIFOs when trigger sent via RUI EVM interface Confirm DAQ CERN PH CMD 2008 Page 13 of 54 3 RU builder application FIFOs RU Builder User Manual 3 2 2 EVM FIFOs when triggers are sent via the TA EVM interface Figure 5 shows the internal FIFOs of the EVM The EVM tells the TA the capacity of the RU builder by sending it trigger credits step 1 One trigger credit represents the ability to build one event Given a credit the TA sends the EVM the trigger data of an event step 2 The EVM pairs the trigger data with a free event ID step 3 The EVM also requests the RUs to readout the event s data step 4 A BU with the ability to build an event will ask
15. 384 A BU has a single FIFO called discardFIFO for FU discard messages and one FIFO per FU called requestFIFO for FU request messages Knowing that a BU can service a maximum of 64 FUs and assuming that a single FU will never have more than 1024 outstanding requests for events rubuilder bu Application discardFIFOCapacity 1024 x maximum number of FUs 1024 x 64 65536 rubuilder bu Application requestFIFOCapacity 1024 The default values of the RU exported parameters were calculated as follows The blockFIFO of the RU is responsible for buffering incoming super fragment data Assuming a RUI only gives as many super fragments to a RU as the TA gives triggers to the EVM then in the worst case the blockFIFO must hold the blocks of as many super fragments as there are event ids in the RU builder rubuilder ru Application blockFIFOCapacity nbEvtIdsInBuilder x size of a super fragment block size 4096 x 16K 4KB 16384 The parameters of the form _Packing should not normally be modified as they have only been tested with the default value of 8 However the user may modify them if they are experiencing performance problems with the RU builder DAQ CERN PH CMD 2008 Page 35 of 54 10 RU builder application web pages RU Builder User Manual 1ORU builder application web pages The BU EVM and RU applications each have default and debug web pages This chapter describes the parameters displayed on these pag
16. ATH XDAQ ROOT bin export LD LIBRARY _PATH LD LIBRARY PATH XDAQ ROOT lib Step 4 Run the XDAQ executive xdaq exe h HOST p PORT e XDAQ ROOT etc default profile c CONFIGURATION_FILE XML Where osr and porr are the same as those given in step 2 and CONFIGURATION FILE XML is the file modified from chapter 12 DAQ CERN PH CMD 2008 Page 51 of 54 13 RU builder self test RU Builder User Manual Step 4 Open a web browser and enter the following URL http HOST PORT Where HOST and PORT are the same as those of steps 2 and 3 You should now see the HyperDAQ page of the XDAQ executive you started in step 3 It should look similar to the following screenshot i Zones default Groups profile AQ http localhost 65432 urn xdaq application lid 3 designed by J Gutleber L Orsini HyperDAQ Q ve Q Control Panel Cluster Explore executive pt http PeerTrans hyperdag xrelay urn xdaq urnxdaq urn xdaq urm xdaq application lid 0 application lid 1 application lid 3 application lid 4 pt fifo PeerTrans rubuilder tester rubuilder evm rubuilder ru rubuilder bu urn xdaq urn xdaq urn xdaq urn xdaq urn xdaq application lid s application lid 12 application lid 13 application lid 14 application lid 15 Figure 15 HyperDAQ web page for self test Step 5 Go to the default web page of rubuilder tester Application by clicking on the rubuilder tester link Your web browser should now
17. Count The 12O message frame count for 12O_BU_CONFIRM oe eee messages See LO_BU_CONFIRM_LogicalCount IZ2O0_RU_SEND_Payload The payload in bytes transferred by I20_RU_SEND oe eee messages See ZO_RU_SEND_LogicalCount I2O0_RU_SEND_LogicalCount The number of super fragments requested from the RUs BUs request super fragments from the BUs using I20_RU_SEND messages I20_RU_SEND_1I20Count The I20 message frame count for 12O_RU_SEND messages sasha cane eceeeceneeeiacencsceeateceseseceneseheneseet Oo Oe OU aaao IZ20_BU_CACHE_Payload The payload in bytes transferred by 120_BU_CACHE a E messages See I2O_BU_CACHE_LogicalCount 20_BU_CACHE_LogicalCount The number of super fragments received from the RUs RUs send BUs super fragment data using 120_BU_CACHE ee ee E ee S Leo axial a ne rn en I20_BU_CACHE_120Count The 12O message frame count for 12O_BU_CACHE messages See 120_BU_CACHE_LogicalCount Table 5 BU Default web page Standard monitoring parameters DAQ CERN PH CMD 2008 Page 39 of 54 10 RU builder application web pages RU Builder User Manual rcmsStateListener The class name and instance number of the RCMS state listener selfDriven The threading model used by the application A value of true indicates self driven whereas a value of false indicates event driven See chapter 6 RU builder threading models for more information workLoopName The name of the worker thread when the application is using the self driven
18. ED e g The EVM is in the Halted state The RU builder is said to be flushed when it is empty and the data rate is 0 The sample period of the rate calculation is defined by the value of the following exported parameter rubuilder evm Application monitoringSleepSec Halting a RU builder application causes it to discard destroy all the data in its internal data structures and to release all incoming 120 messages DAQ CERN PH CMD 2008 Page 32 of 54 9 Exported configuration parameters RU Builder User Manual 9 Exported configuration parameters Configuration parameters need to be set before an application is sent a Configure SOAP message Table 2 lists the exported control parameters of each type of RU builder application The type and default value of each parameter is given Parameter name nbEvitidsInBuilder ageMessages msgAgeL imitDiNSec exitOnFail blockFIFOCapacity discardFlIFOCapacity 20_EVM_ALLOCATE_CLEAR_Packing maxEvtsUnderConstruction requestFlFOCapacity 2O0_RU_SEND_Packing UnsignedInteger32 sendCreditsWithDispatchFrame riggerFIFOCapacity UnsignedInteger32 2O0_RU_READOUT_Packing UnsignedInteger32 2O0_TA_CREDIT_Packing UnsignedInteger32 blockFIFOCapacity UnsignedInteger32 16384 BU rubuilder bu Application EVM rubuiilder evm Application RU rubuilder ru Application Table 2 Exported configuration parameters The default values are set when the RU builder application is instantiated The d
19. END messages See RO_RU_SEND_LogicalCount_BUn I2O_RU_SEND_LogicalCount_BUn The number of requests for super fragments received from a specific BU BUs request super fragments from RUs using I2Z0_RU_SEND messages I20O_RU_SEND_1I20Count_BUn The 12O message frame count for 12O_RU_SEND messages eaaa SEC IZO_LRU_SEND_LogicalCount BUN IZ20_BU_CACHE_Payload The payload in bytes transferred by 120_BU_CACHE ee eee eee ene era messages See I2O_BU_CACHE_LogicalCount IZ20_BU_CACHE_LogicalCount The number of super fragments sent to the BUs RUs send super fragment data to the BUs using 120_BU_CACHE ENEE ON Dar CO DC EEA EEEE E oll oie A ee oe ANETE TE OE De ene Oe En I20_BU_CACHE_ 20Count The 12O message frame count for 12O_BU_CACHE messages See 120_BU_CACHE_LogicalCount Table 15 RU Default web page Standard monitoring parameters rcmsStateListener The class name and instance number of the RCMS state listener selfDriven The threading model used by the application A value of true indicates self driven whereas a value of false indicates event driven See chapter 6 RU builder threading models for more information workLoopName The name of the worker thread when the application is using the self driven threading model See chapter 6 RU builder threading models for more information monitoringSleepSec The number of seconds the monitoring thread should sleep between monitoring information updates tolerateCSCFaults Specifi
20. Partition xmlns soapenc http schemas xmlsoap org soap encoding xmlns xc http xdaq web cern ch xdaq xsd 2004 XMLConfiguration 30 xmins xsi http www w3 org 2001 XMLSchema instance gt lt i2o0 protocol xmlns i20 http xdag web cern ch xdag xsd 2004 1I20Configuration 30 gt lt i2o0 target class rubuilder evm Application instance 0 tid 23 gt lt i2o0 target class rubuilder ru Application instance 0 tid 24 gt lt i2o0 target class rubuilder bu Application instance 0 tid 25 gt lt i20 protocol gt lt xc Context url http HOST PORT gt lt xc Module gt XDAQ ROOT 1lib librubuilderutils so lt xc Module gt lt xc Module gt XDAQ ROOT 1lib libxdaq2rc so lt xc Module gt lt xc Application class rubuilder tester Application id 12 instance 0 network local gt lt xc Module gt XDAQ ROOT 1lib librubuildertester so lt xc Module gt lt xc Application class rubuilder evm Application id 13 instance 0 network local gt lt xc Module gt XDAQ ROOT 1ib librubuilderevm so lt xc Module gt lt xc Application class rubuilder ru Application id 14 instance 0 network local gt lt xc Module gt XDAQ ROOT 1ib librubuilderru so lt xc Module gt lt xc Application class rubuilder bu Application id 15 instance 0 network local gt lt xc Module gt XDAQ ROOT 1ib libr
21. The Compact Muon Solenoid Experiment TriDAS Trigger and Data Acquisition RU Builder User Manual Version rubuilder G V01 12 00 August 21 2008 Authors R Mommsen S Murray CI identifier EVB D_ 18306 CERNY 2 Fermilab Revision History a irr ee July 31 2003 Finalization of version 1 0 J Gutleber May 24 2004 2 0 Updated for version 2 0 of the EVB S Murray now referred to as the RU builder November 1 2004 2 1 Updated for version 2 1 of the RU S Murray builder May 2 2005 3 0 Updated for version 3 0 of the RU S Murray builder July 4 2005 3 2 Updated for version 3 2 of the RU S Murray builder August 16 2005 3 3 Updated for version 3 3 of the RU S Murray builder August 26 2005 Updated for version 3 4 of the RU S Murray builder October 5 2005 Updated for version 3 5 of the RU S Murray builder October 27 2005 Updated for version 3 6 of the RU S Murray builder January 30 2006 3 7 1 Updated for version 3 7 1 of the RU S Murray builder March 29 2006 3 8 Updated for version 3 8 of the RU S Murray builder May 8 2006 3 9 Updated for version 3 9 of the RU S Murray builder September 18 2006 3 9 8 Updated for version 3 9 8 of the RU S Murray builder August 7 2007 RUBUILDER_G_V1_3_0 Updated for the latest version of S Murray the RU builder October 11 2007 rubuilder_G_V01_06_00 Updated for the latest version of S Murray the RU builder Added a descri
22. _DISCARD ee eee eee messages See 12O_BU_DISCARD_LogicalCount IZ20_FU_TAKE_Payload The payload in bytes transferred by 12O_FU_TAKE a E messages See I2O_FU_TAKE_LogicalCount IZ20_FU_TAKE_LogicalCount The number of events sent to the FUs BUs send event data ooo eee e eee eeeeeeeeeeeeeeeee to the FUs using ZO_FU_TAKE messages I20_FU_TAKE_I20Count The 120 message frame count for 12O_FU_TAKE messages oe eee eect eect e eee eeeeeee SEE RO_FU_TAKE_LogicalCount eee IZO0_EVM_ALLOCATE_CLEAR_Payload The payload in bytes transferred by IZO_EVM_ALLOCATE_CLEAR messages See sate epee eee eeens nerean o MANTO CATE eee I20_EVM_ALLOCATE_CLEAR_LogicalCount The number of event IDs that have been requested plus the number of event IDs that have been recycled cleared BUs request new event IDs from the EVM and recycle clear old event IDs using ZO_EVM_ALLOCATE_CLEAR messages DAQ CERN PH CMD 2008 Page 38 of 54 10 RU builder application web pages RU Builder User Manual IZO_EVM_ALLOCATE_CLEAR_120Count The 120 message frame count for IZO_EVM_ALLOCATE_CLEAR messages See I2Z0_EVM_ALLOCATE_CLEAR_LogicalCount IZ20_BU_CONFIRM_Payload The payload in bytes transferred by 120_BU_CONFIRM Bede tee Seats ete cpedeaesdeeueseesesesasle messages See 12O_BU_CONFIRM_LogicalCount IZ20_BU_CONFIRM_LogicalCount The number of event IDs received from the EVM The EVM sends event IDs to the BU using I20_BU_CONFIRM_LogicalCount messages I20_BU_CONFIRM_1I20
23. acilitate efficient lookups in the EVM and RUs A negative value is interpreted as meaning this parameter has not been set If this parameter is not set when the BU is configured then the BU will use its instance of MUM occ eeeeee eee ee eee eens evminstance The BU uses this parameter to determine the instance number of the EVM which manages it If this parameter is not set then the BU will use the first EVM it finds in the configuration given to its XDAQ executive A negative value is interpreted as meaning this parameter has not been set alee Ue eee ere USL AT ee i A ee maxEvtsUnderConstruction The maximum number of events the BU can build concurrently blockFlFOCapacity The capacity of the event data block FIFO DAQ CERN PH CMD 2008 Page 36 of 54 10 RU builder application web pages RU Builder User Manual discardFlFOCapacity The capacity of the discard FIFO Table 4 BU Default web page Standard configuration parameters DAQ CERN PH CMD 2008 Page 37 of 54 10 RU builder application web pages RU Builder User Manual runNumber The current run number nbEventsInBU The occupancy of the BU as the number of events currently in the BU This number is calculated by subtracting the number of events sent to the FU or dropped from the total number of events built deltaT The duration of the last delta t in seconds with respect to poses E A monitoring events that pass through the BU deltaN The umber of events that passed th
24. ata Acquisition system RU Readout Unit EVB Event builder RUI Readout Unit Input EVM Event Manager TA Trigger Adapter FED Front End Driver TDR Technical Design Report FSTN Finite State Transition Network TriDAS Trigger and Data Acquisition FU Filter Unit XDAQ Cross platform data acquisition toolkit 1 5 RU builder application class names The following table shows the names of the RU builder applications and their corresponding C class names The C class names are used when making XDAQ configurations for the RU builder They are also used on the web pages of the running RU builder applications Application name C class name Acronym Full name Builder unit rubuilder bu Application Event manager rubuilder evm Application Filter unit rubuilder fu Application Readout unit rubuilder ru Application Readout unit input rubuilder rui Application Trigger adapter rubuilder ta Application Table 1 RU builder application class names DAQ CERN PH CMD 2008 Page 8 of 54 2 RU builder overview RU Builder User Manual 2 RU builder overview 2 1 How the RU builder connects to the components around it The RU builder is a component of a larger system called the event builder EVB The EVB is a distributed application that reads out event fragments from one set of nodes and assembles them into entire events in another set of nodes Two scenarios dictate how the RU builder connects to the components around it In the first
25. cal DAQ or test beam system E aaa ae T A I E E E ees ia aaa A E T E EE 18 Figure 9 External IZO interfaces of the RU builder eer Ee 19 Figure 10 TA E VM interface sequence Cia oan is iecciandscesswesreeedesssewsiiwssoncinad ducwabsawaedassseuceawiads 20 Figure 11 RU RUI interface sequence diagram eeeceseecccccceeeseesseecceceeseaaeesseeeceeeeeeaaeenees 22 Pisure 12 BU FUinterlace SequenCe dido traMeesarerr a E saiaaeandec dustin 23 Ficure 13 FSIN ofa RU builder appliCationis cncsvssacerscasensicvetanesstansSesesedenetieaeeetieectanessiaubetaes 28 Pipe 14 BU EVM and RU ESTING ore aE EAA E EAA EE 29 Figure 15 HyperDAQ web page Tor self test cctcseccscccsdiancasntsasiavavendcasensswccnhlenntssetsaiiaeaveadoesaunsds 52 Figure 16 rubuilder tester Application Web page ccccsseceecccceceeeeeeeeeaeaaeeessssseeeeeeeeeeeees 52 Figure 17 rubuilder tester Application control Web page sssseeeeeeeeeeesssssssssssssseeeeereeeees 53 Picure IS Rumnine RU DUJE C 2234 scot canst N dacetthebodacecisuataetataatandes 53 List of Tables Table 1 RU builder application class names essessssoeeensssssssseeerrssssssssceerssssssssceerssssssssseeeeo 8 Table 2 Exported Conmmeuration Paranmiele ts eera e E E ER 33 Table 3 BU Default web page Data flow through node parameters ccceeeeeeeeees 36 Table 4 BU Default web page Standard configuration parameters sssseeeeeeeeees 37 Table
26. d state i N N Fatal error The contents of all internal data i structures are discarded when the application is halted j The application releases all incoming 120 message The application cannot be recovered frames when it is in the Halted state from the Failed state Figure 13 FSTN of a RU builder application DAQ CERN PH CMD 2008 Page 28 of 54 6 Application state machines RU Builder User Manual 6 2 BU EVM and RU finite state machines Halted Halted Configure Configure Enable Enable Send request to s ofevgne eeubigiaiisaie wll ae Failed Failed EMMET ig leah eel credits to TA set of event Halted Configure Ready Enable lahiqve Clerc MismatchDetected E Tolerating TimedOut tolerating true Mismatch tolerating fa fse TimedOut Tolerating MismatchDetected BackPressuring TimedOut tolerating false TimedOut BackPressuring Figure 14 BU EVM and RU FSTNs The FSTNs specific to each type of RU builder application are shown in figure 10 DAQ CERN PH CMD 2008 Page 29 of 54 6 Application state machines RU Builder User Manual DAQ CERN PH CMD 2008 Page 30 of 54 7 Starting the RU builder RU Builder User Manual 7 Starting the RU builder The RU builder is part of a larger system called the event builder EVB Besides run control the RU builder communicates with a TA one or more RUIs and one of more FUs The RU builder cannot be started at any arbitrary
27. e number of triggers received from the RUI The RUI sends the EVM trigger data using IZO0_EVMRU_DATA_READY messages DAQ CERN PH CMD 2008 Page 42 of 54 10 RU builder application web pages User Manual RU Builder The 12O message frame count for IZO0_EVMRU_DATA_READY messages See IZ0_EVMRU_DATA_READY_LogicalCount The payload in bytes transferred by I20O_RU_READOUT messages See 120_RU_READOUT_LogicalCount The number of event number event ID pairs received from the EVM The EVM sends the RUs event number Event ID pairs using IZO_RU_READOUT messages The 120 message frame count for messages I2O_RU_READOUT See I20_RU_READOUT_LogicalCount The payload in bytes transferred by IZO0_EVM_ALLOCATE_CLEAR messages See I20_EVM_ALLOCATE_CLEAR_LogicalCount The number of requests for new event IDs plus the number of old event IDs to be recycled cleared received from the BUs The BUs request new event IDs from the EVM and ask the EVM to recycle clear old ones using IZ0_EVM_ALLOCATE messages The 120 message frame count for IZO_EVM_ALLOCATE_CLEAR messages See IZ0_EVM_ALLOCATE_CLEAR_LogicialCount The payload in bytes transferred by 120_BU_CONFIRM messages See 120_BU_CONFIRM_LogicalCount The number of event IDs sent to the BUs The EVM sends event IDs to the BU using I20_BU_CONFIRM messages The 12O message frame count for 12O_BU_CONFIRM messages See 120_BU_CONFIRM_LogicalCount Table 10 EVM
28. efault values have been chosen with the goal of covering the majority of use cases for the RU builder A user should rarely need to diverge from these default values DAQ CERN PH CMD 2008 Page 33 of 54 9 Exported configuration parameters RU Builder User Manual The following assumptions were made when calculating the default values of the RU builder s configuration parameters A RU builder is composed of 64 BUs and 64 RUs A RU has 64MB of physical memory for caching super fragments An event is 1MB An event is made up of 64 super fragments 1 per RU of equal size therefore the size of a super fragment is 16KB The block size size of an 12O message frame used to transport event data is 4KB The RUIs only give as many events to the RUs as the TA gives triggers to the EVM The maximum number of FUs per BU is 64 A FU will never have more than 1024 outstanding requests for events Fast control messages are sent if they are older than 1 second The packing factor of fast control messages is 8 The need to know the total number of event ids in the RU builder is common to all three types of RU builder applications The total number of event ids in the RU builder is a function of RU memory Assuming each RU has 64MB of memory for buffering super fragments and that the size of an event is 1 MB rubuilder bu Application nbEvtIdsInBuilder rubuilder evm Application nbEvtIdsInBuilder rubuilder ru Application nbEvtIdsInBuild
29. er sum of the memory of all RUs size of an event 64 x 64MB 1MB 4096 The BUs and EVM send fast control messages Fast control messages are sent if they are older than 1 second rubuilder bu Application ageMessages rubuilder evm ApplicationageMessages true rubuilder bu Application msgAgeLimitDtMSec rubuilder evm Application msgAgeLimitDtMSec 1000 It has been assumed that all events have the fixed size of IMB If the RU builder is to build events of varying sizes then the appropriate safety factor needs to be taken into account when calculating the number of event ids in the RU builder DAQ CERN PH CMD 2008 Page 34 of 54 9 Exported configuration parameters RU Builder User Manual The default values of the BU control parameters were calculated as follows To prevent a BU from monopolizing event ids each BU has a maximum number of event ids it can acquire at any moment in time Assuming all BUs are equal each BU is allowed to acquire rubuilder bu Application maxEvtsUnderConstruction nbEvtIdsInBuilder number of BUs 4096 64 64 The blockFIFO of a BU see figure 2 in section 3 2 is responsible for buffering incoming event data In the worst case this FIFO would have to buffer the blocks of all outstanding requests for event data rubuilder bu Application blockFIFOCapacity maxEvtIdsUnderConstruction x size of an event block size 64 x 1 MB 4 KB 64 x256 16
30. es This chapter has been written to be used as a reference by DAQ operators This chapter is divided into three sections one for each type of application Each section contains a set of tables which describe the parameters displayed on the default and debug web pages of the corresponding application type 10 1 BU web pages throughput bytes sec 1 The event throughput in bytes per second average bytes The average size of an event rate events sec 1 The number of events per second rms bytes The RMS of the event size Table 3 BU Default web page Data flow through node parameters fragmentSetsUrl If useFragmentSet is set to true then this parameter must be set to the URL of the fragment sets file The BU will use this file to determine which RUs ee S BS La e e useFragmentSet Indicates whether the BU is using a fragments sets file or the value of rulnstances to determine which RUs are participating in event building If set to true the BU will use the fragments sets file specified by eee tragmentSetsUrl else the BU will use the contents of rulnstances fragmentSetld The BU can use a fragment sets file to determine which RUs are participating in event building If useFragmentSet is set to true and fragmentSetsUrl specifies a valid location of a valid fragment sets file then this parameter specifies which fragment set within the file is to be Index BU indices are internal to the RU builder application They are used to f
31. es whether or not the RU should tolerate CSC local DAQ faults dummyBlockSize When generateDummySuperFragments is set to true this parameter specifies the size in bytes of a dummy super fragment data block generateDummySuperFragments Specifies whether or not the RU should generate dummy super fragments dummyFedPayloadSize When generateDummySuperFragments is set to true this parameter specifies the payload size in bytes of a FED fragment within a dummy super fragment fedSourcelds When generateDummySuperFragments is set to true this parameter specifies the source ID of each fragment within a dummy super fragment maxPairAgeMSec The maximum age of an event number event ID pair in milliseconds A negative value means forever expiredPairDetectionSleepUSec The number of microseconds the thread checking for an expired pair sleeps between checks tolerateFaults Specifies whether or not the RU should service BUs with empty super DAQ CERN PH CMD 2008 Page 47 of 54 10 RU builder application web pages RU Builder User Manual fragments when in the TimedOut state Only relevant if maxPairAgeMSec is not negative Table 16 RU Debug web page Debug configuration parameters DAQ CERN PH CMD 2008 Page 48 of 54 10 RU builder application web pages RU Builder User Manual Description measured TimeOutSec The seconds component of the time period that passed and triggered the finite state machine of the application
32. file to determine which RUs are participating in event building If useFragmentSet is set to true and fragmentSetsUrl specifies a valid location of a valid fragment sets file then this parameter specifies which fragment set within the file is to be used nbEvtldsInBuilder The total number of event IDs in the RU builder triggerFIFOCapacity The capacity of the trigger FIFO taClass The class name of the TA talnstance The instance number of the TA Table 9 EVM Default web page Standard configuration parameters DAQ CERN PH CMD 2008 Page 41 of 54 10 RU builder application web pages RU Builder User Manual LCL ITAL 6610121600100 01 3 EE nbTriggersInEVM The occupancy of the EVM as the number of triggers ee a nee nner E L E LLE S A E deltaT The duration of the last delta t in seconds with respect to e ee eee eee monitoring triggers that pass through the EVM deltaN The number of triggers that passed through the EVM in the ee rE a deltaSumOfSquares The sum of the squares with respect to the sizes of the a CE ee deltasumOfSizes The sum of the trigger sizes that passed through the EVM in Ear CO TOE A 1 en SI a O a foundTA Specifies whether or not the EVM has found the TA The ee P A Wi Ue Siecle NEN itie CONNGUTEO o ere ruBuilderlsFlushed Tri state variable specifying whether or not the RU builder is flushed O FALSE 1 TRUE 2 UNDEFINED e g The EVM is in the Halted state The RU builder is said to be flushed when
33. hts the most important points with regards to configuring the RU builder The default values of the RU builder control parameters have been chosen for the majority of use cases The user should rarely need to diverge from these values The RU builder is dependent on the instance numbers of the BUs EVM FUs RUs and TA RUs must be assigned instance numbers from 0 to the number of RUs 1 BUs must be assigned instance number from 0 to the number of BUs 1 The EVM must be assigned instance number 0 The TA must be assigned instance number 0 The RU builder has the following configuration restrictions A single BU can service a maximum of 64 FUs The sum of the maximum number of event ids each BU can have at any moment in time rubuilder bu Application maxEvtsUnderConstruction must not exceed the total number of event ids in the RU builder rubuilder bu Application nbEvtIdsInBuilder rubuilder evm Application nbEvtIdsInBuilder rubuilder ru Application nbEvtIdsInBuilder If the total number of event ids is exceeded then there is no guarantee that the EVM will be able to buffer BU requests for event ids or that the RUs will be able to buffer BU requests for event data The configuration parameters of a RU builder application must be set before it is configured DAQ CERN PH CMD 2008 Page 54 of 54
34. iagram The EVM communicates with the TA using a credit based mechanism The EVM tells the TA the current capacity of the RU builder by sending the TA a trigger credit count step 1 One trigger credit represents the RU builder s ability to build one event The TA should only send the EVM trigger data for as many events as the EVM has given the TA credits step 2 The TA is responsible for getting receiving trigger data from the trigger and for providing backpressure to the trigger as necessary The I20_TA CREDIT MESSAGE FRAME C structure is as follows typedef struct 120 TA CREDIT MESSAGE FRAME I20 PRIVATE MESSAGE FRAME PvtMessageFrame U32 nbCredits I20 TA CREDIT MESSAGE FRAME PI20 TA CREDIT MESSAGE FRAME The EVM must fill nbCredits DAQ CERN PH CMD 2008 Page 20 of 54 4 RU builder 120 interface RU Builder User Manual The I20_ EVENT DATA BLOCK MESSAGE FRAME C structure is as follows typedef struct I20 EVENT DATA BLOCK MESSAGE FRAME I20 PRIVATE MESSAGE FRAME PvtMessageFrame U32 U32 U32 U32 U32 U32 U32 U32 U32 eventNumber nbBlocksInSuperFragment blockNb eventid buResourceld fuTransactionid nbSuperFragmentsInEvent superFragmentNb padding I20 EVENT DATA BLOCK MESSAGE FRAME PI20 EVENT DATA BLOCK MESSAGE FRAME The TA must fill eventNumber nbBlocksInSuperFragment blockNb A Thee RU builder only supports single block trigger data Therefore the TA must always set
35. message until they can be processed The full ResourceFIFO stores which in memory events are built ep3 Jf RU no Sepa blockFIFO Confirm LLL Resource table Step 5 Allocate new and or clear previous 1 ep 1 p E fullResourceFIFO discardFIFO Je janani Step 6 FU Allocate discard FU take Figure 3 BU FIFOs DAQ CERN PH CMD 2008 Page 12 of 54 3 RU builder application FIFOs RU Builder User Manual 3 2 EVM FIFOs The EVM is responsible for controlling the flow of event data through the RU builder To understand the internal FIFOs of the EVM it is necessary to know its dynamic behavior As explained in Section 2 1 How the RU builder connects to the components around it triggers are sent to the EVM using the RUI EVM interface when the RU builder is connected to the rest of the EVB and via the TA EVM interface when connected within a local DAQ or test beam system This section is divided into two sub sections tone describing the EVM FIFOs when the triggers are sent via the RUI EVM interface and the other describing the EVM FIFOs when triggers are sent via the TA EVM interface 3 2 1 EVM FIFOs when triggers are sent via the RUI EVM interface Figure 4 shows the internal FIFOs of the EVM The RUI sends the EVM the trigger data of an event step 1 The EVM pairs the trigger data with a free event ID step 2 The EVM also requests the RUs to readout the event s data step 3 A BU with the ability to build an event
36. moment in time its start up must be synchronized with that of the TA RUIs and FUs The RU builder and the EVB components it interacts with are XDAQ applications and as such depend on one or more peer transports to communicate with each other These peer transports must be up and running before these applications try to communicate with each other To start the RU builder and the components it interacts with run control should do the following in order 1 Start the necessary peer transports so that the RU builder and its surrounding applications can communicate 2 Start the TA if it is present so that is can service credits from the EVM 3 Start the RU builder so that it can receive super fragments from the RUIs Start the RUIs and FUs so they can start pushing in super fragments and extracting events respectively The RU builder is a distributed application whose nodes BUs EVM and RUs need to be started up in a specific order To put the RU builder into the state where it will build events run control should do the following in order Send Configure to all of the RU builder applications Send Enable to all of the RUs Send Enable to the EVM Send Enable to the BUs A UW N State changes are synchronous There is no need to poll the state of an application after a request to change state The RUs have to be enabled first because they have to be ready to receive event number event ID pairs from the EVM The EVM can start se
37. nbBlocksInSuperFragment to 1 and blockNb to 0 DAQ CERN PH CMD 2008 Page 21 of 54 4 RU builder 120 interface RU Builder User Manual 4 2 2 _RU RUI interface The RU RUI interface specifies how a RUI passes super fragments to a RU Figure 11 is a sequence diagram describing the protocol between the RUI and the RU RUI RU Function code I20 EVMRU DATA READY C structure 120 EVENT DATA BLOCK MESSAGE FRAME gt Figure 11 RU RUI interface sequence diagram A super fragment is composed of one or more I20 EVENT DATA BLOCK MESSAGE FRAMEs The I20 EVENT DATA BLOCK MESSAGE FRAME C structure is as follows typedef struct I20 EVENT DATA BLOCK MESSAGE FRAME I20 PRIVATE MESSAGE FRAME PvtMessageFrame U32 eventNumber U32 nbBlocksInSuperFragment U32 blockNb U32 eventId U32 buResourceld U32 fuTransactionId U32 nbSuperFragmentsInEvent U32 superFragmentNb U32 padding I20 EVENT DATA BLOCK MESSAGE FRAME PI20 EVENT DATA BLOCK MESSAGE FRAME The RUI must fill eventNumber nbBlocksInSuperFragment blockNb The nbBlocksInSuperFragment field gives the number of blocks the super fragment is composed of The blockNb field indicates the block s position within the super fragment Blocks are numbered from 0 to nbBlocksInSuperFragment 1 DAQ CERN PH CMD 2008 Page 22 of 54 4 RU builder 120 interface RU Builder User Manual 4 2 3 BU FU interface The BU FU interface specifies how A FU requests events f
38. nding these pairs immediately after it has been enable The EVM has to be enabled before the BUs so that it is ready to service their requests for event ids BUs can start requesting event ids as soon as they are enabled Enabling the EVM causes it to send an initial trigger credit count to the TA if it is present The number of initial credits is equal to the total number of event ids in the RU builder As soon as trigger data arrives at the EVM the EVM sends event number event ID pairs to the RUs As explained in the previous paragraph this is why the RUs have to be enabled before the EVM Enabling a BU causes it to send its initial request for event ids to the EVM The number of initial event ids requested is equal to the maximum number of event ids the BU is allowed to acquire at any single moment in time DAQ CERN PH CMD 2008 Page 31 of 54 8 Stopping the RU builder RU Builder User Manual 8 Stopping the RU builder The current version of RU builder foresees two ways of stopping Stop the trigger and event data entering the RU builder Halt all of the RU builder applications When stopping the trigger and event data entering the RU builder it is useful to know when the RU builder is flushed This can be found out by reading the following tri state exported parameter of the EVM xsd unsignedint rubuilder evm Application ruBuilderIsFlushed The possible values of this parameter are O FALSE 1 TRUE 2 UNDEFIN
39. ng models event driven and self driven Knowledge of these threading models is required in order to correctly configure the RU builder Only one threading model can be active during the lifetime of an application The threading model used is chosen on the first ever configure of an application The following exported parameters allow the user of the RU builder applications to choose which threading model is used xsd boolean rubuilder bu application selfDriven xsd boolean rubuilder evm application selfDriven xsd boolean rubuilder ru application selfDriven A value of true indicates the self driven threading model whereas a value of false indicates event driven The default threading model is event driven A RU builder application contains message FIFOs to store incoming messages until they can be processed In both the event driven and thread driven models the underlying peer transport thread s push incoming messages onto these FIFOs Which thread s pop messages off these FIFOs depends on the threading model being used In the case of the event driven threading model the peer transport thread which pushed a message onto a message FIFO is also the same thread which pops and processes messages of the message FIFOs In the case of the self driven threading model a separate worker thread pops and processes messages from the message FIFOs A counting semaphore is used to synchronize the worker thread with the peer transport thread s
40. ns This overview will help the reader understand the message counters displayed on the web pages of the running applications The second section of this chapter goes into the details of the external 120 interfaces of RU builder This will help the reader integrate the RU builder with the DAQ components around it 4 1 Overview of the RU builder 120 interface All the 120 messages of the EVB including the internal and external messages of the RU builder are defined in the package TriDAS dagq interface The 120 function codes of all the RU builder 12O messages are given in the file TriDAS dagq interface shared include i2oXFunctionCodes h The C structures that define the 120 messages are in the file TriDAS dag interface evb include i20EVBMsgs h The 20 interface of the RU builder is subject to change The description of the interface provided by this document cannot be relied upon to be valid beyond this release DAQ CERN PH CMD 2008 Page 16 of 54 4 RU builder 120 interface RU Builder User Manual As explained in Section 2 1 How the RU builder connects to the components around it triggers are sent to the EVM using the RUI EVM interface when the RU builder is connected to the rest of the EVB and via the TA EVM interface when connected within a local DAQ or test beam system Figures 7 and 8 show the RU builder 12O messages in these two scenarios I20 EVMRU DATA READ I20 EVMRU DATA READY I20 RU READOUT I20 RU_SEND
41. nt is composed of I20 EVENT DATA BLOCK FRAMEs The nbSuperFragmentsInEvent superFragmentNb nbBlocksInSuperFragment blockNb fields are used to identify the position of an event data block within an event An event is composed of one trigger super fragment plus N RU super fragments where N is the number of RUs Therefore the nbSuperFragmentsInEvent field is set to the number of RUs plus 1 The superFragmentNb field is numbered from 0 to nbSuperFragmentInEvent 1 The blockNb field is numbered from 0 to nbBlocksInSuperFragment 1 The buResourceld field is an opaque handle that a FU should use to identify events resources to be discarded The fuTransactionlId field is the FU transaction ID of the FU request that caused the BU to reply with the current event DAQ CERN PH CMD 2008 Page 25 of 54 4 RU builder 120 interface RU Builder User Manual The I20_BU_ DISCARD C structure is as follows typedef struct 120 BU DISCARD MESSAGE FRAME I20 PRIVATE MESSAGE FRAME PvtMessageFrame U32 n U32 buResourceId 1 I20 BU DISCARD MESSAGE FRAME PI2O BU DISCARD MESSAGE FRAME The FU must fill n buResourceld The n field specifies the number of events resources to be discarded The buResourcelTd field is an array of the ids of the BU resources to be discarded DAQ CERN PH CMD 2008 Page 26 of 54 5 RU builder threading models RU Builder User Manual 5 RU builder threading models The BU EVM and RU applications have two threadi
42. nts in the cleared event ID FIFO The number of elements in the free event ID FIFO The number of elements in the trigger FIFO The number of elements in the pair FIFO The number of elements in the request FIFO CERN PH CMD 2008 Page 44 of 54 10 RU builder application web pages RU Builder User Manual Table 12 EVM Debug web page Debug monitoring parameters DAQ CERN PH CMD 2008 Page 45 of 54 10 RU builder application web pages RU Builder User Manual 10 3 RU web pages throughput bytes sec 1 The super fragment throughput in bytes per second average bytes The average size of a super fragment rate events sec 1 The number of super fragments per second rms bytes The RMS of super fragment size Table 13 RU Default web page Data flow through node parameters nbEvtldsInBuilder The total number of event IDs in the RU builder blockFlFOCapacity The capacity of the block FIFO Table 14 RU Default web page Standard configuration parameters T LSA O deltaT The duration of the last delta t in seconds with respect to be SEENE A monitoring super fragments that pass through the RU n deltaN Number of super fragments that passed through the RU in the eave Dv ESSE OPS F019 ee deltaSumOfSquares The sum of the squares with respect to the sizes of the super fragments in bytes that passed through the RU in the last ac deltaSumOfSizes The sum of the super fragment sizes
43. ption of the two ways to send triggers to the EVM Added a simple description of all the RU builder 120 messages Added a description of the two RU builder threading models Added more information about the contents of the RU builder application web pages August 21 2008 rubuilder_G_V01_12_00 Updated naming of eventNumber R Mommsen monitoring parameters and document the fault _ tolerant behavior of the RU CI Record rea pm O CI Identifier EVB_D_18306 Description Describes the RU builder for an integrator who will put the builder into a full DAQ system and for a DAQ operator who needs to understand how the RU builder works in order to efficiently diagnose problems Point of Contact R Mommsen Remigius Mommsen cern ch S Murray Steven Murray cern ch Physical Location http cms ru builder web cern ch cms ru builder RUBUILDER G V1 6 0 doc 10 11 12 Table of Contents Introduction 1 1 Document purpose and scope 2 Intended readership 3 References 4 Definitions Acronyms and Abbreviations 5 RU builder application class names RU builder overview 2 1 How the RU builder connects to the components around it 2 2 What the EVB does 2 3 The RU builder applications RU builder application FIFOs 3 1 BU FIFOs 3 2 EVM FIFOs 3 2 1 EVM FIFOs when triggers are sent via the RUI EVM interface 3 2 2 EVM FIFOs when triggers are sent via the TA EVM interface 3 3 RU FIFOs RU builder I20 interface 4 1
44. questing The allocate field is an array of FU transaction ids and fragment sets For each event a FU requests the FU fills in the fuTransactionId field and the fset field of a BU_ALLOCATE C structure and puts it in the allocate array The fuTransactionId field is a transaction ID that a FU can use to match its requests with the events it receives A BU treats the fuTransactionlId field as being opaque in other words it is not interpreted A BU will send back a copy of the fuTransactionId field in each of the I20 EVENT DATA BLOCK MESSAGE FRAMEs that make up the requested event The fset field is a fragment set identifier Fragment sets are a way to describe partial events The fset field is ignored by the BU in this version of the RU builder because this version does not support partial events DAQ CERN PH CMD 2008 Page 24 of 54 4 RU builder 120 interface RU Builder User Manual The I20_ EVENT DATA BLOCK MESSAGE FRAME C structure is as follows typedef struct I20 EVENT DATA BLOCK MESSAGE FRAME I20 PRIVATE MESSAGE FRAME PvtMessageFrame U32 eventNumber U32 nbBlocksInSuperFragment U32 blockNb U32 eventId U32 buResourceld U32 fuTransactionId U32 nbSuperFragmentsInEvent U32 superFragmentNb U32 padding I20 EVENT DATA BLOCK MESSAGE FRAME PI20 EVENT DATA BLOCK MESSAGE FRAME The FU should only read nbSuperFragmentsInEvent superFragmentNb nbBlocksInSuperFragment blockNb buResourcelId fuTransactionId An eve
45. r Specifies whether or not the application has found the RCMS state listener eventiIdFIFOElements The number of elements in event ID FIFO blockFIFOElements The number of elements in the block FIFO fullResourceFIFOElements The number of elements in the full resource FIFO discardFIFOElements The number of elements in the discard FIFO requestFIFOsElements The number of elements in the request FIFO Table 7 BU Debug web page Debug monitoring parameters DAQ CERN PH CMD 2008 Page 40 of 54 10 RU builder application web pages RU Builder User Manual 10 2 EVM web pages throughput bytes sec The trigger data throughput in bytes per second 1 average bytes The average size of a trigger s data rate events sec 1 The number of triggers per second rms bytes The RMS of the trigger data size Table 8 EVM Default web page Data flow through node parameters fragmentSetsUrl If useFragmentSet is set to true then this parameter must be set to the URL of the fragment sets file The EVM will use this file to determine which RUs are participating in event building useFragmentSet Indicates whether the EVM is using a fragments sets file or the value of rulnstances to determine which RUs are participating in event building If set to true the EVM will use the fragments sets file specified by fragmentSetsUrl else the EVM will use the contents of rulnstances fragmentSetld The EVM can use a fragment sets
46. ributed XDAQ application that is part of a larger system called the event builder EVB The CMS data acquisition group is presently developing the EVB as described in the TriDAS TDR 1 This document explains how to obtain build and configure version rubuilder_G_V0O1_06_00 of the RU builder 1 1 Document purpose and scope This document has two goals The first is to enable the reader to integrate the RU builder into a running system and the second is to help the reader understand how the RU builder works so that they may be able to diagnose any problems which may occur A running system is composed of the RU builder itself a trigger source one or more event data sources one or more data sinks and a run control system The RU builder cannot run without the components just listed This document describes how to obtain build and configure the RU builder This document does not describe the other components of a running system such as run control software or how to setup a trigger or event data source Although this document does describe some of the internal workings of the RU builder the reader is referred to the comments within the source code as the authoritative guide on the subject The code has been structured and commented so that it can be easily read and understood It is recommended to use doxygen to generate documentation from the code as compatible comment tags have been used If the reader is not familiar with doxygen
47. rom a BU A BU sends an event to a FU A FU tells a BU to discard an event Figure 12 is a sequence diagram describing the protocol between a BU anda FU FU BU Function code I20 BU ALLOCATE C structure I20 BU ALLOCATE MESSAGE FRAME Function code I20 FU TAKE C structure I20 EVENT DATA BLOCK MESSAGE FRAME Vy wa e Function code I20 BU DISCARD C structure I20 BU DISCARD MESSAGE FRAME Figure 12 BU FU interface sequence diagram A FU requests a BU to allocate it one or more events step 1 In response the BU asks the FU to take the requested event data as a set of event data blocks step 2 When a FU has finished processing one or more events it tells the BU to discard them step 3 The BU FU interface of this version of the RU builder does not support partial events Partial events may be supported in a future version DAQ CERN PH CMD 2008 Page 23 of 54 4 RU builder 120 interface RU Builder User Manual The I20 BU ALLOCATE MESSAGE FRAME C structure and its companion BU_ALLOCATE C structure are as follows typedef struct BU ALLOCATE U32 fuTransactionIid U32 fset BU_ALLOCATE PBU_ ALLOCATE typedef struct 120 BU ALLOCATE MESSAGE FRAME I20 PRIVATE MESSAGE FRAME PvtMessageFrame U32 n BU ALLOCATE allocate 1 I20 BU ALLOCATE MESSAGE FRAME PI20 BU ALLOCATE MESSAGE FRAME The FU must fill n allocate The n field specifies the number of events the FU is re
48. rough the BU in the last en SLG S deltaSumOfSquares The sum of the squares with respect to the sizes of the events in bytes that passed through the BU in the last en S LG S deltaSumOfSizes The sum of the event sizes in bytes that passed through the r BU inthe last Gea Ti ccccnceacceseceeceiiaeceaieacececaneccats stateName eee ee eeeeeeee Lhe current state of the application aaa lastEventNumberFromEVM The last event number received from the EVM lastEventNumberFromRUs The last event number received from the RUs lastEventNumberToFUs The last event number sent to the FUs nbEvtsBuilt The number of events built since the BU was last enabled nbEventsDropped ee The number of events dropped by the BU a IZ0_BU_ALLOCATE_Payload The payload in bytes transferred by 12O0_BU_ALLOCATE nn ee ee eee Reece Cm messages See I2O_BU_ALLOCATE_LogicalCount IZ20_BU_ALLOCATE_LogicalCount The number of events requested by the FUs FUs request Oooo eeeeeeeeeeeeeeee EVENES from the BUs using I2O_BU_ALLOCATE messages I20_BU_ALLOCATE_120Count The 12O message frame count for 12O_BU_ALLOCATE ee E messages See IZO_BU_ALLOCATE_LogicalCount I20_BU_DISCARD_Payload The payload in bytes transferred by 120_BU_DISCARD oe cee A E messages See 2O_BU_DISCARD_LogicalCount I20_BU_DISCARD_LogicalCount The number of events discarded by the FUs FUs tells the ee eee BUs to using I2O_BU_DISCARD messages I20_BU_DISCARD_120Count The 12O message frame count for 12O_BU
49. the EVM to allocate it an event step 5 Within such a request a BU will normally give back the ID of an event to be cleared For each cleared event ID the EVM sends a trigger credit to the TA and makes the ID a free event ID step 6 The EVM confirms the allocation of an event by sending the requesting BU the event ID and trigger data of the allocated event step 7 The triggerFIFO clearedEventIdFIFO and requestFIFO store incoming messages until they can be processed The pairFIFO keeps track of the event ID trigger data pairs that have yet to be sent to requesting BUs The freeEventIdFIFO stores the ids of free events for which trigger credits have been sent to the TA RU Readout EVM Step 4 Tri triggerFIFO Step 3 eae LA ss 0 gt HHH Credit Step 1 clearedEventIdFIFO freeEventIdFIFO SERRE ITT Allocate and or clear Step 6 tep 5 cecwlesicele T H am a fs BU Confirm tep 7 J Figure 5 EVM FIFOs DAG CERN PH CMD 2008 Page 14 of 54 0 The RU can be set into a fault tolerant mode in which it continues to RU Builder User Manual 3 3 RU FIFOs A RU is responsible for buffering super fragments until they are request by the BUs To understand the internal FIFOs of a RU it is necessary to know its dynamic behavior Figure 6 shows the internal FIFOs of a RU The EVM sends a RU an event ID trigger event number pair when it asks the RU to readout the corresponding event s data step 1 In parallel
50. threading model See chapter 6 RU builder threading models for more information ageMessages Specifies whether or not control messages should be aged i e whether or not a control message should be sent if it is too old as opposed to only when it is full msgAgeLimitDtMSec How old a message can be in milliseconds before it will be sent if ageMessages Is set to true monitoringSleepSec Number of seconds the monitoring thread should sleep between monitoring information updates dropEventData If the value is set to true then the BU will drop an event when it has finished building it I2O_RU_SEND_Packing The packing factor of I2O_RU_SEND messages IZO_EVM_ALLOCATE_CLEAR_Packing The packing factor of I2ZO_EVM_ALLOCATE_CLEAR messages monitoringFilename The name of the file to which monitoring information should be written rulnstances The instance numbers of the RUs that will participate in event building If this parameter is not set then it assumed that all RUs will participate in event building This parameter is ignored if useFragment sets is set to true oldMessageSenderSleepUSec The number of micro seconds between checks for old messages dumpTriggersToLogger Specifies whether or not triggers should be dumped to the logger This is a debugging aid Dumping triggers to the logger can have a serious impact on the performance of the BU Table 6 BU Debug web page Debug configuration parameters foundRcmsStateListene
51. to move to the TimedOutTolerating or TimedOutBackPressuring state measuredTimeOutUSec The microseconds component of the time period that passed and triggered the finite state machine of the application to move to the TimedOutTolerating or TimedOutBackPressuring state foundRcmsStateListener Specifies whether or not the application has found the RCMS state listener blockFIFOElements The number of elements in the block FIFO pairFIFOElements The number of elements in the pair FIFO requestFIFOsElements The number of elements in the request FIFO nbSuperFragmentsReady The number of super fragments that are ready to be consumed by the BUs lowestBulnstance The lowest BU instance number found in the XDAQ configuration highestBulnstance The highest BU instance number found in the XDAQ configuration bulnstances The instance numbers of all the BUs found in the configuration Table 17 RU Debug web page Debug monitoring parameters DAQ CERN PH CMD 2008 Page 49 of 54 11 How to install the RU builder RU Builder User Manual 11 How to install the RU builder Install the XDAQ coretools powerpack and regular worksuite packages using the instructions at the following URL http xdaqwiki cern ch index php Main_Page 12 Example configuration file The following example configuration file shows how to configure a 1x1 RU builder to run on a single XDAQ executive and service HTTP requests on port xosr Porr lt xc
52. ubuilderbu so lt xc Module gt lt xc Context gt lt xc Partition gt A Two import rules for the module ta 1 The module tags for librubuilderutils so and libxdaq2rc so must appear once within each context tag where there is a module tag for librubuilderbu so librubuilderevm so and or librubuilderru so 2 The module tags for librubuilderutils so and libxdaq2rc so must appear within the context tag before any of the module tags librubuilderbu so librubuilderevm so or librubuilderru so DAQ CERN PH CMD 2008 Page 50 of 54 13 RU builder self test RU Builder User Manual 13 RU builder self test This section explains how to perform the self test of the RU builder This test helps determine whether or not the RU builder has been successfully installed The self test consists of the rubuilder tester Application plus one EVM one RU and one BU all running on the same XDAQ executive The EVM is told to generate dummy triggers the RU is told to generate dummy super fragments and the BU is told to drop the events it builds The step by step instructions to run the self test are Step 1 Install the RU builder as described in chapter 11 Step 2 Modify the configuration file of chapter 12 so that osT porr refers to the port on which the XDAQ executive shall service HTTP requests Step 3 Setup the following shell environment variables export XDAQ ROOT opt xdaq export XDAQ DOCUMENT ROOT XDAQ ROOT htdocs export PATH P
53. ytes When generateDummyTriggers is set to true this parameter specifies the trigger source ID that is to be put into each dummy trigger The packing factor of I2O_TA_CREDIT messages The packing factor of I2O_RU_READOUT messages The instance numbers of the RUs that will participate in event building If this parameter is not set then it assumed that all RUs will participate in event building This parameter is ignored if useFragment sets is set to true The number of micro seconds between checks for old messages Specifies whether or not triggers should be dumped to the logger This is a debugging aid Dumping triggers to the logger can have a serious impact on the performance of the EVM Table 11 EVM Debug web page Debug configuration parameters foundRcmsStateListener eventIdGaugeName eventlIdGaugeValue dummyEventNumber nbCredits ToBeSent clearedEventldFIFOElements freeEventIdFIFOElements triggerFIFOElements pairFlFOElements requestFIFOElements DAQ Specifies whether or not the application has found the RCMS state listener The name of the gauge responsible for measuring the number of event ids resources in use The current value of the gauge responsible for measuring the number of event ids resources in use When generateDummyTriggers is set to true this parameter specifies the event number of the next dummy trigger The number of outstanding trigger credits to be sent to the TA The number of eleme

RU Builder User Manual

Contents

Download Pdf Manuals

Related Search

Related Contents