User manual
Contents
1. 52 Wi fn INSTITUTO DE PLASMAS ur E FUS O NUCLEAR T CNICO Acronyms Definition AC __ AC Coupled Advanced Configuration Environment AC ACE Analogue to Digital Converter Application Programming Interface ATCA Advanced Telecommunications Computing Architecture Advanced Technology Extended BAR Base Address Registers Compact Flash CF CompactFlash gt gt gt CODAS JET Control and Data Acquisition System CTTS Composite Timing and Triggering System Data Acquisition System DDR2 Double data rate two synchronous dynamic random access memory Db o Decibl Joint European Torus _ OP Operating System Peripheral Component Interconnect Express PCISIG Peripheral Component Interconnect Special Interest Group Pulse Height Analyser PICMG PCI Industrial Computer Manufacturers Group Phase locked loop PUR Pile Up Rejection 53 fi INSTITUTO SUPERIOR T CNICO y gh fn INSTITUTO DE PLASMAS E FUSAO NUCLEAR Reference Clock Real Time Applications Interface Surface Acoustic Wave Synchronous Dynamic Random Access Memory Small Form factor Pluggable transceiver System on chip Serial Peripheral Interface System ACE Spread Spectrum Clock Texas Instruments UpStream Transient Recorder and Processing VCXO Voltage controlled oscillator 54 A fi IT INSTITUTO DE PLASMAS INSTITUTO E FUSAO2. e Hardware trigger external trigger e Programmable delay after start acquisition occurrence Range 0 to 4294967295 2432 The delay time is calculated and written in us 48 fi INSTITUTO SUPERIOR T CNICO gh fn INSTITUTO DE PLASMAS E FUS O NUCLEAR y 4 3 Data Transfer Data is transferred to the host trough DMA transfers either by using the classic interrupt or by fast polling without interrupts Some parameters are the same for the two acquisition modes The size of each DMA in bytes range gt 4 to 4096 each DMA packet followed by an MSI interrupt The number of DMA buffer to be used a maximum of 16 buffers default value range gt 1 to 6553502216 gt Default value 16 Number of samples in bytes that host wants to receive read from memory range 4 to 4294967296 2432 Offline mode only The size of each acquisition to DDR2 in bytes gt 4 to 2G 2432 Offline mode only 4 4 Data Format Data format changes with the acquisition mode The data format is equivalent of a number of words depending on the memory filled length which format is presented at table 4 1 e Raw acquisition The word length is always of 16 bits The order is presented at table 4 1 e Burst acquisition The word length consists of a user defined number of samples PWIDTH for each detected event and related timestamp TS the order is presented at table 1 The word length changes w
3. COMPACT FLAS CP FTOVITHLNI OVILLO DRW 02 15 12 2007 RCE Figure 3 11 SYSACE CF controller Diagram Block 25 ne yi o a INSTITUTO DE PLASMAS INSTITUTO FUSAO NUCLEAR TECNICO y 7 3 3 2 Test JTAG TSTJTAG to configuration JTAG CFGJTAG setup When no CF is inserted the test JTAG TSTJTAG to configuration JTAG CFGJTAG setup can be used Using this setup the 2 FPGAs can be configured via JTAG from a JTAG compliant tool After downloading the FPGAs programs the system must be resetted 3 3 3 MPU setup SYSACE is connected to the FPGAs through the JTAG chain for configuration and through MPU port of SYSACE for allowing the FPGA of block 1 to control SYSACE and access the CF not implemented The ability to communicate with CF through MPU port allows the user to perform many operations such as being able to switch the programming or in system program new bitstreams or other files into the CF card 3 4 PCle Switch ExpressLane PEX 8516 device 7 offers PCI Express switching capability enabling users to add scalable high bandwidth non blocking interconnection to a wide variety of applications This device has a flexible port width configuration and is used as a fan out application The upstream US and downstream DS ports are all configured as x1 although the board design is compliant with x4 There is the possibility of US port connected to ATCA channels be chosen from two different ATCA chan
4. EPLOS 302HLN Compliant with RS485 NU standard interface j j 1920 GPIO SFP amp EM High speed serial protocol PCIe Cages over fibre can be used for Interface Block 1 several gigabit Interfaces up 3 tested to 2 5 Gbps NU j 2138 COPO SEB amp EM highspeed serial protocol Pele Cages over fibre can be used for Interface Block 2 several gigabit Interfaces up tested to 2 5 Gbps NU Mimi e a SOI SIRIA Iaia Ri SIRIA SIRIA IRR nni RR 3 6 Intelligent Platform Management Controller IPMC The module has a vertical receptacle 38 way adaptor J24 to connect an IPMC Figure 3 1 This IPMC is a microcontroller module that should reside in the ATCA module aiming to control and monitor the operations of its host This IPMC should be responsible for the communication between shelf manager and the several components in the shelf IPMC plays the role of relaying data to and from shelf manager 8 Shelf manager Is the command center of the shelf It assures a proper operation of boards and rest of the shelf It monitors the system s health retrieves inventory information and controls the performances of the Field Replaceable units as setting fan level 31 fn INSTITUTO DE PLASMAS gara E FUSAO NUCLEAR Up to now there is no IPMC on the acquisition module As soon as this feature 1s implemented the SYSACE failsafe mode will be enabled see section 3 3 3 7 LED Indicators 3 7 1 Front panel LED
5. e Software or e Hardware Attaching a trigger signal to o Front panel unipolar lemo EPL 00 250 NTN External LVTTL 3 3V trigger 5V tolerant Polarity any o Rear panel differential Lemo LEMO EPL OS 302 HLN Compliant to RS485 standard interface 5V 2 4 Time Stamping Specifications Time resolution depends on the frequency rate for example F 400 MHz gt T 2 5 ns Over days 2 x 2 5 ns of time span 2 5 Storage Capabilities e Gbytes of data memory distributed in two blocks of four channels 500 Mbytes per channel e IGB of CompactFlash card memory containing FPGA images every time the system 1s powered up the FPGA is programmed with these images 11 x a f INSTITUTO DE PLASMAS D E FUS O NUCLEAR fi INSTITUTO SUPERIOR TECNICO y 2 6 Processing Capabilities Two XC4VFX60 10FF1152C FPGAs for real time pulse processing e Pulse height analyzer PHA e Pile up rejection PUR 2 Interface Capabilities e ATCA based module e ATCA Fabric channel x1 PCI Express Although the board design is prepared to work with x4 PCIe links Compliant with PCIe rev1 1 3 e General Purpose Input Output GPIO links provided by ATCA Zone 2 connectors ATCA Fabric channels 3 to 13 x1 Aurora e Four GPIO 4 Small Form Pluggable SFP cages to allow gigabit links other than those provided by ATCA Zone 2 connectors 3 System Description The data acquisit
6. 00100 0000001000 00000 4100000080000 ALL 1100005915400 00D0D01409 aan aan Pedala EE as Figure 3 15 Location of the debug LED indicators at the module layout 32 fi INSTITUTO SUPERIOR TECNICO gh fn INSTITUTO DE PLASMAS E FUSAO NUCLEAR y 3 8 Jumpers settings The jumper location is showed at Figure 3 3 3 8 1 SW 1 SYSACE failsafe MODE SWI on il SYSACE ERRLED NM SYSACE RESET FPGA PROGRAM SYSACE ERRLED Figure 3 16 SW1 SYSACE failsafe mode jumper view Default settings In order to System ACE failsafe mode detects a failed configuration attempt the SW 1 must be on Without IPMC this feature is disabled Not usedf 3 8 2 SW 2 SYSACE configuration switch SW2 CFGMODE X CFGADDR LX GND CFGADDR1 LX CFGADDRO LIX 1 O Figure 3 17 SW2 SYSACE configuration switch view Default settings 36 gh fn INSTITUTO DE PLASMAS E FUSAO NUCLEAR fi INSTITUTO SUPERIOR TECNICO Table 3 10 SW2 SYSACE configuration SW2 Status Note refer to Figure 3 17 ON 0 GND Inhibits the SYSACE from configuring after reset or power up and can only be CFGMODE commanded by the MPU control e eee eee OFF 1 Allows SYSACE to configure mode behavior after reset or power up default o a ee CEE emman lola CFGADDR 2 0 0x000 This allows up to 8 different ace files at the CF default 3 8 3 SW5 PEX C
7. ne yi o a INSTITUTO DE PLASMAS INSTITUTO FUSAO NUCLEAR TECNICO y 7 setting the beginning of each pulse event for time reconstruction thus allowing time resolved spectroscopy After storing the triggered event on a temporary buffer the system waits for another trigger Meanwhile if pile up occurs then it has to be resolved off line The system allows the selection of two trigger modes by level and by edge In the former a voltage level comparison is performed between successive samples and a user defined threshold value while on the latter a linear regression is done to the rising edge of the event If the difference between consecutive samples is higher than a slope value an event has occurred This type of triggering is more accurate in respect to the level one since it is more insensitive to time jitter For this acquisition mode the read bandwidth doesn t depend on the sampling rate but instead on the event counting rate TRP 250 amp TRP 400 e Segment width PWIDTH 512 samples per event e 512 x 2 bytes 2 us 1 28 us 1024 bytes 1KB per channel e 4GB IkB 4 Mevents s If the counting rate is of 1 Mevent s the memory will be filled up within 4s e If the PWIDTH 256 gt the filling time will increase to s e x PCIe link gt 2 5 Gb s for 4 channels for one direction the mdlue can attain 200 MB s If in the future the module will work with x4 PCIe link this value will increase to 800 MB s 3 11 3 P
8. DE PLASMAS E FUSAO NUCLEAR y Looking at Figure 3 8 at FWHM there are 4 bins this is equivalent of an idle channel of 1 8 LSB This same study was performed for all module channels Figure 3 9 presents TRP 400 channel 1 In order to understand if the signal conditioning also introduces noise to the input channel a similar test was performed adding the differential amplifier to the test and connected the input to ground this study is showed at Figure 3 10 Inputs Shorted to Vcm Ea 8252 Y 21 3185 Y 21 3948 E a Ke 6264 Y 12 6202 entage O Per FWHM 4 Bin gt 1 8 LSB 8256 8257 8258 8259 8260 8261 8262 8263 8264 8265 8266 8267 8268 code number Figure 3 9 TRP 400 module Noise histogram of the module channel with inputs Shorted to Vcy Idle channel noise is 1 8 LSB 22 fi e IT INSTITUTO DE PLASMAS INSTITUTO E FUSAO NUCLEAR THENICO f 7 No input at differential amplifier 14 E M o Ia oO Percentage 2 FWHM 7 Bin gt 2 8 LSB 8200 8205 8210 8215 8220 8225 8230 code number Figure 3 10 TRP 400 module Noise histogram of the module channel 1 with no signal applied to the input channel before the differential amplifier The idle channel noise in FWHM LSB as well as the expected resolution of the module input chanels are presented at Table 3 2 Table 3 2 TRP 400 module Idle channel noise in FWHM LSB and expected resolution of ADS5474 ADC
9. NUCLEAR T CNICO y ANNEX A User manual for setting an acquisition on a Linux terminal 55 INSTITUTO SUPERIOR TECNICO fn INSTITUTO DE PLASMAS E FUSAO NUCLEAR y ANNEX B Digitizer Module Design Guidelines 58 INSTITUTO SUPERIOR TECNICO dr fn INSTITUTO DE PLASMAS E FUSAO NUCLEAR ANNEX C Schematic of the digitizer module 76 Gi f n INSTITUTO DE PLASMAS E FUS O NUCLEAR
10. TEMPORAL TimeStamp 44 bits Resolu o at 2 5 ns XC4FX60 1152 REGISTERS ATCA Virtex 4 Parameters Sincronism MLVD distribution ATCA Master Slave Switch PNO E IRSA LR E AA RE E A CS Master Slave E la Configuration PLL SPI Processed Mode Based on the Trapezoidal Filter 43 ds yi o n INSTITUTO DE PLASMAS INSTITUTO FUSAO NUCLEAR T CNICO y Figure 3 22 Block Diagram of FPGA architecture 3 11 1 Raw Mode At raw data mode all acquired data is stored in the DDR2 memory The module data transfer bandwidth 1s different for each version TRP250 and TRP 400 TRP 250 e One sample gt 13 bit 2bytes e One channel gt 2 bytes 250 MHz 500 MB s e digitizer module gt 8 channels 4 GB s Local memory The module has 4 GB of local memory At free running mode the module memory will take Is to fill it up Real time transfer 1t isn t supported to this acquisition mode TRP 400 e One sample gt 14 bit 2bytes e One channel gt 2 bytes 400 MHz 800 MB s e digitizer module gt 8 channels 16 GB s Local memory 0 5 s of raw acquisiton Real time transfer 1t isn t supported to this acquisition mode 3 11 2 Burst Mode During burst mode for each event detection any new incoming event is neglected that is the system does not present dead time Every time an event occurs a user defined number of samples PWIDTH are stored with an associated timestamp 44
11. choosing slot n 12 Annex A From the point of view of the Operating System OS each address points to a different device using the same Linux Device driver section 3 1 If more than one module is to be used for each module two different devices must be created at the OS All system endpoints can be configured and acquired simultaneously Using the Linux terminal all the endpoints will have a delay time between each configuration starting acquisition 47 ds yi o n INSTITUTO DE PLASMAS INSTITUTO FUSAO NUCLEAR T CNICO y All the configurations of the module are presented at the annex B 4 1 Operating mode The module supports two distinct operation modes which will be interfaced by either Linux or RTAI Real time mode not implemented Is based on fast polling without interruptions and only includes processing data otherwise it wouldn t be possible to transfer raw data in real time through PCIe without losing data If the 4 channels were acquiring simultaneously at 250 MHz 13 bits 16 bits it would be necessary a transfer rate of 2 GB s however as the module is working with only one PCIe lane only 800MB s are available Offline mode Is used to retrieve the acquired data from DDR2 memory using interruption after DMA transfer Data is ready to be retrieved of DDR2 memory when the 2 GBytes are written or after a user request 4 2 Start Acquisition The user can choose between e Software Trigger
12. gt gt gt 5V DCDC Power 48V gt gt gt 12V Distribution ATX ATCA SLOT 1 x16 PCle D Interface MS z D 3 3 3 o le mb o da 5 me e e e Dip DIL 9 2J o A DIK 4 ul Endpoint LC FPGA Tali A 4 ADCs A ane Root Complex DCDC Power 48V gt gt gt 12V 7 Distribution DAL a PCIE Guriirh N Switch n Pi lut y P PLL et Controller Module 2 p 2 T gt 2 A k ATCA SLOT 4 ATCA SLOT 5 AT A SLOT 6 ATCA SLOT 7 EEE ENO Figure 3 1 Block diagram of an ATCA System up to 5 digitizer modules 3 1 Digitizer Module Architecture This digitizer module has a form factor large enough to accommodate eight acquisition channels per module The module has eight analog input channels divided in two symmetric blocks Figure 3 1 At these speeds 400 and 250 MHZ only fast programmable devices especially Field Program Gate Arrays FPGAs can be used for data processing and transfer The module contains two FPGAs each controlling a memory bank of up to 2 GB and four acquisition channels and providing a gigabit communications interface thereby reducing circuit complexity and cost by acting as a system on chip SoC device The memory will be implemented with double data rate DDR2 synchronous dynamic random access memory SDRAM providing up to 4 GB per module Figure 3 2 represents a simplified block diagram of architecture
13. indicators Front panel LED indicators are presented at Table 3 7 and can be seen at Figure 3 13 These LEDs are divided into 3 different information groups 1 CF being inserted and the 2 FPGAs being properly programmed 11 Indicates how the 3 PCIe links and respective lanes are working properly and 11 ATCA mandatory LEDs that informs if the module is properly inserted and detected and provide basic feedback about failures and out of service states Table 3 7 Front panel LED indicators LED Label Note Dl CF SYSACE error and status LED refer to Table 3 3 BAD sii PCIe LINK sila i Mv T ci Link refer to Table 3 8 n gt 2 blue LED HP ATCA specification Hot Swap mandatory om 3 Amber OOS ATER specification CULO Service mandatory 3 7 1 1 PCIe LINK LEDs The D4 D5 D6 and D7 are composed by 4 LEDs each The configuration is presented at Figure 3 14 looking to the front panel 32 N fi INSTITUTO SUPERIOR T CNICO ipfn UP LEDs D 3 Left Up LED TO D 1 Left Down LED e DOWN LEDs Figure 3 14 PCle Link LED INSTITUTO DE PLASMAS E FUS O NUCLEAR D 4 Right Up LED D 2 Right Down LED Looking at the front panel the LED order is from the left to the right or from up to down if the module is in vertical position D5 D6 D7 D4 Figure 3 13 Table 3 8 PCIE Link LEDs The Link features are default Hardware wired refer to Tab
14. of the digitizer module and Figure 3 3 the general view of the module 14 m f n INSTITUTO DE PLASMAS E FUS O NUCLEAR Front Panel Transient Recorder amp Processor Rear Panel IPMC CompactFLASH REA System ACE Aurora x1 Multi Gigabit EN Seton ACE JTAG ens Digital Link Aurora x1 Aurora x1 e PADRE ii l Input protection amp ADC 14 bits ki Passive anti aliasing filter 400 MHz ee CTTS Clock Input protection amp ADC 14 bits Passive anti aliasing filter 400 MHz Discretes OPAMP ADS5474 Input protection amp ADC 14 bits Jitter Passive anti aliasing filter 400 MHz TBLOCK 417 clock Attenuator Discretes OPAMP ADS5474 A SINGLE ENDED Analog input 100 MHz PCle PCle RST reference clock PCle Clock Aurora x1 Input protection amp ADC 14 bits Aurora xi Passive anti aliasing filter 400 MHz Discretes OPAMP ADS5474 SINGLE ENDED Analog input Aurora x1 Logic analyser Clock Synthesis Socket i amp Distribution Aurora x1 ES Aurora x1 Aurora x1 DDR2 DRAM 2 GBytes Blue LED SODIMM LED1 SYSTEM ACE J23 ATCA POWER DOS ATCA LDOs LDOs re LDOs VOC ay 1 5V PEX e sv en 5V ADCs i IPMB POLA POLA POLA POLA POLA power contro
15. 000000000000000000000000000000000000000000000000000000 17 32 1 Input Channel Performance 18 LL daa 19 AA A o UC E E eee 21 3 3 system ACE CE COntronler 444545 5 5454145454045c05540 4oo 48e 00440003050 o 80006e048 24 3 3 1 CF to configuration JTAG CFGJTAG setup kek k k k k 25 3 3 2 Test JTAG TSTJTAG to configuration JTAG CFGJTAG setup 26 DD Mpa 26 E TS SIC I 26 35 External COMNCCUOS sarria 29 ot Pila 29 SE PA Leni 30 3 6 Intelligent Platform Management Controller IPMC s sssss 31 A DA ANAN MM sas asa aiar 32 de ed Front pane LED telle a LES emenike branda k a kak n en dl w G lme mini ape 32 wT B PEI CINE LED aaa a rrar pPp 7 EE 32 Sele Debug DEP in le ili Ss pamela l eld i 33 3 0 SUMIPOCES SCUUIIOS ninia ninia 36 x a INSTITUTO DE PLASMAS a E FUS O NUCLEAR INSTITUTO T CNICO p 3 8 1 SW 1 SYSACE failsafe MODE 36 3 8 2 SW2 SYSACE configuration switceh k k k 36 3 8 3 SW5 PEX Configuration switch cd 37 ANN l b N NN NN A MM 38 3 10 Clock Generation umana 38 3 10 1 CORNO AO PP n a GA A en lima Dam non ga dinle oi 39 ZLI gt e e i ER PE YERE br 39 3 10 1 2 Global Synchronization Synthesizer 39 2 10 1 3 PCleCioceSynlhesizer A A cidade 41 Oli CO DES Clock A o 41 3 10 2 A
16. 1 YES Mem OxFFFFFF88 0 128 NO BAR2 NO Be BAR3 NO o BAR4 NO o BARS NO While BARO is enabled and has 1Mbyte range it will NOT BE DECODED by V1 0 of Firmware 46 ne yi o a INSTITUTO DE PLASMAS INSTITUTO FUSAO NUCLEAR TECNICO y 7 There are a few important issues that developer must be aware for current firmware version a BARO although is defined it will not respond and no access to this zone should be made b No Long Addressing 64 bit is supported IMPORTANT c Module supports only MSI Message Signalling Interface d Data Retrieval from module local memory must be performed using direct memory access from module to Host and the use of MSI to manage the transfer e Access to outside specified register and memory zones may lead to unspecified behaviour f The accesses to the module by the host can be divided into three categories 1 Commands 2 Configuration and 3 Data Retrieval The behavior of digitizer module as a sequence of the interaction between the board firmware and the host driver is presented at Annex B 4 Digitizer Module Operation The user s first action is to configure the right endpoint module s block section 3 Where block 1 first PCIe endpoint section 3 4 1s configured by addressing the logical slot number of the crate correspondent to the place where it is inserted slot n and block 2 second PCle endpoint 1s configured by
17. 3 18 SW5 PEX Configuration switch view Default status US port is the ATCACHI PEXishard wredprogrammed 37 Figure 3 19 Global clock synthesis and distribution of module references 40 Figure 3 20 Synthesis and distribution of PCIe related module clocks 41 Figure 3 21 TRP 400 acquisition and DDR2 clocks source example 42 Figure 3 22 Block Diagram of FPGA architecture kek kek 44 n yi o INSTITUTO DE PLASMAS INSTITUTO E FUSAO NUCLEAR T CNICO f 7 Digitizer Technical Specifications 1 Introduction The development of advanced acquisition Systems in ATCA Advanced Telecommunications Computing Architecture PICMG 2008 1 2 has provide the means for building a new high rate data acquisition system Current Developer User Manual presents the system architecture and details about acquisition capabilities The purpose of this manual is to describe the functionality and operating modes of an 8 channel 14 13 bit 400 250 MHz 4 GB DDR2 Memory ATCA based Transient Recorder and Processing TRP module digitizer part of the data acquisition system Two digitizer modules will be described 1 14 bit 400 MHZ TRP 400 and 11 13 bit 150 MHZ TRP 250 2 System Specifications 2 1 Analog Input characteristics e Fight analogue channels at o 400 MSPS using AD5474 from TI digitizer TRP
18. 400 or o 250 MSPS using AD5444 from TI digitizer TRP 250 e Resolution o 14 bit digitizer TRP 400 or o or 13 bit digitizer TRP 250 ne yi o a INSTITUTO DE PLASMAS INSTITUTO E FUSAO NUCLEAR TECNICO 7 7 e AC coupled Input Range 0 V gt 1 1 V 50 Q e DC coupled Input Range 1 1V 50 Q e Lemo EPL 00 250 NTN 2 2 Acquisition Configuration e Two Blocks block 1 block 2 of 4 channels each Each block can be acquired simultaneously but data is retrieved from each block separately With an acquisition rate of 400 MHz only two channels at each block can be acquired simultaneously without loosing data 4 channels can be acquired simultaneously only at 200 MHZ e Programmable acquisition rate VCXO N refer to e Table 2 1 Table 2 1 Programmable acquisition rate for TRP 400 and TRP 250 digitizers DIGITIZER i VCXO MHz N Acquisition Rate MHz TRP 400 800 2 400 4 200 5 160 8 100 10 80 nn l le_ 290 __ TRP 250 1000 4 250 5 200 8 125 10 100 16 62 5 e Free running ADCS acquisition mode acquisition up to memory filling o Raw data o Segmented data burst mode or o Processed data 10 L a fn INSTITUTO DE PLASMAS D E FUSAO NUCLEAR fi INSTITUTO SUPERIOR TECNICO e XC4FX60 FPGA from Xilinx for enhanced and upgradeable acquisition processing and or control 2 3 Trigger Source Trigger source can be given by
19. AA A SE REY db dani 42 SIT FPGA ACI CCU E animacion atacaba 42 3 11 1 A MO id 44 3 11 2 Burst MODO andina 44 3 11 3 Processed MOS ainia 45 3 11 4 Host Data NA yasa d saya yaya yan a nda ER goi 46 4 Digitizer Module Operationy cccccccsssssssssssssssssssssssssssssssssssssssssssssssssssssssseees 47 4 1 o a toreni tepon sbo asere niise s eisato rerit ro s sbo riSai 48 42 Start MADA 48 Es Daa Tran sTo lari 49 dd Dali fol 49 5 Special Handling and Operation Consider ations sssssssssssssssssssssseseeess 50 6 TRP 400 Module Insertion at the ATCA Ct ccccccccsssssssscccccccccccesseees 51 DO OVO Yl A Der 52 PRCT OVI rrr rere RS RR ER E 53 PING Gra ES DSR e ARA EN RE RODE 55 User manual for setting an acquisition on a Linux terminal 55 AB ai ND AR a E RR 58 Digitizer Module Design Guidelines ssr0s000000009050000099090000000000000000050058600099990000000009 58 ANNETC in 76 Schematic of the digitizer module ss s ssssscessssssse 76 ds yi o n INSTITUTO DE PLASMAS INSTITUTO FUSAO NUCLEAR T CNICO y Table List Table 2 1 Programmable acquisition rate for TRP 400 and TRP 250 digitizers 10 Table 3 1 TRP 250 module Idle channel noise in FWHM LSB and expected PESOMI LIO HOF ADS AAA ME O A a 21 Table 3 2 TRP 400 module Idle channel noise in FWHM LSB and expected ESOO OF Ful 5947 GN Y A pp rrr r
20. CE failsafe mode see section 3 8 1 In this mode 1f the SYSACE Controller detects a failed configuration attempt it automatically reboots back to a redefined configuration image This only occurs if the IPMC is onboard otherwise this feature 1s disabled 24 j a f INSTITUTO DE PLASMAS D E FUS O NUCLEAR fi INSTITUTO SUPERIOR TECNICO y 3 3 1 CF to configuration JTAG CFGJTAG setup The SYSACE CF controller is the primary mean of configuring the two FPGAs XC4VFX 11FF1152C present on the digitizer module through the JTAG Joint Test Action Group interface figure 3 5 During configuration the SYSACE has full control of the JTAG signals Virtex 4 FPGA 1s configured by loading application specific configuration data the bitstream file into internal memory Because Xilinx FPGA internal memory is volatile it must be configured each time it 1s powered up The bitstream is loaded through special configuration pin and configuration mode In the digitizer the bitstream is loaded through the JTAG interface JTAG Boundary scan configuration mode No system reset is needed SYSTEM ACE DIAGRAM BLOCK MPU INTERFACE PARALLEL CABLE IV SYSTEM ACE CONTROLLER PC4 PC4 TCK gt PC4 TMS a PC4 TDI _ j E CFG TCK CFG TMS CFG TDI CFG TDO BUFFER 3 3V 2 5V AOVAJALNI IVILOHOI H WI H C E D Q H Z H E YU Ay D Q Er SFTOVATHINI DVIL ISHL HOVAITHINI DVIL NOTLYANILANO
21. Digitizer Module SR TR ATCA User Manual V1 1 Rita Costa Pereira n yi o a INSTITUTO DE PLASMAS INSTITUTO E FUSAO NUCLEAR TECNICO y Document Configuration Company IPFN IST instituto de Plasmas e Fus o Nuclear Instituto Superior T cnico Av Rovisco Pais 1 1049 001 Lisbon Portugal R C Pereira ritacp lei fis uc pt VERSION 1 1 O fn INSTITUTO DE PLASMAS E FUS O NUCLEAR Historic 2009 09 26 2012 1 17 ne yi o a INSTITUTO DE PLASMAS INSTITUTO FUSAO NUCLEAR TECNICO y 7 Contents Document Configuration rica ise 2 E ARR 3 rr lm E RR 4 UDK 6 Foresti 7 Digitizer Technical Specifications rrrrrrrrrrrrrrrrererirrereerioeericeericicrerio ie cioe 9 FE MIU 9 2 SIRO ani 9 2 1 Analog Input characteristics 41 4j ss x s ssssssssssssseseeeresesosssseeseseseee 9 2 2 Acquisition CONHSUration ina 10 2 5 L Y O GUT solleva 11 2 4 Time Stamping Specificat OIS8 s ssscssseeeeseeeeeeeeeeeeee 11 PAS AGS AA NCS SARAN NNN A N NN a ya a 11 2 6 PROCESSING Capabilif eS s u z z cc cccccscccssceeseeeeeeeeese 12 2 7 Interface Capabilities aia ais 12 3 System Description AAA RR RP EN ARR A SPD EE YA 12 3 1 Digitizer Module Architecture 000000000000000000000000000000000000000000000000000000000000 14 32 Analog to Digital Front End 000000000
22. I A E 19 Figure 3 6 TRP 250 module Noise histogram of the module channel 1 with inputs Shorted to Vem Idle channel noise is 0 4 LSB eee keke kek keke kk 20 Figure 3 7 TRP 250 module Noise histogram of the module channel 1 with no signal applied to the input channel before the differential amplifier 20 Figure 3 8 Noise histogram with inputs shorted from ADS5474 datasheet Idle channel Ae SI E o EE A E EEE xxx pg 21 Figure 3 9 TRP 400 module Noise histogram of the module channel 1 with inputs shorted to Vem Idle channel noise is 1 8 LSB rrenan 22 Figure 3 10 TRP 400 module Noise histogram of the module channel 1 with no signal applied to the input channel beforethediffteren alampliier 23 Figure 3 11 SYSACECFcontrollerDiagramBlock 23 Figure 3 12 PCleswitchingdiagramblock 2 Pieire o Palle DE EE E E AR E EMR N BE rrr REK E A AN a_m 30 fi gh fn INSTITUTO DE PLASMAS E FUS O NUCLEAR y T CNICO tico 1d POE LAO CED ri H r 33 Figure 3 15 Location of the debug LED indicators at the module layout 35 Figure 3 16 SW1 SYSACE failsafe mode jumper view Default settings 36 Figure 3 17 SW2 SYSACE configuration switch view Default settings 36 Figure
23. Idle channel noise FWHM LSB Resolution ADS5474 Figure 3 5 4 1 8 x 12 2 bit CH 1 ADS5474 Figure 3 6 4 1 8 i amp 12 2 bit CH 1 Diff Amplifier Figure 3 7 7 2 8 11 2 bit 23 ne yi o a INSTITUTO DE PLASMAS INSTITUTO E FUSAO NUCLEAR TECNICO 7 7 3 3 System ACE CF Controller The Xilinx System ACE SYSACE CompactFlash CF configuration controller allows a Type I CF card inserted on the CF socket Pl present on the front panel Figure 3 3 and Figure 3 13 to program the FPGA through the JTAG port The System ACE controller supports up to eight configuration images on a single CF card The configuration address switch allows the user to choose which of the eight configuration Images to use see section 3 8 2 SYSACE error and status double LED indicates the operational state of the SYSACE controller refer to Table 3 3 Every time a CF card is inserted into the SYSACE socket a configuration operation is initiated Pressing the SYSACE reset button re programs the FPGAs because of the PCle core everytime the FPGA is re programmed the system must also restarted Table 3 3 System Ace Error and Status LED LED LED STATUS System ACE Status Blinking Red ERROR No Compact Flash card 1s present Solid Red ERROR Error condition during configuration Blinking Green OK Configuration operation is ongoing Solid Green OK Successful FPGA program download The board also features a SYSA
24. S idle channel noise This figure of merit act as an ease way to understand the resolution of the described channel input architecture The ADC output should be zero when at its input no signal is applied The idle channel noise quantifies this leakage from zero 18 fi INSTITUTO SUPERIOR T CNICO Gr fn INSTITUTO DE PLASMAS E FUSAO NUCLEAR 3 2 1 1 TRP 250 Noise Histogram With Inputs Shorted Percentage 4109 4110 4111 4112 4113 4114 4115 4116 Code Number Figure 3 5 Noise histogram with inputs shorted from ADS5444 datasheet Idle channel noise is 0 4 LSB Looking at Figure 3 5 at Full Width Half Maximum FWHM there is 1 bin that it is equivalent of an idle channel of 0 4 LSB This same study was performed for all module channels Figure 3 6 presents TRP 250 channel 1 In order to understand if the signal conditioning also introduces noise to the input channel a similar test was performed adding the differential amplifier to the test and connected the input to ground this study is showed at Figure 3 7 19 NA INSTITUTO SUPERIOR T CNICO fn INSTITUTO DE PLASMAS E FUSAO NUCLEAR y Inputs Shorted to Ven n O la N la O i in J O Percentage hj q hj O FWHM 1 Bin gt 0 4 LSB _R mM O mM 4110 4111 4112 4113 4114 4115 4116 code number Figure 3 6 TRP 250 module Noise histogram of the module channel 1 with inputs shor
25. SUPERIOR TECNICO Figure 3 13 Front Panel All the front panel connectors are presented at Table 3 5 Figure 3 13 Table 3 5 Front panel connectors Con Label Mechanical Electrical specifications Polarity Specifications EPL 00250 NTN 0 1 1V 50 Q AC J9 CLOCK EPL 00 250 NTN LVTTL 3 3 V supports 5V Any From 1 to 125MHz mom peppe J10 START EPL 00 250 NIN LVTTL 3 3 V supports SV i Any em om zamma mma SA mm mm mm mm mm une zanna mma mm mm mm mma umman zamma mma amma mem mm mm mm mm mm J11 SYNC EPL 00 250 NTN LVTTL 3 3 V supports 5V Not used NU HT Saas ees eg Flash 3 5 2 Rear Panel Control Trigger and Timing System compliant to JET CTTS is composed by two signals CTTS Clock and CTTS trigger Table 3 6 The small form factor SFP host connector and EMI cages can be used to implement a high speed serial protocol over fibre These connectors support data rates up to 2 5 Gbps and such as PCle AURORA Infiniband Gigabit Ethernet or Fibre Channel NU These connectors are for General Purpose I O GPIO use Table 3 6 30 fi INSTITUTO SUPERIOR TECNICO Table 3 6 Rear panel connectors l pfn INSTITUTO DE PLASMAS E FUS O NUCLEAR Con Label Mechanical Electrical specifications Notes Specifications J30 CTTS LEMO 05 type i Compliant with RS485 NU Clock EPL 0S 302 HLN standard interface 55 TRG
26. alization OK BLL Block TT Red ERROR not initialized Initialization Status mi mm ia i m i E i iI I I m m mm DSI DS3 Dsi DS5 DS11 DS10 DS9 DS12 Not Used DS13 34 INSTITUTO DE PLASMAS E FUS O NUCLEAR INSTITUTO SUPERIOR T CNICO 0040000 llhbblbbblilbhhhlkbkllilkkklkllllkkli li nn damat u aaa 0004200000000 0000000000000 0000 c AZAAZAIAZZIZIA ZIZAZAZIZA AIZIZAZIA LZI e a e t eecouuetneeeeee 11555111515551551155541555155443 026646464400646864646464448880646464488080806444868 n eee naa a AMA cano BABA AMA an Eee gz2cajli i AAA ee kbb kkk kkk diil AAA ALLI e e e d a tee ee et a a sape aeapoagagcano paga ana pa COVA URINA FITERRPRAFIRRRRRA TRE Ed do o do dd do a a o Do do do dl a o do Do o ds do o Ds ds 0004440000040 4000000400000 aan pagaia Cig 946886664646640688666666 Block reaeeeeeeeaeaaeaeaeeeaaad AAA AAA ALA AAA LA a0ocoeopposgaoosoeoeppsasso 8000000100000 0010000 uu ee o e e e e AAA e e o o a e a s n uas een ae tea 1400000154000 AMADA 1000000140000 D0507400 0000001500000 ARE SSE SE SOU 40000000400000004000 40001000400000004000
27. at annec D 51 ne yi o a INSTITUTO DE PLASMAS INSTITUTO E FUSAO NUCLEAR TECNICO 7 7 Bibliography 1 ADVANCEDTCA PICMG amp 3 0 REVISION 2 0 ADVANCEDTCA BASE SPECIFICATION MARCH 18 2005 2 PCISIG 2008 SEPTEMBER 29 PERIPHERAL COMPONENT INTERCONNECT SPECIAL INTERES2 GROUP RETRIEVED FROM PERIPHERAL COMPONENT INTERCONNECT SPECIAL INTEREST GROUP http www pcisig com 3 ADVANCEDTCA PCI EXPRESS ADVANCED SWITCHING FOR ADVANCEDTCA SYSTEMS MAY 21 2003 4 A J N BATISTA J SOUSA AND C A F VARANDAS ATCA DIGITAL CONTROLLER HARDWARE FOR VERTICAL STABILIZATION OF PLASMAS IN TOKAMAKS REVIEW OF SCIENTIF INSTRUMENTS VOL 77 NO 10 OCT 2006 5 R C PEREIRA J SOUSA A M FERNANDES F PATRICIO B CARVALHO A NETO C A F VARANDAS G GORINI M TARDOCCHI D GIN A SHEVELEV ATCA DATA ACQUISITION SYSTEM FOR GAMMA RAY SPECTROMETRY FUSION ENGINEERING AND DESIGN VL 83 IS 2 3 PG 341 345 2008 6 A NETO J SOUSA B CARVALHO H FERNANDES R C PEREIRA A M FERNANDES VARANDAS G GORINI M TARDOCCHI D GIN A SHEVELEV AND K KNEUPNER THE CONTROL AND DATA ACQUISITION SOFTWARE FOR THE GAMMA RAY SPECTROSCOPY ATCA SUB SYSTEMS OF THE JET EP2 ENHANCEMENTS FUSION ENGINEERING AND DESIGN VL 83 IS 2 3 PG 346 349 2008 7 www plxtech com 8 MARK OVERGAARD REMOTE RELIABLE FIRMWARE UPGRADE ON PICMG BOARD MANAGEMENT CONTROLLERS COMPACTPCI AND ADVANCEDTCA SYSTEMS MAY 2005
28. erface RTAI which supplies a layer of VxWorks compatibility This API can transfer data to from the on board double data rate DDR2 synchronous dynamic random access memory SDRAM or stream out processed data through the x1 PCle link the design is able of x4 PCle link as well as control all the board functionalities by using a set of specially developed commands and communication Structure The digitizer module is divided into 2 blocks block 1 block 2 of 4 channels each The module represents two PCIe endpoints section 3 4 Each endpoint represents one block of the module Where endpoint 1 is in charge of channel 1 2 3 4 of the module and endpoint 2 is in charge of channel 5 6 7 and 8 of the module section 3 1 Figure 3 1 and Figure 3 3 PCle implements a dual simplex link which implies that data is transmitted an d received simultaneously on a transmit and receive lane The aggregate bandwidth assumes simultaneously traffic in both directions To obtain the aggregate bandwidth numbers multiply 2 5Gbits s by 2 for each direction then multiply by number of lanes and finally divide by 10 bits per Byte to account for the 8 to 10 bit encoding For 1 lane 0 5GBytes s 2 lanes 1GBytes s and 4 lane 2 GB s 13 uan gi f n INSTITUTO DE PLASMAS E FUS O NUCLEAR ATCA MLVDS Digitizer Module ATCA SLOT 3 Initialization Sys RST amp Clock PCle 100 MHz SSC Atenuador 3 E B DCDC de Jitter FPGA A 48V
29. howed at Table 2 1 are allowed by programming properly the EEPROM The hardware wired configuration is the first option of the Table 3 4 27 ne yi o a INSTITUTO DE PLASMAS INSTITUTO E FUSAO NUCLEAR TECNICO y Table 3 4 Possible configurations for the digitizer PCle switch Defenition of the DS and US ports N ofPCle US Port DSPort Notes Ports This configuration allows up to 4 x4 x4 x4 x4 3 digitizer modules default hardware wired 3 x8 x4 x4 5 digitizer modules EEPROM 4 x8 x4 x2 x2 3 digitizer modules EEPROM 4 x8 x2 x4 x2 3 digitizer modules EEPROM 4 x8 x2 x2 x4 3 digitizer modules EEPROM gt This configuration plus the controller s PEX configuration de select one of the PCle switches of the controller will allow up to 3 or 5 digitizer modules If the Controller will remain with the 3 PCle switches then only 2 digitizer modules will be possible 28 q Df a INSTITUTO DE PLASMAS A E FUS O NUCLEAR INSTITUTO SUPERIOR T CNICO 3 5 External Connectors 3 5 1 Front Panel IPFN TRP 400 err G o M pP A G T L A S H Gi CHB B E G2 o K 1 CHC A N cHD N L O G CH A Q5 n P U CHB T A ue za E Q7 6 Ca K CHD 2 li D5 D6 TE Ea wi D2 li E D4 E vs CLOCK START 880 SYNC 29 x a f INSTITUTO DE PLASMAS D E FUS O NUCLEAR fi INSTITUTO
30. ion system DAQ system is based on the ATCA PICMG 3 0 standar The ATCA base specifications define a board and architecture sharing a common backplane with interconnections based on a full mesh of serial gigabit communication links Each slot is interconnected to all others through x1 x2 and x4 links with a maximum throughput capacity of 2 GByte s Only x1 links are implemented Calculation of this bandwidth number The transmission reception rat is 2 5 Gbits s per lane per direction To support a greater degree of robustness during data transmission and reception each byte of data transmitted is converted into a 10 bit code via an 8b 10b encoder in the transmitter device In other words for every Byte of data to be transmitted 10 bits of encoded data are actually transmitted The result 1s 25 additional overhead to transmit a byte data 25 loss in transmission performance 12 L a fn INSTITUTO DE PLASMAS D E FUS O NUCLEAR fi INSTITUTO SUPERIOR T CNICO The ATCA system 1s composed by ATCA shelf sub rack 14 or 5 slots Processor blade a low cost ATX motherboard mounted on an ATCA carrier module connected to the ATCA backplane through an x16 PCIe link 3 2 GByte s 4 Upto 5 digitizer modules 5 An application programming interface API was developed based on the native functions device driver of Fedora 8 Linux based operating system OP with real time applications int
31. ith PWIDTH 49 lil fn INSTITUTO DE PLASMAS gara E FUSAO NUCLEAR T CNICO e Processed acquisition The word length consists of the detected event energy value of 32 bits and related TS of 32 bits The word length 1s always 64 bits Table 4 1 Acquisition mode word format Acquisition Mode Word Format Word 1 Raw MSB ce 15 16 bits 16b Burst Processed Energy 32b TS 32b Energy 32b TS 32b 5 Special Handling and Operation Considerations Due to sensitive devices on the module some precautions must be taken when handling and operating the board Failed to comply with these procedures may result in hardware damage and or malfunction e Observe electrostatic precautions when handling the boards Electronic components especially analogue circuits are sensible to ESD e Use the board on good ventilated computer cases Temperature above 65 C die temperature may cause permanent damage 50 n yi o n INSTITUTO DE PLASMAS INSTITUTO FUSAO NUCLEAR T CNICO y 6 TRP 400 Module Insertion at the ATCA Crate The handles insertion detail is the critical step on the module insertion procedure Both levers must be actuated synchronously Guiding the module to the backplane ATCA socket is done automatically by a guiding block near the ATCA plugs on the module there is a metallic block of 2cm A more detailed explanation of this insertion is presented
32. l 1 2V 1 8V 2 5V 3 3V 5 5V 48V Hardware PEM200F Address Figure 3 2 Module block diagram 15 INSTITUTO DE PLASMAS E FUS O NUCLEAR INSTITUTO SUPERIOR T CNICO CLK 4 TRG CTTS Inputs DC DC 48 V 12V DC DC 48 V 12V nane mek ar TEL IA J J pspeD7DaJ9J10J11 E Sd mms mme se nea ne pe co 3 Red o 4 o A e PCle LS hi Link K LEDs TRG Inputs SYNC Figure 3 3 Digitizer module Layout TOP view 16 fn INSTITUTO DE PLASMAS fari E FUS O NUCLEAR T CNICO 3 2 Analog to Digital Front End Figure 3 4 shows a simplified architecture of one of the four analog AC coupled input channels Rg2 Rpi_a2 ED 340R 1 16W 1 NA R 0 1uF lt Re lt Rpi_b2 j Rpi_c2 NU 0 9 9R 56R2 lt EPL 00 250 NTN Rpi at CAC1 340R 1 16W 1 E IN NANA KAA Vp Rf NAA 348R 1 16W 1 il e Rgl ji SN 3 VOUTp Rfilter2 ku VIN_ADC_p T vw Cfilter1 li 2 7pF Rfilteri NV Vin ADC n 100R a lt Rpi_b1 an geet S 56R2 ka BZX84C5V1 SOT Figure 3 4 TRP 400 Analog Block Input AC coupled lo This THS4513 amplifier circuit is designed for minimum gain of 0dB THS 4513 provides up to 10dB of gain with a low pass filter limiting the bandwidth to ISOMHz for 400 MHz ADCs and 100 MHz for 250 MHZ ADCs This circuit converts the single ended AC coupled input to differential and sets the proper input common
33. l blocks 1 and 2 Table 3 13 The common part is provided by several external clocks oscillators and two phase locked loops PLLs 1CS874003 02 from IDT and AD9510 from analog devices Each of the two equal blocks is provided by one PLL AD9510 with external VCXO from Silicon Laboratories 1GHz 800 MHz 3 3v LVDS 20ppm 14 peak to peak jitter 2ps rms jitter followed by clock distribution system The common AD9510 operate as the other two but with 1 GHz external VCXO also from Silicon Laboratories AD9510 is 38 ne yi o a INSTITUTO DE PLASMAS INSTITUTO E FUSAO NUCLEAR TECNICO 7 7 partitioned into two parts the PLL section for synchronization purposes and distribution section this last section 1s in use Table 3 13 Clock generation Integrated circuits IC i Quant i Supplier Distribui o Frequency AD9510 3 Analog i Common PLL Devices Block ICS874003 002 1 IDT Common PLL SG 636 1 Epson Common 33 MHz oscilattor Toyocom EG 2101CA 1 Epson Common 125 MHz oscilattor Toyocom Si550 VCXO l Silicon Lab Common I GHz VCXO SISSO VCXO 2 Siliconlab Block TRP 250 1GHz TRP 400 800 MHz 3 10 1 Common part 3 10 1 1 33 MHz oscillator Provides a single ended low frequency clock necessary to the SYSACE and also distributed to both FPGAs to allow a low frequency to cope with serial peripheral interface SPI fre
34. le 3 4 LED PCIE Link LED order D4 PEX to ATCA CH 1 D4 1 Lane3 ON D4 2 Lane ON wak D4 3 Lanel ON aka ceceno kii Lane ON D5 Block 2 FPGA to PEX D5 1 Lane3 OFF l D5_2 Lane2 OFF hen DS 3 Lanel OFF M A I E a DA Lane0 ON ___ D6 PEX to ATCA CH 12 D6 1 Lane3 OFF Not Used D6 2 Lane2 OFF D6 3 Lanel OFF Al HA DO 4 Lane OFF __ D7 Block 1 FPGA to PEX D7_1 Lane3 OFF l D7_2 Lane2 OFF ihren D7_3 Lanel OFF D7_4 LaneO ON 3 2 Debug LED indicators da gh fn INSTITUTO DE PLASMAS gara E FUSAO NUCLEAR T CNICO Debug LED indicators are presented at Table 3 9 and its location at the module layout is presented at Figure 3 15 These LEDs are divided into 4 different information groups 1 FPGA Status 11 DDR2 status 111 PLL status and iv Not defined can be used for future debugging functions for future applications of this ATCA system Table 3 9 Debug LED indicators LED Status Note DS8 Red ERROR FPGA programming Block 1 FPGA DONE 3 Off FPGA programmed OK RA D Red ERROR FPGA programming Block 2 FPGA DONE off FPGA programmed OK Ds15 Green initialization OK DDR Block 1 Red ERROR not initialized irani Status ae ni Green initialization OK DER EKA TT Red ERROR not initialized Initialization Status mms E ARE Ra Red ERROR not initialized Initialization Status ae r Green initi
35. mode voltage of the ADCs Figure 3 4 presents the circuit for the ADS5454 400 MHz the 100 ohms resistors and 2 7 pF capacitor between the THS4513 outputs and ADS5474 inputs along with the input capacitance of the ADS5474 limit the bandwidth of the signal to 150 MHz 3 dB 17 L a fn INSTITUTO DE PLASMAS D E FUSAO NUCLEAR fi INSTITUTO SUPERIOR TECNICO The conditioning circuit also presents a 7 attenuator composed by Rpi a Rpi b Rpi_c used to adjust signal level while controlling impedance mismatch and isolating circuit stages Offset can be software programmable although not used and not tested The channels can also be DC coupled where the CAC capacitors are replaced by two O resistors and two Rpy resistors were added to avoid violating the input common mode voltage range Vicr of the op amp as the source and the input termination resistor are referenced to ground There are two drawbacks for this configuration 1 It requires additional current from the power supply 11 Increases the noise gain of the circuit In order to minimize the noise of the DC coupling circuit the power reference of 5 V VCC5_CH next to the Rpy resistor was filtered with a 0 1 uF capacitor and a ferrite bead This circuit allowed filtering the high frequency electronic noise 3 2 1 Input Channel Performance From the ADC datasheets the noise histogram with inputs shorted inputs tied to common mode gives a RM
36. nels this choice 1s settled by hardware strapping dependent on were the controller is placed at the crate and the two DS ports are connected to each block FPGA PCIe endpoint as illustrated in Figure 3 12 26 INSTITUTO SUPERIOR gh fn INSTITUTO DE PLASMAS E FUS O NUCLEAR y T CNICO x4 BLK1 FPGA PCle endpoint m poni Upstream port ATCA Zone 2 Connectors n Han PEX8516 x4 Upstream port Na J am port x4 Blk2 FPGA DRW 05 11 06 2008 RCP PCle Figure 3 12 PCle switching diagram block This switch doesn t allow an US port with spread spectrum clocking SSC and a downstream port with constant frequency clock internal oscillator Consequently the 100 MHz PCle reference must be supplied by the root complex controller module Virtex 4 PCIe endpoint must be synchronously clocked to the root complex and the clock input must be 250 MHz The 100 MHz PCle reference must be multiplied to 250 MHZ while at the same time remaining compliant to the jitter specifications required by the Virtex 4 Multigigabit Transceivers MGTs To cope with this a jitter attenuator and clock distributor was used to distribute the 100 MHz PEX clock and generate and distribute the two 250 MHz PCIe endpoint clocks The PCle switch can be configured by hardware switches or hardware wired or by an EEPROM Electrically Erasable Programmable Read Only Memory memory The following configurations s
37. onfiguration switch SW5 O UP PORTO LX EEPRN XT GND Figure 3 18 SW5 PEX Configuration switch view Default status US port is the ATCA CHI PEX is hard wired programmed Table 3 11 SW5 PEX Configuration SW5 3 Status Note refer to Figure 3 18 ON 0 GND US port is CHI default UP_PORTO io We OFF 1 US port is CH2 i TT eee TIT TO TN Serial EEPROM present OFF 1 PEX is Hard wired programmed default Serial EEPROM not present 37 fi INSTITUTO SUPERIOR TECNICO gh fn INSTITUTO DE PLASMAS E FUSAO NUCLEAR 3 9 PUSHBUTTONS PB Table 3 12 Digitizer module Pushbuttons PB Function Notes Once the pushbutton PB pressed PBI PEX_RST PEX 1s resetted without rebooting the system SW4 SYSACE_RST SYSACE CF controller 1s resetted without rebooting system SW3 PPC RST PowerPC inside each FPGA 1s resetted without rebooting system This feature isn t used and this signal can be used for another operation to be defined IPMC not implemented SW6 IPMC_RST IPMC is resetted without rebooting system SW HANDLE SWITCH IPMC informs the shelf manager that a module has been inserted to the system Not implemented Table 3 12 shows all the PB present at the digitizer module each name and function 3 10 Clock Generation The clock generation section of the digitizer module can be divided into three parts a common one and the two equa
38. quencies The three AD9510 PLLs are programmed through SPI This clock is generated by a local Oscillator SG 636 from Epson Toyocom 3 10 1 2 Global Synchronization Synthesizer The global synchronization Synthesizer is responsible for generating most important module clock references maintaining phase lock with source references It 39 L a INSTITUTO DE PLASMAS rn E FUS O NUCLEAR fi INSTITUTO SUPERIOR T CNICO uses a clock distribution with integrated PLL chip AD9510 from Analog Devices and a VCXO from Silicon Laboratories Figure 3 19 The PLL can use the following selectable clock reference sources hardware coded 1 Local 125 MHz LVPECL low jitter SAW oscillator 25 ps peak to peak jitter 100ppm directly connected to the common PLL reference clock Use EG2101 from Epson Default 2 External unipolar LEMO J30 Can have 1 to 10 MHz It is compatible with JET CTTS_CLK 3 Block 1 FPGA LVPECL 2 5V IO standard output This is sourced by FPGA infrastructure y GHz VCXO CLK Local 100 MHz CLK BLK1 REFCLK 125 MHz BLK2 REFCLK 125 MHz RESSE FPGA_SR_CLK _ REFCLK UNUSED me Local 125 MHz CLK MN BLK1 CLKMGT2 250 MHz gt EXT REFCLK Vv BLK1 CLKMGT10 250 MHz J gt LVDS ADS9510 BLK2 CLKMGT2 250 MHz BLK2 CLKMGT10 250 MHz Figure 3 19 Global clock synthesis and distribution of mod
39. rocessed Mode At processed data mode per each detected event 48 bytes are stored where 8 bytes have the timestamp and channel information and the lasting 40 bytes have the pulse energy information For processing data PHA and PUR were applied to each detected 45 yi e a INSTITUTO DE PLASMAS D E FUSAO NUCLEAR O ON TECNICO event and the final data gathered ina word of 64 bits with the information of pulse energy related channel and time of pulse occurrence For this acquisition mode the read bandwidth doesn t depend on the sampling rate but instead on the event counting rate TRP 250 amp TRP 400 e Word length 64 bytes per event per channel e 512 x 2 bytes 2 us 1 28 us 1024 bytes 1KB per channel e 4GB IkB 4 Mevents s If the counting rate is of 1 Mevent s the memory will be filled up within 4s e If the PWIDTH 256 gt the filling time will increase to s e xl PCIe link gt 2 5 Gb s for 4 channels for one direction the module can attain 200 MB s If in the future the module will work with x4 PCIe link this value will increase to 800 MB s 3 11 4 Host Data Interface The TRP 400 module is compliant with PCI Express V1 0 PCISIG 2008 for communication protocol with the crate host controller Table 3 14 presents the currently defined Base Address Registers BAR for PCle Table 3 14 Currently Defined PCle BAR Structure BAR Enabled Type Value Size kB Add_64bit BARO YES Mem OxFFFO0008 1000 NO BAR
40. ted to Ven Idle channel noise is 0 4 LSB No input at differential amplifier x 4111 Y 36 113 Percentage ho h mM _ in FWHM 3 Bin gt 1 2 LSB 4106 4107 4108 4109 4110 4111 4112 4113 4114 4115 code number Figure 3 7 TRP 250 module Noise histogram of the module channel 1 with no signal applied to the input channel before the differential amplifier 20 n yi o INSTITUTO DE PLASMAS INSTITUTO E FUSAO NUCLEAR T CNICO The idle channel noise in FWHM LSB as well as the expected resolution of the module input channels are presented at Table 3 1 Table 3 1 TRP 250 module Idle channel noise in FWHM LSB and expected resolution of ADS5444 ADC Idle channel noise i FWHM LSB Resolution ADS5444 Figure 3 5 l do 04 Ew 12 6 bit CH 1 ADS5444 Figure 3 6 1 04 126 bit CH 1 Diff Amplifier Figure 3 7 3 1 2 amp 11 8 bit 3 2 1 2 TRP 400 NOISE HISTOGRAM WITH INPUTS SHORTED 25 400 MSPS 20 o 5 15 w ms E D 10 5 O rt n 0h 90 nota 0 lt F 10 460 K 0 O0 SC YX JI a t0 d0 K O909000 Tere e r e Nav Gi GI i GI AI J QI JjJ J IJI QI J J NUUN I Q N J CJ J O co co co e do Y e O CO 0 0 0 t0 d0 cC WD oC QO 00 oo Output Code Figure 3 8 Noise histogram with inputs shorted from ADS5474 datasheet Idle channel noise is 1 8 LSB 21 NA fi INSTITUTO SUPERIOR T CNICO fn INSTITUTO
41. ule references The Global Synchronization Synthesizer generates the following clock references 1 BLKx REFCLK clock reference 125 MHz for each acquisition block 2 Local Reference for PCIe clocking 100 MHz This is used when no System Controller is present 3 Four 250 MHz clocks divided into 2 clocks for each module block directly connected to the FPGA MGTs dedicated clocks 40 ne yi o n INSTITUTO DE PLASMAS INSTITUTO FUSAO NUCLEAR T CNICO y 7 3 10 1 3 PCle Clock Synthesizer When the controller module is present it can supply a reference clock through the root complex on the ATCA Zone 2 CLK3B Annex C for module PEX and PCIe endpoints Virtex 4 firmware The references are synthesized by ICS874003 02 from IDT Figure 3 20 This is the default operation mode When no ATCA controller is present the reference is supplied by the Global Synchronisation Synthesizer LVPECL BLK1 PCle 250 MHz CLK SSC 100 MHz CLK via B REFCLK BLK2 PCle 250 MHz CLK LVDS Local 100 MHz CLK PEX 100 MHz CLK 1CS874003 Figure 3 20 Synthesis and distribution of PCle related module clocks 3 10 1 4 CTTS clock CTTS clock is supplied by differential LEMO through rs485 interface and is connected to each FPGA for synchronization purpose The acquisition clock must be synchronized with the system clock for example in JET all acguisition channels must be synchronized
42. with the JET s clock 1MHz If the acquisition system has only one acquisition module this signal is used as the internal clock clk int to synchronize the system If more than one acquisition module is used than a master module must be settled This settlement is hardware coded by means of a jumper JMP1 or can be done by the host using the IPMC not implemented Once the master is settled the block 1 FPGA knows that must send the CTTS clock through the DS91C176 M LVDS transceiver to the crate backplane by setting the transmit enable of the transceiver and simultaneously receive it to the block 1 FPGA 41 ds yi o n INSTITUTO DE PLASMAS INSTITUTO FUSAO NUCLEAR T CNICO y CTTS backplane clock supplied by a zone 2 ATCA connector connected to each FPGA Synchronization purpose All the slave modules will only receive the clock from the backplane the transmit enable 1s always off All system modules will use the CTTS backplane clock as clk int and all system acquisition channels will be synchronous with clk int 3 10 2 Equal Blocks BLOCK 1 2 PLL AD9510 is in charge of respectively 4 acquisition clocks and the 2 DDR2 controller clocks Figure 3 21 presents an example of a TRP 400 digitizer module The clock reference is given by the global Synchronization Synthesizer section 3 10 1 2 The acquisition clock can be chosen from 8 different values refer to Table 2 1 e MHz VCXO GLK 50 MH
43. y 23 Table 3 3 System Ace Error and Status LED keke kek k k ka 24 Table 3 4 Possible configurations for the digitizer PCle switch Defenition of the DS ORRORI 28 Table 3 gt 5 Front pan l Connectors aeroespacial i rina 30 Table 5 6 Rear panel CONNECLOLS iaa 31 Table 3 7 Front panel LED Mel G N miii ect rie 32 Table 3 8 PCIE Link LEDs The Link features are default Hardware wired refer to TADE AA Ay nn PPP ERR er 33 Table 3 9 Depus LED GC AL OL Si apo 34 Table 3 10 SW2 SYSACE configuration kk k k kak k k 37 Table 3 11 SW 5 PEX Conti curation rinite pierino 37 Table 3 12 Digitizer module PushDUttonS kek kek k 38 Table 3 13 Clockgenerattonlntegratedcircults 39 Table 3 14 Currently Defined PCIe BAR Structure 46 Table 4 1 Acquisition mode word format Li 50 ne yi o n INSTITUTO DE PLASMAS INSTITUTO FUSAO NUCLEAR T CNICO f 7 Figures List Figure 3 1 Block diagram of an ATCA System up to 5 digitizer modules 14 Figure 3 2 Moduleblockdiagram K k 15 Figure 3 3 Digitizer module Layout TOP vevw 16 Figure 3 4 TRP 400 AnalogBlocklinputACcoupled 17 Figure 3 5 Noise histogram with inputs shorted from ADS5444 datasheet Idle channel I
44. z to 400 MHz ADC A CLK 50MHz to 400 MHz ADC B CLK gt LVPECL 50MHz to 400 MHz ADC C CLK gt 50 MHz to 400 MHz ADC B CLK BLK REFCLK 125 MHz REFCLK INIS _ 200 MHz DDR2 operating CLK ADS9510 REF CLK of 200 MHz for the IDELAY of DDR2 controller LVDS UNUSED Figure 3 21 TRP 400 acquisition and DDR2 clocks source example 3 11 FPGA Architecture FPGA 1s directly connected to the free running ADC channels acting as Temporary data buffer Real time event manager 42 f INSTITUTO DE PLASMAS E FUSAO NUCLEAR D Time stamping and Performing some high speed algorithms like digital level trigger detection PHA with pile up rejection PUR Each acquisition block of four input channels has its own control and resources The firmware allows a continuous acquisition mode where data is continuously stored from an initial trigger until memory filling or acquisition disabled by software The stored data can be raw segmented burst mode or processed Data retrieval is executed in post shot offline either raw segmented or processed data Figure 3 22 gt Register gt nas a ESSE w O o Memory x 13b 14b 13b 14b 13b 14b 13b 14b 4b Register 32b ab in n IE Ne m _TRP 400 M U T ex Segmented Mode Interface ATCA E Registers 32b Acquisition Window x R CONTADOR
Download Pdf Manuals
Related Search