The MUSIC User`s Manual
Contents
1. Compiling MUSIC Installing third party libraries required by MUSIC MUSIC requires a local installation of the FFTW and GSL libraries for its basic functionality These are very common libraries and are usually found pre installed at many scientific computing facilities MUSIC makes use of the shared memory parallelization paradigm and is itself OpenMP parallelized In order to also perform the FFTs in parallel the multithread version of FFTW needs to be installed In addition the HDF5 Hierarchical Data Format Library v5 can be installed Note that some output formats ENZO amp the generic format will be unavailable if the HDF 5 library is not installed The user will be guided through the necessary steps to perform a user level installation of all these libraries in what follows Installing FFTW required MUSIC uses the FFTW Fastest Fourier Transform in the West to perform the FFI based convolution of white noise with the matter transfer function MUSIC supports both FFTW2 and FFIWS for multi threaded shared memory machines Note that FFTW2 does not support transforms of very large arrays gt 32bit so that for very large refinement volumes FFTW3 needs to be used FFTW version 2 The library should be installed for multi threaded machines to allow highest performance on multi core processors In order to download and install the library follow these steps 1 Download the source code of the library in version 2 1 5 from the webs2. MULTI SCALE INITIAL CONDITIONS The MUSIC User s Manual written by Oliver Hahn version 1 0 Tuesday June 11 2013 Table of Contents Introduction Description of the method Cookbook for setting up a zoom simulation with MUSIC Compiling MUSIC Installing third party libraries required by MUSIC Installing FFTW required Installing GSL required Installing HDF 5 optional Adjusting the Makefile and Compiling MUSIC Running MUSIC The configuration file The setup Section The cosmology Section The random Section The output Section The poisson Section Output plug ins gadget grafic2 enzo tiOSy art cart nyx generic A sample configuration file Extending MUSIC Adding a new output plug in Adding a new transfer function plug in Adding a new region generator plug in MUSIC User s Manual CO Q O onwxnr RA A o 10 11 12 12 13 13 13 14 14 14 14 14 14 15 16 16 16 16 Density field in a 100 Mpc h box with two initial levels of Example of an N body simulation of a deeply nested re refinement generated with MUSIC gion of 6 initial levels generated with MUSIC and evolved with Gadget 2 to achieve an effective resolution of 8192 with 11603 particles in the high res region Introduction Description of the method MUSIC MUlti Scale Initial Conditions generates cosmological initial conditions for a hierarchical set of nested regions A detailed description of the
3. Alternatively in the MUSIC parameter file it is possible to specify the following extra parameters in section output which only apply to the Gadget 2 output plugin gadget_usekpc if set to yes the length unit will be in kpc i e UnitLength_in_cm becomes bool 3 08568025e21 the velocity unit remains the same gadget_usemsol if set to yes the mass unit will be in solar masses i e UnitMass_in_g bool becomes 1 989e33 the velocity unit remains the same gadget_num_files this will split the initial conditions file into several files necessary for very large int particle numbers gt 2 or for convenience reasons gadget_coarsetype Gadget particle type to be used for coarse particles 2 3 or 5 default is 5 int gadget_longids Use 64bit integers for particle IDs Default if the parameter is not given is 32bit bool if less than 29 particles 64bit if more grafic2 This is the format for RAMSES The specific options for the GRAFIC output plugin in section output are ramses_nml if set to yes a partial RAMSES nameiist file will be generated It can be bool used as a Starting point for the simulation namelist file A mask ic_refmap will be output that tags the masked refinement region ramses old_nml if set to yes a RAMSES namelist file will be generated containing the bool base grid structure Instead of a mask field a geometric refinement hierarchy will be written out It can be used as a starting point for the simulat
4. method can be found in the code paper Hahn amp Abel 2011 htto arxiv org abs 1 103 6031 We kindly refer the reader to that paper for all technical aspects as well as performance and validation of the code Cookbook for setting up a zoom simulation with MUSIC The procedure for setting up a zoom simulation follows typically the procedure of 4 steps given below Note that resolution levels in MUSIC are specified by their linear power 2 exponent i e a resolution of 128 cells or particles corresponds to level 7 log2 128 7 We use the term lower for levels synonymously with coarser and higher with finer Run a unigrid dark matter only pre flight simulation In order to set up unigrid initial conditions with MUSIC select first the desired resolution for this pre flight simulation Assume we want to run a 128 simulation the coarse grid level has to be set to log2 128 7 Since we want to run a unigrid simulation both levelmin and levelmax in section setup should be set to 7 Also the coarse grid seed needs to be chosen now and must not be changed afterwards To do this we set seed 7 in section random to the desired random seed This seed determines the large scale structure and we will only add subgrid noise when performing refinement later Now set the box size starting redshift all the cosmological parameters and the input transfer function in the respective sections Finally select the output plugin in section
5. output for the code with which you wish to perform this pre flight simulation Finally run MUSIC with the configuration file that contains all your settings and start your simulation Note that you also have to explicitly specify a redshift at which to generate the initial conditions MUSIC User s Manual 3 Identify a region of interest e g a halo and trace back the region to the initial conditions in order to obtain the extent of its Lagrangian patch Once the simulation has finished the region of interest has to be identified Save all dark matter particle IDs contained in the volume In the next step identify the positions of these particles in the initial conditions file as output by MUSIC Compute the bounding box of these particles and allow for some safety boundary Set up MUSIC to resample this Lagrangian patch at the desired resolution and possibly with baryons Insert the coordinates of the lower left corner of this bounding box into ref offset in section setup the lengths into ref_extent Set the maximum refinement level in Levelmax Do not change the seed you had specified for the full box simulation nor add new seeds for lower levels You can adjust levelmin if you do not specify seeds for these new coarser levels Finally add seeds for all additional finer levels including levelmax aS seed to section random Add baryons if desired by saying baryons yes in section setup and run MUSIC with this new configuration file
6. Experimental feature to give the subvolume a kick opposite to its predicted bool motion over time to minimize movement of the high resolution region with respect to the grid rest frame Region box default The default region generator is the cuboid box region specified by giving the coordinates of the bounding box of the high resolution region ref offset Offset of the refinement region Given as three comma separated real values in 3 real units of the box length e g 0 3 0 4 0 14 You can only specify ref_offset or ref_center not both MUSIC User s Manual 9 rer Center LSet sexe eene ref dims Region ellipsoid Center of the refinement region Given as three comma separated real values in 3 real units of the box length e g 0 3 0 4 0 14 You can only specify ref_offset or ref_center not both Extent of the refinement region Given as three comma separated real values in 3 real units of the box length e g 0 3 0 4 0 14 You can only specify ref_extent or ref_dims not both Instead of ref_extent also the resolution of the high resolution region in grid cells 3 int can be specified directly You can only specify ref_extent or ref_dims not both Instead of the default cuboid regions a set of points can be specified that define the high resolution region When using ellipsoid a minimum volume bounding ellipsoid will be fit to these points For particle codes only high resolution particle
7. ailable in this section There is support for various region specifications via plug ins that can be selected via the region parameter Currently MUSIC implements two region generator plugins that will be described below boxlength The size of the full simulation box in comoving Mpc h real ZStart The starting redshift for the simulation real region The method used to generate the refinement hierarchy Currently the methods string box and ellipsoid are supported If this parameter is not specified it defaults to box levelmin The level of the coarse grid which covers the full simulation box The number int specified is the 2 log of the number of grid cells per dimension e g a level of 7 corresponds to 2 128 cells dimension MUSIC User s Manual 8 levelmax The maximum refinement level in the simulation This value has to be larger or int equal to levelmin This sets the effective resolution in the refinement region to 2ievelmax cells dimension levelmin TE The level of the coarse grid when the density grid is computed the convolution int with the transfer function is performed This can be set to a number larger than levelmin the density field will be averaged down after the convolution has been performed and the Poisson solver is invoked Essentially this improves the accuracy of the coarser density modes when a low resolution in the coarsest level is desired in the final simulation Force _equal_extent Forces the refinement region t
8. erent random numbers Random numbers from file Instead of a seed number integer also a string can be given which gives the path and location of a file containing the white noise for the level The file needs to contain 2 numbers and should be in GRAFIC format i e a FORTRAN unformatted file written by the following FORTRAN in the GRAFIC2 source code file ic4 f MUSIC User s Manual 11 print Writing random numbers used in ic4 to filename open 11 file filename form unformatted rewind 11 writer mpd np2 nps 1seed do T pS write Liy Cee Cry 271r pe len Z end do close 11 where np1 np2 np3 have to be equal to 2 and iseed is an integer containing the seed not used further f is an array of double or single precision real numbers with the random values NOTE THAT FOR HISTORICAL REASONS THERE IS A FACTOR OF 1 CURRENTLY WITH WHICH GRAFIC2 NOISE HAS TO BE MULTIPLIED IN ORDER TO GET COMPATIBLE ICS This can be achieved inside MUSIC by specifying grafic sign yes in section random The output Section Output in MUSIC is performed using output plug ins The parameters that are given in this section will be parsed by each plugin itself Therefore the specific parameters available depend on the output plug in chosen However all output plugins are required to work in a minimal setting when no options are given besides a filename Then format the string identifier that selects the plug in Standard plug ins are s
9. es are determined by it This is always true for the lowest seed given It is also possible to provide only a seed for level 8 in which case the random values will be averaged down outside the refinement region Note that in the current implementation seed values are intimately tied to the level i e the same seed number given for two levels does not generate the same phases The seed for the lowest level specified which does not have to coincide with levelmin determines the phases Higher level seeds are just used to refine the noise Thus if you want to keep phases identical between simulations do not change any of the seeds for any of the levels that the two simulations have in common How does the parallel random number generator work The implementation of the parallel random number generator is as follows In order to be able to draw reproducible random numbers without the need to always draw all numbers at the highest resolution the grids are broken down into subcubes with independent random seeds These seeds are computed from the level seed and the position of the subcube and are thus unique Only random numbers for those subcubes which intersect regions of interest need to be drawn those effectively allowing for efficient and parallelizable random number generation on large grids The size of the subcubes can be chosen via the parameter cubesize not that changing cubesize for two simulations with the same seeds specified leads to diff
10. ion namelist file ramses_pvar_idx writes a refinement mask out as a passive variable field this parameter int specifies the index of the variable filename becomes ic_pvar_ lt idx gt default is 1 ramses_pvar_val value to which the passive variable should be initialized default is 0 5 real Note that Gadget 2 allows for a zoom region particle mesh that can be enabled in the Makefile Please consult the Gadget 2 manual for this option MUSIC User s Manual 13 enzo The ENZO output plugin has no extra parameters at this time It outputs by default a template parameter file parameter file txt in the output directory which contains the grid set up etc and can be used as a starting point for setting up a simulation It is not a complete parameter file to run a simulation other parameters need to be added The sample parameter file AMRCosmologySimulation enzo that comes with ENZO should be used for guidance tipsy The tipsy output plugin has the extra parameters listed below tipsy eps this specifies how the softening is calculated for the particles You can real specified one real number that specifies the force softening for each particle in units of the mean linear particle separation on each refinement level i e a value of 0 05 corresponds to 1 20 of the mean linear separation The default value if the parameter is not given is 0 05 tipsy native if set to yes native output will be written out if set to no all data will be bool pi
11. ite http www fftw org or follow this direct link which may however be outdated in the future 2 Open a terminal and expand the gzipped tar archive 3 Run the configure script with the following parameters configure prefix S HOME local enable type prefix enable threads enable float where HOME local will install the library into a directory local inside of your home directory You should set this to a directory that exists and where you are allowed to write files MUSIC User s Manual 4 Add enable float if you want to compile for single precision omit it if you want to compile for double precision Omit enable threads if you do not want to compile the multi threaded version of the library 4 Compile and install the library make install 5 Ifyou want to install both the single and double precision versions repeat from step 3 and omit add enable float when configuring in step 3 in order to compile also for double single precision FFTW version 3 The library should be installed for multi threaded machines to allow highest performance on multi core processors In order to download and install the library follow these steps 6 10 Download the source code of the library in version 3 x from the website htto Avww fftw org or follow this direct link which may however be outdated in the future Open a terminal and expand the gzipped tar archive Run the configure script with the fol
12. llipsoid cc MUSIC User s Manual 16
13. lowing parameters configure prefix S HOME local enable openmp enable float where HOME local will install the library into a directory local inside of your home directory You should set this to a directory that exists and where you are allowed to write files Add enable float if you want to compile for single precision omit it if you want to compile for double precision Omit enable openmp if you do not want to compile the multi threaded version of the library Compile and install the library make install If you want to install both the single and double precision versions repeat from step 3 and omit add enable float when configuring in step 3 in order to compile also for double single precision Installing GSL required MUSIC requires the GNU scientific library GSL to compute integrals and generate random numbers To download and install the library follow these steps 1 Download the source code of the library from the website lhtto Awww gnu org software gsl or follow this direct link to version 1 9 which may however be outdated in the future Open a terminal and expand the gzipped tar archive Run the configure script with the following parameters configure prefix SHOME 1local where HOME local will install the library into a directory local inside of your home directory You should set this to a directory that exists and where you are allowed to write files Com
14. o be a cube with edge length equal to the largest bool length determined from the region parameters padding Number of grid cells in intermediate levels when using levelmax gt levelmin 1 int surrounding the nested grids The extent of an intermediate level is N 2 2 padding if N is the number of cells in the next finer level overlap Number of extra padding cells for subgrids when computing the transfer int function convolutions These are discarded when computing the displacements but greatly reduce errors due to boundary effects blocking factor This parameter can be used to require the dimensions of initial grids to be int multiples of blocking factor Not used if parameter is not present This is mainly necessary to optimize for block based AMR align_top Require subgrids to be always aligned with the coarsest grid This is necessary bool for some codes ENZO but not for others Gadget periodic TE This controls whether the transfer function kernel is periodic or not The bool convolution is always periodic Should be set to yes baryons Set to yes if also initial conditions for baryons shall be generated bool use_2ZLPT Set to yes if 2nd order Lagrangian perturbation theory shall be used to compute bool particle displacements and velocities use_LLA Set to yes if the baryonic density field shall be computed using a second order bool expansion of the local Lagrangian approximation LLA See Section 5 3 in Hahn amp Abel 2011 center vel
15. on file as a command line parameter e g sI MUS TO my LESON All specifics regarding the setup of the initial conditions to be generated and options to control the code behaviour are given in this file The available options are detailed below MUSIC can be run using multiple threads on multi core shared memory machines The number of threads that MUSIC is allowed to use is taken from the OMP_NUM_THREADS environment variable This variable can be set for bash shell users by executing export OMP NUM THREADS 4 to allow a maximum of 4 threads for any subsequent call of MUSIC during the current shell session For tesh users the respective command would be setenv OMP NUM THREADS 4 The configuration file The MUSIC configuration file is organized into sections whose names appear in square brackets and options which are assigned a value with the equal operator Currently the following sections exist setup General setup for the simulation box and the refinement hierarchy as well as code behavior control cosmology Cosmological parameters and the transfer function are defined here random The random seeds and the behavior of the random number generator output The ouput plug in is selected and parameters for output poisson Behavior of the multi grid Poisson solver is controlled The setup Section General setup for the simulation box and the refinement hierarchy as well as code behavior control is defined using the options av
16. on of the power spectrum real Power law index of the density perturbation spectrum after inflation real 10 transfer Name of the transfer function plug in to be used Depending on the choice there string are different additional parameters that need to be set MUSIC comes with the following default plug ins e BBKS for the Bardeen fit to the transfer function without baryon features eisenstein for the Eisenstein amp Hu fit for the CDM transfer function with baryon features CAMB for CAMB output transfer functions tabulated The filename of the additional option transfer file has to indicate the file from which the tabulated transfer function shall be read YHe Helium abundance This is used to compute the initial gas temperature if baryons real are present Will only be used if the simulation code supports reading temperature fields Optional Default value 0 248 gamma Adiabatic exponent This is used to compute the initial gas temperature if baryons real are present Will only be used if the simulation code supports reading temperature fields Optional Default value 5 3 The random Section In this section the random seeds for the various levels should be given in the form seed level seedval i e e g seed 7 152521 seed 8 532211 In this example the random values generated for level 7 i e a grid with 128 cells are used as constraints for the next finer values i e the large scale mod
17. on options FFTW3 yes MULTITHREADFFTW yes SINGLEPRECISION no HAVEHDF 5 yes HAVEBOXLIB no itt tH HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE HE E HE HE HE HE HE HE HE E HE E HE E HE HE HE HE HE HE HE E E H H compiler and path settings CC g OPT 03 CFLAGS Wall fopenmp LFLAGS fopenmp lgsl lgslcblas CPATHS I I HOME local include I opt local include I usr local include LPATHS L HOME local lib L opt local lib L usr local lib MUSIC User s Manual 6 The first three options configure basic functionality of the code FFTWS MULTITHREADFF TW SINGLEPRECISION HAVEHDF5 HAVEBOXLIB set to yes if you are using FFTW8 no if you are using FFTW2 set to yes if you configured FFTW with the enable threads option set to yes if you want to run MUSIC in single precision mode and you have compiled a version of FFIWx with enable float set to yes if you have installed the HDF5 library and want to create output for ENZO set to yes if you have installed the boxtlib library and want to create output for Nyx needs the path in BOXLIB_HOME You also need to add the paths where you installed the third party libraries mentioned in the previous section to CPATHS and LPATHS just replace the bold parts MUSIC User s Manual Running MUSIC MUSIC is run by specifying the path name of a configurati
18. ped through XDR External Data Representation encoding Default is no i e XDR encoded values art The ART output plugin has no extra parameters at this time cart The Chicago ART output plugin has no extra parameters at this time nyx The nyx output plugin has no extra parameters at this time generic The generic plugin can be used to output simple HDF 5 data arrays for all fields that can be readily used for subsequent analysis visualization with other tools MUSIC User s Manual 14 A sample configuration file setup boxlength zstart levelmin Tevelmin TE levelmax padding overlap rer OFESEL rer eRLCent align top baryons UGS ART use GEA periodic ITE cosmology Omega m Omega L Om ga p HO sigma 8 nspec transfer random seed 7 seed 8 output format filename Shite back poisson fit fine accuracy pre smooth post smooth smoother laplace Order grad Order MUSIC User s Manual 100 Or 276 0 724 20S TOS Gee 0 961 elsenstein 12345 23456 gadget2 Les gadget dat yes NO le 5 This creates a 100Mpc h box with a 128 base grid and a refinement region at 256 effective resolution with a corner point at 40 40 40 Moc h and a side length of 20 Mpc nh in all directions 15 Extending MUSIC Adding a new output plug in New output plugins can be added any time to the directory plugins Note that the naming convention is that output plugins start with ou
19. pile and install the library make install MUSIC User s Manual 5 Installing HDF5 optional Some MUSIC output plug ins such as ENZO and the MUSIC generic format require the Hierarchical Data Format v 5 HDF5 If you want to use any of these plug ins you need to install HDF5 Follow these steps to install 1 Download the source code of the library from the website lhtto Avww hdfgroup org HDE5 or follow this direct link to the most recent version which may however be outdated in the future 2 Open a terminal and expand the gzipped tar archive 3 Run the configure script with the following parameters configure prefix HOME 1local where HOME local will install the library into a directory local inside of your home directory You should set this to a directory that exists and where you are allowed to write files 4 Compile and install the library make install Remark Boxlib and Nyx Nyx requires boxlib to be installed The path to the installed version should be given in the MUSIC Makefile Further details are however beyond the scope of this manual Adjusting the Makefile and Compiling MUSIC Ideally you have to only adjust the paths to third party libraries and your compiler in the Makefile Below are the relevant sections at the beginning of the Makefile You might need to adjust the bold expressions iat ar at at ae aE aE ae aE aE ae aE aE AE a aE AE aa EAE aE aaa aaa eae Compile time configurati
20. s in this region will be written out for grid codes masks will be written out that reflect the shape of the ellipsoid region point file region POING Shiite ASCII file containing point coordinates of the particles in the Lagrangian 3 real patch of the region of interest These need to be given in box coordinates i e 0 1 one point per line optional If the simulation from which the coordinates in region_point_file 3 int have been determined was already a zoom simulation the coordinates in the IC file need to be shifted back to their original position the zoom region had been centered Specify here the shift that was applied in the centering of the zoom region as has been output by MUSIC in the line e g Domain will be shifted by 12 12 12 If you specify the shift also the levelmin of that simulation needs to be specified as below region_point_levelmin optional but req if region point shift is specified The levelmin 3 real of the zoom in simulation that has been run to determine the points that determine the region The cosmology Section Specific settings for cosmological parameters and physical properties are set in this section Omega m Omega L Omega D HO sigma 8 nspec MUSIC User s Manual The total matter density parameter now real The cosmological constant density parameter now real The baryon density parameter now real The Hubble constant now in km s Mpc real Normalizati
21. tput The output plugins need to derive from the class output plugin declared in file output hh and implement all the write functions The implementation of the grafic2 plugin in file plugins output grafic2 cc can be taken as a simple example Note that the plugin has to register itself using the abstract factory pattern with the plugin creator by adding a line similar to this to the end of the new plugin file namespace output plugin creator concret e lt my output plugin gt creator myplugin name When MUSIC is recompiled using make the plugin will be automatically available by specifying output format myplugin name in the parameter file no other MUSIC source file needs to be touched Adding a new transfer function plug in New transfer functions e g new fits modified fits or new reading modules for tabulated functions can be added via the plug in mechanism These plugins need to derive from the class transfer function plugin defined in transfer function hh Examples can be found in the plugins directory Adding a new region generator plug in New region generators can be added via the plug in mechanism These generate a bounding box that initializes the MUSIC hierarchy They are queried cell by cell upon output to provide support for masking The region generator needs to derive from the class region generator plugin defined in region generator hh An example is the ellipsoid generator in plugins region e
22. tring generic gadget grafic2 for RAMSES e g and enzo filename name and path of the file or directory depending on the plug in where the output string data will be stored Further options specific to the chosen output plugin are possible They are listed below The poisson Section laplace_order order of the finite difference approximation for the Laplacian operator int grad_order order of the finite difference approximation for the gradient operator int accuracy the residual norm required to establish convergence of the multigrid solver real smoother name of the smoothing sweep method used in the multigrid method string gs Gauss Seidel jacobi Jacobi sor Successive Overrelaxation pre_smooth number of pre smoothing sweeps int post_smooth number of post smoothing sweeps int fit fine controls whether the hybrid Poisson solver shall be used see paper for details bool MUSIC User s Manual 12 Output plug ins gadget For Gadget the high resolution particles are written out as Gadget type O for gas if baryons are enabled and as type 1 for dark matter The coarse level particles are all of type 5 and correspond to total matter particles no SPH particles will be present outside the high resolution region By default initial conditions for Gadget are generated with the following units UnitLength in cm 3 08568025e24 aL Mpc UnitMass in g 1 989e43 1 0e10 solar masses UnitVelocity in cm per s led 1 km sec
Download Pdf Manuals
Related Search