Practical lab 1
Contents
1. DJF amp JJA seasons Change to your case directory s cd SUSER t torial Make a new script cp run_testOl csh run_test0l_restart csh Edit the restart run script in emacs vi or your editor of choice emacs nw run test0l_restart csh Note the argument to emacs nw no window is an easy way to run inside your shell window instead of popping up a new one Look through the script for the text startup Change itto continue Continue Run cont But first check to make sure you made the change you meant to If you run another startup job it will overwrite the previous run 6 grep continue run_test0Ol_restart csh bsub U reservation id lt run_test01l_restart csh Given the instruction to continue CAM looks in the wrkdir for pointer text files that indicate the latest restart files generated by each component of the model For instance ptmp bundy test02 gt ls 1 rpointer rw r r 1 bundy ncar 302 Jul 23 13 04 rpointer atm rw r r 1 bundy ncar 257 Jul 23 13 04 rpointer drv Sow En 1 bundy ncar 257 Jul 23 13 04 rpointer ice rw r r 1 bundy ncar 257 Jul 23 13 04 rpointer lnd Sey eee 1 bundy ncar 33 Jul 23 13 04 rpointer ocn ptmp bundy test02 gt cat rpointer atm test02 cam2 r 0000 01 01 03600 nc More to do Congratulations You now know the basics of configure build amp run as well as restart If you want more to do here are some suggestions 1 Make a new case Run2. and use storm instead of bluefire 6 Or look at the source code for an idea of how the model runs The main physics interface is models atm cam src physics cam physpkg F90 or take a step back to see how each timestep is controlled dynamics physics coupling etc in models atm cam src dynamics fv stepon F90
3. set camroot fis01 cgd ccr hannay cam_tutorial_cam3_6_48 CAM run script directories We don t have room to build and run the model on home so we use the ptmp user directories Set paths to needed directories Swrkdir is a working directory under which the model will be built and run set wrkdir ptmp SLOGNAME blddir is the directory where model will be compiled set blddir Swrkdir Scase bld rundir is the directory where the model will be run set rundir Swrkdir S case cfgdir is the directory containing the CAM configuration scripts set cfgdir Scamroot models atm cam bld The script makes these directories if they don t already exist mkdir p Sru undir echo cannot create rundir amp amp exit 1 mkdir p SBLddIr echo cannot create Sblddir amp amp exit 1 mkdir p SUS sre echo cannot create susr sre amp amp exit 1 CAM run script configure gmake As discussed previously we set some script variables to pass to configure set dycore fv dycore is the dynamical core sld eul or fv set resolution 4x5 resolution for fv 1 9x2 5 2x25 4x5 or 10x set runtype startup run type startup continue or branch The script looks to see if an executable exists If it does we assume this has already been done and skip the configure step If the executable doesn t exist call configure If an executable doesn t exist build one if x blddir cam t
4. the model for 6 hours writing data out every hour Look in the CAM users manual for a description of the namelist parameter nhttrq You ll probably need to find out the model timestep for this dynamical core and resolution hint it is in the standard out stdout a text file from your previous model run as well as in the source code deltat You can also use mfilt to change the number of time samples in each history file Look at the resulting history files using ncdump v time date datesec file to see if it is working 2 Either in a new case or rerunning the one from 1 add a field to the history tape You can find a list of the available fields in the stdout from your first model run The namelist parameter is fincl as referenced in the user manual You can also remove all the default fields from the history file empty_htapes and instead put a few of your choice 3 Using fincl2 fincl3 etc and multiple values for nhtfrg repeat the model run with the following history files hO default fields printed out every hour hi T TS PS Q instantaneous every 3 hours h2 T TS PS Q averaged every 6 hours More to do cont 4 Look through the namelist parameters in the users manual or search the source code for namelist and look in the files for newer parameters for other things to change 5 And of course you can always look at the data using ncview or your favorite analysis package Remember to be a good citizen
5. time so we often have to resubmit a job several times This is how you tell the model to restart branch to continue an existing run with changes CAM run script mpirun and stdout To run the model with tasks amp threads we must use mpirun with the hybrid_launch instructions Nptrun lst Ju usrt locat bin yb id launch Sb lddir7fcam When it is done running the standard text output error will be in a log file in your case directory or wherever you specified it in the BSUB directives You can browse this output or quickly tell if the model successfully completed by looking for this line QS KIANA BND OF MODEL RUN Seer e ea CAM output ncdump While the model is running it will write output into history files in the wrkdir ptmp SUSER test01 The CAM history files are written out monthly by default and have the naming convention SCASE cam2 h0O YYYY MM nc e g test01 cam2 h0 0000 01 nc To get a snapshot of the data in the file use ncdump 6 cd ptmp SUSER test0l ncdump h test01 cam2 h0 0000 01 nc Pipe into less or more or save in a text file for browsing ncdump h test01 cam2 h0 0000 01 nc less ncdump h test01 cam2 h0 0000 01 nc gt dump txt See all the data in text format by leaving off the header h option ncdump test01 cam2 h0 0000 01 nc Or view one or more fields ncdump v TS PS test01 cam2 h0 0000 01 nc CAM output ncview For a graphical view much more useful there is a tool c
6. CAM Tutorial configure build amp run Dani Coleman bundy ucar edu July 27 2009 CAM ts a subset of CCSM Data Sea Ice Documentation of CAM Scientific description http www ccsm ucar edu models atm cam docs description User s guide http www ccsm ucar edu models atm cam docs usersguide Download code http www ccsm ucar edu models atm cam docs CCSM information http www ccsm ucar edu NCAR supercomputer bluefire IBM clustered Symmetric MultiProcessing SMP system based on the Power6 chip Nodes Each of the 128 nodes contains 32 processors Disk storage 5 GB home directory backed up and not subject to scrubbing 250 GB ptmp directory not backed up and is subject to scrubbing Parallelization Shared memory SMP OpenMP OMP Distributed memory SPMD Message Passing Interface MPI CAM usually uses a combination of the two called hybrid mode All of this information and more available online http www cisl ucar edu docs bluefire be_quickstart html Log in to bluefire and change shell Log on to bluefire cryptocard ssh bluefire Token_Response lt enter the number from your crypto card here gt Set shell This tutorial is written using t c shell tcsh Feel free to use whatever you want You can find out which shell you re running by typing shell If you want to change to tcsh shell follow these instructions rsh bems Follow the prompts to change your shell
7. alled ncview We should be good citizens and move to one of the analysis machines storm1 or storm4 bluefire is really just for running the model ssh Y storm1 UCAS Token Response Crytocard required Bluefire s ptmp directory is mounted in a different place on storm cd biptmp SUSER test0l nceview test0l cam2 h0 0000 0Ol nc Click on 2D vars and peruse Or 3D vars Finally print a screen shot by selecting print and then printing to a file This file or others can be your test cases for transferring data to your own computer next page Copying data from bluefire Use ftp server 1 Copy data from bluefire storm to ftp server on bluefire ftp ftp cgd ucar edu login anonymous password email cd incoming put filename 2 Copy data from ftp server to local machine local machine ftp ftp cgd ucar edu login anonymous password email cd incoming get filename Note On my Windows Vista machine found an ftp program by going to the start menu and searching for ftp It found ftp exe clicked on that and it opened an ftp window ftp gt From there you need to open a connection fto gt open ftp cgd ucar edu Continue Run In order to do this the model must have completed 1 year The first step was running the model for 1 year Now let s restart it to run for another year The standard atmosphere model diagnostic package requires over 14 months of data in order to get full
8. hen cd Sblddir echo cd Sblddir failed amp amp exit 1 Scfaqdir contigure verbose dyn Sdycore res Sresolution usr_src Susr_src lt user modifications directory phys cam3_5 1 lt specify the version of physics to use ntasks S ntasks nthreads SOMP_NUM_THREADS echo configure failed amp amp exit 1 After configure the model is compiled gmake J8 gt amp MAKE out echo CAM build failed CAM run script check config amp make Since the model should have already run check the files configure and gmake have created change to the build directory cd ptmp SUSER test01 bld If the directory doesn t exist and your job is no longer in the queue there is a problem get help ls There may be a few or a lot of files depending on where the model is in the building process First configure creates text files feel free to look at them using less more or an editor Makefile instructions on how to build compile for gmake Filepath text file list of which directories to search for source code config_cache xml cache of settings used by configure Can be used to copy the case misc h preproc h fortran files archaic config_cache_cice xml Similar files for the sea ice model CICE _cppdefs The Gmake creates Srcfiles a list of all the source code files found under Filepath Depends a list of how the files depend on each other object o and module mod files for each of the source code f
9. iles cam the executable CAM run script build namelist The model our executable ptmp USER test01 b1d cam needs input instructions These are provided through fortran namelists created by the CAM build namelist tool First we create a little text file in the blddir that will be passed to build namelist cat lt lt EOF gt namelist_options amp cCamexp stop_n 2y stop_option nsteps npr_yz 072r 2r Oy EOF you can view it less ptmp SUSER test01 bld namelist_options Then we invoke the build namelist tool with some of its options and our text file options Scfgdir build namelist s case case runtype runtype infile namelist_options echo build namelist failed amp amp exit 1 This creates several small text files in blddir that are then moved to wrkdir ls ptmp SUSER test01 _in CAM run script namelist options For this run we re using these options is the number of days to integrate units depend on stop_ option stop_n stop_option can be set to nsteps ndays nmonths nyears npr_yz Gives information on how to break up the global domain on multiple processors We normally don t need to set it but running at this low resolution requires it The 8 2 2 8 setting is only good for 16 processors build namelist option runtype startup for a run starting from initial conditions continue to continue an existing run bluefire only allows 6 hours of wall clock
10. n look for output see following slides If the response is No job found then the job exited too quickly we expect this run to take about 30 minutes and there is a problem CAM run script BSUB While that job is running we ll look at the run script and see what the run is doing First you ll see a bunch of queuing system directives denoted by BSUB These are set to what we need today on bluefire BSUB BSUB BSUB BSUB BSUB BSUB BSUB BSUB BSUB BSUB poe 16 use LSF openmp elim exclusive use of node comment out to share number of total processes must be compatible with ntasks amp npr_yz below span ptile 16 out Jd out Jd regular 0 20 37591047 testOl number of processes on each node number of CPUs on each node must be compatible with procs node amp nthreads below output file error file when same as output o e are merged queue wall clock limit HH MM Default is six hours 6 00 Project number for this tutorial job name your choice Settings are coded as follows settings you should never change settings related to parallelization processes amp threads that you can change when you know what you re doing settings you ll need to be able to change after this tutorial settings you can change anytime you want CAM run script parallelization The parallelization is accomplished by setting e the number of tasks distribu
11. r by hand If you configure outside of the queue make sure the set ntasks setting is equivalent to BSUB n CAM run script setenv Most of the environment variables are bluefire specific But when you run on another machine you ll need to set the NetCDF paths setenv INC_NETCDF usr local include setenv LIB NETCDF usr local 1ib64 r4i4 And you ll need to set the path the the datasets that CAM reads in downloaded with the source code Root of datasets for the CAM needs to be customized unless running at NCAR Contains the initial and boundary data for the CAM setenv CSMDATA s cgd csm inputdata CAM run script case Give your model run a case name This should change any time you change the model For example today we ll run test01 When you make changes later on this week you ll have anew case name Give a unique case name to this model run set case testOl We re not going to modify source code today but we ll set up the model to look for any modifications whenever it compiles We make a directory called mods_ case and in the future copy any modified files there set wsr sre blhome SLOGNAME tutorial mods_Scase Specify the path of the source code for CAM You can continue to point to this in the future this will be the location of the un tarred code you download Root of the source code for the CAM needs to be customized the root directory contains the subdirectory models
12. ted memory SPMD MPI e the number of threads per task shared memory SMP OMP In general you want ntasks nthreads total number of CPUS There are 32 CPUs on each bluefire node On complication is that bluefire works most efficiently if you actually run double the number of threads multi threading so there are 64 virtual CPUs per node So for bluefire the calculation is ntasks nthreads 64 virtual CPUs Our software engineers tell us that running 4 threads per task is a good configuration for CAM For this tutorial we re going to use 1 node which has 32 CPUs The equation gives us the number of tasks to ask for ntasks 64 CPUs 4 threads 16 total tasks CAM run script parallelization cont The number of threads is set as an environment variable in the run script setenv OMP_NUM_THREADS 4 The number of tasks is set in the BSUB directives BSUB n 16 number of total processes And we also need to set how many tasks per node BSUB R span ptile 16 number of processes on each node and that we want exclusive use of the node BSUB x exclusive use of node comment out to share Note that this is explained in the bluefire documentation http www cisl ucar edu docs bluefire be_quickstart html Also note that CAM currently needs to know the number of tasks when configuring for the sea ice model to build So we set a script variable ntasks in the run script either from the batch queue environment LSB_HOSTS o
13. to tcsh The change may take up to 60 minutes to propagate Meanwhile invoke the tcsh for this session by typing tesh Run script configure There are 3 steps to running CAM configure create the files that will allow us to compile amp run build compile the source code and make an executable run execute the model The CAM distribution includes tools for these steps but it isn t as easy as installing the latest version of Firefox Configure e just a perl script in the code distribution models atm cam bld configure e set options that are required for compiling e resolution 4x5 degree e dynamical core finite volume e version of the physics we want to use cam3_5_ 1 e which directory holds code modifications If you change any of these things you must reconfigure creates the files necessary for us to compile CAM Makefile Filepath etc in the model build directory Run script build and run Build This is the verb for compile source code and create the executable All of this work is done for us by a UNIX tool called gmake Using the files created by configure gmake knows where to find source code how the source code files depend on each other and what compiling options are required Run To run the model we give it a list of input options that don t have to be built into the executable for example how long to run which fields we want on the output datasets and invoke the parallel run
14. tool used on bluefire These basic steps configure build amp run are set up in a csh script that comes with the model distribution Your Turn The rest of the tutorial is for you to do at your own pace We start by submitting the run script and then examining it while the run is going Set up run directory We will have use two directories 1 run or case directory on nome for scripts and modified source code 2 work directory on ptmp for model output object files and executable This will be created automatically by the run script 1 run or case directory cd change into home directory g mkdir t t rial make a new directory cd tutorial change into directory copy the script to this directory cp blhome bundy tutorial run i1bm tutorial csh runutest0l sesh oO BTW useful unix commands pwd print current directory back return to previous directory set case cwd set a variable called case to current directory o cd Scase change to case directory from anywhere Submit test run of CAM Submit the run script now bsub U reservation id lt run_test0Ol csh Output Job lt 726040 gt is submitted to queue lt regular gt Make sure it is running bjobs Output JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME 726040 bundy PEND regular bel005en camrun Jul 23 12 56 STAT PEND status is pending means it is waiting in the queue When STAT RUN the job is running and you ca
Download Pdf Manuals
Related Search