Recent changes are in highlighted in RED
A description of the changes is available here.
Since d0runjob is a UPS package, available versions are obtained by the command :
ups list -a d0runjob
d0runjob has replaced mc_runjob with more or less the same functionalities. Some documentation is available here
The changes compared to mc_runjob are :
############################ Init.csh ####################
# to be sourced ( c-shell environment is assumed )
# If declaring/storing files in SAM
# setup sam_cp
setup -j d0runjob <version>
setup -j sam_client v2_1_23
source ${SAM_CLIENT_DIR}/etc/init_sam.csh ${SAM_CLIENT_DIR}
# Place the local directory where the paramFile sits in the PYTHONPATH.
setenv PYTHONPATH ${PYTHONPATH}:.
setenv RTE_DIR <D0code directory>
setenv CARDDIR <Cardfile directory>
setenv JOBCACHE <A working directory>
################################################
RTE_DIR is the directory where the releases are available.
CARDDIR is the directory where versions of the cardfiles package are available.
At the CCIN2P3, most versions are available at /afs/in2p3.fr/group/d0/packages/cardfiles.
See if it contains the version you are looking for, or extract
it from CVS ( cvs export -d <version> -r <version> cardfiles ).
JOBCACHE is the working directory where the job will be executed (in interactive mode).
These environment variable values are superseded by the d0runjob command options (see below) :
--rtedir=<value>
--carddir=<value>
--jobcache=<value>
Then, you just have to source the initialization script to create your environment.
WARNING : as the d0runjob package is using a more recent version of Python than any D0RunII release you should not execute a setup D0RunII vxx.yy.zz command when using d0runjob.
Remark : At CCIN2P3 the MinBias files are stored in HPSS (even on /sps), so you must not define the variable MinBiDir ( as on clued0).
The list of files to be used is recorded in the file as defined by the minbidataset parameter in the digitized section of the paramFile (see later).
[->contents]
There is at the CCIN2P3 a Python script to do it :
It returns on the screen or in myparamFile the parameters of the request. The paramFile name should have the extension '.py'.
Here is an example for request 55052.
######################## req_55052.py #################
import SAM
from SamUtility.CaseInsensitiveDictionary import CaseInsensitiveDictionary
params={'digitized': CaseInsensitiveDictionary({
'StagerCmd' : 'D0rfcp',
'calorimeternoise' : 'off',
'd0release' : 'p20.08.02',
'frameworkrcpname' : 'runD0Sim_noCalNoise_run2b.rcp',
'mergeminbias' : 'on',
'minbidataset' : '/afs/in2p3.fr/throng/d0/info_data/mcp20/zerob_p200801_RunIIbMC_REALsample_2.py',
'minbiopt' : 'Fixed',
'numminbi' : '1.0',
}), 'global': CaseInsensitiveDictionary({
'datatier' : 'reconstructed',
'description' : 'CSMC Pythia gam-Z(+jets)->ee(+jets) m=60-130 GeV (store d0gstar)',
'groupname' : 'dzero',
'phase' : 'mcp20',
'producedbyname' : 'jaffre',
'producedforname' : 'tadams',
'requestid' : '55052',
'runtype' : 'Monte Carlo',
'stream' : 'notstreamed',
}), 'reconstructed': CaseInsensitiveDictionary({
'StagerCmd' : 'D0rfcp',
'appfamily' : 'reconstruction',
'appname' : 'd0reco',
'appversion' : 'p20.08.02',
'd0release' : 'p20.08.02',
'frameworkrcpname' : 'runD0reco_mc.rcp',
}), 'generated': CaseInsensitiveDictionary({
'cardfiledir' : 'dzero',
'cardfileversion' : 'v01-00-00',
'collisionenergy' : '1960.0',
'd0release' : 'p20.08.02',
'decay' : 'ee_sm.n',
'etagt' : '-5.0',
'etalt' : '5.0',
'generator' : 'pythia',
'kinmassgt' : '60.0',
'kinmasslt' : '130.0',
'pdflibfunc' : 'LHPDFCTEQ6L1',
'production' : 'gam-z',
'ptgt' : '0.0',
'ptlt' : '-1.0',
'topmass' : '175.0',
'useevtgen' : 'on',
}), 'simulated': CaseInsensitiveDictionary({
'StagerCmd' : 'D0rfcp',
'd0release' : 'p20.08.02',
'geometry' : 'plate-run2b',
'keepparticlecalenergy' : 'off',
'samstoreoutput' : 'd0gstar',
}), 'relocate': CaseInsensitiveDictionary({
'tasks' : 'reconstructed',
'copymode' : 'd0rfcp',
'targetdir' : '/hpss/in2p3.fr/home/U/USER',
'targetnode' : 'cchpssd0.in2p3.fr',
})}
#######################################################
The getRequestParm.py script is special to the CCIN2P3 as it adapts the
original file extracted from SAM to the CCIN2P3 environment.
Look at the minbidataset parameter in d0sim or the relocate task which by default copies the produced events output files in HPSS.
D0runjob.py (alias d0_runjob) is a utility to run/check a MC request. A single job is prepared/submitted.
Usage:
d0_runjob [-flags] <--requiredOptions> [--options]
-flags:
-c to check (default)
-h help
-r to run (incompatible with check)
-s to create a shell script
-v verbose mode
--requiredOptions :
--paramFile= # file containing request params
and/or ( --paramFile has priority)
--requestId= # info obtained from requestId
--options:
--carddir= : full path to cardfiles
( default is env var CARDDIR )
--check : equivalent of -c (INTERACTIVE use)
--copybackDir= : (used in BATCH mode) to retrieve some files from the worker
--help : equivalent of -h
--jobname= : jobname
--jobcache= : directory where job is executed
default ( in INTERACTIVE mode : env var JOBCACHE )
( in BATCH mode : local directory
--jobtype= : choose between (MCchain is default)
: ['MCchain', 'TMBAnalyze', 'GenD0gSim', 'D0Gstar', 'GenCafe', 'SimReco', 'GenRecoCafe', 'Merge', 'D0GSimReco', 'MergeCafe', 'D0Reco', 'D0Sim', 'Gen', 'GenD0g']
(see below to see the correspondance with the performed tasks)
--loglevel= : INFO (default),
: WARNING
: DEBUG
--minbidir= : Minbias directory if needed
( default is env var MinBiDir if defined )
--nEvents= : nb of events to generate and/or process (default: 0 )
--rtedir= : code distribution directory
( default is env var RTE_DIR )
--run : equivalent of -r (INTERACTIVE mode)
--submit= : BATCH job option string ( assumed -s)
--inputfilename=: (i.e a generator event file)
--inputfilelist=: name of the file containing the list of files to process in a single job i.e files to merge)
--mdinputfilelist= : name of the file containing the list of MetaData files
--provinputfilelist= : name of the file containing the list of provenances files
--runnumber= : run number for the generated events ( if not set it is read from the runnumberfile )
--runnumberfile= : name of the file the run number counter
(default is $HOME/run.number)
--site= : choose between (ccin2p3, clued0, ...)
used in outputfile naming,
triggers the making of the shell script for batch submission
initialises 'OriginName' and 'FacilityName' variables in metadata
--skiprecords= : record nb to skip on input file (default: 0)
--noSAM : ( default ) skip any declaration and transfert to SAM
--samparams=filename: Allow declaration and transfert to SAM/Enstore
using parameters to be found in <filename> file
see here.
Additional options are offered with the script D0runjobs. It accepts all
the options of d0_runjob and more see below:
D0runjobs is a utility to submit several D0runjob jobs.
Usage:
D0runjobs [-flags] --options
Some of the options are special to this script and not transmitted
to D0runjob.py
Some are in common to his script and D0runjob.py, they are
transmitted to D0runjob after some changes
The other are transmitted unchanged.
-flags:
-h help
-v verbose mode
D0runjobs.py options only ( i.e filtered out ):
--jobprefix : jobprefix ( mandatory )
A counter is maintained in file 'Runjobs' on your home directory for each job prefix, and a jobname is defined as -<3digit-value>
--repeat : nb of jobs to submit
: force to 1 for a merge or a generator job
: otherwize it equals the nb of input files to process
--skipfiles : to skip files on an inputfilelist
--inputfilesperjob : nb of files to input per job (default: 1)
Common options to D0runjob.py
--help : equivalent of -h
--inputfilename : (i.e a generator event file)
--inputfilelist : (i.e. list of d0gstar files)
--skiprecords : record nb to skip on input file (default: 0)
--nEvents= : nb of events to generate and/or process ( mandatory )
The description of the various jobtypes follows:
| Jobtype | Task labels |
|---|---|
| MCchain | generated, simulated, digitized, reconstructed |
| Gen | generated |
| D0Gstar | simulated |
| D0Sim | digitized |
| D0Reco | reconstructed |
| GenD0g | generated, simulated |
| GenD0gSim | generated, simulated, digitized |
| SimReco | digitized, reconstructed |
| D0GSimReco | simulated, digitized, reconstructed |
| Merge | merge |
| TMBAnalyze | root-tuple |
| MergeCafe | merge, root-tuple |
| GenCafe | generated, root-tuple |
| GenRecoCafe | generated, reconstructed, root-tuple |
'relocate': CaseInsensitiveDictionary({
'copymode' : 'cp',
'targetdir' : '/sps/d0/user',
'tasks' : 'reconstructed',
})
'samstoreoutput' : 'dst,thumbnail'
[sam] samtemplocation = <node>:<location> destinationpath = <stationnode>:<durablelocation> stationname = <station> samusername = <user>samtemplocation is a disk location which is accessible from the SAM station (ccin2p3-analysis).
If you are working with sam_client, forget the rest of this section and see next one.
There are necessary changes in your environment that you have to add in your Init.csh file
# DOMAIN_CAPABILITY_MAP = {
# 'fnal.gov' : [ 'local_cp', ],
# }
and redefine the SAM_CP_CONFIG_FILE to be yours by :
setenv SAM_CP_CONFIG_FILE MY-sam_cp_config.py
############################ Init.csh ####################
# to be sourced ( c-shell environment is assumed )
setup -j d0runjob <version>
setup -j sam_client <version>
source ${SAM_CLIENT_DIR}/etc/init_sam.csh ${SAM_CLIENT_DIR}
setenv SAM_CP_CONFIG_FILE ${SAM_CLIENT_DIR}/etc/sam_cp_config.py
# to import sam_cp_config
setenv PYTHONPATH ${PYTHONPATH}:${SAM_CLIENT_DIR}/etc
# Place the local directory where the paramFile sits in the PYTHONPATH.
setenv PYTHONPATH ${PYTHONPATH}:.
.....
.....
(The -j option in the setup command is to ignore the package dependencies.
D0runjob assumes that there is no '_' in any field except the <prod-decay-model> one.
<prefix>_<release>_<NumEv-xxxx>_<prod-decay-model>_<SAMgroup>_<phase>_<facilityname>_<requestid>_<generalizedtimestamp>
<prefix> : pythia, alpgenpythia, d0gstar, d0sim, tmb, dst,....
<release> : D0RunII release version for the task
<NumEv-xxxx> : xxxx is the no of events contained in the file
<prod-decay-model> : a string build from the production and decay parameter values in MCrequest used to get also the cardfile name
<SAMgroup> : 'dzero' is the default
<phase> : at present 'mcp17' or 'mcp20' to differentiate RunIIa and RunIIb production
<facilityname> : site where the job is run (in practice --site= d0runjob argument) Neither column nor underscore are allowed
<requestid> : for private production, the username is used instead of the requestid
<generalizedtimestamp> : a unique number built from the IP number of the working node and time stamp (defined by d0runjob)
Since the number of '_' is changing in the production and decay fields ( some cardfile builder forgot
also about conventions), d0runjob assumes simply that there are :
3 '_' preceeding the <prod-decay-model> string
5 '_' following it
to decode an input filename.
With version v07-10-21, it is thus possible to merge generated files. Those files should be produced with a datatier = unmerged-generated (a parameter to add in the paramFile). It is also possible to choose the prefix for the merged file (instead of the MERGE prefix chosen by default), using the parameter OutputFilePrefix (see also [->File naming convention]).
To perform this task, one has to provide as input the list of files to merge (parameter --inputfilelist= and the list of the metadata files which described those files (parameter --mdinputfilelist=. A metadata file is produced for each event file. Its name is identical to the event filename with a '.py' extension; it is to be found in the task subdirectory of the returned tarball in copybackDir directory. The merge task gets its parameters from the merge dictionary in the paramFile, as illustrated below:
'merge' : CaseInsensitiveDictionary({
'appfamily' : 'merge',
'appname' : 'merge',
'appversion' : 'p20.09.02',
'd0release' : 'p20.09.02',
'MergeSamMetadata' : 'thumbnail',
# 'OutputFilePrefix' : '',
'BasicScript' : 'copyd0om.py',
'UseMaxopt' : '1',
}),
To check that you did not forget any parameter in your paramFile, in particular the parameters needed to edit the cardfile you have chosen. The generator cardfile as used by the task will appear in the job generated sub-directory in the $JOBCACHE directory. The DEBUG loglevel may also be useful.
d0_runjob --check --jobtype=Gen --paramFile=<your_paramFile> --loglevel=DEBUG
To run interactively the generation step only on a few events:
d0_runjob --run --jobtype=Gen --paramFile=<your_paramFile> --nEvents=9
To submit in batch a job to generate/simulate/digitize/reconstruct events:
d0_runjob --site=ccin2p3 --paramFile=<your_paramFile> --jobtype=MCchain --nEvents=250 --jobname=myjob-001 --copybackDir=`pwd`/MyProd --submit='-l platform=LINUX,T=200000,M=500MB'
To submit a d0gstar job on a generated file:
d0_runjob --site=ccin2p3 --paramFile=<your_paramFile> --inputfilename=</a/generated/file/withlotsofevents> --skiprecords=0 --nEvents=250 --jobtype=D0Gstar --jobname=d0gstar-001 --copybackDir=`pwd`/MyProd --submit='-l platform=LINUX,T=200000,M=500MB'
To submit all 40 d0gstar/sim/reco jobs on a generated file with 10Kevents:
D0runjobs --site=ccin2p3 --paramFile=<your_paramFile> --inputfilename=</a/generated/file/withlotsofevents>
--nEvents=250 --jobtype=D0GSimReco --jobprefix=d0gsimreco
--copybackDir=`pwd`/MyProd --submit='-l platform=LINUX,T=200000,M=500MB'
--repeat=40
[->contents]
Yes, simply edit the paramFile and add in the dictionary part of the task the line :
'customexecutable':'/full/path/to_my_private/Executable',
'Customexecutable' is a better variable to use than 'Executable' as its value will appear in the metadata; its usage will be enforced in a future release.
Of course, this executable has to be built with the release version specified in the dictionary.
Customization has been extended starting from v07-10-16.
If the processing task will run several customized executables ( i.e. with mcatnloherwig ) then
one has to use the variable 'UserExeDir' in the "global" part of the paramFile
'userexedir' : '/full/path/to_my_private_executables',
This directory will be searched first for the existence of the executables.
This method works only if in the task(s) the executable name(s) is (are) the official one(s). However, it can be used in connection with the 'CustomExecutable' variable, leaving out the directory path, as in :
'customexecutable' : 'MyExecutable',
Yes, since version v07.09.11 (however full functionality only from v07-09-19).
You have to add in the task section of the paramFile where private RCPs are needed, the following 2 parameters :
Since v07-10-07 rcpcustomlocations is not needed.
Since v07-10-08 all RCPs in the <rcpcustomdir> directory will be copied to the job work area ( the usual RCP directory structure is assumed).
Since version v07-09-11, a new run number is automatically attributed when a generator task is submitted. The counter is by default saved in the file named 'run.number' in your home directory.
It is also possible with new options of d0_runjob :
To select the stream to be stored in SAM (when this is allowed ), one has to provide a 'samstoreoutput':'value' input in the task section of the request or of the paramFile. The value is mapped to the package and RCP names of the 'WriteEvent' RCP file. The useful and present mapping is :
package RCP stream
------------ ------------- -----------
d0_mcpp_gen MCWriteEvent generated
simpp simWriteEventFull d0gstar
d0sim D0SimWriteRDC d0sim
d0sim D0SimWriteRDC_keepCAEP d0sim
d0reco D0recoWriteEvent dst
thumbnail tmbWriteEvent thumbnail