Using D0 data at CCIN2P3

This page is meant be to a simple introduction to the handling of D0 data at the IN2P3 computing center. I will focus on the use of root-tuple / root-trees or thumbnail data.

1. The mass storage: HPSS

D0 does not have any large disk space avaible at CCIN2P3, so all of the data are resident on tape and access via HPSS. Please make sure to read the FAQ before using HPSS.

HPSS presents to the user data files as files in a (virtual) filesystem. Each user have a home space area under the "directory" ccmdrs11:/hpss/in2p3.fr/home/u/username (with a naming convention similar to that of the AFS home directories). Experiment data are put under different directories ( cchpssd0:/hpss/in2p3.fr/group/d0 and directories specific to the ccin2p3-analysis SAM station) in which most D0 users are NOT allowed to write. In addition to the directory, one has to specify the HPSS server that handles the data: ccmdrs11 for home directories and cchpssd0 for D0 data at the time of writing.

HPSS defines Classes Of Service (COS) that can be set for each file. In the past (pre-Aug 2007), COS 21 and 22 were used for Dzero, this is no longer the case and COS 0 should be used.

Although the data files appear to be in directories, they are not readily available via the usual UNIX access methods. Data are accessed via RFIO.

2. Accessing files in HPSS via RFIO

RFIO is a protocol that allows to acces files on a remote machine (an RFIO server). It has been initially developped at CERN as part of the shift software. It is use at CCIN2P3 to access files in HPSS.

RFIO is a C/C++ API that can be used in e.g. experiment software but it also defines a set of commands: rfdir, rfmkdir, rfchmod, rfrename, rfrm, rfstat, rfcp (see the man pages for details). rfcp can be use to retrieve files from HPSS:

 rfcp ccmdrs11:/hpss/in2p3.fr/home/d/duflot/lostjets.reco lostjets.reco 

It can also be used to write a file to HPSS, but make sure to set the COS of the file by setting the RFCP_HPSSCOS environment variable to the COS value. Note that rfdir is slow and is NOT the right way to find out what's in HPSS. You should keep track of what's in your area separately. Most of the official D0 data files are declared to SAM. The recommended way to create a list of file in HPSS from a dataset definition or dimensions is

setup sam
listCCFiles.py --datasetdef CSskim-3LEP-PASS2-p14.06.01

That will create a file called CSskim-3LEP-PASS2-p14.06.01.allfiles that contains the full path in HPSS of files corresponding to the definition. If there are files corresponding to the definition that are not at CCIN2P3, they will be shown on the standard output. If you don't have a dataset definition, you can use dimensions (as in sam translate constraints / sam list files) with the --dimension option. Note: in the past, it was suggested that a simple sam translate constraints would list the files in HPSS, however it would not give the HPSS directory and files are no longer stored in a single directory since we are using the SAM station at CCIN2P3 to retrieve files.
Due to the specific structure and access patterns of root files and the large buffer used by RFIO, accessing root-tuples directly in HPSS is slow. It is recommended to copy the file to the worker disk via rfcp or to use xrood.
1. Accessing files in the framework

Since p14, the D0 I/O package supports the idea of a stager command and automatically invoque staging for filenames of the form server:path. By default, when such a filename is encountered, the io package will try to copy the file on the current directory with the "rcp" command (i.e. do "rcp server:file ."). The default staging directory in <io_packages ReadEvent> is the current directory which is fine for batch jobs. You would need to change that for interactive jobs to, say, /scratch/username. If you're not using that RCP, please check the staging directory used. The stager command to be used if rfcp or better $THRONG_DIR/scripts/D0rfcp that handles retries (necessary when HPSS is under high load). You can specify the stager command on the command line for framework jobs via the -stager option. The advantage of using this staging method is that rfcp is able to use load-balancing of HPSS servers, so we will benefit from improved HPSS performances.

2. Accessing files in cafe

To access CAF tree with cafe, the method is very similar to that of the framework:

• specify your file list obtained with listCCFiles.py as stageinlist:my.list, the keywork stageinlist will trigger cafe to try and copy each file before using it.
• Use the following options in your config file or on the command line

cafe.Stager: 		/afs/in2p3.fr/home/throng/d0/scripts/D0rfcp  
cafe.StagingDirectory: 	.
That specify the method and target directory for cafe to copy the files. Of course, you can change the StagingDirectory to be another directory.

3. Accessing files in /sps

There are currently two GPFS areas dedicated to d0: /sps/d0 and /sps/d0usr. However, GPFS is not very efficient to deliver files to multiple batch jobs. You may try to use the stager as described above but leaving /bin/cp (or the default) as cafe.Stager

4. Common Sample Group skims at CCIN2P3
Some of the recent CS skims are imported at CCIN2P3:
• Up-to-date skims to analyze NP and 2EMhighpt in 2009
• CSG_CAF__PASS3_p18.13.01
• CSG_CAF__PASS2_p21.10.00
• CSG_CAF__PASS4_p21.10.00_p20.12.00
• CSG_CAF__PASS4_p21.10.00_p20.12.01
• CSG_CAF__PASS4_p21.10.00_p20.12.02
• CSG_CAF__PASS4_p21.10.00_p20.12.04
• CSG_CAF__PASS4_p21.12.00_p20.12.05

One can create lists of files in HPSS for the CSG skims as explained above.

It happens that files are declared bad after they've been put in SAM and some of them could have been transfered to CCIN2P3 before they were declared bad. It's always good to re-generate your file list once you know that a given dataset is frozen (e.g. an official dataset for conferences) just to make sure you don't have bad files in your list. On a best effort basis, we're trying to physically remove the files from HPSS if we know that they've been declared bad, so it's possible that a file was present in your file list and suddenly seems to be have disapeared from HPSS.



5. D0, HPSS and the batch system

In order to better optimize access to D0 data in HPSS and make sure that your jobs are submitted only when HPSS is available, you should use the following option of qsub to submit your jobs :

• -l u_hpps_d0_stress for short jobs (e.g. less than 200000 units) or jobs that do a lot of I/O with HPSS
• -l u_hpss_d0 for other jobs
Since April 2007, users are asked to specify the resource -l u_users_d0 when submitting jobs (except for official D0 production jobs).
6. Importing files in HPSS via bbftp

Although one can import files in HPSS by copying them to a disk scratch area and put them in HPSS via an rfcp, this is no very efficient. The recommended way is to use bbftp to put files directly in HPSS.

On clued0, the bbftp client is available as

/home/duflot/Linux/bbftp

This is the version of bbftp that implements a "passive mode" like the one for ftp and thus is not blocked by the clued0 firewall. bbftp needs a command file like

setoption remoterfio
setremotecos 0
put /work/thamber-clued0/duflot/lostjets.reco cchpssd0:/hpss/in2p3.fr/home/d/duflot/lostjets.reco

Don't forget to set the correct COS. You can launch the transfer by

bbftp -i bbftp.command -u duflot -p 3 -m ccbbftp.in2p3.fr

where 3 streams are used to transfer data. This is a good value for the current capacity of the link between CCIN2P3 and FNAL, the maximum being around 5. bbftp will ask for your password at CCIN2P3. For an automatic procedure, one can also use RSA keys for autentification with the ccbbftp machine. You should
1. generate a key on the machine you want to transfert from:

   d0mino> cd ~/.ssh
   d0mino> ssh-keygen -t rsa -f identity.d0mino.bbftp   (return for password)
     

2. Copy the public key (e.g. identity.d0mino.bbftp.pub) to your public area at CCIN2P3
3. Send a mail to user.support@in2p3.fr explaining that you want to transfer data from machine xxx with RSA key autentification and point to your public key file. System administrators will have to install this key on the bbftp server. This key cannot be used for anything alse but launching the bbftp server on ccbbftp from machine xxx.