DPM SE how-to
Jump to navigation
Jump to search
Introduction
This page describes how to work with the Disk Pool Manager Storage Element of the Bern ATLAS T2 cluster UNIBE-LHEP
Resources available
Status as of September 2013:
| Space Token | Reserved size | Path | Comment |
|---|---|---|---|
| UNIBE-LHEP_DATADISK | 350 TB | /dpm/lhep.unibe.ch/home/atlas/atlasdatagroupdisk/ | DDM endpoint in the ND cloud - Managed by DDM only, Authorized FQANs: atlas/Role=production |
| UNIBE-LHEP_LOCALGROUPDISK | 71.78 TB | /dpm/lhep.unibe.ch/home/atlas/atlaslocalgroupdisk/ | DDM endpoint in the ND cloud - Managed by DDM only, Authorized FQANs: atlas/Role=production, atlas/ch |
| UNIBE-LHEP_SCRATCHDISK | 14.65 TB | /dpm/lhep.unibe.ch/home/atlas/atlasscratchdisk/ | DDM endpoint in the ND cloud - Managed by DDM only, Authorized FQANs: atlas |
| No token area for other VOs | 21.93 TB | /dpm/lhep.unibe.ch/home/<vo name> | Authorized FQANs: aec, atlas/Role=NULL, crypto, dteam, life, ltpc, ops, ops/NGI, ops/NGI/Germany, ops/NGI/Switzerland, ops/ROC, ops/ROC/GermanySwitzerland, ops/Role=lcgadmin, smscg, t2k.org, tutor |
| ... | ... | ... | ... |
Setting up your environment
- consider adding the following to your bash profile
source /terabig/atlsoft/LocalSoft/glite/glite-UI-current/external/etc/profile.d/grid-env.sh && unset GT_PROXY_MODE export DPM_HOST=dpm.lhep.unibe.ch export DPNS_HOST=dpm.lhep.unibe.ch
- you need your standard grid credentials for the appropriate VO, either atlas or ltpc
voms-proxy-init -voms atlas:/atlas/ch voms-proxy-init -voms <vo name>
- if down the line you get errors about libshift and/or libRFIO, these should be cured by prepending the following to your PATH (this is certainly needed if opening files directly in ROOT or Athena):
export LD_LIBRARY_PATH=/terabig/atlsoft/LocalSoft/glite/glite-UI-current/lcg/lib:${LD_LIBRARY_PATH}
For first time users
- some useful information and a tutorial are found here:
https://svnweb.cern.ch/trac/lcgdm/wiki/Dpm https://svnweb.cern.ch/trac/lcgdm/wiki/Dpm/User/Tutorial
- if you plan to use the non token space (for e.g. quicker interactive access) you will need most of what is covered in the tutorial. Please clean up after yourself after creating test files, dirs, etc. For any other work, it is highly advised to use the LOCALGROUPDISK and run in grid fashion.
Getting and accessing data on to/from the DPM SE
UNIBE-LHEP_DATADISK space token
- Pledged area, no local users allowed
UNIBE-LHEP_LOCALGROUPDISK space token
- DDM subscription via Data Transfer Request Interface (DaTRI): https://twiki.cern.ch/twiki/bin/view/Atlas/DataTransferRequestInterface
- You need an account from the Panda web page, which is not the same as any account you already have at CERN
- Request that from a browser that has your certificate loaded
- Once registered, follow this link to the data subscription page: http://pandamonitor.cern.ch:25980/server/pandamon/query?mode=ddm_req
- To use the data in Grid jobs, choose ND as cloud and UNIBE-LHEP_LOCALGROUPDISK as destination site
- NOTE: all files in the space token area can also be accessed directly with all tools listed below, provided the users finds their way into the directory tree. However: please refrain from deleting and moving individual files from the token area, as this will make the entire dataset not usable and create nasty inconsistencies in the DDM catalogues.
Non token space
- You can use this space to store both DDM data (via dq2) and any other file (mainly via GFAL/lcg-* utils or RFIO/rf* clients. For convenience, one can also use the dpns-* commands as well, but these work on the namespace only and do not perform data I/O) - *NOTE*: Non token space for ATLAS has been basically discontinued. Only <100GB of space are available to store small files or testing.
dq2
- When to use dq2-get or DDM subscriptions: https://twiki.cern.ch/twiki/bin/view/Atlas/DQ2ClientsHowTo#When_to_use_dq2_get_or_DDM_subsc
- You're all familiar with dq2: https://twiki.cern.ch/twiki/bin/view/Atlas/DQ2ClientsHowTo
- A nice and comprehensive tutorial on DDM/dq-2: http://indico.ific.uv.es/indico/getFile.py/access?contribId=3&resId=0&materialId=slides&confId=207
dpns-*
- These commands will not allow you perform any I/O, but only interact with the name-space, namely creating/deleting directories, list, rename file/dirs, change permissions and so on. Sometimes it is convenient to use some of these commands.
- The commands available are the following (see man pages for full details):
dpns-chgrp dpns-entergrpmap dpns-listgrpmap dpns-ls dpns-modifyusrmap dpns-rm dpns-setacl dpns-chmod dpns-enterusrmap dpns-listusrmap dpns-mkdir dpns-ping dpns-rmgrpmap dpns-chown dpns-getacl dpns-ln dpns-modifygrpmap dpns-rename dpns-rmusrmap
- Usage examples
# dpns-ls /dpm/lhep.unibe.ch/home/atlas/user/gsciacca # /ngse3/data/data10_7TeV/aod/jettauetmiss > dpns-mkdir /dpm/lhep.unibe.ch/home/atlas/user/gsciacca/testdir # /ngse3/data/data10_7TeV/aod/jettauetmiss > dpns-ls /dpm/lhep.unibe.ch/home/atlas/user/gsciacca testdir # dpns-mkdir -p /dpm/lhep.unibe.ch/home/atlas/user/gsciacca/testdir/testdir1/testdir2/testdir3 # dpns-ls -R /dpm/lhep.unibe.ch/home/atlas/user/gsciacca/ /dpm/lhep.unibe.ch/home/atlas/user/gsciacca/: testdir /dpm/lhep.unibe.ch/home/atlas/user/gsciacca//testdir: testdir1 /dpm/lhep.unibe.ch/home/atlas/user/gsciacca//testdir/testdir1: testdir2 /dpm/lhep.unibe.ch/home/atlas/user/gsciacca//testdir/testdir1/testdir2: testdir3 /dpm/lhep.unibe.ch/home/atlas/user/gsciacca//testdir/testdir1/testdir2/testdir3:
-p With this option, rfmkdir creates dirname by creating all the
non-existing parent directories first.
# dpns-rm -r /dpm/lhep.unibe.ch/home/atlas/user/gsciacca/testdir
-r Recursive mode.
- Be warned: none of the remove/delete commands ever ask for confirmation!
RFIO
- Please use XROOTD if working with ROOT files
- This is as close as it gets to interacting with data as if the storage were mounted locally as in NFS
- The commands available are the following (see man pages for full details):
rfcat rfchmod rfcomm rfcp rfdf rfdir rfmkdir rfrename rfrm rfstat rftp
- Usage examples
# rfdir /dpm/lhep.unibe.ch/home/atlas/user drwxrwxr-x 1 110 109 0 Apr 06 11:24 kruker # rfmkdir /dpm/lhep.unibe.ch/home/atlas/user/gsciacca # rfcp testfile.txt /dpm/lhep.unibe.ch/home/atlas/user/gsciacca 141 bytes in 1 seconds through local (in) and bond0 (out) (0 KB/sec) # rfdir /dpm/lhep.unibe.ch/home/atlas/user/gsciacca -rw-rw-r-- 1 109 109 141 Apr 28 14:34 testfile.txt # rfcat /dpm/lhep.unibe.ch/home/atlas/user/gsciacca/testfile.txt ************************************ hostname: lheppc51.unibe.ch date: Thu Apr 28 11:49:14 CEST 2011 ************************************ # rfcp /dpm/lhep.unibe.ch/home/atlas/user/gsciacca/testfile.txt testfile1.txt 141 bytes in 0 seconds through bond0 (in) and local (out) # md5sum testfile.txt b547254b9b53c0916c3865a5aa825107 testfile.txt # md5sum testfile1.txt b547254b9b53c0916c3865a5aa825107 testfile1.txt # rfrm /dpm/lhep.unibe.ch/home/atlas/user/gsciacca/testfile.txt
- Please note that rfcp has no useful options, least it does recursive copy, and does not understand wildcards, so you'll need something clever to achieve that, like bash loops and sometimes use the dpns-* commands for convenience:
# for x in `ls /terabig/gsciacca/tmp/`; do rfcp /terabig/gsciacca/tmp/${x} /dpm/lhep.unibe.ch/home/atlas/user/gsciacca/${x}_copy; done
956780 bytes in 0 seconds through local (in) and bond0 (out)
441795 bytes in 0 seconds through local (in) and bond0 (out)
438527 bytes in 1 seconds through local (in) and bond0 (out) (428 KB/sec)
126913 bytes in 1 seconds through local (in) and bond0 (out) (123 KB/sec)
114967 bytes in 1 seconds through local (in) and bond0 (out) (112 KB/sec)
# rfdir /dpm/lhep.unibe.ch/home/atlas/user/gsciacca/
-rw-rw-r-- 1 109 109 956780 Apr 28 15:32 tempfile1_copy
-rw-rw-r-- 1 109 109 441795 Apr 28 15:32 tempfile2_copy
-rw-rw-r-- 1 109 109 438527 Apr 28 15:32 tempfile3_copy
-rw-rw-r-- 1 109 109 126913 Apr 28 15:32 tempfile4.root_copy
-rw-rw-r-- 1 109 109 114967 Apr 28 15:32 tempfile5.root_copy
# dpns-ls /dpm/lhep.unibe.ch/home/atlas/user/gsciacca/
tempfile1_copy
tempfile2_copy
tempfile3_copy
tempfile4.root_copy
tempfile5.root_copy
# for x in `dpns-ls /dpm/lhep.unibe.ch/home/atlas/user/gsciacca/`; do rfrm /dpm/lhep.unibe.ch/home/atlas/user/gsciacca/${x}; done
GFAL
- These are higher-level clients that allow you more sophisticated control over data, both DDM data and your local files. For example you can use an LFC to give your files unique LFN (Logical File Name), create replica of files using their LFN, etc without having to worry about the actual path to the file, if you wish so. You can also interact with files in the Space Token area if you need to. The syntax of these commands tends to be a bit complex.
- The commands available are the following (see man pages for full details)
lcg-aa lcg-get-checksum lcg-job-monitor lcg-ls lcg-rf lcg-bringonline lcg-getturls lcg-job-status lcg-ManageVOTag lcg-sd lcg-cp lcg-gt lcg-la lcg-ra lcg-stmd lcg-cr lcg-info lcg-lg lcg-rep lcg-tags lcg-del lcg-infosites lcg-lr lcg-replica-manager lcg-uf
- Commands to interact with the LHC File Catalog are (see man pages for full details)
lfc-chgrp lfc-delcomment lfc-getacl lfc-ln lfc-modifygrpmap lfc-rename lfc-rmusrmap lfc-chmod lfc-entergrpmap lfc-listgrpmap lfc-ls lfc-modifyusrmap lfc-rm lfc-setacl lfc-chown lfc-enterusrmap lfc-listusrmap lfc-mkdir lfc-ping lfc-rmgrpmap lfc-setcomment
- Usage examples
- copy, list and delete of a single file to a pre-selected destination (not using the LFC)
# lcg-cp /terabig/gsciacca/tmp/tempfile1 srm://$DPNS_HOST:8446/srm/managerv2?SFN=/dpm/lhep.unibe.ch/home/atlas/user/gsciacca/tempfile1 # lcg-ls srm://$DPNS_HOST:8446/srm/managerv2?SFN=/dpm/lhep.unibe.ch/home/atlas/user/gsciacca/ /dpm/lhep.unibe.ch/home/atlas/user/gsciacca//tempfile1 # lcg-cp srm://$DPNS_HOST/dpm/lhep.unibe.ch/home/atlas/user/gsciacca/tempfile1 /terabig/gsciacca/tmp/tempfile_copy # md5sum tempfile1 tempfile_copy cb8432b25b2ad8eda801f36ed6c0c53d tempfile1 cb8432b25b2ad8eda801f36ed6c0c53d tempfile_copy # lcg-del -l srm://$DPNS_HOST/dpm/lhep.unibe.ch/home/atlas/user/gsciacca/tempfile1
- To see a bit more of what's happening:
-v,--verbose
Verbose mode. You can specify it twice for extra verbose mode.
- Please note:
- lcg-* commands deal with single files only, no recursive, no wildcards, no directories
- lcg-del will NOT prompt for confirmation
- Please note:
- LFC is no longer used in central ATLAS DDM operations, the service will be retired in the near future, so the following section is practically obsolete for ATLAS
- To use the LFC you will need to set the following
# export LFC_HOST=prod-lfc-atlas-central.cern.ch
- Create your personal path on the LFC. For convenience you can choose to store that path into a convenient env variable: this will shorten your LFNs, as the path is automatically prepended to it
# lfc-mkdir -p /grid/atlas/gsciacca # export LFC_HOME=/grid/atlas/gsciacca # lfc-ls -l $LFC_HOME # lfc-ls -l #equivalent to the above since you have defined LFC_HOME
- copy-register, list, copy back and delete of a single file using the LFC
# lcg-cr -v -d dpm.lhep.unibe.ch -l lfn:file1 /terabig/gsciacca/tmp/tempfile1
Using grid catalog type: lfc
Using grid catalog : prod-lfc-atlas-central.cern.ch
Checksum type: None
SE type: SRMv2
Destination SURL : srm://dpm.lhep.unibe.ch/dpm/lhep.unibe.ch/home/atlas/generated/2011-05-13/file64a236bf-fb72-40ab-8672-fde99eb75f47
Source SRM Request Token: 7b28883c-0746-4815-a136-d46219228541
Source URL: file:/terabig/gsciacca/tmp/tempfile1
File size: 956780
VO name: atlas
Destination specified: dpm.lhep.unibe.ch
Destination URL for copy: gsiftp://dpmdisk02.lhep.unibe.ch/dpmdisk02.lhep.unibe.ch:/mnt/storage3/atlas/2011-05-13/file64a236bf-fb72-40ab-8672-fde99eb75f47.22305.0
# streams: 1
956780 bytes 4328.18 KB/sec avg 4328.18 KB/sec inst
Transfer took 1010 ms
Using LFN: lfn:/grid/atlas/gsciacca/file1
Using GUID: guid:ee16c60f-11a8-4b69-b844-0b1b2e2e3efa
Registering LFN: /grid/atlas/gsciacca/file1 (ee16c60f-11a8-4b69-b844-0b1b2e2e3efa)
Registering SURL: srm://dpm.lhep.unibe.ch/dpm/lhep.unibe.ch/home/atlas/generated/2011-05-13/file64a236bf-fb72-40ab-8672-fde99eb75f47 (ee16c60f-11a8-4b69- b844-0b1b2e2e3efa)
guid:ee16c60f-11a8-4b69-b844-0b1b2e2e3efa
# lcg-lr lfn:file1 srm://dpm.lhep.unibe.ch/dpm/lhep.unibe.ch/home/atlas/generated/2011-05-13/file64a236bf-fb72-40ab-8672-fde99eb75f47
# lcg-cp lfn:file1 /terabig/gsciacca/tmp/tempfile2 # md5sum /terabig/gsciacca/tmp/tempfile1 /terabig/gsciacca/tmp/tempfile2 cb8432b25b2ad8eda801f36ed6c0c53d /terabig/gsciacca/tmp/tempfile1 cb8432b25b2ad8eda801f36ed6c0c53d /terabig/gsciacca/tmp/tempfile2
# lcg-del -a -v lfn:file1 VO name: atlas SE type: SRMv2 [lfn:/grid/atlas/gsciacca/file1] srm://dpm.lhep.unibe.ch/dpm/lhep.unibe.ch/home/atlas/generated/2011-05-13/file64a236bf-fb72-40ab-8672-fde99eb75f47 - DELETED [guid:ee16c60f-11a8-4b69-b844-0b1b2e2e3efa] srm://dpm.lhep.unibe.ch/dpm/lhep.unibe.ch/home/atlas/generated/2011-05-13/file64a236bf-fb72-40ab-8672-fde99eb75f47 - UNREGISTERED [guid:ee16c60f-11a8-4b69-b844-0b1b2e2e3efa] lfn:/grid/atlas/gsciacca/file1 - UNREGISTERED
XROOTD
The xrootd protocol has been demonstrated to be considerably faster for direct data access than RFIO. In addition, one can open directly files from ROOT
Setup CVMFS first
# export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
# source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh
Setup XROOTD and copy files
# localSetupEmi # localSetupXRootD
- Commands available are the following:
xrd xrdb xrdcp xrdfs xrdpwdadmin xrdstagetool xrdadler32 xrdcopy xrdcp-old xrdgsiproxy xrdsssadmin
# voms-proxy-init --voms atlas:/atlas/ch # xrdcp root://dpm.lhep.unibe.ch//dpm/lhep.unibe.ch/home/atlas/atlaslocalgroupdisk/rucio/user/lmarti/e3/07/user.lmarti.016146._00001.susyNt.root /tmp
Setup and open files from ROOT
# localSetupEmi # localSetupROOT
- Please note:
- Do NOT swap the two setups, otherwise it breaks (!)
# voms-proxy-init --voms atlas:/atlas/ch # root -l root://dpm.lhep.unibe.ch//dpm/lhep.unibe.ch/home/atlas/atlaslocalgroupdisk/rucio/user/fmeloni/4d/ce/user.fmeloni.004139._00001.fileout.strip1.root
ARC
ARC clients talk to SRMv2.2 SEs, such as the DPM. Set up your ARC environment first (after generating a proxy valid for your VO)
# cd /terabig/atlsoft/LocalSoft/nordugrid-arc/nordugrid-arc-standalone-0.8.1.1-2 && source setup.sh && cd
- Usage examples
- copy, list and delete of a single file to a pre-selected destination (not using the LFC)
# ngcp -d2 HelloWorld.jar srm://dpm.lhep.unibe.ch/dpm/lhep.unibe.ch/home/life/smscg_HelloWorld.jar # ngls srm://dpm.lhep.unibe.ch/dpm/lhep.unibe.ch/home/life # ng?? don't know this yet...
Advanced use
- NOTE: not sure whether this section is still relevant. Refer to the XROOTD section
In order to read files in from ROOT and Athena, you will need to pickup the appropriate version of libshift.so by prepending its location to your PATH:
export LD_LIBRARY_PATH=/terabig/atlsoft/LocalSoft/glite/glite-UI-current/lcg/lib:${LD_LIBRARY_PATH}
- NOTE: you will need to run a 32-bit version of ROOT
Open files from within ROOT
- Example
# source /terabig/atlsoft/LocalSoft/glite/glite-UI-current/external/etc/profile.d/grid-env.sh && unset GT_PROXY_MODE
# export DPM_HOST=dpm.lhep.unibe.ch
# export DPNS_HOST=dpm.lhep.unibe.ch
# voms-proxy-init -voms atlas:/atlas/ch
# export AtlasSetup=/afs/cern.ch/atlas/software/dist/AtlasSetup
# alias asetup='source $AtlasSetup/scripts/asetup.sh'
# cat .asetup
[defaults]
default32 = True
briefprint = True
testarea = "~/testarea"
multi = True
# asetup 16.6.0
# export LD_LIBRARY_PATH=/terabig/atlsoft/LocalSoft/glite/glite-UI-current/lcg/lib:${LD_LIBRARY_PATH}
# cat testSE.C
{
TFile* f = new TFile();
f -> Open("rfio:/dpm/lhep.unibe.ch/home/atlas/atlaslocalgroupdisk/group10/perf-jets/data11_7TeV/group10.perf-jets.data11_7TeV.00178109.physics_JetTauEtmiss.merge.AOD.f354_m765.rel166.v7.110329003509_EXT0/group10.perf-jets.17195_011959.EXT0._00001.qcd.root__DQ2-1301921925");
f -> ls();
f -> Close();
}
# root -b testSE.C
Open files from within Athena
- Example here...
DPM use guide for t2k.org
- Initialisation:
export DPM_HOST=dpm.lhep.unibe.ch export DPNS_HOST=dpm.lhep.unibe.ch export LFC_HOST=lfc.gridpp.rl.ac.uk voms-proxy-init --voms t2k.org
- List of files, e.g.:
lfc-ls /grid/t2k.org/nd280/production005/F/mcp/neut/2010-11-water/magnet/beamb/anal
- Check if file is present at LHEP, e.g.:
lcg-lr lfn:/grid/t2k.org/nd280/production005/F/mcp/neut/2010-11-water/magnet/beamb/anal/oa_nt_beam_90210004-0071_z3t2ik4agm77_anal_000_prod005magnet201011waterb-bsdv01.root
- if "dpm.lhep.unibe.ch" is in the list, the file is at LHEP
- Copy file to disk, e.g.:
lcg-cp srm://dpm.lhep.unibe.ch/dpm/lhep.unibe.ch/home/t2k.org/production005/F/mcp/neut/2010-11-water/magnet/beamc/anal/oa_nt_beam_90410010-0087_sfs3crv64jvl_anal_000_prod005magnet201011waterc-bsdv01.root local_file_name.root
Scripts for running Highland etc. on large data sets
- To create a Highland flat tree of an entire run, use this script: create_flat_tree_mass.sh
Using files in root directly (not recommended)
There are two ways to open files directly with root:
- Using RFIO (must be enabled while compiling root), e.g.:
root -l rfio:////dpm/lhep.unibe.ch/home/t2k.org/production005/F/mcp/neut/2010-11-water/magnet/beamc/anal/oa_nt_beam_90410010-0087_sfs3crv64jvl_anal_000_prod005magnet201011waterc-bsdv01.root
- Using GFAL (must be enabled while compiling root), e.g.:
root -l gfal:srm://dpm.lhep.unibe.ch/dpm/lhep.unibe.ch/home/t2k.org/production005/F/mcp/neut/2010-11-water/magnet/beamc/anal/oa_nt_beam_90410010-0087_sfs3crv64jvl_anal_000_prod005magnet201011waterc-bsdv01.root
Both methods are often quite slow compared to local file acces (2x to 10x, depending on situation). It is recommended to copy the file to local disk first using lcg-cp (see above).
Administrative commands
Do not use the following commands, unless you know exactly what you are doing.
- Copy file to LHEP, e.g.:
lcg-cp -v --vo t2k.org \ srm://t2ksrm.nd280.org/nd280data/production005/F/mcp/neut/2010-11-water/magnet/beamb/anal/oa_nt_beam_90210004-0071_z3t2ik4agm77_anal_000_prod005magnet201011waterb-bsdv01.root \ srm://dpm.lhep.unibe.ch/dpm/lhep.unibe.ch/home/t2k.org/production005/F/mcp/neut/2010-11-water/magnet/beamb/anal/oa_nt_beam_90210004-0071_z3t2ik4agm77_anal_000_prod005magnet201011waterb-bsdv01.root
- Do not use lcg-rep, as it will register the file with the t2k global catalog (which we are not supposed to do)!
- If you run commands in loops, add the following options to prevent it to hang from time to time:
--connect-timeout 120 --sendreceive-timeout 120 --bdii-timeout 120 --srm-timeout 120
- Delete a file ("-l" prevents removal from the catalog), e.g.:
lcg-del -l srm://dpm.lhep.unibe.ch/dpm/lhep.unibe.ch/home/t2k.org/production005/F/mcp/neut/2010-11-water/magnet/beamb/anal/oa_nt_beam_90210020-0010_neqobffjtosu_anal_000_prod005magnet201011waterb-bsdv01.root