How to use the clusters

From LHEP Wiki
Revision as of 14:06, 29 May 2017 by Lhep (talk | contribs)
Jump to navigation Jump to search
  • The AEC clusters can only be accessed via Grid tools. There is no local login on to the clusters for direct batch submission.
    The cluster endpoints are: ce01.lhep.unibe.ch, ce02.lhep.unibe.ch


  • Prerequisite: you need a valid user eScience certificate in order to access the Grid. For all Swiss institutes, the certificate issuing and yearly renewal is handled by the Science IT Support unit at the University of Bern. Follow the procedure detailed here. The procedure creates a file userkey.pem. This is your private key, keep it safe.


  • You will receive the signed certificate by email from SEE-GRID CA ithin 4 working days. The certificate itself is an attachment to the email: <serial nr>.pem. Copy it to your home directory and change the name to usercert.pem. Put both files in the following directory (create it if it isn't there already) and set the correct permissions:
           mkdir $HOME/.globus
           chmod go-rx $HOME/.globus

           chmod 400 $HOME/.globus/userkey.pem 
           chmod 600 $HOME/.globus/usercert.pem

If the $HOME/.globus directory holding the certificate resides in an afs home-directory (e.g. lxplus), the directory has to be further secured using afs-tools in addition to set the normal unix file access permissions. 

           fs setacl -dir $HOME/.globus -acl system:anyuser l

The user's Grid certificate/key pair (usercert.pem and userkey.pem) can be copied to any other machine to access the Grid simply by copying the $HOME/.globus directory. The security measures described above have to be repeated.


  • Load the certificate on to your browser. This will allow you to access any web-based resources accessible only to grid users. Run the following command:
           openssl pkcs12 -export -in usercert.pem -inkey userkey.pem -out mycert.p12

The command creates the bundle mycert.p12 which can be imported into your browser (ans also mail client, OS keychain, etc.). Procedure for importing the .p12 bundle differ from browser to browser. E.g. with Firefox: Preferences->Advanced->Certificates->View certificate->Import. Safari makes use of the Mac OS keychain.


  • Optional: restore a backup of your certificate. You can recreate the certificate/key pair from the .p12 bundle. Export the bundle from your browser and run the following commands:
           openssl pkcs12 -in mycert.p12 -clcerts -nokeys -out $HOME/.globus/usercert.pem
           openssl pkcs12 -in mycert.p12 -nocerts -out $HOME/.globus/userkey.pem


  • Enroll in the appropriate Virtual Organisation (VO) for your experiment, by visiting their VOMS (Virtual Organization Management Service) server. On the AEC clusters, the following experiments are currently supported

atlas - VOMS server

fermilab - VOMS server

t2k.org - VOMS server



Make sure you have a file ~/.voms/vomses with this line inside (create it with your editor): "aec" "voms.lhep.unibe.ch" "15027" "/DC=com/DC=quovadisglobal/DC=grid/DC=switch/DC=hosts/C=CH/ST=Bern/L=Bern/O=Universitaet Bern/CN=voms.lhep.unibe.ch" "aec"

Install your submission client, the ARC (Advanced Resource Connector), documentation can be found in [1]. This gives you the arc tools like arcsub, arcls, arcstat, arcget etc. With those you can submit and manage your jobs.

To set up ARC (standalone) choose the needed configuration from [2] and download it. Then, for linux, extract and source: 

     # tar -xvf nordugrid-arc-standalone-<your version>.tgz
     # cd nordugrid-arc-standalone-<your-version>
     # . ./setup.sh

The ARC environment should be set, you can make a proxy (default valid 12 hours) 

     # arcproxy --voms aec
      Type your password:

To test it (other clusters at UNIBE are ce01.lhep.unibe.ch and nordugrid.unibe.ch) : 

     #arctest -c ce01.lhep.unibe.ch -J 1

To check your job go to: [3]. Now describe a real job in the xrsl language, submit and retrive it: 

    # arcsub -c ce.lhep.unibe.ch myjob.xrsl (returns a job identifier gsiftp://.... if everything is ok)
    # arcget gsiftp://..

Here an xrsl file example (documentation here : http://www.nordugrid.org/documents/xrsl.pdf): 

  $&(executable=myjobscript.sh)
  (inputfiles=
  (myjob.exe myjob.exe)
  )
  (* comments within stars *)
  (outputfiles=("/" ""))
  (jobname=MyVeryFirstJob)
Retrieved from "https://wiki.lhep.unibe.ch/index.php?title=How_to_use_the_clusters&oldid=627"