AEC-LHEP Hardware information: Difference between revisions

From LHEP Wiki
Jump to navigation Jump to search
No edit summary
No edit summary
 
(89 intermediate revisions by the same user not shown)
Line 1: Line 1:


BERN ATLAS T2 2016 Clusters - ce0.lhep.unibe.ch, ce02.lhep.unibe.ch, dpm.lhep.unibe.ch, bdii.lhep.unibe.ch
BERN ATLAS Tier-2, Tier 3 - 2020 Systems


* Summary :
* Grid HT Cluster '''ce01''' (5.5k cores)
            '''ce01.lhep CLUSTER''':
** Node summary
            ce01:      2*2.40GHz Intel Xeon E5645 6-Core with 24GB RAM - ROCKS 6.2 and nordugrid-arc-5.0.3-1.el6.x86_64 (epel)
** Scheduler: SLURM 18.08.8-1.el7 (scheduler on ce01)
            mds-2-1:  1*2.40GHz Intel Xeon E5620 4-Core with 24GB RAM for Lustre MDS (lustre-2.5.3)
** Shared File System: Lustre 2.12.2-1.el7
            oss-2-x:  2*2.10GHz Intel E5-2620 v2 4-Core with 16GB RAM for Lustre OSS (lustre-2.5.3, 6 nodes each with 32*HDD, 31TB sw raid0)
*** 1 x MDS, 6 x OSS: 204 x 1T HDDs (184T)
            wn-2-x:
                      2*2.53GHz Intel Xeon E5540  8-core + HT (45 Nodes) 540 cores with 2GB RAM/core (or 720 cores with 1.33GB RAM/core)
                      2*2.53GHz Intel Xeon E5540  8-core + HT ( 4 Nodes)  32 cores with 2GB RAM/core (same HW as above with 1 dead CPU)
                      2*2.67GHz Intel Xeon X5650 12-core + HT ( 6 Nodes) 144 cores with 2GB RAM/core
                      2*2.60GHz Intel Xeon E5-2650 v2 8-core + HT (4 Nodes) 128 cores with 2GB RAM/core
                      2*2.40GHz Intel Xeon E5-2630 v3 8-core + HT (8 Nodes) 256 cores with 2GB RAM/core


            '''ce02.lhep CLUSTER''':
* Grid Compute Elements '''ce01.lhep.unibe.ch, ce02.lhep.unibe.ch, ce03.lhep.unibe.ch'''
            ce02       2*2.40GHz Intel Xeon E5620 4-core with 24GB RAM - ROCKS 6.2 and nordugrid-arc-5.0.4-1.el6.x86_64 (epel)
ce01.lhep.unibe.ch has address 130.92.140.201
            mds-0-1   1*2.40GHz Intel Xeon E5620 4-core with 24GB RAM for Lustre MDS (lustre-2.5.3-2)
ce02.lhep.unibe.ch has address 130.92.140.202
            oss-0-x   2*2.30GHz AMD Opteron 2376 4-core with 16GB RAM for Lustre OSS (lustre-2.5.3, 11 nodes each with 1*HD 2TB)
ce03.lhep.unibe.ch has address 130.92.140.203
            wn-0-x:
* Grid Storage Element '''dpm.lhep.unibe.ch ''' (2PB)
                      2*2.30GHz AMD Opteron 2376 4-core (15 Nodes) 120 cores with 2GB RAM/core
dpm.lhep.unibe.ch has address 130.92.140.100
                      4*2.30GHz AMD Opteron 8356 4-core (20 Nodes) 320 cores with 2GB RAM/core
dpm.lhep.unibe.ch has IPv6 address 2001:620:400:fea8::100
                      1*2.50GHz Intel Xeon E5420 4-core + HT (47 Nodes) 376 cores with 2GB RAM/core
* Grid Information System '''bdii.lhep.unibe.ch '''
# host bdii.lhep.unibe.ch
  bdii.lhep.unibe.ch has address 130.92.140.121
   bdii.lhep.unibe.ch has address 130.92.140.122
* Squid for CVMFS '''cvmfs.lhep.unibe.ch'''
# host cvmfs.lhep.unibe.ch
  cvmfs.lhep.unibe.ch has address 130.92.252.18
   cvmfs.lhep.unibe.ch has address 130.92.252.19
* Interactive services
ui00.lhep.unibe.ch has address 130.92.140.20
ui01.lhep.unibe.ch has address 130.92.140.11
ui01.lhep.unibe.ch has IPv6 address 2001:620:400:fea8::11
ui02.lhep.unibe.ch has address 130.92.140.12
ui03.lhep.unibe.ch has address 130.92.140.13
ui04.lhep.unibe.ch has address 130.92.140.14
ui05.lhep.unibe.ch has address 130.92.140.15


            '''dpm.lhep STORAGE ELEMENT''':
            dpm        2*2.40GHz Intel Xeon E5620 4-core + HT with 16GB RAM and dpm-1.8.8-4.el5 (epel)
            dpmdisk0x  1*2.67GHz Intel Xeon X5650 6-core with 12GB RAM and dpm-1.8.8-4.el6.x86_64(epel), 6 nodes, 3*raid6 each, 436.52TB net in DPM)
            dpmdisk04  1*2.33GHZ Intel Xeon E5345 4-core with  8GB RAM and dpm-1.8.10-1.el6.x86_64 (epel), 1 node 1*raid6, 21.83TB net in DPM)


            '''bdii.lhep INFORMATION SYSTEM''':
EGI:
            service running on dpm.lhep: emi-bdii-site-1.0.1-1.el5     
* [https://goc.egi.eu/portal/index.php?Page_Type=Site&id=524 GOCDB Entry] - UNIBE-LHEP
* ROC: [https://goc.egi.eu/portal/index.php?Page_Type=NGI&id=11 NGI_CH] - Certification status: Certified.


* Hardware :
            '''ce01.lhep''':
            12-core Xeon E5645 @ 2.40GHz FrontEnd
            96 SUN Blade X6048 (workers, ~1/2 dead)
            6 IBM x3650 M3 (workers)
            12 Dalco (workers)
            6 Dalco Terraserver (Lustre)


MONITORING:
* [http://www-f9.ijs.si/atlas/grid/monitor/loadmon.php?display=vo=Switzerland NorduGrid Swiss ATLAS]
* [http://wiki.lhep.unibe.ch/index.php/AEC-LHEP_Monitoring_information More monitoring information TO BE REVIEWED]


            '''ce02.lhep''':
            26 SUN Fire X2200 (Lustre and workers)
            20 SUN Blade X8840 (workers)
            47 DELL PE 1950
            8-core Xeon E5620 @ 2.40GHz FrontEnd.
            '''dpm.lhep''':
            8-core Xeon E5620 DPM head node
            6 Xeon X5650 (6-core) with 18 12-port Areca 1280/1880 (RAID6) each
            1 Xeon E5-2609 v2 with 1 Areca 1880 (RAID6) (DPM disk servers).
            '''Network''':
            Dell PowerConnect 2724 (WAN), NetGear ProSafe GSM7248R and GS748T (LAN) switches.


* Benchmark (not measured):
NETWORK:
            Quad-Core AMD Opteron(tm) Processor 2376, 2.3GHz (SUN Fire X2200): 5.676 HEP-SPEC06/core (*)
• IPv4: 130.92.140.0/24, GW: 130.92.140.1 (130.92.140.5 - 130.92.140.250)
             Quad-core AMD Opteron 8356, 2.3GHz (SunBlade X8440): 7.4c97 HEP-SPEC/core (**)
• IPv6: 2001:620:400:fea8::/64, GW: 2001:620:400:fea8::1 (2001:620:400:fea8::10 - 2001:620:400:fea8::109)
           
 
            (*)  http://hepix.caspur.it/afs/hepix.org/project/ptrack/spec-cpu2000.html (first row in table, average, SL4 x86_64)
 
            (**) https://wiki.chipp.ch/twiki/bin/view/LCGTier2/BenchMarks
SYSTEMS' TOPOLOGY:
[[Media:Topology-dec2020.png]] - Updated Dec 2020
 
 
SYSTEMS' POWER BUDGET:
[[Media:PowerBudget-apr2021.png]] - Updated Apr 2021
 
 
* '''Hardware Summary :'''
'''ce01.lhep CLUSTER''':
ce01:      2*2.40GHz Intel Xeon E5645 6-Core with 24GB RAM - ROCKS 7.0 and Nordugrid ARC 6.10.2-2.el7 (epel)
ce02:      2*2.60GHz Intel Xeon E5-2650 v2 8-Core +HT with 64GB RAM - Nordugrid ARC 6.10.2-2.el7 (epel)
ce03:     2*2.60GHz Intel Xeon E5-2670  0 8-Core +HT with 96GB RAM - Nordugrid ARC 6.10.2-2.el7 (epel)
mds-2-1:  1*2.40GHz Intel Xeon E5620 4-Core with 24GB RAM for Lustre MDS (lustre-2.12.2-1.el7)
oss-2-x:  2*2.10GHz Intel Xeon E5-2620 v2 6-Core with 32GB RAM for Lustre OSS (lustre-2.12.2-1.el7, 6 nodes each with 34*HDD, 31TB sw raid0)
wn-1-1:    2*2.10GHz Intel Xeon E5-2620 v4 8-core + HT (1 node) 32 cores with total 256GB RAM
            8*GeForce RTX 2080ti GPUs
wn-1-x    2*2.40GHz Intel Xeon E5620  4-core + HT ( 4 nodes)  64 cores with 2.0GB RAM/core
wn-2-x:                   
            2*2.60GHz Intel Xeon E5-2650 v2  8-core + HT (4 Nodes)  128 cores with 2.0GB RAM/core  "Ivi-Bridge EP"  Launch date: Q3'13
            2*2.40GHz Intel Xeon E5-2630 v3  8-core + HT (8 Nodes)  256 cores with 2.0GB RAM/core  "Haswell EP"      Launch date: Q3'14
            2*2.20GHz Intel Xeon E5-2630 v4  10-core + HT (12 Nodes) 480 cores with 2.4GB RAM/core  "Broadwell EP"    Launch date: Q1'16
            2*2.60GHz Intel Xeon E5-2670  0  8-core + HT (12 Nodes)  384 cores with 2.0GB RAM/core   "Sandy Bridge EP" Launch date: Q3'12
            2*2.60GHz Intel Xeon E5-2670  0  8-core + HT (54 Nodes) 1728 cores with 2.0GB RAM/core  "Sandy Bridge EP" Launch date: Q3'12
            2*2.80GHz Intel Xeon E5-2680 v2  10-core + HT (39 Nodes) 1560 cores with 3.2GB RAM/core  "Ivi Bridge EP"  Launch date: Q3'13
             2*2.80GHz AuthenticAMD EPYC 7742 64-core + HT ( 4 Nodes) 1024 cores with 2.0GB RAM/core  "AMD EPYC 7742"  Launch Date: Q3'19
 
'''dpm.lhep STORAGE ELEMENT''':
dpm            2*2.60GHz Intel Xeon E5-2650 v2  8-core + HT with 96GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel )
dpmdisk0[1-3]  1*2.67GHz Intel Xeon X5650      6-core      with 24GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 3* 20TB (18.19TB net in DPM) RAID6 each
dpmdisk04      2*2.50GHZ Intel Xeon E5-2609 v2  4-core      with 32GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 1* 60TB ( 54.57TB net in DPM) RAID6
dpmdisk0[5-6]  1*2.67GHz Intel Xeon X5650      6-core      with 24GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 3* 30TB ( 27.28TB net in DPM) RAID6 each
dpmdisk07      1*2.67GHz Intel Xeon X5650      6-core + HT with 24GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 3* 40TB ( 36.38TB net in DPM) RAID6
dpmdisk08      2*2.10GHz Intel Xeon E5-2620 v4  8-core + HT with 32GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 2*100TB ( 90.95TB net in DPM) RAID6
                                                                                                                          1*140TB (127.33TB net in DPM) RAID6
dpmdisk09      2*2.10GHz Intel Xeon E5-2620 v4  8-core + HT with 96GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 2*378TB (343.76TB net in DPM) RAID6
 
'''bdii.lhep INFORMATION SYSTEM''':
bdii0[1-2]    2*VMs run by ID with emi-bdii-site-1.0.1-1.el6 (UMD-4)
 
'''cvmfs.lhep SQUID FOR CVMFS''':
cvmfs0[1-2]    2*VMs run by ID with frontier-squid-4.13-5.1 (cern-frontier)
 
'''Interactive services''':
ui01:      2*2.40GHz Intel Xeon E5620      4-Core + HT with  64GB RAM (login node) with 3.7TB local user storage /mnt/storage1 (incl. backup of terabig)
ui02:      2*2.40GHz Intel Xeon E5620      4-Core + HT with  64GB RAM (login node) with 3.7TB local user storage /mnt/storage1
  ui03:     2*2.40GHz Intel Xeon E5-2630 v3  8-Core + HT with 128GB RAM (login node) with 3.7TB local user storage /mnt/storage1
ui04:      2*2.40GHz Intel Xeon E5-2640 v4 10-Core + HT with 128GB RAM (login node) with 3.7TB local user storage /mnt/storage1
ui05:      2*2.40GHz Intel Xeon E5-2640 v4 10-Core + HT with 128GB RAM (login node) with 3.7TB local user storage /mnt/storage1
homes:    2*2.50GHz Intel Xeon E5-2609 v2  4-Core      with 32GB RAM shared home directories: 8.7TB exported to the ui's as /terabig
nfs02:    1*2.40GHz Intel Xeon E5-2609 v2  4-Core      with 16GB RAM shared storage area:      44TB exported to the ui's as /ngse5
 
'''Virtualisation service''':
kvm02      2*2.67GHz Intel Xeon X5650      6-core +HT with  64GB with KVM and 3.5T storage for guests     
 
  '''Network''':
  2 Brocade FCX648 (4*10GbE, 48*1GbE each) with 2*10GbE uplink (Scienze DMZ)
 
  2 Dell Force10 (48+48*1GbE vLAN stack) private LAN ce02.lhep CLUSTER)
  2 Dell Force10 (2*24+24*1GbE vLANs stack) private LAN ce01.lhep CLUSTER, 2nd vLAN unused)
  1 Mellanox QDR (30*40Gbit) private IB LAN ce01.lhep CLUSTER
  x QDR switches built-in the SUN Blade X6048 rack. private IB LAN ce01.lhep CLUSTER
 
* '''Benchmark HEPspec06''' (weighed average across all node types):
 
            '''ce01.lhep'''
          NODES * TYPE (cores)  HEPSPEC/node (total)  HEPSPEC/core  CPU model
            4 * 32-core  (128)    370.36  (1481.44)    11.57        Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz "Ivi-Bridge EP"
            8 * 32-core  (256)    406.31  (3250.48)    12.7          Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz "Haswell EP"
          12 * 40-core  (480)    496.28  (5955.36)    12.41        Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz "Broadwell EP"
          12 * 32-core  (384)    366.08  (4392.96)    11.44        Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz  "Sandy Bridge EP"
          54 * 32-core  (1728)    366.08 (19768.32)    11.44        Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz  "Sandy Bridge EP"
          39 * 40-core  (1560)    488.40 (19047.60)    12.21        Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz "Ivi Bridge EP"
            4 * 256-core (1024)  3158.79 (12635.16)    12.34        AMD EPYC 7742 64-Core Processor
          =========================================================
          133            5560            66531.32
 
          In arc.conf:
          [infosys/glue1]
          processor_other_description=Cores=41.8, Benchmark=11.97-HEP-SPEC06
          ...
          [queue:CLUSTER]
          benchmark=HEPSPEC 11.97
          benchmark=Si2k 2993
 
          [queue:CLUSTER-HM]
          benchmark=HEPSPEC 12.21
          benchmark=Si2k 3052
          allowaccess=uboone
          allowaccess=dune
          allowaccess=ops


* Available resources (workers):
          [queue:CLUSTER-GPU]
            19*8  AMD Opteron(tm) 2376:        862.752 HEP-SPEC06
          benchmark=HEPSPEC 11.97
            20*16 Quad-core AMD Opteron 8356:  2399.04 HEP-SPEC06
          benchmark=Si2k 2993
            ---------------------------------------------------------------------------------------------------------
            Total: 3261.8 HEP-SPEC06


* Network : Gateway 130.92.139.1. DNS 130.92.9.53 (default), 130.92.9.52. Subnet mask 255.255.255.0
           
* GOCDB Entry: https://gocdb4.esc.rl.ac.uk/portal/index.php?Page_Type=View_Object&object_id=476&grid_id=0
* ROC: NGI_CH - Certification status: Certified.




Line 74: Line 151:
! Host !! IP !! Service !!  OS !! CPU !! Comment  
! Host !! IP !! Service !!  OS !! CPU !! Comment  
|-
|-
! lheplsw1.unibe.ch !!  130.92.139.201  !!  48-Port Switch  !!    !! Brocade FCX648 !! Connected to eth1 on FE (on 10GbE NIC) !!  
! lheplsw1.unibe.ch !!  130.92.139.201  !!  48+4 Port Switch  !!    !! Brocade FCX648 !! Connected to eth0 (10GbE NIC) on ce01 and eth2 (10GbE NIC) on ce02 !!
|-
! ce01.lhep.unibe.ch !!  130.92.140.201  !!  Rocks Front End (FE) 6.2 x86_64 !!  SLC 6.7 x86_64 !! 2 Quad Xeon E5645 @2.4GHz !! Rocks FE, SLURM master, ARC CE, Web Server, Ganglia
|-
! ce02.lhep.unibe.ch !!  130.92.140.202  !!  Rocks Front End (FE) 6.2 x86_64 !!  SLC 6.7 x86_64 !! 2 Quad Xeon E5620 @2.4GHz !! Rocks FE, SLURM master, ARC CE, Web Server, Ganglia
|-
! nfs01.lhep.unibe.ch !!  130.92.140.31  !! 10TB NFS server !!  SLC 6.3 x86_64 !! 1 Quad Xeon E5620 @2.4GHz !! Obsolete as sw server, some backups (also nfs-2-1, 10.2.255.100)
|-
! nfs02.lhep.unibe.ch !!  130.92.140.32  !! 37TB NFS server !!  SLC 6.3 x86_64 !! 1 Quad Xeon E5530 @2.4GHz !! Local NFS server for ATLAS, exports /ngse5
|-
! mds-2-1 !!  10.1.255.200  !! 503GB Lustre MDS/MDT ce01 !! SLC 6.7 x86_64 !! 1 Quad Xeon E5620 @2.4GHz !! MDS/MDT (/dev/mapper/vg00-mdt LV on SSDs for the MDT), Managed by ce01
|-
! mds-0-1 !!  10.1.255.200  !! 110GB Lustre MDS/MDT ce02 !! SLC 6.7 x86_64 !! 1 Quad Xeon E5620 @2.4GHz !! MDS/MDT (/dev/sda2 LV on SSDs for the MDT), Managed by ce02
|-
! oss-2-[1-6] !! 10.1.255.[11-16]  !! Lustre OSS ce01 !! SLC 6.7 x86_64 !! 2 Exa Xeon E5-2620 v2 @2.1GHz !! Managed by ce01
|-
|-
! ce.lhep.unibe.ch !! 130.92.139.200 !! Rocks Front End (FE) 5.3 x86_64 !! CentOS 5.3 x86_64 !! 2 Quad Xeon E5620 @2.4GHz !! Rocks FE, SGE, ARC FE, Web Server, Ganglia
! oss-0-[1-12] !! 10.1.255.[11-22] !! Lustre OSS ce02 !! SLC 6.7 x86_64 !! 2 Dual AMD Opteron 2218 or 2376  @2.3GHz !! Managed by ce02
|-
|-
! !! 10.1.225.254  !! 48 Ports Switch  !!   !! !!  
! wn-2-[1-96] !! 10.1.255.[101-196] !! Worker Nodes ce01 !! SLC 6.7 x86_64 !! 2 Quad Xeon E5540 @2.53GHz !! Managed by ce01
|-
|-
! nas-0-2 !! 10.1.255.253  !! 10TB NFS server !! CentOS 5.3 !! 1 Quad Xeon E5620 @2.4GHz !! ATLAS and VO software (also lhepat02.unibe.ch)
! wn-2-[101-106] !! 10.1.255.[91-96] !! Worker Nodes ce01 !! SLC 6.7 x86_64 !! 2 Exa Xeon X5650 @2.67GHz !! Managed by ce01
|-
|-
! nas-0-1 !! 10.1.255.252  !! 1.1TB Lustre MDS/MDT !! CentOS 5.3 !! 1 Quad Xeon E5620 @2.4GHz !! MDS/MDT (currently not in production)
! wn-2-[111-114] !! 10.1.255.[201-204] !! Worker Nodes ce01 !! SLC 6.7 x86_64 !! 4 Quad Xeon E5-2650 v2 @2.6GHz !! Managed by ce01
|-
|-
! compute-0-0/2 !! 10.1.255.xxx  !! Lustre !! CentOS 5.3 !! 2 Dual AMD Opteron 2214 !! Managed by ce.lhep.unibe.ch
! wn-2-[115-122] !! 10.1.255.[205-212] !! Worker Nodes ce01 !! SLC 6.7 x86_64 !! 4 Quad Xeon E5-2630 v3 @2.4GHz !! Managed by ce01
|-
|-
! compute-0-3 !! 10.1.255.248 !! Lustre MDS/MDT !! CentOS 5.3 !! 2 Dual AMD Opteron 2214 !! Managed by ce.lhep.unibe.ch
! wn-0-[1-15] !! 10.1.255.[101-115] !! Worker Nodes ce02 !! SLC 6.7 x86_64 !! 2 Dual AMD Opteron 2376 @2.3GHz !! Managed by ce02
|-
|-
! compute-0-4/5/6/7 !! 10.1.255.xxx  !! Lustre  !! CentOS 5.3 !! 2 Quad AMD Opteron 2376 !! Managed by ce.lhep.unibe.ch
! wn-0-[16-62] !! 10.1.255.[116-162] !! Worker Nodes ce02 !! SLC 6.7 x86_64 !! 2 Quad Xeon E5420 @2.5GHz !! Managed by ce02
|-
|-
! compute-0-8 to 26 !! 10.1.255.xxx !! Worker Node 0-8 to 0-26  !!  CentOS 5.3 !! 2 Quad AMD Opteron 2376 !! Managed by ce.lhep.unibe.ch
! wn-1-[1-20] !! 10.1.255.[201-220] !! Worker Nodes ce02 !!  SLC 6.7 x86_64 !! 4 Quad AMD Opteron 8356 @2.3GHz !! Managed by ce02
|-
|-
! compute-1-1 to 19 !! 10.1.255.xxx !! Worker Node 1-1 to 1-19 !!  CentOS 5.3 !! 4 Quad AMD Opteron 8356 !! Managed by ce.lhep.unibe.ch
! dpm.lhep.unibe.ch !! 130.92.139.211 !! DPM head node !!  SLC 5.7 !! 2 Quad Xeon E5620 @2.4GHz !! DPM Head node
|-
|-
! dpm.lhep.unibe.ch !! 130.92.139.211 !! DPM head node !!  SLC 5.7 !! 2 Quad Xeon E5620 @2.4GHz !! DPM Head node and disk pool
! dpmdisk[01-03].lhep.unibe.ch !! 130.92.140.[211-213] !! DPM pool nodes !!  SLC 6.8 x86_64 !! 1 Exa Xeon X5650 @2.67GHz !! DPM 60TB disk pools
|-
|-
! dpmdisk01(to 3).lhep.unibe.ch !! 130.92.139.211 to 213 !!  DPM pool node !!  SLC 5.7 !! 1 Hexa Xeon X5650 @2.67GHz !! DPM 60TB disk pool
! dpmdisk04.lhep.unibe.ch !! 130.92.140.214 !!  DPM pool node !!  SLC 6.8 x86_64 !! 1 Quad Xeon E5-2609 v2 @2.50GHz !! DPM 22TB disk pools
|-
|-
! dpmdisk04.lhep.unibe.ch !! 130.92.139.214 !!  DPM pool node !!  SLC 5.7  !! 1 Quad Xeon E5345 @2.33 GHz !! DPM 22TB disk pool
! dpmdisk[05-07].lhep.unibe.ch !! 130.92.140.[215 to 219] !!  DPM pool nodes  !!  SLC 6.8 x86_64 !! Exa Xeon X5650  @2.67GHz !! DPM 90/120TB disk pools
|-
|-
! dpmdisk05(to 9).lhep.unibe.ch !! 130.92.139.215 to 219 !!  DPM pool node  !!  SLC 5.x !! ?? !! Reserved for DPM disk pool
! kvm02.lhep.unibe.ch !! 130.92.140.22 !!  KVM host !!  SLC 6.8 x86_64 !! 2 Exa Xeon X5650 @2.67GHz !! Virtualization host for KVM
|-
|-
! kvm01.lhep.unibe.ch !! 130.92.139.150 !! KVM host !!  SLC 5.7 !! 2 Dual AMD Opteron 2218 @2.6GHz !! Virtualization host for KVM
! ui0[1-3].lhep.unibe.ch !! 130.92.140.1[1-3] !! Interactive login servers !!  SLC 6.8 x86_64 !! 2 Quad Xeon E5620 and 2 Oct Xeon E5-2630 v3 @2.4GHz !! 3.7TB local storage
|-
! homes.lhep.unibe.ch !! 130.92.140.10 !! Home directories server !!  SLC 6.8 x86_64  !! 2 Quad Xeon E5-2609 v2 !! 8.7TB nfs homedirs
|-
! nfs02.lhep.unibe.ch !! 130.92.140.32 !! Local storage server !!  SLC 6.8 x86_64  !! 2 Quad Xeon E5530 !! 37TB nfs storage area
|-
|-
! ... !!  ... !!  ... !!  ...  !! ... !!  ...|-
! ... !!  ... !!  ... !!  ...  !! ... !!  ...|-


|}
|}

Latest revision as of 23:07, 6 April 2021

BERN ATLAS Tier-2, Tier 3 - 2020 Systems

  • Grid HT Cluster ce01 (5.5k cores)
    • Node summary
    • Scheduler: SLURM 18.08.8-1.el7 (scheduler on ce01)
    • Shared File System: Lustre 2.12.2-1.el7
      • 1 x MDS, 6 x OSS: 204 x 1T HDDs (184T)
  • Grid Compute Elements ce01.lhep.unibe.ch, ce02.lhep.unibe.ch, ce03.lhep.unibe.ch
ce01.lhep.unibe.ch has address 130.92.140.201
ce02.lhep.unibe.ch has address 130.92.140.202
ce03.lhep.unibe.ch has address 130.92.140.203
  • Grid Storage Element dpm.lhep.unibe.ch (2PB)
dpm.lhep.unibe.ch has address 130.92.140.100
dpm.lhep.unibe.ch has IPv6 address 2001:620:400:fea8::100
  • Grid Information System bdii.lhep.unibe.ch 
# host bdii.lhep.unibe.ch
  bdii.lhep.unibe.ch has address 130.92.140.121
  bdii.lhep.unibe.ch has address 130.92.140.122
  • Squid for CVMFS cvmfs.lhep.unibe.ch
# host cvmfs.lhep.unibe.ch
  cvmfs.lhep.unibe.ch has address 130.92.252.18
  cvmfs.lhep.unibe.ch has address 130.92.252.19
  • Interactive services
ui00.lhep.unibe.ch has address 130.92.140.20
ui01.lhep.unibe.ch has address 130.92.140.11
ui01.lhep.unibe.ch has IPv6 address 2001:620:400:fea8::11
ui02.lhep.unibe.ch has address 130.92.140.12
ui03.lhep.unibe.ch has address 130.92.140.13
ui04.lhep.unibe.ch has address 130.92.140.14
ui05.lhep.unibe.ch has address 130.92.140.15


EGI:


MONITORING:


NETWORK: 

• IPv4: 130.92.140.0/24, GW: 130.92.140.1 (130.92.140.5 - 130.92.140.250)
• IPv6: 2001:620:400:fea8::/64, GW: 2001:620:400:fea8::1 (2001:620:400:fea8::10 - 2001:620:400:fea8::109)


SYSTEMS' TOPOLOGY: Media:Topology-dec2020.png - Updated Dec 2020


SYSTEMS' POWER BUDGET: Media:PowerBudget-apr2021.png - Updated Apr 2021


  • Hardware Summary :
ce01.lhep CLUSTER:
ce01:      2*2.40GHz Intel Xeon E5645 6-Core with 24GB RAM - ROCKS 7.0 and Nordugrid ARC 6.10.2-2.el7 (epel)
ce02:      2*2.60GHz Intel Xeon E5-2650 v2 8-Core +HT with 64GB RAM - Nordugrid ARC 6.10.2-2.el7 (epel)
ce03:      2*2.60GHz Intel Xeon E5-2670  0 8-Core +HT with 96GB RAM - Nordugrid ARC 6.10.2-2.el7 (epel)
mds-2-1:   1*2.40GHz Intel Xeon E5620 4-Core with 24GB RAM for Lustre MDS (lustre-2.12.2-1.el7)
oss-2-x:   2*2.10GHz Intel Xeon E5-2620 v2 6-Core with 32GB RAM for Lustre OSS (lustre-2.12.2-1.el7, 6 nodes each with 34*HDD, 31TB sw raid0)
wn-1-1:    2*2.10GHz Intel Xeon E5-2620 v4 8-core + HT (1 node) 32 cores with total 256GB RAM
           8*GeForce RTX 2080ti GPUs
wn-1-x     2*2.40GHz Intel Xeon E5620  4-core + HT ( 4 nodes)  64 cores with 2.0GB RAM/core
wn-2-x:                    
           2*2.60GHz Intel Xeon E5-2650 v2   8-core + HT (4 Nodes)   128 cores with 2.0GB RAM/core   "Ivi-Bridge EP"   Launch date: Q3'13
           2*2.40GHz Intel Xeon E5-2630 v3   8-core + HT (8 Nodes)   256 cores with 2.0GB RAM/core   "Haswell EP"      Launch date: Q3'14
           2*2.20GHz Intel Xeon E5-2630 v4  10-core + HT (12 Nodes)  480 cores with 2.4GB RAM/core   "Broadwell EP"    Launch date: Q1'16
           2*2.60GHz Intel Xeon E5-2670  0   8-core + HT (12 Nodes)  384 cores with 2.0GB RAM/core   "Sandy Bridge EP" Launch date: Q3'12
           2*2.60GHz Intel Xeon E5-2670  0   8-core + HT (54 Nodes) 1728 cores with 2.0GB RAM/core   "Sandy Bridge EP" Launch date: Q3'12
           2*2.80GHz Intel Xeon E5-2680 v2  10-core + HT (39 Nodes) 1560 cores with 3.2GB RAM/core   "Ivi Bridge EP"   Launch date: Q3'13
           2*2.80GHz AuthenticAMD EPYC 7742 64-core + HT ( 4 Nodes) 1024 cores with 2.0GB RAM/core   "AMD EPYC 7742"   Launch Date: Q3'19

dpm.lhep STORAGE ELEMENT:
dpm            2*2.60GHz Intel Xeon E5-2650 v2  8-core + HT with 96GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel )
dpmdisk0[1-3]  1*2.67GHz Intel Xeon X5650       6-core      with 24GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 3* 20TB (18.19TB net in DPM) RAID6 each
dpmdisk04      2*2.50GHZ Intel Xeon E5-2609 v2  4-core      with 32GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 1* 60TB ( 54.57TB net in DPM) RAID6 
dpmdisk0[5-6]  1*2.67GHz Intel Xeon X5650       6-core      with 24GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 3* 30TB ( 27.28TB net in DPM) RAID6 each
dpmdisk07      1*2.67GHz Intel Xeon X5650       6-core + HT with 24GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 3* 40TB ( 36.38TB net in DPM) RAID6
dpmdisk08      2*2.10GHz Intel Xeon E5-2620 v4  8-core + HT with 32GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 2*100TB ( 90.95TB net in DPM) RAID6
                                                                                                                         1*140TB (127.33TB net in DPM) RAID6
dpmdisk09      2*2.10GHz Intel Xeon E5-2620 v4  8-core + HT with 96GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 2*378TB (343.76TB net in DPM) RAID6

bdii.lhep INFORMATION SYSTEM:
bdii0[1-2]     2*VMs run by ID with emi-bdii-site-1.0.1-1.el6 (UMD-4)

cvmfs.lhep SQUID FOR CVMFS:
cvmfs0[1-2]    2*VMs run by ID with frontier-squid-4.13-5.1 (cern-frontier)

Interactive services:
ui01:      2*2.40GHz Intel Xeon E5620       4-Core + HT with  64GB RAM (login node) with 3.7TB local user storage /mnt/storage1 (incl. backup of terabig)
ui02:      2*2.40GHz Intel Xeon E5620       4-Core + HT with  64GB RAM (login node) with 3.7TB local user storage /mnt/storage1 
ui03:      2*2.40GHz Intel Xeon E5-2630 v3  8-Core + HT with 128GB RAM (login node) with 3.7TB local user storage /mnt/storage1
ui04:      2*2.40GHz Intel Xeon E5-2640 v4 10-Core + HT with 128GB RAM (login node) with 3.7TB local user storage /mnt/storage1 
ui05:      2*2.40GHz Intel Xeon E5-2640 v4 10-Core + HT with 128GB RAM (login node) with 3.7TB local user storage /mnt/storage1 
homes:     2*2.50GHz Intel Xeon E5-2609 v2  4-Core       with 32GB RAM shared home directories: 8.7TB exported to the ui's as /terabig 
nfs02:     1*2.40GHz Intel Xeon E5-2609 v2  4-Core       with 16GB RAM shared storage area:      44TB exported to the ui's as /ngse5

Virtualisation service:
kvm02      2*2.67GHz Intel Xeon X5650      6-core +HT with   64GB with KVM and 3.5T storage for guests      

 Network:
 2 Brocade FCX648 (4*10GbE, 48*1GbE each) with 2*10GbE uplink (Scienze DMZ)

 2 Dell Force10 (48+48*1GbE vLAN stack) private LAN ce02.lhep CLUSTER)
 2 Dell Force10 (2*24+24*1GbE vLANs stack) private LAN ce01.lhep CLUSTER, 2nd vLAN unused)
 1 Mellanox QDR (30*40Gbit) private IB LAN ce01.lhep CLUSTER
 x QDR switches built-in the SUN Blade X6048 rack. private IB LAN ce01.lhep CLUSTER
  • Benchmark HEPspec06 (weighed average across all node types):
           ce01.lhep
          NODES * TYPE (cores)   HEPSPEC/node (total)  HEPSPEC/core  CPU model
           4 * 32-core   (128)    370.36  (1481.44)    11.57         Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz "Ivi-Bridge EP"
           8 * 32-core   (256)    406.31  (3250.48)    12.7          Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz "Haswell EP"
          12 * 40-core   (480)    496.28  (5955.36)    12.41         Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz "Broadwell EP"
          12 * 32-core   (384)    366.08  (4392.96)    11.44         Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz  "Sandy Bridge EP"
          54 * 32-core  (1728)    366.08 (19768.32)    11.44         Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz  "Sandy Bridge EP"
          39 * 40-core  (1560)    488.40 (19047.60)    12.21         Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz "Ivi Bridge EP"
           4 * 256-core (1024)   3158.79 (12635.16)    12.34         AMD EPYC 7742 64-Core Processor
          =========================================================
          133            5560             66531.32

          In arc.conf:
          [infosys/glue1]
          processor_other_description=Cores=41.8, Benchmark=11.97-HEP-SPEC06
          ...
          [queue:CLUSTER]
          benchmark=HEPSPEC 11.97
          benchmark=Si2k 2993

          [queue:CLUSTER-HM]
          benchmark=HEPSPEC 12.21
          benchmark=Si2k 3052
          allowaccess=uboone
          allowaccess=dune
          allowaccess=ops

          [queue:CLUSTER-GPU]
          benchmark=HEPSPEC 11.97
          benchmark=Si2k 2993



Host IP Service OS CPU Comment
lheplsw1.unibe.ch 130.92.139.201 48+4 Port Switch Brocade FCX648 Connected to eth0 (10GbE NIC) on ce01 and eth2 (10GbE NIC) on ce02
ce01.lhep.unibe.ch 130.92.140.201 Rocks Front End (FE) 6.2 x86_64 SLC 6.7 x86_64 2 Quad Xeon E5645 @2.4GHz Rocks FE, SLURM master, ARC CE, Web Server, Ganglia
ce02.lhep.unibe.ch 130.92.140.202 Rocks Front End (FE) 6.2 x86_64 SLC 6.7 x86_64 2 Quad Xeon E5620 @2.4GHz Rocks FE, SLURM master, ARC CE, Web Server, Ganglia
nfs01.lhep.unibe.ch 130.92.140.31 10TB NFS server SLC 6.3 x86_64 1 Quad Xeon E5620 @2.4GHz Obsolete as sw server, some backups (also nfs-2-1, 10.2.255.100)
nfs02.lhep.unibe.ch 130.92.140.32 37TB NFS server SLC 6.3 x86_64 1 Quad Xeon E5530 @2.4GHz Local NFS server for ATLAS, exports /ngse5
mds-2-1 10.1.255.200 503GB Lustre MDS/MDT ce01 SLC 6.7 x86_64 1 Quad Xeon E5620 @2.4GHz MDS/MDT (/dev/mapper/vg00-mdt LV on SSDs for the MDT), Managed by ce01
mds-0-1 10.1.255.200 110GB Lustre MDS/MDT ce02 SLC 6.7 x86_64 1 Quad Xeon E5620 @2.4GHz MDS/MDT (/dev/sda2 LV on SSDs for the MDT), Managed by ce02
oss-2-[1-6] 10.1.255.[11-16] Lustre OSS ce01 SLC 6.7 x86_64 2 Exa Xeon E5-2620 v2 @2.1GHz Managed by ce01
oss-0-[1-12] 10.1.255.[11-22] Lustre OSS ce02 SLC 6.7 x86_64 2 Dual AMD Opteron 2218 or 2376 @2.3GHz Managed by ce02
wn-2-[1-96] 10.1.255.[101-196] Worker Nodes ce01 SLC 6.7 x86_64 2 Quad Xeon E5540 @2.53GHz Managed by ce01
wn-2-[101-106] 10.1.255.[91-96] Worker Nodes ce01 SLC 6.7 x86_64 2 Exa Xeon X5650 @2.67GHz Managed by ce01
wn-2-[111-114] 10.1.255.[201-204] Worker Nodes ce01 SLC 6.7 x86_64 4 Quad Xeon E5-2650 v2 @2.6GHz Managed by ce01
wn-2-[115-122] 10.1.255.[205-212] Worker Nodes ce01 SLC 6.7 x86_64 4 Quad Xeon E5-2630 v3 @2.4GHz Managed by ce01
wn-0-[1-15] 10.1.255.[101-115] Worker Nodes ce02 SLC 6.7 x86_64 2 Dual AMD Opteron 2376 @2.3GHz Managed by ce02
wn-0-[16-62] 10.1.255.[116-162] Worker Nodes ce02 SLC 6.7 x86_64 2 Quad Xeon E5420 @2.5GHz Managed by ce02
wn-1-[1-20] 10.1.255.[201-220] Worker Nodes ce02 SLC 6.7 x86_64 4 Quad AMD Opteron 8356 @2.3GHz Managed by ce02
dpm.lhep.unibe.ch 130.92.139.211 DPM head node SLC 5.7 2 Quad Xeon E5620 @2.4GHz DPM Head node
dpmdisk[01-03].lhep.unibe.ch 130.92.140.[211-213] DPM pool nodes SLC 6.8 x86_64 1 Exa Xeon X5650 @2.67GHz DPM 60TB disk pools
dpmdisk04.lhep.unibe.ch 130.92.140.214 DPM pool node SLC 6.8 x86_64 1 Quad Xeon E5-2609 v2 @2.50GHz DPM 22TB disk pools
dpmdisk[05-07].lhep.unibe.ch 130.92.140.[215 to 219] DPM pool nodes SLC 6.8 x86_64 Exa Xeon X5650 @2.67GHz DPM 90/120TB disk pools
kvm02.lhep.unibe.ch 130.92.140.22 KVM host SLC 6.8 x86_64 2 Exa Xeon X5650 @2.67GHz Virtualization host for KVM
ui0[1-3].lhep.unibe.ch 130.92.140.1[1-3] Interactive login servers SLC 6.8 x86_64 2 Quad Xeon E5620 and 2 Oct Xeon E5-2630 v3 @2.4GHz 3.7TB local storage
homes.lhep.unibe.ch 130.92.140.10 Home directories server SLC 6.8 x86_64 2 Quad Xeon E5-2609 v2 8.7TB nfs homedirs
nfs02.lhep.unibe.ch 130.92.140.32 Local storage server SLC 6.8 x86_64 2 Quad Xeon E5530 37TB nfs storage area
... ... ... ... ... -
Retrieved from "https://wiki.lhep.unibe.ch/index.php?title=AEC-LHEP_Hardware_information&oldid=684"