AEC-LHEP Hardware information: Difference between revisions

From LHEP Wiki
Jump to navigation Jump to search
No edit summary
No edit summary
 
(93 intermediate revisions by the same user not shown)
Line 1: Line 1:


BERN ATLAS T2 2016 Clusters - ce0.lhep.unibe.ch, ce02.lhep.unibe.ch, dpm.lhep.unibe.ch, bdii.lhep.unibe.ch
BERN ATLAS Tier-2, Tier 3 - 2020 Systems


* Summary :
* Grid HT Cluster '''ce01''' (5.5k cores)
            '''ce01.lhep''':
** Node summary
            540 2.53 GHz Intel(R) Xeon(R) E5540 Worker Cores (45 Nodes) with 2GB RAM per core (or 720 cores with 1.33GB RAM per core)
** Scheduler: SLURM 18.08.8-1.el7 (scheduler on ce01)
            32  2.53 GHz Intel(R) Xeon(R) E5540 Worker Cores (4 Nodes) with 2GB RAM per core (same HW as above with 1 dead CPU)
** Shared File System: Lustre 2.12.2-1.el7
            144 2.67 GHz Intel(R) Xeon(R) X5650 Worker Cores (6 Nodes) with 2GB RAM per core
*** 1 x MDS, 6 x OSS: 204 x 1T HDDs (184T)
            128 2.60 GHz Intel(R) Xeon(R) E5-2650 v2 Worker Cores (4 Nodes) with 2GB RAM per core
            256 2.40 GHz Intel(R) Xeon(R) E5-2630 v3 Worker Cores (8 Nodes) with 2GB RAM per core
            186TB Lustre FS (6 nodes, 204 spindles)


            '''ce02.lhep''':
* Grid Compute Elements '''ce01.lhep.unibe.ch, ce02.lhep.unibe.ch, ce03.lhep.unibe.ch'''
             120 2.30 GHz AMD Opteron 2376 Worker Cores (15 Nodes) with 2GB RAM per core
ce01.lhep.unibe.ch has address 130.92.140.201
             320 2.30 GHz AMD Opteron 8356 Worker Cores (20 Nodes) with 2GB RAM per core
ce02.lhep.unibe.ch has address 130.92.140.202
             376 2.50 GHz Intel(R) Xeon(R) E5420 (47 Nodes) with 2GB RAM per core
ce03.lhep.unibe.ch has address 130.92.140.203
             20TB Lustre FS (11 nodes, 11 spindles).
* Grid Storage Element '''dpm.lhep.unibe.ch ''' (2PB)
           
dpm.lhep.unibe.ch has address 130.92.140.100
            '''dpm.lhep''':
dpm.lhep.unibe.ch has IPv6 address 2001:620:400:fea8::100
            562TB DPM Storage Element (net in DPM: 460TB) with 7 disk servers and 19*arrays (RAID 6)
* Grid Information System '''bdii.lhep.unibe.ch '''
# host bdii.lhep.unibe.ch
  bdii.lhep.unibe.ch has address 130.92.140.121
  bdii.lhep.unibe.ch has address 130.92.140.122
* Squid for CVMFS '''cvmfs.lhep.unibe.ch'''
# host cvmfs.lhep.unibe.ch
  cvmfs.lhep.unibe.ch has address 130.92.252.18
  cvmfs.lhep.unibe.ch has address 130.92.252.19
* Interactive services
ui00.lhep.unibe.ch has address 130.92.140.20
ui01.lhep.unibe.ch has address 130.92.140.11
ui01.lhep.unibe.ch has IPv6 address 2001:620:400:fea8::11
ui02.lhep.unibe.ch has address 130.92.140.12
ui03.lhep.unibe.ch has address 130.92.140.13
ui04.lhep.unibe.ch has address 130.92.140.14
ui05.lhep.unibe.ch has address 130.92.140.15
 
 
EGI:
* [https://goc.egi.eu/portal/index.php?Page_Type=Site&id=524 GOCDB Entry] - UNIBE-LHEP
* ROC: [https://goc.egi.eu/portal/index.php?Page_Type=NGI&id=11 NGI_CH] - Certification status: Certified.
 
 
MONITORING:
* [http://www-f9.ijs.si/atlas/grid/monitor/loadmon.php?display=vo=Switzerland NorduGrid Swiss ATLAS]
* [http://wiki.lhep.unibe.ch/index.php/AEC-LHEP_Monitoring_information More monitoring information TO BE REVIEWED]
 
 
NETWORK:
• IPv4: 130.92.140.0/24, GW: 130.92.140.1 (130.92.140.5 - 130.92.140.250)
• IPv6: 2001:620:400:fea8::/64, GW: 2001:620:400:fea8::1 (2001:620:400:fea8::10 - 2001:620:400:fea8::109)
 
 
SYSTEMS' TOPOLOGY:
[[Media:Topology-dec2020.png]] - Updated Dec 2020
 
 
SYSTEMS' POWER BUDGET:
[[Media:PowerBudget-apr2021.png]] - Updated Apr 2021
 
 
* '''Hardware Summary :'''
'''ce01.lhep CLUSTER''':
ce01:      2*2.40GHz Intel Xeon E5645 6-Core with 24GB RAM - ROCKS 7.0 and Nordugrid ARC 6.10.2-2.el7 (epel)
ce02:      2*2.60GHz Intel Xeon E5-2650 v2 8-Core +HT with 64GB RAM - Nordugrid ARC 6.10.2-2.el7 (epel)
ce03:      2*2.60GHz Intel Xeon E5-2670  0 8-Core +HT with 96GB RAM - Nordugrid ARC 6.10.2-2.el7 (epel)
mds-2-1:  1*2.40GHz Intel Xeon E5620 4-Core with 24GB RAM for Lustre MDS (lustre-2.12.2-1.el7)
oss-2-x:  2*2.10GHz Intel Xeon E5-2620 v2 6-Core with 32GB RAM for Lustre OSS (lustre-2.12.2-1.el7, 6 nodes each with 34*HDD, 31TB sw raid0)
wn-1-1:    2*2.10GHz Intel Xeon E5-2620 v4 8-core + HT (1 node) 32 cores with total 256GB RAM
             8*GeForce RTX 2080ti GPUs
wn-1-x    2*2.40GHz Intel Xeon E5620  4-core + HT ( 4 nodes)  64 cores with 2.0GB RAM/core
wn-2-x:                   
            2*2.60GHz Intel Xeon E5-2650 v2  8-core + HT (4 Nodes)  128 cores with 2.0GB RAM/core  "Ivi-Bridge EP"  Launch date: Q3'13
            2*2.40GHz Intel Xeon E5-2630 v3  8-core + HT (8 Nodes)   256 cores with 2.0GB RAM/core   "Haswell EP"      Launch date: Q3'14
             2*2.20GHz Intel Xeon E5-2630 v4  10-core + HT (12 Nodes) 480 cores with 2.4GB RAM/core   "Broadwell EP"    Launch date: Q1'16
             2*2.60GHz Intel Xeon E5-2670  0  8-core + HT (12 Nodes) 384 cores with 2.0GB RAM/core  "Sandy Bridge EP" Launch date: Q3'12
            2*2.60GHz Intel Xeon E5-2670  0  8-core + HT (54 Nodes) 1728 cores with 2.0GB RAM/core  "Sandy Bridge EP" Launch date: Q3'12
            2*2.80GHz Intel Xeon E5-2680 v2  10-core + HT (39 Nodes) 1560 cores with 3.2GB RAM/core   "Ivi Bridge EP"  Launch date: Q3'13
             2*2.80GHz AuthenticAMD EPYC 7742 64-core + HT ( 4 Nodes) 1024 cores with 2.0GB RAM/core  "AMD EPYC 7742"  Launch Date: Q3'19
 
'''dpm.lhep STORAGE ELEMENT''':
dpm            2*2.60GHz Intel Xeon E5-2650 v2  8-core + HT with 96GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel )
dpmdisk0[1-3]  1*2.67GHz Intel Xeon X5650      6-core      with 24GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 3* 20TB (18.19TB net in DPM) RAID6 each
dpmdisk04      2*2.50GHZ Intel Xeon E5-2609 v2  4-core      with 32GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 1* 60TB ( 54.57TB net in DPM) RAID6
dpmdisk0[5-6]  1*2.67GHz Intel Xeon X5650      6-core      with 24GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 3* 30TB ( 27.28TB net in DPM) RAID6 each
dpmdisk07      1*2.67GHz Intel Xeon X5650      6-core + HT with 24GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 3* 40TB ( 36.38TB net in DPM) RAID6
dpmdisk08      2*2.10GHz Intel Xeon E5-2620 v4  8-core + HT with 32GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 2*100TB ( 90.95TB net in DPM) RAID6
                                                                                                                          1*140TB (127.33TB net in DPM) RAID6
dpmdisk09      2*2.10GHz Intel Xeon E5-2620 v4  8-core + HT with 96GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 2*378TB (343.76TB net in DPM) RAID6
 
'''bdii.lhep INFORMATION SYSTEM''':
bdii0[1-2]    2*VMs run by ID with emi-bdii-site-1.0.1-1.el6 (UMD-4)
 
'''cvmfs.lhep SQUID FOR CVMFS''':
cvmfs0[1-2]    2*VMs run by ID with frontier-squid-4.13-5.1 (cern-frontier)
 
'''Interactive services''':
ui01:      2*2.40GHz Intel Xeon E5620      4-Core + HT with  64GB RAM (login node) with 3.7TB local user storage /mnt/storage1 (incl. backup of terabig)
ui02:      2*2.40GHz Intel Xeon E5620      4-Core + HT with  64GB RAM (login node) with 3.7TB local user storage /mnt/storage1
ui03:      2*2.40GHz Intel Xeon E5-2630 v3  8-Core + HT with 128GB RAM (login node) with 3.7TB local user storage /mnt/storage1
ui04:      2*2.40GHz Intel Xeon E5-2640 v4 10-Core + HT with 128GB RAM (login node) with 3.7TB local user storage /mnt/storage1
ui05:      2*2.40GHz Intel Xeon E5-2640 v4 10-Core + HT with 128GB RAM (login node) with 3.7TB local user storage /mnt/storage1
homes:    2*2.50GHz Intel Xeon E5-2609 v2  4-Core      with 32GB RAM shared home directories: 8.7TB exported to the ui's as /terabig
nfs02:    1*2.40GHz Intel Xeon E5-2609 v2  4-Core      with 16GB RAM shared storage area:      44TB exported to the ui's as /ngse5
 
'''Virtualisation service''':
kvm02      2*2.67GHz Intel Xeon X5650      6-core +HT with  64GB with KVM and 3.5T storage for guests     
 
  '''Network''':
  2 Brocade FCX648 (4*10GbE, 48*1GbE each) with 2*10GbE uplink (Scienze DMZ)
 
  2 Dell Force10 (48+48*1GbE vLAN stack) private LAN ce02.lhep CLUSTER)
  2 Dell Force10 (2*24+24*1GbE vLANs stack) private LAN ce01.lhep CLUSTER, 2nd vLAN unused)
  1 Mellanox QDR (30*40Gbit) private IB LAN ce01.lhep CLUSTER
  x QDR switches built-in the SUN Blade X6048 rack. private IB LAN ce01.lhep CLUSTER
 
* '''Benchmark HEPspec06''' (weighed average across all node types):


             '''bdii.lhep''':
             '''ce01.lhep'''
             service running on dpm.lhep            
          NODES * TYPE (cores)  HEPSPEC/node (total)  HEPSPEC/core  CPU model
            4 * 32-core  (128)    370.36  (1481.44)    11.57        Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz "Ivi-Bridge EP"
             8 * 32-core  (256)    406.31  (3250.48)    12.7          Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz "Haswell EP"
          12 * 40-core  (480)    496.28  (5955.36)    12.41        Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz "Broadwell EP"
          12 * 32-core  (384)    366.08  (4392.96)    11.44        Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz  "Sandy Bridge EP"
          54 * 32-core  (1728)    366.08 (19768.32)    11.44        Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz  "Sandy Bridge EP"
          39 * 40-core  (1560)    488.40 (19047.60)    12.21        Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz "Ivi Bridge EP"
            4 * 256-core (1024)  3158.79 (12635.16)    12.34        AMD EPYC 7742 64-Core Processor
          =========================================================
          133           5560            66531.32


* Hardware :
          In arc.conf:
            '''ce01.lhep''':
          [infosys/glue1]
            96 SUN Blade X6048 (workers, ~1/2 dead)
          processor_other_description=Cores=41.8, Benchmark=11.97-HEP-SPEC06
            6 IBM x3650 M3 (workers)
          ...
            12 Dalco (workers), 6 Dalco Terraserver (Lustre)
          [queue:CLUSTER]
            12-core Xeon E5645 @ 2.40GHz FrontEnd
          benchmark=HEPSPEC 11.97
            '''ce02.lhep''':
          benchmark=Si2k 2993
            26 SUN Fire X2200 (Lustre and workers)
            20 SUN Blade X8840 (workers)
            47 DELL PE 1950
            8-core Xeon E5620 @ 2.40GHz FrontEnd.
            '''dpm.lhep''':
            8-core Xeon E5620 DPM head node
            6 Xeon X5650 (6-core) with 18 12-port Areca 1280/1880 (RAID6) each
            1 Xeon E5-2609 v2 with 1 Areca 1880 (RAID6) (DPM disk servers).
            '''Network''':
            Dell PowerConnect 2724 (WAN), NetGear ProSafe GSM7248R and GS748T (LAN) switches.


* Benchmark (not measured):
          [queue:CLUSTER-HM]
            Quad-Core AMD Opteron(tm) Processor 2376, 2.3GHz (SUN Fire X2200): 5.676 HEP-SPEC06/core (*)
          benchmark=HEPSPEC 12.21
            Quad-core AMD Opteron 8356, 2.3GHz (SunBlade X8440): 7.4c97 HEP-SPEC/core (**)
          benchmark=Si2k 3052
           
          allowaccess=uboone
            (*)  http://hepix.caspur.it/afs/hepix.org/project/ptrack/spec-cpu2000.html (first row in table, average, SL4 x86_64)
          allowaccess=dune
            (**) https://wiki.chipp.ch/twiki/bin/view/LCGTier2/BenchMarks
          allowaccess=ops


* Available resources (workers):
          [queue:CLUSTER-GPU]
            19*8  AMD Opteron(tm) 2376:        862.752 HEP-SPEC06
          benchmark=HEPSPEC 11.97
            20*16 Quad-core AMD Opteron 8356:  2399.04 HEP-SPEC06
          benchmark=Si2k 2993
            ---------------------------------------------------------------------------------------------------------
            Total: 3261.8 HEP-SPEC06


* Network : Gateway 130.92.139.1. DNS 130.92.9.53 (default), 130.92.9.52. Subnet mask 255.255.255.0
           
* GOCDB Entry: https://gocdb4.esc.rl.ac.uk/portal/index.php?Page_Type=View_Object&object_id=476&grid_id=0
* ROC: NGI_CH - Certification status: Certified.




Line 63: Line 151:
! Host !! IP !! Service !!  OS !! CPU !! Comment  
! Host !! IP !! Service !!  OS !! CPU !! Comment  
|-
|-
! lheplsw1.unibe.ch !!  130.92.139.201  !!  48-Port Switch  !!    !! Brocade FCX648 !! Connected to eth1 on FE (on 10GbE NIC) !!  
! lheplsw1.unibe.ch !!  130.92.139.201  !!  48+4 Port Switch  !!    !! Brocade FCX648 !! Connected to eth0 (10GbE NIC) on ce01 and eth2 (10GbE NIC) on ce02 !!
|-
! ce01.lhep.unibe.ch !!  130.92.140.201  !!  Rocks Front End (FE) 6.2 x86_64 !!  SLC 6.7 x86_64 !! 2 Quad Xeon E5645 @2.4GHz !! Rocks FE, SLURM master, ARC CE, Web Server, Ganglia
|-
! ce02.lhep.unibe.ch !!  130.92.140.202  !!  Rocks Front End (FE) 6.2 x86_64 !!  SLC 6.7 x86_64 !! 2 Quad Xeon E5620 @2.4GHz !! Rocks FE, SLURM master, ARC CE, Web Server, Ganglia
|-
! nfs01.lhep.unibe.ch !!  130.92.140.31  !! 10TB NFS server !!  SLC 6.3 x86_64 !! 1 Quad Xeon E5620 @2.4GHz !! Obsolete as sw server, some backups (also nfs-2-1, 10.2.255.100)
|-
! nfs02.lhep.unibe.ch !!  130.92.140.32  !! 37TB NFS server !!  SLC 6.3 x86_64 !! 1 Quad Xeon E5530 @2.4GHz !! Local NFS server for ATLAS, exports /ngse5
|-
! mds-2-1 !!  10.1.255.200  !! 503GB Lustre MDS/MDT ce01 !! SLC 6.7 x86_64 !! 1 Quad Xeon E5620 @2.4GHz !! MDS/MDT (/dev/mapper/vg00-mdt LV on SSDs for the MDT), Managed by ce01
|-
! mds-0-1 !!  10.1.255.200  !! 110GB Lustre MDS/MDT ce02 !! SLC 6.7 x86_64 !! 1 Quad Xeon E5620 @2.4GHz !! MDS/MDT (/dev/sda2 LV on SSDs for the MDT), Managed by ce02
|-
! oss-2-[1-6] !! 10.1.255.[11-16]  !! Lustre OSS ce01 !! SLC 6.7 x86_64 !! 2 Exa Xeon E5-2620 v2 @2.1GHz !! Managed by ce01
|-
|-
! ce.lhep.unibe.ch !! 130.92.139.200 !! Rocks Front End (FE) 5.3 x86_64 !! CentOS 5.3 x86_64 !! 2 Quad Xeon E5620 @2.4GHz !! Rocks FE, SGE, ARC FE, Web Server, Ganglia
! oss-0-[1-12] !! 10.1.255.[11-22] !! Lustre OSS ce02 !! SLC 6.7 x86_64 !! 2 Dual AMD Opteron 2218 or 2376  @2.3GHz !! Managed by ce02
|-
|-
! !! 10.1.225.254  !! 48 Ports Switch  !!   !! !!  
! wn-2-[1-96] !! 10.1.255.[101-196] !! Worker Nodes ce01 !! SLC 6.7 x86_64 !! 2 Quad Xeon E5540 @2.53GHz !! Managed by ce01
|-
|-
! nas-0-2 !! 10.1.255.253  !! 10TB NFS server !! CentOS 5.3 !! 1 Quad Xeon E5620 @2.4GHz !! ATLAS and VO software (also lhepat02.unibe.ch)
! wn-2-[101-106] !! 10.1.255.[91-96] !! Worker Nodes ce01 !! SLC 6.7 x86_64 !! 2 Exa Xeon X5650 @2.67GHz !! Managed by ce01
|-
|-
! nas-0-1 !! 10.1.255.252  !! 1.1TB Lustre MDS/MDT !! CentOS 5.3 !! 1 Quad Xeon E5620 @2.4GHz !! MDS/MDT (currently not in production)
! wn-2-[111-114] !! 10.1.255.[201-204] !! Worker Nodes ce01 !! SLC 6.7 x86_64 !! 4 Quad Xeon E5-2650 v2 @2.6GHz !! Managed by ce01
|-
|-
! compute-0-0/2 !! 10.1.255.xxx  !! Lustre !! CentOS 5.3 !! 2 Dual AMD Opteron 2214 !! Managed by ce.lhep.unibe.ch
! wn-2-[115-122] !! 10.1.255.[205-212] !! Worker Nodes ce01 !! SLC 6.7 x86_64 !! 4 Quad Xeon E5-2630 v3 @2.4GHz !! Managed by ce01
|-
|-
! compute-0-3 !! 10.1.255.248 !! Lustre MDS/MDT !! CentOS 5.3 !! 2 Dual AMD Opteron 2214 !! Managed by ce.lhep.unibe.ch
! wn-0-[1-15] !! 10.1.255.[101-115] !! Worker Nodes ce02 !! SLC 6.7 x86_64 !! 2 Dual AMD Opteron 2376 @2.3GHz !! Managed by ce02
|-
|-
! compute-0-4/5/6/7 !! 10.1.255.xxx  !! Lustre  !! CentOS 5.3 !! 2 Quad AMD Opteron 2376 !! Managed by ce.lhep.unibe.ch
! wn-0-[16-62] !! 10.1.255.[116-162] !! Worker Nodes ce02 !! SLC 6.7 x86_64 !! 2 Quad Xeon E5420 @2.5GHz !! Managed by ce02
|-
|-
! compute-0-8 to 26 !! 10.1.255.xxx !! Worker Node 0-8 to 0-26  !!  CentOS 5.3 !! 2 Quad AMD Opteron 2376 !! Managed by ce.lhep.unibe.ch
! wn-1-[1-20] !! 10.1.255.[201-220] !! Worker Nodes ce02 !!  SLC 6.7 x86_64 !! 4 Quad AMD Opteron 8356 @2.3GHz !! Managed by ce02
|-
|-
! compute-1-1 to 19 !! 10.1.255.xxx !! Worker Node 1-1 to 1-19 !!  CentOS 5.3 !! 4 Quad AMD Opteron 8356 !! Managed by ce.lhep.unibe.ch
! dpm.lhep.unibe.ch !! 130.92.139.211 !! DPM head node !!  SLC 5.7 !! 2 Quad Xeon E5620 @2.4GHz !! DPM Head node
|-
|-
! dpm.lhep.unibe.ch !! 130.92.139.211 !! DPM head node !!  SLC 5.7 !! 2 Quad Xeon E5620 @2.4GHz !! DPM Head node and disk pool
! dpmdisk[01-03].lhep.unibe.ch !! 130.92.140.[211-213] !! DPM pool nodes !!  SLC 6.8 x86_64 !! 1 Exa Xeon X5650 @2.67GHz !! DPM 60TB disk pools
|-
|-
! dpmdisk01(to 3).lhep.unibe.ch !! 130.92.139.211 to 213 !!  DPM pool node !!  SLC 5.7 !! 1 Hexa Xeon X5650 @2.67GHz !! DPM 60TB disk pool
! dpmdisk04.lhep.unibe.ch !! 130.92.140.214 !!  DPM pool node !!  SLC 6.8 x86_64 !! 1 Quad Xeon E5-2609 v2 @2.50GHz !! DPM 22TB disk pools
|-
|-
! dpmdisk04.lhep.unibe.ch !! 130.92.139.214 !!  DPM pool node !!  SLC 5.7  !! 1 Quad Xeon E5345 @2.33 GHz !! DPM 22TB disk pool
! dpmdisk[05-07].lhep.unibe.ch !! 130.92.140.[215 to 219] !!  DPM pool nodes  !!  SLC 6.8 x86_64 !! Exa Xeon X5650  @2.67GHz !! DPM 90/120TB disk pools
|-
|-
! dpmdisk05(to 9).lhep.unibe.ch !! 130.92.139.215 to 219 !!  DPM pool node  !!  SLC 5.x !! ?? !! Reserved for DPM disk pool
! kvm02.lhep.unibe.ch !! 130.92.140.22 !!  KVM host !!  SLC 6.8 x86_64 !! 2 Exa Xeon X5650 @2.67GHz !! Virtualization host for KVM
|-
|-
! kvm01.lhep.unibe.ch !! 130.92.139.150 !! KVM host !!  SLC 5.7 !! 2 Dual AMD Opteron 2218 @2.6GHz !! Virtualization host for KVM
! ui0[1-3].lhep.unibe.ch !! 130.92.140.1[1-3] !! Interactive login servers !!  SLC 6.8 x86_64 !! 2 Quad Xeon E5620 and 2 Oct Xeon E5-2630 v3 @2.4GHz !! 3.7TB local storage
|-
! homes.lhep.unibe.ch !! 130.92.140.10 !! Home directories server !!  SLC 6.8 x86_64  !! 2 Quad Xeon E5-2609 v2 !! 8.7TB nfs homedirs
|-
! nfs02.lhep.unibe.ch !! 130.92.140.32 !! Local storage server !!  SLC 6.8 x86_64  !! 2 Quad Xeon E5530 !! 37TB nfs storage area
|-
|-
! ... !!  ... !!  ... !!  ...  !! ... !!  ...|-
! ... !!  ... !!  ... !!  ...  !! ... !!  ...|-


|}
|}

Latest revision as of 23:07, 6 April 2021

BERN ATLAS Tier-2, Tier 3 - 2020 Systems

  • Grid HT Cluster ce01 (5.5k cores)
    • Node summary
    • Scheduler: SLURM 18.08.8-1.el7 (scheduler on ce01)
    • Shared File System: Lustre 2.12.2-1.el7
      • 1 x MDS, 6 x OSS: 204 x 1T HDDs (184T)
  • Grid Compute Elements ce01.lhep.unibe.ch, ce02.lhep.unibe.ch, ce03.lhep.unibe.ch
ce01.lhep.unibe.ch has address 130.92.140.201
ce02.lhep.unibe.ch has address 130.92.140.202
ce03.lhep.unibe.ch has address 130.92.140.203
  • Grid Storage Element dpm.lhep.unibe.ch (2PB)
dpm.lhep.unibe.ch has address 130.92.140.100
dpm.lhep.unibe.ch has IPv6 address 2001:620:400:fea8::100
  • Grid Information System bdii.lhep.unibe.ch 
# host bdii.lhep.unibe.ch
  bdii.lhep.unibe.ch has address 130.92.140.121
  bdii.lhep.unibe.ch has address 130.92.140.122
  • Squid for CVMFS cvmfs.lhep.unibe.ch
# host cvmfs.lhep.unibe.ch
  cvmfs.lhep.unibe.ch has address 130.92.252.18
  cvmfs.lhep.unibe.ch has address 130.92.252.19
  • Interactive services
ui00.lhep.unibe.ch has address 130.92.140.20
ui01.lhep.unibe.ch has address 130.92.140.11
ui01.lhep.unibe.ch has IPv6 address 2001:620:400:fea8::11
ui02.lhep.unibe.ch has address 130.92.140.12
ui03.lhep.unibe.ch has address 130.92.140.13
ui04.lhep.unibe.ch has address 130.92.140.14
ui05.lhep.unibe.ch has address 130.92.140.15


EGI:


MONITORING:


NETWORK: 

• IPv4: 130.92.140.0/24, GW: 130.92.140.1 (130.92.140.5 - 130.92.140.250)
• IPv6: 2001:620:400:fea8::/64, GW: 2001:620:400:fea8::1 (2001:620:400:fea8::10 - 2001:620:400:fea8::109)


SYSTEMS' TOPOLOGY: Media:Topology-dec2020.png - Updated Dec 2020


SYSTEMS' POWER BUDGET: Media:PowerBudget-apr2021.png - Updated Apr 2021


  • Hardware Summary :
ce01.lhep CLUSTER:
ce01:      2*2.40GHz Intel Xeon E5645 6-Core with 24GB RAM - ROCKS 7.0 and Nordugrid ARC 6.10.2-2.el7 (epel)
ce02:      2*2.60GHz Intel Xeon E5-2650 v2 8-Core +HT with 64GB RAM - Nordugrid ARC 6.10.2-2.el7 (epel)
ce03:      2*2.60GHz Intel Xeon E5-2670  0 8-Core +HT with 96GB RAM - Nordugrid ARC 6.10.2-2.el7 (epel)
mds-2-1:   1*2.40GHz Intel Xeon E5620 4-Core with 24GB RAM for Lustre MDS (lustre-2.12.2-1.el7)
oss-2-x:   2*2.10GHz Intel Xeon E5-2620 v2 6-Core with 32GB RAM for Lustre OSS (lustre-2.12.2-1.el7, 6 nodes each with 34*HDD, 31TB sw raid0)
wn-1-1:    2*2.10GHz Intel Xeon E5-2620 v4 8-core + HT (1 node) 32 cores with total 256GB RAM
           8*GeForce RTX 2080ti GPUs
wn-1-x     2*2.40GHz Intel Xeon E5620  4-core + HT ( 4 nodes)  64 cores with 2.0GB RAM/core
wn-2-x:                    
           2*2.60GHz Intel Xeon E5-2650 v2   8-core + HT (4 Nodes)   128 cores with 2.0GB RAM/core   "Ivi-Bridge EP"   Launch date: Q3'13
           2*2.40GHz Intel Xeon E5-2630 v3   8-core + HT (8 Nodes)   256 cores with 2.0GB RAM/core   "Haswell EP"      Launch date: Q3'14
           2*2.20GHz Intel Xeon E5-2630 v4  10-core + HT (12 Nodes)  480 cores with 2.4GB RAM/core   "Broadwell EP"    Launch date: Q1'16
           2*2.60GHz Intel Xeon E5-2670  0   8-core + HT (12 Nodes)  384 cores with 2.0GB RAM/core   "Sandy Bridge EP" Launch date: Q3'12
           2*2.60GHz Intel Xeon E5-2670  0   8-core + HT (54 Nodes) 1728 cores with 2.0GB RAM/core   "Sandy Bridge EP" Launch date: Q3'12
           2*2.80GHz Intel Xeon E5-2680 v2  10-core + HT (39 Nodes) 1560 cores with 3.2GB RAM/core   "Ivi Bridge EP"   Launch date: Q3'13
           2*2.80GHz AuthenticAMD EPYC 7742 64-core + HT ( 4 Nodes) 1024 cores with 2.0GB RAM/core   "AMD EPYC 7742"   Launch Date: Q3'19

dpm.lhep STORAGE ELEMENT:
dpm            2*2.60GHz Intel Xeon E5-2650 v2  8-core + HT with 96GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel )
dpmdisk0[1-3]  1*2.67GHz Intel Xeon X5650       6-core      with 24GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 3* 20TB (18.19TB net in DPM) RAID6 each
dpmdisk04      2*2.50GHZ Intel Xeon E5-2609 v2  4-core      with 32GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 1* 60TB ( 54.57TB net in DPM) RAID6 
dpmdisk0[5-6]  1*2.67GHz Intel Xeon X5650       6-core      with 24GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 3* 30TB ( 27.28TB net in DPM) RAID6 each
dpmdisk07      1*2.67GHz Intel Xeon X5650       6-core + HT with 24GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 3* 40TB ( 36.38TB net in DPM) RAID6
dpmdisk08      2*2.10GHz Intel Xeon E5-2620 v4  8-core + HT with 32GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 2*100TB ( 90.95TB net in DPM) RAID6
                                                                                                                         1*140TB (127.33TB net in DPM) RAID6
dpmdisk09      2*2.10GHz Intel Xeon E5-2620 v4  8-core + HT with 96GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 2*378TB (343.76TB net in DPM) RAID6

bdii.lhep INFORMATION SYSTEM:
bdii0[1-2]     2*VMs run by ID with emi-bdii-site-1.0.1-1.el6 (UMD-4)

cvmfs.lhep SQUID FOR CVMFS:
cvmfs0[1-2]    2*VMs run by ID with frontier-squid-4.13-5.1 (cern-frontier)

Interactive services:
ui01:      2*2.40GHz Intel Xeon E5620       4-Core + HT with  64GB RAM (login node) with 3.7TB local user storage /mnt/storage1 (incl. backup of terabig)
ui02:      2*2.40GHz Intel Xeon E5620       4-Core + HT with  64GB RAM (login node) with 3.7TB local user storage /mnt/storage1 
ui03:      2*2.40GHz Intel Xeon E5-2630 v3  8-Core + HT with 128GB RAM (login node) with 3.7TB local user storage /mnt/storage1
ui04:      2*2.40GHz Intel Xeon E5-2640 v4 10-Core + HT with 128GB RAM (login node) with 3.7TB local user storage /mnt/storage1 
ui05:      2*2.40GHz Intel Xeon E5-2640 v4 10-Core + HT with 128GB RAM (login node) with 3.7TB local user storage /mnt/storage1 
homes:     2*2.50GHz Intel Xeon E5-2609 v2  4-Core       with 32GB RAM shared home directories: 8.7TB exported to the ui's as /terabig 
nfs02:     1*2.40GHz Intel Xeon E5-2609 v2  4-Core       with 16GB RAM shared storage area:      44TB exported to the ui's as /ngse5

Virtualisation service:
kvm02      2*2.67GHz Intel Xeon X5650      6-core +HT with   64GB with KVM and 3.5T storage for guests      

 Network:
 2 Brocade FCX648 (4*10GbE, 48*1GbE each) with 2*10GbE uplink (Scienze DMZ)

 2 Dell Force10 (48+48*1GbE vLAN stack) private LAN ce02.lhep CLUSTER)
 2 Dell Force10 (2*24+24*1GbE vLANs stack) private LAN ce01.lhep CLUSTER, 2nd vLAN unused)
 1 Mellanox QDR (30*40Gbit) private IB LAN ce01.lhep CLUSTER
 x QDR switches built-in the SUN Blade X6048 rack. private IB LAN ce01.lhep CLUSTER
  • Benchmark HEPspec06 (weighed average across all node types):
           ce01.lhep
          NODES * TYPE (cores)   HEPSPEC/node (total)  HEPSPEC/core  CPU model
           4 * 32-core   (128)    370.36  (1481.44)    11.57         Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz "Ivi-Bridge EP"
           8 * 32-core   (256)    406.31  (3250.48)    12.7          Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz "Haswell EP"
          12 * 40-core   (480)    496.28  (5955.36)    12.41         Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz "Broadwell EP"
          12 * 32-core   (384)    366.08  (4392.96)    11.44         Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz  "Sandy Bridge EP"
          54 * 32-core  (1728)    366.08 (19768.32)    11.44         Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz  "Sandy Bridge EP"
          39 * 40-core  (1560)    488.40 (19047.60)    12.21         Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz "Ivi Bridge EP"
           4 * 256-core (1024)   3158.79 (12635.16)    12.34         AMD EPYC 7742 64-Core Processor
          =========================================================
          133            5560             66531.32

          In arc.conf:
          [infosys/glue1]
          processor_other_description=Cores=41.8, Benchmark=11.97-HEP-SPEC06
          ...
          [queue:CLUSTER]
          benchmark=HEPSPEC 11.97
          benchmark=Si2k 2993

          [queue:CLUSTER-HM]
          benchmark=HEPSPEC 12.21
          benchmark=Si2k 3052
          allowaccess=uboone
          allowaccess=dune
          allowaccess=ops

          [queue:CLUSTER-GPU]
          benchmark=HEPSPEC 11.97
          benchmark=Si2k 2993



Host IP Service OS CPU Comment
lheplsw1.unibe.ch 130.92.139.201 48+4 Port Switch Brocade FCX648 Connected to eth0 (10GbE NIC) on ce01 and eth2 (10GbE NIC) on ce02
ce01.lhep.unibe.ch 130.92.140.201 Rocks Front End (FE) 6.2 x86_64 SLC 6.7 x86_64 2 Quad Xeon E5645 @2.4GHz Rocks FE, SLURM master, ARC CE, Web Server, Ganglia
ce02.lhep.unibe.ch 130.92.140.202 Rocks Front End (FE) 6.2 x86_64 SLC 6.7 x86_64 2 Quad Xeon E5620 @2.4GHz Rocks FE, SLURM master, ARC CE, Web Server, Ganglia
nfs01.lhep.unibe.ch 130.92.140.31 10TB NFS server SLC 6.3 x86_64 1 Quad Xeon E5620 @2.4GHz Obsolete as sw server, some backups (also nfs-2-1, 10.2.255.100)
nfs02.lhep.unibe.ch 130.92.140.32 37TB NFS server SLC 6.3 x86_64 1 Quad Xeon E5530 @2.4GHz Local NFS server for ATLAS, exports /ngse5
mds-2-1 10.1.255.200 503GB Lustre MDS/MDT ce01 SLC 6.7 x86_64 1 Quad Xeon E5620 @2.4GHz MDS/MDT (/dev/mapper/vg00-mdt LV on SSDs for the MDT), Managed by ce01
mds-0-1 10.1.255.200 110GB Lustre MDS/MDT ce02 SLC 6.7 x86_64 1 Quad Xeon E5620 @2.4GHz MDS/MDT (/dev/sda2 LV on SSDs for the MDT), Managed by ce02
oss-2-[1-6] 10.1.255.[11-16] Lustre OSS ce01 SLC 6.7 x86_64 2 Exa Xeon E5-2620 v2 @2.1GHz Managed by ce01
oss-0-[1-12] 10.1.255.[11-22] Lustre OSS ce02 SLC 6.7 x86_64 2 Dual AMD Opteron 2218 or 2376 @2.3GHz Managed by ce02
wn-2-[1-96] 10.1.255.[101-196] Worker Nodes ce01 SLC 6.7 x86_64 2 Quad Xeon E5540 @2.53GHz Managed by ce01
wn-2-[101-106] 10.1.255.[91-96] Worker Nodes ce01 SLC 6.7 x86_64 2 Exa Xeon X5650 @2.67GHz Managed by ce01
wn-2-[111-114] 10.1.255.[201-204] Worker Nodes ce01 SLC 6.7 x86_64 4 Quad Xeon E5-2650 v2 @2.6GHz Managed by ce01
wn-2-[115-122] 10.1.255.[205-212] Worker Nodes ce01 SLC 6.7 x86_64 4 Quad Xeon E5-2630 v3 @2.4GHz Managed by ce01
wn-0-[1-15] 10.1.255.[101-115] Worker Nodes ce02 SLC 6.7 x86_64 2 Dual AMD Opteron 2376 @2.3GHz Managed by ce02
wn-0-[16-62] 10.1.255.[116-162] Worker Nodes ce02 SLC 6.7 x86_64 2 Quad Xeon E5420 @2.5GHz Managed by ce02
wn-1-[1-20] 10.1.255.[201-220] Worker Nodes ce02 SLC 6.7 x86_64 4 Quad AMD Opteron 8356 @2.3GHz Managed by ce02
dpm.lhep.unibe.ch 130.92.139.211 DPM head node SLC 5.7 2 Quad Xeon E5620 @2.4GHz DPM Head node
dpmdisk[01-03].lhep.unibe.ch 130.92.140.[211-213] DPM pool nodes SLC 6.8 x86_64 1 Exa Xeon X5650 @2.67GHz DPM 60TB disk pools
dpmdisk04.lhep.unibe.ch 130.92.140.214 DPM pool node SLC 6.8 x86_64 1 Quad Xeon E5-2609 v2 @2.50GHz DPM 22TB disk pools
dpmdisk[05-07].lhep.unibe.ch 130.92.140.[215 to 219] DPM pool nodes SLC 6.8 x86_64 Exa Xeon X5650 @2.67GHz DPM 90/120TB disk pools
kvm02.lhep.unibe.ch 130.92.140.22 KVM host SLC 6.8 x86_64 2 Exa Xeon X5650 @2.67GHz Virtualization host for KVM
ui0[1-3].lhep.unibe.ch 130.92.140.1[1-3] Interactive login servers SLC 6.8 x86_64 2 Quad Xeon E5620 and 2 Oct Xeon E5-2630 v3 @2.4GHz 3.7TB local storage
homes.lhep.unibe.ch 130.92.140.10 Home directories server SLC 6.8 x86_64 2 Quad Xeon E5-2609 v2 8.7TB nfs homedirs
nfs02.lhep.unibe.ch 130.92.140.32 Local storage server SLC 6.8 x86_64 2 Quad Xeon E5530 37TB nfs storage area
... ... ... ... ... -
Retrieved from "https://wiki.lhep.unibe.ch/index.php?title=AEC-LHEP_Hardware_information&oldid=684"