AEC-LHEP Hardware information: Difference between revisions

From LHEP Wiki
Jump to navigation Jump to search
No edit summary
No edit summary
 
(63 intermediate revisions by the same user not shown)
Line 1: Line 1:


BERN ATLAS Tier-2 2016 Systems - '''ce0.lhep.unibe.ch''', '''ce02.lhep.unibe.ch''', '''dpm.lhep.unibe.ch''', '''bdii.lhep.unibe.ch'''
BERN ATLAS Tier-2, Tier 3 - 2020 Systems


* [http://ce01.lhep.unibe.ch/ganglia/ Ganglia ce01]
* Grid HT Cluster '''ce01''' (5.5k cores)
* [http://ce02.lhep.unibe.ch/ganglia/ Ganglia ce02]
** Node summary
* [http://nagios.lhep.unibe.ch/nagios/ Nagios UNIBE-LHEP] (restricted)
** Scheduler: SLURM 18.08.8-1.el7 (scheduler on ce01)
* [http://www.nordugrid.org/monitor/atlas/ NorduGrid ATLAS]
** Shared File System: Lustre 2.12.2-1.el7
*** 1 x MDS, 6 x OSS: 204 x 1T HDDs (184T)


* Grid Compute Elements '''ce01.lhep.unibe.ch, ce02.lhep.unibe.ch, ce03.lhep.unibe.ch'''
ce01.lhep.unibe.ch has address 130.92.140.201
ce02.lhep.unibe.ch has address 130.92.140.202
ce03.lhep.unibe.ch has address 130.92.140.203
* Grid Storage Element '''dpm.lhep.unibe.ch ''' (2PB)
dpm.lhep.unibe.ch has address 130.92.140.100
dpm.lhep.unibe.ch has IPv6 address 2001:620:400:fea8::100
* Grid Information System '''bdii.lhep.unibe.ch '''
# host bdii.lhep.unibe.ch
  bdii.lhep.unibe.ch has address 130.92.140.121
  bdii.lhep.unibe.ch has address 130.92.140.122
* Squid for CVMFS '''cvmfs.lhep.unibe.ch'''
# host cvmfs.lhep.unibe.ch
  cvmfs.lhep.unibe.ch has address 130.92.252.18
  cvmfs.lhep.unibe.ch has address 130.92.252.19
* Interactive services
ui00.lhep.unibe.ch has address 130.92.140.20
ui01.lhep.unibe.ch has address 130.92.140.11
ui01.lhep.unibe.ch has IPv6 address 2001:620:400:fea8::11
ui02.lhep.unibe.ch has address 130.92.140.12
ui03.lhep.unibe.ch has address 130.92.140.13
ui04.lhep.unibe.ch has address 130.92.140.14
ui05.lhep.unibe.ch has address 130.92.140.15


* [https://goc.egi.eu/portal/index.php?Page_Type=Site&id=524 GOCDB Entry]
* ROC: NGI_CH - Certification status: Certified.
* Network : Gateway 130.92.140.1. DNS 130.92.9.53 (default), 130.92.9.52. Subnet mask 255.255.255.0
* [http://web.lhep.unibe.ch/topology-june2016.pdf Systems' Topology]


EGI:
* [https://goc.egi.eu/portal/index.php?Page_Type=Site&id=524 GOCDB Entry] - UNIBE-LHEP
* ROC: [https://goc.egi.eu/portal/index.php?Page_Type=NGI&id=11 NGI_CH] - Certification status: Certified.


* '''Summary :'''
            '''ce01.lhep CLUSTER''':
            ce01:      2*2.40GHz Intel Xeon E5645 6-Core with 24GB RAM - ROCKS 6.2 and nordugrid-arc-5.0.3-1.el6.x86_64 (epel)
            mds-2-1:  1*2.40GHz Intel Xeon E5620 4-Core with 24GB RAM for Lustre MDS (lustre-2.5.3)
            oss-2-x:  2*2.10GHz Intel E5-2620 v2 4-Core with 16GB RAM for Lustre OSS (lustre-2.5.3, 6 nodes each with 32*HDD, 31TB sw raid0)
            wn-2-x:
                      2*2.53GHz Intel Xeon E5540  8-core + HT (45 Nodes) 540 cores with 2GB RAM/core (or 720 cores with 1.33GB RAM/core)
                      2*2.53GHz Intel Xeon E5540  8-core + HT ( 4 Nodes)  32 cores with 2GB RAM/core (same HW as above with 1 dead CPU)
                      2*2.67GHz Intel Xeon X5650 12-core + HT ( 6 Nodes) 144 cores with 2GB RAM/core
                      2*2.60GHz Intel Xeon E5-2650 v2 8-core + HT (4 Nodes) 128 cores with 2GB RAM/core
                      2*2.40GHz Intel Xeon E5-2630 v3 8-core + HT (8 Nodes) 256 cores with 2GB RAM/core


            '''ce02.lhep CLUSTER''':
MONITORING:
            ce02      2*2.40GHz Intel Xeon E5620 4-core with 24GB RAM - ROCKS 6.2 and nordugrid-arc-5.0.4-1.el6.x86_64 (epel)
* [http://www-f9.ijs.si/atlas/grid/monitor/loadmon.php?display=vo=Switzerland NorduGrid Swiss ATLAS]
            mds-0-1    1*2.40GHz Intel Xeon E5620 4-core with 24GB RAM for Lustre MDS (lustre-2.5.3-2)
* [http://wiki.lhep.unibe.ch/index.php/AEC-LHEP_Monitoring_information More monitoring information TO BE REVIEWED]
            oss-0-x    2*2.30GHz AMD Opteron 2376 4-core with 16GB RAM for Lustre OSS (lustre-2.5.3, 11 nodes each with 1*HD 2TB)
            wn-0-x:
                      2*2.30GHz AMD Opteron 2376 4-core (15 Nodes) 120 cores with 2GB RAM/core
                      4*2.30GHz AMD Opteron 8356 4-core (20 Nodes) 320 cores with 2GB RAM/core
                      1*2.50GHz Intel Xeon E5420 4-core + HT (47 Nodes) 376 cores with 2GB RAM/core


            '''dpm.lhep STORAGE ELEMENT''':
            dpm        2*2.40GHz Intel Xeon E5620 4-core + HT with 16GB RAM and dpm-1.8.8-4.el5 (epel)
            dpmdisk0x  1*2.67GHz Intel Xeon X5650 6-core with 12GB RAM and dpm-1.8.8-4.el6.x86_64(epel), 6 nodes, 3*raid6 each, 436.52TB net in DPM)
            dpmdisk04  1*2.33GHZ Intel Xeon E5345 4-core with  8GB RAM and dpm-1.8.10-1.el6.x86_64 (epel), 1 node 1*raid6, 21.83TB net in DPM)


            '''bdii.lhep INFORMATION SYSTEM''':
NETWORK:
            service running on dpm.lhep: emi-bdii-site-1.0.1-1.el5     
• IPv4: 130.92.140.0/24, GW: 130.92.140.1 (130.92.140.5 - 130.92.140.250)
• IPv6: 2001:620:400:fea8::/64, GW: 2001:620:400:fea8::1 (2001:620:400:fea8::10 - 2001:620:400:fea8::109)


* '''Hardware :'''
 
            '''ce01.lhep''':
SYSTEMS' TOPOLOGY:
            1 Dalco FrontEnd 12-core Xeon E5645 @ 2.40GHz
[[Media:Topology-dec2020.png]] - Updated Dec 2020
            1 Dalco Lustre mds 4-core Xeon E5620 @ 2.40GHz
 
             96 SUN Blade X6048 (workers, ~1/2 dead)
 
            6 IBM x3650 M3 (workers)
SYSTEMS' POWER BUDGET:
             12 Dalco (workers)
[[Media:PowerBudget-apr2021.png]] - Updated Apr 2021
            6 Dalco Terraserver (Lustre)
 
             '''ce02.lhep''':
 
            1 Dalco FrontEnd 8-core Xeon E5620 @ 2.40GHz
* '''Hardware Summary :'''
            1 Dalco Lustre mds 4-core Xeon E5620 @ 2.40GHz
'''ce01.lhep CLUSTER''':
            26 SUN Fire X2200 (Lustre and workers)
ce01:      2*2.40GHz Intel Xeon E5645 6-Core with 24GB RAM - ROCKS 7.0 and Nordugrid ARC 6.10.2-2.el7 (epel)
            20 SUN Blade X8840 (workers)
ce02:      2*2.60GHz Intel Xeon E5-2650 v2 8-Core +HT with 64GB RAM - Nordugrid ARC 6.10.2-2.el7 (epel)
            47 DELL PE 1950 (workers)
ce03:      2*2.60GHz Intel Xeon E5-2670  0 8-Core +HT with 96GB RAM - Nordugrid ARC 6.10.2-2.el7 (epel)
            '''dpm.lhep''':
mds-2-1:  1*2.40GHz Intel Xeon E5620 4-Core with 24GB RAM for Lustre MDS (lustre-2.12.2-1.el7)
            1 DELL PE 1950 (DPM head node)  
oss-2-x:  2*2.10GHz Intel Xeon E5-2620 v2 6-Core with 32GB RAM for Lustre OSS (lustre-2.12.2-1.el7, 6 nodes each with 34*HDD, 31TB sw raid0)
            6 Dalco Terraserver (DPM pool nodes) with 3*12-port Areca 1280/1880 (RAID6) each
wn-1-1:    2*2.10GHz Intel Xeon E5-2620 v4 8-core + HT (1 node) 32 cores with total 256GB RAM
            1 Dalco Terraserver (DPM pool node)  with 1*Areca 1880 (RAID6)
            8*GeForce RTX 2080ti GPUs
            '''Network''':
wn-1-x    2*2.40GHz Intel Xeon E5620 4-core + HT ( 4 nodes)  64 cores with 2.0GB RAM/core
            2 Brocade FCX648 (4*10GbE, 48*1GbE each) with 1*10GbE uplink (public LAN)
wn-2-x:                   
            1 NetGear ProSafe GSM7248R and GS748T (public LAN)
             2*2.60GHz Intel Xeon E5-2650 v2  8-core + HT (4 Nodes)  128 cores with 2.0GB RAM/core  "Ivi-Bridge EP"  Launch date: Q3'13
            2 Dell Force10 (48+48*1GbE vLAN stack) private LAN ce02.lhep CLUSTER)
            2*2.40GHz Intel Xeon E5-2630 v3  8-core + HT (8 Nodes)   256 cores with 2.0GB RAM/core  "Haswell EP"      Launch date: Q3'14
            2 Dell Force10 (2*24+24*1GbE vLANs stack) private LAN ce01.lhep CLUSTER, 2nd vLAN unused)
            2*2.20GHz Intel Xeon E5-2630 v4  10-core + HT (12 Nodes) 480 cores with 2.4GB RAM/core  "Broadwell EP"    Launch date: Q1'16
            1 Mellanox QDR (30*40Gbit) private IB LAN ce01.lhep CLUSTER
             2*2.60GHz Intel Xeon E5-2670  0  8-core + HT (12 Nodes)  384 cores with 2.0GB RAM/core  "Sandy Bridge EP" Launch date: Q3'12
            x QDR switches built-in the SUN Blade X6048 rack. private IB LAN ce01.lhep CLUSTER
            2*2.60GHz Intel Xeon E5-2670  0  8-core + HT (54 Nodes) 1728 cores with 2.0GB RAM/core  "Sandy Bridge EP" Launch date: Q3'12
            2*2.80GHz Intel Xeon E5-2680 v2  10-core + HT (39 Nodes) 1560 cores with 3.2GB RAM/core  "Ivi Bridge EP"  Launch date: Q3'13
             2*2.80GHz AuthenticAMD EPYC 7742 64-core + HT ( 4 Nodes) 1024 cores with 2.0GB RAM/core  "AMD EPYC 7742"  Launch Date: Q3'19
 
'''dpm.lhep STORAGE ELEMENT''':
dpm            2*2.60GHz Intel Xeon E5-2650 v2  8-core + HT with 96GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel )
dpmdisk0[1-3]  1*2.67GHz Intel Xeon X5650      6-core      with 24GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 3* 20TB (18.19TB net in DPM) RAID6 each
dpmdisk04      2*2.50GHZ Intel Xeon E5-2609 v2  4-core     with 32GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 1* 60TB ( 54.57TB net in DPM) RAID6
dpmdisk0[5-6]  1*2.67GHz Intel Xeon X5650      6-core      with 24GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 3* 30TB ( 27.28TB net in DPM) RAID6 each
dpmdisk07      1*2.67GHz Intel Xeon X5650      6-core + HT with 24GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 3* 40TB ( 36.38TB net in DPM) RAID6
dpmdisk08      2*2.10GHz Intel Xeon E5-2620 v4  8-core + HT with 32GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 2*100TB ( 90.95TB net in DPM) RAID6
                                                                                                                          1*140TB (127.33TB net in DPM) RAID6
dpmdisk09      2*2.10GHz Intel Xeon E5-2620 v4  8-core + HT with 96GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 2*378TB (343.76TB net in DPM) RAID6
 
'''bdii.lhep INFORMATION SYSTEM''':
bdii0[1-2]    2*VMs run by ID with emi-bdii-site-1.0.1-1.el6 (UMD-4)
 
'''cvmfs.lhep SQUID FOR CVMFS''':
cvmfs0[1-2]    2*VMs run by ID with frontier-squid-4.13-5.1 (cern-frontier)
 
'''Interactive services''':
ui01:      2*2.40GHz Intel Xeon E5620      4-Core + HT with  64GB RAM (login node) with 3.7TB local user storage /mnt/storage1 (incl. backup of terabig)
ui02:      2*2.40GHz Intel Xeon E5620      4-Core + HT with  64GB RAM (login node) with 3.7TB local user storage /mnt/storage1
ui03:      2*2.40GHz Intel Xeon E5-2630 v3  8-Core + HT with 128GB RAM (login node) with 3.7TB local user storage /mnt/storage1
ui04:      2*2.40GHz Intel Xeon E5-2640 v4 10-Core + HT with 128GB RAM (login node) with 3.7TB local user storage /mnt/storage1
ui05:      2*2.40GHz Intel Xeon E5-2640 v4 10-Core + HT with 128GB RAM (login node) with 3.7TB local user storage /mnt/storage1
  homes:    2*2.50GHz Intel Xeon E5-2609 v2  4-Core      with 32GB RAM shared home directories: 8.7TB exported to the ui's as /terabig
nfs02:    1*2.40GHz Intel Xeon E5-2609 v2  4-Core      with 16GB RAM shared storage area:      44TB exported to the ui's as /ngse5
 
'''Virtualisation service''':
kvm02      2*2.67GHz Intel Xeon X5650      6-core +HT with  64GB with KVM and 3.5T storage for guests     
 
  '''Network''':
  2 Brocade FCX648 (4*10GbE, 48*1GbE each) with 2*10GbE uplink (Scienze DMZ)
 
  2 Dell Force10 (48+48*1GbE vLAN stack) private LAN ce02.lhep CLUSTER)
  2 Dell Force10 (2*24+24*1GbE vLANs stack) private LAN ce01.lhep CLUSTER, 2nd vLAN unused)
  1 Mellanox QDR (30*40Gbit) private IB LAN ce01.lhep CLUSTER
  x QDR switches built-in the SUN Blade X6048 rack. private IB LAN ce01.lhep CLUSTER


* '''Benchmark HEPspec06''' (weighed average across all node types):
* '''Benchmark HEPspec06''' (weighed average across all node types):


             '''ce01.lhep'''
             '''ce01.lhep'''
            NODES * TYPE (cores)       HEPSPEC/node (total)
          NODES * TYPE (cores)   HEPSPEC/node (total) HEPSPEC/core  CPU model
            8 * 32-core (256)         406.31 (3250.48)
            4 * 32-core  (128)    370.36  (1481.44)    11.57        Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz "Ivi-Bridge EP"
            4 * 32-core (128)         370.36 (1481.44)
            8 * 32-core   (256)   406.31 (3250.48)   12.7          Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz "Haswell EP"
            6 * 24-core (144)         232.50 (1395.00)
          12 * 40-core  (480)    496.28  (5955.36)    12.41        Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz "Broadwell EP"
            4 * 8-core ( 32)         118.48 ( 473.92)
          12 * 32-core   (384)   366.08  (4392.96)    11.44         Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz  "Sandy Bridge EP"
             45 * 12-core (540)         143.06 (6437.07)
          54 * 32-core (1728)   366.08 (19768.32)    11.44        Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz  "Sandy Bridge EP"
            ============================================
          39 * 40-core (1560)   488.40 (19047.60)    12.21        Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz "Ivi Bridge EP"
             67          1100                  13037.91 Installed Capacity
             4 * 256-core (1024)   3158.79 (12635.16)   12.34        AMD EPYC 7742 64-Core Processor
           
          =========================================================
            In arc.conf arc.conf:
          133            5560             66531.32
            processor_other_description='Cores=16.42, Benchmark=11.85-HEP-SPEC06'
 
            ...
          In arc.conf:
            jobreport_options="urbatch:500,archiving:/var/spool/nordugrid/urs-archive-jura,topic:/queue/global.accounting.cpu.central,gocdb_name:UNIBE-LHEP,benchmark_type:HEPSPEC,benchmark_value:11.85,use_ssl:true"
          [infosys/glue1]
           
          processor_other_description=Cores=41.8, Benchmark=11.97-HEP-SPEC06
            '''ce02.lhep'''
          ...
            NODES * TYPE (cores)        HEPSPEC/node (total)
          [queue:CLUSTER]
            20 * 16-core (320)          123.74 (2474.7 )
          benchmark=HEPSPEC 11.97
            15 *  8-core (120)          72.72 (1090.87)
          benchmark=Si2k 2993
            47 *  8-core (376)          72.14 (3390.7 )
 
            ============================================
          [queue:CLUSTER-HM]
            82           816                  6956.27 Installed Capacity
          benchmark=HEPSPEC 12.21
            Total: 3261.8 HEP-SPEC06
          benchmark=Si2k 3052
          allowaccess=uboone
          allowaccess=dune
          allowaccess=ops
 
          [queue:CLUSTER-GPU]
          benchmark=HEPSPEC 11.97
           benchmark=Si2k 2993
 
              
              
            In arc.conf arc.conf:
            processor_other_description='Cores=9.9, Benchmark=8.52-HEP-SPEC06'
            ...
            jobreport_options="urbatch:1000,archiving:/var/spool/nordugrid/urs-archive-jura,topic:/queue/global.accounting.cpu.central,gocdb_name:UNIBE-LHEP,benchmark_type:HEPSPEC,benchmark_value:8.52,use_ssl:true"




Line 139: Line 185:
! dpm.lhep.unibe.ch !! 130.92.139.211 !! DPM head node  !!  SLC 5.7  !! 2 Quad Xeon E5620 @2.4GHz !! DPM Head node
! dpm.lhep.unibe.ch !! 130.92.139.211 !! DPM head node  !!  SLC 5.7  !! 2 Quad Xeon E5620 @2.4GHz !! DPM Head node
|-
|-
! dpmdisk[01-03].lhep.unibe.ch !! 130.92.140.[211-213] !!  DPM pool node !!  SLC 6.8 x86_64  !! 1 Exa Xeon X5650 @2.67GHz !! DPM 60TB disk pools
! dpmdisk[01-03].lhep.unibe.ch !! 130.92.140.[211-213] !!  DPM pool nodes !!  SLC 6.8 x86_64  !! 1 Exa Xeon X5650 @2.67GHz !! DPM 60TB disk pools
|-
! dpmdisk04.lhep.unibe.ch !!  130.92.140.214 !!  DPM pool node !!  SLC 6.8 x86_64  !! 1 Quad Xeon E5-2609 v2 @2.50GHz !!  DPM 22TB disk pools
|-
|-
! dpmdisk04.lhep.unibe.ch !! 130.92.140.214 !!  DPM pool node !!  SLC 6.8 x86_64 !! 1 Quad Xeon E5345 @2.33 GHz !! DPM 22TB disk pools
! dpmdisk[05-07].lhep.unibe.ch !! 130.92.140.[215 to 219] !!  DPM pool nodes  !!  SLC 6.8 x86_64 !! Exa Xeon X5650  @2.67GHz !! DPM 90/120TB disk pools
|-
|-
! dpmdisk[05-07].lhep.unibe.ch !! 130.92.140.[215 to 219] !!  DPM pool node  !!  SLC 6.8 x86_64 !! Exa Xeon X5650 @2.67GHz !! DPM 90/120TB disk pools
! kvm02.lhep.unibe.ch !! 130.92.140.22 !!  KVM host !!  SLC 6.8 x86_64 !! 2 Exa Xeon X5650 @2.67GHz !! Virtualization host for KVM
|-
|-
! kvm01.lhep.unibe.ch !! 130.92.139.150 !! KVM host !!  SLC 6.8 x86_64  !! 2 Dual AMD Opteron 2218 @2.6GHz !! Virtualization host for KVM
! ui0[1-3].lhep.unibe.ch !! 130.92.140.1[1-3] !! Interactive login servers !!  SLC 6.8 x86_64  !! 2 Quad Xeon E5620 and 2 Oct Xeon E5-2630 v3 @2.4GHz !! 3.7TB local storage
|-
! homes.lhep.unibe.ch !! 130.92.140.10 !! Home directories server !!  SLC 6.8 x86_64  !! 2 Quad Xeon E5-2609 v2 !! 8.7TB nfs homedirs
|-
! nfs02.lhep.unibe.ch !! 130.92.140.32 !! Local storage server !!  SLC 6.8 x86_64  !! 2 Quad Xeon E5530 !! 37TB nfs storage area
|-
|-
! ... !!  ... !!  ... !!  ...  !! ... !!  ...|-
! ... !!  ... !!  ... !!  ...  !! ... !!  ...|-


|}
|}

Latest revision as of 23:07, 6 April 2021

BERN ATLAS Tier-2, Tier 3 - 2020 Systems

  • Grid HT Cluster ce01 (5.5k cores)
    • Node summary
    • Scheduler: SLURM 18.08.8-1.el7 (scheduler on ce01)
    • Shared File System: Lustre 2.12.2-1.el7
      • 1 x MDS, 6 x OSS: 204 x 1T HDDs (184T)
  • Grid Compute Elements ce01.lhep.unibe.ch, ce02.lhep.unibe.ch, ce03.lhep.unibe.ch
ce01.lhep.unibe.ch has address 130.92.140.201
ce02.lhep.unibe.ch has address 130.92.140.202
ce03.lhep.unibe.ch has address 130.92.140.203
  • Grid Storage Element dpm.lhep.unibe.ch (2PB)
dpm.lhep.unibe.ch has address 130.92.140.100
dpm.lhep.unibe.ch has IPv6 address 2001:620:400:fea8::100
  • Grid Information System bdii.lhep.unibe.ch 
# host bdii.lhep.unibe.ch
  bdii.lhep.unibe.ch has address 130.92.140.121
  bdii.lhep.unibe.ch has address 130.92.140.122
  • Squid for CVMFS cvmfs.lhep.unibe.ch
# host cvmfs.lhep.unibe.ch
  cvmfs.lhep.unibe.ch has address 130.92.252.18
  cvmfs.lhep.unibe.ch has address 130.92.252.19
  • Interactive services
ui00.lhep.unibe.ch has address 130.92.140.20
ui01.lhep.unibe.ch has address 130.92.140.11
ui01.lhep.unibe.ch has IPv6 address 2001:620:400:fea8::11
ui02.lhep.unibe.ch has address 130.92.140.12
ui03.lhep.unibe.ch has address 130.92.140.13
ui04.lhep.unibe.ch has address 130.92.140.14
ui05.lhep.unibe.ch has address 130.92.140.15


EGI:


MONITORING:


NETWORK: 

• IPv4: 130.92.140.0/24, GW: 130.92.140.1 (130.92.140.5 - 130.92.140.250)
• IPv6: 2001:620:400:fea8::/64, GW: 2001:620:400:fea8::1 (2001:620:400:fea8::10 - 2001:620:400:fea8::109)


SYSTEMS' TOPOLOGY: Media:Topology-dec2020.png - Updated Dec 2020


SYSTEMS' POWER BUDGET: Media:PowerBudget-apr2021.png - Updated Apr 2021


  • Hardware Summary :
ce01.lhep CLUSTER:
ce01:      2*2.40GHz Intel Xeon E5645 6-Core with 24GB RAM - ROCKS 7.0 and Nordugrid ARC 6.10.2-2.el7 (epel)
ce02:      2*2.60GHz Intel Xeon E5-2650 v2 8-Core +HT with 64GB RAM - Nordugrid ARC 6.10.2-2.el7 (epel)
ce03:      2*2.60GHz Intel Xeon E5-2670  0 8-Core +HT with 96GB RAM - Nordugrid ARC 6.10.2-2.el7 (epel)
mds-2-1:   1*2.40GHz Intel Xeon E5620 4-Core with 24GB RAM for Lustre MDS (lustre-2.12.2-1.el7)
oss-2-x:   2*2.10GHz Intel Xeon E5-2620 v2 6-Core with 32GB RAM for Lustre OSS (lustre-2.12.2-1.el7, 6 nodes each with 34*HDD, 31TB sw raid0)
wn-1-1:    2*2.10GHz Intel Xeon E5-2620 v4 8-core + HT (1 node) 32 cores with total 256GB RAM
           8*GeForce RTX 2080ti GPUs
wn-1-x     2*2.40GHz Intel Xeon E5620  4-core + HT ( 4 nodes)  64 cores with 2.0GB RAM/core
wn-2-x:                    
           2*2.60GHz Intel Xeon E5-2650 v2   8-core + HT (4 Nodes)   128 cores with 2.0GB RAM/core   "Ivi-Bridge EP"   Launch date: Q3'13
           2*2.40GHz Intel Xeon E5-2630 v3   8-core + HT (8 Nodes)   256 cores with 2.0GB RAM/core   "Haswell EP"      Launch date: Q3'14
           2*2.20GHz Intel Xeon E5-2630 v4  10-core + HT (12 Nodes)  480 cores with 2.4GB RAM/core   "Broadwell EP"    Launch date: Q1'16
           2*2.60GHz Intel Xeon E5-2670  0   8-core + HT (12 Nodes)  384 cores with 2.0GB RAM/core   "Sandy Bridge EP" Launch date: Q3'12
           2*2.60GHz Intel Xeon E5-2670  0   8-core + HT (54 Nodes) 1728 cores with 2.0GB RAM/core   "Sandy Bridge EP" Launch date: Q3'12
           2*2.80GHz Intel Xeon E5-2680 v2  10-core + HT (39 Nodes) 1560 cores with 3.2GB RAM/core   "Ivi Bridge EP"   Launch date: Q3'13
           2*2.80GHz AuthenticAMD EPYC 7742 64-core + HT ( 4 Nodes) 1024 cores with 2.0GB RAM/core   "AMD EPYC 7742"   Launch Date: Q3'19

dpm.lhep STORAGE ELEMENT:
dpm            2*2.60GHz Intel Xeon E5-2650 v2  8-core + HT with 96GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel )
dpmdisk0[1-3]  1*2.67GHz Intel Xeon X5650       6-core      with 24GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 3* 20TB (18.19TB net in DPM) RAID6 each
dpmdisk04      2*2.50GHZ Intel Xeon E5-2609 v2  4-core      with 32GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 1* 60TB ( 54.57TB net in DPM) RAID6 
dpmdisk0[5-6]  1*2.67GHz Intel Xeon X5650       6-core      with 24GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 3* 30TB ( 27.28TB net in DPM) RAID6 each
dpmdisk07      1*2.67GHz Intel Xeon X5650       6-core + HT with 24GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 3* 40TB ( 36.38TB net in DPM) RAID6
dpmdisk08      2*2.10GHz Intel Xeon E5-2620 v4  8-core + HT with 32GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 2*100TB ( 90.95TB net in DPM) RAID6
                                                                                                                         1*140TB (127.33TB net in DPM) RAID6
dpmdisk09      2*2.10GHz Intel Xeon E5-2620 v4  8-core + HT with 96GB RAM and the dmlite DOMA stack 1.14.2-1.el7 (epel), 2*378TB (343.76TB net in DPM) RAID6

bdii.lhep INFORMATION SYSTEM:
bdii0[1-2]     2*VMs run by ID with emi-bdii-site-1.0.1-1.el6 (UMD-4)

cvmfs.lhep SQUID FOR CVMFS:
cvmfs0[1-2]    2*VMs run by ID with frontier-squid-4.13-5.1 (cern-frontier)

Interactive services:
ui01:      2*2.40GHz Intel Xeon E5620       4-Core + HT with  64GB RAM (login node) with 3.7TB local user storage /mnt/storage1 (incl. backup of terabig)
ui02:      2*2.40GHz Intel Xeon E5620       4-Core + HT with  64GB RAM (login node) with 3.7TB local user storage /mnt/storage1 
ui03:      2*2.40GHz Intel Xeon E5-2630 v3  8-Core + HT with 128GB RAM (login node) with 3.7TB local user storage /mnt/storage1
ui04:      2*2.40GHz Intel Xeon E5-2640 v4 10-Core + HT with 128GB RAM (login node) with 3.7TB local user storage /mnt/storage1 
ui05:      2*2.40GHz Intel Xeon E5-2640 v4 10-Core + HT with 128GB RAM (login node) with 3.7TB local user storage /mnt/storage1 
homes:     2*2.50GHz Intel Xeon E5-2609 v2  4-Core       with 32GB RAM shared home directories: 8.7TB exported to the ui's as /terabig 
nfs02:     1*2.40GHz Intel Xeon E5-2609 v2  4-Core       with 16GB RAM shared storage area:      44TB exported to the ui's as /ngse5

Virtualisation service:
kvm02      2*2.67GHz Intel Xeon X5650      6-core +HT with   64GB with KVM and 3.5T storage for guests      

 Network:
 2 Brocade FCX648 (4*10GbE, 48*1GbE each) with 2*10GbE uplink (Scienze DMZ)

 2 Dell Force10 (48+48*1GbE vLAN stack) private LAN ce02.lhep CLUSTER)
 2 Dell Force10 (2*24+24*1GbE vLANs stack) private LAN ce01.lhep CLUSTER, 2nd vLAN unused)
 1 Mellanox QDR (30*40Gbit) private IB LAN ce01.lhep CLUSTER
 x QDR switches built-in the SUN Blade X6048 rack. private IB LAN ce01.lhep CLUSTER
  • Benchmark HEPspec06 (weighed average across all node types):
           ce01.lhep
          NODES * TYPE (cores)   HEPSPEC/node (total)  HEPSPEC/core  CPU model
           4 * 32-core   (128)    370.36  (1481.44)    11.57         Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz "Ivi-Bridge EP"
           8 * 32-core   (256)    406.31  (3250.48)    12.7          Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz "Haswell EP"
          12 * 40-core   (480)    496.28  (5955.36)    12.41         Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz "Broadwell EP"
          12 * 32-core   (384)    366.08  (4392.96)    11.44         Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz  "Sandy Bridge EP"
          54 * 32-core  (1728)    366.08 (19768.32)    11.44         Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz  "Sandy Bridge EP"
          39 * 40-core  (1560)    488.40 (19047.60)    12.21         Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz "Ivi Bridge EP"
           4 * 256-core (1024)   3158.79 (12635.16)    12.34         AMD EPYC 7742 64-Core Processor
          =========================================================
          133            5560             66531.32

          In arc.conf:
          [infosys/glue1]
          processor_other_description=Cores=41.8, Benchmark=11.97-HEP-SPEC06
          ...
          [queue:CLUSTER]
          benchmark=HEPSPEC 11.97
          benchmark=Si2k 2993

          [queue:CLUSTER-HM]
          benchmark=HEPSPEC 12.21
          benchmark=Si2k 3052
          allowaccess=uboone
          allowaccess=dune
          allowaccess=ops

          [queue:CLUSTER-GPU]
          benchmark=HEPSPEC 11.97
          benchmark=Si2k 2993



Host IP Service OS CPU Comment
lheplsw1.unibe.ch 130.92.139.201 48+4 Port Switch Brocade FCX648 Connected to eth0 (10GbE NIC) on ce01 and eth2 (10GbE NIC) on ce02
ce01.lhep.unibe.ch 130.92.140.201 Rocks Front End (FE) 6.2 x86_64 SLC 6.7 x86_64 2 Quad Xeon E5645 @2.4GHz Rocks FE, SLURM master, ARC CE, Web Server, Ganglia
ce02.lhep.unibe.ch 130.92.140.202 Rocks Front End (FE) 6.2 x86_64 SLC 6.7 x86_64 2 Quad Xeon E5620 @2.4GHz Rocks FE, SLURM master, ARC CE, Web Server, Ganglia
nfs01.lhep.unibe.ch 130.92.140.31 10TB NFS server SLC 6.3 x86_64 1 Quad Xeon E5620 @2.4GHz Obsolete as sw server, some backups (also nfs-2-1, 10.2.255.100)
nfs02.lhep.unibe.ch 130.92.140.32 37TB NFS server SLC 6.3 x86_64 1 Quad Xeon E5530 @2.4GHz Local NFS server for ATLAS, exports /ngse5
mds-2-1 10.1.255.200 503GB Lustre MDS/MDT ce01 SLC 6.7 x86_64 1 Quad Xeon E5620 @2.4GHz MDS/MDT (/dev/mapper/vg00-mdt LV on SSDs for the MDT), Managed by ce01
mds-0-1 10.1.255.200 110GB Lustre MDS/MDT ce02 SLC 6.7 x86_64 1 Quad Xeon E5620 @2.4GHz MDS/MDT (/dev/sda2 LV on SSDs for the MDT), Managed by ce02
oss-2-[1-6] 10.1.255.[11-16] Lustre OSS ce01 SLC 6.7 x86_64 2 Exa Xeon E5-2620 v2 @2.1GHz Managed by ce01
oss-0-[1-12] 10.1.255.[11-22] Lustre OSS ce02 SLC 6.7 x86_64 2 Dual AMD Opteron 2218 or 2376 @2.3GHz Managed by ce02
wn-2-[1-96] 10.1.255.[101-196] Worker Nodes ce01 SLC 6.7 x86_64 2 Quad Xeon E5540 @2.53GHz Managed by ce01
wn-2-[101-106] 10.1.255.[91-96] Worker Nodes ce01 SLC 6.7 x86_64 2 Exa Xeon X5650 @2.67GHz Managed by ce01
wn-2-[111-114] 10.1.255.[201-204] Worker Nodes ce01 SLC 6.7 x86_64 4 Quad Xeon E5-2650 v2 @2.6GHz Managed by ce01
wn-2-[115-122] 10.1.255.[205-212] Worker Nodes ce01 SLC 6.7 x86_64 4 Quad Xeon E5-2630 v3 @2.4GHz Managed by ce01
wn-0-[1-15] 10.1.255.[101-115] Worker Nodes ce02 SLC 6.7 x86_64 2 Dual AMD Opteron 2376 @2.3GHz Managed by ce02
wn-0-[16-62] 10.1.255.[116-162] Worker Nodes ce02 SLC 6.7 x86_64 2 Quad Xeon E5420 @2.5GHz Managed by ce02
wn-1-[1-20] 10.1.255.[201-220] Worker Nodes ce02 SLC 6.7 x86_64 4 Quad AMD Opteron 8356 @2.3GHz Managed by ce02
dpm.lhep.unibe.ch 130.92.139.211 DPM head node SLC 5.7 2 Quad Xeon E5620 @2.4GHz DPM Head node
dpmdisk[01-03].lhep.unibe.ch 130.92.140.[211-213] DPM pool nodes SLC 6.8 x86_64 1 Exa Xeon X5650 @2.67GHz DPM 60TB disk pools
dpmdisk04.lhep.unibe.ch 130.92.140.214 DPM pool node SLC 6.8 x86_64 1 Quad Xeon E5-2609 v2 @2.50GHz DPM 22TB disk pools
dpmdisk[05-07].lhep.unibe.ch 130.92.140.[215 to 219] DPM pool nodes SLC 6.8 x86_64 Exa Xeon X5650 @2.67GHz DPM 90/120TB disk pools
kvm02.lhep.unibe.ch 130.92.140.22 KVM host SLC 6.8 x86_64 2 Exa Xeon X5650 @2.67GHz Virtualization host for KVM
ui0[1-3].lhep.unibe.ch 130.92.140.1[1-3] Interactive login servers SLC 6.8 x86_64 2 Quad Xeon E5620 and 2 Oct Xeon E5-2630 v3 @2.4GHz 3.7TB local storage
homes.lhep.unibe.ch 130.92.140.10 Home directories server SLC 6.8 x86_64 2 Quad Xeon E5-2609 v2 8.7TB nfs homedirs
nfs02.lhep.unibe.ch 130.92.140.32 Local storage server SLC 6.8 x86_64 2 Quad Xeon E5530 37TB nfs storage area
... ... ... ... ... -
Retrieved from "https://wiki.lhep.unibe.ch/index.php?title=AEC-LHEP_Hardware_information&oldid=684"