AEC-LHEP Hardware information
Jump to navigation
Jump to search
BERN ATLAS Tier-2 2020 Systems
- Grid HT Cluster ce01 (5.5k cores)
- Node summary
- Scheduler: SLURM 18.08.8-1.el7 (scheduler on ce01)
- Shared File System: Lustre 2.12.2-1.el7
- 1 x MDS, 6 x OSS: 204 x 1T HDDs (184T)
- Grid Compute Elements ce01.lhep.unibe.ch, ce02.lhep.unibe.ch, ce03.lhep.unibe.ch
ce01.lhep.unibe.ch has address 130.92.140.201 ce02.lhep.unibe.ch has address 130.92.140.202 ce03.lhep.unibe.ch has address 130.92.140.203
- Grid Storage Element dpm.lhep.unibe.ch (2PB)
dpm.lhep.unibe.ch has address 130.92.140.100 dpm.lhep.unibe.ch has IPv6 address 2001:620:400:fea8::100
- Grid Information System bdii.lhep.unibe.ch
# host bdii.lhep.unibe.ch bdii.lhep.unibe.ch has address 130.92.140.121 bdii.lhep.unibe.ch has address 130.92.140.122
- Squid for CVMFS cvmfs.lhep.unibe.ch
# host cvmfs.lhep.unibe.ch cvmfs.lhep.unibe.ch has address 130.92.252.18 cvmfs.lhep.unibe.ch has address 130.92.252.19
EGI:
- GOCDB Entry - UNIBE-LHEP
- ROC: NGI_CH - Certification status: Certified.
MONITORING:
NETWORK:
• IPv4: 130.92.140.0/24, GW: 130.92.140.1 (130.92.140.5 - 130.92.140.250) • IPv6: 2001:620:400:fea8::/64, GW: 2001:620:400:fea8::1 (2001:620:400:fea8::10 - 2001:620:400:fea8::109)
- [File:Topology-dec2020.png]
- Systems' Power Budget - TO BE UPDATED
- Summary :
ce01.lhep CLUSTER:
ce01: 2*2.40GHz Intel Xeon E5645 6-Core with 24GB RAM - ROCKS 7.0 and 6.6.0-1.el7 (epel)
mds-2-1: 1*2.40GHz Intel Xeon E5620 4-Core with 24GB RAM for Lustre MDS (lustre-2.12.2-1.el7)
oss-2-x: 2*2.10GHz Intel E5-2620 v2 6-Core with 32GB RAM for Lustre OSS (lustre-2.12.2-1.el7, 6 nodes each with 34*HDD, 31TB sw raid0)
wn-2-x:
2*2.53GHz Intel Xeon E5540 8-core + HT (13 Nodes) 208 cores with 3GB RAM/core
1*2.53GHz Intel Xeon E5540 8-core + HT ( 3 Nodes) 24 cores with 6GB RAM/core (same HW as above with 1 dead CPU)
2*2.67GHz Intel Xeon X5650 12-core + HT ( 6 Nodes) 144 cores with 2GB RAM/core
2*2.50GHz Intel Xeon E5420 8-core + HT ( 6 Nodes) 24 cores with 2GB RAM/core
2*2.60GHz Intel Xeon E5-2650 v2 8-core + HT (4 Nodes) 128 cores with 2GB RAM/core
2*2.40GHz Intel Xeon E5-2630 v3 8-core + HT (8 Nodes) 256 cores with 2GB RAM/core
2*2.20GHz Intel Xeon E5-2630 v4 10-core + HT (12 Nodes) 480 cores with 2.4GB RAM/core
ce02.lhep CLUSTER:
ce02 2*2.40GHz Intel Xeon E5620 4-core with 24GB RAM - ROCKS 6.2 and nordugrid-arc-5.0.4-1.el6.x86_64 (epel)
mds-0-1 1*2.40GHz Intel Xeon E5620 4-core with 24GB RAM for Lustre MDS (lustre-2.5.3-2)
oss-0-x 2*2.30GHz AMD Opteron 2376 4-core with 16GB RAM for Lustre OSS (lustre-2.5.3, 11 nodes each with 1*HD 2TB)
wn-0-x:
2*2.30GHz AMD Opteron 2376 4-core (15 Nodes) 120 cores with 2GB RAM/core
4*2.30GHz AMD Opteron 8356 4-core (20 Nodes) 320 cores with 2GB RAM/core
1*2.50GHz Intel Xeon E5420 4-core + HT (47 Nodes) 376 cores with 2GB RAM/core
dpm.lhep STORAGE ELEMENT:
dpm 2*2.40GHz Intel Xeon E5620 4-core + HT with 16GB RAM and dpm-1.8.8-4.el5 (epel)
dpmdisk0x 1*2.67GHz Intel Xeon X5650 6-core with 12GB RAM and dpm-1.8.8-4.el6.x86_64(epel), 6 nodes, 3*raid6 each, 436.52TB net in DPM)
dpmdisk04 1*2.33GHZ Intel Xeon E5-2609 v2 4-core with 8GB RAM and dpm-1.8.10-1.el6.x86_64 (epel), 1 node 1*raid6, 21.83TB net in DPM)
bdii.lhep INFORMATION SYSTEM:
service running on dpm.lhep: emi-bdii-site-1.0.1-1.el5
Interactive services:
ui01: 2*2.40GHz Intel Xeon E5620 4-Core + HT with 64GB RAM (login node) with 3.7TB local user storage /mnt/storage1 (incl. backup of terabig)
ui02: 2*2.40GHz Intel Xeon E5620 4-Core + HT with 64GB RAM (login node) with 3.7TB local user storage /mnt/storage1
ui03: 2*2.40GHz Intel Xeon E5-2630 v3 8-Core + HT with 128GB RAM (login node) with 3.7TB local user storage /mnt/storage1
homes: 2*2.50GHz Intel Xeon E5-2609 v2 4-Core with 32GB RAM shared home directories: 8.7TB exported to the ui's as /terabig
nfs02: 2*2.40GHz Intel Xeon E5530 4-Core with 24GB RAM shared storage area: 37TB exported to the ui's at /ngse5
Virtualisation service:
kvm02 2*2.67GHz Intel Xeon X5650 6-core +HT with 64GB with KVM and 3.5T storage for guests
- Hardware :
ce01.lhep:
1 Dalco FrontEnd 12-core Xeon E5645 @ 2.40GHz
1 Dalco Lustre mds 4-core Xeon E5620 @ 2.40GHz
96 SUN Blade X6048 (workers, ~1/2 dead)
6 IBM x3650 M3 (workers)
24 Dalco (workers)
6 Dalco Terraserver (Lustre)
ce02.lhep:
1 Dalco FrontEnd 8-core Xeon E5620 @ 2.40GHz
1 Dalco Lustre mds 4-core Xeon E5620 @ 2.40GHz
3 SUN X4500 "Thumpers" (Lustre)
26 SUN Fire X2200 (Lustre and workers)
20 SUN Blade X8840 (workers)
47 DELL PE 1950 (workers)
dpm.lhep:
1 DELL PE 1950 (DPM head node)
6 Dalco Terraserver (DPM pool nodes) with 3*12-port Areca 1280/1880 (RAID6) each
1 Dalco Terraserver (DPM pool node) with 1*Areca 1880 (RAID6)
Interactive
3 DELL PE 1950 (ui's) 16-core (HT) Xeon E5620 and 32-core (HT) Xeon E5-2630 v3 and 4GB RAM per core
1 Dalco NFS server 8-core Xeon E5-2609 v2 and Areca 1880 8/12 (RAID6)
1 Dalco NFS server 8-core Xeon E5530 and Areca 1280/1280ML 24-Port (RAID6)
Virtualisation
1 Dalco KVM server 24-core Xeon 5650 @2.67GHz
Network:
2 Brocade FCX648 (4*10GbE, 48*1GbE each) with 1*10GbE uplink (public LAN)
1 NetGear ProSafe GSM7248R and GS748T (public LAN)
2 Dell Force10 (48+48*1GbE vLAN stack) private LAN ce02.lhep CLUSTER)
2 Dell Force10 (2*24+24*1GbE vLANs stack) private LAN ce01.lhep CLUSTER, 2nd vLAN unused)
1 Mellanox QDR (30*40Gbit) private IB LAN ce01.lhep CLUSTER
x QDR switches built-in the SUN Blade X6048 rack. private IB LAN ce01.lhep CLUSTER
- Benchmark HEPspec06 (weighed average across all node types):
ce01.lhep
NODES * TYPE (cores) HEPSPEC/node (total)
12 * 40-core (480) 496.28 (5955.36)
8 * 32-core (256) 406.31 (3250.48)
4 * 32-core (128) 370.36 (1481.44)
6 * 24-core (144) 232.50 (1395.00)
3 * 8-core ( 24) 72.14 ( 216.42)
3 * 8-core ( 24) 118.48 ( 355.44)
13 * 16-core (208) 154.39 (2007.07)
============================================
49 1264 14661.21 Installed Capacity
In arc.conf arc.conf:
processor_other_description='Cores=25.8, Benchmark=11.6-HEP-SPEC06'
...
jobreport_options="urbatch:500,archiving:/var/spool/nordugrid/urs-archive-jura,topic:/queue/global.accounting.cpu.central,gocdb_name:UNIBE-LHEP,benchmark_type:HEPSPEC,benchmark_value:11.85,use_ssl:true"
ce02.lhep
NODES * TYPE (cores) HEPSPEC/node (total)
20 * 16-core (320) 123.74 (2474.7 )
15 * 8-core (120) 72.72 (1090.87)
47 * 8-core (376) 72.14 (3390.7 )
============================================
82 816 6956.27 Installed Capacity
Total: 3261.8 HEP-SPEC06
In arc.conf arc.conf:
processor_other_description='Cores=9.9, Benchmark=8.52-HEP-SPEC06'
...
jobreport_options="urbatch:1000,archiving:/var/spool/nordugrid/urs-archive-jura,topic:/queue/global.accounting.cpu.central,gocdb_name:UNIBE-LHEP,benchmark_type:HEPSPEC,benchmark_value:8.52,use_ssl:true"
| Host | IP | Service | OS | CPU | Comment | |
|---|---|---|---|---|---|---|
| lheplsw1.unibe.ch | 130.92.139.201 | 48+4 Port Switch | Brocade FCX648 | Connected to eth0 (10GbE NIC) on ce01 and eth2 (10GbE NIC) on ce02 | ||
| ce01.lhep.unibe.ch | 130.92.140.201 | Rocks Front End (FE) 6.2 x86_64 | SLC 6.7 x86_64 | 2 Quad Xeon E5645 @2.4GHz | Rocks FE, SLURM master, ARC CE, Web Server, Ganglia | |
| ce02.lhep.unibe.ch | 130.92.140.202 | Rocks Front End (FE) 6.2 x86_64 | SLC 6.7 x86_64 | 2 Quad Xeon E5620 @2.4GHz | Rocks FE, SLURM master, ARC CE, Web Server, Ganglia | |
| nfs01.lhep.unibe.ch | 130.92.140.31 | 10TB NFS server | SLC 6.3 x86_64 | 1 Quad Xeon E5620 @2.4GHz | Obsolete as sw server, some backups (also nfs-2-1, 10.2.255.100) | |
| nfs02.lhep.unibe.ch | 130.92.140.32 | 37TB NFS server | SLC 6.3 x86_64 | 1 Quad Xeon E5530 @2.4GHz | Local NFS server for ATLAS, exports /ngse5 | |
| mds-2-1 | 10.1.255.200 | 503GB Lustre MDS/MDT ce01 | SLC 6.7 x86_64 | 1 Quad Xeon E5620 @2.4GHz | MDS/MDT (/dev/mapper/vg00-mdt LV on SSDs for the MDT), Managed by ce01 | |
| mds-0-1 | 10.1.255.200 | 110GB Lustre MDS/MDT ce02 | SLC 6.7 x86_64 | 1 Quad Xeon E5620 @2.4GHz | MDS/MDT (/dev/sda2 LV on SSDs for the MDT), Managed by ce02 | |
| oss-2-[1-6] | 10.1.255.[11-16] | Lustre OSS ce01 | SLC 6.7 x86_64 | 2 Exa Xeon E5-2620 v2 @2.1GHz | Managed by ce01 | |
| oss-0-[1-12] | 10.1.255.[11-22] | Lustre OSS ce02 | SLC 6.7 x86_64 | 2 Dual AMD Opteron 2218 or 2376 @2.3GHz | Managed by ce02 | |
| wn-2-[1-96] | 10.1.255.[101-196] | Worker Nodes ce01 | SLC 6.7 x86_64 | 2 Quad Xeon E5540 @2.53GHz | Managed by ce01 | |
| wn-2-[101-106] | 10.1.255.[91-96] | Worker Nodes ce01 | SLC 6.7 x86_64 | 2 Exa Xeon X5650 @2.67GHz | Managed by ce01 | |
| wn-2-[111-114] | 10.1.255.[201-204] | Worker Nodes ce01 | SLC 6.7 x86_64 | 4 Quad Xeon E5-2650 v2 @2.6GHz | Managed by ce01 | |
| wn-2-[115-122] | 10.1.255.[205-212] | Worker Nodes ce01 | SLC 6.7 x86_64 | 4 Quad Xeon E5-2630 v3 @2.4GHz | Managed by ce01 | |
| wn-0-[1-15] | 10.1.255.[101-115] | Worker Nodes ce02 | SLC 6.7 x86_64 | 2 Dual AMD Opteron 2376 @2.3GHz | Managed by ce02 | |
| wn-0-[16-62] | 10.1.255.[116-162] | Worker Nodes ce02 | SLC 6.7 x86_64 | 2 Quad Xeon E5420 @2.5GHz | Managed by ce02 | |
| wn-1-[1-20] | 10.1.255.[201-220] | Worker Nodes ce02 | SLC 6.7 x86_64 | 4 Quad AMD Opteron 8356 @2.3GHz | Managed by ce02 | |
| dpm.lhep.unibe.ch | 130.92.139.211 | DPM head node | SLC 5.7 | 2 Quad Xeon E5620 @2.4GHz | DPM Head node | |
| dpmdisk[01-03].lhep.unibe.ch | 130.92.140.[211-213] | DPM pool nodes | SLC 6.8 x86_64 | 1 Exa Xeon X5650 @2.67GHz | DPM 60TB disk pools | |
| dpmdisk04.lhep.unibe.ch | 130.92.140.214 | DPM pool node | SLC 6.8 x86_64 | 1 Quad Xeon E5-2609 v2 @2.50GHz | DPM 22TB disk pools | |
| dpmdisk[05-07].lhep.unibe.ch | 130.92.140.[215 to 219] | DPM pool nodes | SLC 6.8 x86_64 | Exa Xeon X5650 @2.67GHz | DPM 90/120TB disk pools | |
| kvm02.lhep.unibe.ch | 130.92.140.22 | KVM host | SLC 6.8 x86_64 | 2 Exa Xeon X5650 @2.67GHz | Virtualization host for KVM | |
| ui0[1-3].lhep.unibe.ch | 130.92.140.1[1-3] | Interactive login servers | SLC 6.8 x86_64 | 2 Quad Xeon E5620 and 2 Oct Xeon E5-2630 v3 @2.4GHz | 3.7TB local storage | |
| homes.lhep.unibe.ch | 130.92.140.10 | Home directories server | SLC 6.8 x86_64 | 2 Quad Xeon E5-2609 v2 | 8.7TB nfs homedirs | |
| nfs02.lhep.unibe.ch | 130.92.140.32 | Local storage server | SLC 6.8 x86_64 | 2 Quad Xeon E5530 | 37TB nfs storage area | |
| ... | ... | ... | ... | ... | - |