AEC-LHEP Hardware information

BERN ATLAS Tier-2 2016 Systems - ce0.lhep.unibe.ch, ce02.lhep.unibe.ch, dpm.lhep.unibe.ch, bdii.lhep.unibe.ch

• Ganglia ce01
• Ganglia ce02
• Nagios UNIBE-LHEP (restricted)
• NorduGrid ATLAS

• GOCDB Entry
• ROC: NGI_CH - Certification status: Certified.
• Network : Gateway 130.92.140.1. DNS 130.92.9.53 (default), 130.92.9.52. Subnet mask 255.255.255.0
• Systems' Topology

• Summary :

           ce01.lhep CLUSTER:
           ce01:      2*2.40GHz Intel Xeon E5645 6-Core with 24GB RAM - ROCKS 6.2 and nordugrid-arc-5.0.3-1.el6.x86_64 (epel)
           mds-2-1:   1*2.40GHz Intel Xeon E5620 4-Core with 24GB RAM for Lustre MDS (lustre-2.5.3)
           oss-2-x:   2*2.10GHz Intel E5-2620 v2 4-Core with 16GB RAM for Lustre OSS (lustre-2.5.3, 6 nodes each with 32*HDD, 31TB sw raid0)
           wn-2-x:
                      2*2.53GHz Intel Xeon E5540  8-core + HT (45 Nodes) 540 cores with 2GB RAM/core (or 720 cores with 1.33GB RAM/core)
                      2*2.53GHz Intel Xeon E5540  8-core + HT ( 4 Nodes)  32 cores with 2GB RAM/core (same HW as above with 1 dead CPU)
                      2*2.67GHz Intel Xeon X5650 12-core + HT ( 6 Nodes) 144 cores with 2GB RAM/core
                      2*2.60GHz Intel Xeon E5-2650 v2 8-core + HT (4 Nodes) 128 cores with 2GB RAM/core
                      2*2.40GHz Intel Xeon E5-2630 v3 8-core + HT (8 Nodes) 256 cores with 2GB RAM/core

           ce02.lhep CLUSTER:
           ce02       2*2.40GHz Intel Xeon E5620 4-core with 24GB RAM - ROCKS 6.2 and nordugrid-arc-5.0.4-1.el6.x86_64 (epel)
           mds-0-1    1*2.40GHz Intel Xeon E5620 4-core with 24GB RAM for Lustre MDS (lustre-2.5.3-2)
           oss-0-x    2*2.30GHz AMD Opteron 2376 4-core with 16GB RAM for Lustre OSS (lustre-2.5.3, 11 nodes each with 1*HD 2TB)
           wn-0-x:
                      2*2.30GHz AMD Opteron 2376 4-core (15 Nodes) 120 cores with 2GB RAM/core 
                      4*2.30GHz AMD Opteron 8356 4-core (20 Nodes) 320 cores with 2GB RAM/core
                      1*2.50GHz Intel Xeon E5420 4-core + HT (47 Nodes) 376 cores with 2GB RAM/core

           dpm.lhep STORAGE ELEMENT:
           dpm        2*2.40GHz Intel Xeon E5620 4-core + HT with 16GB RAM and dpm-1.8.8-4.el5 (epel)
           dpmdisk0x  1*2.67GHz Intel Xeon X5650 6-core with 12GB RAM and dpm-1.8.8-4.el6.x86_64(epel), 6 nodes, 3*raid6 each, 436.52TB net in DPM)
           dpmdisk04  1*2.33GHZ Intel Xeon E5345 4-core with  8GB RAM and dpm-1.8.10-1.el6.x86_64 (epel), 1 node 1*raid6, 21.83TB net in DPM)

           bdii.lhep INFORMATION SYSTEM:
           service running on dpm.lhep: emi-bdii-site-1.0.1-1.el5      

          Interactive services:
           ui01:      2*2.40GHz Intel Xeon E5620      4-Core + HT with  64GB RAM (login node) with 3.7TB local user storage /mnt/storage1 (incl. backup of terabig)
           ui02:      2*2.40GHz Intel Xeon E5620      4-Core + HT with  64GB RAM (login node) with 3.7TB local user storage /mnt/storage1 
           ui03:      2*2.40GHz Intel Xeon E5-2630 v3 8-Core + HT with 128GB RAM (login node) with 3.7TB local user storage /mnt/storage1
           homes:     2*2.50GHz Intel Xeon E5-2609 v2 4-Core with 32GB RAM shared home directories: 8.7TB exported to the ui's as /terabig 
           nfs02:     2*2.40GHz Intel Xeon E5530      4-Core with 24GB RAM shared storage area:      37TB exported to the ui's at /ngse5

• Hardware :

           ce01.lhep:
            1 Dalco FrontEnd 12-core Xeon E5645 @ 2.40GHz 
            1 Dalco Lustre mds 4-core Xeon E5620 @ 2.40GHz
           96 SUN Blade X6048 (workers, ~1/2 dead)
            6 IBM x3650 M3 (workers)
           12 Dalco (workers)
            6 Dalco Terraserver (Lustre)
           ce02.lhep:
            1 Dalco FrontEnd 8-core Xeon E5620 @ 2.40GHz
            1 Dalco Lustre mds 4-core Xeon E5620 @ 2.40GHz
           26 SUN Fire X2200 (Lustre and workers)
           20 SUN Blade X8840 (workers)
           47 DELL PE 1950 (workers)
           dpm.lhep:
            1 DELL PE 1950 (DPM head node) 
            6 Dalco Terraserver (DPM pool nodes) with 3*12-port Areca 1280/1880 (RAID6) each
            1 Dalco Terraserver (DPM pool node)  with 1*Areca 1880 (RAID6)
           Interactive
            3 DELL PE 1950 (ui's) 16-core (HT) Xeon E5620 and 32-core (HT) Xeon E5-2630 v3 and 4GB RAM per core
            1 Dalco NFS server 8-core Xeon E5-2609 v2 and Areca 1880 8/12 (RAID6)
            1 Dalco NFS server 8-core Xeon E5530 and Areca 1280/1280ML 24-Port (RAID6)
           Network:
            2 Brocade FCX648 (4*10GbE, 48*1GbE each) with 1*10GbE uplink (public LAN)
            1 NetGear ProSafe GSM7248R and GS748T (public LAN)
            2 Dell Force10 (48+48*1GbE vLAN stack) private LAN ce02.lhep CLUSTER)
            2 Dell Force10 (2*24+24*1GbE vLANs stack) private LAN ce01.lhep CLUSTER, 2nd vLAN unused)
            1 Mellanox QDR (30*40Gbit) private IB LAN ce01.lhep CLUSTER
            x QDR switches built-in the SUN Blade X6048 rack. private IB LAN ce01.lhep CLUSTER

• Benchmark HEPspec06 (weighed average across all node types):

           ce01.lhep
           NODES * TYPE (cores)        HEPSPEC/node (total)
            8 * 32-core (256)          406.31 (3250.48)
            4 * 32-core (128)          370.36 (1481.44)
            6 * 24-core (144)          232.50 (1395.00)
            4 *  8-core ( 32)          118.48 ( 473.92)
           45 * 12-core (540)          143.06 (6437.07)
           ============================================
           67           1100                  13037.91 Installed Capacity
           
           In arc.conf arc.conf:
           processor_other_description='Cores=16.42, Benchmark=11.85-HEP-SPEC06'
           ...
           jobreport_options="urbatch:500,archiving:/var/spool/nordugrid/urs-archive-jura,topic:/queue/global.accounting.cpu.central,gocdb_name:UNIBE-LHEP,benchmark_type:HEPSPEC,benchmark_value:11.85,use_ssl:true"
           
           ce02.lhep
           NODES * TYPE (cores)        HEPSPEC/node (total)
           20 * 16-core (320)          123.74 (2474.7 )
           15 *  8-core (120)           72.72 (1090.87) 
           47 *  8-core (376)           72.14 (3390.7 )
           ============================================
           82            816                   6956.27 Installed Capacity
           Total: 3261.8 HEP-SPEC06
           
           In arc.conf arc.conf:
           processor_other_description='Cores=9.9, Benchmark=8.52-HEP-SPEC06'
           ...
           jobreport_options="urbatch:1000,archiving:/var/spool/nordugrid/urs-archive-jura,topic:/queue/global.accounting.cpu.central,gocdb_name:UNIBE-LHEP,benchmark_type:HEPSPEC,benchmark_value:8.52,use_ssl:true"

Host	IP	Service	OS	CPU	Comment
lheplsw1.unibe.ch	130.92.139.201	48+4 Port Switch		Brocade FCX648	Connected to eth0 (10GbE NIC) on ce01 and eth2 (10GbE NIC) on ce02
ce01.lhep.unibe.ch	130.92.140.201	Rocks Front End (FE) 6.2 x86_64	SLC 6.7 x86_64	2 Quad Xeon E5645 @2.4GHz	Rocks FE, SLURM master, ARC CE, Web Server, Ganglia
ce02.lhep.unibe.ch	130.92.140.202	Rocks Front End (FE) 6.2 x86_64	SLC 6.7 x86_64	2 Quad Xeon E5620 @2.4GHz	Rocks FE, SLURM master, ARC CE, Web Server, Ganglia
nfs01.lhep.unibe.ch	130.92.140.31	10TB NFS server	SLC 6.3 x86_64	1 Quad Xeon E5620 @2.4GHz	Obsolete as sw server, some backups (also nfs-2-1, 10.2.255.100)
nfs02.lhep.unibe.ch	130.92.140.32	37TB NFS server	SLC 6.3 x86_64	1 Quad Xeon E5530 @2.4GHz	Local NFS server for ATLAS, exports /ngse5
mds-2-1	10.1.255.200	503GB Lustre MDS/MDT ce01	SLC 6.7 x86_64	1 Quad Xeon E5620 @2.4GHz	MDS/MDT (/dev/mapper/vg00-mdt LV on SSDs for the MDT), Managed by ce01
mds-0-1	10.1.255.200	110GB Lustre MDS/MDT ce02	SLC 6.7 x86_64	1 Quad Xeon E5620 @2.4GHz	MDS/MDT (/dev/sda2 LV on SSDs for the MDT), Managed by ce02
oss-2-[1-6]	10.1.255.[11-16]	Lustre OSS ce01	SLC 6.7 x86_64	2 Exa Xeon E5-2620 v2 @2.1GHz	Managed by ce01
oss-0-[1-12]	10.1.255.[11-22]	Lustre OSS ce02	SLC 6.7 x86_64	2 Dual AMD Opteron 2218 or 2376 @2.3GHz	Managed by ce02
wn-2-[1-96]	10.1.255.[101-196]	Worker Nodes ce01	SLC 6.7 x86_64	2 Quad Xeon E5540 @2.53GHz	Managed by ce01
wn-2-[101-106]	10.1.255.[91-96]	Worker Nodes ce01	SLC 6.7 x86_64	2 Exa Xeon X5650 @2.67GHz	Managed by ce01
wn-2-[111-114]	10.1.255.[201-204]	Worker Nodes ce01	SLC 6.7 x86_64	4 Quad Xeon E5-2650 v2 @2.6GHz	Managed by ce01
wn-2-[115-122]	10.1.255.[205-212]	Worker Nodes ce01	SLC 6.7 x86_64	4 Quad Xeon E5-2630 v3 @2.4GHz	Managed by ce01
wn-0-[1-15]	10.1.255.[101-115]	Worker Nodes ce02	SLC 6.7 x86_64	2 Dual AMD Opteron 2376 @2.3GHz	Managed by ce02
wn-0-[16-62]	10.1.255.[116-162]	Worker Nodes ce02	SLC 6.7 x86_64	2 Quad Xeon E5420 @2.5GHz	Managed by ce02
wn-1-[1-20]	10.1.255.[201-220]	Worker Nodes ce02	SLC 6.7 x86_64	4 Quad AMD Opteron 8356 @2.3GHz	Managed by ce02
dpm.lhep.unibe.ch	130.92.139.211	DPM head node	SLC 5.7	2 Quad Xeon E5620 @2.4GHz	DPM Head node
dpmdisk[01-03].lhep.unibe.ch	130.92.140.[211-213]	DPM pool nodes	SLC 6.8 x86_64	1 Exa Xeon X5650 @2.67GHz	DPM 60TB disk pools
dpmdisk04.lhep.unibe.ch	130.92.140.214	DPM pool node	SLC 6.8 x86_64	1 Quad Xeon E5345 @2.33 GHz	DPM 22TB disk pools
dpmdisk[05-07].lhep.unibe.ch	130.92.140.[215 to 219]	DPM pool nodes	SLC 6.8 x86_64	Exa Xeon X5650 @2.67GHz	DPM 90/120TB disk pools
kvm02.lhep.unibe.ch	130.92.140.22	KVM host	SLC 6.8 x86_64	2 Exa Xeon X5650 @2.67GHz	Virtualization host for KVM
ui0[1-3].lhep.unibe.ch	130.92.140.1[1-3]	Interactive login servers	SLC 6.8 x86_64	2 Quad Xeon E5620 and 2 Oct Xeon E5-2630 v3 @2.4GHz	3.7TB local storage
homes.lhep.unibe.ch	130.92.140.10	Home directories server	SLC 6.8 x86_64	2 Quad Xeon E5-2609 v2	8.7TB nfs homedirs
nfs02.lhep.unibe.ch	130.92.140.32	Local storage server	SLC 6.8 x86_64	2 Quad Xeon E5530	37TB nfs storage area
...	...	...	...	...	-