Shared Cluster Hardware System Standards for Compute Nodes
To maximize Shared Cluster System management effectiveness, and to provide the highest quality computing services to Shared Cluster customers, compute nodes added to WEXAC must meet minimum standards.
For detailed information, please refer to our WEXAC policy on the policy information page.
Four distinct compute node types are defined:
- CPU intensive.
- Memory intensive.
- I/O intensive.
- GPU.
Following are the standards for these nodes, as of February 2013:
- HP SL230s blades.
- High quality, tightly integrated hardware, in terms of thermal capacity, power, rack mounting capability and parts used.
- Dual six-core 2.3 GHz Intel Xeon E5-2630 or more powerful CPUs.
- A minimum of 2 GBs of memory per core, with the ability to expand to 2 TBs for a single memory intensive node.
- A 500 to 600 GB SATA or SAS hard drive per node.
- A Gigabit Ethernet or 10 Gigabit Ethernet port, or an InfiniBand connection.
- QDR InfiniBand interconnect for MPI jobs; Ethernet interconnect for standard cluster jobs.
- A 3-year warranty
To purchase WEXAC nodes, contact Vadim Malkin at extension 6078.
System Standards for Compute Nodes
| Type |
CPU |
RAM |
Disk |
Price |
| A (CPU + I/O) |
2 x Intel E5-2630, 12 cores |
32 GB |
2 x 300 SAS |
$4,400 |
| B (CPU) |
2 x Intel E5-2630, 12 cores |
32 GB |
1 x 500 SATA |
$3,900 |
| C (GPU) |
2 x Intel E5645 + 2 x Nvidia Tesla M2050 (896 cores) |
48 GB |
2 x 300 SAS |
$8,100 |
| D Mem 128 |
2 x Intel E5-2630, 12 cores |
128 GB |
2 x 300 SAS |
$6,950 |
| E Mem 48 |
2 x Intel E5645, 12 cores |
48 GB |
2 x 300 SAS |
$4,500 |
The above standards will be evaluated periodically and updated as per best price/performance.
WEXAC consists of 1,400 cores on nodes configured with identical Operating Systems and Application Stack installations.
Network and Interconnect
WEXAC nodes feature both an InfiniBand interconnect and a 1/10 Gigabit Ethernet network port, while others have a Gigabit Ethernet or 10 Gigabit Ethernet network port. The Ethernet port is dedicated to inbound and outbound WEXAC storage system traffic and additionally handles various administrative functions. InfiniBand is used for inter-node MPI-type communication. Use of these two interconnects maximizes performance within the cluster. WEXAC is the very first Weizmann Institute cluster that utilizes both QDR IB and 10 Gigabit Ethernet connections.
Storage
A high performance, high-availability system provides storage for WEXAC.
The storage directory structure is as follows:
- /home/lab/lab_name - Parent directory of a member lab
- /home/lab/lab_name/user_name - member lab user's home directory
- /apps - common applications, binaries location
- /sharedDB - common public scientific databases (e.g - NT/NR, PDB, etc,). Users can ask to add databases to this location.
Our maximum storage and recommended limits are as follows:
| Function |
Limit |
Notes |
| Maximum number of files per member lab |
100 million |
It is recommended that you TAR files when jobs end, to reduce the number of files in your lab, to assure backup cost savings and to improve storage performance. |
| Maximum number of files per non-member lab |
20 million |
It is recommended that you TAR files when jobs end, to reduce the number of files in your lab, to assure backup cost savings and to improve storage performance. |
| Recommended maximum number of files per directory |
5 million |
A higher number of files may result in performance degradation. It is recommended that you TAR files when jobs end, do avoid exceeding this limit and to assure backup cost savings. |
| Recommended maximum file size |
50 GBs |
|
| Recommended maximum number of subdirectories per directory |
100,000 |
A higher number of subdirectories may result in performance degradation. |
Compute Nodes on WEXAC
| CN# |
Type |
Cores |
GPU |
Memory (GB) |
Owner |
| 101 |
IBM |
896 GPU cores |
Tesla S2050 |
24 |
Public |
| 102-103 |
HP type C |
896 GPU cores |
Tesla S2050 |
48 |
Public |
| 104-106 |
HP type D |
12 |
|
96 |
Public |
| 107-121 |
HP type B |
12 |
|
24 |
Public |
| 500-524 |
HP type B |
12 |
|
24 |
Fleishman lab |
| 200-238 |
Fujitsu |
8 |
|
16 |
Sorek lab |
| 070-072 |
IBM |
12 |
|
72 |
Lancet lab |
| 073-075 |
|
12 |
|
24 |
Hanna lab |
| 080 |
|
12 |
|
24 |
Schreiber lab |
| 081 |
HP type E |
12 |
|
48 |
Schreiber lab |
| 090-091 |
|
8 |
|
32 |
Tawfik lab |
| 400-401 |
|
8 |
|
16 |
Koren lab |
| 600 |
|
8 |
|
16 |
Pilpel lab |
| 610 |
|
8 |
|
36 |
Sobel lab |
| 630 |
|
8 |
|
16 |
EM unit |
