The solution
Designed around liquid cooling
Compute, storage, network and software, delivered and supported as one platform.
The tender
Requirement | Delivery
£2 million to succeed the end of life Hawk system, integrating existing Hawk nodes.
Minimum 7,000 compute cores
High performance network
Minimum 1PB, expandable to 2PB
300TB home storage
Cluster management software stack
Liquid cooled infrastructure
Falcon today
Falcon in full
Falcon is Cardiff University’s HPC and AI cluster, run by ARCCA. It brings together three parts in one environment: the system bought at tender, the nodes added since, and Hawk nodes moved from the Redwood data centre. Falcon is housed at Cloud Centres’ Tier III-certified data centre north of Cardiff, in a suite built for direct liquid cooling.
Initial system
- 30 standard compute nodes, each with two AMD EPYC 9654 processors (192 cores per node; 5,760 cores in total)
- 8 high-memory compute nodes (192 cores per node; 1,536 cores)
- 1 GPU node with 8 NVIDIA L40S GPUs
- 1 GPU node with 4 NVIDIA H100 NVL GPUs
- 3 direct liquid-cooled racks
Added since installation
Since the initial system was installed, Cardiff and its research groups have expanded Falcon:
- 13 standard compute nodes (2,496 cores), including 2 for the School of Chemistry
- 3 compute nodes (256 cores) for Earth and Environmental Sciences
- 1 GPU node with 8 NVIDIA L40S GPUs for Physics and Astronomy
- 1 GPU node with 4 NVIDIA H200 NVL GPUs for Physics and Astronomy
- 1 additional direct liquid-cooled rack
Integrated from Hawk
Hawk nodes moved from the Redwood data centre now run alongside the new hardware:
- 14 Dell GPU servers with NVIDIA V100 GPUs
- 68 Dell C6525 compute nodes with AMD EPYC Rome processors. 32 are Hawk’s core AMD nodes, and 36 came from research-funded expansions, including 16 for Physics and 16 for Chemistry.
- 15 Lenovo compute nodes from the star formation expansion
Storage
- DDN EXAScaler parallel file system: 1.1 PB usable (274 TB NVMe flash and 1.1 PB HDD), 37 GB/s read and 30 GB/s write, and can scale to 11.4 PB
- DDN IntelliFlash N6200 home storage: 504 TB usable, and can expand to 4 PB
Networking and management
- InfiniBand spine-leaf fabric
- 2 high-availability login nodes, supporting up to 100 concurrent users
- 2 high-availability management nodes
Software
- TrinityX and OpenHPC: the operating system, compilers, debuggers, profilers and more than 2,000 packages
- Slurm workload manager, customised with Cardiff University
- EasyBuild, with Cardiff’s scientific application suite compiled and installed
- A user wiki covering cluster use and example workloads
Liquid Cooling as a service
Falcon was designed around a simple idea. Direct to chip liquid cooling is not a standalone technology to bolt on to a cluster. It has to be combined with the operational expertise required to deploy, integrate and support the whole platform as one environment.That is what BIOS IT means by liquid cooling as a service: one team accountable for the cooling, the compute, the storage, the network and the software stack working together, not four vendors' worth of separate support lines.-
More compute in a smaller footprint.
-
Improved energy efficiency against air cooled equivalents.
-
A quieter environment than traditional air cooled systems.
-
Capacity built in for future expansion.
Achievements
First Supermicro direct to chip liquid cooled cluster in UK academia.Relocated the platform's housing mid project to a purpose built environment better suited to liquid cooling, without disrupting ARCCA's users.NVIDIA Deep Learning courses made available to Cardiff's research community as part of the partnership.