The solution

Designed around liquid cooling

Compute, storage, network and software, delivered and supported as one platform.

The tender

Requirement | Delivery

The brief

£2 million to succeed the end of life Hawk system, integrating existing Hawk nodes.

Required

Minimum 7,000 compute cores

Delivered7,296cores across 38 compute nodes
Required

High performance network

Delivered200Gb3:1 blocking HDR InfiniBand
Required

Minimum 1PB, expandable to 2PB

Delivered1.1PBusable DDN Lustre main storage
Required

300TB home storage

Delivered504TBnet usable DDN IntelliFlash N6200 hybrid storage
Required

Cluster management software stack

DeliveredTrinityXClusterVision cluster management stack
Required

Liquid cooled infrastructure

Delivered3 DLC racksdirect to chip liquid cooling

Falcon today

Falcon in full

Falcon is Cardiff University’s HPC and AI cluster, run by ARCCA. It brings together three parts in one environment: the system bought at tender, the nodes added since, and Hawk nodes moved from the Redwood data centre. Falcon is housed at Cloud Centres’ Tier III-certified data centre north of Cardiff, in a suite built for direct liquid cooling.

10,048CPU cores across 54 compute nodes
1.1PBusable DDN Lustre storage
4direct liquid‑cooled racks
24NVIDIA GPUs across 4 GPU nodes
UKThe first Supermicro DLC cluster in academia
01

Initial system

  • 30 standard compute nodes, each with two AMD EPYC 9654 processors (192 cores per node; 5,760 cores in total)
  • 8 high-memory compute nodes (192 cores per node; 1,536 cores)
  • 1 GPU node with 8 NVIDIA L40S GPUs
  • 1 GPU node with 4 NVIDIA H100 NVL GPUs
  • 3 direct liquid-cooled racks
02

Added since installation

Since the initial system was installed, Cardiff and its research groups have expanded Falcon:

  • 13 standard compute nodes (2,496 cores), including 2 for the School of Chemistry
  • 3 compute nodes (256 cores) for Earth and Environmental Sciences
  • 1 GPU node with 8 NVIDIA L40S GPUs for Physics and Astronomy
  • 1 GPU node with 4 NVIDIA H200 NVL GPUs for Physics and Astronomy
  • 1 additional direct liquid-cooled rack
03

Integrated from Hawk

Hawk nodes moved from the Redwood data centre now run alongside the new hardware:

  • 14 Dell GPU servers with NVIDIA V100 GPUs
  • 68 Dell C6525 compute nodes with AMD EPYC Rome processors. 32 are Hawk’s core AMD nodes, and 36 came from research-funded expansions, including 16 for Physics and 16 for Chemistry.
  • 15 Lenovo compute nodes from the star formation expansion

Storage

  • DDN EXAScaler parallel file system: 1.1 PB usable (274 TB NVMe flash and 1.1 PB HDD), 37 GB/s read and 30 GB/s write, and can scale to 11.4 PB
  • DDN IntelliFlash N6200 home storage: 504 TB usable, and can expand to 4 PB

Networking and management

  • InfiniBand spine-leaf fabric
  • 2 high-availability login nodes, supporting up to 100 concurrent users
  • 2 high-availability management nodes

Software

  • TrinityX and OpenHPC: the operating system, compilers, debuggers, profilers and more than 2,000 packages
  • Slurm workload manager, customised with Cardiff University
  • EasyBuild, with Cardiff’s scientific application suite compiled and installed
  • A user wiki covering cluster use and example workloads

Liquid Cooling as a service

Falcon was designed around a simple idea. Direct to chip liquid cooling is not a standalone technology to bolt on to a cluster. It has to be combined with the operational expertise required to deploy, integrate and support the whole platform as one environment.
That is what BIOS IT means by liquid cooling as a service: one team accountable for the cooling, the compute, the storage, the network and the software stack working together, not four vendors' worth of separate support lines.

Achievements

First Supermicro direct to chip liquid cooled cluster in UK academia.
Relocated the platform's housing mid project to a purpose built environment better suited to liquid cooling, without disrupting ARCCA's users.
NVIDIA Deep Learning courses made available to Cardiff's research community as part of the partnership.