HPC Performance Tuning on a Managed Computing Platform

In this post, MCP means a managed computing platform: a high-performance computing (HPC) cluster that a provider or a central IT team runs for you. You submit jobs; they own the hardware, operating system and scheduler. (It's not about the Model Context Protocol used by AI tools.) This guide covers what you can tune yourself on such a platform, what to ask your provider for, and how to tell whether a change actually helped.

What you control and what the provider controls

You control	The provider controls
The resources your job script requests	Hardware, BIOS settings and CPU power profiles
Your code, compiler and compiler flags	Operating system and kernel parameters
How processes and threads are placed on cores	The network fabric and its topology
How your data is laid out and written	Filesystem configuration
Which partition or node type you use	Scheduler policies, priorities and limits

A lot of generic server-tuning advice (overclocking, sysctl kernel settings, switching off background services) only applies if you run the cluster yourself. On a managed platform, raise those with your provider. The rest of this guide focuses on the left-hand column.

Measure before you change anything

Pick a few numbers and record them for a representative job:

• Wall time: how long the job took from start to finish.
• Queue wait: how long it sat before starting.
• CPU efficiency: CPU time used divided by (wall time × cores allocated).
• Peak memory per node.
• Throughput: jobs or simulations completed per day, if you run many.

On a Slurm cluster, sacct gives you most of these after a job finishes:

sacct -j 123456 --format=JobID,JobName,Elapsed,TotalCPU,MaxRSS,State

Many sites also install seff, which prints CPU and memory efficiency in one line:

seff 123456

A CPU efficiency of 25% usually means you asked for four times more cores than the code uses. That's the cheapest fix there is.

For the platform itself, standard benchmarks tell you what the hardware can do: HPL (LINPACK) for floating-point performance, STREAM for memory bandwidth, IOR for the parallel filesystem and the OSU micro-benchmarks for MPI latency and bandwidth. SPEC's suites are another option. Run the relevant one before and after a change, on the same node type.

Request the right resources

The scheduler can only place your job well if you tell it what the job needs. A typical hybrid MPI and OpenMP job script:

#!/bin/bash
#SBATCH --job-name=sim
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=32
#SBATCH --cpus-per-task=2
#SBATCH --time=04:00:00
#SBATCH --exclusive

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
srun --cpu-bind=cores ./simulation input.dat

• Match ranks × threads to the cores on each node. Here, 32 MPI ranks × 2 threads fill a 64-core node. Running more threads than cores (oversubscription) slows everything down.
• Set a realistic --time. Many clusters use backfill scheduling, which starts short jobs in gaps between big ones. An honest time limit often gets your job running sooner than the maximum allowed.
• Don't over-request memory or nodes. Spare resources sit idle, count against your allocation and make the job harder to schedule.
• Use --exclusive for benchmarking and for jobs sensitive to noisy neighbours, so other jobs don't share your nodes.

If some work is more urgent than the rest, ask your provider which quality-of-service (QOS) levels or priority options your account has, rather than requesting more resources.

Use every core you pay for

• Know the node. lscpu shows sockets, cores per socket, NUMA nodes and supported instruction sets.
• Compile for the hardware. With GCC or Clang, -O3 -march=native enables the vector instructions (AVX2, AVX-512) the CPU supports. Build on the same node type you run on, or the binary may use instructions another node type lacks. Check whether your provider offers vendor compilers or tuned maths libraries (BLAS, FFT) as modules: they're often faster than building your own.
• Bind processes to cores. srun --cpu-bind=cores stops the operating system moving processes between cores mid-run, which wastes cache.
• Test hyperthreading rather than assuming it helps. Floating-point-heavy codes often run as fast or faster with one thread per physical core.

CPU frequency and turbo behaviour are set by the provider through BIOS and power profiles. If your results vary a lot between identical runs, ask whether the nodes use a performance profile.

Memory and NUMA placement

On a multi-socket node, each CPU has its own local memory. Reaching memory attached to the other socket is slower; this is non-uniform memory access (NUMA).

• numactl --hardware shows the NUMA nodes and how much memory each has.
• For a single process, bind it and its memory to one NUMA node: numactl --cpunodebind=0 --membind=0 ./app. Under Slurm, combine --cpu-bind=cores with --mem-bind=local.
• Mind first touch. Linux places a memory page on the NUMA node of the thread that first writes to it. Initialise large arrays in parallel, with the same threads that will later compute on them, not in a serial loop at start-up.
• Access memory in order. Loop over arrays in the order they're stored (row by row in C, column by column in Fortran) so the cache does its job.
• Watch peak memory from sacct (MaxRSS). A job that swaps or gets killed for running out of memory needs more memory per rank, not more ranks.

Storage and I/O

Storage is often the hidden bottleneck, especially when many ranks read or write at once.

• Use the right tier. Node-local SSD or NVMe scratch (often available as $TMPDIR; check your provider's documentation) for temporary files. The parallel filesystem, such as Lustre or IBM Storage Scale (formerly GPFS), for shared input and output. Home directories, often on NFS, for code and small files, not heavy I/O.
• Avoid thousands of small files. They overload the filesystem's metadata servers. Pack them into an archive or a single HDF5 file.
• Stripe large shared files on Lustre so several storage servers serve them in parallel:

lfs getstripe /scratch/$USER/results
lfs setstripe -c 8 /scratch/$USER/results

New files created in that directory are then spread across eight storage targets. Leave small files at the default stripe count; striping them only adds overhead.

• Write collectively. For many ranks writing one dataset, MPI-IO or parallel HDF5 usually beats every rank writing its own file.
• Move finished results to cheaper archive storage so scratch stays fast for running jobs.

Network and communication

Tightly coupled MPI codes depend on the interconnect, usually InfiniBand or high-speed Ethernet with RDMA (RoCE). RDMA lets the network card move data directly between nodes' memory without involving the CPU.

• Make sure MPI is using the fast network. A misconfigured MPI can silently fall back to TCP over a slower network. If ibstat is available, it shows whether InfiniBand ports are active and their speed. Use the MPI modules your provider supplies; they're built for the fabric.
• Keep nodes close together. In Slurm, --switches=1 asks for nodes under one network switch, which reduces latency. You can give a maximum wait, such as --switches=1@30:00, after which Slurm takes any nodes.
• Measure it. Run the OSU latency and bandwidth benchmarks between two of your allocated nodes and compare them with the figures your provider publishes.

Profile the code

Most of a program's time is usually spent in a small part of the code. Find it before optimising anything:

• perf and Intel VTune (part of Intel oneAPI) show hotspots with little overhead.
• gprof (compile with -pg) gives a simple function-level profile.
• Valgrind's Cachegrind shows cache misses, but runs the program many times slower, so use a small input.
• For MPI programs, a profiler such as mpiP or Score-P shows how much time goes to communication rather than computation.

Profile a short, representative run. Then work on the top two or three hotspots: better memory access, vectorised loops, or parallelising a serial section.

Check how far the job scales

Run the same problem on 1, 2, 4 and 8 nodes and compare the wall times. If doubling the nodes cuts the time by much less than half, you've passed the useful scaling point; extra nodes just burn allocation. This is a strong scaling test. For codes where the problem size grows with the machine, run a weak scaling test instead: double the problem and the nodes together and check that the time stays roughly flat.

Adding nodes (scaling out) is usually more flexible than moving to bigger nodes (scaling up), but only up to the point where communication starts to dominate.

Containers: most shared HPC clusters don't allow Docker, because it needs a root-owned daemon. They support Apptainer (formerly Singularity), which runs containers as your own user and works with MPI and GPUs. Docker and Kubernetes fit better for services than for tightly coupled MPI jobs.

Monitoring and good habits

• Use the provider's dashboards and job reports; many platforms run Prometheus and Grafana, Nagios or Zabbix behind them.
• Keep job scripts in version control alongside the code, so you know exactly which settings produced which result.
• Write down every tuning change and its measured effect. Six months later, that note is the difference between a known-good configuration and guesswork.
• Rerun your benchmarks after major updates to the platform, compilers or MPI libraries.

Where to start

Run seff or sacct on your last ten jobs. If CPU efficiency is low, fix the resource requests first: it costs nothing and frees up allocation. Then bind processes to cores, check your I/O pattern, and only then start profiling the code. If you also run machine learning workloads on the same platform, see our guide to deploying Python machine learning models on MCP servers. The same profile-first approach applies to web code too: our PHP testing and profiling tools guide covers it for WordPress.