HPC & Data Tools / Built for research
Make HPC work easier to run, understand, and improve.
I build practical software for researchers and HPC users: clearer job performance, simpler distributed setup, and faster ways to explore data and storage.
From the job to the data
- 01Understand the runJob telemetry and resource use
- 02Scale the analysisDistributed Dask workflows
- 03Explore the dataZarr stores and file system usage
Why these tools exist
Less time wrestling with HPC. More time on the research.
See what jobs actually used
Turn resource telemetry into a clearer picture of CPU, memory, I/O, and GPU use, so the next run is easier to plan.
HPC Dashboard · logger-rs
Get distributed work running
Spend less effort assembling a Dask environment and more effort analysing the datasets that matter.
dask_setup
Understand data and storage
Inspect Zarr stores quickly and find what is using space in large directory trees.
zarr-dump · rudu
HPC Dashboard + logger-rs
From a job script to useful answers.
logger-rs records what your workload uses. HPC Dashboard reads those files and helps you understand the run.
- 01Mark the workloadWrap the science in
telemetry_startandtelemetry_stop. Setup and data staging stay outside the measurement. - 02Collect job telemetrylogger-rs samples resource use while the job runs and writes the results to shared storage.
- 03Review the runOpen the data in HPC Dashboard to inspect CPU, memory, I/O, and GPU use and plan the next request.
From the Gadi PBS example
module use /g/data/gb02/modules
module load logger-rs
source logger-rs.sh
export HPC_DASHBOARD_DIR="/g/data/ab12/dashboard/$USER"
telemetry_start
mpirun -np "$PBS_NCPUS" ./my_model
telemetry_stopUse your own allocation, storage path, and workload. The measured region contains the model run.
View the complete example job ↗The tools
Practical software for real HPC problems.
Explore each project to see how it works and find the source code.
- Featured tool
HPC Dashboard
A desktop dashboard for inspecting HPC job performance, including CPU, memory, I/O, and GPU use, with practical recommendations for future runs.
HPCMonitoringDesktop App Explore project - 02 / 05
logger-rs
A Rust telemetry collector that samples job resource use and writes it to NDJSON for analysis in HPC Dashboard.
RustHPCTelemetry Explore project - 03 / 05
dask_setup
Helpers for starting and managing distributed Dask workloads on HPC systems, including PBS and Slurm clusters.
PythonDaskDistributed Computing Explore project - 04 / 05
zarr-dump
A Rust command-line tool for quickly inspecting Zarr store metadata in a familiar NetCDF-style view.
RustZarrCLI Explore project - 05 / 05
rudu
A Rust-powered disk usage tool with parallel scanning and memory limits, designed to handle large and deep file systems.
RustCLIHPC Explore project
Get in touch
Have an HPC problem to solve?
Tell me what you are trying to run, understand, or improve. I would be glad to hear about the problem.