7  Computing Resources

Because our work is quantitative, research computing skills are an important part of your toolset. We use Python as our primary programming language, with pyemu as the main interface to PEST++ and FloPy for working with MODFLOW 6.

7.1 Setting Up Your Local Environment

7.1.1 Python with Miniforge

We use Miniforge to manage Python environments. Miniforge is a lightweight conda-compatible package manager that uses the conda-forge channel by default.

  1. Download and install Miniforge from github.com/conda-forge/miniforge
  2. Each project in the lab maintains an environment.yml file that specifies all required packages. Create and activate an environment from it:
mamba env create -f environment.yml
mamba activate <env_name>

Here is an example environment.yml from the mf_insar project:

name: mf_insar
channels:
  - conda-forge
  - nodefaults
dependencies:
  - python=3.11
  - pandas<3.0
  - numpy>=1.25
  - pip
  - jupyter
  - jupytext
  - jupyterlab
  - openpyxl
  - xlrd
  - netcdf4
  - pyshp
  - rasterio
  - rasterstats
  - fiona
  - descartes
  - pyproj
  - shapely
  - geos
  - geojson
  - geopandas
  - xarray
  - rioxarray
  - uxarray
  - pyyaml
  - rtree
  - pyvista
  - vtk
  - imageio
  - requests
  - pytest
  - statsmodels
  - modflowapi
  - pymetis
  - conda-pack
  - scikit-image
  - pip:
      - git+https://github.com/pypest/pyemu.git@develop
      - git+https://github.com/modflowpy/flopy.git@develop
      - pypestutils
      - scores
      - dataretrieval
  1. Download MODFLOW 6 and PEST++ executables from their latest GitHub releases. For example, on macOS (Apple Silicon):
# MODFLOW 6 — check https://github.com/MODFLOW-USGS/modflow6/releases for the latest version
curl -L -O https://github.com/MODFLOW-USGS/modflow6/releases/download/6.X.X/mf6.6.X.X_mac.zip
unzip mf6.6.X.X_mac.zip

# PEST++ — check https://github.com/usgs/pestpp/releases for the latest version
curl -L -O https://github.com/usgs/pestpp/releases/download/5.X.X/pestpp-5.X.X-mac-arm64.zip
unzip pestpp-5.X.X-mac-arm64.zip

Replace X.X with the current release version numbers. Place the executables somewhere on your PATH (e.g. ~/bin/) or in your project directory.

Tip

Download executables once per project and commit them to your GitHub repository. This ensures everyone working on the project uses the same version of MODFLOW 6 and PEST++, which is important for reproducibility.

7.1.2 Git and GitHub

All code should be version controlled and hosted in the markovich-lab GitHub organization. See the Introduction chapter for our code sharing practices.

7.2 The Markovich Lab Cluster

Our cluster is located in Northrop 207 and consists of the following hardware:

  • Head node: Mac mini M4 Pro (64 GB RAM, 14 CPU cores) — this is where you submit jobs and interact with the cluster
  • Worker nodes: 6× Mac mini M4 Pro (48 GB RAM, 14 CPU cores each) — these execute your model runs in parallel
  • Storage: Synology RS1221+ NAS (48 TB usable) — for archiving completed runs and shared lab data
  • Total compute: 84 CPU cores across 6 worker nodes

The cluster uses HTCondor for job scheduling and PEST++ with the PANTHER run manager for parallel model execution.

7.3 Getting Access to the Cluster

To get access to the cluster, you will need:

  1. A user account on the head node — Katie will create this for you during onboarding. You will receive a username and temporary password.
  2. Tailscale installed on your laptop — Tailscale is a VPN that allows you to securely connect to the cluster from anywhere.

7.3.1 Installing Tailscale

  1. Download Tailscale from tailscale.com/download
  2. Install and open the app
  3. Sign in — Katie will send you an invitation link to join the lab’s Tailscale network
  4. Once approved, the head node will appear in your Tailscale network

7.3.2 Connecting to the Head Node

Once Tailscale is running, connect via SSH:

ssh yourusername@100.115.148.79

You will be prompted for your password on first login. We recommend setting up SSH keys to avoid entering your password every time:

# On your laptop, generate a key pair (skip if you already have one)
ssh-keygen -t ed25519 -C "your.email@unm.edu"

# Copy your public key to the head node
ssh-copy-id yourusername@100.115.148.79

7.4 Shared Storage

The NAS is mounted at /Volumes/labdata on the head node and all worker nodes. Use this for:

  • Archiving completed PEST++ runs
  • Sharing data with other lab members

Your personal directory is at /Volumes/labdata/students/yourusername/. Please keep active run directories on the head node’s local disk (~/) and move completed runs to the NAS when finished.

# Archive a completed run directory to NAS
tar -czf /Volumes/labdata/students/yourusername/my_run.tar.gz ~/my_run

7.5 Submitting Jobs with HTCondor

HTCondor manages job scheduling on the cluster. Jobs are submitted from the head node using a submit file.

# HTCondor submit file for a PEST++ parallel run

# Job type — vanilla is standard for most jobs
universe = vanilla

# The script that runs on each worker
executable = worker.sh

# Arguments passed to worker.sh: head node IP and PANTHER port
arguments  = 192.168.1.10 4004

# Log files for monitoring and debugging
log        = log/job.log    # HTCondor event log
output     = log/job.out    # Worker stdout
error      = log/job.err    # Worker stderr

# Resource requests per worker
request_cpus   = 1          # CPU cores per worker
request_memory = 4g         # RAM per worker
request_disk   = 12g        # Scratch disk per worker

# Transfer input files to each worker before running
should_transfer_files   = YES
when_to_transfer_output = ON_EXIT
transfer_input_files    = model.tar.gz, mf_env.tar.gz

# Number of workers to launch (one per realization)
queue 84

Submit with:

condor_submit my_job.sub

7.5.1 Useful HTCondor Commands

Command Description
condor_submit job.sub Submit a job
condor_q Show your queued and running jobs
condor_status Show all worker nodes and their status
condor_rm <jobid> Remove a job from the queue
condor_q -better-analyze Diagnose why a job isn’t running

7.6 Getting Started with PEST++ on the Cluster

A good starting point is the lab’s tutorial repository (link to be added), which contains a simple synthetic MODFLOW 6 model and a step-by-step PEST++ ensemble calibration workflow designed to run on the cluster. Work through this before setting up your own project.

For the underlying concepts and more advanced workflows, the GMDSI notebooks repository is the most comprehensive resource available and closely mirrors how we approach parameter estimation in the lab.

7.7 UNM Center for Advanced Research Computing (CARC)

UNM’s Center for Advanced Research Computing (CARC) provides access to significantly larger compute resources and is free to UNM faculty, staff, and students.

Important

Our current high-throughput PEST++ workflow (PANTHER run manager with HTCondor) does not translate directly to CARC’s shared HPC environment, as CARC’s network configuration does not support the TCP/IP master/worker communication pattern that PANTHER requires. CARC is best suited for other compute-intensive analyses that are truly parallel in nature — for example, large-scale ensemble simulations, machine learning model training, or analyses that can be expressed as independent batch jobs via SLURM.

7.7.1 CARC Systems

CARC’s current primary cluster is Easley, a mixed CPU/GPU cluster with:

  • 65 compute nodes
  • 4,160 total CPU cores
  • 23.3 TB total RAM
  • NVIDIA L40s and H100 GPUs

CARC uses SLURM for job scheduling.

7.7.2 Getting Access to CARC

Katie does not currently have an active CARC project. If you have a need for CARC resources, discuss it with Katie and she will set up a project.

  1. Create a CARC account at carc.unm.edu/new-users/getting-started-at-carc1.html using your UNM email
  2. Ask Katie to add you to the lab’s CARC project via ColdFront
  3. Once added, you can log in via SSH and submit jobs

7.7.3 CARC Resources

7.8 Learning Resources

7.8.1 Groundwater Fundamentals

  • GW-Project: Analytical Groundwater Modeling — free interactive textbook covering the fundamentals of groundwater flow, a good starting point if you are new to the field
  • pycap-dss — Python decision support tool for evaluating analytical solutions for drawdown and streamflow depletion from high-capacity wells; useful for simple well response analyses

7.8.2 MODFLOW 6 and FloPy

7.8.3 PEST++ and Parameter Estimation

7.8.4 Time Series Analysis

  • Pastas — Python package for time series analysis of groundwater head data; useful for understanding aquifer responses to stresses

7.8.5 High Throughput Computing

7.8.6 Python and Scientific Computing

7.9 Getting Help

If you have trouble accessing the cluster or submitting jobs, post in the lab’s Discussions page or contact Katie directly.