[develop] Update CI/CD pipeline to work properly on Gaea C6 by MichaelLueken · Pull Request #1384 · ufs-community/ufs-srweather-app

MichaelLueken · 2026-08-18T14:00:42Z

DESCRIPTION OF CHANGES:

.cicd/scripts/wrapper_srw_ftest.sh - Updated job_id to include | awk -F';' '{print $1}' to allow the Functional WorkflowTaskTests stage to run on Gaea C6 following the SLURM update

Type of change

Bug fix (non-breaking change which fixes an issue)

TESTS CONDUCTED:

gaeac6.intel
Jenkins
coverage test suite

DEPENDENCIES:

None

DOCUMENTATION:

None

ISSUE:

N/A

CHECKLIST

My code follows the style guidelines in the Contributor's Guide
I have performed a self-review of my own code using the Code Reviewer's Guide
My changes do not require updates to the documentation (explain).
- This is an update to the Jenkins CI/CD scripts
My changes generate no new warnings
New and existing tests pass with my changes

CONTRIBUTORS

@kbooker79 - for discovering the appended ;c6 at the end of the job_id on Gaea C6 following the SLURM update and providing the fix.

* .cicd/scripts/wrapper_srw_ftest.sh - Updated job_id to include | awk -F';' '{print $1}' to allow the Functional WorkflowTaskTests stage to run on Gaea C6 following the SLURM update

MichaelLueken · 2026-08-18T14:12:29Z

Since the SLURM upgrade on Gaea C6, the Functional WorkflowTaskTests have been failing with messages similar to the following:

Running: sbatch -A epic --parsable /gpfs/f6/bil-fire8/scratch/Michael.Lueken/ufs-srweather-app/gaeac6/.cicd/scripts/sbatch_srw_ftest.sh
Waiting ten seconds for node to initialize
squeue: error: Invalid job id: 220189234;c6
Job has completed.
Missing results file! 
exit 1

During the SLURM upgrade, the Job ID was updated to include a semi-colon and the cluster the job was submitted on.

Setting the Job ID to just the Job ID and removing the semi-colon and the cluster, allows the Functional WorkflowTaskTests to run on the machine:

Running: sbatch -A epic --parsable /gpfs/f6/bil-fire8/scratch/Michael.Lueken/ufs-srweather-app/gaeac6/.cicd/scripts/sbatch_srw_ftest.sh
Waiting ten seconds for node to initialize
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes

MichaelLueken · 2026-08-18T15:59:43Z

The Gaea C6 Jenkins testing has successfully passed following the update in this PR.

Please review and approve this PR so that the nightly testing can once again resume on Gaea C6.

* .cicd/scripts/wrapper_srw_ftest.sh - Add a separate run for Derecho, since the fix doesn't work with PBS

MichaelLueken · 2026-08-18T17:59:11Z

The update to wrapper_srw_ftest.sh works on SLURM, but fails on Derecho (PBS). Updated to maintain current job submission on Derecho, and apply the fix for SLURM.

MichaelLueken · 2026-08-18T18:58:44Z

The update at hash 2ac20d3 allows the Functional WorkflowTaskTests to run on Derecho:

Running: qsub -A NRAL0032  /glade/derecho/scratch/mlueken/ufs-srweather-app/derecho/.cicd/scripts/qsub_srw_ftest.sh
Waiting ten seconds for node to initialize
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
qstat: 7145594.desched1 Job has finished, use -x or -H to obtain historical job information
Job has completed.
exit 0

It is also passing on Gaea C6:

Running: sbatch -A epic --parsable /gpfs/f6/bil-fire8/scratch/Michael.Lueken/ufs-srweather-app/gaeac6/.cicd/scripts/sbatch_srw_ftest.sh
Waiting ten seconds for node to initialize
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job has completed.
exit 0

Reopening PR now.

RatkoVasic-NOAA

[Ratko.Vasic@gaea65:/gpfs/f6/bil-fire8/world-shared/Ratko.Vasic/ufs-srweather-app/gaeac6/.cicd/scripts]# ./wrapper_srw_ftest.sh
Running: sbatch -A epic --parsable /gpfs/f6/bil-fire8/world-shared/Ratko.Vasic/ufs-srweather-app/gaeac6/.cicd/scripts/sbatch_srw_ftest.sh
Waiting ten seconds for node to initialize
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job has completed.
exit 0

Approved.

natalie-perlin · 2026-09-03T20:50:56Z

@RatkoVasic-NOAA @MichaelLueken - how do I run this test on gaeac6?
This does not work running from a terminal on Gaeac6 for me. Do I need to run it from Jenkins site?

MichaelLueken · 2026-09-03T20:55:57Z

@RatkoVasic-NOAA @MichaelLueken - how do I run this test on gaeac6? This does not work running from a terminal on Gaeac6 for me. Do I need to run it from Jenkins site?

@natalie-perlin - please follows these steps to clone my branch and run the Functional WorkflowTaskTests:

You will need to clone the branch so that it is ufs-srweather-app/gaeac6 (i.e., git clone -b bf/GaeaC6_CICD_fix git@github.com:MichaelLueken/ufs-srweather-app.git ufs-srweather-app/gaeac6)
You will need to set the following environmental variables:

SRW_PLATFORM="gaeac6"
SRW_COMPILER="intel"
WORKSPACE= "path_to_ufs-srweather_app" (i.e., "/gpfs/f6/bil-fire8/scratch/Michael.Lueken/ufs-srweather-app")
SRW_PROJECT="epic"

cd .cicd/scripts
./srw_build.sh (to build the SRW using the cicd scripts)
./wrapper_srw_ftest.sh (to run the Functional WorkflowTaskTests)

Please let me know if you have any questions or encounter issues.

natalie-perlin · 2026-09-03T21:59:53Z

All done!.. Worked well on gaeac6:

[Natalie.Perlin@gaea62:/gpfs/f6/bil-fire8/scratch/Natalie.Perlin/SRW/ufs-srweather-app/gaeac6/.cicd/scripts]$ ./wrapper_srw_ftest.sh 
Running: sbatch -A epic --parsable /gpfs/f6/bil-fire8/scratch/Natalie.Perlin/SRW/ufs-srweather-app/gaeac6/.cicd/scripts/sbatch_srw_ftest.sh
Waiting ten seconds for node to initialize
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job is still running. Check again in two minutes
Job has completed.
exit 0

Approving it now.

natalie-perlin

Thank you, approved!

Feature/lbcs fcst cycles - bring recent changes from ufs-community#1384 (ufs-community#1384)

[bf/GaeaC6_CICD_fix]:

d7f0eb2

* .cicd/scripts/wrapper_srw_ftest.sh - Updated job_id to include | awk -F';' '{print $1}' to allow the Functional WorkflowTaskTests stage to run on Gaea C6 following the SLURM update

MichaelLueken added the Priority: HIGH label Aug 18, 2026

MichaelLueken added this to SRW App's PR Reviewers Aug 18, 2026

MichaelLueken added the run_we2e_coverage_tests Run the coverage set of SRW end-to-end tests label Aug 18, 2026

MichaelLueken marked this pull request as ready for review August 18, 2026 15:57

MichaelLueken requested review from EdwardSnyder-NOAA, JeffBeck-NOAA, RatkoVasic-NOAA, WeirAE, benkozi, chan-hoo, christinaholtNOAA, gsketefian, gspetro-NOAA, jdkublnick, jkbk2004, natalie-perlin, panll, priyarpillai, rickgrubin-noaa, selbigmtnwx23 and ywangwof as code owners August 18, 2026 15:57

MichaelLueken marked this pull request as draft August 18, 2026 17:48

[bf/GaeaC6_CICD_fix]:

2ac20d3

* .cicd/scripts/wrapper_srw_ftest.sh - Add a separate run for Derecho, since the fix doesn't work with PBS

MichaelLueken marked this pull request as ready for review August 18, 2026 18:58

MichaelLueken moved this to In Review in SRW App's PR Reviewers Aug 18, 2026

RatkoVasic-NOAA approved these changes Sep 3, 2026

View reviewed changes

natalie-perlin approved these changes Sep 4, 2026

View reviewed changes

MichaelLueken merged commit 570c047 into ufs-community:develop Sep 4, 2026
7 of 8 checks passed

github-project-automation Bot moved this from In Review to Done in SRW App's PR Reviewers Sep 4, 2026

MichaelLueken mentioned this pull request Sep 4, 2026

Updates to the authoritative develop branch since the SRW v3.0.0 release (04/09/2025) #1283

Open

MichaelLueken deleted the bf/GaeaC6_CICD_fix branch September 4, 2026 13:35

natalie-perlin added a commit to natalie-perlin/ufs-srweather-app that referenced this pull request Sep 4, 2026

Merge pull request #13 from MichaelLueken/feature/lbcs_fcst_cycles

bee7b50

Feature/lbcs fcst cycles - bring recent changes from ufs-community#1384 (ufs-community#1384)

MichaelLueken removed this from SRW App's PR Reviewers Sep 4, 2026

Navigation Menu

Uh oh!

[develop] Update CI/CD pipeline to work properly on Gaea C6 - #1384

[develop] Update CI/CD pipeline to work properly on Gaea C6#1384
MichaelLueken merged 2 commits into
ufs-community:developfrom
MichaelLueken:bf/GaeaC6_CICD_fix

MichaelLueken commented Aug 18, 2026 •

edited

Loading

Uh oh!

MichaelLueken commented Aug 18, 2026 •

edited

Loading

Uh oh!

MichaelLueken commented Aug 18, 2026

Uh oh!

MichaelLueken commented Aug 18, 2026

Uh oh!

MichaelLueken commented Aug 18, 2026

Uh oh!

RatkoVasic-NOAA left a comment

Uh oh!

natalie-perlin commented Sep 3, 2026

Uh oh!

MichaelLueken commented Sep 3, 2026

Uh oh!

natalie-perlin commented Sep 3, 2026

Uh oh!

natalie-perlin left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

MichaelLueken commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

DESCRIPTION OF CHANGES:

Type of change

TESTS CONDUCTED:

DEPENDENCIES:

DOCUMENTATION:

ISSUE:

CHECKLIST

CONTRIBUTORS

Uh oh!

MichaelLueken commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MichaelLueken commented Aug 18, 2026

Uh oh!

MichaelLueken commented Aug 18, 2026

Uh oh!

MichaelLueken commented Aug 18, 2026

Uh oh!

RatkoVasic-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

natalie-perlin commented Sep 3, 2026

Uh oh!

MichaelLueken commented Sep 3, 2026

Uh oh!

natalie-perlin commented Sep 3, 2026

Uh oh!

natalie-perlin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

MichaelLueken commented Aug 18, 2026 •

edited

Loading

MichaelLueken commented Aug 18, 2026 •

edited

Loading