[develop] Port SRW to Ursa - #1322
Merged
Merged
Conversation
RatkoVasic-NOAA
requested review from
EdwardSnyder-NOAA,
JeffBeck-NOAA,
MichaelLueken,
WeirAE,
benkozi,
chan-hoo,
christinaholtNOAA,
gsketefian,
gspetro-NOAA,
jdkublnick,
jkbk2004,
natalie-perlin,
panll,
priyarpillai,
rickgrubin-noaa,
selbigmtnwx23 and
ywangwof
as code owners
September 17, 2025 13:59
Collaborator
|
I'm able to successfully build the SRW App on Ursa using your It looks like Ursa is calling HTAR using the name of the service node rather than the user that submitted the job. The service nodes can't authenticate into HPSS, so the HTAR command fails, resulting in the necessary files not being found and retrieved. Did you experience this while testing on Ursa? It's unclear if this is an issue with the build of HPSS/HTAR on Ursa, or how we should go about ensuring that the user that launched the experiment is used for authenticating purposes. |
Collaborator
Author
|
I didn't have that problem when I was running on Ursa (GNU and Intel). I'm running again now. Will let you know how it went. |
Collaborator
|
While submitting jobs, I've been seeing that the However, |
Collaborator
Author
|
Not sure what is going on, here is what I see (my latest Ursa/Intel run - fundamental tests): And in And: What is your path, so I can take a look? |
Collaborator
Sure, please see |
Collaborator
|
It seems like the majority of the six 3km WE2E tests that wouldn't submit the It's not clear to me how to further adjust the |
Collaborator
|
Of the six 3km tests that failed originally, the following four still fail due to OOM: The |
Collaborator
|
This is the status of the six 3km WE2E tests on Ursa: All failed in |
Collaborator
Author
|
I used Ursa: If we run on Ursa using Hera's job card, it fails since we are asking for too many nodes. Maybe some other partition allows more nodes? I didn't go through Ursa's documentation. |
Collaborator
|
I was able to get all 3km WE2E tests to successfully pass by:
I'll try running the full comprehensive test suite on Ursa with this modification now. |
Collaborator
|
The comprehensive WE2E tests have successfully passed on Ursa, following the changes from the above comment: The UFS Fire WE2E tests have successfully passed: The AQM WE2E test failed. In The Following the above changes to the modulefiles, the AQM WE2E tests successfully passed: I was able to successfully build the SRW using GNU and the fundamental WE2E tests passed: The UFS Fire GNU WE2E tests also passed: The SRW successfully builds on Ursa using the Jenkins build script. The |
Collaborator
Author
|
@MichaelLueken Great news, could you please create PR to my branch with necessary changes? |
Collaborator
|
I have opened PR #16 in your branch. This addresses issues with the AQM WE2E test on Ursa. I have also wrapped running the Jenkins run script for the Intel coverage WE2E suite on Ursa. All tests successfully passed: Once you have merged in PR #16, I should be able to approve this PR. Thank you very much! |
[ursa] AQM WE2E test updates
MichaelLueken
approved these changes
Sep 26, 2025
MichaelLueken
left a comment
MichaelLueken
left a comment
Collaborator
There was a problem hiding this comment.
Thanks, @RatkoVasic-NOAA!
Approving PR now.
EdwardSnyder-NOAA
requested changes
Sep 29, 2025
EdwardSnyder-NOAA
left a comment
EdwardSnyder-NOAA
left a comment
Collaborator
There was a problem hiding this comment.
Left a couple of comments; just looking for clarity on a few things. Otherwise, built the SRW App with Intel and GNU on Ursa using the devbuild.sh script.
The fundamental suite passed with Intel:
----------------------------------------------------------------------------------------------------
Experiment name | Status | Core hours used
----------------------------------------------------------------------------------------------------
grid_RRFS_CONUScompact_25km_ics_RRFS_lbcs_RRFS_suite_RRFS_sas_202 COMPLETE 17.55
custom_ESGgrid_Central_Asia_3km_20250929174955 COMPLETE 51.96
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot COMPLETE 20.57
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2025092 COMPLETE 54.28
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20250929175 COMPLETE 23.47
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2025092917505 COMPLETE 39.05
----------------------------------------------------------------------------------------------------
Total COMPLETE 206.88
And tested one WE2E test with GNU, which passed too:
----------------------------------------------------------------------------------------------------
Experiment name | Status | Core hours used
----------------------------------------------------------------------------------------------------
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20250929192 COMPLETE 27.24
----------------------------------------------------------------------------------------------------
Total COMPLETE 27.24
…nd Ursa, and paths to data on Hera and Ursa
Update documentation
EdwardSnyder-NOAA
approved these changes
Sep 30, 2025
MichaelLueken
added
the
run_we2e_coverage_tests
Collaborator
|
The |
Collaborator
|
The NOAA Cloud WE2E tests failed due to missing staged data on those platforms. The WE2E tests on Derecho, Hera Intel, and Ursa Intel were aborted due to the Test phase lasting longer than 8 hours. The WE2E tests on Hera GNU and Ursa GNU failed while trying to pull data from HPSS. These jobs normally fail initially, but rocotorewind/rocotoboot corrects the failure. The Hercules and Orion WE2E tests successfully passed. The Hera and Ursa WE2E tests are being run manually. Once complete, the summary logs will be pasted in this PR. Once they pass, this PR will be merged. |
Collaborator
|
The Ursa Intel WE2E coverage tests have successfully passed: The Hera Intel WE2E coverage tests have successfully passed: |
Collaborator
|
The Ursa GNU coverage WE2E tests have successfully passed: The Hera GNU coverage WE2E tests have successfully passed: The Derecho fundamental WE2E tests have successfully passed: The Gaea C6 coverage WE2E tests have successfully passed: Merging this PR now. |
github-project-automation
Bot
moved this from In Review
to Done
in SRW App's PR Reviewers
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DESCRIPTION OF CHANGES:
Port SRW to new Ursa machine
Type of change
TESTS CONDUCTED:
DEPENDENCIES:
NONE
DOCUMENTATION:
Included
ISSUE:
Closes #1188 issue
CHECKLIST
LABELS (optional):
A Code Manager needs to add the following labels to this PR:
NOTE for Code Manager:
Once merged, add new machine to this template. While there, replace "Cheyenne" with "Derecho".