-
Notifications
You must be signed in to change notification settings - Fork 17
Add "dash" options to the bash scripts, and some additional refactoring #15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
mpaiao
wants to merge
15
commits into
monanadmin:develop
Choose a base branch
from
mpaiao:use_dash-develop
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1. All bash scripts in scripts_CD-CT/scripts now use dash-based flags to specify the
settings. For example, when the previous configuration for script 3.run_model.bash
used to be:
- Original:: ./3.run_model.bash GFS 1024002 2024010100 48
- Proposed: ./3.run_model.bash -e GFS -r 1024002 -i 2024010100 -f 48 [...]
The main advantage of the proposed is that it is more transparent, and also does not
require any specific order for the arguments.
2. Multiple settings that were hardcoded across multiple scripts are now arguments:
-d OUTPUT_DIAG_INTERVAL: this specifies how frequently output variables should be
generated.
-i INPUT_PATH: if provided, this replaces the default input data path with specific
paths in setenv.bash. This can be useful when testing new datasets.
-l NLEV: the number of vertical levels
-v VARTABLE: if provided, the suffix for specific variable tables (e.g., OPER, GFS)
-z: this deletes all paths containing compiled code and output paths
(sources, execs, datain, dataout and run.*)
3. Script 0.run_all.bash was rewritten such that all settings can be specified by editing
the beginning of the script (as it used to be), or through the dash-based flags. This
is the only script that has default settings. In addition, a new flag step (-s [0-4])
can be used to specify which step to run. Options are 1-4 for the numbered scripts,
and 0 to run all steps. This change eliminates the need for commenting/uncommenting
steps. Ultimately, the idea is for most users to call 0.run_all.bash only and use the
step argument to control the steps, and let 0.run_all.bash parse arguments to the other
numbered scripts.
4. Script 2.pre_processing.bash now accepts an optionial flag for overwriting the static
file (-o). If this flag is provided, the step will generate the static file again even
if the file already exists. This option is relevant for when testing MONAN developments
that change the contents of the static file (e.g., soil colour maps).
5. All scripts were revised so all the standard resolutions (10, 15, 24, 120 km) were
consistent and supported in all steps. In addition, the number of grid cells when
reprojecting MONAN's output to regular longitude/latitude grids was slightly revised in
scripts 4.run_post.bash and make_template.bash. The original calculation assumed 1
degree ~ 100 km, which makes the reprojected grid a bit coarser than it needs to be.
The new calculation uses the Earth's total surface area and the number of grid cells as
follows:
- delta = sqrt ( 4 * pi * (180 /pi)^2 / NumberOfGridCells )
- PointsPerDegree = round (1/Delta)
6. A few minor additional bug fixes and edits for clarity.
1. Added option -h to show usage to all scripts. 2. Added option -m12 to try to run older versions of MONAN (based on 1.2.0-rc) on Jaci. If this runs, I will make another commit to add some post-processing code too.
…flag, and edited setenv.bash to provide a unique name for the true/false version of the -m12 flag.
…PUT_PATH must update the system-specific setenv.bash
It could be a PBS problem. I attempted to fix by defining the paths DIR_SCRIPTS and DIR_DADOS based on the actual location of setenv.bash instead of $(pwd)
…nge when switching to another model)
…ith scripts_CD-CT.
…ripts_CD-CT can handle MONAN versions based on 1.2.0-rc. These can be removed again once the versions containing Noah-MP are fully integrated into later releases.
… to setenv.bash. The same directory names were set in every bash script, so defining them in setenv.bash helps streamline the code and avoids inconsistencies.
…ranch, so the output was not properly set. I also slightly edited the logic for reporting the error, so all expected files that were missing are listed in the output.
…MONAN code (and python requirements) on Jaci.
Author
|
All tests on Egeon and Jaci ran successfully, so I am flagging this as ready for review. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request supersedes #12, as it is now compatible with egeon and jaci.
Goals of this pull request
scripts_CD-CT, by defining an optional flag-m12. This was done because NoahMP is not available in the most recent MONAN version, and it was relatively straightforward to isolate the code needed for older versions of MONAN. This does add the python scripts back because we need the functionality to group vertical levels, but we can remove this oncefeature/MONAN-757-NFis merged to the newest version.0.run_all.bashand just call it multiple times for each step.Main updates and refactoring:
scripts_CD-CT/scriptsnow use dash-based flags to specify the settings. For example, below is an example of the differences between the original and proposed call for script3.run_model.bash:./3.run_model.bash GFS 1024002 2024010100 48./3.run_model.bash -e GFS -r 1024002 -i 2024010100 -f 48 [...]The main advantage of the proposed is that it is more transparent, and also does not require any specific order for the arguments.
Script
0.run_all.bashwas rewritten such that all settings can be specified by editing the beginning of the script (as it used to be), or through the dash-based flags. This is the only script that has default settings. In addition, a new flag step (-s [0-4]) can be used to specify which step to run. Options are 1-4 for the numbered scripts, and 0 to run all steps. This change eliminates the need for commenting/uncommenting steps. Ultimately, the idea is for most users to call0.run_all.bashonly and use the step argument to control the steps, and let0.run_all.bashparse arguments to the other numbered scripts.Multiple settings that were hardcoded across multiple scripts are now arguments:
-d OUTPUT_DIAG_INTERVAL: this specifies how frequently output variables should be generated.-h: this prints the list of all options for each script.-i INPUT_PATH: if provided, this replaces the default input data path with specific paths insetenv.bash. This can be useful when testing new datasets.-l NLEV: the number of vertical levels-v VARTABLE: if provided, the suffix for specific variable tables (e.g., OPER, GFS)-z: this deletes all paths containing compiled code and output paths (sources,execs,datain,dataoutandrun.*)Script
2.pre_processing.bashnow accepts an optionial flag for overwriting the static file (-o). If this flag is provided, the step will generate the static file again even if the file already exists. This option is relevant for when testing MONAN developments that change the contents of the static file (e.g., soil colour maps).All scripts were revised so all the standard resolutions (3, 10, 15, 24, 30, 60, 120 km) are consistent and supported in all steps. In addition, the number of grid cells when reprojecting MONAN's output to regular longitude/latitude grids was slightly revised in scripts
4.run_post.bashandmake_template.bash. The original calculation assumed 1 degree ~ 100 km, which makes the reprojected grid a bit coarser than it needs to be. The new calculation uses the Earth's total surface area and the number of grid cells as follows:Most paths were defined multiple times across all scripts, sometimes using relative paths. This was causing some glitches. Following @marcelopaivaramos suggestion, paths are now set and exported on
setenv.bash.A few minor additional bug fixes and edits for clarity.
Testing and caveats
All numbered scripts have been successfully tested on both Jaci and Egeon. For the tests I used the following MONAN versions:
release/1.4.3-rc)feature/MONAN-757-NFCaveats:
feature/MONAN-757-NFwill not work on Jaci because it lacks the compiler settings in MONAN's Makefile. My fork offeature/MONAN-757-NFis identical to the original one, except that it adds the specific compiler settings for Jaci.setenv_PBS_ian.bashfrom `` tocdo/2.5.4. The older module had trouble reading NetCDF. It looks like cdo was not used until I added back some commands for older versions of MONAN, so maybe this doesn't affect anything.