[template] create templates for use in generating actions - #1282

Draft

cjac wants to merge 131 commits into

GoogleCloudDataproc:mainfrom

LLC-Technologies-Collier:template-gpu-20241219

Commits on Jan 9, 2025

[template] generate gpu/install_gpu_driver.sh from templates
cjac
committed
4f49f65
new hold nvidia packages function ; moved variable definition around a bit
cjac
committed
1dae02b
added two new gpu functions: configure_mig_cgi and enable_mig
cjac
committed
e97e376
templatized version of mig.sh
cjac
committed
310bb9d
comment fix-up
cjac
committed
912ebe7
nvidia-container-toolkit repo setup changes are working on rocky8
cjac
committed
87965de
defining variables in the generator script instead of duplicating in the root template ; do not hold nvidia packages in the prepare function
cjac
committed
93fe4cc
tested with debian12
cjac
committed
b82aadc
tested on 8x H100s with bookworm
cjac
committed
dd98436
created and called function enable_and_configure_mig
cjac
committed
b4dabad
moved comment to correct function
cjac
committed
edeab28
do not point to local rpm pgp key
cjac
committed
0e8946c
store completion signal files in their own directory
cjac
committed
c44195a
excessive sudo
cjac
committed
31d1a9e
install spark rapids in all cases
cjac
committed
4a3a8cd
merged spark-rapids functions into general gpu util_functions template
cjac
committed
6ab36a5
correcting variable name
cjac
committed
ef36694
using new function name
cjac
committed
af69141
driver version for 12.4.0 had not been tested in a while and had become incorrect
cjac
committed
d59d5e6
expanding non-default version tests ; adding utility function to verify pyspark
cjac
committed
8a9e00a
reduced boot disk size to 50GB
cjac
committed
1113855
skipping old cuda on new images ; sizing instances to build
cjac
committed
7034739
skipping older debuntu when cuda version not specified
cjac
committed
2873f49
refactor into functions
cjac
committed
576b32f
moved secure-boot utility functions and common environment setup into common/util_functions
cjac
committed
b03dc57
refactored exit_handler
cjac
committed
bf98d85
declaring constants prior to running functions
cjac
committed
4320953
removed old variables, included a current one which does not get exercise
cjac
committed
2b0947b
do not break if variable undefined
cjac
committed
5dbc1f2
order of operations error fixed with parantheses.
cjac
committed
c5d46d3
using lower xgboost version for older dataproc images
cjac
committed
7be62b3
test whether the variable is defined before testing its value
cjac
committed
41c327a
refactor the xgboost installer a little
cjac
committed
b70477b
only minor changes
cjac
committed
073ed1f
explicitly notifying at the completion of the main function
cjac
committed
4f66a51
moved trap outside of the template
cjac
committed
f8a9b7d
stop / start instead of restart
cjac
committed
19520b4
skipping install on gpu-less systems more quickly
cjac
committed
f659ec5
install_dependencies is called from base template prep function
cjac
committed
af817f0
re-thought about the dependencies install time
cjac
committed
2e7441b
refactored configure_gpu_exclusive_mode to fewer lines
cjac
committed
29631a0
refactored gpu-related code out of common function library ; less reactive to not having GPU
cjac
committed
7ea7653
being more surgical about signing material usage
cjac
committed
70349a6
removed dependency on pciutils ; defined is_debuntu with other os comparison functions
cjac
committed
b5473c5
again I meant elif
cjac
committed
be3dbf6
fall back on metadata value if modulus_md5sum variable undefined
cjac
committed
3ca8c91
switch to other build_dir variable assignment
cjac
committed
83d5ccc
parens
cjac
committed
93fdb30
allow failure when grepping PCI devices for 10DE
cjac
committed
917f4b6
removed listing of nodes_include ; does not work in custom-images context
cjac
committed
668db72
min spark version supported by newer rapids is insufficient ; xgboost version is fine where it is at
cjac
committed
2193c28
skipping fewer tests
cjac
committed
992d83a
simplified rapids / xgboost default version logic
cjac
committed
7170872
ubuntu sometimes takes a while to bring gcloud online
cjac
committed
b33cb27
only using 24.08.1 on 2.2 images ; fix a typo in a comment
cjac
committed
c56440a
refactored ; these files should be quite similar now
cjac
committed
f10df49
returning spark-rapids/* to master ; this version of these templates was used to generate a spark-rapids/spark-rapids.sh which passes all master tests
cjac
committed
f3a103e
return test suite to master
cjac
committed
0ac57a0
do not run all tests ; also do not retry failures
cjac
committed
b3e5618
expanding non-default version tests ; adding utility function to verify pyspark ; disk size correction
cjac
committed
b4e99ee
reverting to master
cjac
committed
e4eab7b
reverting test_spark-rapids.py to master
cjac
committed
95b17ac
do not consider templates as changed files
cjac
committed
212b9af
using nvsmi for some error protection
cjac
committed
e9b9e5d
corrected comments
cjac
committed
adf4312
defining xpath variables as local
cjac
committed
dfcd8b0
tested on 2.1-ubuntu20
cjac
committed
9bb4d66
using tests from https://github.com/GoogleCloudDataproc/initialization-actions/pull/1275
cjac
committed
17f0fe8
reducing resources for build cluster ; pause for gcloud
cjac
committed
f42a86d
exercising spark-rapids from this template
cjac
committed
811ad03
improved header documentation
cjac
committed
4378edd
generated from templates in commit d5f7ffb7cf19852e48ce17c9ffae3640e7b19ca2
cjac
committed
992bd14
replacing java spark tests with pyspark tests
cjac
committed
1d84952
pyspark test code
cjac
committed
88ccfec
corrected function signature
cjac
committed
282ca0c
fixing order of operations for setting default cuda version ; removed excess cuda default logic ; too many gs:// ; testing 2.0-rocky8 instead of 2.1-rocky8
cjac
committed
d6e9809
including verify_pyspark.py in data list
cjac
committed
e3df6f2
verifying with gcloud dataproc jobs submit pyspark instead of spark ; skipping all tests that use ssh
cjac
committed
89fe31b
re-enable ssh tests
cjac
committed
e221ede
refactored ssh command retry code into the base class
cjac
committed
c9950a8
remembered the imports ; sleep a random period
cjac
committed
8143d4c
A100->H100
cjac
committed
834f7d5
fixing whitespace for python
cjac
committed
3d83795
moved knox variables to common env ; renamed ambiguous variable name
cjac
committed
7718e5a
remove gpu related code from dask action
cjac
committed
aded30b
changing failure to warning
cjac
committed
f553371
removing more gpu stuff from dask
cjac
committed
0439a8d
moved MASTER global variable to common/util_functions
cjac
committed
0251358
correct variable name
cjac
committed
a643c9a
moved hold_nvidia_packages out of common environment prepare into gpu env prep ; removed accidental inclusion of original spark-rapids.sh
cjac
committed
d6867d9
added comments and timing collection
cjac
committed
6a7d10d
no need to consider unsupported dataproc < 2.0 image versions ; reducing instance type a little
cjac
committed
1ab3f8d
using "dask-scheduler" instead of "dask scheduler"
cjac
committed
598b690
wait for dask scheduler before starting worker
cjac
committed
81c7d28
using variable instead of my own cluster master name
cjac
committed
48906e1
corrected syntax errors ; dump log on service failure
cjac
committed
510e520
refactored some common code ; setting default value for metadata attribute correctly
cjac
committed
9e9f872
added new function is_ramdisk ; keeping conda cache in its own directory ; same for pip cache ; refactored pip setup and teardown
cjac
committed
7480a23
calling functions from refactored pip setup/teardown
cjac
committed
6b73d22
moved knox dask config to templates/dask/util_functions
cjac
committed
2e45a75
added copyright to templates/legal/license_header
cjac
committed
33fdd38
latest generated action
cjac
committed
4f974c5
removed redundant template disclaimer
cjac
committed
75d8e32
setup and tear-down for actions which work with conda
cjac
committed
34fce25
* refactored common conda installer functionality from dask.sh.in and
cjac
committed
bbe062e
tested rapids.sh init action with dataproc-repro
cjac
committed
10f1698
templates/dask/dask.sh.in,
cjac
committed
8a4cbd9
refactor yarn functions into their own template
cjac
committed
b01b867
refactor mig functions into their own template
cjac
committed
c6c09db
state before gpu rebranch
cjac
committed
88f9f7f
templates/common/util_functions:
cjac
committed
119f1b1
templates/dask/util_functions:
cjac
committed
d45e16b
refactored spark variable definition and reduced excess lines by bulking the readonly operations
cjac
committed
a7b4707
development on these scripts will happen in the spark-rapids-template-20241225 branch
cjac
committed
35ca704
revert dask/ to master
cjac
committed
43232b2
moving that .in suffix to the correct variable
cjac
committed
4b6e520
reverted to master ; changes ended up in gpu-template-20250107
cjac
committed
4a024e0
including libtemplate-perl as a dependency
cjac
committed
f00e2f8
moved to dask-template-20250104
cjac
committed
7118ebf
moved to gpu-template-20250107
cjac
committed
f2b50f7
* include version in template disclaimer
cjac
committed
900c10a

Commits on Jan 10, 2025

migrated rapids.sh base template to rapids-template-20250106
cjac
committed
bef08b1
script to generate all actions from templates
cjac
committed
aa792c3
spark prepare steps belong in common
cjac
committed
824bcf8

Commits on Jan 12, 2025

less noise in temp directory
cjac
committed
374ff96

Commits on Jan 16, 2025

Commits on Jan 20, 2025

create function to harden sshd config ; execute it before repairing old backports
cjac
committed
0c3eb51

Commits on Jan 23, 2025

reviewed #1275 and brought closer to parity
cjac
committed
576bbb6

Commits on Jan 27, 2025

Merge branch 'GoogleCloudDataproc:master' into template-gpu-20241219
cjac
authored
07949a9

Commits on Jan 29, 2025

changes from testing PR #1275
cjac
committed
989b445