Sometimes 24 hours are not enough to train your model. Yet big compute providers like the J\u00fclich Supercomuting Centre enforce 24h job limits to ensure everyone gets access.<\/p>\n\n\n\n
Chaining jobs allows us to split work into parts without having to submit every part independently by hand. Here is a minimal example:<\/p>\n\n\n\n
# python_train.py\n\nimport pickle\nimport os\nimport subprocess\n\nif __name__ == '__main__':\n if not os.path.exists('model.pkl'):\n model_var = 0\n else:\n with open('model.pkl', 'rb') as f:\n model_var = pickle.load(f)\n\n model_var += 1\n with open('model.pkl', 'wb') as f:\n model = pickle.dump(model_var, f)\n print(f\"saved model: {model_var}\")\n if model_var < 5:\n subprocess.call(\"sbatch\",\n \"run_chain.sh\")\n else:\n print(\"Model trained! Done!\")<\/code><\/pre>\n\n\n\nThe script above uses pickle to simulate your code’s results. Store it in a file called python_train.py<\/code>. It checks the model_var<\/code> and resubmits your code until a condition is met. A slurm file like the one below lets you run this example:<\/p>\n\n\n\n#!\/bin\/bash\n#\n#SBATCH -A TODO:your-account-name\n#SBATCH --nodes=1\n#SBATCH --job-name=test_chain\n#SBATCH --output=test_chain-%j.out\n#SBATCH --error=test_chain-%j.err\n#SBATCH --time=00:05:00\n#SBATCH --partition develbooster\n\nmodule load Python\n\npython python_train.py<\/code><\/pre>\n\n\n\nCall the file run_chain.sh<\/code>.<\/p>\n\n\n\nSubmit everything with sbatch run_chain.sh<\/code> and voil\u00e0, your job executes itself in a chain. This is useful if you are dealing with deep learning code that needs more than 24h to converge. The pickle code in the example is meant to simulate model storage and loading from disc.<\/p>\n","protected":false},"excerpt":{"rendered":"Sometimes 24 hours are not enough to train your model. Yet big compute providers like the J\u00fclich Supercomuting Centre enforce 24h job limits to ensure everyone gets access. Chaining jobs allows us to split work into parts without having to submit every part independently by hand. Here is a minimal example: The script above uses … <\/p>\n