mir/criu
2
0
Fork 0
mirror of https://github.com/checkpoint-restore/criu synced 2025-09-05 16:55:43 +00:00
Code Issues Releases Wiki Activity

cuda: prevent task lockup on timeout error

Browse Source 
When creating a checkpoint of large models, the `checkpoint` action of
`cuda-checkpoint` can exceed the CRIU timeout. This causes CRIU to fail
with the following error, leaving the CUDA task in a locked state:

	cuda_plugin: Checkpointing CUDA devices on pid 84145 restore_tid 84202
	Error (criu/cr-dump.c:1791): Timeout reached. Try to interrupt: 0
	Error (cuda_plugin.c:139): cuda_plugin: Unable to read output of cuda-checkpoint: Interrupted system call
	Error (cuda_plugin.c:396): cuda_plugin: CHECKPOINT_DEVICES failed with
	net: Unlock network
	cuda_plugin: finished cuda_plugin stage 0 err -1
	cuda_plugin: resuming devices on pid 84145
	cuda_plugin: Restore thread pid 84202 found for real pid 84145
	Unfreezing tasks into 1
		Unseizing 84145 into 1
	Error (criu/cr-dump.c:2111): Dumping FAILED.

To fix this, we set `task_info->checkpointed` before invoking
the `checkpoint` action to ensure that the CUDA task is resumed
even if CRIU times out.

Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
This commit is contained in:
Radostin Stoyanov
2024-12-14 21:14:58 +00:00
committed by Andrei Vagin
parent 5513a33300
commit 59b022db35
1 changed files with 2 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
plugins/cuda/cuda_plugin.c
View File

@@ -391,14 +391,14 @@ int cuda_plugin_checkpoint_devices(int pid)
if (resume_restore_thread(restore_tid, &save_sigset)) {
return -1;
}
task_info->checkpointed = 1;
status = cuda_process_checkpoint_action(pid, ACTION_CHECKPOINT, 0, msg_buf, sizeof(msg_buf));
if (status) {
pr_err("CHECKPOINT_DEVICES failed with %s\n", msg_buf);
goto interrupt;
}
task_info->checkpointed = 1;
interrupt:
int_ret = interrupt_restore_thread(restore_tid, &save_sigset);