torch.cuda.OutOfMemoryError: CUDA out of memory, GPU UP but utilization SLOW?
torch.cuda.OutOfMemoryError: CUDA out of memory, GPU UP but utilization SLOW?

torch.cuda.OutOfMemoryError: CUDA out of memory, GPU UP but utilization SLOW?

Hello

I was using this repo yesterday: https://github.com/saifhassan/Wav2Lip-HD

For some reason it worked yesterday but in the end of the night + the day after it is no longer functioning, it tells me: OUT OF MEMORY:

But I have LOT of GPU:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 15.82 GiB. GPU 0 has a total capacty of 23.99 GiB of which 0 bytes is free. Of the allocated memory 21.37 GiB is allocated by PyTorch, and 13.42 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I am bit confused, how can I resolve this?

What can I investigate?

(And yes I have restarted the pc to make sure there is nothing else running)

Oh and for some reason low utilisation % but high gpu?

https://imgur.com/KJpVXSb

submitted by /u/Unreal_777
[link] [comments]