這個問題在這里已經有了答案： 如何防止 tensorflow 分配整個 GPU 記憶體？ (16 個答案) 3天前關閉。
我在使用 TensorFlow (2.9.1) 時遇到了一個奇怪的問題。定義分布式訓練策略后，我的 GPU 記憶體似乎已滿。
重現的步驟很簡單：
import tensorflow as tf
strat = tf.distribute.MirroredStrategy()
在第一行（匯入 TensorFlow）之后，nvidia-smi輸出：
Fri Jun 10 03:01:47 2022
-----------------------------------------------------------------------------
| NVIDIA-SMI 470.103.01 Driver Version: 470.103.01 CUDA Version: 11.4 |
|------------------------------- ---------------------- ----------------------
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=============================== ====================== ======================|
| 0 Quadro P6000 Off | 00000000:04:00.0 Off | Off |
| 26% 25C P8 9W / 250W | 0MiB / 24449MiB | 0% Default |
| | | N/A |
------------------------------- ---------------------- ----------------------
| 1 Quadro P6000 Off | 00000000:06:00.0 Off | Off |
| 26% 20C P8 7W / 250W | 0MiB / 24449MiB | 0% Default |
| | | N/A |
------------------------------- ---------------------- ----------------------
-----------------------------------------------------------------------------
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
-----------------------------------------------------------------------------
在第二行代碼之后，nvidia-smi輸出：
Fri Jun 10 03:02:43 2022
-----------------------------------------------------------------------------
| NVIDIA-SMI 470.103.01 Driver Version: 470.103.01 CUDA Version: 11.4 |
|------------------------------- ---------------------- ----------------------
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=============================== ====================== ======================|
| 0 Quadro P6000 Off | 00000000:04:00.0 Off | Off |
| 26% 29C P0 59W / 250W | 23951MiB / 24449MiB | 0% Default |
| | | N/A |
------------------------------- ---------------------- ----------------------
| 1 Quadro P6000 Off | 00000000:06:00.0 Off | Off |
| 26% 25C P0 58W / 250W | 23951MiB / 24449MiB | 0% Default |
| | | N/A |
------------------------------- ---------------------- ----------------------
-----------------------------------------------------------------------------
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1833720 C python 23949MiB |
| 1 N/A N/A 1833720 C python 23949MiB |
-----------------------------------------------------------------------------
GPU 記憶體幾乎已滿？還有一些終端輸出：
2022-06-10 03:02:37.442336: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-06-10 03:02:39.136390: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 23678 MB memory: -> device: 0, name: Quadro P6000, pci bus id: 0000:04:00.0, compute capability: 6.1
2022-06-10 03:02:39.139204: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 23678 MB memory: -> device: 1, name: Quadro P6000, pci bus id: 0000:06:00.0, compute capability: 6.1
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1')
關于為什么會發生這種情況的任何想法都會有所幫助！有關我的配置的其他詳細資訊：
- Linux 上的 Python 3.10.4 [GCC 7.5.0]
- 張量流 2.9.1
- cuda/11.2.2 cudnn/v8.2.1
uj5u.com熱心網友回復：
默認情況下，Tensorflow 將映射幾乎所有的 GPU 記憶體：官方指南。這是出于性能原因：通過分配 GPU 記憶體，它減少了記憶體增長通常會導致的延遲。
您可以嘗試使用tf.config.experimental.set_memory_growth以防止它立即填滿所有記憶體。這篇StackOverflow 帖子也有一些很好的解釋。
轉載請註明出處，本文鏈接：https://www.uj5u.com/caozuo/489707.html