RTX-30xx GPU에서 CUDA 인식을 잘 못할때 해결법(+ Tensorflow gpu 연결 확인)

$ python
>>> from tensorflow.python.client import device_lib
>>> print(device_lib.list_local_devices())

2021-06-24 18:50:18.130906: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2021-06-24 18:50:18.205333: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
2021-06-24 18:50:18.232026: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce RTX 3070 major: 8 minor: 6 memoryClockRate(GHz): 1.725
pciBusID: 0000:2b:00.0
2021-06-24 18:50:18.232140: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll
2021-06-24 18:50:18.233282: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cublas64_100.dll'; dlerror: cublas64_100.dll not found
2021-06-24 18:50:18.233966: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cufft64_100.dll'; dlerror: cufft64_100.dll not found
2021-06-24 18:50:18.234557: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'curand64_100.dll'; dlerror: curand64_100.dll not found
2021-06-24 18:50:18.235152: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cusolver64_100.dll'; dlerror: cusolver64_100.dll not found
2021-06-24 18:50:18.235756: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cusparse64_100.dll'; dlerror: cusparse64_100.dll not found
2021-06-24 18:50:18.236336: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudnn64_7.dll'; dlerror: cudnn64_7.dll not found
2021-06-24 18:50:18.236399: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1641] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2021-06-24 18:50:18.340081: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-06-24 18:50:18.340209: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0
2021-06-24 18:50:18.340927: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N
[name: "/device:CPU:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 1321658220477975012
]

GPU를 CUDA+CUDnn을 이용해서 잘 연결했지만, 위의 코드를 쳐봤을때 다음과 같이 CPU만 인식하는 경우가 있다.

(필자의 경우, 예전 버전 Tensorflow-gpu==1.15을 설치했을때 다음과 같은 현상이 일어났다.)

 

로그를 자세히 살펴보면

2021-06-24 18:50:18.232140: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll
2021-06-24 18:50:18.233282: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cublas64_100.dll'; dlerror: cublas64_100.dll not found
2021-06-24 18:50:18.233966: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cufft64_100.dll'; dlerror: cufft64_100.dll not found
2021-06-24 18:50:18.234557: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'curand64_100.dll'; dlerror: curand64_100.dll not found
2021-06-24 18:50:18.235152: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cusolver64_100.dll'; dlerror: cusolver64_100.dll not found
2021-06-24 18:50:18.235756: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cusparse64_100.dll'; dlerror: cusparse64_100.dll not found
2021-06-24 18:50:18.236336: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudnn64_7.dll'; dlerror: cudnn64_7.dll not found
2021-06-24 18:50:18.236399: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1641] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...

다음과 같이 나오는 것을 볼 수 있는데 cuda-toolkit을 잘 인식하지 못해서 GPU 등록을 취소해버렸다는 정보를 얻을 수 있다.

 

해결방법

cuda가 설치된 경로를 가서 위의 로그 파일에서 없다고 된 파일을 새로 만들어주면 된다.

$ sudo find / -name "cufft64*"
# cuda toolkit의 설치 경로를 모를 시 찾아준다.

$ sudo cp cufft64_xx.dll cufft64_100.dll
# 위에 로그 파일에서 없다고 뜨는 파일을 이름을 동일하게 만들어준다.