解决RuntimeError: cuda runtime error (30) : unknown error at /pytorch/aten/src/THC/THCGeneral.cpp:70&N...
2018-11-14 21:35
1931 查看
解决两个问题:
(1)RuntimeError: cuda runtime error (30) : unknown error at /pytorch/aten/src/THC/THCGeneral.cpp:70 (2)NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
之前还好好的代码 时隔一个月就出现了问题
haiou@gpu4:/home/gpu4/haiou/newdata/tqa$ cat f2 WARNING:root:WARN: Duplicate embedding found for ; /home/gpu4/haiou/newdata/tqa/src_0826/layers.py:227: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument. alpha_flat = F.softmax(scores.view(-1, y.size(1))) THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=70 error=30 : unknown error Traceback (most recent call last): File "src_0826/main.py", line 129, in <module> b_info,b_dev_acc, b_test_acc = run_epoch(args,model,train_data,dev_data,test_data,checkpoint_path) File "src_0826/main.py", line 68, in run_epoch model.train(cur_train_data) File "/home/gpu4/haiou/newdata/tqa/src_0826/model.py", line 64, in train pred_proba = self.network(*feed_input) File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__ result = self.forward(*input, **kwargs) File "/home/gpu4/haiou/newdata/tqa/src_0826/tmn_v21_update.py", line 101, in forward self.mfunction(p_hiddens, q_hiddens, c_hiddens,p_mask, q_mask, c_mask) File "/home/gpu4/haiou/newdata/tqa/src_0826/tmn_v21_update.py", line 222, in NA_TriMatching x_mask = p_mask) File "/home/gpu4/haiou/newdata/tqa/src_0826/tmn_v21_update.py", line 185, in tri_matching init_mem = torch.zeros(dim[0],dim[1],dim[2]).float().cuda() #zero mem RuntimeError: cuda runtime error (30) : unknown error at /pytorch/aten/src/THC/THCGeneral.cpp:70
询问了实验室的小伙,then发现问题:
gpu4@gpu4:~$ nvidia-smi NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running. gpu4@gpu4:~$ nvidia- nvidia-bug-report.sh nvidia-container-runtime-hook nvidia-debugdump nvidia-installer nvidia-settings nvidia-xconfig nvidia-container-cli nvidia-cuda-mps-control nvidia-detector nvidia-modprobe nvidia-smi nvidia-container-runtime nvidia-cuda-mps-server nvidia-docker nvidia-persistenced nvidia-uninstall
然后解决问题:
好吧,兜兜转转后续儿重装,不是安装cuda就相对简单了:
安装NVIDIA驱动程序(如果事先有装但失败的,先卸载,所以要卸载。。)
$ sudo apt remove --purge nvidia*
1)禁用nouveau驱动
终端中运行:
$ lsmod | grep nouveau
2)关闭图形化界面
$ sudo service lightdm stop
3)安装驱动
NVIDIA驱动安装也有多种方式,runfile文件形式的安装和apt安装
apt安装:
a.添加Graphic Drivers PPA
$ sudo add-apt-repository ppa:graphics-drivers/ppa $ sudo apt-get update
寻找合适的驱动版本
$ ubuntu-drivers devices
c.我的显示是415
$ sudo apt-get install nvidia-415
d.安装完成后,重启
$ sudo reboot
e.这时,执行下面的命令,查看驱动的安装状态
$ sudo nvidia-smi
查看安装版本:
nvidia-settings --v
阅读更多
参考链接:https://blog.csdn.net/Jonms/article/details/79318566?utm_source=blogxgwz2
相关文章推荐
- Pytorch 读取参数错误 RuntimeError: cuda runtime error (10) : invalid device ordinal
- SHUTDOWN_MSG: Shutting down NameNode at java.net.UnknownHostException: datanode:unknown error解决办法
- 安装opencv出现:cudalegacy/src/graphcuts.cpp:120:54: error: ‘NppiGraphcutState’ has not been declared
- make[2]: *** [modules/cudalegacy/CMakeFiles/opencv_cudalegacy.dir/src/graphcuts.cpp.o] Error 1 make[
- pytorch runtime error(59):device-side assert triggered at XXX
- 类似"ERROR at line 633: Unknown command '\''"错误的一种解决办法
- RuntimeError: invalid argument 1: must be strictly positive at /pytorch/torch/lib/TH/generic/THTenso
- ERROR at line 206: Unknown command '\''.解决办法
- 安装opencv出现:cudalegacy/src/graphcuts.cpp:120:54: error: ‘NppiGraphcutState’ has not been declared
- make: *** [.build_release/cuda/src/caffe/layers/detection_output_layer.o] Error 1解决
- Cocos2d-x 3.0 TestCpp android编译出错 解决 error: expected ';' at end of member declaration
- cudalegacy/src/graphcuts.cpp:120:54: error: ‘NppiGraphcutState’ has not been declared
- RuntimeError: invalid argument 4: out of range at pytorch/torch/lib/TH/generic/THTensor.c:439
- mysql导入时出现"ERROR at line : Unknown command '\''."的解决办法
- CMake Error at cuda_compile_generated_warp.cu.o.cmake:264 (message)
- 错误解决:Error(10028):Can't resolve multiple constant drivers for net “ ” at **.v
- 关于QT中 **.cpp:1:1: error: stray '\377' in program 错误解决
- Pytorch源码安装(附加可能出现的问题解决)
- 关于apktool反编译出错Error occured at code address 0的解决办法
- 关于系统蓝屏stop:c0000135 unknown hard Error unknow的解决