OOM when allocating tensor with shape[96,3,299,299] and type float on /job:localhost/replica:0/task:0/device:GPU:0
2018-10-14 17:07
1946 查看
单个GPU启动任务时报OOM的错误:
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[96,3,299,299] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D-0-TransposeNHWCToNCHW-LayoutOptimizer = Transpose[T=DT_FLOAT, Tperm=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](fifo_queue_Dequeue/_1557, PermConstNHWCToNCHW-LayoutOptimizer)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. [[Node: train_op/_1567 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_6943_train_op", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
报错GPU内存不足,就使用2个GPU,使用2个GPU的时候,发现有一块GPU是使用率空闲的,但是内存是满的。添加如下代码:
from keras import backend as K config = tf.ConfigProto() config.gpu_options.allow_growth=True sess = tf.Session(config=config) K.set_session(sess)
参考:https://github.com/keras-team/keras/issues/6031
相关文章推荐
- Tensorflow Deep MNIST: Resource exhausted: OOM when allocating tensor with shape[10000,32,28,28]
- ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[4096]
- ResourceExhaustedError: OOM when allocating tensor with shape[10,256,400,528]
- OOM when allocating tensor with shape
- 显存不够----ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[4096]
- ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[4096]
- ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[100,35,35,64
- Operation was explicitly assigned to /job:ps/task:0/device:CPU:0 but available devices are [ /job:localhost/replica:0/task:0/cpu:0 ]
- tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shap
- device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/dev
- hands on machine learning with sklearn and tensorflow 附录B 翻译与整理(1)概要
- 《Hands-on Machine Learning with Scikit-Learn and TensorFlow》 读书笔记
- TypeError: List of Tensors when single Tensor expected - when using const with tf.random_normal
- 集成算法(chapter 7 - Hands on machine learning with scikit learn and tensorflow)
- hands on machine learning with sklearn and tensorflow 附录B-扩充整理 2关于获取数据
- 《Hands-on Machine Learning with Scikit-Learn and TensorFlow》 读书笔记
- chapter2 of OReilly.Hands-On.Machine.Learning.with.Scikit-Learn.and.TensorFlow
- Enable SPI 1.0 and 1.1 with device tre overlays on BeagleBone
- AWS EC2 服务建立云端Deep Learning开发环境 -- GPU with Tensorflow and Caffe
- 170620 How to install tensorflow with GPU on linux