Caffe: Data layer prefetch queue empty
2015-10-20 16:30
453 查看
资源1
The problem was that the text files for training and test data were not shuffled, but sorted. Thus, test was done only on the images of same class, and parameters were learned from the images of that same class as well, leading to 100% accuracy very quickly.
资源2
So can you get the right result ( accuracy and loss) while finetune , in my result it seems only slow down the test process.
资源3
I think it is the problem of power supply, not the problem of Caffe.
总结:
原因:CPU数据读入慢于GPU计算,导致GPU处于空闲,使得计算速度下降,迭代时间间隔扩大。nvidia-smi显示为:GPU功率不断大幅度变化,GPU电压不稳定。
可能诱因:服务器同时在进行其他工作,如大规模读写数据(频繁使用IO),数据无法读入;CPU同时进行其他工作,负载极重,无法正常读取数据;CPU内存爆表,所有程序速度变慢。
解决办法:
1 使用lmdb替换txt文件,加快数据读取速度;
2 避免IO被频繁使用;
3 可以考虑不加入test.txt
The problem was that the text files for training and test data were not shuffled, but sorted. Thus, test was done only on the images of same class, and parameters were learned from the images of that same class as well, leading to 100% accuracy very quickly.
资源2
So can you get the right result ( accuracy and loss) while finetune , in my result it seems only slow down the test process.
资源3
I think it is the problem of power supply, not the problem of Caffe.
总结:
原因:CPU数据读入慢于GPU计算,导致GPU处于空闲,使得计算速度下降,迭代时间间隔扩大。nvidia-smi显示为:GPU功率不断大幅度变化,GPU电压不稳定。
可能诱因:服务器同时在进行其他工作,如大规模读写数据(频繁使用IO),数据无法读入;CPU同时进行其他工作,负载极重,无法正常读取数据;CPU内存爆表,所有程序速度变慢。
解决办法:
1 使用lmdb替换txt文件,加快数据读取速度;
2 避免IO被频繁使用;
3 可以考虑不加入test.txt
相关文章推荐
- 最长递增子序列(longest increasing subsequence)
- JAVA Eight Queens
- Android UI-去除SpannableString下划线
- WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster uito ensure
- mysql 如何设置自动增长序列 sequence(一)
- 关于Xilium.CefGlue(CEF3) 加载mp4格式失败
- NDK错误 org.gradle.process.internal.ExecException: Process 'command with non-zero exit value 2
- leetcode Permutation Sequence
- 详细整理:UITableView优化技巧
- wcf 远程服务器返回了意外响应: (413) Request Entity Too Large。
- NSURLRequest POST方式请求
- 获取<meta-data>元素的value属性的值
- StringBuilder和StringBuffer区别
- easyui datagrid 诡异的无法显示问题
- 二、通过纯代码创建的UICollectionView的简单使用
- EasyUi Grid以POST方式传送参数绑定
- conn.setAutoCommit(true)和(false)的区别
- Ubuntu中crosstool-ng编译出错:[ERROR] gtype-desc.c:8829:18: error: subscripted value is neither array nor
- Android ui utils-简单实用的Android界面工具
- UE3代码阅读需知