caffe---create自己的数据出现的各种bug

目前bug主要是create_imagenet.sh（来源于examples/imagenet）生成lmdb数据时产生的

bug 1  mkdir  *_val_lmdb failed 

这个一般是因为指定路径下已经存在了该文件，导致出现冲突问题，我最开始对于这问题是每次都手动敲码删除该文件，最后发现自己很笨，可以直接加个语句到create_imagenet.sh中：rm -rf $EXAMPLE/mytask_train_lmdb
rm -rf $EXAMPLE/mytask_val_lmdb


 

bug 2 找不到指定路径下的图片could not open or find file

第一个情况是我在windows cmd下生成的txt标签文件，这里路径是反斜杠，我没有注意到。解决的最好办法就是打开txt文件，将反斜杠替换为斜杠。要么就是在linux下运行make_list.py就不会出现这个问题了。



第二种情况，这个着实困扰了我好久，怎么也搞不懂，路径明明对着了，为啥就不对呢？百思不得其解。。。最后才发现是python里面的转义字符 \t 搞的鬼  在图片名和标签之间的空格用\t表示的，解决这个问题的办法是用 ‘ ’代替了，好了：

#fout.write('%s\t%d\n'%(image_list[i][0], image_list[i][1]))
fout.write('%s%s%d\n'%(image_list[i][0], ' ',image_list[i][1]))#space not \t



正确情况，开始生成lmdb 数据比较大啊 378430图像  比较耗时



代码一

make_list.py

import fnmatch,os
import random
import numpy as np
import argparse

def list_image(root, recursive, exts):
image_list = []
if recursive:
cat = {}
for path, subdirs, files in os.walk(root,True):
print path
for fname in files:
fpath = os.path.join(path,fname)
suffix = os.path.splitext(fname)[1].lower()
if os.path.isfile(fpath) and (suffix in exts):
if path not in cat:
cat[path] = len(cat)
image_list.append((os.path.relpath(fpath, root), cat[path]))
# print fpath,cat[path]
else:
for fname in os.listdir(root):
fpath = os.path.join(root, fname)
suffix = os.path.splitext(fname)[1].lower()
if os.path.isfile(fpath) and (suffix in exts):
image_list.append((os.path.relpath(fpath, root), 0))
return image_list

def write_list(path_out, image_list):
with open(path_out, 'w') as fout:
for i in xrange(len(image_list)):
#fout.write('%d \t %d \t %s\n'%(i, image_list[i][1], image_list[i][0]))
#fout.write('%s\t%d\n'%(image_list[i][0], image_list[i][1]))
fout.write('%s%s%d\n'%(image_list[i][0], ' ',image_list[i][1]))#space not \t
def make_list(prefix_out, root, recursive, exts, num_chunks, train_ratio):
image_list = list_image(root, recursive, exts)
random.shuffle(image_list)
N = len(image_list)
chunk_size = (N+num_chunks-1)/num_chunks
for i in xrange(num_chunks):
chunk = image_list[i*chunk_size:(i+1)*chunk_size]
if num_chunks > 1:
str_chunk = '_%d'%i
else:
str_chunk = ''
if train_ratio < 1:
sep = int(chunk_size*train_ratio)
write_list(prefix_out+str_chunk+'_train.txt', chunk[:sep])
write_list(prefix_out+str_chunk+'_val.txt', chunk[sep:])
else:
write_list(prefix_out+str_chunk+'.txt', chunk)

def main():
parser = argparse.ArgumentParser(
formatter_class=argparse.ArgumentDefaultsHelpFormatter,
description='Make image list files that are\
required by im2rec')
parser.add_argument('root', help='path to folder that contain images.')
parser.add_argument('prefix', help='prefix of output list files.')
parser.add_argument('--exts', type=list, default=['.bmp','.bmp'],
help='list of acceptable image extensions.')
parser.add_argument('--chunks', type=int, default=1, help='number of chunks.')
parser.add_argument('--train_ratio', type=float, default=1.0,
help='Percent of images to use for training.')
parser.add_argument('--recursive', type=bool, default=True,
help='If true recursively walk through subdirs and assign an unique label\
to images in each folder. Otherwise only include images in the root folder\
and give them label 0.')
args = parser.parse_args()

make_list(args.prefix, args.root, args.recursive,
args.exts, args.chunks, args.train_ratio)

if __name__ == '__main__':
main()
代码二
create_imagenet.sh 
#!/usr/bin/env sh
# Create the imagenet lmdb inputs
# N.B. set the path to the imagenet train + val data dirs
EXAMPLE=examples/mytask
DATA=/mnt/hgfs/caffe
TOOLS=build/tools
TRAIN_DATA_ROOT=/mnt/hgfs/caffe/train/
VAL_DATA_ROOT=/mnt/hgfs/caffe/val/
# Set RESIZE=true to resize the images to 256x256. Leave as false if images have
# already been resized using another tool.
RESIZE=true
if $RESIZE; then
RESIZE_HEIGHT=256
RESIZE_WIDTH=256
else
RESIZE_HEIGHT=0
RESIZE_WIDTH=0
fi
if [ ! -d "$TRAIN_DATA_ROOT" ]; then
echo "Error: TRAIN_DATA_ROOT is not a path to a directory: $TRAIN_DATA_ROOT"
echo "Set the TRAIN_DATA_ROOT variable in create_imagenet.sh to the path" \
"where the ImageNet training data is stored."
exit 1
fi
if [ ! -d "$VAL_DATA_ROOT" ]; then
echo "Error: VAL_DATA_ROOT is not a path to a directory: $VAL_DATA_ROOT"
echo "Set the VAL_DATA_ROOT variable in create_imagenet.sh to the path" \
"where the ImageNet validation data is stored."
exit 1
fi
echo "Creating train lmdb..."
rm -rf $EXAMPLE/mytask_train_lmdb
rm -rf $EXAMPLE/mytask_val_lmdb
GLOG_logtostderr=1 $TOOLS/convert_imageset \
--resize_height=$RESIZE_HEIGHT \
--resize_width=$RESIZE_WIDTH \
--shuffle \
$TRAIN_DATA_ROOT \
$DATA/train.txt \
$EXAMPLE/mytask_train_lmdb
echo "Train lmdb done!"
echo "Creating val lmdb..."
GLOG_logtostderr=1 $TOOLS/convert_imageset \
--resize_height=$RESIZE_HEIGHT \
--resize_width=$RESIZE_WIDTH \
--shuffle \
$VAL_DATA_ROOT \
$DATA/val.txt \
$EXAMPLE/mytask_val_lmdb
echo "val lmdb done!"
echo "Done."