Creating HDF5 files for caffe

In this post, I will describe how to create HDF5 files for caffe using a set of images. For a full list of the formats supported by caffe, visit this url. Take into account that this is a very general way of creating the HDF5 files, you should be able to easily adapt the code to fulfil your needs (e.g. adding another dataset to the file, changing the names of the dataset).

The 3 major ways to input files to caffe are:

1. Using an HDF5 file. 
2. Using an LMDB file. 
3. Creating a list of the paths to the images in a txt file. 

I strongly recommend creating either HDF5 files or LMDB to train your caffe network for a fast training in comparison with using a list. The Python notebook with the code to create the HDF5 files is here.

For this section, the code contained in the Python notebook is very self-explicative and most of the methods have less than 10 lines of code, so reading them should be straight forward. I'll just explain the high level idea of what the code is doing:

1. Using a directory with images.
2. Read all the images.
3. Scale the images from 0 to 1.
4. Create subimages of all the images.
5. Create a numpy array with all the subimages.
6. Give the array the shape required by caffe:
   • Number of items.
   • Number of channels.
   • Height.
   • Width.
7. Create an H5 with this array.

This logic is coded in the method getimagesforcaffe. Step 4 is required in most cases, we usually don't work with complete images when training a network, we create small sub-images and input them to the network.

Most of the times we also want to create sub-images using a stride to get more samples. The method I provided to create the sub-images is getsubimages. You can configure how to create the sub-images using the parameters, by default, it creates 30 x 30 pixel images, with a stride of 10 and without cropping the image.

The python notebook also has a method to show the images contained in the h5 file, just for testing purposes, the method is called showh5images.