Wednesday 3 August 2016

Creating HDF5 files for caffe

In this post, I will describe how to create HDF5 files for caffe using a set of images. For a full list of the formats supported by caffe, visit this url. Take into account that this is a very general way of creating the HDF5 files, you should be able to easily adapt the code to fulfil your needs (e.g. adding another dataset to the file, changing the names of the dataset).

The 3 major ways to input files to caffe are:

  1. Using an HDF5 file. 
  2. Using an LMDB file. 
  3. Creating a list of the paths to the images in a txt file. 
I strongly recommend creating either HDF5 files or LMDB to train your caffe network for a fast training in comparison with using a list. The Python notebook with the code to create the HDF5 files is here.

For this section, the code contained in the Python notebook is very self-explicative and most of the methods have less than 10 lines of code, so reading them should be straight forward. I'll just explain the high level idea of what the code is doing:
  1. Using a directory with images.
  2. Read all the images.
  3. Scale the images from 0 to 1.
  4. Create subimages of all the images.
  5. Create a numpy array with all the subimages.
  6. Give the array the shape required by caffe:
    • Number of items.
    • Number of channels.
    • Height.
    • Width.
  7. Create an H5 with this array.
This logic is coded in the method getimagesforcaffe. Step 4 is required in most cases, we usually don't work with complete images when training a network, we create small sub-images and input them to the network.

Most of the times we also want to create sub-images using a stride to get more samples. The method I provided to create the sub-images is getsubimages. You can configure how to create the sub-images using the parameters, by default, it creates 30 x 30 pixel images, with a stride of 10 and without cropping the image.

The python notebook also has a method to show the images contained in the h5 file, just for testing purposes, the method is called showh5images.

Creating a diagram for a caffe neural network architecture

In this tutorial, I will explain how to save an image with the diagram of a caffe neural network architecture The Python notebook with the code is in this url.

If we want to create a visualisation for the caffe neural network, we can use the Python library. The first requirement is to install GraphViz an pydot. You can find instructions here for GraphViz.

Now, these are the Python imports required:

import sys

from google.protobuf import text_format

caffe_library = 'caffelibrary/'
sys.path.insert(0, caffe_library)
import caffe;
import caffe.draw

This is the method we can use to create the diagram:

def create_network_architecture_diagram(networkfilepath, fileextextension='png'):
    with open(networkfilepath, "r") as f:
        net_architecture_text = f.read()

    net_architecture = caffe.proto.caffe_pb2.NetParameter()
    text_format.Merge(net_architecture_text, net_architecture)
    caffe.draw.draw_net_to_file(net_architecture, networkfilepath + '.' + fileextextension, "LR")


We can optionally select the format of the output, any file extension supported by GraphViz is valid, e.g.: png, jpg, svg, eps. An example of how to use it is shown here:

create_network_architecture_diagram("Path to the prototxy network file", "eps")

The following diagram corresponds to the architecture of the network "net surgery" provided in the caffe examples:


Monday 1 August 2016

Testing a Caffe Model using Python

I've been completely absorbed by my master degree at UCL and my part time job at Satalia, but it is time to start a series of posts about the work I'm developing for my dissertation. 




I'll start with this posts related to the caffe framework

This first tutorial is about how to test a pre-trained model of Caffe using caffe's Python library. Using Python for testing Caffe is very useful, however in contrast with the installation docs, there is not much documentation on how to use Caffe’s Python library apart from the Python notebooks found in the Github repository. 

The python notebook using this code can be found here.



Let's start with the Python library imports that we will need:

from pylab import *
caffe_library = '../caffelibrary/'

import sys
sys.path.insert(0, caffe_library)
import caffe
import os

import numpy as np
import matplotlib.pyplot as plt
from PIL import Image
from sklearn import preprocessing
import scipy.misc

In general, caffe networks are trained using H5 or LMDB files containing multiple images. However, for testing, it is much simpler to input one image every time and check the results. In order to do this, it's better to work with a copy of the prototxt network file and remove all the data layers used for training and modify the data input layers used for testing in the following way:


layer {
  name: "data"
  type: "Input"
  top: "inputdata"
  
  input_param {
    shape {
      dim: 1
      dim: # of channels
      dim: height
      dim: width
    }
  }
  
    include { phase: TEST }
}


The height and width of the image can be changed later to match the input size that you want to test. In order to load a pre-trained model, we need the caffe model and the modified prototxt file with the architecture of the network. The following code shows how to load the network:


MODEL_ARQUITECTURE = 'PATH TO prototxt FILE'
MODEL = 'PATH TO PRE-TRAINED MODEL'
net = caffe.Net(MODEL_ARQUITECTURE, MODEL, caffe.TEST)


Because we want to test the current state of the network, we have to use as thelast parameter for the method: caffe.TEST. That tells the process that we want to run the test phase of the network. Once we have the model in memory, we want to test it using images. In order to do this, we need to scale the input images to the format we used for the training. Caffe training is done using numpy arrays with 8 bit unsigned int or images scaled into floats that go from 0 to 1. To do the later, we can use the following method:
scaler = preprocessing.MinMaxScaler()

def scaleimage(image):    
    ascolumns = image.reshape(-1, 3)
    t = scaler.fit_transform(ascolumns)
    transformed = t.reshape(image.shape)     
        
    return transformed


After scaling, we can use the following method to create the inputs for testing the network:


def createcaffeinput(imagepath):
    # Open the image
    image = np.array(Image.open(imagepath))
    # Scale (0 - 1)
    image = scaleimage(image)
    
    image = image.transpose(2, 0, 1)

    image = image[np.newaxis, :, :]
    
    return image


Note that the image is scaled first, and then the shape is transformed. When loading images in python, the shape is loaded in this way: [height, width, channels], but caffe needs the shape in this fashion: [channels, height, width]. After creating the inputs for caffe, we need to add them to the data blobs of the network. The following method has 3 parameters, net which is the network, layername corresponding to the layer where the image will be added, and finally imagepath containing the path to the image that will be loaded into the caffe blob.


def inputimage(net, layername, imagepath):
    caffeinput = createcaffeinput(imagepath)
    net.blobs[layername].reshape(*caffeinput.shape)
    net.blobs[layername].data[...] = caffeinput


So, if our network has two data layer inputs: clean and dirty, the following code will fill the data blobs and run a forward pass:


inputimage(net, 'clean', IMAGE_FILE)
inputimage(net, 'dirty', IMAGE_FILE_DIRTY)
out = net.forward()


The variable out will contain the output that generates the network (the value of the loss function). After running a forward pass, we are interested in looking at the results. If we suppose that the last layer containing such results is called result, we can access it in the following way:


netcolor.blobs['result'].data


This python object will be a numpy array with the same shape that your result layer has in caffe. If the result layer output are images, the array will be a 4 dimension array. The first dimension contains the number of elements processed and the other dimensions correspond to the actual result image, so the first element of the result blob is theresult image. This is the code to access such element:


result = netcolor.blobs['result'].data[0]


Note how we are only interested in the first element of the array. We can also save this blob to our hard drive as an image with the following code:


resultimage = result.transpose(1, 2, 0)
scipy.misc.imsave('result.jpg', resultimage)


We need to do a transpose before saving to have the right shape required by the method.


This tutorial has come to an end, please let me know about any comments!

Friday 15 May 2015

An straetegy to easily add new tasks to a big project

The authors of Managing the unmanageable came to visit our office and they gave us insights to improve our processes as well as some methods to organize projects. Something that is very interesting about the book, is that it contains valuable and innovative information about managing software projects, and in my opinion the strategies and recommendations can be applied to lots of projects, even if they are not related to software. The one that I'm about to talk about is -I believe- designed for big projects. So let's start talking about the context, so you can find why doing this will be useful and probably the ideas that brought this method into something real.

When you're getting started with a big project, you'll create a first list of tasks that will be developed, from this list, there will come more and more tasks as the project gets developed, so, figuring out how much effort is required for each task can take a lot of time if you don't have anything that leverage the way that this happens, so I'll present you with the steps required for this strategy:
  • Divide the project in tasks
  • Assign each task a number that represents how difficult the task is
  • Sort the tasks
Once that the tasks are sorted, as you can probably imagine, it's very easy to keep the list sorted when more tasks area added to the project, and to estimate the effort required for them,  you just need to compare them with a similar task. This are some of the advantages that you can find when you use this method:

  • Adding new tasks is easy, you can compare similar tasks and make a good bet on how much effort it's required for the task and where in the level of effort required does this task falls into place
  • You can give a more accurate delivery date for the project, because it's more easy to develop
  • Adding big pieces to the project should take less time to organize and make the estimates






Tuesday 16 December 2014

Cómo hacer mejores estimados de tiempo y recomendaciones para reducir las fechas de entrega de proyectos

Realizar buenas estimaciones de tiempo para los proyectos es muy importante. Mantiene los objetivos, fechas y presupuesto en un buen estado. A veces decidimos tomar el camino fácil y comenzamos con estimados de tiempo poco fundamentados para los proyectos. Esto es una mala práctica; es necesario revisar la documentación, las experiencias y las personas involucradas con el fin de realizar una mejor estimación y tener una base sólida que la sustente. La mejor forma de obtener datos para apoyar nuestros estimados, es a través de proyectos anteriores y situaciones similares, Esto es fácil de lograr si mantenemos una bitácora del presupuesto que en realidad se requirió y de las fechas de entrega para todos los proyectos.

Comencemos con los pasos necesarios para tener buenas estimaciones de tiempo:

Dividir el proyecto en tareas. Éste es el primer paso que debes realizar. Cada tarea debe de tomar entre dos y cinco horas para ser completada, algunas tomarán más o menos tiempo, por ejemplo, ciertas tareas son muy importantes y necesitan estar aisladas, otra situación, puede ser que la tarea requiera completarse toda de una sola vez y no es posible dividir. Cuándo nos encontremos dividiendo tareas, debemos tratar de mantenerlas lo más aisladas posible unas de otras, ya que es más sencillo realizar estimados para pequeñas tareas en comparación con el esfuerzo requerido para estimar tareas grandes o con muchas relaciones entre ellas; esto te puede mantener enfocado y  encontrar requerimientos adicionales / documentación / personas que van a ser necesarios para completar cada tarea. Esto es de gran ayuda también cuándo buscas tareas anteriores con las cuáles quieres comparar las actuales y ayuda a entender si otras tareas deben ser realizadas por otros equipos. Por ejemplo, si deseamos obtener cuentas de prueba para un sistema, es posible que el departamento encargado de manejar las cuentas, tenga datos más relevantes para nuestras pruebas, porque ellos conocen mejor cómo funciona el sistema y que cuentas les interesa probar.

Revisar proyectos anteriores para encontrar situaciones similares. Ésta es la mejor forma para conocer cuánto tiempo tomará realizar un proyecto similar. Aquí debemos tomar en cuenta que si el nuevo proyecto es muy parecido al anterior, probablemente tomaremos menos tiempo en realizarlo, puesto que ya hemos superado la curva de aprendizaje necesaria para las tareas que requirieron investigación y la lectura de diversa documentación.

Juego de poker. Cuándo se esté calculando la cantidad requerida para una tarea, es una buena idea tener la opinión de uno o dos miembros experimentados del equipo. Ellos siempre ven algo diferente y pueden ayudar a realizar una mejor estimación visualizando algún dato que no tomamos en cuenta. Es importante mencionar que si las estimaciones difieren mucho de persona a persona, es posible que no tengamos suficiente información para determinar cuánto puede tomar realizar una tarea.

Pasos previos y pasos posteriores. Muchas tareas tienen pasos que se deben realizar antes de comenzar la tarea por ejemplo: crear un nuevo repositorio, una nueva cuenta de prueba, esperar la configuración de un ambiente de pruebas. Estos se deben tomar en cuenta como parte del estimado, asimismo, cuándo la tarea se finaliza, casi siempre hay una serie de pasos posteriores, por ejemplo: enviar un correo a los solicitantes de la tarea, crear notas para la liberación a producción, dar de baja cuentas de prueba, etc.

Factores externos. ¿El equipo de soporte está muy ocupado para configurar un ambiente de prueba?
¿El área de finanzas no puede proveer la información en las fechas de ejecución del proyecto? ¿Podemos liberar el sistema en la fecha que seleccionamos, o esto no está en la ventana de liberación para el área afectada?. Todas estas preguntas son importantes cuándo establecemos la fecha de liberación de un proyecto; de nuevo, cuándo hemos estado en situaciones similares, podemos usar la experiencia para determinar que factores externos debemos tomar en cuenta.

En resumen, hay muchas cosas que se deben tomar en consideración, pero algo que debemos tener en mente, es que debemos mantenernos lo más "optimistas" posible, no podemos estar pensando que mañana va a ocurrir un "Armageddon" para fundamentar que el proyecto no se entregará a tiempo. Se pueden dar situaciones que retrasen la entrega, pero deben de ser resueltas en el momento en el que se presentan y el proyecto debe de ser reorganizado adecuadamente.

Finalmente escribiré sobre algunas formas de entregar el proyecto en menos tiempo y con menos esfuerzo:


  • Involucra a todos los departamentos interesados en el proyecto. Los proyectos de desarrollo de software también deben incluir a miembros de los departamentos interesados o afectados por el proyecto. Se les debe asignar tareas como: búsqueda de cuentas de prueba, análisis de resultados, etc. Esto tendrá como resultado, la revisión a tiempo de los requerimientos de los departamentos y la planeación de los siguientes pasos; por otro lado, los departamentos podrán conocer cómo va el proyecto, qué expectaciones tener, dar retroalimentación a tiempo y entregar información relevante para las pruebas.
  • Sistemas prefabricados. Muchos proyectos contienen un objetivo similar al de algún software existente en el mercado. Si puedes comprar un sistema que tenga una buena relación costo/beneficio/calidad, debes considerar comprarlo y apropiarlo a las necesidades del proyecto. Debes comprar un sistema confiable que haya sido usado por varios usados, esto garantiza que tengas una cantidad importante de pruebas que validen su funcionamiento y reducirá drásticamente el tiempo del proyecto que estás ejecutando.
  • Contratar compañías expertas. Si el software que vas a desarrollar es muy genérico, es posible que exista una empresa dedicada a realizar el sistema que necesitas en una cantidad menor de tiempo y que probablemente te sorprenda con un buen precio. Esto te permitirá enfocarte en otros proyectos que sí requieren la atención explícita de tu equipo y realizarás más cosas en menos tiempo.
Esto es una guía básica que espero encuentren útil al momento de realizar estimaciones para proyectos. Ésta es la versión original. Estaré traduciendo varias entradas, con la finalidad de llegar a más usuarios. 

¡Espero hayan disfrutado la entrada!

Monday 15 December 2014

Making good estimates and ways to reduce delivery dates



Making good estimates is very important because it keeps the project goals, dates and budget in a healthy state. Sometimes we take the easy path and just throw numbers for the budget and the time that it takes to deliver the project, but that's a bad practice, we must review documentation, experiences and the people involved in order to give a good estimate and support our numbers. The best way to obtain support data is through old projects and similar situations, this information should be easy to obtain if we keep track of what was the real budget and the real dates of delivery of all the projects.

First I will talk about the required steps to have good time estimates:

  • Divide the project in tasks. This is the first thing you need to do. Each task should take around two through five hours to complete, some tasks might take more time or less time, for example, certain tasks are very important and they need to be isolated, or it could also happen that they need to be completed all at once. Something else to take into account when dividing tasks, is to keep them as self independent as possible, meaning that there's no dependency on another tasks. It's always easy to estimate the time required for small items than the one required for big items, this can keep you focused and find additional requirements/documentation/people that you will need to complete each task. This also helps when you're looking for similar situations in the past, it's easier to find a matching similar situation when tasks are divided properly. Another good thing about this is that it helps in understanding if some tasks should be made by another team, for example, in some cases, testing accounts can be provided by another department because they know better what are the things that they want to test.
  • Review old projects and find similar situations. This is the best way to know how much will it take to create a similar project. Take into account that if the new project is really similar, we may end up finishing it sooner than expected. This is because in most projects, there's a learning curve for items that needs discovery and reading documentation.
  • Poker game. When deciding the amount of time required for a task, it's a good idea to have the opinion of one or two experimented members of the team, they always see something that is missing for the other individuals, and that could be key to set the estimate for the task. Keep in mind that if the estimates provided are very different, is a smell of insufficient data to set the required time for completing a task.
  • Pre-steps, Post-steps. It's important to know if there are another things needed to start and to finish the task and add it to the estimate. For example, you might need to create a branch for working on the task, set up a new account for a system, wait for a testing environment, etc. Those are the pre-steps, but when you finish the action items of the task, there might also be another post-steps, like creating release notes, sending out an email to the business owners, deleting testing accounts, etc.
  • External factors. Network team is too busy to set up a testing environment? Working with the financial area in a time that they cannot provide information fast? Can you release the software project in the date you selected?. This and many other questions are important when creating an estimate and deciding on dates of delivery, if you have been in a similar situation before, it could be easy to know when some external factor will be affecting the date of delivery.

There are many things to take into consideration! but something to keep in mind is that you should stay as "optimistic" as possible, meaning that you cannot put as a support to your estimate, the fact that there could be an "Armageddon" tomorrow, and your project might not be completed on time.

There will be situations that can delay the delivery date of the project, but they should be addressed at the moment that they happen, and the project should be reorganized accordingly. Another thing that I would like to talk about is ways to deliver the project in less time, and with less effort:

  • Get all the team interested in the project involved. Software projects are not completely developed by the software team. If there are going to be changes to the some department systems, it's a good idea to have them involved as much as possible, and even assign some tasks to them (like finding testing accounts, providing data analysis, etc). The outcome of this will be early checking the requirements of the departments and based on that, planning the next steps, on the other hand, the departments can know how's the project going, what are the expectations, provide early feedback about features, and give relevant data for testing.
  • Buying systems. Many projects are similar to current systems in the market. If you can buy a system that has a decent trade-off between cost/value/quality, think about buying it and customizing according to your needs. You should buy a very known system in order to trust that it has been deeply tested and that can reduce the time required for delivery.
  • Request help from another company. If you have the budget, it can be useful to have another company help you, specially if they're experts on the system you want to develop, this can be cheaper than expected, because they have passed the learning curve and can save you a lot of time, and allow you to focus on another projects that can only be developed by your team.
That's it for now. I hope you enjoyed this entry. Let me know if you have any questions or if you would like me to talk about another topic. Many thanks readers!


Sunday 9 November 2014

Benefits of SEO (Search engine optimization)

My last post was an introduction guide for a successful website. Now I will present you the benefits of going through the search engine optimization process.

  1. Increased traffic. Generally speaking, the main objective of a website is to reach a bigger audience, it doesn't matter if your website is a store or if it's for helping other people with some topics, the bigger the traffic, the better results you'll have. The reason why traffic is increased is because search engines will place your website in a better position in the seach results, once you finish the SEO process.
  2. Usability is a side result. The website is faster and when you have friendly URLs, your website is easier to navigate for the users, so they will be happy to use your page, content is also improved and they will know were they can reach things. For search engines, it will make it easier to understand your website.
  3. You can track everything. It's much easier to track a website that has gone though the SEO process, because you have a better organized website, the content is interesting, the structure of the URLs can help you with insights on what to improve, you can track which pages are not being visited, get statistics of usage of a specific section, know where your visitors are leaving the page and so on.
  4. Putting ads from other companies. If your website has a lot of visitors, it's very likely that companies related to what you do, would like to have ads in your website. This can give you some money in return and make your website even more popular.
  5. Have an international 24 x 7 website. Once you start to be popular, you can decide to expand your website to reach more countries by creating country specific pages, and of course, your website is alive all the days! it never closes, so, everyone who can reach you, will be able to do so through your website.

There are many other benefits, I just wanted to give you an idea of the importance of SEO. If you like this post, please comment or tell me about a topic that you would like me to write about. See you soon!