Raw QuickTime Video (cropped)		After messing around (Image energy)		Active Contour Model (Visualisation)		Axis Crossover Vector (Visualisation)		Neural network classification

Detecting, Tracking and Classifying Pedestrian Movement using Active Contour Models and Neural Networks

Bit of a mouthful isn't it? My research involves the use of Active Contour Models to detect and track moving objects in a video sequence, and the use of Neural Networks firstly to determine whether or not those moving objects are pedestrian, and secondly to analyse their movement. The axis crossover vector is a technique that has been developed in order to pass active contour models into neural networks. This page explains the work in layman's (or thereabouts) terms; if you're more interested in my work you might like to download PDF files of some of my publications. They are also known to cure insomnia. Finally this page offers links to various bits of free software that may be of use to other people in the field of computer vision and / or neural networks.

Active Contour Models (ACMs)

ACMs are energy minimising splines that can be applied to an image in order to detect contours in the image's energy space. "What does this mean?", you might ask. It means that, when told what types of 'clues' to look for in an image (defined mathematically beforehand in the form of an energy function), an ACM is able to detect the visible outline of an object in an image. By performing this energy minimisation ('searching') in each frame of a video sequence, ACMs are able to track objects as they move around an image.

For the purposes of my research, the Williams and Shah Fast Snake ACM model has been adopted and significantly customised, to allow certain characteristics to be present in the ACM which better suit the task. For instance, the ACM developed in my work is able to predict object movement based upon the object's previous movement in the video sequence, dramatically reducing the search time needed to detect the object's outline in the current video frame. Furthermore the model developed is able to deform itself between video frames, to match the object's shape deformation / articulation, again increasing the model's performance and accuracy.

The model's image energy function (a component of the ACM's energy function) has also been customised, to enhance the edges of moving objects in a video frame.

Further information on the ACMs developed in my research can be found on my ACM page, which will be linked to from here very soon.

Axis Crossover Vectors

In order for an ACM's shape to be usable by a neural network, several pieces of information that are implicit in the contour's representation need to be removed in order to abstract the shape information exclusively.

The axis crossover vector has been developed as part of this work to represent closed loop contours (i.e. the contours from ACMs) as vectors that are independent of the contour's size, location, number of control points, and order of control points.

The axis crossover vector technique is very flexible, in terms of the resolution of shape information which it can contain, and in terms of the types of shape information it can contain. In my work, axis crossover vectors have been optimised for the purposes of describing pedestrian and non-pedestrian shapes; that is, they include a suitable amount of shape information to allow a trained neural network to distinguish one class of object (pedestrians) from other classes of objects (non-pedestrians). Investigations and testing procedures used to validate these optimisations are documented in my research papers.

The axis crossover technique itself is documented in my research papers, and a Mac OS X application for generating these vectors from closed loop contours is available in my software section.

Neural Networks

Although being a general purpose tool in the Artificial Intelligence domain for vector classification, in my work neural networks have been developed specifically to distinguish visual objects according to their shape.

Feedforward error-backpropagation networks have been developed such that they can accurately recognise an object's class (pedestrian / non-pedestrian) independent of viewpoint, that is, independent of the camera's position in the scene relative to the object.

The neural networks developed are trained using computer generated instances of pedestrian and non-pedestrian objects, but achieve 100% decision boundary accuracy when presented with previously unseen examples of these computer generated objects. More significantly, the networks also achieve 100% accuracy when presented with real-world objects, detected in real-world video footage of complex scenes, viewed from any direction on the ground plane.

*Under Construction!*

Software

As part of my research here at the Uni I have developed several bits of software to make my life easier. Some of these bits of software are so badly documented and / or idiosyncratic that it would be futile to release them to the public, whereas other bits are probably reusable to others in the fields of computer vision and / or neural networks. However these applications are not quite ready for general consumption just yet. Below are listed the bits of software which will at some point in the hopefully not too distant future be released. Please note that all software and APIs are written in Objective-C for Mac OS X machines. If you instead use Solaris or FreeBSD and are interested in this software, please let me know at , as these operating systems were used to develop some of the software initially, using C and C++, before being ported to Mac OS X.

Each of the pieces of software below are / will be freely downloadable with no license fees, whether for academic or commercial use. If you're wondering why it is all free, it is because I believe education should be freely available to all. If you find the software invaluable and insist on making a donation however, I would accept a set of Size 10 D'Addario electric or acoustic guitar strings, sent to the postal address listed on my home page.

kPixelKit

kPixelKit is an Objective-C API for performing image processing within the Cocoa API in Mac OS X. The kPixelKit API makes extensive use of the Altivec (a.k.a. Velocity Engine) vector co-processing unit in the PowerPC G4 and G5 chips used in modern macs, and has been optimised for use on macs with more than one processor in them. The API extends Cocoa's NSImage and NSBitmapImageRep classes, allowing operations such as colour tinting, edge detection, motion detection, and other image processing operations. kPixelKit also makes it trivial for several image processing operations to be strung together.

kNeuralKit

kNeuralKit is an Objective-C API for developing neural networks on Mac OS X. The API supports multiple neural architectures: error-backpropagation, SOM, recurrent feedforward, Kohonen, and Hopfield. Further architectures are implementable due to the API's plugin architecture, therefore novel architectures can be developed. In addition the API supplies various weight matrix initialisation functions and neuron activation functions, and again, can be customised to allow for novel techniques to be prototyped and developed.

Neuron Factory

Neuron Factory is a Mac OS X application for developing neural networks using a graphical interface. As such, no programming experience is required in order to use Neuron Factory to develop neural networks. Behind the scenes, Neuron Factory makes heavy use of kNeuralKit to perform its computations, and in addition utilises MPI (Message Passing Interface) extensively to allow the combined computational power of multiple computers to be used as a cluster during the development process. An adapted, more specific variant of Neuron Factory has been used heavily during my PhD to develop copious neural networks in parallel using clusters of Mac OS X and Solaris machines here at the University. The MPI functionality within Neuron Factory allows multiple networks to be developed simultaneously, and in addition enables the computational load of individual machines within the cluster to be load-balanced. Once the Neuron Factory stub application is installed on the other nodes of the computational cluster, the neural network architect need only sit in front of the one machine running the complete Neuron Factory application in order to control any of the machines in the cluster, either individually or globally.

I am currently evaluating Apple's Xgrid technology for the purposes of load-balancing the computational overhead of neural networks. If Xgrid provides a sufficiently powerful alternative to MPI, then I will consider releasing an Xgrid-compatible version of Neuron Factory. If you would be interested in such an application, please let me know by e-mailing me at .

Axis Crossover Vector Generator

One facet of my research necessitated a scale- and location-invariant vector description of human shape to be developed, so that a neural network could be developed to classify objects by their shape. In my research, the axis crossover vector was developed as a mathematical technique which could transform a closed-loop contour into a representation which a neural network could understand for classification purposes. In the case of my research, the neural network was trained to distinguish between pedestrian and non-pedestrian moving objects in video sequences. If you're really gripped by this stuff, you might want to plough through some of my research papers, which describe the process in painstaking detail. However, an axis crossover vector is theoretically more generally applicable than just the use I had of it. So you might want to generate your own axis crossover vectors for your own purposes, but you might not have the time or inclination to familiarise yourself with the underlying maths involved. And that is why the Axis Crossover Vector Generator was born, to do this boring stuff for you. You supply the closed-loop contour (in terms of contour control point (x,y) coordinates stored in a text file), along with the number of axes you'd like to project, and the angles you'd like to project them at, and the app does all the graft for you. Axis Crossover Vector Generator is a Mac OS X application, although the brains of it is written in ANSI C, so if you just want the algorithm (eg. to use on Solaris / FreeBSD etc.) then e-mail me at and I'll send it to you.

Active Contour Model Algorithm

Although it's not an application as such, and nor is it an API, I am providing the internal mechanisms for my Active Contour Model for free download. Many people have expressed an interest in my ACM algorithm, for two reasons: it works and it's free. Furthermore it bypasses the need to familiarise yourself with all of the mathematical mechanisms involved in ACMs. The code is provided as the algorithm for two reasons: packaging the algorithm within the application I developed for my own model would make the algorithm too specific for normal use, but on the other hand I don't have the spare time needed to make a fully fledged API out of the algorithm (not one which would be generally applicable in any case).

The ACM used in my research is a variant of the Williams and Shah Fast Snake ACM model, and as such uses a 'greedy' (admissible) approach to energy minimisation. In other words, it doesn't miss a trick, but this luxury places extra computational load on the processor. I'm putting the final touches to my code so that it's sufficiently tidied up as to be decipherable by other people besides me, but the algorithm when posted will support the following features:

• Spawning control points at specific distances around the initial contour, despite the user specifying sparse coordinates
• Control point contextual locking, allowing corners to form naturally in the contour when a control point is in an 'important' part of the image (as defined by the contour's energy function)
• User-definable energy weight parameters for internal, continuity, curvature, image and locking variables
• Control point momentum between video frames, allowing the contour to predict complex movement and shape deformation by the object being tracked

The algorithm is written in Objective-C, but it would be easily translatable into other object-oriented languages such as C++, Java or Smalltalk. Or of course you could just un-wrap all the object-oriented aspects of it and re-write it in ANSI C if you want a procedural version of the algorithm. I'll leave that as an "exercise for the reader" though, as I quite like the Objective-C version myself.