This is a very incomplete list of libraries that you have to know about. Most of these have been discovered and used during my M.Sc or private projects. I have contributed and written some of them and the others are just libraries that I like.

Maths and Optimisation:

  • Eigen – This is by far the best Matrix and Vector toolkit on the face of the planet. It is entirely implemented in C++ templates which means there are zero issues compiling on any platform. It also has SIMD optimisations for both X86 and ARM NEON platforms. Eigen is used in some big projects and has all the main factorisation algorithms built in.

  • Ceres Solver – I have mentioned this one before. Ceres is an optimisation library built, maintained and used extensively by Google. It has convex and non-convex optimisation algorithms built in, and a simple approach to make it stupidly easy to optimise anything. It supports auto differentiation as well as user defined derivatives functions and linearizations. After some coercing I even got it built and running on iOS.
  • nanoFLANN – If you need k-d trees or a nearest-neighbour implementation then this is the way to go. Once again it is a simple C++ template implementation of k-d trees that can interface directly with Eigen data structures or standard C++ STL vectors. nanoFLANN has muh better performance than OpenCVs FLANN implementation which has become a bit bloated these days.

Image Libraries:

  • GPUImage – Is you need a decent image library for OSX or iOS then you have no way to ignore this. Brad Larson has done a great job on making GPUImage the most sort after imaging toolbox for these platforms. This one leaves Android developers wishing it was cross platform. GPUImage makes full use of the GPU and texture buffers to provide super fast processing even on older iPhone models. From feature detection to chromakeying GPUImage has it all.
  • OpenCV – aka BrokenCV as my personal experience has shown. Even though I am not a huge fan of the library and have a very strong opinion on it becoming over bloated and incompatible with other libraries, it i worth a mention. When it comes to implementing Computer Vision stuff fast, on most platforms, with moderate performance then you can’t really beat it. Although I find there have been a large number of bad design decisions along the way. For instance it relies on Eigen, but is incompatible with Eigen types, you have to convert them first. Everything is wrapped and hidden away in an abstracted manner so you can’t just plug and replace the pieces you need. Let’s hop OpenCV 3 will fix some of this.