## Sunday, August 21, 2011

### OPENCV: STEREO MATCHING

In a previous post I talked about how to calibrate a stereo camera using OpenCV. Today, I would like to talk about the next step. Once your stereo camera is calibrated you can estimate the 3D position (relative to the camera) of any object given its position in the left and right image. For that, we need to calculate the stereo disparity for that object (stereo disparity = the difference in image location of an object seen by the left and right camera). If we want to know the 3D position of all points in a stereo pair of images, then we want to compute a dense disparity map. And that is what this post goes about.

A dense disparity map looks like this:

I am not going to explain the details or the math behind it, I am more of a practical kind of guy. So let's start.

Google ads, probably not very well related to the audience of this blog...

Basically OpenCV provides 2 methods to calculate a dense disparity map:
In this post I will focus on cvFindStereoCorrespondenceBM, this method is based on Konolige's Block Matching Algorithm. The OpenCV call looks like this:

void cvFindStereoCorrespondenceBM(const CvArr* left, const CvArr* right, CvArr* disparity, CvStereoBMState* state)

The structure CvStereoBMState contains all the parameters that are applicable to the algorithm. There is a bunch of them (pre-filtering, Sum of Absolute Difference windows size, disparity-related, post-filtering...). So, to make it easy, I implemented a small Gtk application that takes 2 images (left image and right image), calculates the disparity map using cvFindStereoCorrespondenceBM and allows you to play with the parameters.

The application is written in C and can be downloaded here: StereoBMTuner-1.0. The application depends on the libraries gtk+-2.0, gmodule-2.0 and opencv. Be sure to have them installed in your system.

tar xzvf stereoBMTunner-1.0.tgz
cd StereoBMTunner
make
./main

The last command will execute the application

As you can appreciate, the disparity map generated using the default parameters is hardly similar to the first image on this post. But, you can tune the parameters until you get a clearer disparity map. This video shows the use of the application:

Once the parameters are tuned, the disparity map is much better

It is still not perfect, but it is not so bad either.
Now, to use this application with your own couple of images the only thing you need to do is execute the application like this:

./main -left /path/to/my/image/left -right /path/to/my/image/right

25JKKNMXU6FE

Matt Montag said...

You had me really amazed with that first disparity map image...but now I realize it's just faked.

Martin Peris said...

Hi Matt!

Thanks for your comment, yeah the first disparity map looks great, but not because it is a fake but because it is the "ground truth" disparity map distributed with the stereo scene. This stereo scene is called "Tsukuba" and the "ground truth" was, probably, obtained using structured light techniques.

There are stereo matching algorithms, other than block matching, that can achieve really good results, for example the algorithm based on Graph Cut. But, unfortunately, none of them is capable of constructing a ground-truth-like-quality disparity map in real time. Yet ;)

Best regards,
Martin

Anonymous said...

Thanks so much for this blog. Im just waiting for my USB cams to arrive so I can start experimenting!

Martin Peris said...

Hi!

Thanks for your comment :) I am glad you find it useful, good luck with your experiments!!

Martin.

il-sognatore said...

Hi Martin!
I love you application!
I was trying to elaborate a stereoSGBM version (c++) of your application but i get error cause of the glade version. Which version did u use?
If you have time you can try: just rename your program from .c to .cpp and you'll notice that everything is compiled but u get errors during the running.
I hope we can collaborate ;)
Greetings,
Mattia

Martin Peris said...

Hi Mattia!

Thanks a lot for your comment :) Actually I was thinking about doing the same myself and post it. Let me give it a try during the weekend.

Oh, my version of libglade is 2.6.4

Best regards,
Martin

il-sognatore said...

Problem solved!
just modify the declaration of the all the functions
in

Let me know your results ;)

Mattia

mike said...

Hi Martin,
Thanks for answering my previous question.
I've captured the images using
Minoru 3d camera (http://www.minoru3d.com/) which is a low cost 3d webcam.

It seems that the calibration error using it is about 0.8 while using your images the error is 0.4.

Any chance you can provide which stereo camera (stereo setup) you've used?

Thanks again,
Michael

Martin Peris said...

Hi Mike,

Thanks for your comment :) The images that I provide as a calibration example were taken using a Videre Design Stereo Camera http://www.videredesign.com/

Best regards,
Martin.

chien said...

Mike:

Thanks for the blog. I tried to access the link to "Konolige's Block Matching Algorithm", but didn't have permission to do so. Do you happen to have an electronic copy of it that you could send it to me?

Thanks,

Chien

Martin Peris said...

Hi Chien!

Thanks for reporting the error, I've updated the link to point at the correct URL. You should be able to download the file at this address: http://www.cs.cmu.edu/~motionplanning/papers/sbp_papers/integrated1/konolidge_stereo_vision.pdf

Best regards,
Martin

Anonymous said...

hey,

i am working with opencv.... If I am creating disparity maps of objects, the disparity map is very map, It looks like your first result.... But I can't change parameters cause the program doesn't react...
Thanks a lot.

Best regards
Patrick

Martin Peris said...

Hi Patrick!

Thanks for your comment, I am afraid I would not be able to help you if you don't provide more details, like O.S. Version, OpenCV version and such.

Best regards,
Martin

Anonymous said...

Hi Martin,

first, i have another question. I am capturing images of different positions of the checkerboard (nearly similar to your pictures). But the quality of the disparity of different captured objects are just random. Sometimes better, sometimes worse.
Furthermore, if i repeat the calibration with new calibration pictures and want to create the disparity map of the same scene, the disparity is also random. Am I doing anything wrong? I have to add that I don't/can't play around with the parameters mentionend below. Is that the source of error? Perhaps you can help me.
Thank you very much.

Best regards
Patrick

Sebastian Calleja said...

Martin: Te felicito por tu blog muy completo! Estoy tratando de compilar el codigo que pusiste en este post en windows. Tengo una duda en la parte que se lee el archivo con las imagenes, que extension tiene este?

Saludos

Martin Peris said...

Hola Sebastian,

Muchas gracias por tu comentario. En principio el archivo con las imagenes puede tener la extension que prefieras, que yo recuerde no hay ninguna restriccion en ese sentido.

Buena suerte con Windows :)

Saludos.

Sebastian Calleja said...

Martin: Yo de nuevo. No entiendo muy bien que relacion tiene este post con el de stereo calibration. Me da la impresion de que no es necesario calibrar las camaras para poder tener un un dense dispatiry map. Cual es la diferencia de un dense disparity map con un disparity map?

Saludos y muchas gracias por la respuesta anterior.

Martin Peris said...

Hola Sebastian,

En el siguiente enlace explico claramente porque es necesaria la calibracion para poder obtener un dense disparity map http://blog.martinperis.com/2011/08/stereo-vision-why-camera-calibration-is.html

En realidad "disparity map" y "dense disparity map" son lo mismo, solo que el nombre "dense disparity map" hace hincapie en que TODOS los pixels de la imagen tienen asignado un valor de disparidad. Para algunas aplicaciones solo es necesario calcular la disparidad en alguno de los puntos de la imagen, no en todos. Esto se conoce como "sparse disparity map".

Espero que esto resuelva tus dudas.

Saludos.

Sebastian Calleja said...

Hola Martin: Estuve revisando el codigo de Stereo Calibration y usas la funcion stereoRectifyUncalibrated(). Se supone que esta funcion puede ser usada para rectificar sin haber calibrado las camaras? Una vez calibradas las camaras puedo rectificar dos imagenes que no contengan el tablero de ajedrez, pero hallan sido sacada con las mismas camaras?

Martin Peris said...

Hola Sebastian,

Si te fijas, la llamada a stereoRectifyUncalibrated() nunca se llega a realizar, solo esta ahi porque el codigo es una modificacion de un ejemplo de un libro y no quite las cosas "no necesarias". Esta funcion puede rectificar el par de imagenes estereo sin necesidad de conocer los parametros intrinsecos de las camaras y su posicion relativa, pero el resultado puede ser bastante malo si no se corrige primero la distorsion introducida por las lentes. Mas info sobre esta funcion en: http://opencv.willowgarage.com/documentation/camera_calibration_and_3d_reconstruction.html#stereorectifyuncalibrated

Una vez calibradas las camaras, por supuesto que puedes rectificar imagenes que no contengan el tablero de ajedrez, seria muy tonto no poder hacerlo verdad? jejeje

Saludos.

Miguel said...

Hola Martin.

Estoy realizando mi proyecto fin de carrera en Windows, ¿Hay alguna posibilidad de poder ejecutar el Stereo-BM-Tuner en Windows? Tu blog me está siendo de gran ayuda, muchas gracias.

Martin Peris said...

Hola Miguel,

Me alegro de que el blog te sea util. En principio, las librerias que uso para StereoBMTuner (gtk+-2.0, gmodule-2.0 y opencv) tambien estan disponibles para Windows, asi que deberia funcionar. No digo que sea facil, porque igual te toca cambiar alguna cosa, pero deberia ser posible.

Espero que puedas portarlo a windows sin demasiados problemas.

Un saludo.

Anonymous said...

Hi Martin,
can you just say anything about the BMState parameters? I try to tune my disparity map but I'm a little bit confused by the range and the step size of these parameters. If I choose the wrong values the cvFindStereoCorrespondenceBM() function crashes and my whole program breaks. Why preFilterSize or SADWindowSize accept just values which are calculable by modulo 2 (%2) for example?
In addition, I don't understand where the range / interval of these paramters starts or ends.
Hope you get my problem and you could give me some hints.
Cheers, Peter

Martin Peris said...

Hi Peter!,

I would recommend you to read pages 438-444 of the book "Learning OpenCV: Computer Vison with the OpenCV Library" for further details. Or cvFindStereoCorrespondenceBM() documentation.

Anyway, preFilterSize and SADWindowSize only accept odd values from 5 to 21, which translates into a window size of 5x5 to 21x21. The value must be odd so there is the same number of pixels around the center pixel of the window in any direction. The minimum(5) and maximum(21) values were a design decision of the people from OpenCV, a window with less than 5x5 pixels would not contain enough information to perform the matching and a window with more than 21x21 pixels would make the algorithm perform very slow.

Another important point is that the numberOfDisparities must be multiple of 16, this has something to do with some kind of processor optimization.

I hope this could solve your doubts.
Best regards,
Martin.

Ronald said...

Hi Martin,
its me again ... its real fun to work with your samples. You know how to get your reader's interest =) This topic is very fascinating!!

Now I still wondering why I got such a bad disparity map.
http://imageshack.us/photo/my-images/607/disparity.png/

Using following parameters:
preFilterSize = 15
preFilterCap = 19
minDisparity = 0
numberOfDisparities = 64
textureThreshold = 0
uniquenessRatio = 0

Tried to vary the parameters, but I didn't get better visual results =( Do you know what I have to change for a good output?
Cheers,
Ronald

Martin Peris said...

Hi Ronald!

I am glad to see that you are having fun :)
I saw your disparity map, and the problem is not so much the parameters for the block matching algorithm but the block matching algorithm itself.

Looking at your picture one can see that there is many areas with very low texture, those areas are really hard to match with block matching algorithm. Actually, low textured areas have been a pain in the neck for researchers on the field for decades. And still is.

You could try out other, more robust, matching methods included in OpenCV. But they are more computationally expensive (sayonara real-time performance).

I hope this helps.
Martin.

Ronald said...
This comment has been removed by a blog administrator.
Ronald said...

Hi Martin,
Ok, I will try to use cvCreateStereoGCState ...

But you use the block matching algorithm, too. Why your disparity results looking much better? =)
Kind Regards,
Ronald

p.s. for the last post!

Martin Peris said...

Hi Ronald!

The sample stereo pair (University of Tsukuba "head and lamp" scene) that I used in this post has very few areas with low texture, which makes it "easy" for the block matching algorithm to generate a decent disparity map. But that rarely happens in real-world conditions :(

Good luck with Graph Cuts method :) You can also try out SGBM (Semi-Global Block Matching) method which is available in OpenCV as well.

Best regards,
Martin

maxwell said...

Hi Martin,

first of all, thank you very much for all your never-ending help to all the help-seeking OpenCV-users. Unfortunately, the documentation is quite bad in the stereo-field, so we are all dependent on people like you! Thank you very much for that!

I finally managed to get the SGBM-algorithm running, the results look quite good. But, unfortunately, it is quite slow. I'm using images with apprx. 730x530 pixels (after rectification and cropping), 128 disparities, fullDP = false, SADWindowSize=5 on a i7 desktop-computer. The SGBM-part takes almost 300ms, which seems to be pretty much to me. Could you give me a benchmark on how long your algorithms take?

Thank you very much,

Max

tina said...

Hi Martin Peris,

hope you are fine and enjoying your work.
Well i am really impressed with your work.
Actually i am also doing stereo calibration for detecting an object .

I did almost the same as i read from many blogs .
My result of depth map is shown via this link :http://s1096.photobucket.com/albums/g330/Labi_Darling/?action=view&current=Untitled15.mp4

Please tell me that my disparity map is so bad ?? ? ?
I am using regular texture to check my disparity map even then my result seems so bad.
Any idea for the current issue will be appreciated .
Is it possible to contact with you via email?
My email id is : rubi_faith@yahoo.com

tina said...

Hy Marin,

i want to access your code of the above program.
thanks

Martin Peris said...

Hi tina!

I took a look at the video that you attached. The problem is in your calibration step, from the video it looks like you are only taking a few images for calibration with the chessboard in a very similar position on each frame. For better results you should capture many more frames for calibration, with the chessboard in as many positions and orientations as possible.

A better calibration should improve the quality of your disparity map. I would recommend you to read this post if you havent't read it yet: http://blog.martinperis.com/2011/01/opencv-stereo-camera-calibration.html

I hope this helps. By the way, you can get the source code of the stereo matching program here: http://www.martinperis.com/stereobmtuner/stereoBMTuner-1.0.tgz

Best regards,
Martin.

Liz said...

Hi Martin!
Thank you very much for your blogpost. I've done all the steps and it works at first go :)
But there's one thing, I don't understand: In your previous post, you've calibrated the cameras and got several .xml files with the calculated matrices. Are those matrices used in this stereo matching algorithm? Probably I'm just blind and don't see the connection between the programs... ;)
Best regards,
Liz

Martin Peris said...

Hi Liz! Thanks for your comment.

Actually, you are not blind :P The program on this post is assuming that the images have been already rectified and undistorted.

The rectification and undistortion step is explained in the post about stereo camera calibration, I paste here the part that is useful for you:

"The files m*.xml are the distortion models of the cameras. So we will need these matrices to undo the distortion of the images caused by the lens. Using the cvRemap() function:

cvRemap(imgLeftOrig, imgLeftUndistorted, mx1, my1);
cvRemap(imgRightOrig, imgRightUndistoreted, mx2, my2);"

Anyway, thanks for pointing this out, I should make another post puting it all together: from image capture to 3D reconstruction.

Best regards,
Martin

Liz said...

Hi Martin!

Thank you for your fast answer! Now, I think I've got it :)

I calibrate the camera with stereo_calibrate.cpp and my own set of chessboard images, then use the matrices mx1, mx2,... to rectify and distort a pair of stereo images. The rectified images are loaded in the stereo matching program from this post. After adjusting the parameters, a hopefully nice looking depth image is created. And with that depth image and matrix Q, it should be possible to create a 3D image (either with your code from the other post or with reprojectImageTo3D()).

Best regards,
Liz

azer89 said...

hi, i would like to say thank you since your tool is really handy for tuning BM parameters :D
I am Windows user and your progam is written in Linux and using GTk+
at fist it was little difficult to port them into Windows but i've done it, i put all the steps in my blog
http://azerdark.wordpress.com/2012/04/28/compiling-gtk-app-on-windows/

tina said...

Hy Matrin,

Thanks for such a good reply .
I am capturing 20 frames for disparity map.
Actually i have a confusion also .
I did single camera calibration and i used this function cvInitUndistortMap() for camera distortion.
and when i did stereo vision , i used this cvUndistortPoints function for undistortion map .
Could you tell me the difference between them.
Thanks for such a nice help .

Anonymous said...

Hi Martin,

I tried cvFindStereoCorrespondenceGC, but I get a black screen no matter how I tune the parameters. Have you ever tried cvFindStereoCorrespondenceGC ?

Thanks!

rida said...

hi Martin, i need urgent help.
I calibrated stereo camera and i have calibration error is 0.456 some thing like that .
I have just concern with disparity map . Problem is my closer objects are darker than farthest object which is not correct. calibration is good enough. Could you please tell me tell could be the reason of getting wrong disparity map .

alfredo yohanes said...

hi rida, i made a mistake like you did. did you set your left cameras as the right cameras? that was the reason for me when i have closer objects are darker than farthest object

rida said...

Hi Alfredo,

The disparity map from my camera is really strange .
and its hard to get clear disparity map because of texture . hand has not many texture .which create problem in getting disparity map of hand .Do you have any idea regarding that ?
thanks

rida said...

Hy Alfredo ,Martin and all,

look at this my disparity map.
http://s1268.photobucket.com/albums/jj562/ridasana/?action=view&current=Untitled38.mp4

I just want to segment hand, but my nearest object seems so dark as compare to farthest objects.

if you see my left camera set same as right .
also if you see in disparity map . i have double hand when i move my hand.

rida said...

hy all,

sorry to disturb you again .
Could you tell me rang to set BM parameter slide bars .
like H S V have range . Could u tell me what are ranges of BM parameter ?
Thanks

rida said...

hy Disparity result on highly depended on rectification and calibration while calibration and rectification highly depend on chessboard rotation and translation .

Because only one time i got expected disparity map but mistakenly i did not capture video.
i think for my project i can't take this risk like this . for example if i did not get expected disparity map then run again and try it .

its time consuming .

rida said...

So what should i do?
I can't run it again and again if i do't get expected result .
In my opinion , i think i should take capture more frames so that may be with more rotation and translation, i may be will get expected disparity map .

Could any one please say me something.
Its urgent . thanks

rida said...

its urgent

Anonymous said...

Hi
How do I save the disparity image.
cvSaveImage
cvSave are both giving segmentation fault.

Kyle said...

fantastic tool, thanks!

Wu said...

Excellent Work , you are the best Martin ,
preFilterType
preFilterSize
preFilterCap
minDisparity
minimum disparity
numberOfDisparities
textureThreshold
uniquenessRatio
speckleWindowSize
speckleRange
trySmallerWindows
disp12MaxDiff

the documentation is not helpfull at all ... any explanation please , I'm new to this field ...

Gianfranco said...

great work!

I am working with this stuff too and the documentation is like wu said! its not so helpfull and i think in the opencv 2.4.3 just the SGBM algorithm is avaliable!
I found a good example of stereo match in the opencv folder(C:\opencv\samples\python2)but its not so easy to understand all the parameters.

do you have any tip?

Đức Anh Lê said...

Hi martin, thank you for your tool.
But can u help me to use your tool in Windows, i don't have much exp with Linux,
Thank you again.

Terry Rea said...

Well done Martin!

You have a very informative blog. I have worked my way through your camera calibration and stereo matching tutorial and successfully duplicated your results. Your work appears focused on stereo cameras; I am working on an effort with a single camera to build Shape/Structure from Motion point clouds. Are you doing any work in that field?

Terry

jose said...

Por casualidad lo hiciste en c++ ?

Darío Rosas said...

The depth map in what space is it?

yaman neameh said...
This comment has been removed by the author.
yaman neameh said...

Hello,

the result of 3d reprojection using StereoSGBM algorithm is the X,Y,Z coordinates of each pixel in the depth image.

points = PointCollection.ReprojectImageTo3D(disparityMap, Q);

by taking the first element of this result:

points[0] = { X= 414.580017 Y= -85.03029 Z= 10000.0 }

I'm confused here!! to which pixel this point refers to ? and why it is not like this X=0,Y=0,Z=10000.0!

zohir said...

merci

Judy Reinhold said...

Hi
The last time I used it, it worked fine. I ended up doing a clean install of Ubuntu 12.04 for use with a TI SDK. When I try to run it now, I get a segment fault. (Ubuntu 12.04 64-bit, OpenCV 2.4.6, PCL 1.7).
Thanks
Ralph

Judy Reinhold said...

The segment fault appears to be due to access rights. I ran as

sudo ./main

It ran without a fault, but the graphic did not come up. I assume it is a GTK issue. I'll look at that.

akilan said...

Hi

Thanks for ur great job.
I gone thru some of the publications on semi global match(SGM). To me SGBM looks similar. Can clear me the difference between SGM and SGBM?

Thanks

Harish Jamakhandi said...

Hi Martin,

We are a set of students working with stereo vision which we plan to implement and get the real time distance of the objects from the camera. Now that we are getting a good disparity we are finding it difficult to keep the brightness or the pixel value of the disparity constant bcoz of which my distance varies with the same object in same position. Plz suggest us a method to keep the pixel values of the disparity image constant. We use visual studio and Windows 7 and two distinct but exactly same in features Logitech cameras

Suhas S R said...

Hi Martin,
Its good work, it help me a lot!
Actually I am doing for 2592x1944 resolution images, but my disparity is not upto the mark. Can you please guide me the values for SGBM
At the moment I gave
int windowSize = 9;
int disparityRange = 416;
int preFilterCap = 52;
int minDisparity = -103;
int uniquenessRatio = 15;
int speckleWindowSize = 150;
int speckleRange = 2;
int dispMaxDiff = 10;
bool dynamicP = false;
int smoothP1 =1944;
int smoothP2 = 7776;

vicky singh said...

it would be great if you provide breif explanation or probably a link on how BM (block matching) in opencv works/implemented...

Vaishu D said...

can any one tell me that what is the difference between correlation map and disparity map?

Naga Bharath said...

Hi Martin...

I tried using your code for changing the parameters of the algorithms. I have downloaded GTK+-2.0 package and tried downloading Gmodule-2.0 in Ubuntu..but I couldnot get any Gmodule package. I tried making the file with out Gmodule, i have got errors

/usr/bin/ld: cannot find -lcufft
/usr/bin/ld: cannot find -lnpps
/usr/bin/ld: cannot find -lnppi
/usr/bin/ld: cannot find -lnppc
/usr/bin/ld: cannot find -lcudart

Thanks and Regards
Naga Bharath

jumpjack2 said...

I can't understand if and how I can use OpenCV to get a depth map from a stereo pair like this:
http://sci.esa.int/science-e-media/img/76/ESA_Rosetta_OSIRIS_SiteJ_Anaglyph.png

Is OpenCV just a library or a complete SW?!?

i can't tune the parameters until i get a clearer disparity map
help !!!
(stereobmtuner:5509): Gtk-WARNING **: Could not find signal handler 'on_adjustment1_value_changed'
....
..........
........
.............
(stereobmtuner:5509): Gtk-WARNING **: Could not find signal handler 'on_adjustment9_value_changed'