3D Realtime Face Reconstruction with Matlab and Kinect

14 02 2011

This post has been updated and moved to a new dedicated website HERE

I managed to map the texture on the pointgrid aquired from the Kinect 3D image.

This is what my program does:

  1. Gets 3D + live image
  2. Converts 3D coordinates to real-world coordinates
  3. Makes a trigonial mesh
  4. Fills in the entire trigonial mesh with the correct colors (from live image)
  5. Starts from 1


  • ~2 seconds per frame (0.5 FPS)
  • Accurate to ~5mm
  • Works from 50cm to at least 700cm
  • 0.3 MP Resolution
  • If not set to realtime (=postprocessing) framerate would be 30FPS

See below this youtube of my face, which consists of around 40.000 datapoints:
This post has been updated and moved to a new dedicated website HERE

And then here follows the code. Looks basic? It was, i spent a total of 2 hours on it.

%% acquire imagesfor k=1:2option.adjust_view_point = true;% Acquire rgb and depth imagetic[rgb, depth] = mxNiImage(context, option);
real_XYZ = mxNiConvertProjectiveToRealWorld(context, depth); % return XYZ[mm, mm, mm] on the world coordinate system
%some croppings%real_XYZ=real_XYZ(200:400,300:400,:);%rgb=rgb(200:400,300:400,:);
[M]=-mn32m3(real_XYZ); %dit duurt erg lang[Mrgb]=mnrgb2mrgb(rgb); %dit duurt erg lang
itmp=sum(M~=0); %locate existing infostmp=itmp~=0; %positive if there is info
itmp=M(3,:)>-800; stmp=itmp==1;
trisurf(t,p(:,1),p(:,2),p(:,3),1:size(p,1),'edgecolor','none')axis equal;view(200,66)shading interp
%real_X = real_XYZ(:,:,1);%real_Y = real_XYZ(:,:,2);%real_Z = real_XYZ(:,:,3);
% update FIGURE% set(h1,'CData',depth); % set(h2,'CData',rgb); %set(h3,'CData',rgb); %set(h4,'CData',depth);%set(h4,'AlphaData',double(depth/50));%%tmp = real_XYZ(:);%tmp(find(real_XYZ==0)) = NaN;%tmp = reshape(tmp,[height width 3]);
drawnow;disp(['itr=' sprintf('%d',k) , ' : FPS=' sprintf('%f',1/toc)]);end% Delete the context object mxNiDeleteContext(context);

Another 3D face reconstruction




4 responses

23 03 2011

Is posible to create a partial reconstruction of an object by using few images?If it’s so can you please send me an e mail or make a post with some code?Thanks in advance

23 03 2011
Tim Zaman

i’ve updated the matlab code section in the top. I would also advice you to check out the VLSIFT libraries for matlab! happy coding.

17 02 2011
Tim Zaman

its a triangular surface plot. For that you need to make the delaunay triangles first (line 15). trisurf(Tri,X,Y,Z,C) displays triangles defined in the m-by-3 face matrix Tri as a surface. Each row of Tri defines a single triangular face by indexing into the vectors or matrices that contain the X, Y, and Z vertices. The color is defined by the vector C. The Colormatrix C is the one we took from the RGB camera on the kinect :).

17 02 2011

Thank you for your program .
line 16 , what is that?

%d bloggers like this: