It has been observed that one's nose is never so happy as when it is thrust into the affairs of another, from which some physiologists have drawn the inference that the nose is devoid of the sense of smell. –– Ambrose Bierce, "The Devil's Dictionary"
Vision, Image Processing and Graphics
Introduction
During the Summer I worked on several computer vision projects most of which are still ongoing. I had time, nevertheless, to work in an example that would allow me to introduce myself and my work to you, my students.
As an introduction to my classes I wanted to give my students some useful hints as to when it would be prudent to approach me with a question, or comment. If you're observant you would see that the degree of alertness of your humble professor is in direct proportion to the displacement of his glasses along the curve of his nose. If it gets too low, there is not much sense in seeking help from me, because I might then be in more dire need of help myself. To help you visualize what I mean, I modified a portrait of my profile to display a graded scale on my nose in the risky zone. You will see it at the end of this document.
Computer science is very much like cooking for yourself. You have an more or less precise idea of something you would like to eat, and you must be hungry. Then, you look inside your refrigerator and see what's in your pantry. You fetch the ingredients that suit you the best, and see which pots, pans and utensils are left clean. You then apply a mixture of creativity, past experience, and good luck. After some time you get the results and you enjoy them.
My ingredients are this time a single picture of my profile. My kitchen is my computer, and my pots, pans and utensils will be for this time, Mathematica 6, Xfig, the GIMP, high school algebra, elementary calculus, and geometry. Of course, I'm also using my experience and whatever common sense I'm left. My hunger is my eagerness to get this application done in a reasonable amount of time with some decency in the presentation. I hope you enjoy it!
Acquiring data
We read in a picture onto which we would like to map a graded scale on the nose profile. Ginger took this picture with my little camera.
In[2]:=
Out[2]=
The previous picture is scaled to 30% of the original, which is a big picture.
The graded scale was rapidly drawn with the OpenSource software Xfig, and then imported.
In[3]:=
Out[3]=
The data of the drawing looks like this. Notice that it's given in RGB triples with the number 255 coding for white, and 0 for black.
In[4]:=
Out[4]=
In[5]:=
Out[5]=
In[6]:=
Out[6]=
The picture includes much more than just the numerical data. It has rules and other meta-data. Here we take the RGB numbers and discard the rest.
Similarly, we separate the numerical data from the drawing.
In[7]:=
Out[7]//Shallow=
In[8]:=
Out[8]=
In[9]:=
This is a plot of the numerical data corresponding to the first plane of the data matrix.
In[10]:=
Out[10]=
As you can see, we have now single numbers instead of the RGB triples.
In[11]:=
Out[11]//MatrixForm=
In[12]:=
Out[12]=
In[13]:=
In[14]:=
Out[14]=
Mapping the graded scale on a cylinder
We would like to map the scale data on a cylinder with radius one. This requires only an elementary computation, but we need to draw a figure.
In[15]:=
Out[15]=
Imagine we're looking to the cylinder from above. The plane with the scale drawing looks as the blue line. From the figure above it's easy to see that , for a given value of z, the coordinates of a point (X,Y) on the image plane when projected to the cylinder are given by
and 
, where r is the radius of the cylinder and X is the distance from the image plane to the origin. This is coded in Mathematica as follows:
In[16]:=
When mapped the points look like this.
In[17]:=
Out[17]=
This looks allright but we would like to join the points with lines to achieve a more continuous effect.
In[18]:=
Out[18]=
This almost works but we see that the points originated by the numbers are giving us many undesired lines, so we remove the numbers from the data by taking only the region to the right of the numbers.
In[19]:=
In[20]:=
Out[20]=
Mapping these data on the cylinder we get the following:
In[21]:=
In[22]:=
Out[22]=
It still has many undesired segments. Notice that they are characterized by their long length. The segments we are interested in are small, going from neighbor to neighbor. There is a function "Split" in Mathematica that partitions a list according to a given criterion. We split them whenever the distance between two consecutive points is greater than, or equal, to 0.5.
We then take those lists and prepend the header "Line" to them, so that they can be interpreted as vertices of a polygonal line by the Graphics3d function.
In[23]:=
Out[23]=
We now look to the data from the side and use thicker, colored segments.
In[24]:=
Out[24]=
This is good, although we can perceive some gaps between segments of the vertical axis, and my nose does not really look like a cylinder. Is there a way to map the scale on a shape that looks more like a nose? The answer is, yes!
Finding the nose profile
We would like to extract the shape of my nose from the picture. To do that we will have to segment, or separate, the nose from the rest of the picture.
One way to do this is by classifying regions on the image according to their color. Since my nose has more or less the same color everywhere, with slight variations on intensity and saturation, we see that it's reasonable to work with the hue.
The original picture encodes color in the RGB (Red, Green, Blue) system. We need to convert these values to another color system called HSI (Hue, Saturation, Intensity). The following function translates RGB to HSI data. The algorithm is given in Digital Image Processing, by Gonzalez and Woods.
In[25]:=
We will work with a subimage.
In[26]:=
In[27]:=
Out[27]=
In[28]:=
Out[28]=
The subimage is still too big, so we take a portion of it. Notice the structure of the RGB data.
In[29]:=
Out[29]=
In[30]:=
Out[30]=
We now apply the RGB to HSI function described above.
In[31]:=
In[32]:=
Out[32]=
Displaying the results we get the following figure. We see that indeed, the nose looks as being distinguished by reddish hues.
In[33]:=
Out[33]=
To segment the nose we need only apply a couple of thresholds, since we know that in HSI the reddish hues are in the vicinity of 0 (modulo 1). We use Mathematica's Manipulate function to create an interactive graphical interface to play around and find the right values.
In[34]:=
Out[34]=
To find the edge we will apply Sobel operators which are really difference operators that work very much like the vertical and horizontal derivatives of the image. They measure change, and the intensity in the thresholded image jumps from 0 in the background to 1 in the area of the nose, elsewhere it's constantly 0 or 1.
In[68]:=
In[69]:=
In[70]:=
Out[70]=
We now can apply another threshold this image. To select the best value we need to experiment, so we build an interactive interface.
In[38]:=
Out[38]=
It is now easy to get the coordinates of the edge points by using Mathematica 's Position function, which gives a list of coordinates of points having the value of 1.
In[71]:=
In[72]:=
Out[72]=
Since we now know what to do with larger images, we clear a few unnecessary arrays from the memory, and bring a bigger subimage containing my nose.
In[73]:=
In[74]:=
Out[74]=
In[75]:=
In[76]:=
Out[76]=
We apply the same steps as above, only we use the values for the thresholds found in the previous experiments.
In[77]:=
In[78]:=
In[79]:=
This is the extracted edge.
In[80]:=
Out[80]=
In[81]:=
In[82]:=
Out[82]=
It has too many points, so we drop 100 points from the top, which don't look as very close to the real profile.
In[83]:=
We now fit a continuous function to the data. I like the equation
,
because I can control the steepness of the curvature in the bottom end by plugging in another number than 40 in the second term, and it has a nice, small curvature in the other end.
In[84]:=
Out[84]=
This is the "nose" polynomial.
In[85]:=
Out[85]=
We need this curve to vary consistently with our known ranges for x,y,z. Its variables range from 15 to about 750, and from 1 to 220; whereas we would like them to go from 0 to 1 and from 0 to 160, respectively.
We use a simple linear relation between both domains. It can be visualized as a segment of a straight line.
In[86]:=
Out[86]=
This line's equation is 
= 
+ 
, where
In[87]:=
In[88]:=
Out[88]=
The other normalizing relation is 
, where
In[89]:=
In[90]:=
In[91]:=
The following code finds the coordinates of the black pixels in the scale data; the ones that we would like to map on the nose.
In[92]:=
Now we use the normalizing relations in our mapping equation. The trick is to multiply the x coordinate found in the cylindrical mapping, by the shape information from the "nose" polynomial, which is made to vary with respect to the vertical coordinate in the scale data. You may as well say we're building a separable polynomial surface.
In[93]:=
We partition the points list as above, leaving only chains of near neighbors.
In[94]:=
To achieve a continuous effect in the scale's central axis (remember we didn't like the gaps) we find all points with y=33 in the scale image data.
In[95]:=
And we define an uninterrupted polygonal line for the axis.
In[96]:=
We now map all these lines on the 3D-space and project them on our screen as looked from a point in the right side looking to a point a little bit above the center of the nose. We clip the segments that are on the far side to be consistent with the fact that my nose is not transparent.
In[113]:=
Out[113]=
This is it! We write the figure on a file.
In[114]:=
Out[114]=
Using an OpenSource picture manipulating program (the GIMP), we blend the two images and get the final result. Now you can learn how to assess your professor's alertness by locating the position of his glasses on the scale on his nose.
In[115]:=
Out[115]=
| Created by Wolfram Mathematica 6.0 (26 July 2007) |  |