Thursday, March 26, 2009

Natural feature tracking on NVIDIA Tegra

Monday, March 16, 2009

VS2005 project error while opening mobile device projects

When I open my project written on VS2005, one of my *.vcproj file cannot be opened with this error message.




The message says there are duplicated attributes in the *.vcproj file. To solve the problem, open the file *.vcproj file in a text editor, such as Notepad or Wordpad, and find the specified lines.
There may be something like this :



Remove one of them and save *.vcproj file.

I'm not sure why this happens, but it may be a bug in VS2005. Maybe, it does not occur on VS 2008.

Sunday, March 15, 2009

Sekai camera : A new demo

Sekai camera showed one of future augmented reality services for information retrieval from an environment. However, there were no mentions how it works. Here is another sekai camera demo. I think that the information is not attached to a specific object. Instead, the tags flows around the user's position and the user should select one of them to see what the others left there. As such, no recognition method is required. The position of the user is an important cue. 

More details can be found here


Previous demo. 




A new demo. 

Tuesday, March 10, 2009

A tip to make the AR applications on mobile phones faster


When rendering the incoming video frames as a background texture, try to turn off linear interpolation property.

On Direct3D :
D3DMTSS_MAGFILTER and D3DMTSS_MINFILTER

It will increase texture rendering speed much. In my case, using Direct3D, the performance is almost improved by the factor 1.5 after disabling linear interpolation. What we sacrifice for this performance improvement is the rendered texture of a video frame. Since there is no interpolation, the rendered background image will have aliasing effect but it may not a big problem.


Weird texture mapping on Direct3D Mobile

Well, full screen video rendering is necessary on mobile phone-based AR. So, I tried full screen texture mapping with an image file. It is quite simple and works well on emulator as I expected.



However, it shows weird results on my device as shown below. The texture does not occupy the entire screen. It seems that the texture coordinates are wrong. But I used the same code without any changes.



The same code shows different results on the emulator and the device. What is the problem ??
I digged this few hours and finally I found what I did is a little bit wrong when I create texture object.

When I create my texture for background, I wrote the code as below:

D3DMXCreateTexture(m_pd3dmDevice,
MY_IMG_WIDTH,
MY_IMG_HEIGHT,
D3DMX_DEFAULT, D3DMUSAGE_DYNAMIC,
D3DMFMT_R5G6B5,
D3DMPOOL_SYSTEMMEM,
&m_pBGTexture) ;


The parameters MY_IMG_WIDTH and MY_IMG_HEIGHT are 320 and 240, respectively. The function D3DMXCreateTexture automatically change the texture size when the width and height paramters are not 2^n. Thus, my texture's dimensions become (512,256) internally.
When I update the texture data in the rectangle (0,0)~(320,240) area, the remaining region in (512,256) of the texture still have no data (black). Thus, when I draw a quad using texture coordinates from 0 to 1, the entire texture (512,256) is mapped on the quad. Consequently, I see the squeezed image on my device's screen. By changing the texture coordinates of the vertices of the quad, my problem is solved now.

I'm not sure why it works well on the emulator. Maybe the emulator supports rectangular texture dimensions, not 2^n ?

Monday, March 9, 2009

CreateImageSurface failure in Direct3D Mobile

For augmented reality applications, it is required to render real-time video sequences on the background. Usually, we use texture mapping for this on both OpenGL(ES) or Direct3D (mobile).

In Direct3D Mobile (D3DM), there are two choises usually used by users.

1. Lock the texture, copy pixel data from input video, and unlock the texture.
2. Create an image surface where the video is copied. Lock the surface, copy pixel data from input video, unlock the surface, and update the texture with CopyRect method.

When I implement both methods(to choose a faster one), both methods runs well on the emulaltor. However, I encountered several problems on my smart phone (Samsung M480, Windows Mobile 6.0).

At first, lockable texture is not supported by M480. When I checked the capability using this code,

if(caps.SurfaceCaps & D3DMSURFCAPS_LOCKTEXTURE)

the result was false and calling D3DMXCreateTexture function with D3DMUSAGE_LOCKABLE option failed, of course.

Then, I tried the 2nd method, using image surface but the code did not work again. This time, the error occurred when creating surface. I tried to find documents about this problem on the web, but get nothing. After I tried many different parameters, I finally figured out it.

The CreateImageSurface function always fails when I created the image surface with the format D3DMFMT_R8G8B8 on my device, M480. After I changed the image format to D3DMFMT_R5G6B5, which may be natively supported the hardware on M480, the code works well.

We need to check the capability of the hardware we are working on when we want to work with lockable things (such as textures, vertex buffers, or iamge surfaces). Even though the code runs well on the emulator, it does not gaurantee the code works well on the device. Another thing we have to consider is that try another format when your device cannot create an image surface.

I'm not sure how the textures and image surfaces work on the other devices. Does Direct3D Mobile provide the same functionalities on all the devices using WM6, or does the functionalities depend on hardware driver implementations of vendors ??

Sunday, March 8, 2009

Transformation matrices between OpenGL and Direct3D

In these days, I am working on an augmented reality (AR) applications on mobile phones. The mobile phone I'm working on adopted Windows Mobile as its OS. I'd prefer iPhone, but the project manager chose the WM6-based mobile phones. :-(.
Anyway, at first, I started to port conventional OpenGL codes to WM6. As the 3D rendering library, I had two choices, OpenGL ES and Direct3D Mobile.

I tried OpenGL ES, since I could re-use my marker tracking codes I wrote on my desktop. However, the mobile phone I'm using shows very poor performance for AR application. The low performance may be due to the lack of floating point unit and 3D graphics acceleration chip.

Then, I tried Direct3D Mobile (D3DM) samples provided with SDK and D3DM seems to show better performance (faster framerate). Finally, I decided to port my OpenGL codes to D3DM.

I could use most of my codes since they are platform-independent C++ codes. The only problem is using my marker tracking module. The marker tracking module assumes the right-handed coordinate system, but D3DM uses left-handed coordinate system. Of course, Direct3D provides functions for right-handed coordinate system, but D3DM does not have those functions (Some of functionalities are missing in D3DM).

There are many questions and discussions about this problem on communities on the web. However, most of them does not give me a clear answer. After I searched and wrote some equations, I finally found how to change a transformation between OpenGL and Direct3D (well, between right-handed coordinate system to left-handed coordinate system).

The answer is quite simple.

1. From OpenGL Modelview matrix to D3DM view matrix : We just need to transpose the matrix, and change the signs of some elements.



2. From OpenGL projection matrix to D3DM projection matrix : In the same way, we transpose the matrix and change some elements' signs.



Now, we can use your parameters or transformations in both OpenGL and Direct3D.