Monday, July 27, 2009

A simple and nice way to video background without OpenGL ES texturing on the iPhone

When we make a mobile augmented reality application, video background is required. As all we know, video background requires tedious jobs. On desktop PCs, we can do it just calling glDrawPixels. But on mobile phones we need to do it through OpenGL ES texture and uploading texture data is quite slow on most mobile phones since they do not have GPUs that are fast enough to update textures in real-time.In AR applications, video background is just for displaying what we see through the camera on the screen. While trying to make video background faster, I found a simple way to do it without OpenGL ES texturing. The idea is using two views, one for video background and another for OpenGL ES rendering.

The Cocoa API allows us to add multiple child views to a window object. What I did was adding two views to our window. One is the 'previewView' of the PLCameraController and another is the OpenGL ES View. The 'previewView' of PLCameraController is a subclass of UIView and it displays video preview coming from the camera.

In the function applicationDidFinishLaunching of the application's AppDelegates.m (whatever..), add the code something like this. What the code does is just adding two views to the window.

PLCameraController *cam = [PLCameraController sharedInstance];
UIView *cam_view = [cam previewView] ;
[cam startPreview];
cam_view.frame = CGRectMake(0,0, 320, 480);

glView = [[EAGLView alloc]
initWithFrame:CGRectMake(0,0,320,480)] ;
glView.opaque = NO ;
glView.alpha = 1.0 ;
[window addSubview:cam_view];
[window addSubview:glView] ;

Then, we need to do one more thing. When you render OpenGL scene, we need to clear the scene with a color that have alpha value of 0.0. By making the OpenGL ES view transparent at the beginning of the rendering process, we will see the live video preview in the background.
So, just change the alpha value of clearing color like this.

glClearColor(0,0,0,0) ;

Run your application, then you will see a live video behind your OpenGL scene like the video below.

The pros and cons of this method are:
  • Pros : You don't need to manage textures or OpenGL ES things for video background. Even you don't need to care background update. Useful for developers who just need video background with OpenGL ES.
  • Cons : The background video seems not to be synchronized with the OpenGL ES. I do not dig on this, but if you do image processing with video (like object tracking), the background video may not be synchronized with your rendering result.
The important thing is performance of this method for video background for augmented reality applications.I tested my method with Oolong Engine on the iPhone 3G (OS 2.2.1). The simple skeleton example of Oolong engine runs at 60 fps as it is (before I add PLCameraController's previewView view).

The rendering speed decreased to about 28 fps after I added video background. It means the rendering became slower and I expect it is due to rendering with full-screen transparency. Note that the fps is not the video stream's framerate, which is just 15 fps. It is OpenGL ES rendering speed. It is much slower than when the video background is not applied, but I think the performance is not bad for AR applications.

I tested this method on the iPhone 3G with 2.2.1, but it may work on iPhone 3GS with 3.0 or higher.

Update (2009. 08. 11)

I tested the same code on iPhone 3GS (OS Ver. is 3.0.1). The rendering speed is improved up to 41 fps, which is +13 fps compared to the old iPhone 3G. I think it is because of the better CPU and Graphics chip. The preview is still 15 fps, which is for still image capturing. If we can get the preview for video capturing we will be able to use 30 fps preview video on 3GS, but the interface is not known until now.


  1. A friend of mine did something similar for me but instead of using a moving image, just put a graphic in place which was viewable depending on the compass readout from the 3GS.

    I'd love to see your app running on a 3GS as well. If Apple can clear up these issues and with the dual-core ARM chip available soon, iPhone v4 will be very exciting for this industry.

  2. Hi, mj.
    My 3GS will come to me in next week. I'm really interested in how fast this method works on 3GS.

  3. So I tried this in 3.1, doesn't work. Do you know how to make it work with the new UIImagepPickerController?

  4. Maybe it'll work with UIImagePickerController.overlay = UIView (think it was something like that)

  5. Could you please share a sample of code for the 3.1 version ? I can't display my glView properly... It is behind the camera view...

  6. Hello Miaou.

    If the glView is behind the preview, maybe you added the glView to the window before adding the preview. To make the glView in front of the camera preview, you have to add the glView to the window after adding the camera's preview view.