Immersion requires a large field of view. This could be achieved by putting a large curved spherical display on your face, but alas such technology is prohibitively expensive. A more affordable solution to increasing the field of view is to look at small ubiquitous rectangular displays through lenses:

Lenses placed close to your eyes greatly increase your field of view, but there is a cost: the image becomes spherically distorted. The larger the field of view, the more distorted the image. This post is a quick summary of three different approaches to undistorting the image, all of which have been implemented in JavaScript for various WebVR-related projects.

Here is a closer look at the lens distortion of a typical head mounted display. The lenses cause a pincushion effect:

The solution is to apply barrel distortion to the image. When we look at it through the distorting lenses, the image looks neutral:

Lens distortion is well understood mathematically, governed by equations like these, with distortion coefficients corresponding to the particular lens. To undo the distortion properly, we also need to calculate the centers of the eyes, which requires knowing a bit about the geometry of the display and the enclosure itself. This can all be done, even on the web! I summarize a few implementation options below.

1. Fragment based solution (bad)

The simplest way to using two pass rendering. First, we render the left and right eyes onto a texture, and then process that texture with a fragment (pixel) shader, moving each pixel inward in relation to the centroid of the eye:

This is the first and simplest method, which is also the least efficient, since each pixel is processed separately. The first version of the WebVR Boilerplate implemented this method.

2. Mesh based solution (better)

Rather than processing each pixel separately, we distort the vertices of a relatively sparse mesh (40x20 works well).

This can save some direct computation and let the GPU do a fair amount of interpolation. Rather than having to apply to every single pixel (1920 * 1080 ~= 2e6), we do the calculation for every vertex in the mesh (40 * 20 = 800). The result is a significant reduction (3 magnitudes or so) of computation, and a nice boost in performance. The WebVR Polyfill currently implements this approach.

Applying distortion isn't the only expensive part in this rendering method. A lot of time is wasted copying the whole scene to an intermediate texture.

3. Vertex displacement based solution (best)

This brings us to the most efficient method of the three, which eliminates the need to render to an intermediate texture in the first place. In this approach, the geometry itself is distorted using a custom vertex shader. The idea is that knowing the position of the camera, we can displace vertices in such a way that the resulting 2D render is already barrel distorted. In this case, no shader pass is needed, and we save the expensive step of copying the rendering into a texture.

This method does require a certain vertex density on every mesh that is being deformed. Imagine the simple case of a large, 4-vertex rectangle being rendered very close to the camera. Distorting these vertices would still yield a 4-vertex flat rectangle, and clearly there's no barreling effect. Because of this, this is method does not generalize without extra work on the developer's part.

This approach is used in the Cardboard Design Lab and in the open sourced VR View project. Geometry-based distortion can also result in sharper looking renderings, since the two pass approach can cause aliasing, especially if the intermediate texture is small. You can read more about this distortion method in this helpful explainer.