Overview

DeltaRealTime merges the high-fidelity playback of Delta Media server with real-time interactive assets (e.g. from Unity, or Unreal Engine) to create real-time high fidelity, interactive 3D scenes, whether for a game, exhibit, simulation environment or themed attraction.

DeltaRealTime achieves this by matching camera definitions for pre-rendered playback media and real-time frames, and then compositing pre-rendered and real-time content based on depth and mathematical definition (stencil) information, on a per pixel basis.

3D Depth

In order to merge playback and real-time sequences, we compare the depth buffer for each pixel of the real-time asset with that of the playback frame, to determine which is in front. In the basic mode of DeltaRealTime, any pixel of a real-time asset that is not in front, is not rendered, by using the alpha channel. In this way, real-time assets moving about in the playback frame will move in and out, and disappear behind features in the pre-rendered background. Depending on the range of depth values required, the depth buffers can be 8, 16 or 24-bit (see Playback and Real-time Media Formats). The greater the range, the better the effect.

Depth and alpha alone cannot blend the two layers and the visual result depends entirely on the properties and quality of the RGB playback and real-time imagery (see Basic (No Stencil) Mode).

Stencil feed

In the full implementation mode of DeltaRealTime, the Delta compositor also takes account of a stencil feed, which provides further data in RGB channels to modify each pixel. If depth buffer comparison shows that the real-time pixel is in front, then the stencil is used to re-evaluate the real-time pixel: is it opaque, semi-transparent, or does it shade or brighten the playback pixel behind it? It could be any combination, depending on the R, G and B values in the stencil pixel. This gives the means of truly blending hi-fidelity interactive displays, using the power of Delta Media Server and the Delta compositing engine (see How the Stencil Works).

Terminology

The following terms are used in this guide:

Playback sequence
Pre-rendered colour image media.

Real-time feed
Interactive gaming assets, to be introduced into the playback media (normally input into Delta through a capture card or Spout shared texture).

Depth sequence
A separate pixel-matched media sequence, carrying depth values for each pixel of the corresponding colour media. In this document, illustrations are portrayed as 8-bit depth, in greyscale from white (nearest) to black (deepest).

Stencil feed
Matched to the pixels of the real-time feed, per-pixel data values are carried in the three colour channels: alpha (in blue), semi-transparency (in green) and illumination (in red).

Culling
Where a stencil is not used, and a pixel in a real-time asset is (by depth comparison) to be occluded by the background, the real-time image pixel does not need to be rendered, so is simply ‘culled’, thus reducing processing load.