Render state

In this guide we will dive more into RenderState and how to modify it.

Image description

One way to think of render state is as instruction on how to produce a certain output image. The same render state should produce the same image. Being a vanilla javascript object, render state is also serializable. This should allow you to reproduce the same image on another computer or another time. It can also be useful for unit testing, since mocking and validating state objects are a lot easier than images.

Context

The render state does not live in a vacuum, however. It has to be rendered on a physical device, whose browser, GPU and drivers might deviate slightly in how pixels are rendered. Different versions of our API may also interpret it differently, again producing differences. Also, the render state does not specify the full geometry of your scene. Sometimes it merely points to where geometry can be streamed from. The process of downloading/streaming will happen over time, producing several frames of progressive refinement with varying level of detail. The final quality of the image is also constrained by the limitations of your device, described in DeviceProfile.

Baseline

Each view starts with a default render state, defaultRenderState. This render state will produce nothing but a basic gray gradient background and the Novorender watermark. Details of all the default values can be viewed in the source code linked from the reference documentation.

Immutability

A common problem in javascript (and programming in general) is understanding how state should be modified. Let's consider a hypothetical example, loosely inspired by three.js:

interface Object3D {
   vec3 position;
   quat rotation;
   ...
}

How do we change the position of this object in a way that will actually be rendered as expected?

Option 1: In-place mutation

vec3.copy(obj.position, [x, y, z]);

Option 2: Property mutation

obj.position = [x, y, z];

Option 3: Read-copy-update

obj = { ...obj, position: [x, y, z] };

In the first two options, it's not obvious to the rendering engine that something has changed, much less what has changed. Quite often this results in additional method calls to notify the engine that recomputation are required.

obj.updateWorldMatrix(); // Do I have to call this, and if so, when? Why aren't my updates reflected in the view?

Using typescript's readonly keyword to enforce immutability on this hypothetical example, it becomes obvious that only option 3 is possible, since the other two produces compile time errors.

interface Object3D {
   readonly ReadonlyVec3 position;
   readonly ReadonlyQuat rotation;
   ...
}

Option 3 also happens to be the one that makes it easy for the engine to detect that this object has indeed changed and needs to be reevaluated.

// lurking deep inside the 3D engine somewhere...
if (prevObj !== newObj) {
    // update internal and GPU state.
}

Yes, copying is slower than mutation, but that cost is easily recovered by the efficiency of basic reference comparisons to detect changes. Immutable style of coding may feel a bit awkward and alien to you at first, but it significantly reduces to potential for mistakes. Not to mention the simplicity of understanding, use, implementation and a host of other benefits.

How to modify render state.

Now that we've discussed the why, let's see how render state should be changed. Each view has a copy of the render state of the previously rendered frame View.prevRenderState. To save energy and battery life, it will not re-render unless the state changes. We essentially tell it to re-render by assigning a brand new render state object.

However, there's no need to copy sub state that hasn't changed. In fact, doing so would force the update of internal state unnecessarily, which can be a costly affair. So, for the sake of efficient and correctness, it's important that we reuse all the sub objects that hasn't changed! This is a common paradigm in functional programming languages, such as Haskell and Elm. Unfortunately, javascript is not a particularly functional language, so we end up with something like this:

const newState = {
    ...prevState, // preserve all the state sub-objects that we're not changing.
    background: {
        ...background, // preserve other background state.
        color: [0, 0, 0.1, 1], // introduce changes as new objects/copies.
    },
};

tip

See MDN for more details on the ... spread syntax!

To simplify matters and help encourage correct update of render state, we made the View.modifyRenderState function.

view.modifyRenderState({ background: { color: [0, 0, 0.1, 1] } });

This function will recursively traverse the changes argument, which is a partial render state object, and apply to a new copy of the current render state. You can also combine several changes into a single call.

// multiple changes in single call.
view.modifyRenderState({
    background: { color: [0, 0, 0.1, 1] },
    tonemapping: { exposure: 1 },
});

modifyRenderState() does not instantly trigger a re-render and can safely be called multiple times.

Detecting changes

As part of diagnostics or exploring, you can look for changes by comparing View.renderState and View.prevRenderState.

const { renderState, prevRenderState } = view;
const hasBackgroundChanged = renderState.background !== prevRenderState?.background;
const hasBackgroundColorChanged = renderState.background.color !== prevRenderState?.background?.color;
const hasBackgroundColorRedChanged = renderState.background.color[0] !== prevRenderState?.background?.color?.[0];

Due to the way immutable changes trickle up to the parent object, hasBackgroundChanged will be true for any changes to the background state, including color.

Note that the state is not actually updated until just prior to a frame being rendered. You can override or assign the View.render method to inspect the actual state that just got rendered.

Validating render state.

For performance reasons, render state validation is normally not performed before rendering. You can perform validation explicitly, however.

// ...apply state changes above
const errors = view.validateRenderState();
for (const error of errors) {
    console.warn(error);
}

This validation will only check state changes since the last rendered frame. It will perform basic range checks and values that could result in a run-time exception, such as making sure expected integers are indeed integers.

Some examples will do validation for you and report back validation errors, like the one below:

Change the grid.distance to 100 instead of -100 to rectify the problem.

Pitfalls and performance issues.

Immutability comes at a cost, which may not always be trivial. This is especially true of large arrays, such as RenderStateHighlightGroup.objectIds. The cost of copying such arrays may be considerable, both in terms of memory bandwidth, CPU time and garbage collection pressure. Also, recall that the engine retains the previous copy of the render state for reference. In an iteractive session, you may want to use a temporary group for frequent changes of highlights and merge into the main group at the end.

Dynamic objects are described in their entirety by render state. With the exception of typed arrays, javascript does not have a very efficient memory layout for such data. Typed arrays are not easy to work with, however. For this reason, most of the dynamic objects render state is defined by vanilla JS objects, despite the extra memory overhead.

Also keep in mind that this data is then copied into memory that is GPU accessible, nearly doubling the memory footprint. In the future, we may offer an option to "freeze" objects and thus remove the CPU copy. For now, extra care should be taken to keep dynamic geometry at modest complexity and memory footprint, particularly on mobile devices.

The hardest problem to spot may be where you inadvertently create an unnecessary copy of state that hasn't changed. Not only can this be costly in and of itself, but it forces the engine to update its internal state as well. Worse yet, it will cause the entire frame to be re-rendered, even when there's no visible changes. On a workstation, this is not a big problem, but anything that runs on a battery will suffer.

To help diagnose duplicate render state problems, you may want to include an optional frame counter in your app.

Tips and tricks

Sadly, immutability is not the default in javascript/typescript. To help your team avoid accidental in-place mutation, you should use the typescript readonly keyword whenever possible, preferably on both sides of array, like illustrated below.

interface MyObject {
    readonly items: readonly number[];
}

We strive to do this everywhere, except where the intention is explicitly that of mutation. When mutation is possible, such as on camera controller properties, it's usually backed up by some logic to handle it gracefully.

You may find typescript const assertions useful for making members readonly implicitly.

The new javascript copying array methods, such as with(), toSpliced() and toSorted() can be useful for manipulating arrays in a non-mutating way.

The already mentioned spread operator ... is also an invaluable tool to create modified copies in a safe and readable manner, both for arrays and objects.

We've also exposed the mergeRecursive function that we use to merge partial state changes internally.