VR Renderer

Overview

The VR renderer records stereo Vulkan rendering for OpenXR. Visibility and draw command generation run on GPU buffers.

Scene geometry is stored in shared GPU buffers. Per-frame CPU work updates transforms, frustum data, and command recording.

OpenXR Integration

The VR renderer integrates with OpenXR. OpenXR provides:

• Runtime Abstraction: VR runtime selection through OpenXR
• Input Management: VR controllers, hand tracking, and spatial anchors
• Compositor Integration: Swapchain image submission to the VR compositor
• Session Management: Session initialization, frame timing, and cleanup

The OpenXR integration handles:

• View Configuration: Manages stereo rendering for left/right eyes with proper field-of-view and projection matrices
• Frame Synchronization: Coordinates with VR runtime for frame pacing and presentation timing
• Space Management: Handles reference spaces (local, stage, view) for proper object positioning
• Action System: Provides input abstraction for controller interactions

Rendering Pipeline Architecture

The recorded mesh path uses primitive culling, optional parallel LOD selection, meshlet binning allocation, count dispatch, meshlet unpacking, and draw preparation. MeshletCulling resources are still created, but the recorded mesh path skips the meshlet culling dispatch.

Pipeline Stages

Shader Stage Details

1. Primitive Culling (Compute Shader)

Purpose: Routes visible primitives to the mesh shader path, vertex shader path, or LOD path.

Inputs:

• Local bounds and per-object transformation matrices
• Dual-eye frustum planes (left/right eye view frustums)
• Active instance IDs

Outputs:

• culling_survivors[] raw primitive IDs for the mesh shader path
• vs_visible_instances[] raw primitive IDs for the vertex shader path
• lod_visible_primitives[] visible LOD primitive records
• Atomic counters for each output stream

Algorithm:

• Computes world-space bounds from local bounds and world matrices.
• Tests each primitive's bounding sphere against both eye frustums.
• Writes raw primitive IDs or visible LOD primitive records; pipeline binning runs later in MeshletBinningAllocator.comp.

2. Optional LOD Prefix Sum and Cluster Selection (Compute Shaders)

Purpose: Selects visible LOD clusters for primitives with cluster LOD data.

Inputs:

• lod_visible_primitives[] from primitive culling
• Cluster LOD data and cluster group data
• LOD configuration and per-object transforms

Outputs:

• lod_cluster_survivors[]
• lod_cluster_survivor_count

Algorithm:

• ClusterPrefixSum.comp computes prefix-sum offsets and an indirect dispatch.
• ClusterLodSelection.comp runs one thread per candidate cluster.
• Selected clusters feed meshlet binning as single-meshlet allocations.

3. Meshlet Binning Allocation (Compute Shader)

Purpose: Allocates per-pipeline meshlet ranges.

Inputs:

• culling_survivors[] and cull_count
• lod_cluster_survivors[] and lod_cluster_survivor_count
• Primitive meshlet metadata
• Cluster LOD data

Outputs:

• allocations[]
• pipeline_meshlet_counts[]

Algorithm:

• Processes regular primitive survivors and LOD cluster survivors.
• Uses subgroup operations to reduce per-pipeline atomic contention.
• Writes allocation records for meshlet unpacking.

4. Count Dispatch (Compute Shader)

Purpose: Converts survivor counts into a dispatch command for meshlet unpacking.

Inputs:

• cull_count
• lod_cluster_survivor_count

Outputs:

• Indirect dispatch buffer for MeshletUnpacking.comp

Algorithm:

• Computes workgroup count for survivor_count + lod_cluster_survivor_count.

5. Meshlet Unpacking (Compute Shader)

Purpose: Expands allocation records into visible meshlet records.

Inputs:

• Allocation records from meshlet binning
• Survivor counters

Outputs:

• binned_visible_meshlet_info[] organized by pipeline bins

Algorithm:

• Runs from an indirect dispatch produced by Count Dispatch.
• Writes VisibleMeshletInfo records consumed by the mesh shader.

6. Draw Command Preparation (Compute Shader)

Purpose: Generates GPU-executable draw commands from visibility data

Inputs:

• Per-pipeline meshlet counts from meshlet binning

Outputs:

• Indirect draw command buffer for mesh shader dispatch

Algorithm:

• Converts meshlet counts into vkCmdDrawMeshTasksIndirectEXT commands
• Dispatches one workgroup; shader specialization runs one invocation per pipeline.

7. Mesh Shader Assembly (Graphics Shader)

Purpose: Transforms meshlets into rasterizable geometry

Inputs:

• Unified vertex buffer (all scene vertices)
• Unified meshlet buffer (vertex/triangle metadata)
• Unified index buffer (triangle connectivity)
• Per-object transformation matrices
• Material and texture binding data

Outputs:

• Transformed vertices in clip space
• Texture coordinates and normals
• Primitive assembly for rasterization

Algorithm:

• Each mesh shader workgroup processes one meshlet
• Fetches vertex data from unified buffers using meshlet offsets
• Applies object transformations and generates triangle primitives
• Outputs vertex attributes for fragment shading

8. Fragment Shading (Graphics Shader)

Purpose: Computes final pixel colors and applies materials

Inputs:

• Interpolated vertex data from mesh shader
• Material properties and texture samplers
• Lighting information (when applicable)

Outputs:

• Final color values for each pixel
• Depth values for depth testing

Variants:

• Flat Color: Simple uniform color application
• Textured: Samples from texture arrays using UV coordinates
• Lightmapped: Combines albedo textures with precomputed lighting (WIP)

Buffer Architecture

Static Geometry Buffers

Created once at scene load - Never modified during rendering:

• VertexBuffer: All vertex positions, normals, UVs in a single buffer
• MeshletBuffer: Per-meshlet metadata (counts, offsets, material indices)
• MeshletTriangleBuffer: Triangle index data for all meshlets
• MeshBuffer: Mesh-level organization (which meshlets belong to which mesh)
• MeshPrimitiveBuffer: Material and texture assignments per primitive
• TextureArray: Bindless texture array for all scene textures

Dynamic Per-Frame Buffers

Updated each frame by CPU:

• PerObjectSSBO: Object world matrices, color multipliers, visibility flags
• FrustumBuffer: Current frustum planes for both VR eyes
• ViewProjectionBuffer: Combined view-projection matrices per eye

GPU-Generated Intermediate Buffers

Written by compute shaders during pipeline execution:

• BinnedVisibleMeshletIndexBuffer: Visible meshlets organized by pipeline
• MeshletCounterBuffer: Atomic counters tracking visible meshlets per pipeline
• IndirectDrawBuffer: Final draw commands for mesh shader dispatch

Data Flow Summary

1. CPU Preparation: Updates per-frame matrices and frustums
2. GPU Culling: Primitive visibility and path routing
3. LOD Selection: Optional prefix sum and cluster selection
4. Binning and Unpacking: Pipeline bins and visible meshlet records
5. Command Generation: Convert per-pipeline counts to draw commands
6. Geometry Assembly: Mesh shaders transform meshlets to triangles
7. Pixel Shading: Fragment shaders compute final colors
8. Compositing: OpenXR presents stereo images

See GPU-Driven Rendering Pipeline for the stage-level buffer flow.