Audio Integration Plan: miniaudio + Steam Audio

Overview

Two-layer audio system:

1. miniaudio (v0.11.23) — audio backend: device output, decoding (WAV/FLAC/MP3), mixing, node graph. Single-header C library, public domain/MIT-0.
2. Steam Audio (v4.8.1) — spatial DSP: HRTF binaural rendering, occlusion, reverb, propagation. Apache 2.0.

Steam Audio plugs into miniaudio's node graph as a custom processing node. An official reference exists: engine_steamaudio.c in the miniaudio repo.

Playback Modes

Mode	Path	Use Case
Flat	ma_sound → engine endpoint	UI sounds, music, non-spatial effects
Spatial	ma_sound → SteamAudioBinauralNode → engine endpoint	3D positioned sounds with HRTF

Audio Pipeline Flow

Audio thread (miniaudio callback, ~5ms period):
  Flat:    decode → mix → output
  Spatial: decode → SteamAudioNode.process_pcm_frames() → mix → output
                    ├── deinterleave input
                    ├── iplBinauralEffectApply(direction, HRTF)
                    └── reinterleave output
 
Game thread (Engine::update):
  1. AudioAssetPipeline::tick() — process async loads
  2. Sync listener position from AudioListener + WorldTransform
  3. For each spatial AudioSource + WorldTransform:
     - Compute direction relative to listener
     - Write to SteamAudioNode (lock-free)

Critical Config

Both libraries must agree:

• Sample rate: 48000 Hz
• Frame size: 1024 frames (ma_engine.periodSizeInFrames == IPLAudioSettings.frameSize)
• Format: 32-bit float

Asset Pipeline Integration

Follows the existing TextureAssetPipeline / ModelAssetPipeline pattern in FrameProcessing.cpp.

AudioAssetPipeline (2-stage state machine)

Stage 1: submitLoadAudio(async)
  - Submit AudioLoader::load(path) to thread pool
  - Returns future<AudioData> (decoded PCM: sample rate, channels, frames, float*)
  - Supported formats: WAV, FLAC, MP3 (miniaudio built-in decoders)
 
Stage 2: retrieveAudioData(async)
  - Poll futures with rate limit (maxAudioLoadsPerFrame = 8)
  - On ready: audioAssetManager->add(path, audioAsset)
  - Hook: audioEngine->onAudioLoaded(audioAsset)
    → Creates ma_sound from decoded data
    → If spatial: creates SteamAudio binaural node, wires into node graph
    → Stores handle for playback API

Simpler than textures — no header pre-pass needed, no GPU upload staging. Audio stays in CPU memory.

Streaming (large files)

For music/ambient tracks: don't decode fully, use ma_sound in streaming mode. miniaudio handles streaming internally. Just register the handle, no pipeline stages needed.

File Structure

New Files

Engine/include/Engine/Audio/
├── AudioEngine.h          // Main subsystem: owns ma_engine + IPLContext + IPLHRTF
├── AudioAsset.h           // Asset class: path, sample rate, channels, duration, PCM data
├── AudioAssetManager.h    // Maps path → AudioAsset*, dedup, lookup
├── AudioLoader.h          // Static: load(path) → AudioData (uses ma_decoder)
├── AudioSource.h          // Per-instance playback: wraps ma_sound + optional SteamAudio node
├── SteamAudioNode.h       // Custom ma_node wrapping IPL binaural/direct effects
 
Engine/src/Engine/Audio/
├── AudioEngine.cpp
├── AudioAsset.cpp
├── AudioAssetManager.cpp
├── AudioLoader.cpp
├── AudioSource.cpp
└── SteamAudioNode.cpp

AudioAsset

class AudioAsset : public Asset {
    uint32_t sampleRate;
    uint32_t channels;        // 1 = mono (required for spatial), 2 = stereo
    uint64_t frameCount;
    float durationSeconds;
    std::vector<float> pcmData;  // decoded samples (empty if streaming)
    bool streaming = false;
};

AudioLoader

struct AudioData {
    uint32_t sampleRate;
    uint32_t channels;
    uint64_t frameCount;
    std::vector<float> pcmData;
};
 
class AudioLoader {
public:
    static AudioData load(const std::filesystem::path& path);
    // Uses ma_decoder internally — supports WAV, FLAC, MP3
};

AudioAssetPipeline (in FrameProcessing)

class AudioAssetPipeline {
    struct AudioLoadFuture {
        Asset::Path path;
        std::future<AudioData> future;
    };
 
    std::vector<Asset::Path> assetQueue_;
    std::vector<AudioLoadFuture> loadFutures_;
    AudioAssetManager* audioAssetManager_;
    AudioEngine* audioEngine_;
    int maxAudioLoadsPerFrame_ = 8;
 
public:
    void submitAsset(const Asset::Path& path);
    void tick();  // called each frame, polls futures
};

ECS Components

namespace Ecs {
    struct AudioSource {
        Asset::Path clipPath;
        float volume = 1.0f;
        bool loop = false;
        bool spatial = false;   // false = flat (miniaudio only), true = HRTF (Steam Audio)
        bool autoPlay = false;
    };
 
    struct AudioListener {};  // Tag component on camera/player entity
}

Existing Files to Modify

File	Change
Engine/include/Engine/Ecs/EcsData.h	Add Ecs::AudioSource, Ecs::AudioListener
Engine/include/Engine/Ecs/FrameProcessing.h	Add AudioAssetPipeline class
Engine/src/Engine/Ecs/FrameProcessing.cpp	Implement pipeline (2-stage async)
Engine/include/Engine/Mesh/AssetManager.h	Add AudioAssetManager*, AudioAssetRef, getAsset<Engine::Assets::Audio>()
Engine/src/Engine/Mesh/AssetManager.cpp	Implement audio asset manager integration
Engine/include/Engine/Core/Engine.h	Add AudioEngine* member, createAudioEngine()
Engine/src/Engine/Core/Engine.cpp	Init in run(), update in update(), destroy in cleanup()
Engine/CMakeLists.txt	FetchContent for miniaudio + Steam Audio

CMake Integration

# miniaudio — header only
FetchContent_Declare(miniaudio
    GIT_REPOSITORY https://github.com/mackron/miniaudio.git
    GIT_TAG        0.11.23)
 
# Steam Audio — prebuilt binaries
FetchContent_Declare(steamaudio
    URL https://github.com/ValveSoftware/steam-audio/releases/download/v4.8.1/steamaudio_4.8.1.zip)

Engine Lifecycle Integration

Follows PhysicsEngine pattern:

• Init: After createPhysicsWorld() in Engine::run() — create AudioEngine, init ma_engine + IPLContext + IPLHRTF
• Update: In Engine::update() — pipeline tick, sync listener, update spatial source directions
• Cleanup: In Engine::cleanup() — destroy in reverse order

glTF Extension (optional)

Audio sources on entities via custom extension in glTF:

"extensions": {
    "VULKANSCHNEE_audio": {
        "clip": "sounds/impact.wav",
        "volume": 0.8,
        "spatial": true,
        "loop": false
    }
}

GltfLoader::processModel() would extract these and queue to AudioAssetPipeline::submitAsset(), same as lightmap texture extraction.

Future Extensions (incremental)

Feature	Steam Audio API	Complexity
Occlusion	IPLDirectEffect + scene geometry	Medium
Transmission	Material properties on walls	Medium
Reverb/Reflections	IPLReflectionEffect + baked/real-time sim	High
Ambisonics	IPLAmbisonicsDecodeEffect	Low
Custom HRTFs	SOFA file loading into IPLHRTF	Low