VibeOS TechnologiesResearch & Development

Technical overview / VibeVR

Document: VT-VR-01Revision: 1.0 Implemented architecture

VibeVR Generative VR Architecture

VR Device turns a spoken design request into a structured, explodable object and presents it through a procedural Three.js scene on desktop or in WebXR.

Primary interface
Voice-led VR Device
Model boundary
OpenAI speech / Gemini generation
Render runtime
Three.js / WebGL / WebXR
Persistence
Current session only
01

AI Integration Architecture

Voice, model, procedural rendering, and immersive runtime boundaries

VibeVR AI integration architecture Implemented architecture

VR Device

Voice-led workspace for requesting, viewing, and inspecting generated holographic objects

Voice requestLive transcriptDesktop viewImmersive VR

Orchestration & Translation Layer

Transcribes intent, grounds the generation contract, parses structured output, and mediates the render boundary

Speech transcriptionPrompt contractJSON parsingRender gate

AI Model

Produces a structured multi-part object specification from the transcribed design request

GeminiOpenAI transcriptionApproved model endpoints

Procedural 3D Runtime

Builds primitive geometry and physical materials, then mounts the object in the live scene

Three.jsWebGLPhysical materialsExploded view

WebXR Interaction

Presents the current in-memory object through desktop controls or an immersive headset session

Controller raysRotate & zoomXR calibrationSession state
Runtime boundary. The model supplies a structured object proposal. The application owns render eligibility, procedural scene construction, and the active WebXR session.
Figure 1 — VibeVR implementation of the AI Integration Architecture. Model output crosses parsing and render gates before it becomes live scene geometry.
02

Input, model & runtime services

Implemented components behind VR Device

Browser audio capture

Explicit microphone permission and a press-and-hold interaction capture the user's spoken object request as WebM audio.

MediaRecorder / audio-webm

Speech transcription

A server boundary sends the recording to an approved OpenAI speech model and returns plain transcript text.

No browser-held service credential

AI model

Gemini receives the bounded design contract and produces the current implementation's structured multi-part object specification.

Approved model boundary

Object specification

The generated contract describes 30–60 named parts using a closed set of primitives, metric transforms, colors, and physical material properties.

JSON-only generation contract

Three.js runtime

Accepted parts become procedural geometry and MeshPhysicalMaterial instances in a continuously rendered WebGL scene.

No imported 3D asset required

WebXR session

The same active object can be inspected on desktop or through an immersive VR session with controllers and calibrated headset height.

Immersive-vr / local-floor

03

Object generation lifecycle

From spoken intent to interactive hologram

  1. 01

    Capture

    The user grants microphone access, holds the voice control, and describes the object. Releasing the control completes an audio/webm recording.

  2. 02

    Transcribe

    The client uploads the recording to a protected transcription route. Empty or undersized input exits without starting object generation.

  3. 03

    Constrain

    Fixed instructions define the allowed primitives, required part count, transforms, colors, physical material fields, and JSON response shape.

  4. 04

    Generate

    Gemini converts the transcript into a named, explodable object specification containing 30–60 individual parts.

  5. 05

    Parse

    The response passes through direct JSON parsing and an object-extraction fallback, then must expose a non-empty parts array to continue.

  6. 06

    Construct

    Each part maps to supported Three.js geometry and a physical material whose metalness, roughness, and opacity are bounded by application code.

  7. 07

    Mount

    The previous hologram is disposed, the new object group is mounted at the desk focal anchor, and rotation and zoom state reset.

  8. 08

    Inspect

    The render loop supports auto-spin, rotation, zoom-driven exploded inspection, preset selection, controller rays, and immersive VR.

04

Hologram generation network

Detailed flow inside the VibeVR orchestration and rendering path

Hologram generation network Voice-to-XR execution

Microphone Permission & Voice Intent

The user explicitly grants microphone access and holds the voice control while describing the object to generate

getUserMediaPress and holdVoice-only prompt

WebM Audio Capture

MediaRecorder collects browser audio and rejects a recording that is too small to contain a usable request

audio/webmRelease to sendMinimum blob gate

Speech Transcription

A protected server boundary sends the recording to an approved speech model and returns transcript text

Multipart uploadSpeech modelTranscript response

Transcript Gate

The interface exposes the transcript and stops before generation when the returned text is empty

Visible transcriptTrim checkEarly exit

Generation Contract Assembly

Fixed instructions convert the request into a JSON-only contract for an explodable engineering object with 30–60 parts

Closed shape setMetric transformsMaterial fields30–60 parts

Structured Object Generation

The AI model returns a named object specification containing primitive geometry, transforms, color, and surface properties

GeminiJSON objectPart specification

Parse & Render Eligibility Gate

The response is parsed with a JSON fallback, then accepted only when it contains a non-empty parts array

JSON.parseObject extractionNon-empty parts

Geometry & Material Construction

Each accepted part becomes supported Three.js geometry with bounded physical material values

Six primitivesMeshPhysicalMaterialClamped surfaces

Scene Replacement

The previous object is disposed and the generated group is mounted above the hologram desk focal anchor

Dispose previousMount groupReset pose

Desktop / WebXR Inspection

The render loop supports auto-spin, rotation, zoom, exploded inspection, controller selection, and immersive headset calibration

WebGL render loopExploded viewXR controllersIn-memory session
Runtime boundary. Generated specifications exist only in the current browser session. The three shelf examples are deterministic presets, not model-generated history.
Figure 2 — Verified voice-to-XR execution path. Generated JSON is a procedural construction contract, not an imported 3D asset.
05

Responsibility & trust boundaries

Authority by implementation component

VR Device

Responsible for

Microphone permission, recording state, transcript display, generation status, current object name, and interaction controls

Cannot do directly

Does not hold model credentials or create arbitrary geometry outside the accepted contract

Transcription model

Responsible for

Converting the submitted WebM recording into transcript text

Cannot do directly

Does not decide object structure, materials, or scene behavior

Generation model

Responsible for

Proposing the named parts, primitive shapes, transforms, colors, and material values for the requested object

Cannot do directly

Cannot mount meshes, enter WebXR, or persist a generated object

Orchestration & translation

Responsible for

Prompt constraints, model mediation, JSON parsing, and the minimum parts-array eligibility check

Cannot do directly

Does not provide full field-by-field schema validation in the current implementation

Three.js / WebXR runtime

Responsible for

Geometry construction, material bounds, scene replacement, animation, controllers, and the active immersive session

Cannot do directly

Does not save generated specifications or treat shelf presets as generated history

06

Generation & runtime controls

Implemented safeguards around voice, model output, and WebXR

Explicit input permission

Audio capture starts only after browser microphone permission and a direct press-and-hold action.

User-controlled recording

Early input gates

Very small recordings and empty transcripts stop before object generation, preserving the current scene.

No prompt on unusable input

Closed geometry vocabulary

The generation contract limits output to sphere, cylinder, box, torus, cone, and ring primitives.

Predictable procedural construction

Material bounds

Metalness, roughness, and opacity are clamped when materials are built, even when generated values exceed expected ranges.

Client-side numeric safeguards

Render eligibility

A parsed response must contain at least one part before it can replace the currently displayed hologram.

Previous object retained on rejection

XR capability fallback

WebXR availability and session startup are checked independently; desktop rendering remains the base presentation mode.

Immersive mode is optional

Failure behavior: microphone denial, recording failure, transcription or generation errors, empty transcripts, malformed model output, empty part arrays, and WebXR startup failure surface an error or stop the path while preserving the current scene where possible.
07

Runtime state & persistence

What exists during and after a generation session

Current transcript

The latest returned speech text is displayed in the interface for the active page session

Current object spec

The accepted generated specification exists in memory and drives the currently mounted Three.js object

Current render object

Constructed meshes, materials, transforms, rotation, zoom, and exploded-view state live in the active scene

Preset catalog

Arc Reactor, Repulsor Gauntlet, and Jet Turbine are deterministic specifications bundled with the application

Generated history

Not implemented: generated objects are not added to the shelf, database, browser storage, or a saved-scene library

Reload behavior

A reload or preset switch discards the generated object unless the user generates it again