tomkrikorian/AGENTS.MD

## AGENTS.MD

      
    Raw
  

              AGENTS.MD
            
          
    AGENTS.MD – Meta Wearables Device Access Toolkit (DAT) iOS 0.2


Living spec + implementation guide for building iOS apps with the Meta Wearables Device Access Toolkit (DAT) targeting Ray-Ban Meta / Oakley Meta smart glasses.


0. Goals of this document

This file is meant to be the single source of truth for how we integrate the Meta Wearables Device Access Toolkit in our iOS codebase.
It should help any AI agent / human developer to:

Understand the capabilities and limits of the SDK.
Follow a consistent architecture (registration → permissions → device selection → sessions → streaming / photo / audio).
Reuse common patterns (session management, error handling, device selectors, photo capture, etc.).
Quickly find API details (types, enums, methods) without re-opening the docs.

Whenever the SDK changes, update this file first.

1. High-level overview

1.1 What DAT does

The Wearables Device Access Toolkit (DAT) lets an iOS app connect to supported Meta smart glasses and use them as remote sensors & outputs:

Camera: live video streaming + still photo capture.
Microphones: voice input via glasses mic over Bluetooth.
Speakers: audio playback through glasses speakers.

All application logic runs on the phone. Glasses provide sensor data and audio I/O; there is no app runtime on the glasses.
1.2 Supported devices (currently)


Ray-Ban Meta (Gen 1)
Ray-Ban Meta (Gen 2)
Oakley Meta HSTN

Future devices may be added; the SDK exposes DeviceType / compatibility utilities.
1.3 Core concepts


Registration: one-time handshake between our app and the Meta AI companion app. After registration, our app appears in the Meta AI "App Connections" list.
Permissions: camera permission is granted per-app but confirmed per-device through Meta AI. Microphone/speaker access is handled via iOS audio + Bluetooth permissions.
Devices: AI glasses linked to the Meta AI app and visible through DAT.
Device selector: object deciding which device a given session targets (auto or specific).
Sessions:

StreamSession: media session for live video + optional photo capture.
DeviceStateSession: session focused on tracking device state.


Publishers / async streams: provide state updates, frames, errors, photo data, registration state, device lists.


2. iOS integration checklist

2.1 Prerequisites


Xcode project with a registered bundle identifier.
Access to Meta Wearables DAT GitHub repo (meta-wearables-dat-ios).
Meta AI companion app installed on device and compatible glasses paired.

2.2 Info.plist configuration

We must configure:

Custom URL scheme for callbacks from Meta AI.
Query scheme to allow fb-viewapp.
External accessory protocol for Meta wearables.
Background modes for Bluetooth / external accessory.
Bluetooth usage description.
MWDAT dictionary for AppLinkURLScheme + MetaAppID.

Template (adjust myexampleapp and MetaAppID):
<!-- Custom URL scheme for callbacks from Meta AI -->
<key>CFBundleURLTypes</key>
<array>
  <dict>
    <key>CFBundleTypeRole</key>
    <string>Editor</string>
    <key>CFBundleURLName</key>
    <string>$(PRODUCT_BUNDLE_IDENTIFIER)</string>
    <key>CFBundleURLSchemes</key>
    <array>
      <string>myexampleapp</string>
    </array>
  </dict>
</array>

<!-- Allow Meta AI (fb-viewapp) to call the app -->
<key>LSApplicationQueriesSchemes</key>
<array>
  <string>fb-viewapp</string>
</array>

<!-- External Accessory protocol for Meta Wearables -->
<key>UISupportedExternalAccessoryProtocols</key>
<array>
  <string>com.meta.ar.wearable</string>
</array>

<!-- Background modes for Bluetooth and external accessories -->
<key>UIBackgroundModes</key>
<array>
  <string>bluetooth-peripheral</string>
  <string>external-accessory</string>
</array>

<key>NSBluetoothAlwaysUsageDescription</key>
<string>Needed to connect to Meta Wearables</string>

<!-- Wearables Device Access Toolkit configuration -->
<key>MWDAT</key>
<dict>
  <key>AppLinkURLScheme</key>
  <string>myexampleapp://</string>
  <key>MetaAppID</key>
  <string>0</string> <!-- 0 for dev mode, real ID for production -->
</dict>

If Info.plist is preprocessed, the :// suffix may get stripped unless the -traditional-cpp flag is used (Apple TN2175).

2.3 Add SDK via Swift Package Manager


Xcode → File > Add Package Dependencies…
URL: https://github.com/facebook/meta-wearables-dat-ios/
Pick a tagged release (e.g. v0.2.0.x).
Add package to all targets that require wearables access.
In Swift files using the SDK, import the specific modules:

import MWDATCore       // Core features, registration, device discovery
import MWDATCamera     // Camera streaming, StreamSession
import MWDATMockDevice // (Debug only) Mock device support
2.4 Configure SDK on launch

Call once during app startup:
func configureWearables() {
  do {
    try Wearables.configure()
  } catch {
    assertionFailure("Failed to configure Wearables SDK: \(error)")
  }
}
2.5 URL handling for Meta AI callbacks

When Meta AI returns to the app (registration, permissions), we must forward the URL to the SDK.
SwiftUI (WindowGroup level):
.onOpenURL { url in
    guard let components = URLComponents(url: url, resolvingAgainstBaseURL: false),
          components.queryItems?.contains(where: { $0.name == "metaWearablesAction" }) == true else {
        return
    }
    
    Task {
        do {
            _ = try await Wearables.shared.handleUrl(url)
        } catch {
            print("Handle URL failed: \(error)")
        }
    }
}
UIKit (AppDelegate / SceneDelegate):
func handleWearablesCallback(url: URL) async throws {
  _ = try await Wearables.shared.handleUrl(url)
}

3. Registration, devices & permissions

3.1 Registration

Registration binds our app to the user’s glasses via Meta AI:

One-time per install / account.
User is deep-linked to Meta AI to confirm, then sent back via callback URL.
After registration, our app appears in Meta AI App Connections list.
User can unregister in Meta AI; we can also start unregistration in-app.

API:
func startRegistration() throws {
  try Wearables.shared.startRegistration()
}

func startUnregistration() throws {
  try Wearables.shared.startUnregistration()
}
Observe registration state asynchronously:
let wearables = Wearables.shared

Task {
  for await state in wearables.registrationStateStream() {
    // Update registration UI (connected / not connected / in progress)
  }
}
3.2 Device discovery

Devices (glasses) are discovered via devicesStream():
Task {
  for await devices in wearables.devicesStream() {
    // devices: [Device]
    // Update our device list UI / selection
  }
}
Device represents a pair of glasses.
Key members:

identifier: DeviceIdentifier — unique ID.
name: String — human-readable device name (may be empty).
nameOrId() -> String — name or identifier fallback.
deviceType() -> DeviceType — e.g. Ray-Ban Meta.
compatibility() -> Compatibility — compatibility with this DAT version.
linkState — connection state.
addCompatibilityListener(_:) — listen for compatibility changes.
addLinkStateListener(_:) — listen for link state changes.

3.3 Permissions

Camera permission is managed by DAT through Meta AI:

Granted at app level but confirmed per device.
Flow is handled in Meta AI, with options like Allow once / Allow always / Deny.

API pattern:
var cameraStatus: PermissionStatus = .denied

cameraStatus = try await wearables.checkPermissionStatus(.camera)

if cameraStatus != .granted {
  cameraStatus = try await wearables.requestPermission(.camera)
}
Notes:

If user denies, we must handle gracefully (disable features, show explanation).
With multiple devices, permission is granted if any linked device has approved.

Microphone / speaker access uses Bluetooth HFP/A2DP and is governed by iOS audio permissions and routing (not a DAT permission type). See audio section.

4. Device selection & state sessions

4.1 Device selectors

All session-based operations (e.g. StreamSession) use a DeviceSelector.
AutoDeviceSelector

final class AutoDeviceSelector: DeviceSelector {
  public init(wearables: WearablesInterface)

  var activeDevice: DeviceIdentifier?

  public func activeDeviceStream() -> AnyAsyncSequence<DeviceIdentifier?>
}
Behavior:

Automatically selects the “best” device.
Picks the first connected device from the devices list.
If no device is connected, falls back to the first device.
activeDeviceStream() updates whenever the device list changes.

Use when we want smart default selection with minimal UI friction.
SpecificDeviceSelector

class SpecificDeviceSelector: DeviceSelector {
  public init(device: DeviceIdentifier)

  var activeDevice: DeviceIdentifier? { get }

  public func activeDeviceStream() -> AnyAsyncSequence<DeviceIdentifier?>
}
Behavior:

Always targets a specific device (by identifier).
activeDeviceStream() yields that device and then completes.

Use when user explicitly selects a device in UI.
4.2 DeviceStateSession

DeviceStateSession monitors state changes for the selected device.
final class DeviceStateSession {
  public init(deviceSelector: DeviceSelector)

  var state: /* device state type – see full docs */

  public func start()
  public func stop()
}
Use this when we need ongoing device state monitoring separate from media streaming.

5. StreamSession – video streaming & photo capture

5.1 Overview

StreamSession manages a media streaming session from glasses:

Live video frames from camera.
On-demand still photo capture during streaming.
Real-time state updates.
Error reporting.

final class StreamSession {

  // Constructors
  public init(deviceSelector: DeviceSelector)
  public init(streamSessionConfig: StreamSessionConfig,
              deviceSelector: DeviceSelector)

  // Properties
  var state: StreamSessionState { get }
  var streamSessionConfig: StreamSessionConfig { get }
  var statePublisher: /* Publisher<StreamSessionState> */ { get }
  var videoFramePublisher: /* Publisher<VideoFrame> */ { get }
  var photoDataPublisher: /* Publisher<PhotoData> */ { get }
  var errorPublisher: /* Publisher<StreamSessionError> */ { get }

  // Methods
  public func start()
  public func stop()
  public func capturePhoto(format: PhotoCaptureFormat) -> Bool
}
New sessions start in .stopped state.
5.2 StreamSessionConfig

Configuration for a streaming session:
struct StreamSessionConfig {

  var videoCodec: VideoCodec
  var resolution: StreamingResolution
  var frameRate: UInt

  public init(videoCodec: VideoCodec,
              resolution: StreamingResolution,
              frameRate: UInt)

  public init() // defaults: raw codec, medium resolution, 30 FPS
}
5.3 StreamingResolution

Valid 9:16 resolutions:
enum StreamingResolution: CaseIterable {
  case high   // 720 x 1280
  case medium // ~504 x 896
  case low    // 360 x 640

  var videoFrameSize: VideoFrameSize { get }
}
Internally the SDK may use additional steps (576x1024, 540x960, 432x768) for laddering.
5.4 VideoCodec

enum VideoCodec {
  case raw // raw decompressed frames
}
Currently only .raw is exposed.
5.5 StreamSessionState

enum StreamSessionState {
  case stopping
  case stopped
  case waitingForDevice
  case starting
  case streaming
  case paused
}
Transitions:

.stopped → .waitingForDevice (no compatible device at start() time).
.stopped → .starting → .streaming (device available).
Any → .stopping → .stopped (on stop() or error).
.streaming ↔ .paused (device / system pauses and resumes session).

5.6 StreamSessionError

enum StreamSessionError: Error, Equatable {
  case internalError
  case deviceNotFound(DeviceIdentifier)
  case deviceNotConnected(DeviceIdentifier)
  case timeout
  case videoStreamingError
  case audioStreamingError
  case permissionDenied
}
Emitted via errorPublisher on various failures (connection, permissions, timeouts, internal issues).
User-Friendly Error Mapping Example:
private func formatStreamingError(_ error: StreamSessionError) -> String {
  switch error {
  case .internalError:
    return "An internal error occurred. Please try again."
  case .deviceNotFound:
    return "Device not found. Please ensure your device is connected."
  case .deviceNotConnected:
    return "Device not connected. Please check your connection and try again."
  case .timeout:
    return "The operation timed out. Please try again."
  case .videoStreamingError:
    return "Video streaming failed. Please try again."
  case .audioStreamingError:
    return "Audio streaming failed. Please try again."
  case .permissionDenied:
    return "Camera permission denied. Please grant permission in Settings."
  @unknown default:
    return "An unknown streaming error occurred."
  }
}
5.7 VideoFrame & VideoFrameSize

struct VideoFrame: Sendable {
  var sampleBuffer: CMSampleBuffer { get } // read-only

  /// Converts to UIImage safely.
  public func makeUIImage() -> sending UIImage?
}

struct VideoFrameSize {
  public init(width: UInt, height: UInt)

  var width: UInt
  var height: UInt
}

Important: sampleBuffer is shared; do not mutate attachments, timing, or pixel buffer. Use makeUIImage() for safe conversion.
5.8 PhotoCaptureFormat & PhotoData

enum PhotoCaptureFormat: Sendable {
  case heic
  case jpeg
}

struct PhotoData: Sendable {
  public init(data: Data, format: PhotoCaptureFormat)

  let data: Data
  let format: PhotoCaptureFormat
}
Usage with StreamSession:

Call capturePhoto(format:) while streaming.
If it returns true, subscribe to photoDataPublisher for PhotoData.
Streaming pauses during capture and resumes automatically after.

5.9 Example: start stream & display frames

let wearables = Wearables.shared
let deviceSelector = AutoDeviceSelector(wearables: wearables)

let config = StreamSessionConfig(
  videoCodec: .raw,
  resolution: .low,
  frameRate: 24
)

let session = StreamSession(
  streamSessionConfig: config,
  deviceSelector: deviceSelector
)

// Track loading state
var hasReceivedFirstFrame = false

let stateToken = session.statePublisher.listen { state in
  Task { @MainActor in
    // Update UI (e.g. show streaming / paused / stopped)
  }
}

let frameToken = session.videoFramePublisher.listen { frame in
  guard let image = frame.makeUIImage() else { return }
  Task { @MainActor in
    // Display the image in a preview surface
    // Set hasReceivedFirstFrame = true
  }
}

Task {
  await session.start()
}
5.10 Example: photo capture

let accepted = session.capturePhoto(format: .jpeg)

if !accepted {
  // No active session, capture already in progress, or capture failed
}

let photoToken = session.photoDataPublisher.listen { photoData in
  let data = photoData.data
  if let image = UIImage(data: data) {
    // Save, share, or process the image
  }
}
5.11 Bandwidth & quality laddering


Streaming uses Bluetooth Classic.
SDK automatically manages bandwidth:

First lowers resolution (High → Medium → Low).
Then reduces frame rate (e.g. 30 → 24 → 15 FPS, never below 15).


Effective image quality may be lower than configured resolution because of adaptive per-frame compression.
Requesting lower resolution and/or frame rate can yield better perceived quality with fewer artifacts.


6. Audio: microphones & speakers

6.1 Bluetooth profiles

Glasses audio uses:

A2DP — high-quality, output-only audio (media).
HFP — two-way audio (mic + speaker) for voice interactions.

DAT shares mic/speaker access with the system Bluetooth stack; we control it via AVAudioSession.
6.2 AVAudioSession configuration (iOS)

Typical setup for HFP:
let audioSession = AVAudioSession.sharedInstance()
try audioSession.setCategory(.playAndRecord,
                             mode: .default,
                             options: [.allowBluetooth])
try audioSession.setActive(true,
                           options: .notifyOthersOnDeactivation)
Notes:

Use .playAndRecord category for input + output.
Combine with .allowBluetooth and/or .allowBluetoothA2DP as appropriate.
Listen for AVAudioSession.routeChangeNotification to handle device connect/disconnect.

6.3 Coordinating audio with streaming

When using HFP and streaming at the same time:

Ensure HFP audio session is fully configured before starting a StreamSession that uses audio.
Avoid fixed-sleep hacks in real code; prefer state-based detection (route change events).

Pseudo:
func startStreamSessionWithAudio() async {
  // 1. Set up HFP
  startAudioSession()

  // 2. Wait for audio route to be ready (replace with real logic)
  try? await Task.sleep(nanoseconds: 2 * NSEC_PER_SEC)

  // 3. Start streaming
  await streamSession.start()
}
6.4 Practical tips


For voice-centric features, favor stable HFP routing; keep video resolution/FPS conservative.
For visual-centric features, be aware that HFP can reduce audio quality; test UX carefully.
Log route changes and StreamSessionError.audioStreamingError for diagnostics.


7. Session lifecycle & device behavior

7.1 Sessions vs transactions


Device sessions (what we use via StreamSession): long-lived, sensor/output access.
Transactions: short, system-owned interactions (notifications, “Hey Meta” triggers) – not controlled by us.

7.2 Device-driven state changes

Session state can change when:

User opens another experience via system gesture.
Another app or system component starts a device session.
Glasses are removed, folded, or moved out of range → Bluetooth disconnect.
User removes our app from Meta AI companion app.
Meta AI ↔ glasses connectivity drops.

We must not guess the cause; just react to state (StreamSessionState, device link state, availability).
7.3 Device availability


Hinge position and wear detection affect connectivity but are not exposed directly.
Key observable effects:

Closing hinges → Bluetooth disconnect, active streams stop, sessions go to .stopped.
Opening hinges → Bluetooth may reconnect, but sessions do not auto-restart. We must explicitly start sessions once device is available and user initiates.


7.4 Lifecycle checklist


Subscribe to registrationStateStream and devicesStream.
Before starting streaming:

Verify registered.
Verify camera permission granted.
Verify at least one compatible device available.


React to StreamSessionState:

.streaming: show active UI; process frames.
.paused: keep connection; pause heavy processing; show paused UI.
.stopped: release resources; show idle UI; allow restart.


Handle StreamSessionError values (especially .permissionDenied, .deviceNotConnected, .deviceNotFound, .timeout).


8. Recommended architecture (for agents & humans)

Use an agent-based modular architecture so both humans and AI tools can work safely.
8.1 Modules / agents


WearablesBootstrapAgent

Calls Wearables.configure() at startup.
Checks environment (Info.plist, MetaAppID).
Sets global flags (dev mode vs production).


RegistrationAgent

Wraps startRegistration(), startUnregistration().
Handles callback URL → handleWearablesCallback(url:).
Exposes a simplified RegistrationState to rest of the app.


DeviceAgent

Wraps devicesStream().
Maintains list of Device objects and selection logic.
Creates AutoDeviceSelector and SpecificDeviceSelector as needed.


PermissionsAgent

Wraps camera permission APIs.
Exposes computed permission state and helper methods:

ensureCameraPermission() → Bool
requireCameraPermission { ... }


SessionAgent

Owns StreamSession instances.
Manages StreamSessionConfig.
Subscribes to statePublisher and errorPublisher.
Provides high-level states: .idle, .starting, .running, .paused, .stopping, .error.


CameraAgent

Subscribes to videoFramePublisher and photoDataPublisher.
Handles conversion to UIImage or CV/ML inputs.
Encapsulates capture flows (trigger, save, share).


AudioAgent

Configures AVAudioSession.
Manages Bluetooth routing, route changes.
Provides high-level methods for starting/stopping audio use.


UIAgent

Connects agents to SwiftUI / UIKit views.
Owns view models modeling the whole lifecycle:

Not registered → needs registration
Registered but no permission
Permission granted but no device
Device ready, session not started
Session active / paused / stopped / error


MockAgent

Uses Mock Device Kit when available.
Simulates devices, streams, errors.
Enables automated tests and offline dev.


8.2 Coding guidelines


Keep direct SDK calls inside agents; UI interacts with our abstractions only.
Prefer async/await and structured concurrency over nested callbacks.
Treat publishers / async streams as authoritative sources of truth.
Always handle StreamSessionError & permission failures gracefully.
Log key events (state changes, errors, route changes) for telemetry & debugging.

8.3 Mock Device Implementation (Debug)

For development without physical devices, MWDATMockDevice allows simulating connection, state, and media.
Setup in Debug Build:
#if DEBUG
import MWDATMockDevice

class MockManager {
    let mockDeviceKit = MockDeviceKit.shared
    var mockDevice: MockDevice?

    func pairMockDevice() {
        // Creates and pairs a simulated Ray-Ban Meta device
        let device = mockDeviceKit.pairRaybanMeta()
        self.mockDevice = device
        
        // Simulate physical state to allow streaming
        device.powerOn()
        device.unfold()
        device.don() // "Put on" the glasses
    }
    
    func setupMockMedia(videoURL: URL, imageURL: URL) async {
        guard let cameraKit = (mockDevice as? MockDisplaylessGlasses)?.getCameraKit() else { return }
        
        // Feed a local video file as the camera stream
        await cameraKit.setCameraFeed(fileURL: videoURL)
        
        // Set the image to be returned when capturePhoto() is called
        await cameraKit.setCapturedImage(fileURL: imageURL)
    }
}
#endif
Mock Controls:

powerOn() / powerOff()
fold() / unfold(): Closing hinges stops the stream.
don() / doff(): Wearing state affects auto-pause behavior.

8.4 Common UI Patterns

Video Rendering in SwiftUI:
struct StreamView: View {
    let videoFrame: UIImage
    
    var body: some View {
        GeometryReader { geometry in
            Image(uiImage: videoFrame)
                .resizable()
                .aspectRatio(contentMode: .fill)
                .frame(width: geometry.size.width, height: geometry.size.height)
                .clipped()
        }
        .edgesIgnoringSafeArea(.all)
    }
}
Loading State:
Use a boolean flag hasReceivedFirstFrame (set to true on the first video frame emission) to toggle between a loading spinner and the video view. This prevents showing a black screen while the connection initializes.

9. Edge cases & pitfalls


Permission denial loop: if user keeps denying camera access, avoid spamming prompts; instead surface instructions to enable permissions in Meta AI.
Multiple devices: don’t assume only one; ensure selectors and UI handle multiple glasses.
Session conflicts: only one session per device; handle failures when another app already has a session.
Bluetooth instability: expect transient disconnects; implement reconnection / recovery strategies.
Capture without streaming: capturePhoto returns false if no active session or capture in progress.
Mutating sampleBuffer: never mutate VideoFrame.sampleBuffer.
Quality expectations: communicate potential quality degradation due to bandwidth to stakeholders/UX.


10. Maintenance rules for AGENTS.MD

Whenever we:

Upgrade the DAT SDK version.
Add new capabilities (codecs, resolutions, devices).
Change our architecture around sessions, permissions, audio.

We must:

Update all relevant API signatures & enums here.
Add sections for any new major features.
Mark removed/changed APIs as deprecated and document migration paths.

This keeps AI agents and human developers aligned on the current integration contract.
No results found