Interface Frame

A single frame, as seen by the camera. This is backed by a C++ HostObject wrapping the native GPU buffer. At a 4k resolution, a raw Frame can be 12MB in size.

Example

const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  console.log(`Frame: ${frame.width}x${frame.height} (${frame.pixelFormat})`)
}, [])
interface Frame {
    bytesPerRow: number;
    height: number;
    isMirrored: boolean;
    isValid: boolean;
    orientation: Orientation;
    pixelFormat: PixelFormat;
    planesCount: number;
    timestamp: number;
    width: number;
    getNativeBuffer(): NativeBuffer;
    toArrayBuffer(): ArrayBuffer;
    toString(): string;
}

Properties

bytesPerRow height isMirrored isValid orientation pixelFormat planesCount timestamp width

Methods

getNativeBuffer toArrayBuffer toString

Properties

Readonly bytesPerRow

bytesPerRow: number

Returns the amount of bytes per row.

Readonly height

height: number

Returns the height of the frame, in pixels.

Readonly isMirrored

isMirrored: boolean

Returns whether the Frame is mirrored (selfie camera) or not.

Readonly isValid

isValid: boolean

Whether the underlying buffer is still valid or not. A Frame is valid as long as your Frame Processor (or a runAsync(..) operation) is still running

Readonly orientation

orientation: Orientation

Represents the orientation of the Frame, relative of what the desired output orientation is.

For example, if the phone is held in 'portrait' mode and the Frame's orientation is 'landscape-left', it is 90° rotated relative to the phone's rotation.

To make the frame appear up-right, one would need to counter-rotate it by 90°. Such counter-rotations should not actually rotate pixels in the buffers, but instead be handled via flags or transforms to avoid any performance overheads.

For example in MLKit, the caller just needs to pass the Frame's orientation to it's detect(...) function and it will interpret buffers in that target orientation.

See

See "Orientation"

Readonly pixelFormat

pixelFormat: PixelFormat

Represents the pixel-format of the Frame.

Readonly planesCount

planesCount: number

Returns the number of planes this frame contains.

Readonly timestamp

timestamp: number

Returns the timestamp of the Frame relative to the host sytem's clock.

Readonly width

width: number

Returns the width of the frame, in pixels.

Methods

getNativeBuffer

  • getNativeBuffer(): NativeBuffer

  • Get the native platform buffer of the Frame.

    • On Android, this is a AHardwareBuffer*
    • On iOS, this is a CVPixelBufferRef

    The native buffer needs to be manually deleted using NativeBuffer.delete(), and this Frame needs to be kept alive as long as-, or longer than the NativeBuffer.

    Returns NativeBuffer

toArrayBuffer

  • toArrayBuffer(): ArrayBuffer

  • Get the underlying data of the Frame as a uint8 array buffer.

    The format of the buffer depends on the Frame's pixelFormat.

    Note that Frames are allocated on the GPU, so calling toArrayBuffer() will copy from the GPU to the CPU.

    Returns ArrayBuffer

    Example

    const frameProcessor = useFrameProcessor((frame) => {
      'worklet'

      if (frame.pixelFormat === 'rgb') {
        const buffer = frame.toArrayBuffer()
        const data = new Uint8Array(buffer)
        console.log(`Pixel at 0,0: RGB(${data[0]}, ${data[1]}, ${data[2]})`)
      }
    }, [])

toString

  • toString(): string

  • Returns a string representation of the frame.

    Returns string

    Example

    console.log(frame.toString()) // -> "3840 x 2160 Frame"

On This Page