Skip to content

[Android][OpenGL] Support GL_TEXTURE_EXTERNAL_OES in ExternalTexture #1847

Description

@yonsun-msft

For questions, please use the forum.

Is your feature request related to a problem? Please describe.

Android commonly delivers camera and hardware-decoded video frames through
SurfaceTexture. The underlying OpenGL texture uses the
GL_TEXTURE_EXTERNAL_OES target rather than GL_TEXTURE_2D.

These textures cannot be sampled as ordinary 2D textures:

  • They must be bound using GL_TEXTURE_EXTERNAL_OES.
  • GLES 3 shaders must enable GL_OES_EGL_image_external_essl3.
  • They must use samplerExternalOES.
  • They are sampled-only, single-level textures with restricted sampler state.

BabylonNative's current OpenGL ExternalTexture path cannot describe this
texture kind:

  • Graphics::GL::Texture::Descriptor
    has no texture-target/type field. Its target is currently derived from the
    layer count as either a 2D texture or a 2D-array texture.
  • ExternalTexture
    always creates a bgfx texture through bgfx::createTexture2D.

As a result, an externally owned OES texture cannot be wrapped and sampled
directly by BabylonNative. The application must currently perform an additional
full-frame GPU pass:

SurfaceTexture / MediaCodec / camera
    -> GL_TEXTURE_EXTERNAL_OES
    -> samplerExternalOES full-screen draw
    -> RGBA GL_TEXTURE_2D
    -> BabylonNative ExternalTexture

BabylonNative's own Android
NativeCamera
currently demonstrates this workaround: it samples the OES camera texture into
an RGBA 2D framebuffer texture before making that texture available to bgfx.

Although this avoids CPU readback, it still requires an extra destination
texture and a full-frame render pass for every video frame, adding GPU bandwidth,
synchronization, and latency.

Describe the solution you'd like

Allow the OpenGL ExternalTexture integration to represent and sample an
externally owned GL_TEXTURE_EXTERNAL_OES texture directly.

The exact API shape is open for discussion. One possible approach would be to
extend Graphics::GL::Texture::Descriptor, or an equivalent external-texture
descriptor, with a texture kind/target such as ExternalOES.

The resulting texture should:

  • Be bindable as GL_TEXTURE_EXTERNAL_OES.
  • Be sampled through samplerExternalOES on OpenGL ES.
  • Be usable from JavaScript after ExternalTexture::CreateForJavaScript and
    engine.wrapNativeTexture, without first copying it into a
    GL_TEXTURE_2D.
  • Support replacing the underlying OES handle through
    ExternalTexture::Update, if handle replacement is otherwise supported.
  • Preserve the existing externally owned lifetime model; BabylonNative must
    not delete a texture owned by SurfaceTexture or another producer.
  • Detect unavailable OES/ESSL3 extension support and report a clear error.

The implementation should also enforce or document the restrictions of an
external OES texture:

  • Sampled-only; not a Babylon render target.
  • One layer and base mip level zero.
  • No mipmap generation or mipmapped filtering.
  • CLAMP_TO_EDGE wrapping.
  • NEAREST or LINEAR minification/magnification filtering.

The producer would remain responsible for its platform lifecycle, including
calling SurfaceTexture.updateTexImage() on the correct GL context and applying
the matrix returned by SurfaceTexture.getTransformMatrix().

Suggested acceptance criteria:

  1. An Android SurfaceTexture-backed OES texture containing a known test image
    can be wrapped as an ExternalTexture.
  2. A Babylon material can sample and display it without an intermediate
    OES-to-RGBA texture copy.
  3. Updating the SurfaceTexture updates the displayed Babylon texture.
  4. Externally owned texture lifetime and graphics-thread requirements remain
    correct.
  5. Existing 2D, 2D-array, D3D, and Metal external-texture paths are unchanged.
  6. Unsupported sampler states or missing OES extensions fail clearly rather
    than rendering a black texture.

Describe alternatives you've considered

  1. Render OES into an RGBA GL_TEXTURE_2D.
    This is the current working solution, but it adds one full-frame texture and
    GPU render pass per input frame.

  2. Read the decoded frame back to the CPU and upload it again.
    This is substantially more expensive and defeats the purpose of decoding to
    a Surface.

  3. Render the OES texture outside BabylonNative using custom GLES code.
    This avoids the copy but prevents the texture from participating normally
    in Babylon materials, scene composition, and texture lifecycle management.

Requesting a regular 2D texture from the producer is generally not an option
for Android SurfaceTexture/MediaCodec output, whose texture target is
defined as GL_TEXTURE_EXTERNAL_OES.

Additional context

The historical bgfx work used a target-aware overrideInternal operation and an
external-sampler shader abstraction. The currently exposed bgfx API used by
BabylonNative does not appear to retain that target parameter, so implementing
this may require coordinated bgfx and BabylonNative changes.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions