For questions, please use the forum.
Is your feature request related to a problem? Please describe.
Android commonly delivers camera and hardware-decoded video frames through
SurfaceTexture. The underlying OpenGL texture uses the
GL_TEXTURE_EXTERNAL_OES target rather than GL_TEXTURE_2D.
These textures cannot be sampled as ordinary 2D textures:
- They must be bound using
GL_TEXTURE_EXTERNAL_OES.
- GLES 3 shaders must enable
GL_OES_EGL_image_external_essl3.
- They must use
samplerExternalOES.
- They are sampled-only, single-level textures with restricted sampler state.
BabylonNative's current OpenGL ExternalTexture path cannot describe this
texture kind:
Graphics::GL::Texture::Descriptor
has no texture-target/type field. Its target is currently derived from the
layer count as either a 2D texture or a 2D-array texture.
ExternalTexture
always creates a bgfx texture through bgfx::createTexture2D.
As a result, an externally owned OES texture cannot be wrapped and sampled
directly by BabylonNative. The application must currently perform an additional
full-frame GPU pass:
SurfaceTexture / MediaCodec / camera
-> GL_TEXTURE_EXTERNAL_OES
-> samplerExternalOES full-screen draw
-> RGBA GL_TEXTURE_2D
-> BabylonNative ExternalTexture
BabylonNative's own Android
NativeCamera
currently demonstrates this workaround: it samples the OES camera texture into
an RGBA 2D framebuffer texture before making that texture available to bgfx.
Although this avoids CPU readback, it still requires an extra destination
texture and a full-frame render pass for every video frame, adding GPU bandwidth,
synchronization, and latency.
Describe the solution you'd like
Allow the OpenGL ExternalTexture integration to represent and sample an
externally owned GL_TEXTURE_EXTERNAL_OES texture directly.
The exact API shape is open for discussion. One possible approach would be to
extend Graphics::GL::Texture::Descriptor, or an equivalent external-texture
descriptor, with a texture kind/target such as ExternalOES.
The resulting texture should:
- Be bindable as
GL_TEXTURE_EXTERNAL_OES.
- Be sampled through
samplerExternalOES on OpenGL ES.
- Be usable from JavaScript after
ExternalTexture::CreateForJavaScript and
engine.wrapNativeTexture, without first copying it into a
GL_TEXTURE_2D.
- Support replacing the underlying OES handle through
ExternalTexture::Update, if handle replacement is otherwise supported.
- Preserve the existing externally owned lifetime model; BabylonNative must
not delete a texture owned by SurfaceTexture or another producer.
- Detect unavailable OES/ESSL3 extension support and report a clear error.
The implementation should also enforce or document the restrictions of an
external OES texture:
- Sampled-only; not a Babylon render target.
- One layer and base mip level zero.
- No mipmap generation or mipmapped filtering.
CLAMP_TO_EDGE wrapping.
NEAREST or LINEAR minification/magnification filtering.
The producer would remain responsible for its platform lifecycle, including
calling SurfaceTexture.updateTexImage() on the correct GL context and applying
the matrix returned by SurfaceTexture.getTransformMatrix().
Suggested acceptance criteria:
- An Android
SurfaceTexture-backed OES texture containing a known test image
can be wrapped as an ExternalTexture.
- A Babylon material can sample and display it without an intermediate
OES-to-RGBA texture copy.
- Updating the
SurfaceTexture updates the displayed Babylon texture.
- Externally owned texture lifetime and graphics-thread requirements remain
correct.
- Existing 2D, 2D-array, D3D, and Metal external-texture paths are unchanged.
- Unsupported sampler states or missing OES extensions fail clearly rather
than rendering a black texture.
Describe alternatives you've considered
-
Render OES into an RGBA GL_TEXTURE_2D.
This is the current working solution, but it adds one full-frame texture and
GPU render pass per input frame.
-
Read the decoded frame back to the CPU and upload it again.
This is substantially more expensive and defeats the purpose of decoding to
a Surface.
-
Render the OES texture outside BabylonNative using custom GLES code.
This avoids the copy but prevents the texture from participating normally
in Babylon materials, scene composition, and texture lifecycle management.
Requesting a regular 2D texture from the producer is generally not an option
for Android SurfaceTexture/MediaCodec output, whose texture target is
defined as GL_TEXTURE_EXTERNAL_OES.
Additional context
The historical bgfx work used a target-aware overrideInternal operation and an
external-sampler shader abstraction. The currently exposed bgfx API used by
BabylonNative does not appear to retain that target parameter, so implementing
this may require coordinated bgfx and BabylonNative changes.
For questions, please use the forum.
Is your feature request related to a problem? Please describe.
Android commonly delivers camera and hardware-decoded video frames through
SurfaceTexture. The underlying OpenGL texture uses theGL_TEXTURE_EXTERNAL_OEStarget rather thanGL_TEXTURE_2D.These textures cannot be sampled as ordinary 2D textures:
GL_TEXTURE_EXTERNAL_OES.GL_OES_EGL_image_external_essl3.samplerExternalOES.BabylonNative's current OpenGL
ExternalTexturepath cannot describe thistexture kind:
Graphics::GL::Texture::Descriptorhas no texture-target/type field. Its target is currently derived from the
layer count as either a 2D texture or a 2D-array texture.
ExternalTexturealways creates a bgfx texture through
bgfx::createTexture2D.As a result, an externally owned OES texture cannot be wrapped and sampled
directly by BabylonNative. The application must currently perform an additional
full-frame GPU pass:
BabylonNative's own Android
NativeCameracurrently demonstrates this workaround: it samples the OES camera texture into
an RGBA 2D framebuffer texture before making that texture available to bgfx.
Although this avoids CPU readback, it still requires an extra destination
texture and a full-frame render pass for every video frame, adding GPU bandwidth,
synchronization, and latency.
Describe the solution you'd like
Allow the OpenGL
ExternalTextureintegration to represent and sample anexternally owned
GL_TEXTURE_EXTERNAL_OEStexture directly.The exact API shape is open for discussion. One possible approach would be to
extend
Graphics::GL::Texture::Descriptor, or an equivalent external-texturedescriptor, with a texture kind/target such as
ExternalOES.The resulting texture should:
GL_TEXTURE_EXTERNAL_OES.samplerExternalOESon OpenGL ES.ExternalTexture::CreateForJavaScriptandengine.wrapNativeTexture, without first copying it into aGL_TEXTURE_2D.ExternalTexture::Update, if handle replacement is otherwise supported.not delete a texture owned by
SurfaceTextureor another producer.The implementation should also enforce or document the restrictions of an
external OES texture:
CLAMP_TO_EDGEwrapping.NEARESTorLINEARminification/magnification filtering.The producer would remain responsible for its platform lifecycle, including
calling
SurfaceTexture.updateTexImage()on the correct GL context and applyingthe matrix returned by
SurfaceTexture.getTransformMatrix().Suggested acceptance criteria:
SurfaceTexture-backed OES texture containing a known test imagecan be wrapped as an
ExternalTexture.OES-to-RGBA texture copy.
SurfaceTextureupdates the displayed Babylon texture.correct.
than rendering a black texture.
Describe alternatives you've considered
Render OES into an RGBA
GL_TEXTURE_2D.This is the current working solution, but it adds one full-frame texture and
GPU render pass per input frame.
Read the decoded frame back to the CPU and upload it again.
This is substantially more expensive and defeats the purpose of decoding to
a
Surface.Render the OES texture outside BabylonNative using custom GLES code.
This avoids the copy but prevents the texture from participating normally
in Babylon materials, scene composition, and texture lifecycle management.
Requesting a regular 2D texture from the producer is generally not an option
for Android
SurfaceTexture/MediaCodecoutput, whose texture target isdefined as
GL_TEXTURE_EXTERNAL_OES.Additional context
SurfaceTexturedocumentation:https://developer.android.com/reference/android/graphics/SurfaceTexture
GL_OES_EGL_image_external:https://registry.khronos.org/OpenGL/extensions/OES/OES_EGL_image_external.txt
https://registry.khronos.org/OpenGL/extensions/OES/OES_EGL_image_external_essl3.txt
ExternalTexturebackend was introduced in:Implement the OpenGL backend of the ExternalTexture plugin #1780
The historical bgfx work used a target-aware
overrideInternaloperation and anexternal-sampler shader abstraction. The currently exposed bgfx API used by
BabylonNative does not appear to retain that target parameter, so implementing
this may require coordinated bgfx and BabylonNative changes.