Can multimodel support video type ?

[meistudioli]

Hi, I like multimodal support and apply it for e-commerce merchandise submit assistant. Once user uploaded merchandise image, AI will analysis the image and provide extra merchandise information to help them fill form. I wonder is it possible to support video type ? So I can do some video analysis stuff ?

[michaelwasserman]

Chrome (and likely Edge) implementations already support video element input as type "image", we should update docs.

First this API's spec takes image input as ImageBitmapSource.

• The ImageBitmapSource spec only includes: CanvasImageSource or Blob or ImageData
• Chromium implements ImageBitmapSource with many more types, including HTMLVideoElement.

Second, current API impls uses a single snapshot of the video at the element's current playback timestamp. We should specify that behavior, and decide if some other behavior or options are appropriate.

Clients can likely already capture additional video samples (using other APIs), and pass them into this API for additional analysis (e.g. 1 frame per second, etc.). @meistudioli: feel free to explain additional requirements if this doesn't support your use case. Thanks for the question.

[tomayac]

Chrome (and likely Edge) implementations already support video element input as type "image", we should update docs.

FYI for @michaelwasserman: Done in cl/846143000.

[michaelwasserman]

Thanks for updating developer.chrome.com, @tomayac!

We should also aim to update our spec (and the HTML spec?)