Skip to main content
Version: 6.1.0

Vonage ML Transformers

Vonage ML transformers is a library that implements machine learning algorithms for the web. This library is based on @vonage/media-processor, MediaPipe and TFLite

@vonage/media-processor​

Media Processor library is Vonage implementation for insertable streams for supported browsers. Documentation can be found here.

MediaPipe​

MediaPipe library is an open source library under MIT license, this library use for video enhancements.

Sample applications​

Sample applications can be found here.

Background visual effects (out-of-the-box solution)​

This sample uses the Vonage Video Web SDK (OpenTok). OT.Publisher API (setVideoMediaProcessorConnector) function is used to use the Vonage Media Processor Library in a Vonage Video (OpenTok) Web application.

ML assets​

By ML assets we mean the model file and some Mediapipe assets which are not bundled into the library.

During the library operation these are assets fetched from a default location (Vonage AWS Cloud Front CDN). Check this IP whitelistning from AWS Cloud Front CDN.

While the library fetches the ML assets from it we encourge developers using the library this way to host these assets for full control. Please, contact Vonage to get a copy of these assets. Note: These assets can vary between versions.

The library let the developer to specifiy the location of these assets in the following way.

let config: MediaProcessorConfig
config = {
  ...
  // For more information see https://vonage.github.io/ml-transformers-docs/docs/api/interfaces/MediaProcessorBaseConfig#modelasseturipath.
  modelAssetUriPath: 'https://static.opentok.com/ml-transformers/v6.0.0/float16/vonage_selfie_segmenter.tflite',
  // For more information see https://vonage.github.io/ml-transformers-docs/docs/api/interfaces/MediaProcessorBaseConfig#mediapipebaseassetsuri.
  mediapipeBaseAssetsUri: 'https://static.opentok.com/ml-transformers/v6.0.0/mediapipe/0.10.20'
  ...
}

Configure​

Configure post process action.

Blur:​

let config: MediaProcessorConfig
config = {
    transformerType: 'BackgroundBlur',
    radius: BlurRadius.Low | BlurRadius.High | number //Low=5px High=10px number=(number)px
}

Silhouette:​

let config: MediaProcessorConfig
config = {
    transformerType: 'SilhouetteBlur',
    radius: BlurRadius.Low | BlurRadius.High | number //Low=5px High=10px number=(number)px
}

Virtual (image):​

let config: MediaProcessorConfig
config = {
    transformerType: 'VirtualBackground',
    backgroundAssetUri: 'https://some-url-to-image.com'
}

Video:​

let config: MediaProcessorConfig
config = {
    transformerType: 'VideoBackground',
    backgroundAssetUri: 'https://some-url-to-video.com'
}

Create Media Processor​

After configuring which post process is needed, use the helper function to create it VonageMediaProcessor

const processor = await createVonageMediaProcessor(config);
publisher.setVideoMediaProcessorConnector(processor.getConnector());

Change configuration​

To change the post process config in-flight, you can call this method without involving the publisher setBackgroundOptions

await processor.setBackgroundOptions(newConfig);

Disable/enable processing​

You can disable the postprocessing using enable/disable functions.

const processor = await createVonageMediaProcessor(config);
processor.disable();
processor.enable();

Errors, Warnings and Statistics​

isSupported​

Checks if the current browser can run our library.

try {
    await  isSupported();
} catch(e) {
    console.error(e);
}

Emitter Registration​

This solution supports Emittery You can listen event directly on VonageMediaProcessor

processor.on('error', ((eventData: ErrorData) => {
    console.error(eventData);
}))
processor.on('warn', ((eventData: WarnData) => {
    console.warn(eventData);
}))
processor.on('pipelineInfo', ( (eventData: PipelineInfoData) => {
    console.info(eventData)
}))

Frame Drop warning​

If you like to be notified about frame rate drop use setTrackExpectedRate(number) for the expected rate of the process.

processor.setTrackExpectedRate(30)//or any other value.

Statistics​

The API collect statistics for usage and debugging purposes. However, it is up to the user to activate it.

Turn statistics on:​
const  metadata: VonageMetadata = {
    appId:  'video SDK app id',
    sourceType:  'video',
    proxyUrl: 'https://some-proxy.com' //optional
};
setVonageMetadata(metadata)
Turn statistics off: (by default the statistics are off)​
setVonageMetadata(null)

License​

This project is licensed under the terms of the MIT license and is available for free.