@vonage/ml-transformers

Classes

• VonageMediaProcessor
• BackgroundTransformer
• MediapipeHelper

Enumerations

• BlurRadius
• BackgroundTransformerType
• WebglSelfieSegmentationType
• RenderingType
• WarningType
• ErrorFunction
• PipelineInfoData

Interfaces

• MediaProcessorBaseConfig
• AssetBackgroundConfig
• BackgroundBlurConfig
• VirtualBackgroundConfig
• VideoBackgroundConfig
• SilhouetteBlurConfig
• FeatureSupport
• SupportedBrowserFeatures
• FaceDetectionResults
• FaceDetectionOptions
• FaceMeshResults
• FaceMeshOptions
• HandsResults
• HandsOptions
• HolisticResults
• HolisticOptions
• ObjectronResults
• ObjectronOptions
• PoseResults
• PoseOptions
• SelfieSegmentationResults
• SelfieSegmentationOptions

Functions

getVonageSelfieSegmentation

▸ getVonageSelfieSegmentation(): VonageSelfieSegmentation

MediaPipe Selfie Segmentation static metadata.

Returns

VonageSelfieSegmentation

An object containing MediaPipe Selfie Segmentation metadata:

• VERSION

---

getVonageHands

▸ getVonageHands(): VonageHands

Vonage Mediapipe objects info (it will be used by the @mediapipe/drawing_utils package which is not included by this library).

Returns

VonageHands

---

getVonageHolistic

▸ getVonageHolistic(): VonageHolistic

MediaPipe Holistic static metadata. This metadata is needed while using MediaPipe. drawing utils. And control utils.

Returns

VonageHolistic

An object containing MediaPipe Holistic metadata:

• VERSION
• FACE_GEOMETRY
• FACEMESH_LIPS
• FACEMESH_LEFT_EYE
• FACEMESH_LEFT_EYEBROW
• FACEMESH_LEFT_IRIS
• FACEMESH_RIGHT_EYE
• FACEMESH_RIGHT_EYEBROW
• FACEMESH_RIGHT_IRIS
• FACEMESH_FACE_OVAL
• FACEMESH_CONTOURS
• FACEMESH_TESSELATION
• HAND_CONNECTIONS
• POSE_CONNECTIONS
• POSE_LANDMARKS
• POSE_LANDMARKS_LEFT
• POSE_LANDMARKS_RIGHT
• POSE_LANDMARKS_NEUTRAL

---

getVonageFaceDetection

▸ getVonageFaceDetection(): VonageFaceDetection

MediaPipe Face Detection static metadata. This metadata is needed while using MediaPipe. drawing utils. And control utils.

Returns

VonageFaceDetection

An object containing MediaPipe Face Detection metadata:

• VERSION
• FACEDETECTION_LIPS
• FACEDETECTION_LEFT_EYE
• FACEDETECTION_LEFT_EYEBROW
• FACEDETECTION_RIGHT_EYE
• FACEDETECTION_RIGHT_EYEBROW
• FACEDETECTION_FACE_OVAL
• FACEDETECTION_CONTOURS
• FACEDETECTION_TESSELATION

---

getVonageFaceMash

▸ getVonageFaceMash(): VonageFaceMash

MediaPipe Face Mash static metadata. This metadata is needed while using MediaPipe. drawing utils. And control utils.

Returns

VonageFaceMash

An object containing MediaPipe Face Mash metadata:

• VERSION
• FACE_GEOMETRY
• FACEMESH_LIPS
• FACEMESH_LEFT_EYE
• FACEMESH_LEFT_EYEBROW
• FACEMESH_LEFT_IRIS
• FACEMESH_RIGHT_EYE
• FACEMESH_RIGHT_EYEBROW
• FACEMESH_RIGHT_IRIS
• FACEMESH_FACE_OVAL
• FACEMESH_CONTOURS
• FACEMESH_TESSELATION

---

getVonageObjectron

▸ getVonageObjectron(): VonageObjectron

MediaPipe Objectron static metadata. This metadata is needed while using MediaPipe. drawing utils. And control utils.

Returns

VonageObjectron

An object containing MediaPipe Objectron metadata:

• VERSION
• VERSION
• BOX_CONNECTIONS
• BOX_KEYPOINTS

---

getVonagePose

▸ getVonagePose(): VonagePose

MediaPipe Pose static metadata. This metadata is needed while using MediaPipe. drawing utils. And control utils.

Returns

VonagePose

An object containing MediaPipe Pose metadata:

• VERSION
• POSE_CONNECTIONS
• POSE_LANDMARKS
• POSE_LANDMARKS_LEFT
• POSE_LANDMARKS_RIGHT
• POSE_LANDMARKS_NEUTRAL

---

createVonageMediaProcessor

▸ createVonageMediaProcessor(config): Promise<VonageMediaProcessor>

builder function to create MediaProcessor

Parameters

Name	Type	Description
config	BackgroundOptions	see MediaProcessorConfig definition

Returns

Promise<VonageMediaProcessor>

• MediaProcessor provide interface to configure ml-transformer or retrieving connector

---

isSupported

▸ isSupported(wasmAssetUriPath?, tfliteAssetUriPath?): Promise<void>

Check if the current browser is officially supported by the library.

Parameters

Name	Type
wasmAssetUriPath?	string
tfliteAssetUriPath?	string

Returns

Promise<void>

The promise will resolve or reject depending whether the browser is supported or not.

---

getSupportedBrowserFeatures

▸ getSupportedBrowserFeatures(wasmAssetUriPath?, tfliteAssetUriPath?): Promise<SupportedBrowserFeatures>

Detect which features is available in the browser.

Parameters

Name	Type
wasmAssetUriPath?	string
tfliteAssetUriPath?	string

Returns

Promise<SupportedBrowserFeatures>

Supported features by the browser

Type Aliases

MediaProcessorConfig

Ƭ MediaProcessorConfig: BackgroundOptions

MediaProcessorConfig specifies the transformer logic to be performed.

---

RenderingOptions

Ƭ RenderingOptions: Canvas2dOptions | WebglOptions

---

BackgroundOptions

Ƭ BackgroundOptions: BackgroundBlurConfig | VirtualBackgroundConfig | VideoBackgroundConfig | SilhouetteBlurConfig

BackgroundOptions background options

---

MediaPipeResults

Ƭ MediaPipeResults: FaceDetectionResults | FaceMeshResults | HandsResults | HolisticResults | ObjectronResults | SelfieSegmentationResults | PoseResults

MediaPipeResults types of results object of MediaPipe

---

MediaPipeResultsListener

Ƭ MediaPipeResultsListener: <T>(results: T) => Promise<void> | void

Type declaration

▸ <T>(results): Promise<void> | void

ResultsListener callback function from MediaPipe process

Results

• The results object from MediaPipe

Type parameters

Name	Type
T	extends MediaPipeResults

Parameters

Name	Type
results	T

Returns

Promise<void> | void

• can return a promise of void

---

MediaPipeModelType

Ƭ MediaPipeModelType: "face_mesh" | "face_detection" | "hands" | "holistic" | "objectron" | "selfie_segmentation" | "pose"

MediaPipeModelType supported models types

---

MediaPipeModelConfig

Ƭ MediaPipeModelConfig: Object

defines one mediapipe model config

Model Type

• which model is required

Listener

• callback function from the model

Options

• define options for the mediapipe model that is used. for more info check https://google.github.io/mediapipe/getting_started/javascript.html

Assets Uri

(optional) - can be set to get wasm/tflite/js/binarypb assets. Vonage provides static assets. WARNING - using this option moves the responsibility of the assets and the versioning to the user. please keep in mind that the assets list can change between versions! assets are different between mediapipe models.

Type declaration

Name	Type
modelType	MediaPipeModelType
listener	MediaPipeResultsListener
options	FaceDetectionOptions | FaceMeshOptions | HandsOptions | HolisticOptions | ObjectronOptions | SelfieSegmentationOptions | PoseOptions
assetsUri?	string

---

MediapipeConfig

Ƭ MediapipeConfig: Object

MediapipeConfig specified config of mediapipe helper: this structure allows the user to create few mediapipe models that will run in parallel.

Model Types Array

• array of mediapipe models to be loaded and called.

Type declaration

Name	Type
mediaPipeModelConfigArray	MediaPipeModelConfig[]

---

EventMetaData

Ƭ EventMetaData: Object

EventMetaData. Some metadata of the event.

Type declaration

Name	Type	Description
transformerIndex	number	The transformer index in the array of transformers.

---

DropInfo

Ƭ DropInfo: Object

DropInfo gives info about the frame rate of the transformer

Type declaration

Name	Type	Description
requested	number	The rate predicted rate of the track
current	number	The actual rate of the track

---

WarnData

Ƭ WarnData: Object

WarnData - MediaProcessor warning event defintion

Type declaration

Name	Type	Description
eventMetaData	EventMetaData	Meta data of the event
warningType	WarningType	The waring type
dropInfo?	DropInfo	Optional - FPS drop info.

---

ErrorData

Ƭ ErrorData: Object

ErrorData. MediaProcessor error data definition.

Type declaration

Name	Type	Description
eventMetaData	EventMetaData	Metadata of the event.
function	ErrorFunction	The function in which the error occurred.
error	unknown	Type of unknown but moslty will be string

---

EventDataMap

Ƭ EventDataMap: Object

EventDataMap. A map type for all event options from MediaProcessor. Each event will include only one type.

Type declaration

Name	Type	Description
warn	WarnData	Warnings from the insertable streams pipeline.
error	ErrorData	Errors from the insertable streams pipeline.
pipelineInfo	PipelineInfoData	Info from the insertable streams pipeline life cycle.