Virtual Background

Introduction

Virtual background plugin helps in customising one’s background. The customising options are blurring the background or replacing it with a static image. This guide provides an overview of usage of the virtual background plugin of 100ms.

Supported Devices

Virtual background is currently only supported on web in Chrome, Firefox, Brave and Edge browsers.
We have not extended support for Mobile browsers in this release.

Basic Concepts

• blur background - This is an action where the background of the video is blurred. No external image is used/required.
• image background - This is where the plugin will replace the video background with the image provided by user.
• plugin load time - The time taken by plugin to load model. This is in the range of 3-5 seconds for first time. Subsequent loads take less than 100 milliseconds.

Pre-requisites

Get the 100ms VirtualBackground Package

npm install --save @100mslive/hms-virtual-background@latest
 

Import plugin

import HMSVirtualBackgroundPlugin from '@100mslive/hms-virtual-background';
 

Instantiate Virtual Background

This accepts background as a parameter, It can be of 3 types:

• blur - String - This will set the background to blur
• image - HTMLImageElement - This will replace the background to the image provide
• none - String - This will remove the background effect from the video

// background : {'blur' | image | 'none'}
const virtualBackground = new HMSVirtualBackgroundPlugin(background : {'blur' | image | 'none'});
 

Interfaces

Check if plugin is supported

validateVideoPluginSupport can be used to check if the browser/input device is supported or not. It will return True in case of plugin is supported and return error message if it is not.

import { hmsActions } from './hms';

const pluginSupport = hmsActions.validateVideoPluginSupport(virtualBackground);
if (pluginSupport.isSupported) {
    console.log('Platform is supported');
} else {
    const err = pluginSupport.errMsg;
    console.error(err);
}
 

Init(Optional)

Init is used to load model of virtual background for the first time. It takes on an average 3-5 seconds.
Calling init is handled internally by SDK if not done by user, in this case addPlugin call will take 3-5 seconds for the first time and then less than 100 milliseconds in the subsequent calls. Check this section for addPlugin API usage

virtualBackground.init();
 

Init can also be used by user to show a loader icon for background selector component during the plugin loading stage

virtualBackground.init().then(() => console.log('background can be changed now'));
 

Start and Stop Virtual Background

Check this section to use custom pluginFrameRate

import { hmsActions, hmsStore } from './hms';
import { selectIsLocalVideoPluginPresent } from '@100mslive/hms-video-store';

async function toggleVB() {
    const isVirtualBackgroundEnabled = hmsStore.getState(
        selectIsLocalVideoPluginPresent(virtualBackground.getName())
    );
    try {
        if (!isVirtualBackgroundEnabled) {
            // Recommended value
            const pluginFrameRate = 15;
            // add virtual background
            await hmsActions.addPluginToVideoTrack(virtualBackground, pluginFrameRate);
        } else {
            // remove virtual background
            await hmsActions.removePluginFromVideoTrack(virtualBackground);
        }
    } catch (err) {
        console.log('virtual background failure - ', isVirtualBackgroundEnabled, err);
    }
}
 

import { useHMSStore, useHMSActions, selectIsLocalVideoPluginPresent } from '@100mslive/react-sdk';

function VirtualBackground() {
    const isVirtualBackgroundEnabled = useHMSStore(
        selectIsLocalVideoPluginPresent(virtualBackground.getName())
    );
    const hmsActions = useHMSActions();

    const toggleVB = async () => {
        try {
            if (!isVirtualBackgroundEnabled) {
                // Recommended value
                const pluginFrameRate = 15;
                await hmsActions.addPluginToVideoTrack(virtualBackground, pluginFrameRate);
            } else {
                await hmsActions.removePluginFromVideoTrack(virtualBackground);
            }
        } catch (err) {
            console.log('virtual background failure - ', isVirtualBackgroundEnabled, err);
        }
    };

    return <Button onClick={toggleVB} />;
}
 

Change Background

The function setBackground can be used to update the background again later. It accepts string or HTMLImageElement as a parameter Image will get fit to the video by maintaining the aspect ratio. If the aspect ratio of the background image is not the same as the video, the image will be cropped to fit in the background.

// background parameter is explained in Instantiate Virtual Background section
try{
  virtualBackground.setBackground(background : {'blur' | image | 'none'});
}catch(err){
  console.log("Failed to update background", err);
}
 

Tuning pluginFrameRate(Optional)

pluginFrameRate- number - pluginFrameRate helps in controlling the performance and experience of Virtual background plugin. pluginFrameRate translates to the number of frames for which background is detected. Higher value will use more resources (cpu/memory/battery), while making the virtual background experience smooth. Lower value will be generous on resources, while lowering the virtual background smoothness. Recommended value is 15. Values higher than this will not significantly improve virtual background smoothness but will be heavy on resources. For lower end devices value can be in the range of 7-10.

Recommendations For Better User Experience

For enhancing the performance of virtual background plugin we recommend enabling SIMD on the browser. With SIMD enabled, browsers can process multiple data points with each instruction, resulting in a performance boost of more than 2x.

• Chrome - Open chrome://flags/ in browser, search for simd, enable WebAssembly SIMD support and restart.
• Firefox- Open about:config in browser, search for simd, set javascript.options.wasm_simd=true and restart.
• Edge - Open about:flags in browser, search for simd, enable WebAssembly SIMD support and restart.
• Brave - Open about:flags in browser, search for simd, enable WebAssembly SIMD support and restart.