Skip to content

face-mesh-detection

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

face-mesh-detection

README.md

Capacitor ML Kit Face Mesh Detection Plugin

Unofficial Capacitor plugin for ML Kit Face Mesh Detection.1

Use Cases

The Face Mesh Detection plugin is typically used whenever an app needs detailed facial geometry, for example:

  • Face filters and effects: Use the 468 3D face mesh points and triangle surfaces to align virtual makeup, masks, or other effects with a face.
  • Facial feature analysis: Work with the contours of the face oval, eyebrows, eyes, lips, and nose bridge.
  • Fast face localization: Use the BoundingBoxOnly use case to quickly get the bounding boxes of detected faces.
  • Selfie tools: Analyze up to 2 faces in detail using the full face mesh information.

Compatibility

Plugin Version Capacitor Version Status
8.x.x >=8.x.x Active support
6.x.x 6.x.x Deprecated

Installation

You can use our AI-Assisted Setup to install the plugin. Add the Capawesome Skills to your AI tool using the following command:

npx skills add capawesome-team/skills --skill capacitor-plugins

Then use the following prompt:

Use the `capacitor-plugins` skill from `capawesome-team/skills` to install the `@capacitor-mlkit/face-mesh-detection` plugin in my project.

If you prefer Manual Setup, install the plugin by running the following commands and follow the platform-specific instructions below:

npm install @capacitor-mlkit/face-mesh-detection
npx cap sync

Android

Variables

If needed, you can define the following project variable in your app’s variables.gradle file to change the default version of the dependency:

  • $mlkitFaceMeshDetectionVersion version of com.google.mlkit:face-mesh-detection (default: 16.0.0-beta1)

This can be useful if you encounter dependency conflicts with other plugins in your project.

Configuration

No configuration required for this plugin.

Demo

A working example can be found here: robingenz/capacitor-mlkit-plugin-demo

Usage

The following example shows how to detect face meshes in an image.

Detect face meshes in an image

Detect face meshes in an image at a local path. Choose the BoundingBoxOnly use case for fast bounding box detection or the FaceMesh use case to also receive the 468 3D face mesh points and triangle information. Only available on Android:

import { FaceMeshDetection, UseCase } from '@capacitor-mlkit/face-mesh-detection';

const processImage = async () => {
  const { faceMeshs } = await FaceMeshDetection.processImage({
    path: 'path/to/image.jpg',
    useCase: UseCase.FaceMesh,
  });
  return faceMeshs;
};

API

processImage(...)

processImage(options: ProcessImageOptions) => Promise<ProcessImageResult>

Detects face mesh from the supplied image.

Only available on Android.

Param Type
options ProcessImageOptions

Returns: Promise<ProcessImageResult>

Since: 5.3.0

Interfaces

ProcessImageResult

Prop Type Description Since
faceMeshs FaceMesh[] The detected face meshs. 5.3.0

FaceMesh

Represents a face mesh detected by FaceMeshDetector.

When BoundingBoxOnly is selected, FaceMesh only contains valid bounding box.

When FaceMesh is selected, FaceMesh also contains a group of 468 3D face mesh points and related triangle information. Each point is represented by FaceMeshPoint describing a specific position in detected face. The triangle information is a group of 3 FaceMeshPoints representing a valid surface on Face (e.g. a valid small surface on nose tip).

Prop Type Description Since
bounds Rect Returns the axis-aligned bounding rectangle of the detected face mesh. 5.3.0
contours Contours Returns contours with a list of FaceMeshPoint representing the detected face. 5.3.0
faceMeshPoints FaceMeshPoint[] Returns a list of FaceMeshPoint representing the whole detected face. 5.3.0
triangles Triangle[] Returns a list of Triangle representing logical triangle surfaces of detected face. Each Triangle contains 3 FaceMeshPoint, representing 3 points of the triangle surface. The sequence of the 3 points are constant and always counter clockwise in face mesh. 5.3.0

Rect

Rect holds four integer coordinates for a rectangle.

Prop Type Description Since
left number The X coordinate of the left side of the rectangle. 5.3.0
top number The Y coordinate of the top of the rectangle. 5.3.0
right number The X coordinate of the right side of the rectangle. 5.3.0
bottom number The Y coordinate of the bottom of the rectangle. 5.3.0

Contours

Represents contours with their face mesh points.

Prop Type Description Since
faceOval FaceMeshPoint[] Returns all points for the FaceOval contour. 5.3.0
leftEyebrowTop FaceMeshPoint[] Returns all points for the LeftEyebrowTop contour. 5.3.0
leftEyebrowBottom FaceMeshPoint[] Returns all points for the LeftEyebrowBottom contour. 5.3.0
rightEyebrowTop FaceMeshPoint[] Returns all points for the RightEyebrowTop contour. 5.3.0
rightEyebrowBottom FaceMeshPoint[] Returns all points for the RightEyebrowBottom contour. 5.3.0
leftEye FaceMeshPoint[] Returns all points for the LeftEye contour. 5.3.0
rightEye FaceMeshPoint[] Returns all points for the RightEye contour. 5.3.0
upperLipTop FaceMeshPoint[] Returns all points for the UpperLipTop contour. 5.3.0
upperLipBottom FaceMeshPoint[] Returns all points for the UpperLipBottom contour. 5.3.0
lowerLipTop FaceMeshPoint[] Returns all points for the LowerLipTop contour. 5.3.0
lowerLipBottom FaceMeshPoint[] Returns all points for the LowerLipBottom contour. 5.3.0
noseBridge FaceMeshPoint[] Returns all points for the NoseBridge contour. 5.3.0

FaceMeshPoint

Represents a 3D point in face mesh.

The index is an unique ID meaning a fixed position on face, ranging from 0 to 467.

In Point3D, x and y are pixel location of detected face in InputImage. z is also scaled to image size, while the origin will be somewhere in the center of all 468 face mesh points.

Prop Type Description Since
index number Gets the index of the face mesh point, ranging from 0 to 467. For each specific point, the index is a constant value. 5.3.0
point Point3D Gets a 3D point in face mesh. Inside Point3D, X and Y means a 2D position in original image. More information on the Z value: - The unit of measure for the Z value is the same as X and Y. - The smaller the Z value, the closer that landmark is to the camera. - The Z origin is approximately at the center of all 468 face mesh points. Z value will be negative if the point is close to camera and will be positive if the point is away from the camera. 5.3.0

Point3D

Represents a 3D point.

Prop Type Description Since
x number Returns the X value of the point. 5.3.0
y number Returns the Y value of the point. 5.3.0
z number Returns the Z value of the point. 5.3.0

Triangle

Represents a triangle with 3 generic points.

Prop Type Description Since
points FaceMeshPoint[] Returns all points inside the Triangle. 5.3.0

ProcessImageOptions

Prop Type Description Default Since
path string The local path to the image file. 5.3.0
useCase UseCase Sets the use case. When BoundingBoxOnly is selected, the returned FaceMesh only contains bounding box. When FaceMesh is selected, the returned FaceMesh contains bounding box as well as 468 FaceMeshPoint and triangle information. It detects at most 2 faces in this case and it is slower than BoundingBoxOnly. UseCase.FaceMesh 5.3.0

Enums

UseCase

Members Value Description Since
BoundingBoxOnly 0 Return bounding box for detected face. 5.3.0
FaceMesh 1 Return face mesh info for detected face. It detects at most 2 faces in this use case. 5.3.0

FAQ

Which platforms are supported by this plugin?

The processImage(...) method is only available on Android. iOS and Web are not supported by the underlying ML Kit Face Mesh Detection SDK.

What is the difference between the BoundingBoxOnly and FaceMesh use cases?

With BoundingBoxOnly, the result only contains a valid bounding box for each detected face, which is faster. With FaceMesh (the default), the result additionally contains a group of 468 3D face mesh points, contours, and triangle information, but it detects at most 2 faces and is slower.

What exactly is a face mesh point?

A face mesh point is a 3D point on a detected face with an index between 0 and 467, where each index always refers to the same fixed position on the face. The x and y values are pixel locations in the input image, while the z value describes the distance to the camera: the smaller the value, the closer that point is to the camera.

What is the difference between this plugin and the Face Detection plugin?

The ML Kit Face Detection plugin detects faces with bounding boxes, landmarks, contours, and classifications such as smiling probability, and works on Android and iOS. This plugin returns a much denser mesh of 468 3D points and triangle information for detailed facial geometry, but is only available on Android.

Can I use this plugin with Ionic, React, Vue or Angular?

Yes, the plugin is framework-agnostic. It works in any Capacitor app regardless of the web framework, including Ionic with Angular, React, or Vue, as well as plain JavaScript projects.

Related Plugins

Terms & Privacy

This plugin uses the Google ML Kit:

Newsletter

Stay up to date with the latest news and updates about the Capawesome, Capacitor, and Ionic ecosystem by subscribing to our Capawesome Newsletter.

Changelog

See CHANGELOG.md.

License

See LICENSE.

Footnotes

  1. This project is not affiliated with, endorsed by, sponsored by, or approved by Google LLC or any of their affiliates or subsidiaries. ↩