Augmented Web Based Makeup Try-On

2 min read November 4, 2022

The objective of the project was to create a fully custom-made makeup try-on solution built on Big Data, AI, CV, and web AR real-time face augmentation technology, capable of detecting and identifying the exact Pantone of makeup shades, integrating and fully compatible with the Prefect Corp try-on solution. The need to create an entirely new experience arose from the functional limitation of Perfect Corp software in its capability to allow users to try-on multiples makeup combinations for unique look try-on and creation.

The web-based AR solution allows the user to try an entire look with the full Pantone scale and multiple products at once. Integration of AI and Big Data activates a unique feature, which suggests products based on the user’s demographics, geographical location, complexion, face shape, personal preferences, and previous purchase history. One of the core created features grants users the possibility to upload images (of celebrity, blogger, their own, etc. makeup looks), which are then analyzed and matched with a palette of over 10000 shades in order to identify products that are used with maximum precision.

Adaptive facial landmark technology was used to achieve the highest accuracy of facial feature and complexion identification, enabling users to try-on looks and products in real-time. The integration of Shopify allows users to complete the purchase of individual products and/or complete product sets used for the uploaded image look creation.

The new solution was made possible with:

The creation of a custom framework solution allowing to assign hundreds of physical product SKUs to one Perfect Corp SKU, widening the spectrum of products and shades portfolio.
The usage of Big Data collected from users during their try-on experience.
Accurate complexion detection based on data cleaning, shadow, and light spikes removal using color frequency analysis, continuous machine learning, and data mining.
Web-based desktop and mobile device compatible AR.
Creation of easy portfolio management and try-on tool allowing the software owner to navigate through the entire list of SKUs on one screen, manage available palettes for every SKU, check and try on each product in real-time.
Real-time neural network facial feature detection system.
Integration of CIEDE2000 color distance formula for maximum color detection and recognition.
K-mean based color clustering.