Auto-translation used

Why Verigram regularly tests algorithms in the NIST lab

Face and object recognition technologies are becoming more complex and widespread. Today they are used in the field of security, access control, marketing, etc. 

One way to check the quality and reliability of algorithms is to apply for independent testing at the National Institute of Standards and Technology (NIST is a US government agency that develops and promotes standards for a wide range of technologies, including biometrics). NIST's facial recognition scores are rigorous and comprehensive, and the algorithms that have passed these tests are considered among the best in the world.

Some testing goals:

  • Identify areas of improvement of the algorithm. The test shows in detail the performance of the algorithm, which helps us to find problem areas and focus work in them.  
  • Show accuracy and reliability. The test checks algorithms for a variety of factors, including speed and quality of work in difficult conditions. Indeed, the best algorithms that can be relied on in real applications can pass such a test.
  • Transparency of results. The results of the tests are publicly available and are publicly available. Anyone can get acquainted with the work of algorithms from vendors from all over the world.
  • Trust and responsibility. Recognition technologies are powerful tools, but they also raise concerns about privacy and the possibility of abuse. By submitting their technologies to NIST for independent testing, companies demonstrate their commitment to developing and implementing these technologies in a responsible and ethical manner.

The Facial Recognition System Vendors Test (FRVT) conducted by NIST has long been the world's leading independent assessment. FRVT is conducted on a regular basis using a large and diverse set of images. 

What is the NIST Vendor Face Recognition test and how to read the results

Let's figure out what the main test of the manufacturer is, the results of which can often be found in reviews of facial recognition algorithms.

FRVT uses a large and diverse set of images, including full-face and profile photos, images with different lighting and pose variations, as well as images of people from different national and demographic groups. 

The algorithms are tested for the ability to accurately match faces in various scenarios, such as:

  • Identification 1:1: Matching the face in the test image with the face in the image in the gallery. At the same time, the gallery has exactly the image corresponding to the test image (the reference face).
  • Identification 1:N: Search the image gallery to determine if the trial image contains a match with any image in the gallery.

NIST also reports on the accuracy of facial recognition algorithms in various difficult conditions, such as:

  • Low quality images: with poor resolution, noise, or compression.
  • Pose options: Images in which the subject's face is not facing directly into the camera.
  • Lighting options: images with poor lighting conditions, such as overexposure or underexposure.
  • Obstacles: Images in which the subject's face is partially obscured by objects such as glasses, hats, or masks.

NIST uses two metrics to measure the accuracy of facial recognition and matching algorithms:

  • The false—negative identification Coefficient (FNIR) is the proportion of paired comparisons that the algorithm cannot match. In other words, this is the percentage of cases when the algorithm mistakenly fails to identify a famous person.
  • The False Positive Identification Coefficient (FPIR) is the proportion of impostor comparisons that the algorithm matches. In other words, this is the percentage of cases when the algorithm mistakenly identifies a stranger as a famous person.

NIST reports the FNMR and FMR of each algorithm in a range of threshold values. The threshold is the score that the algorithm needs to get in order to return a match. A higher threshold will result in fewer false matches, but also more false mismatches.

In addition to accuracy, NIST also tests the speed and reliability of facial recognition and matching algorithms. The speed of the algorithm is measured by the time it takes to process the image and return a match. The reliability of the algorithm is measured by its ability to work accurately in difficult conditions, such as poor lighting, poor image quality and obstacles (for example, sunglasses, hats and masks).

Case Study: Verigram Test Results - Top 1

Our algorithm has received the highest NIST rating in the category "Identification 1:N" based on a database of 12 million individuals with results FPIR=0.001, FNIR=0.0115.

These results show that the algorithm achieved FNIR 0.0115 with FPIR 0.001 in a photo dataset that contains 12 million faces. This means that the algorithm was able to find and correctly identify a matching face in 98.85% of cases, while a false positive match occurred only in 0.1% of cases. This means that the algorithm will most likely be able to identify a person's face, even if they are wearing glasses, a hat or a mask.

Simply put, this means that the algorithm is very good at recognizing faces in a large database, even in difficult conditions. The first place means that in these conditions our algorithm works better than other vendors.

It is important to note that the performance of facial recognition algorithms may vary depending on the image quality, the size and diversity of the database, as well as the specific conditions in which the algorithm is used. 

The Verigram solution on real data shows higher results than under strict testing conditions.

If you would like to know more information about our Face Recognition or Liveness algorithms testing results, as well as about detecting fraudulent attacks in a broader sense, please write to us at arman.shuakpayev@verigram.kz .

Comments 3

Login to leave a comment

Значительная часть технологий, связанных с распознаванием лиц и объектов, используется в таких ключевых областях, как безопасность и маркетинг. Регулярное тестирование в NIST позволяет не только подтвердить качество алгоритмов, но и определить области для их улучшения.

Reply

Такие результаты говорят о высоком уровне компетентности и качества технологии, и это может быть важным аргументом при выборе и использовании технологии распознавания лиц.

Reply

В эпоху цифровизации и повсеместного использования технологий распознавания лиц для безопасности и автоматизации, такие показатели действительно впечатляют. Жду больше информации о вашем продукте и его применении в реальных условиях. Продолжайте в том же духе!🌟👏

Reply