Your Face Looks Familiar: Intelligence, Analysis, and Attack Methods on ML in Facial Recognition Systems

[Man]

We live in a world where facial recognition systems (hereinafter referred to as FRS) are used almost everywhere: from age validation and online biometric identification to surveillance and payment in the real world. The technological aspects of such algorithms are evolving before our eyes: starting from simple systems based on basic algorithms, we have moved on to systems using machine learning algorithms, but their development is still far from complete.

In our previous research we touched on biometric identification, and this work is a logical continuation of our dive into facial recognition systems and the exploitation of their shortcomings.

For quick navigation through the article

• The Ethical Side of Face Searching
• Facial recognition systems
  • Review of services for searching information about people by images
    • The Eye of God Platform
    • PimEyes Service
    • Search4faces service
  • Overview of Possible Attacks on Facial Recognition Systems
    • Discovered patterns
• Activation patches
  • Creating a test face detection system
  • We use existing approaches to solve the problem
  • Reflection
  • We create activation patches
    • Creating an Activation Patch for Dlib MMOD CNN
      • Step #1: Loading the model
      • Step #2. Patch creation algorithm
      • Step #3. Tests
    • Creating an Activation Patch for MTCNN
      • Creating a Patch for P-Net
      • R-Net, O-Net
• Conclusion

The Ethical Side of Face Searching​

It is important to note that all information used by the services mentioned in the article is obtained from open sources. This is the data that people themselves have provided access to. Services cannot use data obtained from illegal sources (for example, as a result of leaks or leaks).

Facial recognition systems​

The task of face recognition is decomposed into several subtasks solved by different systems. Among such subtasks are:

• Face detection.
• Face alignment.
• Feature extraction.
• Finding a record with a similar set of features (face matching).
• Making the final decision.

The first step is face detection.
At this step, the algorithm determines whether a human face is present in the data at all and where exactly it is located (all irrelevant objects are excluded). This task can be solved using algorithmic approaches or neural networks.
After detecting a face and performing pre-processing, including face alignment and normalization of illumination and contrast parameters, it is necessary to transform the face into a mathematical representation - in the form of a feature vector. This is the second stage. Such a vector allows us to apply mathematical methods to compare faces, determining their degree of similarity based on the distance between the corresponding vectors.

Modern approaches use algorithms for extracting a wider range of features (e.g. HOG, SIFT, SURF, ORB) and also use convolutional neural networks, which are trained to identify human face features based on data.

Such approaches allow not only to detect a face, but also to create a unique representation of its features in numerical form. The resulting feature vector can then be used in a variety of scenarios, including identifying a person by matching them with a database of similar vectors or searching for and comparing faces. In the context of face recognition applications, such tasks may include identifying a person in a photograph, authenticating a user in a system.

It is worth noting the fundamental difference between an online FR system, where you can upload digital photos and work with the system as a service via the Internet, and offline , when we encounter some FR system in the real world (for example, a video surveillance system or a subway fare payment system using face recognition technology). In online systems, we deal with digital data, which allows us to precisely control the adjustment of individual pixels in the image. This makes such systems relatively more stable and predictable in terms of data processing.

At the same time, offline systems that require physical interaction have specific problems such as nonlinear image distortion, camera thermal noise, and color fidelity. These factors make it difficult to attack such systems. In addition, testing these systems is a separate challenge. Unlike online systems, where it is possible to perform thousands of black-box queries and quickly get feedback, similar experiments with offline systems would require significantly more time.
In this study, we looked at online facial recognition systems using face search services as an example.

Review of services for searching information about people by images​

The Eye of God Platform​

The service's special feature is interaction via a telegram bot, and a subscription is required to search by faces. The service has many scenarios for searching for public information about people, in particular, we are interested in the ability to search for information about a person by their face.

PimEyes Service​

This face search service requires a subscription to access the full information, but the information that the service was able to find your photos is available for free.

Search4faces service​

Most of the available data is taken from the social network "VKontakte". A simple and transparent tool for searching faces from open sources.

Overview of Possible Attacks on Facial Recognition Systems​

Most potential attacks on recognition systems are aimed primarily at the face detection component. These are also commonly referred to as evasion attacks. In this context, two main categories of errors are distinguished:

• A type I error occurs when, despite being shown a photograph of a face, the system fails to recognize it in the image;
• A type II error occurs when the system mistakenly classifies an image as a face when it is not.

In addition to detection attacks, extraction attacks are possible - when the feature extraction algorithm produces a representation vector that does not reflect the real characteristics of the face. As a result, the system considers you a different person (misclassification).
Intentionally introducing specific information to affect the system that processes that information is called an adversarial patch. For online systems, the changes made to images may be invisible or barely noticeable, while attacks on offline systems require significant image adjustments to overcome distortions introduced by the real world.

Similar categories apply to attacks on human and object recognition systems.

1. Patch for cheating VGG16 network.

1. Patch to deceive the YOLOv2 detector.

1. Another patch to fool the YOLOv2 detector .

Research into bypassing facial recognition systems is based on a similar principle:

1. Attack on Dlib HOG .
   The researchers created a patch that affects the HOG algorithm, causing it to mistakenly recognize the patch as a face and not recognize the face behind the patch.

1. Attack on Dlib CNN.
   In this case, the researcher used an open source Dlib face detector, mmod_human_face_detector.dat. He transformed the network from Dlib’s proprietary format to a format compatible with PyTorch. Then, he selected a network layer whose activation maximization was of interest. Using the central pixel of the filter output as a loss function, the researcher calculated gradients and modified the image by introducing random rotations and twists. This approach allowed him to create images that maximally activate specific network layers.

Examples of samples obtained by the researcher

This study demonstrates the potential of using type II error.

1. The work of researchers from Skolkovo, who created patches capable of bypassing face detection algorithms using the MTCNN network.

Photo of researchers without patch

Photo with patch

These studies cover the most popular facial recognition technologies.

Discovered patterns​

Most of the attacks we are considering can be carried out using the white box principle - when the attacker has access to full information about the system. In the real world, almost all systems are black boxes. Without source code and feedback, the complexity of attacks increases significantly.

Convolutional neural networks are trained to detect certain visual patterns, if the attacker understands exactly what features the neural network is looking for, they can exploit this. Machine learning algorithms are used for this purpose, for example to generate adversarial examples. In this case, one neural network tries to deceive another. Finally, all attacks have a specific goal. A method that works well against one network will not work against another.

Having noticed these patterns, we formulated the question: is it possible to use the proposed patches to determine whether facial recognition services use any of the above algorithms? And if not, is it possible to create such patches?

Activation patches​

Let's remember that in the context of testing FR systems, it can be quite difficult to quickly access the internal algorithm that processes the user's incoming data (if this is even provided by the service). If the system is active and accessible, then you can conduct several experiments and quickly get a response from the target system. It would be convenient if we had a tool that would allow us to quickly scan the system for the use of popular algorithms and roughly localize the technology stack. For example, by running a certain set of images through a service or device and receiving its response, we could reliably determine which algorithm is used to detect faces or extract features.

If it turns out that a service uses certain open source tools, then you can prepare patches for any type of attack, be it evasion (the network does not find you) or misclassification (you are classified as someone else), since you essentially have an accurate idea of the model that is being used, and often even the model itself.

An activation patch is an information disturbance (image) that triggers a detector, which helps to identify certain algorithms, models, and systems using the black box method.

Creating a test face detection system​

Before we continue, we need to build our pipeline for testing and validating hypotheses. The principle of its operation is quite simple: we load the necessary face recognition algorithms and sequentially check the photos we need, trying to detect a face on them, and highlight the boundaries around the face if it was detected:

- dlib.get_frontal_face_detector() — HOG;
- pytorch_mtcnn(keep_all=True, device="cpu");
- tensorflow_MTCNN();
- dlib.cnn_face_detection_model_v1("mmod_human_face_detector.dat").

Results of the face detection algorithm

We use existing approaches to solve the problem​

1. Dlib HOG Research to Create an Activation Patch

Photo of researcher with patch

To learn how this algorithm works, you can read the article. It is important for us to understand whether HOG is used in the described services. To do this, you can take the visualized face descriptor from the article and check it in our pipeline and in the available services.

Successful HOG detection

The patch results in the unambiguous activation of only one system and does not result in the activation of others.
We will conduct tests in external systems.

Service	Check result
"The Eye of God"	False
Search4faces	False
PimEyes	False

1. Dlib CNN Research for Activation Patch Creation

Let's check how the samples received by the researcher will pass through our pipeline.

It is immediately obvious that this work causes many errors of the first and second kind: in some places hallucinating faces are detected by many systems, in others - by none. The same is true for public services.

Search4faces found a face

"Eye of God" has discovered a face

PimEyes did not find any faces

Service	Check result
"The Eye of God"	True
Search4faces	True
PimEyes	False

1. Exploring MTCNN for Activation Patch Generation

Let's take the researchers' original footage and run it through our face detection pipeline.

Note that none of the algorithms in our test system detects a face. It can be assumed that it is not the patches that are working, but the simple overlap of facial features that are important for detection.

Let's try to test the observation by taking patches from GitHub.

Mixed results

It can be seen that simply overlapping certain areas already results in all three detectors not being able to correctly detect the face area. However, when the image is too small, all detectors detect the face.

We will test original photographs using services.

Service	Check result
"The Eye of God"	False
Search4faces	False
PimEyes	True

Reflection​

Experiments using finished works showed that:

1. While patches are designed to activate or bypass specific systems, they don't always distinguish those systems from others. For example, hallucinating faces can be detected by God's Eye and Search4faces based on the fact that the images actually look very much like faces. At the same time, patches for MTCNN can bypass detection in many services because they cover half of the face.
2. It is not quite correct to call all patches activation patches. The patch for MTCNN, although it can be used to identify MTCNN, does not activate the detector, but suppresses its activation.

We create activation patches​

Creating an Activation Patch for Dlib MMOD CNN​

Our task is to create an image that will trigger the Dlib detector. It is widely used, in particular, by the face_recognition framework.

Step #1: Loading the model​

First, we need the ability to perform mathematical operations on the model. Dlib has its own format for storing models, but it is also possible (thanks to this enthusiast) to load the model into PyTorch and perform any operations on it, in particular, apply forward and backward passes.
Forward pass - data (in our case, an image) is fed to the input layer of the network, all mathematical calculations are performed in the hidden layers, and at the end the result of data processing is output - the area where the face was detected.
Backward pass — the presence of a loss function (MSE) allows us to estimate the contribution of each input parameter to the result obtained from the model relative to a certain rule (loss function). To be more precise, the gradients of the loss function are calculated relative to the weights of the layers — from the output to the input. This process uses the chain rule of differentiation, allowing us to calculate the influence of each weight on the final error.

Once the model is loaded into PyTorch, we can perform forward and backward passes, meaning we can quickly and very accurately use gradient descent to find a unique pattern that the model will perceive as a face.

Step #2. Patch creation algorithm​

We have a network and we would like to create a unique pattern that will be detected by the detector as a face, but will not be one. Let's take a reference image and perform a forward pass through the model, remember the output and visualize it.

Reference image and model response

We need to generate a disturbance that produces a face-like response.

Picture with question and model response

The principle of creating a patch for this model is to pass a reference photo through it, which has a face; to remember the output; then, using the loss function (MSE) and gradient descent, we generate an image from the noise, which, when passing through the network, will create a response that is as close as possible to a real face, while visually the input data will not be a face.

GIF of creating a picture

Graph of the loss function

Note that it is better to stop the algorithm immediately after the first triggering of the original model: the more optimization iterations, the higher the probability of triggering other models. We are not very interested in creating hallucinating faces, so we will take this feature into account.
This algorithm can be applied to any model and system in which the error gradient can be calculated using a computational graph. Thus, a patch can be made for each detector that will allow it to be detected (provided that the detector can be loaded into a deep learning framework).
There are many technologies, but we will limit ourselves to the most popular open source solutions for facial recognition.

Step #3. Tests​

Let's check the patch's performance on a real model without transformation in the PyTorch environment. To do this, we'll run a test in our pipeline.

Testing samples on a real photo

Testing with partial face occlusion

The patch successfully attracts only the model for which it was made, while other detectors do not react to it. This feature distinguishes our patch from hallucinating faces, to which all detectors worked.

With a certain patch application, it is possible to hide the main face, as was the case with HOG, but such use requires modification of the technology, and our task is to detect it.

Service	Check result
"The Eye of God"	False
Search4faces	False
PimEyes	False

It can be concluded that the services do not use the MMOD model from Dlib to detect faces. Patches work for a specific model, if you retrain it from scratch, the result will be different.

Creating an Activation Patch for MTCNN​

MTCNN, or multi-task cascaded convolutional networks, is a method for face detection that uses a cascade of convolutional neural networks to perform multiple tasks in sequence.

The system works in three stages, each of which uses a separate convolutional network that specializes in different aspects of the task:

1. P-Net (proposal network) is the first network responsible for proposing candidates for faces in an image, performing a fast search over the entire image and creating square regions that may contain faces.
2. R-Net (refinement network) is the second network that refines proposals by filtering out most false positives and refining the positioning of faces in the image for subsequent accurate processing.
3. O-Net (output network) is the third network, which performs the final refinement of the positions of detected faces and simultaneously works on recognizing key facial points (eyes, nose, mouth).

These three steps combine to form a powerful tool that can accurately recognize faces in images of varying quality and lighting conditions. In addition, MTCNN can operate in real time, making it popular and applicable to a wide range of tasks. It would be nice to have an activation patch for this network architecture. This task is certainly more complex than it was with Dlib, but the general concepts remain the same.

Creating a Patch for P-Net​

MTCNN has a more complex structure, so you need to dive into the network's internals to conduct an attack. It will no longer be possible to simply remember the outputs and repeat them. For our experiment, we will take the most common implementation of MTCNN on PyTorch — from facenet_pytorch import MTCNN.

It is necessary to follow step by step how the processing of the photo is carried out within the network. The first thing the network will do is to build a pyramid, where the picture will be transformed into a series of smaller pictures.

This is necessary so that the same network can find both large and small faces in the same photograph.

Photo taken from the articlehttps://arxiv.org/pdf/1604.02878.pdf

Next, this group of photographs of different scales is sent to the first network - P-Net.
P-Net will go through all the photos and give out zones where the face may be. If we visualize them, we will see the following.

The file name indicates the scaling factor at which the transformation was performed.
Let's take a reference photo.

The P-Net activation map will be as follows.

We save this data in a separate file, and also save the value of the scale factors - all this will be needed when creating a patch.
We had to patch the library a bit to intercept a few parameters.

What we will need:

• P-Net outputs.
• Values of scale factors.
• The final region of the detected face.
• Keypoints of the detected face.

Note that the reference image only serves to model the desired response from the network and can be any sample on which the network recognizes a face.

Next, we'll load just one P-Net network and write a simple optimization function, the purpose of which will be to generate the most similar response to what we've already successfully intercepted. Then we'll need to create an image from a random set of pixels, which will become a patch during the optimization process.

Let us remember that our task is to ensure that the result that the neural network produces and that it considers to be a real person is not such.

The function to bypass the first network looks like this.

```python
#P net optimization step  
for count, scale in enumerate(scales):  
 
    input_resampled = imresample(input, (int(N * scale + 1), int(N * scale + 1)))  
reg, probs = mtcnn.pnet(input_resampled)  
iter_pnet_loss = loss_pnet(probs[:, 1], outputs_model[count][:, 1])  
history_loss_pnet[count].append(float(iter_pnet_loss))  
pbar.set_description(f"Loss {scale} - {history_loss_pnet[-1]}")  
iter_pnet_loss.backward()  

# Gradient normalization
grad_cpu = input.grad  
grad_cpu = grad_cpu / torch.norm(grad_cpu, 2)  
lr = 2  
input = input.clone().detach() - lr * grad_cpu  
input.requires_grad = True
```

It is necessary to reduce the image (patch) in accordance with the scaling factors that were intercepted during the network pass of the reference face image. Next, it is necessary to perform a forward pass in P-Net, and then calculate the error, which consists of the standard deviation between the previously intercepted responses and the data obtained as a result of the forward pass in the context of each specific scale. Then it is necessary to perform a backward pass, calculate the gradients and go in the opposite direction, reducing the MSE (that is, making the network output as similar as possible to what was intercepted earlier). We got the following patch.

Created patch

Now we need to see how similar our patch's response is to the original. The figure shows the original face with outlined areas (called face candidates and subsequently fed to R-Net).

P-Net activations using reference face

Activation of P-Net by the created patch

The data is very similar, which can also be seen from the decrease in the loss function exponent.

Now we have a patch that causes a response as if the original face is in the image, when it is not. The primary candidates for faces are selected, the areas that will be fed to R-Net to assess how similar they are to faces are outlined. We have bypassed the first network.

It is important to understand that the patch must simultaneously take into account the features of all networks; changing it only for the passage of R-Net will irreversibly change the response of P-Net.

It's worth talking about the other two networks.

R-Net, O-Net​

R-Net only evaluates the selected candidates (photographs obtained after P-Net). It checks to what extent the face is actually in this zone. At first, an approach was chosen according to which it was necessary to optimize each candidate. That is, in each area, through gradient descent, it was necessary to maximize the network's confidence in each candidate, but this, of course, changes the picture, and P-Net will no longer select this zone as a candidate for the face at the next stage of optimization. What to do?

The solution is quite simple: perform a forward pass of the reference face through all three networks and save the final MTCNN decision about where the face is (i.e. the final output of the detector).

Now we use only the area in the optimization that is highlighted in the photo above, because it is in this zone that the face should be (according to the detector's results). Using gradient descent, we need to explain this first to R-Net, making it so that the probability of the assessment in this zone is close to one, and the same needs to be done with O-Net.

So, to create a patch, you need to follow several steps:

1. Initialize a random image that will change during optimization.
2. Pass the image through the P-Net network with all resizing (forward pass).
3. Calculate MSE loss relative to previously intercepted responses from the reference image.

At this stage, a group of candidate faces appears for verification in R-Net. We reject them and propose our candidate, and the only one. This is the area where there is definitely a face, it is also the final area that we received from the network in response to the reference photo.

1. Increase R-Net's confidence that a face is definitely present in a given region by minimizing the MSE between the network's output and one (since one is the maximum confidence), calculating gradients relative to this error, and make a change to the image.
2. Do the same with O-Net.

All this will need to be repeated several times until the loss function estimates reach minimum values.

Scheme of one optimization iteration

Graph of the loss function

The graph shows the loss for all P-Net scaling factors and the errors for O-Net and R-Net.
We get a patch like this.

Patch for MTCNN

Activating P-Net using a patch

We immediately perform tests in our system. The MTCNN implementation in PyTorch and TensorFlow was used.

Testing the patch on a test system

Unfortunately, the patch does not work on any public service we tested.

Service	Check result
"The Eye of God"	False
Search4faces	False
PimEyes	False

Note that the inference of this model on OpenCV in C++ also works on this patch, this may mean that the same model for which we prepared the patch is used there, only transformed and loaded by a completely different computing environment.

Testing a patch, inference in C++

Conclusion​

We have developed a method that allows us to identify machine learning algorithms used in facial recognition systems, even if these systems are black boxes. Our approach is based on the creation of activation patches that, when interacting with the system, allow us to determine the technologies used by their reaction. We have created patches for popular facial detection algorithms, and these patches can be used for research and practical purposes.

HOG patch

Patch for MMOD Dlib model

MTCNN patch

Photo Patches.png - here are the ready patches. Take and use.
More options are available in the GitHub repository.
The main question is: if activation patches really work, then why couldn’t we detect any system using the black box method? The answer is quite obvious: the services we tested do not use open source, namely, specific implementations of the models that were considered in this study.
The results may be of interest:

1. Researchers, white hat hackers.

The paper demonstrated a general algorithm for creating an activation patch. This algorithm will allow anyone to create such a patch if they have access to the model. Of particular interest is the creation of a patch using the black box method, i.e. based on the analysis of responses to queries. However, this topic is beyond the scope of our article.

1. Developers.

An extremely interesting question that comes out of the study is how to protect against such attacks. We tried several methods and got different results:

• Overlaying random noise on an image. The hypothesis was that overlaying random noise would break face detection on an activation patch, but not on a normal face. However, when adding 5% Gaussian noise, the patches for MTCNN remained functional.
• Training a model from scratch. Probably, training a model with weights different from those presented in open source allows achieving the same effect - activation patches generated for the original model stop working.

1. Ordinary users.

The main advice that can be given based on the results of the study is to be conscious when providing access to your data; algorithms for processing information change very quickly, and it is unclear what information people will learn to extract from data in the future and what consequences this may lead to.

1. Civil liberties activists, preachers of anonymity.

One of the interesting conclusions of this work was the creation of the term ethical ML . The idea is to create a new approach to the development of facial recognition systems - embedding an open-source component into the data processing process, the task of which is to detect photographs with a certain visual pattern (patch) and refuse to process such images.

Refusal of cookies, disagreement with the data processing policy on websites, the right to be forgotten - a number of these mechanisms that were introduced by the Federal Law "On Personal Data" and the GDPR can be supplemented by a new approach to the development of facial recognition systems.

The advantages of such a standard would be:

• Independence from the main facial recognition system.

Since it is a standalone component, it can be easily integrated with various systems without changing their core functionality.

• Respect for personal privacy.

This approach allows people to actively manage their privacy, as users can then explicitly indicate that they do not want to be identified.

• Easy to implement and update.

Because it is a separate component, it is easier to update and adapt to new types of patterns or changes in privacy standards.

• Compliance with ethical standards.

This approach allows companies and organizations to comply with ethical standards related to the protection of personal and confidential data.

• The patch can be used to check whether a particular service follows ethical ML standards.

Thank you for reading to the end

Source