Removing password from PC keyboard in 0-60 seconds by heat trace

[Man]

Modern thermal imagers connect to a computer and record the temperature of each pixel in CSV. This data is great for training a neural network. A few months ago, scientists from the University of Glasgow conducted an experiment and developed a really effective ThermoSecure model for recognizing pressed keys by heat trace.

Scientists came to the conclusion that it is possible to relatively reliably remove entered passwords from the keyboard 0-60 seconds after entering characters. Let's see how the machine vision system works step by step.
The experimental setup is shown in the photograph above, with each object labeled. Specifically, the experiment used a €3950 Optris PI 450 infrared camera (A) (resolution 764×480, 80Hz, 40mK NETD, temperature range from -20° C to 100° C, in photograph A), mounted on a tripod next to a Microsoft Wired ISO Keyboard 600 (QWERTY) with ABS keycaps (B). The tripod height and the distance to the object were varied in each session in the ranges of 60–90 cm and 50–121 cm, respectively. In addition, the rotation angle of the keyboard was varied.

On the screen of laptop D, the participant was shown the characters to be entered, while another laptop (C) recorded 16-bit thermal images from the camera. For each thermal image, the Optris APIs wrote a CSV file with pixel temperatures.

ThermoSecure Architecture​

1. Keyboard localization: data collection and annotation, training a model for reliable keyboard recognition in a frame.
2. Keystroke logging: Recognizes the layout, keys pressed, including those pressed multiple times (they are warmer than those pressed once).
3. Determining the order of keystrokes, recognizing the authentication stage against the background of general interaction with the system, that is, against the background of normal operation.

Keyboard contour marking during the annotation process for model training

Before entering the machine learning model, all images undergo pre-processing to suppress noise. This processing includes four main steps:

1. 5x5 median filter overlay
2. Grayscale Conversion
3. Re-imposing the median filter
4. Contrast-Limited Adaptive Histogram Equalization (CLAHE) (doi: 10.1109/ICACCI.2014.6968381)

Four stages of image preprocessing

In the second stage, a Rotated Bounding Box (RBBox) contour is superimposed from Mask R-CNN, a special neural network that performs image segmentation. From this contour, the coordinates of the four vertices of the rectangle are extracted. The model then uses a template keyboard configuration (4 regions, 6 rows) to calculate the coordinates of the keys and associate temperature clusters in the frame with the location of specific keys that are used to enter the password.

Finally, in the last stage, the model uses the coordinates of the keys obtained earlier and the temperature data from each thermal image to obtain the mean, minimum, and maximum temperature of each keycap. These three values (mean, max, min) are then averaged for each key to calculate the correct order in which they should be pressed.

This procedure also includes several stages. After separating the username entry phase from the password entry phase, a list of password characters is determined. Then, the temperature data of each key is used as the transition probability between different states (keys). Using the transition probability and the state probability, the probability of each sequence is calculated, as shown in the following diagram:

In this case, the sequence has the highest probability6fbx9p, although all options are calculated:

( 'Password: ', '6 fbx9p', ' with a probability: ', 0.40740188103110087)
( ' Password : ', '6 fxx9p ', ' with a probability : ' , 0.4071952200350172)
( 'Password: ', 'ffbx9p', 'with a probability: ', 0.40670664210825924)
( ' Password : ', '6 fb99p ', ' with a probability : ' , 0.40659163972726375)
( ' Password : ', '6 f9x9p ', ' with a probability : ' , 0.4065814008654436)
( 'Password: ', 'ffxx9p', 'with a probability: ', 0.40649998111217556)
( ' Password : ', '6 fx99p ', ' with a probability : ' , 0.40638497873118007)
( 'Password: ', '6 bbx9p', ' with a probability: ', 0.4059259564615335)

Based on the results of the experiment, the scientists calculated the probability of recognizing a password based on the heat trace depending on the password length and the time between entering the password and capturing the heat trace. As we can see, short six-character passwords are recognized with 100% accuracy 20 seconds after being entered, and with 83% accuracy 60 seconds after being entered. Other results are shown in the table:

The faster a person types, the lower the probability of recognizing the correct sequence of pressed keys. The accuracy of password recognition increases for users who type hard on the keyboard.

The researchers also concluded that ABS keycaps heat up much more than PBT keycaps, and therefore are more vulnerable to capturing a heat trace.

The scientific article was published on September 15, 2022, in the journal ACM Transactions on Privacy and Security (doi: 10.1145/3563693).

As for the practical conclusions from the study, in public places it is better to cover the keyboard not only when entering a password, but also for 60 seconds after that. Since thermal imagers have become inexpensive mass-produced devices, intruders can use them for espionage.

Source