Sound Keylogger. Determine Pressed Keys by Sound.

[Man]

It is no secret that each key on the keyboard has a unique sound, which depends on its location and other factors. Theoretically, spectrogram analysis allows you to distinguish keys from each other, and by the frequency of pressing, determine which symbol corresponds to each sound. The task of recognizing sounds is simplified by the fact that when typing a coherent text, symbols are well predicted by the dictionary (namely, by the frequency of n-grams for texts).

A two-dimensional spectrogram of a "click" of a single key on a mechanical keyboard looks something like this:

Each key has a slightly different spectrogram.

A sound keylogger is a very interesting task from a security point of view. It is close to fingerprinting a user by their keyboard handwriting (including typing speed, typos, timing between key combinations, etc.).

Theoretically, this makes it possible to:

• register keystrokes by sound;
• identify the user who is working at the keyboard.

All this in the absence of a visual channel, that is, simply by voice communication. For example, during a telephone conversation.

In recent years, several conceptual developments have been developed in this area. One of the first was a tool called keytap by Georgi Gerganov (2018). It is trained for a specific user (the sound of a specific mechanical keyboard). It also works through a browser. You need to enable WebAssembly pthreads and SharedArrayBuffer support in the browser, and also give permission to listen to the microphone, there is a demo.

To test this keylogger, it is recommended to start with two keys whose sound is as different as possible, that is, as far apart as possible. For example, qin pthe case of the QWERTY layout. If the recognition result is less than 100%, the keylogger does not work.

A year and a half later, the same author released another tool, keytap2, which works differently. Instead of learning, it uses statistics on the frequency of letters and n-grams (character sequences) in the English language.

For a successful analysis, it is enough to type a few sentences of coherent text in English - and the program will begin to determine which sound corresponds to which key. See a short video demonstration here.

The online demo has certain limitations. It only works for texts in English. You need to type at least 100 characters of meaningful text. Random keystrokes will not give results. The program works best on loud mechanical keyboards. It is advisable to turn the microphone to maximum sensitivity.

Recently, the most advanced version of keytap3 was released with more accurate n-gram statistics. It still works reliably only with mechanical keyboards and identifies text in English, but it is now much more stable. For example, you can open a page with a demo version of the keylogger on your phone - and put it next to the keyboard.

All the keyloggers mentioned are amateur projects of one enthusiast. Of course, with a competent approach involving qualified specialists and professional equipment, much better recognition accuracy can be achieved.

There are no such tools in the public domain for the Russian language yet. Although a linguistic database for it exists, including the National Corpus of the Russian Language (4.5 million texts), from which one can obtain the frequency of n-grams in Russian-language texts. Obviously, the frequency of keystrokes in the Russian-language layout is very different from the frequency of letters in the English language. It is quite easy to determine from the heat map what language the user is typing in (at least, it is easy to make a binary choice for the Russian/English pair).

Let us repeat that existing tools recognize text only on a loud (mechanical) keyboard and from a close distance.

To prevent information leakage via the audio channel, you can use utilities like Unclack (for macOS) and Hushboard, which automatically turn off the microphone while typing on the keyboard. In addition to security purposes, they also perform a more prosaic function - automatic elimination of extraneous noise ("clicking") during video/audio conferences.

Source