Essential fuzzing

[Father]

Fuzzing is an integral part of the secure development cycle. This is the procedure of feeding a large number, on the order of a billion, samples of input data to the program code that processes them. At the same time, for each sample of input data, a separate code run is performed and its correct operation is verified.

Fuzz testing is the automated detection of implementation errors by sending multiple data samples and analyzing the program's response to them. It is most effective when fuzzing goals are correctly identified and long I / O operations can be eliminated.

Sergey Polunin
Head of the Infrastructure IT Protection Group at Gazinformservis

Fuzzing is always effective, because it's hard to imagine the entire secure development cycle without it. This is actually a method for automated code error detection. You simply send pre-prepared but completely incorrect data to your program and analyze the program's response to it. All sorts of very critical errors like divisions by zero, out-of-bounds arrays, segmentation faults, or race conditions are pretty well detected just during fuzz testing.

The fuzzing testing procedure has two goals:

• generating a set of input data samples that leads to the execution of the largest number of code fragments (achieving a large coverage across lines of code);
• mutating input data so that it does not meet the restrictions and, hypothetically, could lead to undeclared behavior and checking the correct functioning during their processing.

Evgeny Malkin
Head of the Secure Development Department of the Information Security Competence Center of the company "Garda Technologies"

The key thing for a fuzzer is the presence of feedback, which the tool can use to determine the search for new paths, so the "black box" options without instrumentation are initially less effective. You can use different types of fuzzing, but different approaches to creating samples with knowledge of the input data structure will increase efficiency.

You can increase the effectiveness of fuzzing by selecting goals and writing wrappers with modifications for acceleration. At this stage, they resort to "bypassing" or disabling integrity checking, or use sample generation using a given data format/protocol.

Three mailboxes​

There are three methods of fuzzing testing: white box, black box, and gray box. They differ in the available resources of the object under test.

Angelina Razorenova
Senior Information Security Analyst at the Digital Economy League

The white box method uses all available resources, including the source code. Black-box method — only input data and results obtained. The most effective method is the Gray-box method, which uses the results of analysis of the available binary code using RCE in addition to the information detected by the black-box method.

Speaking about the effective use of fuzzing, experts note that when choosing the type, the correct task is set. For example, fuzzing the API of a web application to search for SQL injection via whitebox fuzzing does not make sense, since this process will take a long time to write code and wrappers to test the theory. It is easier and faster to" run " blackbox fuzzing with predefined dictionaries on a running application to check for SQL vulnerability. However, if a library has been implemented that sanitizes input data before sending it to the database, then it is advisable to check it using code fuzzing.

Dmitry Tishkin
Head of the Application Security R-Vision team

If we consider black-box fuzzing, it is effective when searching for standard vulnerabilities, especially server side vulnerabilities, as well as for checking the application's behavior when entering non-standard characters and for conducting negative testing.

White box fuzzing (code-level fuzzing) is useful when checking code sections that have a high cyclomatic complexity, for example, to check that the code performs exactly the functionality embedded in it and there is no error in the conditions.

Code-level fuzzing is suitable for checking various parsers, sanitizers, and validators for their correctness. It also allows you to find errors such as:

• getting into incorrect conditions;
• infinite loops;
• regular expression traversal;
• bypassing the validity of validated values;
• buffer overflow.

In addition, phasers are divided by goal, for example, source-based, when there are project sources, or binary-based, when there are no project sources. Fuzzing can also be divided into two types based on the presence of feedback from the application under test – there is a feedback driven response or there is none . Well, for operations on input data. Generational, mutational, and combined methods.

Effective fuzzing​

All types of fuzzing are effective if applied correctly. Each type solves specific tasks. Experts also note that fuzzing is highly effective with continuous testing. Therefore, we recommend integrating fuzzing into CI/CD pipelines. This provides short feedback loops and allows developers to fix security vulnerabilities before they become a problem.

Alexander Drozdov
Engineer for SDL and Information Security technologies Axiom JDK

Fuzzing is primarily relevant when a software product has an interface that accepts user data as input (for example, a form with fields for entering text or the ability to upload a file for processing it). Fuzzing is usually used to check the correctness of the input data parser, which has a large number of branches during execution. However, you can use fuzzing to check the correctness of any code that processes input data (even if it doesn't come from the external interface).

Most effective fuzzing:

• it uses the response from the program under test (criterion 1). Thanks to this, mutation of input samples will not be performed "blindly", but taking into account the results of processing previous samples. This, in turn, will allow you to achieve greater coverage across lines of code;
• performed when using error sanitizers, which will allow you to detect errors that do not lead to a program crash, but can be a source of a serious problem, up to a security vulnerability;
• performed at high speed, this significantly increases the chances of finding something;
• when generating input data, it takes into account the format of input data. This reduces the number of "idle" fuzzing iterations (when input data does not pass the first stage of verification);
• it relies on information about the coverage of only the code we are interested in.

In addition, efficient fuzzing has good stability – when fuzzing testing, it is recommended, if possible, when collecting coverage from each iteration, not to take into account all sources of entropy that occur during code execution, for example, random number generators. If stability is low, efficiency is lost when working on the coating response.

Advantages and disadvantages​

The advantages of fuzzing are the ability to find errors in complex code that has a large number of branches, and perform this automatically. These are errors that are almost impossible to detect with manual and static analysis. After preparing everything necessary for fuzzing, only machine time will be spent, and it is cheaper than human time.

Fuzzing helps you find deep bugs and vulnerabilities that developers didn't even know existed. Another advantage is the ability to perform a basic vulnerability search using popular dictionaries, which are also used by attackers to increase the application's resistance to hacking.

The pitfalls of fuzzing vary depending on the programming language in which the code under test is written.

Alexander Drozdov
Engineer for SDL and Information Security technologies Axiom JDK

When working with C/C++ code, there may be problems at the compilation stage with instrumentation due to incompatibility of the existing build system with the programming compiler. When linking, undefined symbols may appear in object/executable files for various reasons – using the wrong set of compilation tools, using map files for linking in the build system, setting different visibility levels in the program code. Error sanitizers may interfere with the correct operation of the code under test. More general examples are stability problems due to sources of entropy in the tested code, difficulties when trying to separate the tested code from the entire project with strong connectivity of components.

Dmitry Tishkin
Head of the Application Security R-Vision team

Fuzzing testing requires both large computational and human resources. Because someone has to check fuzzing reports, prepare fuzzing goals, and automate and maintain fuzzing testing. And in the case of whitebox fuzzing, resources are needed to prepare wrappers and to conduct it, which can last for days.

The main disadvantage of fuzzing is that it requires specialists who have a high level of expertise – in the product under test, computer security,low-level details of the program code and its compilation, and nuances of operating systems. You should also consider the cost of hardware resources required for fuzzing.

When is fuzzing not effective?​

Fuzzing helps you find errors that cannot be detected using other testing methods or manual auditing. It is cost-effective and scales easily. Despite its versatility, like any other tool, it may not be effective in certain cases.

Fuzzing may not be effective or applicable in the following cases:

• if the program has no input data or cannot be started without special conditions;
• if the program works with data that cannot be changed (for example, protected files or data stored in encrypted form);
• if the program is too complex and has many possible combinations of input data, which makes fuzzing inefficient and unnecessary in terms of time and resources;
• if the development team does not have sufficient knowledge and skills to conduct fuzzing or cannot fix the detected errors and vulnerabilities.

Denis Isangulov
Head of Testing at NGR Softlab

There is no single best fuzzing method, as each approach has its own advantages depending on what needs to be tested in the program. Some methods can be fast (random and symbolic fuzzing), but do not detect complex errors, while others-hybrid and mutational fuzzing-can be more effective in detecting different types of errors.

Using only fuzz testing can lead to missing real errors or false positives. There is no universal method. Each approach has its own advantages, depending on the specific testing goals of the program.

Conclusion​

On the one hand, a simple and effective solution, but on the other hand, it is complex and requires deep knowledge. Fuzzing is currently one of the most effective dynamic testing techniques. This is no longer just a tool, but a whole approach to testing, which has become the topic of many modern scientific papers and conferences, the peak of the development of technologies for automatic validation of complex software products.

Experts believe that the main rules for effective fuzzing are continuity, a properly chosen fuzzing method for the task, and combining it with other types of testing.