Father
Professional
- Messages
- 2,602
- Reaction score
- 850
- Points
- 113
The article describes the principles and methods of programmatic data recovery for users with no special knowledge.
If the value of the information is very high and / or the storage medium has physical damage, you should think about contacting specialists. In this case, the probability of complete data recovery will be the highest.
When restoring data yourself, you should take into account the following information:
Disc structure
For a better understanding of further actions, let's talk about the logical structure of the disk and file systems. If you are not interested in theory and need to quickly recover data, you can skip the section and go directly to the algorithm of actions.
MBR
Master Boot Record - the master boot record - is located in the zero sector of the disk, contains a part of the executable code, as well as information about the four partitions and which partition is bootable.
If the MBR is healthy, then after gaining control, the code contained in it reads the partition table and transfers control to the code contained in the first sector of the boot partition (VBR - Volume Boot Record), if it contains the 55AAh signature at the end of the sector.
Otherwise, an error message may appear, for example: "Reboot and select a proper boot device" or "Invalid partition table" or "Missing operating system". All of the above messages can occur both due to a malfunction in the MBR or VBR, and for other reasons (the wrong boot disk was selected, etc.).
To repair a damaged MBR or boot partition, there are specialized utilities such as testdisk (Windows) and gpart (Linux), you can also use the "Recovery Console" from the Windows installation disc. Their use will not always give a positive result when recovering data, since in addition to damage to the MBR and VBR, there may be other malfunctions, so these options will not be covered in the article. Although there are plenty of instructions on how to troubleshoot such problems on the network, it is recommended that you first restore important data from the drive, and then try to restore it to its correct operation.
File systems
Any section containing a file system consists of the following parts:
Let's take a closer look at the most popular file systems on user computers: FAT and NTFS.
NTFS
The structure of the NTFS file system is shown schematically in the figure.
The section begins with a boot area that contains information about the section as well as executable code. A copy of the bootloader is most often located at the end of the section.
The next area is the Master File Table (MFT). It is in it that information about directories, files and their attributes is stored. Usually a disk area is reserved for MFT, equal to 12.5% of the partition size. The size of the reserved area can change (up or down), and the table itself can be fragmented.
In addition, a certain area of the section contains a copy of the first 4 service records of the table.
The user data area takes up all of the remaining space on the partition.
When deleting a file using standard OS tools, only a mark is put in the file record that the file has been deleted, and the space it occupies is marked as free. If after deleting the file no writing was made to the disk, the file remained in place, and its recovery is possible.
When formatting a partition, a new MFT is created in place of the old one. Initially, the size of the new table is rather small (several hundred MFT records), so some of the service records of the previous file system can still be restored. The more files were written to a formatted partition, the less chance of successful data recovery.
In this case, the data physically remains in its place, and part of the information about them that was stored in the previous version of the MFT is also retained. These files cannot be read by standard OS tools.
Complete formatting is sometimes referred to as low-level formatting, which is a mistake. Low-level formatting is a technological operation that is performed during the manufacture of a drive, and it cannot be performed using software methods.
In fact, only two types of high-level formatting are available from the operating system: full and fast. Fast formatting simply generates the partition table, and with full formatting, the entire disk is first checked for bad sectors. In Windows XP, this check is performed using a read operation (that is, the data remains in place, which means that in terms of recovery, this situation does not differ from the quick format discussed above), and in Windows 7, sectors are checked by writing, and the data is destroyed irrevocably. and there is no way to restore them.
FAT
The structure of the FAT file system is shown schematically in the figure.
At the beginning of the section there is a VBR, its copy is usually located through 6 sectors. After a certain number of sectors, the file FAT (File Allocation Table) itself is located, followed by its copy.
The file table stores information only about the chains of clusters that are occupied by files. File names and attributes are stored in directories located in the user area.
The user area in FAT starts at the root directory, all other directories and files are located in it. Directory entries point to the first entry in the file table that contains information about the clusters in the file.
When a file is deleted, the first character of the file record is changed to a special code, which means that the file has been deleted. Also, clusters are marked as free and information about the chain of clusters occupied by the file is deleted, which complicates the procedure for recovering fragmented files. The data area remains unchanged, meaning the files can still be recovered.
When formatting, a new file table and root directory are formed, the size of the table is indicated at its beginning. The data itself, in general, remains on the disk. What follows on disk after the newly created table remains intact (that is, information about the location of the files that were on disk before formatting). As new files are written, directory and file structures are overwritten with new data, reducing the likelihood of restoring existing information.
Data recovery methods
There are both purely software methods of data recovery and software and hardware. For the latter, special expensive equipment, relevant experience and knowledge are required, while software recovery using automated utilities is available to almost any PC user. It is this method that will be described below.
Recovering deleted files
To recover files deleted by standard operating system tools, you need to read the service data bypassing the file system interface. In doing so, you can get information about the location of files that are marked as deleted.
If there was no subsequent writing to the disk, then this location will be able to read the desired file.
Reconstruction of a damaged file system
This method is used when the file system has been damaged or formatted. To reconstruct the file system, it is necessary to scan the entire partition to find remnants of service data, on the basis of which the file table will be reproduced and, if successful, access to the files and folders stored in the partition will be obtained.
Recovering after changing the structure of disk partitions
Specialized utilities scan the drive for the presence of file system structures that are on the drive. Based on the scan, a list of possible file systems is built with a preliminary assessment of their state. The next step is to check the found systems for the necessary data.
Recovery by signatures
RAW-recovery - recovery by signatures, used in cases where other methods have not yielded a satisfactory result. In this case, a sector-by-sector scan is performed on the drives for the presence of known signatures (a unique set of characters typical for a particular file type).
No names, logical locations, or attributes are known for the files found. If the files are fragmented, this recovery method will be ineffective.
If the signature of the beginning of the file was found, then the next task is to find the end of the file. Typically, any of the following known signatures are used for this, which may give an unsatisfactory result. Modern programs use methods that use leftover data about the file system and its features to improve the result. In some cases, such algorithms can help to recover even a fragmented file, which is impossible using standard algorithms. For example, the IntelliRAW algorithm used in the UFS Explorer family of programs works in conjunction with file system reconstruction algorithms and uses this information to determine the end of files. This implementation achieves better results than using simple signature-based recovery methods.
A rough recovery can give a good result when the files on the drive are arranged sequentially, without offsets and fragmentation. For example, when recovering files from memory cards of cameras, camcorders, etc.
Data recovery algorithm
We will give the algorithm using the example of the free R.saver program, which is available for non-commercial use in the territory of the former CIS. It is compact, easy to use and does not require installation. Uses algorithms of commercial software UFS Explorer, gives results at the level of professional utilities. The file systems are NTFS, FAT and exFAT.
You can download it from the support page on the official website: http://rlab.ru/tools/rsaver.html.
The archive with the program must be unpacked into a partition different from the one from which the recovery will be carried out. If there is only one partition in the system, the best solution would be to connect the disk with the recovered data to another computer. If this is not possible, you can unpack the program to an external drive (if it is large enough to store the recoverable data).
Selecting a section
After unpacking the program, you need to run it. In the left part of the main window, you can see the drives connected to the system and partitions on them. If the desired section was automatically found and displayed in the list, select it and go to the next item.
If the desired partition is not in the list (for example, the drive was formatted or partitions resized), you can start a search for a lost partition or set it yourself (only if you know the exact parameters of the partition). You can select these functions in the drive's context menu or on the toolbar.
Let's take a closer look at the lost partition search function:
After its launch, a new window will open, in which you need to click on the "Find Now" button, which will start a search on the disk for known file systems. As they are found, the list will be supplemented with information about the found sections. The most likely options will have a blue icon, which means that the partition is in good condition. If the icon is yellow or red, then the found partition is either badly damaged or found by mistake (some file types can have this effect).
From the list of partitions, tick the boxes that best match the parameters of the desired partition: start (in sectors), size (in sectors and megabytes), label, file system type.
If the value in the "size" column is less than the value in the "start" column, then most likely this section is not valid. Typically these "invalid" partitions are disk images stored on the drive.
It is recommended to save the scan results before continuing so that you do not have to scan the disk again if you make an incorrect selection.
Click the Use Selected button to continue.
Scanning
Next, a file manager window will open, which will show the current state of the file system (that is, files and directories visible by the OS). Click the Scan button to start scanning.
If you have selected a partition with the NTFS file system, you will be offered a choice: run a full or quick scan.
The scope of the quick scan is to search for files deleted by standard means of the operating system. This scan only reads service records in the file table and analyzes them.
Full scan is used in most cases. At the same time, the search for the remnants of service records in the file table, virtual reconstruction of the file system and sector-by-sector scanning using IntelliRAW technology are performed. Gives excellent results, but takes longer.
For FAT16 / FAT32 and exFAT file systems, only full scan mode is available. This is due to the fact that, due to the peculiarities of these file systems, in many cases, a quick scan is not enough for effective data recovery.
Saving results
After the scan is complete, the program will show the result of the reconstruction in the form of a tree of files and folders that it could find.
Blue icons display elements that are visible by standard operating system tools, red - those that have been deleted or lost and are inaccessible by standard operating system tools.
In addition, service folders will be available:
To save the recovered data, either select the "Copy to ..." item in the context menu of an element, or click the "Mass selection" button on the toolbar, then select all the necessary elements and click the "Save selected" button.
Remember that you only need to save the recovered data to a partition that is different from the one from which the recovery is being carried out - otherwise the data may be damaged.
It is recommended to save the scan result by clicking the corresponding button on the toolbar so that you do not have to re-scan the section if you did not save all the necessary data for the first time.
If the value of the information is very high and / or the storage medium has physical damage, you should think about contacting specialists. In this case, the probability of complete data recovery will be the highest.
When restoring data yourself, you should take into account the following information:
- Writing to a partition with recoverable information can reduce the likelihood of data recovery, since the data can be overwritten. This also applies to the installation of programs for data recovery and saving recovered files.
- It is advisable to minimize all work with the original drive to increase the chances of recovery. It is good practice to create a sector-by-sector copy of the drive and use various recovery methods on the copy.
- If the storage medium is physically damaged (first of all, it concerns hard drives), then any impact can aggravate the situation. You should continue to restore yourself only if the cost of the data itself is lower than the cost of their restoration by specialists.
- Checking the disk for errors using Windows tools will create additional difficulties for data recovery, up to the complete impossibility of their recovery.
Disc structure
For a better understanding of further actions, let's talk about the logical structure of the disk and file systems. If you are not interested in theory and need to quickly recover data, you can skip the section and go directly to the algorithm of actions.
MBR
Master Boot Record - the master boot record - is located in the zero sector of the disk, contains a part of the executable code, as well as information about the four partitions and which partition is bootable.
If the MBR is healthy, then after gaining control, the code contained in it reads the partition table and transfers control to the code contained in the first sector of the boot partition (VBR - Volume Boot Record), if it contains the 55AAh signature at the end of the sector.
Otherwise, an error message may appear, for example: "Reboot and select a proper boot device" or "Invalid partition table" or "Missing operating system". All of the above messages can occur both due to a malfunction in the MBR or VBR, and for other reasons (the wrong boot disk was selected, etc.).
To repair a damaged MBR or boot partition, there are specialized utilities such as testdisk (Windows) and gpart (Linux), you can also use the "Recovery Console" from the Windows installation disc. Their use will not always give a positive result when recovering data, since in addition to damage to the MBR and VBR, there may be other malfunctions, so these options will not be covered in the article. Although there are plenty of instructions on how to troubleshoot such problems on the network, it is recommended that you first restore important data from the drive, and then try to restore it to its correct operation.
File systems
Any section containing a file system consists of the following parts:
- Boot Loader Region (VBR), which contains information about the partition structure as well as executable code.
- An area of data that is divided into equal parts, called clusters, with each cluster having a unique number. The cluster size is set during formatting.
- The service data area of the file system, which can store information about folders, files, their names, attributes, as well as information about which cluster chains are occupied by this or that file.
Let's take a closer look at the most popular file systems on user computers: FAT and NTFS.
NTFS
The structure of the NTFS file system is shown schematically in the figure.
The section begins with a boot area that contains information about the section as well as executable code. A copy of the bootloader is most often located at the end of the section.
The next area is the Master File Table (MFT). It is in it that information about directories, files and their attributes is stored. Usually a disk area is reserved for MFT, equal to 12.5% of the partition size. The size of the reserved area can change (up or down), and the table itself can be fragmented.
In addition, a certain area of the section contains a copy of the first 4 service records of the table.
The user data area takes up all of the remaining space on the partition.
When deleting a file using standard OS tools, only a mark is put in the file record that the file has been deleted, and the space it occupies is marked as free. If after deleting the file no writing was made to the disk, the file remained in place, and its recovery is possible.
When formatting a partition, a new MFT is created in place of the old one. Initially, the size of the new table is rather small (several hundred MFT records), so some of the service records of the previous file system can still be restored. The more files were written to a formatted partition, the less chance of successful data recovery.
In this case, the data physically remains in its place, and part of the information about them that was stored in the previous version of the MFT is also retained. These files cannot be read by standard OS tools.
Complete formatting is sometimes referred to as low-level formatting, which is a mistake. Low-level formatting is a technological operation that is performed during the manufacture of a drive, and it cannot be performed using software methods.
In fact, only two types of high-level formatting are available from the operating system: full and fast. Fast formatting simply generates the partition table, and with full formatting, the entire disk is first checked for bad sectors. In Windows XP, this check is performed using a read operation (that is, the data remains in place, which means that in terms of recovery, this situation does not differ from the quick format discussed above), and in Windows 7, sectors are checked by writing, and the data is destroyed irrevocably. and there is no way to restore them.
FAT
The structure of the FAT file system is shown schematically in the figure.
At the beginning of the section there is a VBR, its copy is usually located through 6 sectors. After a certain number of sectors, the file FAT (File Allocation Table) itself is located, followed by its copy.
The file table stores information only about the chains of clusters that are occupied by files. File names and attributes are stored in directories located in the user area.
The user area in FAT starts at the root directory, all other directories and files are located in it. Directory entries point to the first entry in the file table that contains information about the clusters in the file.
When a file is deleted, the first character of the file record is changed to a special code, which means that the file has been deleted. Also, clusters are marked as free and information about the chain of clusters occupied by the file is deleted, which complicates the procedure for recovering fragmented files. The data area remains unchanged, meaning the files can still be recovered.
When formatting, a new file table and root directory are formed, the size of the table is indicated at its beginning. The data itself, in general, remains on the disk. What follows on disk after the newly created table remains intact (that is, information about the location of the files that were on disk before formatting). As new files are written, directory and file structures are overwritten with new data, reducing the likelihood of restoring existing information.
Data recovery methods
There are both purely software methods of data recovery and software and hardware. For the latter, special expensive equipment, relevant experience and knowledge are required, while software recovery using automated utilities is available to almost any PC user. It is this method that will be described below.
Recovering deleted files
To recover files deleted by standard operating system tools, you need to read the service data bypassing the file system interface. In doing so, you can get information about the location of files that are marked as deleted.
If there was no subsequent writing to the disk, then this location will be able to read the desired file.
Reconstruction of a damaged file system
This method is used when the file system has been damaged or formatted. To reconstruct the file system, it is necessary to scan the entire partition to find remnants of service data, on the basis of which the file table will be reproduced and, if successful, access to the files and folders stored in the partition will be obtained.
Recovering after changing the structure of disk partitions
Specialized utilities scan the drive for the presence of file system structures that are on the drive. Based on the scan, a list of possible file systems is built with a preliminary assessment of their state. The next step is to check the found systems for the necessary data.
Recovery by signatures
RAW-recovery - recovery by signatures, used in cases where other methods have not yielded a satisfactory result. In this case, a sector-by-sector scan is performed on the drives for the presence of known signatures (a unique set of characters typical for a particular file type).
No names, logical locations, or attributes are known for the files found. If the files are fragmented, this recovery method will be ineffective.
If the signature of the beginning of the file was found, then the next task is to find the end of the file. Typically, any of the following known signatures are used for this, which may give an unsatisfactory result. Modern programs use methods that use leftover data about the file system and its features to improve the result. In some cases, such algorithms can help to recover even a fragmented file, which is impossible using standard algorithms. For example, the IntelliRAW algorithm used in the UFS Explorer family of programs works in conjunction with file system reconstruction algorithms and uses this information to determine the end of files. This implementation achieves better results than using simple signature-based recovery methods.
A rough recovery can give a good result when the files on the drive are arranged sequentially, without offsets and fragmentation. For example, when recovering files from memory cards of cameras, camcorders, etc.
Data recovery algorithm
We will give the algorithm using the example of the free R.saver program, which is available for non-commercial use in the territory of the former CIS. It is compact, easy to use and does not require installation. Uses algorithms of commercial software UFS Explorer, gives results at the level of professional utilities. The file systems are NTFS, FAT and exFAT.
You can download it from the support page on the official website: http://rlab.ru/tools/rsaver.html.
The archive with the program must be unpacked into a partition different from the one from which the recovery will be carried out. If there is only one partition in the system, the best solution would be to connect the disk with the recovered data to another computer. If this is not possible, you can unpack the program to an external drive (if it is large enough to store the recoverable data).
Selecting a section
After unpacking the program, you need to run it. In the left part of the main window, you can see the drives connected to the system and partitions on them. If the desired section was automatically found and displayed in the list, select it and go to the next item.
If the desired partition is not in the list (for example, the drive was formatted or partitions resized), you can start a search for a lost partition or set it yourself (only if you know the exact parameters of the partition). You can select these functions in the drive's context menu or on the toolbar.
Let's take a closer look at the lost partition search function:
After its launch, a new window will open, in which you need to click on the "Find Now" button, which will start a search on the disk for known file systems. As they are found, the list will be supplemented with information about the found sections. The most likely options will have a blue icon, which means that the partition is in good condition. If the icon is yellow or red, then the found partition is either badly damaged or found by mistake (some file types can have this effect).
From the list of partitions, tick the boxes that best match the parameters of the desired partition: start (in sectors), size (in sectors and megabytes), label, file system type.
If the value in the "size" column is less than the value in the "start" column, then most likely this section is not valid. Typically these "invalid" partitions are disk images stored on the drive.
It is recommended to save the scan results before continuing so that you do not have to scan the disk again if you make an incorrect selection.
Click the Use Selected button to continue.
Scanning
Next, a file manager window will open, which will show the current state of the file system (that is, files and directories visible by the OS). Click the Scan button to start scanning.
If you have selected a partition with the NTFS file system, you will be offered a choice: run a full or quick scan.
The scope of the quick scan is to search for files deleted by standard means of the operating system. This scan only reads service records in the file table and analyzes them.
Full scan is used in most cases. At the same time, the search for the remnants of service records in the file table, virtual reconstruction of the file system and sector-by-sector scanning using IntelliRAW technology are performed. Gives excellent results, but takes longer.
For FAT16 / FAT32 and exFAT file systems, only full scan mode is available. This is due to the fact that, due to the peculiarities of these file systems, in many cases, a quick scan is not enough for effective data recovery.
Saving results
After the scan is complete, the program will show the result of the reconstruction in the form of a tree of files and folders that it could find.
Blue icons display elements that are visible by standard operating system tools, red - those that have been deleted or lost and are inaccessible by standard operating system tools.
In addition, service folders will be available:
- [Parent unknown] contains files and folders whose location could not be linked to the root directory
- [IntelliRAW] contains files sorted by type found by signature repair (with IntelliRAW technology)
To save the recovered data, either select the "Copy to ..." item in the context menu of an element, or click the "Mass selection" button on the toolbar, then select all the necessary elements and click the "Save selected" button.
Remember that you only need to save the recovered data to a partition that is different from the one from which the recovery is being carried out - otherwise the data may be damaged.
It is recommended to save the scan result by clicking the corresponding button on the toolbar so that you do not have to re-scan the section if you did not save all the necessary data for the first time.
