Question Can a PNG file with defective bit still be opened ?

[Oblivion77]

Dear all

If a PNG file has 1 or more bit, that is either unreadable, corrupt or broken.

What happens, if the file is opened in MS Paint or another software?
Would an error message appear saying something / file uable to open?
Would the software try to repair or fill the gap?
Is it the same for NTFS and FAT34?

Thank you for replying

Best regards

 

[Krotow]

Not always defective bit make picture file completely unreadable. According to location in file it also may cause picture partially visible, make a local defect somewhere in picture or if wrong bit is located somewhere in metadata - doesn't cause any visible problems at all. Try to open damaged PNG file in various picture viewers and editors or convert to different format.

 

[Oblivion77]

Not always defective bit make picture file completely unreadable. According to location in file it also may cause picture partially visible, make a local defect somewhere in picture or if wrong bit is located somewhere in metadata - doesn't cause any visible problems at all. Try to open damaged PNG file in various picture viewers and editors or convert to different format.

So if a bit is defective / unreadable, it would be the same as 0 / uncharged?

 

[Krotow]

Bit can't be defective. It is binary value - either 1 or 0. It works like that due to design. We can't say that numbers 0 or 6 in basic arithmetic are defective - it all depends from context. If file was copied or transferred with errors, error can flip some bit which now have wrong value - 0 instead of 1 or 1 instead of 0 - and therefore cause byte to which bit belong, have wrong value as well. Like if byte with value 0xA5 (binary 10100101) have 5th bit flipped by transfer error, it will turn into 0xB5 (binary 10110101) - or vice versa if 5th bit will be erroneously flipped from 1 to 0. According to byte location and meaning in file - it can render file unusable or pass without problems.

Copying and transfer errors however are processed differently in particular software used in copying or transferring. In case of interrupted or particularly failed copying/transferring problematic areas can become filled with predefined default values like 0x00, 0xFF, 0xAA etc. Which solely depends from software you are using.

 

[Oblivion77]

Bit can't be defective. It is binary value - either 1 or 0. It works like that due to design. We can't say that numbers 0 or 6 in basic arithmetic are defective - it all depends from context. If file was copied or transferred with errors, error can flip some bit which now have wrong value - 0 instead of 1 or 1 instead of 0 - and therefore cause byte to which bit belong, have wrong value as well. Like if byte with value 0xA5 (binary 10100101) have 5th bit flipped by transfer error, it will turn into 0xB5 (binary 10110101) - or vice versa if 5th bit will be erroneously flipped from 1 to 0. According to byte location and meaning in file - it can render file unusable or pass without problems.

Copying and transfer errors however are processed differently in particular software used in copying or transferring. In case of interrupted or particularly failed copying/transferring problematic areas can become filled with predefined default values like 0x00, 0xFF, 0xAA etc. Which solely depends from software you are using.

Thank you for your reply

I have read your reply thoroughly several times now, and I hope I understand it.

1.
If we only focus on SSD and USB drive = flash memory
Both 1 and 0 bits are charged, but with different volts?

2.
"According to byte location and meaning in file - it can render file unusable or pass without problems."
So a file with an error, that been copied or transfered, is either unusable or changed?

3.
So if I want to find out, if a file has errors /defective bits etc.
What would you suggest?

Thank you again

 

[Ralston18]

Not sure about your "use" or "working" environment and work flows but you can determine if a file has changed.

Powershell: Get-Filehash

FYI:

How to Use the Get-FileHash PowerShell Cmdlet (adamtheautomator.com)

https://stackoverflow.com/questions/67497396/compare-two-hashes-using-powershell

You can easily find other similar links.

Start with the above two links. Try, test, etc. as necessary to determine what meets your requirements.

Then, if and as necessary, do additional searches based on your own search criteria to focus on your specific requirements.

 

[hotaru.hino]

The data in a PNG file is broken up into chunks, all of them having error detecting data using a CRC checksum. Basically, the data is thrown through a math function and the result of that is compared to what's stored for that chunk. If they don't match, then the chunk is considered corrupted. This does not mean however, the data can be fixed. The software doesn't know if a single bit flip, even if that's all that flipped. However, if the software doesn't refuse to open the file outright on one corrupted chunk, it may still load the file, but only part of the image will be corrupted.

There are some other ways to store data with error correcting bits, such as Hamming Code, which can correct a handful of bits depending on which implementation is used. And there are some file systems that store extra data for redundancy in case of errors: https://en.wikipedia.org/wiki/List_of_file_systems#File_systems_with_built-in_fault-tolerance

At the end of the day though, it's practically impossible to correct corrupted data in most situations, because the computer doesn't know how the data was corrupted to begin with. Maybe in the future, we can train AI to figure out the context of the data and attempt to repair it, but I don't think there'll be a universal solution.

 

[fzabkar]

Storage devices read and write files on a sector basis. Wherever possible, faulty bits in a particular sector would be corrected by the storage device, in which case the OS wouldn't see the problem. Otherwise the drive would return a read error for that sector.

If the PNG file does indeed have bad bits, then I would think this would be the result of bad RAM.

I think you can answer your own question by using a hex editor to flip selected bits. HxD is a good freeware editor.

https://mh-nexus.de/en/hxd/

I found this Wikipedia article to be useful:

https://en.wikipedia.org/wiki/Portable_Network_Graphics

Try flipping bits in the IHDR and IDAT chunks.

 

[fzabkar]

I flipped a bit in the IHDR chunk and my image viewer (PaintShop Pro 5.01) complained that the file was not a valid PNG file. Flipping a bit in the IDAT chunk still allows the image to be partially rendered, but there is a horizontal black stripe at the bottom.

http://www.users.on.net/~fzabkar/temp/PNG_test/

However, my browser (Opera) correctly displays the file with the corrupted IDAT. :-?

Here is a nice command line tool for checking the integrity of PNG files:

http://www.libpng.org/pub/png/apps/pngcheck.html

Good PNG file:

C:\>pngcheck.win32.exe -vv C:\OB3350_application.png
File: C:\OB3350_application.png (18669 bytes)
  chunk IHDR at offset 0x0000c, length 13
    667 x 337 image, 8-bit palette, non-interlaced
  chunk tIME at offset 0x00025, length 7: 27 Feb 2022 03:24:31 UTC
  chunk pHYs at offset 0x00038, length 9: 2834x2834 pixels/meter (72 dpi)
  chunk PLTE at offset 0x0004d, length 768: 256 palette entries
  chunk IDAT at offset 0x00359, length 17792
    zlib: deflated, 32K window, maximum compression
    row filters (0 none, 1 sub, 2 up, 3 avg, 4 paeth):
      0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
      0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
      0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
      0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
      0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
      0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
      0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
      0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
      0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
      0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
      0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
      0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
      0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
      0 0 0 0 0 0 0 0 0 0 0 0 (337 out of 337)
  chunk IEND at offset 0x048e5, length 0
No errors detected in C:\OB3350_application.png (6 chunks, 91.7% compression).

Bad PNG file:

C:\>pngcheck.win32.exe -vv C:\OB3350_application_bad_IDAT.png
File: C:\OB3350_application_bad_IDAT.png (18669 bytes)
  chunk IHDR at offset 0x0000c, length 13
    667 x 337 image, 8-bit palette, non-interlaced
  chunk tIME at offset 0x00025, length 7: 27 Feb 2022 03:24:31 UTC
  chunk pHYs at offset 0x00038, length 9: 2834x2834 pixels/meter (72 dpi)
  chunk PLTE at offset 0x0004d, length 768: 256 palette entries
  chunk IDAT at offset 0x00359, length 17792
    zlib: deflated, 32K window, maximum compression
    row filters (0 none, 1 sub, 2 up, 3 avg, 4 paeth):
      0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
      0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
      0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
      0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
      0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
      0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
      0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
      0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
      0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
      0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
      0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
      0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
      0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
      0 0 0 0 0 0 0 0 0 0 0
    zlib: inflate error = -3 (data error)
(336 out of 337)
ERRORS DETECTED in C:\OB3350_application_bad_IDAT.png

Edit: The flipped bit just happens to land on the second last line of the image (line 336 out of 337). On closer examination of the rendered image, there is a thin black border at the bottom edge. This is present in all image viewers except for PaintShop Pro.

If I flip the very first bit (after the IDAT header), the image is completely black or completely white in all my viewers.

 

[Oblivion77]

The data in a PNG file is broken up into chunks, all of them having error detecting data using a CRC checksum. Basically, the data is thrown through a math function and the result of that is compared to what's stored for that chunk. If they don't match, then the chunk is considered corrupted. This does not mean however, the data can be fixed. The software doesn't know if a single bit flip, even if that's all that flipped. However, if the software doesn't refuse to open the file outright on one corrupted chunk, it may still load the file, but only part of the image will be corrupted.

There are some other ways to store data with error correcting bits, such as Hamming Code, which can correct a handful of bits depending on which implementation is used. And there are some file systems that store extra data for redundancy in case of errors: https://en.wikipedia.org/wiki/List_of_file_systems#File_systems_with_built-in_fault-tolerance

At the end of the day though, it's practically impossible to correct corrupted data in most situations, because the computer doesn't know how the data was corrupted to begin with. Maybe in the future, we can train AI to figure out the context of the data and attempt to repair it, but I don't think there'll be a universal solution.

Thank you for your reply

Interesting stuff

"all of them having error detecting data using a CRC checksum"
Does all the chunks have the CRC checksum function?
When is the CRC checksum used, when reading the file, copying etc.?

 

[Oblivion77]

Storage devices read and write files on a sector basis. Wherever possible, faulty bits in a particular sector would be corrected by the storage device, in which case the OS wouldn't see the problem. Otherwise the drive would return a read error for that sector.

If the PNG file does indeed have bad bits, then I would think this would be the result of bad RAM.

I think you can answer your own question by using a hex editor to flip selected bits. HxD is a good freeware editor.

https://mh-nexus.de/en/hxd/

I found this Wikipedia article to be useful:

https://en.wikipedia.org/wiki/Portable_Network_Graphics

Try flipping bits in the IHDR and IDAT chunks.

"faulty bits in a particular sector would be corrected by the storage device"
The whole sector would be corrected to so the sector only contains the data of the file?

When is a sector being corrected by the storage device?

 

[Oblivion77]

Would someone please try to answer my further questions please:

1.
SSD and USB drive = flash memory
Both 1 and 0 bits are charged, but with different volts?

2.
FileComparer, TotalCommander or other software that can check / compare bits of two files
Would tell if one of the files:
Has a bit error
Has corrupted bits
Has flipped bits
Other bit related issues
?

3.
What software can you reccomend to compare metadata of two files?

 

[fzabkar]

I would use a hex editor (eg HxD freeware) to compare the two files.

https://mh-nexus.de/en/hxd/

Otherwise, in a CMD window you could use the FC.EXE command.

C:\>fc /b OB3350_application.png OB3350_application_bad_IDAT.png

Comparing files OB3350_application.png and OB3350_APPLICATION_BAD_IDAT.PNG

000046D0: F0 F1

Here is a corrupted PNG file with a flipped bit in the middle of the IDAT chunk:

http://www.users.on.net/~fzabkar/temp/PNG_test/OB3350_application_bad_IDAT_middle.png

This is the original PNG:

http://www.users.on.net/~fzabkar/temp/PNG_test/OB3350_application.png

 

[fzabkar]

"faulty bits in a particular sector would be corrected by the storage device"
The whole sector would be corrected to so the sector only contains the data of the file?

When is a sector being corrected by the storage device?

The sector is corrected on the SSD before it is transferred to the OS. The ECC feature in the SSD firmware can detect and correct a certain number of flipped bits. If there are too many errors, the sector cannot be repaired and the SSD transfers no data for this sector. It just reports a read error.

 

[Oblivion77]

The sector is corrected on the SSD before it is transferred to the OS. The ECC feature in the SSD firmware can detect and correct a certain number of flipped bits. If there are too many errors, the sector cannot be repaired and the SSD transfers no data for this sector. It just reports a read error.

1.
Faulty bits, do you mean corrupted bits aswell, or only flipped bits?

2.
"The sector is corrected on the SSD before it is transferred to the OS"
Transfered? Whenever the file is opened?

3.
"detect and correct a certain number of flipped bits"
How can it tell that the bits are flipped? It compares it with another file that is identical?

 

[fzabkar]

@emilfrederiksen, I feel that we going round in circles.

Bit = Binary Digit

A bit is either 0 or 1. If a bit should be a 1 but is now a 0, then it is bad or "corrupt". It has flipped from 1 to 0.

If a bit should be a 0 but is now a 1, then it is bad or "corrupt". It has flipped from 0 to 1.

Since a bit can have only two states, a bad (or "corrupt") bit is always "flipped" from one state to the other.

"Transferred" means "read" (as in when a file is "opened"). "Opening" a file entails reading (or transferring) it from the drive to the OS.

As for error detection ...

https://en.wikipedia.org/wiki/Error_correction_code

 

[USAFRet]

bits and bytes.

Here is a 200x200 pixel png. A single field of blue.
797 bytes.

The pic

and then the hex values of the bytes in Notepadd++

Now, we change the value of a single byte, outlined in red.

Resulting in this broken image:

Changing a different byte in that PNG file may make it totally unreadable.

 

[Oblivion77]

“Wherever possible, faulty bits in a particular sector would be corrected by the storage device”
“The sector is corrected on the SSD before it is transferred to the OS. The ECC feature in the SSD firmware can detect and correct a certain number of flipped bits.”
Would it change / correct the stored file on the drive?

Or what is corrected, and where?

How can it tell that the bits are flipped? It compares it with another file that is identical?


CRC checksum

Whats happens if it doesn’t check out /gives same result?


What can cause flipped bits?

Only tranfer and copy errors in RAM?


After I have copied a file to an external drive, I run the fc /b command on the two files

The fc /b command is a guarantee, that no bits are flipped, corrupt etc.?

 

[Oblivion77]

@emilfrederiksen, I feel that we going round in circles.

Bit = Binary Digit

A bit is either 0 or 1. If a bit should be a 1 but is now a 0, then it is bad or "corrupt". It has flipped from 1 to 0.

If a bit should be a 0 but is now a 1, then it is bad or "corrupt". It has flipped from 0 to 1.

Since a bit can have only two states, a bad (or "corrupt") bit is always "flipped" from one state to the other.

"Transferred" means "read" (as in when a file is "opened"). "Opening" a file entails reading (or transferring) it from the drive to the OS.

As for error detection ...

https://en.wikipedia.org/wiki/Error_correction_code

Dear Fzabkar

Thank you again for your many replies

I hope, that you will help me with the following aswell:

1.
“Wherever possible, faulty bits in a particular sector would be corrected by the storage device”
“The sector is corrected on the SSD before it is transferred to the OS. The ECC feature in the SSD firmware can detect and correct a certain number of flipped bits.”
Would it change / correct the stored file on the drive?
Or what is corrected, and where?

2.
How can it tell that the bits are flipped? It compares it with another file that is identical?

3.
CRC checksum
Whats happens if it doesn’t check out /gives same result?

4.
What can cause flipped bits?
Only tranfer and copy errors in RAM?
If you somehow damage internal parts?

5.
After I have copied a file to an external drive, I run the fc /b command on the two files
The fc /b command is a guarantee, that no bits are flipped, corrupt etc.?

Thank you in advance

Best regards

 

[hotaru.hino]

1.
“Wherever possible, faulty bits in a particular sector would be corrected by the storage device”
“The sector is corrected on the SSD before it is transferred to the OS. The ECC feature in the SSD firmware can detect and correct a certain number of flipped bits.”
Would it change / correct the stored file on the drive?
Or what is corrected, and where?

The assumption is yes, if an SSD (or any storage device with ECC) detects an error that it can correct, it will correct the data. Otherwise the computer is wasting cycles realizing the data needs correcting every time it's read.

As for what is corrected and where, it depends on the error correcting method used. For example, if the storage is using Hamming Codes, the amount of bits that can be correct depends on the number of bits set aside for error detection. If the device is using (4, 7) Hamming Code, only a single bit can be corrected. If more than 1 bit is incorrect, then the system can still flag a problem, but there's nothing it can do about it.

2.
How can it tell that the bits are flipped? It compares it with another file that is identical?

The most basic principle is using something called parity. This counts the number of bits that have a value of 1. A bit is used to mark if the number of 1's is even or odd (it depends the protocol). However, this has limitations. For example if the parity bit is detecting an even amount of 1s, then two 1s can flip to 0 and the system won't think anything is wrong.

Another method is to add the values of the data together in some form or fashion before sending it, called a checksum. Then the receiver does the same adding and checks the result. If the result isn't what's expected, then the system knows something happened, but it can't tell what.

Another option is error correcting code, like the aforementioned Hamming Code. Hamming codes can detect where a bit has flipped. But also as mentioned, there are limitations to this.

And while you could have a copy of the data, you need at least 3 total copies of this data to be effective. If you only have 2 and one of them gets corrupted in a way that the above doesn't catch anything, how do you know which one is the correct copy? Having at least 3 makes it so a consensus is easier to determine. However, this is an inefficient way to store data for events that are rare. But if you absolutely need data reliability, then yes, this method works when used with the other methods.

3.
CRC checksum
Whats happens if it doesn’t check out /gives same result?

All you can tell is the data has been corrupted.

4.
What can cause flipped bits?
Only tranfer and copy errors in RAM?
If you somehow damage internal parts?

Damage to the parts, power droops or surges, RF emissions causing a voltage spike, cosmic rays hitting spots in hardware.... There's a lot of things that could happen.

5.
After I have copied a file to an external drive, I run the fc /b command on the two files
The fc /b command is a guarantee, that no bits are flipped, corrupt etc.?

I wouldn't say this is a guarantee. From when the data is picked up from the drive to the time it arrives in RAM and processed, anything could've happened. However, it's reasonable to assume that if your computer is working more or less fine, that this process is reliable enough to be trusted.

 

[fzabkar]

If you want to see how checksums and CRCs work, you can use the Analysis -> Checksums feature of HxD (a free hex editor).

If you want to see how FC /B works, use HxD to change one byte in a file and then use FC to compare the original and modified files.

 

[Oblivion77]

Thank you so much for your long reply

The assumption is yes, if an SSD (or any storage device with ECC) detects an error that it can correct, it will correct the data. Otherwise the computer is wasting cycles realizing the data needs correcting every time it's read.

It still don't understand, how the drive can detect an error and correct, it requires the drive to compare the corrupt file with a flawless version of the file?

Otherwise, its one of the methods you present in 2?

Damage to the parts, power droops or surges, RF emissions causing a voltage spike, cosmic rays hitting spots in hardware.... There's a lot of things that could happen.

Damage to the drive part from outside, would it affect all the bits / the whole drive, or it could be limited to just some parts of the drive / some bits?

I wouldn't say this is a guarantee. From when the data is picked up from the drive to the time it arrives in RAM and processed, anything could've happened. However, it's reasonable to assume that if your computer is working more or less fine, that this process is reliable enough to be trusted.

File A on local drive
Copy file A to USB drive (B)
Create whole file again from the bottom (C)

Compare A and B = no differences encountered
Compare A and C = no differences encountered

In regard of the above method, is it then a guarantee, that no bits are flipped, and all the 3 files are identical?

 

[Oblivion77]

If you want to see how checksums and CRCs work, you can use the Analysis -> Checksums feature of HxD (a free hex editor).

If you want to see how FC /B works, use HxD to change one byte in a file and then use FC to compare the original and modified files.

I have now conducted many experiments with fc /b in cmd
Compared identical pictures
Compared two pictures with minor pixel differences
Compared two identical pictures, but with minor changes in another chunk than iDAT (the picture)

There is no doubt, when 2 files are different og identical

 

[hotaru.hino]

It still don't understand, how the drive can detect an error and correct, it requires the drive to compare the corrupt file with a flawless version of the file?

Otherwise, its one of the methods you present in 2?

It doesn't need an exact copy of the data to correct it. It uses error correcting code to detect and fix errors.

Damage to the drive part from outside, would it affect all the bits / the whole drive, or it could be limited to just some parts of the drive / some bits?

If the damage is localized, it'll only affect that part. Whether or not the entire data store is still considered usable or not is another thing.

File A on local drive
Copy file A to USB drive (B)
Create whole file again from the bottom (C)

Compare A and B = no differences encountered
Compare A and C = no differences encountered

In regard of the above method, is it then a guarantee, that no bits are flipped, and all the 3 files are identical?

Any time the data is in flight, anything can happen to it. If the computer is in an environment that's hostile to electronics, it could easily develop errors along the way.

 

[fzabkar]

It still don't understand, how the drive can detect an error and correct, it requires the drive to compare the corrupt file with a flawless version of the file?

When a sector is written to the drive, the drive calculates the ECC bytes and stores them together with the data bytes. When the drive reads back the sector, it recalculates the ECC. If the new and original ECC bytes don't match, then the drive knows that there was a read error, either in the data bytes or in the ECC bytes. If the number of flipped bits is not too great, the drive is able to determine which bits were flipped. It then automatically and transparently corrects the data before passing it to the host. Otherwise, if the number of flipped bits exceeds the strength of the error correction, the drive returns a read error.

https://hddscan.com/doc/HDD_Tracks_and_Zones.html
https://hddscan.com/doc/images/HDD_Tracks_and_Zones/data.jpg