APFS Object Map Recovery

I spent a day figuring out how to look into a corrupt encrypted APFS partition. Here’s the notes.

If you only care about the answer, jump to the last section.

Kernel Panic

macOS hard panicked on me with a graphics driver corruption error com.apple.iokit.IOAcceleratorFamily2 (which happened after seven “gpuRestart” logs were generated back-to-back).

After a forced reboot and re-login, the system crashed again while applications were starting up. Not good.

After the next reboot, the system started okay, but since applications crashed while starting, they lost all of their historical state. Chrome lost all open windows and tabs. Terminal lost all open windows and tabs. Drat.

Luckily I had some backups. So, off to old backups to restore Chrome windows/tabs and Terminal windows/tabs from old saved states.

Now with windows restored to how they looked at the last backup checkpoint two weeks ago, let’s see what else went wrong during a double crashing reboot.

No Drive

I launched VLC to watch something from my media drive and… my 2 TB USB drive wasn’t mounted.

No problem, just hop into Disk Utility and mount it.

Except—this looks odd.

Odd and wrong. Uninitialized? I’m not uninitialized, you’re uninitialized! Odd and wrong and ohcrapwhereismydrive.

It’s not the end of the world (the drive is just downloaded media), but I worked hard for these files and I’d rather not lose them.

No Data

Let’s click First Aid on AppleAPFSMedia (which is /dev/disk3):

Running First Aid on “AppleAPFSMedia” (disk3)

Fixing damaged partition map.
Invalid disk.

Operation failed…

No bueno.

No fsck

What if we try First Aid on APFS Physical Store disk2s2 instead?

Running First Aid on “” (disk2s2)

Repairing storage system
Performing fsck_apfs -y -x /dev/disk2s2
Checking the container superblock.
Checking the space manager.
Checking the space manager free queue trees.
Checking the object map.
Checking volume.
Checking the APFS volume superblock.
The volume Blue Diamond was formatted by diskmanagementd (945.230.6) and last modified by apfs_kext (945.275.7).
Checking the object map.
error: om: btn: invalid o_oid (0x233b75)
Object map is invalid.
The volume /dev/disk2s2 could not be verified completely.
Storage system check exit code is 8.
Storage system verify or repair failed.

Operation failed…

Even though fsck_apfs can see the volume name now, our overall feeling is still no no no no no no

No diskutil info

What does diskutil apfs list think?

APFS Containers (2 found)
|
+-- <ignoring system drive here>
|
+-- Container ERROR -69808
    ======================
    APFS Container Reference:     disk3
    Size (Capacity Ceiling):      ERROR -69620
    Capacity In Use By Volumes:   ERROR -69620
    Capacity Not Allocated:       ERROR -69620
    |
    +-< Physical Store disk2s2
    |   -----------------------------------------------------------
    |   APFS Physical Store Disk:   disk2s2
    |   Size:                       2000189177856 B (2.0 TB)
    |
    +-> No Volumes

things are looking pretty awful

Apparently, in 2019, Apple has a file system corruptible by crash reboots and also can’t be repaired by their own tools. Everything is fine.

A data recovery tool appears

Searching online for recovering from APFS Object map is invalid errors brings us to a site called “iboysoft” which is a utility with pretty good marketing SEO for file system problems.

Of course each fix of theirs is “our software fixes it! Pay us $99 to recover your data!”

My only only option now is to download a utility called “iboysoft” from an anonymous Chinese company with promises it can recover unreadable APFS partitions.

So we download.

And we run.

And… it shows me all files on the drive! Thanks, “iboysoft,” and shame on you, Apple.

Of course, it only shows you file names and file previews in trial mode. Actually restoring files costs $99.

Before I throw away $99 to recover within-arms-reach files, I do some more searching… either the people behind this utility reverse engineered APFS from half complete docs Apple provides or they are using other public code underneath (spoiler: I never found out if the utility was all custom code or reusing open source code just packaged in a UI¹).

but what about GitHub?

What about APFS recovery tools lurking around on GitHub? If a random anonymous online app can fix my drive, what about public utilities?

A read-only apfs info util

First, we found apfs-tools (via Apple SE: Accessing unmountable APFS Volume on a disk image) currently being maintained/created/updated with a hand written APFS parser — but, it won’t open my encrypted APFS volume, and Apple’s own tools won’t provide a decrypted volume view because diskutil apfs unlockVolume doesn’t recognize the broken APFS volume or drive at all.

A swing and a miss.

A full apfs library

Next up, we find a thing called libfsapfs currently being maintained/created/updated with a different hand written APFS parser, and this one includes support for reading encrypted volumes!

Also it looks like libfsapfs includes some binary utilities so we won’t have to interface with the library directly for quick tests.

Let’s compile it from a release:

CC="xcrun gcc" ./configure
make -j12

libfsapfs includes binaries, but we don’t know where they are or where they got built.

Let’s search:

> find . -type f -perm 0755
./fsapfstools/fsapfsmount
./fsapfstools/fsapfsinfo

Those look useful!

Running ./fsapfsinfo gives us help of:

fsapfsinfo 20191104

Missing source file or device.
Use fsapfsinfo to determine information about an Apple
 File System (APFS).

Usage: fsapfsinfo [ -B bodyfile ] [ -E identifier ]
                  [ -f file_system_index ] [ -F path ]
                  [ -o offset ] [ -p password ]
                  [ -r password ] [ -hHvV ] source

    source: the source file or device

    -B:     output file system information as a bodyfile
    -E:     show information about a specific file system entry or "all"
    -f:     show information about a specific file system or "all"
    -F:     show information about a specific file entry path
    -h:     shows this help
    -H:     shows the file system hierarchy
    -o:     specify the volume offset
    -p:     specify the password
    -r:     specify the recovery password
    -v:     verbose output to stderr
    -V:     print version

oh look, it has a simple -p option to enter the password of an encrypted volume!

and we can use -H to print a list of everything it can see on the filesystem, which, when we run it, ends up being the same list of files discovered by the “iboysoft” recovery utility from before:

./fsapfsinfo -p <passphrase> -H /dev/disk3

It looks like we may be able to avoid paying $99 to recover files from our corrupted APFS volume after all.

Recovering APFS Volume using libfsapfs

The other command is fsapfsmount which seems like our goal. All we really need is to mount our system-unreadable volume so we can copy the files somewhere safe.

Let’s try to mount. The mount command follows normal syntax with the mount point being the last argument.

> mkdir got
> ./fsapfsmount -p <passphrase> -v /dev/disk3 got
fsapfsmount 20191104

mount_osxfuse: the file system is not available (1)
Unable to create fuse channel.

Not available? File system not available? Oh, osxfuse not available. We probably didn’t have mac fuse installed. Let’s fix with:

brew cask install osxfuse

But now we also have to rebuild libfsapfs so it can pick up fuse:

<back to the project root>
CC="xcrun gcc" ./configure
make -j12

Now let’s try the mount again! (you will get a macOS system extension error where you need to manually approve the fuse extension in System Preferences, so do the approval dance then continue from here…)

> ./fsapfsmount -p <passphrase> -v /dev/disk3 got
fsapfsmount 20191104

volicon: missing 'iconpath' option
Unable to create fuse handle.

What’s an iconpath and why do we care?

Back to google… (in reality we tried to mount another couple times, rebooted, then asked google)

We get an unrelated project issue declaring mac fuse requires a path to an icon library to show disks or else it won’t mount anything. sure, why not. remember: defaults matter. Don’t make users search for weird things when the default could just work hardcoded (obviously you need an icon repository to mount your disk, right?).

Back we go to mounting, but we may need to clean up the previous failed mount first.

The previous failed mount did (oddly) use the mount point even though it didn’t mount anything. Let’s free the directory.

> umount got

Now try again (the same command as before, but now with -X volicon=/Library/Filesystems/osxfuse.fs/Contents/Resources/Volume.icns):

> ./fsapfsmount -p <passphrase> -X volicon=/Library/Filesystems/osxfuse.fs/Contents/Resources/Volume.icns /dev/disk3 got
fsapfsmount 20191104

And it worked! At least it worked for me.

We can ls got and… there are all our files! We didn’t have to pay a weird company named “iboysoft” $99 to recover files Apple corrupted away from us. We just had to pay whatever our time was worth for the 5 hours it took to track down all of these solutions.

Now just rsync -av --progress --partial got/* /Volumes/NewDrive/ for about 20 hours. After a mostly complete transfer (you may have some data errors, but rsync will skip over them to continue), you can reformat your corrupt drive as not APFS because APFS shouldn’t be used on external spinning rust drives at all, as we’ve seen through this entire mess. It’s okay though, we didn’t have anything else better to do today than babysit poorly written OS-provided system utilities and sleuth the Internet neck deep in error reports for half a dozen manual workarounds.

Enjoy.

-Matt — @mattsta — ☁mattsta

P.S. not to be rude or ungrateful or anything, but I can’t personally help your data problems. There’s that $99 recovery app you’d probably be better off using if you don’t know how to download github projects or use rsync already. Sorry, but good luck!