Dealing with manual Core Data migrations

Core data migrations are an issue for any app which uses Core Data. We’ve previously written how we deal with them and the measures we take to avoid any issues. Here, we detail the latest improvements in our approach…

Our Core Data schema changes are typically trivial and can be taken care of with a lightweight migration and the use of mapping models. An issue we’ve encountered, however, is that these automatic migrations are a black box – any errors that happen in production can be a mystery as we don’t get detailed error logs when a migration fails. Since it’s also unclear whether a migration has succeeded it’s hard to get metrics to track overall successes/failures.

We wanted to improve this situation to make our migrations even more reliable.

A Proposed Solution

We suspected that the cause of our migration issues were related to high memory usage, either from our app or in general across iOS, causing our app’s main process to be killed and so quitting the app during a migration.

With that in mind we concluded that we need to do a manual, step by step – a.k.a. progressive – migration that we can recover from if an error occurs. This manual process would also allow us to keep the memory use of the process in check.

Apple doesn’t provide too many details on how to perform a manual migration however objc.io have some good suggestions in their Core Data book and their blog.

Initially we decided on the following approach.

1. Create a copy of the database requiring migration, and move it to a backup folder.
2. Perform manual migration on original database.
3. If successful – open the database and proceed with app launch.
4. If unsuccessful – restore the back up copy of the database.

In the majority of cases this worked well but during testing we did see some issues with the database being corrupted after the file copy. We were copying the file using NSFileManager’s moveItemAtURL method.   According to Apple the correct method of copying a store is to use an NSPersistentStoreCoordinator and its replacePersistentStoreAtURL method. After switching to this method database corruption was no longer an issue.

After further testing we decided on the following approach:

1. Create a copy of the database that requires migration and move it to a backup folder.
2. Perform manual migration on the copied database.
3. If successful then replace the original database and proceed with app launch.
4. If unsuccessful then prompt the user to reattempt migration.

A benefit of this strategy is that we maintain a copy of the original database which we can recover from the device if the migration proves impossible, to ensure no data is lost.

Progressive Migration

In a lightweight migration the database schema is migrated from the current version directly to the latest version. For example, a lightweight migration from version 1 to version 3 would go directly from 1–3, bypassing version 2.

In contrast a progressive migration performs a migration for each step in the sequence – so it would perform a migration from version 1 to 2, and then another from 2 to 3. This allows for more granular reporting and memory management during the migration process.

To define each step of the migration we need to use XCMappingModels. Historically we have had to recreate all of the mapping models when we have a new schema model. The new progressive approach means we only need to create one new file for each migration, from the last schema version to the current version.

With the mapping models created we next migrate the data. Similar to the suggestion in the objc.io blog post above we use the NSMigrationManager class and its migrateStoreFromURL method, we wrap this in a method that we call recursively to migrate through each version of the schema until we reach the current version. The outline of the process is below.

1. Check current schema is compatible with final version, if so we are successfully migrated, otherwise:
2. Get the next managed model & mapping model.
3. Initialise NSMigrationManager & call migrateStoreFromURL method with current & target URLs.
4. Repeat until fully migrated.
5. Replace original database with the migrated copy.

Memory considerations

One of the main goals for the manual migration was to reduce the memory footprint of the app while the migration is occurring. Some of our users have large data sets which can cause high memory usage during a migration. Given iOS 13’s aggressive approach to memory management, this can cause apps to exit prematurely.

During testing with a large dataset and a large migration (five schema versions in total) we saw large memory use with our new approach. You can clearly see where each migration step was loaded into memory peaking at 581mb. This is precisely what we want to avoid. 

Even though each migration had completed the data was still being held in memory. To fix this issue we needed to implement an Autorelease pool. Apple provides more info on this here. 

At the end of the autorelease pool block, objects that received an autorelease message within the block are sent a release message—an object receives a release message for each time it was sent an autorelease message within the block.

Each time the performManualMigrationWithSourceURL method is called and finished we want to release any objects associated with the migration. Simply wrapping our method in an @autoreleasepool block provides this functionailty.

After adding this block you can see the total memory footprint is drastically improved as the objects are released once they are finished with.

Summary

As a result of these changes we have much more control over Core Data migrations, the key improvements for us have been:

• More reliable migrations.
• Increased error reporting if an issue does occur.
• Recoverable database in the event of a failed migration.
• Smaller memory footprint during large migrations (a big help in iOS 13).
• Less overhead to create a new migration – we only need to add one file per schema migration.