Example or explanation of Core Data Migration with multiple passes?

My iPhone app needs to migrate its core data store, and some of the databases are quite large. Apple's documentation suggests using "multiple passes" to migrate data to reduce memory use. However, the documentation is very limited and doesn't explain very well how to actually do this. Can someone either point me towards a good example, or explain in detail the process of how to actually pull this off?

I've figured out what Apple hints in their documentation. It's actually very easy but a long way to go before it's obvious. I'll illustrate the explanation with an example. The initial situation is this:

##Data Model Version 1

It's the model you get when you create a project with the "navigation based app with core data storage" template. I compiled it and did some hard hitting with some help of a for loop to create around 2k entries all with some different values. There we go 2.000 events with an NSDate value.

Now we add a second version of the data model, which looks like this:

##Data Model Version 2 The difference is: The Event entity is gone, and we've got two new ones. One which stores a timestamp as a double and the second one which should store a date as NSString.

The goal is to transfer all Version 1 Events to the two new entities and convert the values along the migration. This results in twice the values each as a different type in a separate entity.

To migrate, we choose migration by hand and this we do with mapping models. This is also the first part of the answer to your question. We will do the migration in two steps, because it's taking long to migrate 2k entries and we like to keep the memory footprint low.

You could even go ahead and split these mapping models further to migrate only ranges of the entities. Say we got one million records, this may crash the whole process. It's possible to narrow the fetched entities down with a Filter predicate.

##Back to our two mapping models.

We create the first mapping model like this:

1. New File -> Resource -> Mapping Model

2. Choose a name, I chose StepOne

3. Set source and destination data model

##Mapping Model Step One

The multi pass migration doesn't need custom entity migration policies, however we will do it to get a bit more detail for this example. So we add a custom policy to the entity. This is always a subclass of NSEntityMigrationPolicy.

This policy class implements some methods to make our migration happen. However it's simple in this case so we will have to implement only one method: createDestinationInstancesForSourceInstance:entityMapping:manager:error:.

The implementation will look like this:

###StepOneEntityMigrationPolicy.m

#import "StepOneEntityMigrationPolicy.h"


@implementation StepOneEntityMigrationPolicy

- (BOOL)createDestinationInstancesForSourceInstance:(NSManagedObject *)sInstance 
									  entityMapping:(NSEntityMapping *)mapping 
											manager:(NSMigrationManager *)manager 
											  error:(NSError **)error
{
	// Create a new object for the model context
	NSManagedObject *newObject = 
		[NSEntityDescription insertNewObjectForEntityForName:[mapping destinationEntityName] 
									  inManagedObjectContext:[manager destinationContext]];
	
	// do our transfer of nsdate to nsstring
	NSDate *date = [sInstance valueForKey:@"timeStamp"];
	NSDateFormatter *dateFormatter = [[NSDateFormatter alloc] init];
	[dateFormatter setTimeStyle:NSDateFormatterMediumStyle];
	[dateFormatter setDateStyle:NSDateFormatterMediumStyle];	
	
	// set the value for our new object
	[newObject setValue:[dateFormatter stringFromDate:date] forKey:@"printedDate"];
	[dateFormatter release];
	
	// do the coupling of old and new
	[manager associateSourceInstance:sInstance withDestinationInstance:newObject forEntityMapping:mapping];
	
	return YES;
}

##Final step: the migration itself

I'll skip the part for setting up the second mapping model which is almost identical, just a timeIntervalSince1970 used to convert the NSDate to a double.

Finally we need to trigger the migration. I'll skip the boilerplate code for now. If you need it, I'll post here. It can be found at Customizing the Migration Process it's just a merge of the first two code examples. The third and last part will be modified as follows: Instead of using the class method of the NSMappingModel class mappingModelFromBundles:forSourceModel:destinationModel: we will use the initWithContentsOfURL: because the class method will return only one, maybe the first, found mapping model in the bundle.

Now we've got the two mapping models which can be used in every pass of the loop and send the migrate method to the migration manager. That's it.

NSArray *mappingModelNames = [NSArray arrayWithObjects:@"StepOne", @"StepTwo", nil];
NSDictionary *sourceStoreOptions = nil;

NSURL *destinationStoreURL = [[self applicationDocumentsDirectory] URLByAppendingPathComponent:@"CoreDataMigrationNew.sqlite"];

NSString *destinationStoreType = NSSQLiteStoreType;

NSDictionary *destinationStoreOptions = nil;

for (NSString *mappingModelName in mappingModelNames) {
	NSURL *fileURL = [[NSBundle mainBundle] URLForResource:mappingModelName withExtension:@"cdm"];
	
	NSMappingModel *mappingModel = [[NSMappingModel alloc] initWithContentsOfURL:fileURL];
	
	BOOL ok = [migrationManager migrateStoreFromURL:sourceStoreURL
											   type:sourceStoreType
											options:sourceStoreOptions
								   withMappingModel:mappingModel
								   toDestinationURL:destinationStoreURL
									destinationType:destinationStoreType
								 destinationOptions:destinationStoreOptions
											  error:&error2];
	[mappingModel release];
} 

Notes

• A mapping model ends in cdm in the bundle.

• The destination store has to be provided and should not be the source store. You can after successful migration delete the old and rename the new one.

• I did some changes to the data model after the creation of the mapping models, this resulted in some compatibility errors, which I could only solve with recreating the mapping models.

These questions are related:

https://stackoverflow.com/questions/4875553/memory-issues-migrating-large-coredata-datastores-on-iphone

https://stackoverflow.com/questions/4451951/multiple-pass-core-data-migration-in-chunks-with-ios

To quote the first link:

> This is discussed in the official > documentation in the "Multiple Passes" > section, however it looks like their > suggested approach is to divide up > your migration by entity type, i.e. > make multiple mapping models, each of > which migrate a subset of the entity > types from the complete data model.

Suppose your database schema has 5 entities, e.g. person, student, course, class, and registration to use the standard kind of example, where student subclasses person, class implements course, and registration joins class and student. If you have made changes to all these table definitions, you have to start at the base classes, and work your way up. So, you cannot start with converting registrations, because each registration record depends on having class and students there. So, you would start with migrating only the Person table, copying existing rows into the new table, and filling in whatever new fields are there (if possible) and discarding the removed columns. Do each migration inside an autorelease pool, so that once it is done, your memory is back to start.

Once the Person table is done, then you can convert the student table over. Then hop over to Course and then Class, and the finally the Registration table.

The other consideration is the number of records, if like Person had a thousand rows, you would have to, every 100 or so, execute the NSManagedObject equivalent of a release, which is to tell the managed object context [moc refreshObject:ob mergeChanges:NO]; Also set your stale data timer way low, so that memory is flushed often.