How to sync CoreData and a REST web service asynchronously and the same time properly propagate any REST errors into the UI

Hey, I'm working on the model layer for our app here.

Some of the requirements are like this:

1. It should work on iPhone OS 3.0+.
2. The source of our data is a RESTful Rails application.
3. We should cache the data locally using Core Data.
4. The client code (our UI controllers) should have as little knowledge about any network stuff as possible and should query/update the model with the Core Data API.

I've checked out the WWDC10 Session 117 on Building a Server-driven User Experience, spent some time checking out the Objective Resource, Core Resource, and RestfulCoreData frameworks.

The Objective Resource framework doesn't talk to Core Data on its own and is merely a REST client implementation. The Core Resource and RestfulCoreData all assume you talk to Core Data in your code and they solve all the nuts and bolts in the background on the model layer.

All looks okay so far and initially I though either Core Resource or RestfulCoreData will cover all of the above requirements, but... There's a couple of things none of them seemingly happen to solve correctly:

1. The main thread should not be blocked while saving local updates to the server.
2. If the saving operation fails the error should be propagated to the UI and no changes should be saved to the local Core Data storage.

Core Resource happens to issue all of its requests to the server when you call - (BOOL)save:(NSError **)error on your Managed Object Context and therefore is able to provide a correct NSError instance of the underlying requests to the server fail somehow. But it blocks the calling thread until the save operation finishes. FAIL.

RestfulCoreData keeps your -save: calls intact and doesn't introduce any additional waiting time for the client thread. It merely watches out for the NSManagedObjectContextDidSaveNotification and then issues the corresponding requests to the server in the notification handler. But this way the -save: call always completes successfully (well, given Core Data is okay with the saved changes) and the client code that actually called it has no way to know the save might have failed to propagate to the server because of some 404 or 421 or whatever server-side error occurred. And even more, the local storage becomes to have the data updated, but the server never knows about the changes. FAIL.

So, I'm looking for a possible solution / common practices in dealing with all these problems:

1. I don't want the calling thread to block on each -save: call while the network requests happen.
2. I want to somehow get notifications in the UI that some sync operation went wrong.
3. I want the actual Core Data save fail as well if the server requests fail.

Any ideas?

You should really take a look at RestKit (http://restkit.org) for this use case. It is designed to solve the problems of modeling and syncing remote JSON resources to a local Core Data backed cache. It supports an offline mode for working entirely from the cache when there is no network available. All syncing occurs on a background thread (network access, payload parsing, and managed object context merging) and there is a rich set of delegate methods so you can tell what is going on.

There are three basic components:

1. The UI Action and persisting the change to CoreData
2. Persisting that change up to the server
3. Refreshing the UI with the response of the server

An NSOperation + NSOperationQueue will help keep the network requests orderly. A delegate protocol will help your UI classes understand what state the network requests are in, something like:

@protocol NetworkOperationDelegate
  - (void)operation:(NSOperation *)op willSendRequest:(NSURLRequest *)request forChangedEntityWithId:(NSManagedObjectID *)entity;
  - (void)operation:(NSOperation *)op didSuccessfullySendRequest:(NSURLRequest *)request forChangedEntityWithId:(NSManagedObjectID *)entity;
  - (void)operation:(NSOperation *)op encounteredAnError:(NSError *)error afterSendingRequest:(NSURLRequest *)request forChangedEntityWithId:(NSManagedObjectID *)entity;
@end

The protocol format will of course depend on your specific use case but essentially what you're creating is a mechanism by which changes can be "pushed" up to your server.

Next there's the UI loop to consider, to keep your code clean it would be nice to call save: and have the changes automatically pushed up to the server. You can use NSManagedObjectContextDidSave notifications for this.

- (void)managedObjectContextDidSave:(NSNotification *)saveNotification {
  NSArray *inserted = [[saveNotification userInfo] valueForKey:NSInsertedObjects];
  for (NSManagedObject *obj in inserted) {
    //create a new NSOperation for this entity which will invoke the appropraite rest api
    //add to operation queue
  }
  
  //do the same thing for deleted and updated objects
}

The computational overhead for inserting the network operations should be rather low, however if it creates a noticeable lag on the UI you could simply grab the entity ids out of the save notification and create the operations on a background thread.

If your REST API supports batching, you could even send the entire array across at once and then notify you UI that multiple entities were synchronized.

The only issue I foresee, and for which there is no "real" solution is that the user will not want to wait for their changes to be pushed to the server to be allowed to make more changes. The only good paradigm I have come across is that you allow the user to keep editing objects, and batch their edits together when appropriate, i.e. you do not push on every save notification.

This becomes a sync problem and not one easy to solve. Here's what I'd do: In your iPhone UI use one context and then using another context (and another thread) download the data from your web service. Once it's all there go through the sync/importing processes recommended below and then refresh your UI after everything has imported properly. If things go bad while accessing the network, just roll back the changes in the non UI context. It's a bunch of work, but I think it's the best way to approach it.

[Core Data: Efficiently Importing Data][1]

[Core Data: Change Management][2]

[Core Data: Multi-Threading with Core Data][3]

[1]: http://developer.apple.com/mac/library/documentation/Cocoa/Conceptual/CoreData/Articles/cdImporting.html "Core Data: Efficiently Importing Data" [2]: http://developer.apple.com/mac/library/documentation/Cocoa/Conceptual/CoreData/Articles/cdChangeManagement.html#//apple_ref/doc/uid/TP30001201-CJBDBHCB [3]: http://developer.apple.com/mac/library/documentation/Cocoa/Conceptual/CoreData/Articles/cdMultiThreading.html#//apple_ref/doc/uid/TP40003385-SW1

You need a callback function that's going to run on the other thread (the one where actual server interaction happens) and then put the result code/error info a semi-global data which will be periodically checked by UI thread. Make sure that the wirting of the number that serves as the flag is atomic or you are going to have a race condition - say if your error response is 32 bytes you need an int (whihc should have atomic acces) and then you keep that int in the off/false/not-ready state till your larger data block has been written and only then write "true" to flip the switch so to speak.

For the correlated saving on the client side you have to either just keep that data and not save it till you get OK from the server of make sure that you have a kinnf of rollback option - say a way to delete is server failed.

Beware that it's never going to be 100% safe unless you do full 2-phase commit procedure (client save or delete can fail after the signal from the server server) but that's going to cost you 2 trips to the server at the very least (might cost you 4 if your sole rollback option is delete).

Ideally, you'd do the whole blocking version of the operation on a separate thread but you'd need 4.0 for that.