Offline / Online Data Synchronization Design (Javascript)

I'm currently in the process of writing an offline webapp using all the html5 goodies for offline support. However I'm starting now to think about writing the sync module that will ensure that any offline data gets sent to the server and server data back to the client. Now I'm sure this has been done before, I mean its a pretty classic design issue that affects mobile devices and a plethora of other things. So I'm wondering can anyone point me to some good design resources for this kind of thing?

Now I really do not need to be too sophisticated with this, I mean I'm not handling multiple users accessing the same data and I'm happy not to merge conflicts (just take the latest) but still I would like a design that will allow me those options in the future.

Also, are there any open source projects implementing this type of thing? I'm not above ripping off someone else's code (if license allows) and I'm happy to port.

I had a similar problem. I decided to use a purely JSON in and out approach. The solution I'm taking on form submission is:

1. catch the form submit event
2. check whether or not the user is online
3. if user is online then submit the form as normal form POST
4. if user is offline then stringify a JSON request and store it locally (I decided to use Web SQL Database). Queue table is simply Uri and Payload.

Then I have global event hooks for the online / offline events. When the user comes back online, it checks the queue, and if the queue has items in it, it then sends them through as JSON POST requests.

If you are primarily interested in getting JSON data and caching it for offline usage, then take a look at jquery.offline.

The challenge with synchronizing in both direction is that you need to update the local cached lists with any CRUD work that you have queued.

I'd like to find a more generic way to do this.

My plan for a similar design (not yet tried) is to use something like PouchDB to store the data locally and then sync it with a remote couch instance.

Check out Derby, a Node MVC framework that has some pretty sweet synchronization and conflict resolution features. http://derbyjs.com/

in our team we have already developed app in offline/online mode.

we are using the next following libraries:

• rack-offline
• jquery
• backbonejs
• backbonejs-localStorage
• backbonejs-queues
• jammit

Using rack-offline we are caching all resources files and jst template for rendering content on the page. backbonejs and backbonejs-localStorage helps to make MVC app on the client. it's pretty awesome, you should try it. we are always using localstorage for saving data. when we create post for example model object and saving to the localStorage, we are triggering queues for syncing(also we have by timer background worker for auto running sync process). For each model we have separate sync class that should be run by queue sync trigger. if your navigator.onLine => true we are sending requests to the server with data for updating. if you close browser, anyway you don't loose your data because you have queues in the localStorage. in the next time client will sync data on the first loading with navigator.onLine => true.

How to use rack-offline you can check my small project in the github:

pomodoro-app

Good luck!

I faced the same problem and ended up using an XML-file for storage and git to track changes and commit them automatically, as soon as a connection is available. The sync is done with the usual git commit / push / pull commands in a shell script and a cronjob starting the script. This would also work if you store JSON in a textfile.

I'm currently working on similar webapp. I've decided to make such workflow:

1. Form isn't really submitted - "Submit" button actually saves serialized form data to localStorage (in some queue). This saves from troubles with submit capturing and from writing additional error processing code to handle disconnect during form submission.
2. Transport script is triggered after data saving. It checks online/offline state.
3. When online, it tries to send latest data from queue to server (AJAX request), and deletes it from queue on success (and continues to send next data from queue after short timeout).
4. It shedules re-check after some period of time (by setTimeout()).

If you are up for using the potentially heavy Ext JS / Sencha framework, it has a nice data API with offline (e.g. localStorage) support and a proxy approach for write-thru to local then server. I use Sencha Touch (mobile-specific).

For debugging web storage, check out Weinre.

DerbyJS were probably the best solution. However Derby is still in development and offline support is only in planning and has not yet been implemented. In the Google Group ( http://groups.google.com/group/derbyjs/browse_thread/thread/7e7f4d6d005c219c ) you can find aditional information about what is planned in the future.

I'd personally recommend you write a wrapper on top of the indexedDB API that checks whether you are online/offline.

• if offline, just store in indexedDB and set persisted flag to false on all documents
• if online, get all documents where persisted is false and store them in mongodb or something equivelant on the backend, then store new documents in both indexedDB and on the server with the persisted flag to true.

I've written a small one

You would have to augment the tunnel to set the persisted flag automatically and also tunnel the synchronization of these documents to the backend