Storage module design and API for review

gistfile1.md

Storage

Motivation

There are 3 major types of storage is browsers up to date:

• IndexedDB. The future.
• localStorage. Limited in size and can be observed through the storage event.
• WebSQL. Dropped by the W3C but it's still very present in the wild, with notable platforms like iOS and PhoneGap.

Green: IndexedDB; blue: WebSQL; yellow: localStorage

In the mobile world local cache and storage is very important for various reasons:

• faster loading times
• unreliable connections
• offline use

My particular use case is an app to be used "in the field" where you probably don't have a connection and you need to do data entry. My solution is to save to a local storage and sync to the server when a connection is available. The 5 MB of localStorage may not be enough in my case, but I won't be able to tell until the app is used by real users.

Open questions

• Is there enough interest in this considering the availability of localStorage?
• Is it too early to use these technologies?

Architecture

The goal is to use conditional loading to load one of 3 or 4 sub-modules: indexeddb, websql, localstorage and a fallback to memory.

• IndexedDB uses a key-based storage organized in "object stores".
• WebSQL uses SQL tables. WebSQL can be used as a fallback for object stores by matching a table to a store and using tables with two columns: a key and a value column which saves the object serialized as JSON.
• IndexedDB uses structured cloning which is similar to JSON.parse(JSON.stringify(obj)).

Open questions

• Is falling back to localStorage a good idea considering the limited size and observability?
• Is the whole concept OK or is it too hackish?
• Should this include over-the-wire databases via some protocol?

API design

The API aims to find the lowest common denominator between all implementations. A database requires:

• A name. Required by both IndexedDB and WebSQL
• A version. Required by both IndexedDB and WebSQL
• A structure, which means the name of each object store, tied to the version. Required by both IndexedDB and WebSQL
• A size in bytes. Required by WebSQL

A Storage constructor receives all these parameters and creates an instance with properties with the names of each object store:

var storage = new Y.Storage({
  name: 'my database',
  version: 1,
  size: 5242880,
  stores: ['fooStore', 'barStore']
});

Assert.isObject(storage.fooStore); // true
Assert.isObject(storage.barStore); // true

A Storage instance also has a close method that closes the database connection/session.

A Store object is created for each requested store in a property of the same name. A store can perform the following actions:

• get. Retrieve an object by key
• put. Insert or update an object by key
• remove. Remove an object by key
• clear. Remove all objects from the store
• count. Count the objects in the store

All methods return a promise.

Example:

storage.fooStore.put('some key', { foo: 'bar' }).then(...);

Open questions

• Should Storage inherit from Base in order to use attributes for configuration and maybe fire close or error events? A versionchange event is very important.

Thank you very much for the feedback guys!

Some answers:

@ItsAsbreuk

The attribute-events can be used when a user wants to change the memorysize, or add new stores

Actually you can't do that. In order to change the size and the structure, you need close the database and open a new one with a different version number. Abstracting over that seems hard and impractical.

It would be great if the storage could save and return simple types instead of just a String

Indeed, IndexedDB lets you do that and Storage uses JSON to store objects and arrays in WebSQL.

It would be awesome if we could store Y.Model (or descendents) instances without having to transform the attributes to an object ourself

Absolutely! WebSQL should already do it in my implementation since it passed the object through JSON.stringify which looks for toJSON. The IndexedDB side should look for toJSON.

@caridy

our target should not be browser-level storage mechanism, but any storage mechanism, including over-the-wire

I'd like to include anything local, including web app wrappers like PhoneGap or any other that I find. The reason why I didn't include over-the-wire implementations is that I want to leave the door open for different syncing/conflict resolution algorithms. I want to be able to have Models that sync to Storage and, when available, sync to the server via a REST-like API.

Basically my idea so far has been to handle this at the Model level.

Updated the gist: just promises, no callbacks; all actions return promises. Also added the question to discuss if this should include over-the-wire databases.

Ouch! light bulb

@caridy by over-the-wire do you mean using this in a Node environment to abstract over Mongo, MySQL or whatever? If that's the case, then I'm totally interested. Sadly my side project migrated from Node to Python, so I won't be using YUI on the server to test this in real life.

@juandopazo, yes, I'm talking about Node. And that algorithm for conflict resolution is called "latency compensation" and there are a couple of interesting implementations. The one from Meteor works pretty well, we might be able to use that one. In theory, there is not much different between a REST-like API and IndexedDB.

Aside from that, I think we should use Y.DB instead of Y.Storage. I really liked the implementation from @sdesai for the CRT project.

@caridy I guess I need a bit more research then. I'll look at Meteor and the literature on latency compensation.

I'd love to see @sdesai's implementation!

@juandopazo, this is a good read on latency compensation:

https://groups.google.com/forum/?fromgroups=#!topic/meteor-talk/MxrDzfi0sr8

Quite interesting "latency compensation".
Shouldn't YUI support this inside the core?