

Replication¶

Replication is the ability of a cozy-stack to copy / move all or a subset of its data to another support. It should cover 2 use cases

Devices: The continuous act of syncing change to a subset of the cozy documents and files to and from an user cozy to the same user’s Devices through cozy-desktop and cozy-mobile
Sharing: The continuous act of syncing change to a subset of the cozy documents and files to and from another user’s cozy.

Replication will not be used for Moving, nor Backup. See associated docs in this folder.

CouchDB replication is a well-understood, safe and stable algorithm ensuring replication between two couchdb databases. It is delta-based, and is stateful: we sync changes since a given checkpoint.

Files synchronization¶

Replication of too heavy database (with a lot of attachments) has been, in cozy experience, the source of some bugs. To avoid that (and some other issues), in cozy-stack, the attachments are stored outside of couchdb.

This means we need a way to synchronize files in parallel to couchdb replication.

Rsync is a well-understood, safe and stable algorithm to replicate files hierarchy from one hosts to the other by only transfering changes. It is one-shot and stateless. It could be an inspiration if we have to focus on syncing small changes to big files. The option to use rsync/zsync have been discussed internally (https://github.com/cozy/cozy-stack/pull/57 & Sprint Start 2016-10-24), but for now we should focus on using the files/folders couchdb document for synchronization purpose and upload/download the actual binaries with existing files routes or considering webdav.

Useful golang package: http://rclone.org/ sync folder in FS / Swift (may be we can add a driver for cozy)

Couchdb replication & limitation¶

Couchdb 2 replication protocol is described in details here.

Quick summary¶

The replicator figures out what was the last sequence number lastseqno which was correctly synced from the source database, using a _local/:replicationid document.
The replicator GET :source/changes?since=lastseqno&limit=batchsize and identify which docs have changed (and their open_revs).
The replicator POST :target/_revs_diff to get a list of all revisions of the changed documents the target does not know.
The replicator GET :source/:docid?open_revs=['rev1', ...] several times to retrieve the missing documents revisions.
The replicator POST :target/_bulk_docs with these documents revisions.
Store the new last sequence number

Repeat 2-5 until there is no more changes.

Details¶

In step 5, the replicator can also attempt to PUT :target/:docid if doc are too heavy, but this should not happens in cozy-stack considering there wont be attachment in couchdb.
In step 4, the replicator can optimize by calling GET
The main difference from couchdb 1.X is in the replication history and the manner to determine and store the last sequence number. TODO: actually understand this and how it might increase disk usage if we have a lot of replications.
Couchdb _changes and by extension replication can be either by polling or continuous (SSE / COMET)
In couchdb benchmarking, we understood that the number of couch databases only use a bit of disk space but no RAM or CPU as long as the database is not used. Having a continuous replication active force us to keep the database file open and will starve RAM & FD usage. Moreover, continuous replication costs a lot (cf. ScienceTeam benchmark). To permit an unlimited number of inactive user on a single cozy-stack process, the stack should avoid continuous replication from couchdb.
Two way replication is simply two one way replications

Routes used by replication¶

To be a source of replication, the stack only need to support the following route (and query parameters):

GET :source/:docid get revisions of a document. The query parameters open_revs, revs, latest is necessary for replication.
POST :source/_all_docs is used by current version of cozy-mobile and pouchdb as an optimization to fetch several document’s revision at once.
GET :source/_changes get a list of ID -> New Revision since a given sequence number. The query parameters since, limit are necessary for replication.

To be a target of replication, the stack need to support the following routes:

POST :target/_revs_diff takes a list of ID -> Rev and returns which one are missing.
POST :target/_bulk_docs create several documents at once
POST :target/_ensure_full_commit ensure the documents are written to disk. This is useless if couchdb is configured without delayed write (default), but remote couchdb will call, so the stack should return the expected 201

In both case, we need to support

PUT :both/_local/:revdocid to store the current sequence number.

Stack Sync API exploration¶

Easy part: 1 db/doctype on stack AND remote, no binaries¶

We just need to implement the routes described above by proxying to the underlying couchdb database.

var db = new PouchDB("my-local-contacts");
db.replicate.from("https://bob.cozycloud.cc/data/contacts");
db.replicate.to("https://bob.cozycloud.cc/data/contacts");

To suport this we need to:

Proxy /data/:doctype/_changes route with since, limit, feed=normal. Refuse all filter parameters with a clear error message. (Doc)
Add support of open_revs, revs, latest query parameter to GET /data/:doctype/:docid (Doc)
Proxy the /data/:doctype/_revs_diff (Doc) and /data/:doctype/_bulk_docs routes (Doc) routes
Have /data/:doctype/_ensure_full_commit (Doc) returns 201

This will cover documents part of the Devices use case.

Continuous replication¶

It is impossible to implement it by simply proxying to couchdb (see unlimited inactive users).

The current version of cozy-desktop uses it. TODISCUSS It could be replaced by 3-minutes polling without big losses in functionality, eventually with some more triggers based on user activity.

The big use case for which we might want it is Sharing. But, because Sharing will (first) be stack to stack, we might imagine having the source of event “ping” the remote through a separate route.

Conclusion: Start with polling replication. Consider alternative notifications mechanism when the usecase appears and eventually use time-limited continuous replication for a few special case (collaborative edit).

Realtime¶

One of the cool feature of cozy apps was how changes are sent to the client in realtime through websocket. However this have a cost: every cozy and every apps open in a browser, even while not used keep one socket open on the server, this is not scalable to thousands of users.

Several technology can be used for realtime: Websocket, SSE or COMET like long-polling. Goroutines makes all solution similar performances wise. COMET is a hack, websocket seems more popular and tested (x/net/websocket vs html5-sse-example). SSE is not widely available and has some limitations (headers…)

Depending on benchmarking, we can do some optimization on the feed:

close feeds when the user is not on screen
multiplex different applications’ feed, so each open cozy will only use one socket to the server. This is hard, as all apps live on separate domain, an (hackish) option might be a iframe/SharedWorker bridge.

To have some form of couchdb-to-stack continuous changes monitoring, we can monitor _db_udpates (Doc) which gives us an update when a couchdb has changed. We can then perform a _changes query on this database to get the changed docs and proxy that to the stack-to-client change feed.

Conclusion: We will use Websocket from the client to the stack. We will try to avoid using continuous changes feed from couchdb to the stack. We will optimize if proven needed by benchmarks, starting with “useless” changes and eventually some multiplexing.

To be completed by discussing with Science team.

Current ideas (as understood by Romain)

any filtered replication is unscalable
1 db for all shared docs sharing_db.
Cozy-stack is responsible for saving documents that should be shared in both their dg implemented by 2-way replication between the 2 users sharing_db, filtering is done by computing a list of IDs and then doc_ids (sharing with filters/views are not efficient)
Sharing is performed by batches regularly.
Continuous replication can be considered for collaborative editing.

Proposal by Romain, if we find _selector filter replication performances to be acceptable on very large / very old databases.

No sharing database
A permission, for anything is a mango-style selector.
on every query, the Mango selector is checked at the stack or couchdb level ($and-ing for queries, testing output document, input document)
Sharing is a filtered replication between user’s 1 doctypedb et user’s 2 samedoctypedb
No continuous replication
Upon update, the stack trigger a PUSH replication to its remote or “ping” the remote, and the remote perform a normal PULL replication.

TODO experiment with performance of _selector filtered replication in couchdb2