How to write a connector

Introduction

A connector is a simple script which imports data from other web services and put it in your cozy. Each connector is an independant application, managed by the Cozy Collect application.

To protect your data, each connector runs inside a container in order to sandbox all their interactions with your data.

How does it work ?

A connector is a NodeJS script. The target node version used to run your connector is the current LTS version (8 at the moment).

Like client side apps, connectors communicate with the Cozy Stack using its API, and get an auth token every time they start. They need to register with a manifest, and ask permissions to the user.

To ease the development of a connector, a npm package, named cozy-konnector-libs provides some shared libraries which are adapted to be used for a connector :

But you may need some other npm packages not integrated in cozy-konnector-libs to help you run your connector :

When the connector is started, it also gets some data through environment variables:

Those variables are used by the BaseKonnector and the cozy-client to configure the connection to the Cozy Stack with the right permissions as defined in your manifest.konnector. These are simulated in standalone mode so that you don’t need a real cozy stack to develop your connector.

More information

From the server point of view, each connector is a job that is run periodically via a trigger. More information

Let’s create our first connector

The easiest way to create a new connector is to use cozy-konnector-template:

cozy-konnector-template and standalone mode

git clone https://github.com/cozy/cozy-konnector-template cozy-konnector-monservice
cd cozy-konnector-monservice
yarn # or npm install

note: the Cozy Team uses yarn, but if you prefer npm, just keep using it, everything should just work.

The connector template is ready with demo code to show you how to scrape a fictional website: books.toscrape.com, for which you do not need to have credentials.

As indicated in the README, just run it:

yarn standalone

The first run will create a konnector-dev-config.json file which allows you to configure the input of the connector when running it in the CLI.

{
  "COZY_URL": "http://cozy.tools:8080",
  "fields": {}
}

COZY_URL is for later, but the fields attribute will allow you to define credentials to the target web service, like login and password as if they would come from a real Cozy Stack.

This way to run the connector is the standalone mode. In this mode, cozyClient is stubbed and all data meant to be saved in a cozy is displayed in the standard output and files are directly saved in the root directory of the connector. This is useful to first develop your connector without handling the state of a real Cozy Stack.

You have more documentation about this in the CLI section of the documentation.

Connector structure

Basically, a connector is just a function passed to the BaseKonnector constructor, and which eventually returns a promise:

To create the connector, just create a new instance of BaseKonnector with the function as argument

const {BaseKonnector} = require('cozy-konnector-libs')

module.exports = new BaseKonnector(fields => {
  // use fields to get user credentials and choices
  console.log(fields, 'fields')
})

Fetch operations

Every time the connector is run, it will call the function and wait for the resolution of the returned promise. This function can then log into the remote site, fetch data and save it in the form of an array of objects with specific attributes expected by the different saving functions (saveFiles, addData, filterData, saveBills).

A basic connector workflow involves:

Error handling

If your connector hits an issue fetching or saving the data, it can return an error code by throwing it as an error. The error codes are defined inside the Cozy Collect application and will display an explicit error to the user:

You can get the list of error codes in require('cozy-konnector-libs').errors

const {BaseKonnector, errors} = require('cozy-konnector-libs')

module.exports = new BaseKonnector(fields => {
    // Here, the following message will be displayed in cozy-collect : "Bad credentials. Check the konnector fields and run the connection again."
    throw new Error(errors.LOGIN_FAILED)
})

cozy-konnector-libs

The Cozy Konnector Libs provide several useful methods for common tasks:

Linking with a cozy and dev mode

Once your connector is able to gather data from the targeted web service in standalone mode. Now is the time to put this data in a real cozy. Here comes the dev mode.

But before doing that, your connector needs more setup : a manifest.konnector file and konnector-dev-config.json‘s COZY_URL section.

The manifest

Each connector is described by a manifest. This is a JSON file named manifest.konnector at the root of your code folder. It should include the following minimal information:

{
  "name": "konnector name",
  "type": "node",
  "slug": "konnectorslug",
  "description": "description",
  "source": "git://github.com/cozy/cozy-konnector-thename.git",
  "permissions": {
    "accounts": {
      "description": "Required to get the account's data",
      "type": "io.cozy.accounts",
      "verbs": ["GET"]
    }
  }
}

cozy-konnector-template already has a manifest which you can customize.

You may add some permissions for your own doctype. Here is the detailed list of fields for a connector manifest file.

You can also get more information on permissions in the official cozy-stack documentation

konnector-dev-config.json

If you want to put data from your connector to a real cozy, your must define where to find this cozy, and this must be a cozy for which you have the credentials.

Here comes the COZY_URL in konnector-dev-config.json which defines just that.

Then you just have to run:

yarn dev

And for the first run, the CLI will open a tab in your browser asking you to give permissions to the connector and the connector will save data directly in your cozy. This will validate that your manifest has the needed permissions on the data you want to save.

this is the dev mode

Integration in cozy-collect for all the users

To run a connector, we do not want the cozy to install all dependencies of the connector each time it installs it.

To avoid this, the connectors need to be compiled into one file in a dedicated branch of the repository and the repository needs to be a public git repository. The package.json file from cozy-konnector-template gives you the commands to do this : yarn build and yarn deploy but the last one needs to be configured in package.json

Once your public git repository is configured, you only have to declare it.

Cozy will soon have a store for connectors and you will be able to publish connectors yourself. But at the moment, you need to declare your new connector on the cozy forum. The Cozy team will review your code and add your connector to the Cozy Collect application.

FAQ

When I run my connector, a ghost node process eats all my memory

Cozy-konnector-libs uses cheerio which is great but causes some problems when you try to console.log a cheerio object.

In standalone or dev mode, the BaseKonnector tries to catch errors and display a maximum of details about them. But when the error contains a cheerio object, the problem happens.

If you get this problem, catch the error yourself and only display the message :

.catch(err) {
  console.log(err.message) // good
  console.log(err) // bad
}

How do I scrap a website

Use the request function from cozy-konnector-libs with the proper options

Here’s a sample code that will fetch the login page to get the value of the anti-CSRF token, submit the login form, browse to the bills page and fetch a bill:

const {BaseKonnector, requestFactory} = require('cozy-konnector-libs')
const rq = requestFactory({
  jar: true, // handle the cookies like a browser
  json: false, // do not try to parse the result as a json document
  cheerio: true // automatically parse the result with [cheerio](https://github.com/cheeriojs/cheerio)
})
const moment = require('moment')

module.exports = new BaseKonnector(function fetch (fields) {
  return rq("https://login.remote.web")
  .then($ => { // the result is automatically wrapped with cheerio and you can use it like jQuery
    const form = {
      form: {
        login: fields.login,
        password: fields.password,
        csrf_token: $('[name="csrf_token"]').val(),
      }
    }
    return rq({
      method: 'POST',
      form
    })
  })
  .then($ => rq("https://admin.remote.web/bills"))
  .then($ => {
    return [{date: moment($("#bill_date")), value: $("#bill_value")}]
  })
  .then(entries => addData(entries, 'io.cozy.bills'))
})