Skip to content

Table of contents

Konnectors workflow

TL;DR

Konnectors workflow

Mermaid source file for diagram above

Konnectors models

Types

Accounts

io.cozy.accounts contains information for an account, including those for authentication (technically, they are encrypted by cozy-stack before being saved in CouchDB, but we will show them as the konnectors will see them):

{
    "auth": {
        "login": "000000000",
        "password": "**********"
    },
    "folderPath": "/Administratif/Free Mobile",
    "label": "freemobile",
    "namePath": "Free Mobile"
}

Accounts are manipulated through the /data/ API.

Decryption

The decryption of the credentials is reserved to requests coming from konnectors and services. For services, it is only available on /data/io.cozy.acconts/:account-id with an additional include=credentials parameter.

Aggregator accounts

Some konnectors are based on an aggregator service. An aggregator is declared in the konnector manifest and specify an accountId property. When Cozy-Home detects this property, it checks if an account with this given id exists. If not, it creates it. Every account created for a konnector based on an aggregator is then related to this aggregator account. This relationship is called parent account.

{
    "auth": {
        "login": "000000000",
        "password": "**********"
    },
    "folderPath": "/Administratif/Bankbank",
    "label": "Bankbank",
    "namePath": "Bankbank",
    "relationships": {
      "parent": {
        "data": {
          "_id": "service-aggregator-account",
          "_type": "io.cozy.accounts"
        }
      }
    }
}

Note: you can read more about the accounts doctype here.

Konnectors

io.cozy.konnectors are installed similarly to io.cozy.apps (see doc)

Permissions

Like client-side applications, each konnector has an associated io.cozy.permissions. These permissions are those listed on the manifest of the konnectors, and the stack will add a permission for the files on the folder choosen by the user.

Triggers

io.cozy.triggers are used to define when konnectors are launched. See https://docs.cozy.io/en/cozy-stack/jobs/#post-jobstriggers

Complete flow example

As a user, from the expenses management app, I have a clean flow to configure a connector to retrieve my travel expenses

1 - User is in my-expenses and clicks on [configure travels]

2 - my-expenses triggers an intent

cozy.intents.start("CREATE", "io.cozy.konnectors", {
    category: "transport"
});

3 - the Store app catch the intent, fetch all available konnectors and let the user choose

GET /registry?... HTTP/1.1

4 - the user chooses the trainline konnector. Its manifest looks like this:

{
    "name": "Trainline",
    "type": "konnector",
    "slug": "konnector-trainline",
    "description": "Konnector for trainline . com",
    "source": "https://github.com/konnectors/trainlines.git@build",
    "developer": {
        "name": "XXX",
        "url": "https://www.xxx.fr"
    },
    "version": "3.0.0",
    "licence": "AGPL-3.0",
    "fields": {
        "login": {
            "type": "text"
        },
        "password": {
            "type": "password"
        },
        "advancedFields": {
            "folderPath": {
                "advanced": true,
                "isRequired": false
            }
        }
    },
    "category": "transport",
    "frequency": "weekly",
    "permissions": {
        "events": {
            "description": "Connect train bill with  event in your calendar",
            "type": "io.cozy.events",
            "verbs": ["PATCH"]
        }
    }
}

5 - the user clicks on install, and the install starts:

POST /konnectors/trainline HTTP/1.1

6 - the Store then uses an intent to know which app can configure this konnector:

cozy.intents.start("REDIRECT", "io.cozy.accounts", {
    slug: "trainline"
});

7 - the Store redirects to the Home. The Home asks the user for account config and create a folder to save the PDFs.

8 - the Home also add a Reference from the konnector to the folder to prevent any accidental folder deletion:

POST /files/123-selected-folder-id-123/relationships/referenced_by

{
    "data": [
        {
            "type": "io.cozy.konnectors",
            "id": "io.cozy.konnectors/trainlines"
        }
    ]
}

9 - then the Home can create the account:

POST /data/io.cozy.accounts HTTP/1.1

{
    "account_type": "trainline",
    "auth": {
        "login": "xxxx",
        "password": "yyyyy"
    },
    "folderPath": "/Administrative/Trainline"
}

HTTP/1.1 200 OK

{
    "_id": "123-account-id-123",
    "_rev": "1-asasasasa",
    "account_type": "trainline",
    "auth": {
        "login": "xxxx"
    },
    "folderPath": "/Administrative/Trainline"
}

10 - the Home create the trigger:

POST /jobs/io.cozy.triggers

{
    "data": {
        "attributes": {
            "type": "@cron",
            "arguments": "0 0 0 0 1 1 ",
            "worker": "konnector",
            "message": {
                "konnector": "trainline",
                "account": "5165621628784562148955",
                "folder_to_save": "877854878455"
            }
        }
    }
}

11 - and, finally, the Home runs the konnector for the first time:

POST /jobs/triggers/abc159753/launch HTTP/1.1

12 - the Home uses the realtime to be informed if the konnector fails to login or succeeds to import the PDFs.

If the user wants to use several account, the Home can setup several triggers for the same konnector & various accounts.

Konnector Worker specs

Prepare the execution

The cozy-stack prepares the execution of the konnector by doing these steps:

  • it checks that the konnector is not in maintenance in the registry (except for manual execution)
  • it ensures that the konnector has a folder where it can write its files, and has the permission to write in this folder.

Execute the konnector

Start the konnector through nsjail, passing as ENV variables:

- `COZY_URL`:          the starting instance URL
- `COZY_CREDENTIALS`:  security token to communicate with Cozy
- `COZY_FIELDS`:       JSON-encoded message with the arguments from the trigger
- `COZY_PAYLOAD`:      JSON-encoded payload from the HTTP request (@webhook trigger)
- `COZY_PARAMETERS`:   JSON-encoded parameters from the konnector manifest
- `COZY_LANGUAGE`:     the language field of the konnector (eg. "node" etc.)
- `COZY_LOCALE`:       the locale of the user (eg. "en" etc.)
- `COZY_TIME_LIMIT`:   how much time the konnector can run before being killed
- `COZY_JOB_ID`:       id of the job
- `COZY_TRIGGER_ID`:   id of the trigger that has created the job
- `COZY_JOB_MANUAL_EXECUTION`: whether the job was started manually (in Home) or automatically (via a cron trigger or event)

The konnector process can send events trough its stdout (newline separated JSON object), the konnector worker pass these events to the realtime hub as io.cozy.jobs.events.

  • Only JSON formatted events are forwarded to the client-side through realtime
  • Otherwise formatted lines (such as node Error) will be kept in some system logs.

Konnectors should NOT log the received account login values in production.

Konnector error handling

The konnector can output json formated messages as stated before (the events) and those events will be typed and formatted like this:

{
    type: "messagetype",  // can be "debug", "info", "warning", "error", and "critical"
    message: "message"    // can be any string
}

If there is an error or critical message, the execution will be seen as a failure by cozy-stack. It’s also the case if the konnector reaches the timeout or returns with a non-zero status code.

Note: debug and info level are not transmitted to syslog, except if the instance is in debug mode. It would be too verbose to do otherwise.

The message can start with a known keyword that could have special meanings for cozy-stack or for statistical analysis of konnector health. Known keywords are listed in the konnector tutorial

When a connector throw an error of either LOGIN_FAILED or any error starting with USER_ACTION_NEEDED (with the exception of USER_ACTION_NEEDED.CGU_FORM), meaning that the user needs to execute an action on provider’s website (either changing its credentials or acknowledging something on provider website), then cozy-stack will skip all subsequent automatic executions to avoid querying the provider site and blocking user account. Manual execution by the user will still be possible to unlock the konnector/account when the user thinks it is now ready to be run again.

Account deleted

When an account is deleted, or a konnector is going to be uninstalled, the konnector is executed with the account_deleted field to true, so it can clean the account remotely.

OAuth (and service secrets)

Doctypes

io.cozy.konnectors gives their desiderata for an account

{
    "fields": {
        "account": {
            "doctype": "io.cozy.accounts",
            "account_type": "maif",
            "scope": "openid profile offline_access"
        }
    }
}

io.cozy.accounts contains authentication information for an account, as well as the associated scope

{
    "name": "Mon Compte Maif",
    "account_type": "maif",
    "status": "connected",
    "oauth": {
        "access_token": "akosaksoakso",
        "refresh_token": "okoakozkaozk",
        "scope": "openid profile offline_access"
    }
}

io.cozy.account_types contains the oauth configuration:

{
    "_id": "service.example",
    "grant_mode": "authorization_code",
    "client_id": "the registered client id",
    "client_secret": "client_secret is necessary for server-flow",
    "auth_endpoint": "https://service.example/auth",
    "token_endpoint": "https://api.service.example/token"
}

io.cozy.account_types are not accessible to the applications: they are injected directly in CouchDB in a global database secrets/io-cozy-account_types.

Secrets that are not OAuth

The io.cozy.account_types doctype can also be used for storing secrets that are not related to OAuth. The document still need to be injected manually in secrets/io-cozy-account_types:

{
    "_id": "service.example",
    "grant_mode": "secret",
    "slug": "service",
    "secret": "th1$_1$_th3_s3cr3t!"
}

Note: grant_mode must be secret (or bi_webauth+secret, or bi_webview+secret, or authorization_code+secret), slug must be the slug of the konnector, but secret can be a map instead of a simple string if several secrets are needed for this service.

The secret is given to the konnector in the COZY_PARAMETERS env variable.

Overloading an account type for a given context

It is possible to use a different account type for a given context, by creating a new document with an id prefix by the context name and /. For example, a different secret can be used in the foobar context by injecting this document in secrets/io-cozy-account_types:

{
    "_id": "foobar/service.example",
    "grant_mode": "secret",
    "slug": "service",
    "secret": "th1$_1$_n0t_th3_s4m3_s3cr3t!"
}

Reminder OAuth flow

Service is the website from which konnector aims to retrieve informations. Stack is the cozy stack.

OAuth is divided in 3 steps:

  • Client Registration: the client application (the Stack) needs to be registered with the Service
  • Obtaining & Refreshing Authorization: all the steps from client_id to access_token, the Stack will handle those
  • Using the access_token: ideally, the konnector should only concern itself with this part; it receives an access_token and uses it.

Client Registration

Before beginning the Grant process, most Services require the application to be registered with a defined redirect_uri.

There are a lot of options, which we will choose from when actually implementing konnectors using them.

Manually

Most services requires a human developer to create the client manually and define its redirect_uri. However each instance has its own domain, so for these services, we will need to:

A. Register a “proxy” client, which is a static page performing redirections as needed, as was done for Facebook events in v2 konnectors. We will register a well known cozy domain, like oauth-proxy.cozy.io and registers it with all providers. The use and risks associated with this domain should be made clear to the user.

B. Register each Stack with a redirect_uri on the main stack domain, if we go this way, the register_uri below moves from bob.cozy.rocks/accounts/service/redirect to oauthcallback.cozy.rocks/accounts/service/redirect and the domain will be prepended to the state. This is feasible at cozy scale, but requires more knowledge and work for self-hosters.

Example: Google

For example, these are the steps for creating the google account_type for the stack at .mycozy.cloud:

  • Go to https://console.developers.google.com
  • Select or Create Project (up left near the logo)
  • Enable desired APIs (TBD with usages)
  • Click “Credentials”
  • Create credentials > Oauth Client ID > Web application
  • Set redirectURI to https://oauthcallback.mycozy.cloud/accounts/google/redirect
  • Copy and paste provided Client ID and Client Secret.

Then save the data in the console

http PUT localhost:5984/secrets%2Fio-cozy-account_types/google
grant_mode=authorization_code
redirect_uri="https://oauthcallback.mycozy.cloud/accounts/google/redirect"
token_mode=basic
token_endpoint="https://www.googleapis.com/oauth2/v4/token"
auth_endpoint="https://accounts.google.com/o/oauth2/v2/auth"
client_id=$CLIENT_ID
client_secret=$CLIENT_SECRET

Dynamic Registration Protocol

A few services allows client to register programatically through the Dynamic Client Registration Protocol RFC7591, we should allow the Stack to register using this protocol when we first need to implement a Konnector connecting to such a Service.

No redirect_url enforcement

A few services allows to specify arbitrary redirect_url without registering beforehand as a client.

Authorization Grant flows

webserver flow (Authorization Code Grant)

A. In the cozy app (Home) give a link

<a href="https://bob.cozy.rocks/accounts/service-name/start?
  scope=photos&
  state=1234zyx">

Notes:

  • the scope may depends on other fields being configured (checkboxes), this will be described in json in the konnectors manifest. The format will be determined upon implementation;
  • to limit bandwidth and risk of state corruption, the cozy app should save its state under a random key into localStorage, the key is then passed as the state in this query;
  • a third parameter, slug, can be added to redirect to this app instead of the home.

B. Service lets the user login, allows or denies the scope, then redirects to

https://oauthcallback.cozy.rocks/accounts/service-name/redirect?
  code=AUTH_CODE_HERE&
  state=1234zyx

C. The Stack does Server side this request:

POST /oauth/access_token HTTP/1.1
Host: api.service.example
Content-Type: application/x-www-form-urlencoded

grant_type=authorization_code&
  code=AUTH_CODE_HERE&
  redirect_uri=https://oauthcallback.cozy.rocks/accounts/service-name/redirect&
  client_id=CLIENT_ID&
  client_secret=CLIENT_SECRET

Note: the stack will also send the state parameter on this request. It is not mandatory per the OAuth 2.0 spec, and the skip_state_on_token option can be used to cancel this behavior if the provider throw an error if this parameter is present.

D. The Service responds (server side) with:

{
    "access_token": "ACCESS_TOKEN",
    "token_type": "bearer",
    "expires_in": 2592000,
    "refresh_token": "REFRESH_TOKEN",
    "scope": "read",
    "uid": 100101,
    "info": {
        "name": "Claude Douillet",
        "email": "claude.douillet@example.com"
    }
}

This whole object is saved as-is into a io.cozy.accounts ‘s extras field.

The known fields access_token, refresh_token & scope will be also saved on the account’s oauth itself

E. The Stack redirect the user to the cozy app:

HTTP/1.1 302 Found
Location: https://bob-home.cozy.rocks/?state=1234zyx&account=accountID

The Cozy app checks that the state is expected, and restores its state to the form but whith account completed.

SPA flow (Implicit grant)

A. The cozy app gives a link:

<a href="https://service.example/auth? (url)
    response_type=token&
    client_id=CLIENT_ID&
    redirect_uri=https://bob-home.cozy.rocks/&
    scope=photos&
    state=1234zyx">

See server-flow for state rules.

B. Service lets the user login, allows or denies the scope, then redirects to

https://bob-home.cozy.rocks/?access_token=ACCESS_TOKEN&state=1234zyx

C. The Cozy app adds the token to the io.cozy.accounts and save it before starting konnector.

Accessing data

Once we have an account, the Cozy app starts the konnector with the proper account id. The konnector can then fetch the account and use its access_token to performs a request to fetch data:

POST /resource HTTP/1.1
Host: api.service.example
Authorization: Bearer ACCESS_TOKEN

Refreshing token

When using the server-flow, we also get a refresh_token. It is used to get a new access_token when it expires. However, if konnectors are responsible for refreshing token there is a race condition risk:

(konnector A) GET  https://api.service.com/resource TOKEN1     -> expired
(konnector B) GET  https://api.service.com/resource TOKEN1     -> expired
(konnector A) POST https://api.service.com/token REFRESH_TOKEN -> TOKEN2a
(konnector B) POST https://api.service.com/token REFRESH_TOKEN -> TOKEN2b
(konnector A) GET  https://api.service.com/token TOKEN2a       -> invalid

To avoid this, the stack will be responsible to perform token refresh. A konnector can requires the stack to refresh an account token with an HTTP request.

POST /accounts/:accountType/:accountID/refresh HTTP/1.1
Host: bob.cozy.rocks

Konnectors Marketplace Requirements

The following is a few points to be careful for in konnectors when we start allowing non-cozy developped OAuth konnectors.

  • With SPA flow, because of advanced security concerns (confused deputy problem), cozy should validate the access_token. However, the way to do that depends on the provider and cannot be described in json, it is therefore the responsibility of the konnector itself.

Account types security rules

  • With server flow, an evil account type with proper auth_endpoint but bad token_endpoint could retrieve a valid token as well as cozy client secret. The reviewer of an account_type should make sure both these endpoints are on domains belonging to the Service provider.

Notes for MesInfos experiment

  • MAIF konnector uses the webserver flow without redirect_uri validation
  • Orange konnector uses the client-side proxy but hosted on their own servers (/!\ redirect_uri vs redirect_url)

Webauth

For some banks integration (Paypal, Orange Bank, Revolut…), Budget Insight has a workflow similar to OAuth called Webauth.

It is possible to use this workflow for konnectors by registering an account type with the following parameter:

  • grant_mode, with bi_webauth as the value (or bi_webauth+secret if there is also a secret)
  • redirect_uri, with an URL like https://oauthcallback.mycozy.cloud/accounts/paypal/redirect
  • client_id, with the client ID given by Budget Insight
  • auth_endpoint, with https://{domain}.biapi.pro/2.0/webauth (with the correct domain)
  • reconnect_endpoint, with https://{domain}.biapi.pro/2.0/auth/webview/reconnect (idem).

BI Weview

A new integration of Budget Insight is available for all the proposed banks. This is a workflow similar to OAuth called webview

It is possible to use this workflow for konnectors by registering an account type with the following parameter:

  • grant_mode, with bi_webview+secret as the value
  • redirect_uri, with an URL like https://oauthcallback.mycozy.cloud/accounts/paypal/redirect
  • client_id, with the client ID given by Budget Insight
  • auth_endpoint, with https://{domain}.biapi.pro/2.0/auth/webview/fr/connect (with the correct domain)
  • manage_endpoint, with https://{domain}.biapi.pro/2.0/auth/webview/reconnect (idem).
  • reconnect_endpoint, with https://{domain}.biapi.pro/2.0/auth/webview/reconnect (idem).

Manage

The manage webview can be called with this route:

GET /accounts/:accountType/:accountID/manage?code={code}&connection_id={id}&slug={slug}&state={state} HTTP/1.1
Host: jane.cozy.example

HTTP/1.1 303 See Other
Location: https://domain.biapi.pro/2.0/auth/webview/manage
    ?client_id={client_id}
    &code={code}
    &connection_id={id}
    &redirect_uri=https://oauthcallback.cozy.example/accounts/{accountType}/redirect
    &state={stackState}

At the end of flow, the user will be redirected to https://jane-{slug}.cozy.example/?state={state}.

Reconnect

The reconnection webview can be called with this route:

GET /accounts/:accountType/:accountID/reconnect?code={code}&connection_id={id}&slug={slug}&state={state} HTTP/1.1
Host: jane.cozy.example

HTTP/1.1 303 See Other
Location: https://domain.biapi.pro/2.0/auth/webview/reconnect
    ?client_id={client_id}
    &code={code}
    &connection_id={id}
    &redirect_uri=https://oauthcallback.cozy.example/accounts/{accountType}/redirect
    &state={stackState}

At the end of flow, the user will be redirected to https://jane-{slug}.cozy.example/?state={state}.

Webhook triggers

Webhook trigger for a konnector

  1. The cloudery (or an application) can install the konnector and create an account

  2. It can also create the webhook trigger via a POST /jobs/triggers

Request:

POST /jobs/triggers HTTP/1.1
Host: jane.cozy.example
Content-Type: application/vnd.api+json

{
  "data": {
    "attributes": {
      "type": "@webhook",
      "worker": "konnector",
      "message": {
        "param_from_trigger": "foo"
      }
    }
  }
}

Response:

HTTP/1.1 200 OK
Content-Type: application/vnd.api+json

{
  "data": {
    "type": "io.cozy.triggers",
    "id": "0915b6b0-0c97-0139-5af7-543d7eb8149c",
    "attributes": {
      "type": "@webhook",
      "worker": "konnector",
      "message": {
        "konnector": "mykonnector",
        "param_from_trigger": "foo"
      }
    },
    "links": {
      "self": "/jobs/triggers/0915b6b0-0c97-0139-5af7-543d7eb8149c",
      "webhook": "https://jane.cozy.example/jobs/webhooks/0915b6b0-0c97-0139-5af7-543d7eb8149c"
    }
  }
}

  1. The cloudery can then give the webhook URL to the external service

  2. When the external service has new documents to import, it can call the webhook

Request:

POST /jobs/webhooks/0915b6b0-0c97-0139-5af7-543d7eb8149c HTTP/1.1
Host: jane.cozy.example
Content-Type: application/json

{"param_from_http_body": "bar"}

Response:

HTTP/1.1 204 No Content

  1. The stack will put a job in its queue for the konnector

  2. The stack executes the konnector with:

  • COZY_FIELDS={"param_from_trigger": "foo"}
  • COZY_PAYLOAD={"param_from_http_body": "bar"}
  • etc.

Note: the environnement variables have a limit for their size defined by the linux kernel. If the payload is too big to fit inside the env variable, the stack will put the payload in a temporary file, and will set the COZY_PAYLOAD variable to @ + the filename of this file, like:

COZY_PAYLOAD=@cozy_payload.json

  1. The konnector will fetch the documents from the external service and save them in the Cozy.

Note: if the trigger has a debounce, the COZY_PAYLOAD will be an object with a payloads array, like this:

COZY_PAYLOAD={"payloads": [{"param_from_http_body": "bar"}]}