API

Modules

addData

This function saves the data into the cozy blindly without check You need at least the POST permission for the given doctype in your manifest, to be able to use this function.

Parameters:

  • documents: an array of objects corresponding to the data you want to save in the cozy
  • doctype (string): the doctype where you want to save data (ex: 'io.cozy.bills')
const documents = [
  {
    name: 'toto',
    height: 1.8
  },
  {
    name: 'titi',
    height: 1.7
  }
]

return addData(documents, 'io.cozy.height')
cozyClient

This is a cozy-client-js instance already initialized and ready to use

If you want to access cozy-client-js directly, this method gives you directly an instance of it, initialized according to COZY_URL and COZY_CREDENTIALS environment variable given by cozy-stack You can refer to the cozy-client-js documentation for more information.

Example :

const {cozyClient} = require('cozy-konnector-libs')

cozyClient.data.defineIndex('my.doctype', ['_id'])
hydrateAndFilter

This function filters the passed array from data already present in the cozy so that there is not duplicated data in the cozy. You need at least the GET permission for the given doctype in your manifest, to be able to use this function.

Parameters:

  • documents: an array of objects corresponding to the data you want to save in the cozy
  • doctype (string): the doctype where you want to save data (ex: 'io.cozy.bills')
  • options :
    • keys (array) : List of keys used to check that two items are the same. By default it is set to `['id']'.
    • index (optionnal) : Return value returned by cozy.data.defineIndex, the default will correspond to all documents of the selected doctype.
    • selector (optionnal object) : Mango request to get records. Default is built from the keys {selector: {_id: {"$gt": null}}} to get all the records.
const documents = [
  {
    name: 'toto',
    height: 1.8
  },
  {
    name: 'titi',
    height: 1.7
  }
]

return hydrateAndFilter(documents, 'io.cozy.height', {
  keys: ['name']
}).then(filteredDocuments => addData(filteredDocuments, 'io.cozy.height'))

linkBankOperations

linkBankOperations ( entries, doctype, fields, options = {} )

This function will soon move to a dedicated service. You should not use it. The goal of this function is to find links between bills and bank operations.

mkdirp

Creates a directory and its missing ancestors as needed.

Options :

  • ...pathComponents: one or many path components to be joined
await mkdirp('/foo') // Creates /foo
await mkdirp('/foo') // Does nothing as /foo already exists
await mkdirp('/bar/baz') // Creates /bar, then /bar/baz
await mkdirp('/foo/bar/baz') // Creates /foo/bar, then /foo/bar/baz, not /foo
await mkdirp('/') // Does nothing
await mkdirp('/qux', 'qux2/qux3', 'qux4') // Creates /qux, then /qux/qux2,
                                          // then /qux/qux2/qux3 and
                                          // finally /qux/qux2/qux3/qux4

The function will automatically add a leading slash when missing:

await mkdirp('foo', 'bar') // Creates /foo, then /foo/bar
normalizeFilename

Returns the given name, replacing characters that could be an issue when used in a filename with spaces.

Replaced characters include:

  • Those forbidden on one or many popular OS or filesystem: <>:"/\|?*
  • Those forbidden by the cozy-stack \0, \r and \n
  • Multiple spaces and/or tabs are replaced with a single space
  • Leading & trailing spaces and/or tabs are removed

An exception will be thrown in case there is not any filename-compatible character in the given name.

Parameters:

  • basename is whatever string you want to generate the filename from
  • ext is an optional file extension, with or without leading dot
const { normalizeFilename } = require('cozy-konnector-libs')

const filename = normalizeFilename('*foo/bar: <baz> \\"qux"\t???', '.txt')
// `filename` === `foo bar baz qux.txt`
saveBills

Encapsulate the saving of Bills : saves the files, saves the new data, and associate the files to an existing bank operation

saveBills

Combines the features of saveFiles, hydrateAndFilter, addData and linkBankOperations for a common case: bills. Will create io.cozy.bills objects. The default deduplication keys are ['date', 'amount', 'vendor']. You need the full permission on io.cozy.bills, full permission on io.cozy.files and also full permission on io.cozy.bank.operations in your manifest, to be able to use this function.

Parameters:

  • documents is an array of objects with any attributes with some mandatory attributes :
    • amount (Number): the amount of the bill used to match bank operations
    • date (Date): the date of the bill also used to match bank operations
    • vendor (String): the name of the vendor associated to the bill. Ex: 'trainline' You can also pass attributes expected by saveFiles Please take a look at io.cozy.bills doctype documentation
  • fields (object) this is the first parameter given to BaseKonnector's constructor
  • options is passed directly to saveFiles, hydrateAndFilter, addData and linkBankOperations.
saveFiles

The goal of this function is to save the given files in the given folder via the Cozy API. You need the full permission on io.cozy.files in your manifest to use this function.

  • files is an array of { fileurl, filename } :

    • fileurl: The url of the file. This attribute is mandatory or this item will be ignored
    • filename : The file name of the item written on disk. This attribute is optional and as default value, the file name will be "smartly" guessed by the function. Use this attribute if the guess is not smart enough for you.
  • fields (string) is the argument given to the main function of your connector by the BaseKonnector. It especially contains a folderPath which is the string path configured by the user in collect/home

  • options (object) is optional. Possible options :

    • timeout (timestamp) can be used if your connector needs to fetch a lot of files and if the stack does not give enough time to your connector to fetch it all. It could happen that the connector is stopped right in the middle of the download of the file and the file will be broken. With the timeout option, the saveFiles function will check if the timeout has passed right after downloading each file and then will be sure to be stopped cleanly if the timeout is not too long. And since it is really fast to check that a file has already been downloaded, on the next run of the connector, it will be able to download some more files, and so on. If you want the timeout to be in 10s, do Date.now() + 10*1000. You can try it in the previous code.
    • contentType (string) ex: 'application/pdf' used to force the contentType of documents when they are badly recognized by cozy.
signin

The goal of this function is to provide an handy method to log the user in, on html form pages. On success, it resolves to a promise with a parsed body.

Errors:

  • LOGIN_FAILED if the validate predicate is false
  • INVALID_FORM if the element matched by formSelector is not a form or has no action attribute
  • UNKNOWN_PARSING_STRATEGY if parse is not one of the accepted values: raw, cheerio, json.
  • VENDOR_DOWN if a request throws a RequestError, or StatusCodeError

It does not submit values provided through select tags, except if populated by user with formData.

  • url is the url to access the html form

  • formSelector is used by cheerio to uniquely identify the form in which to log in

  • formData is an object { name: value, … }. It is used to populate the form, in the proper inputs with the same name as the properties of this object, before submitting it. It can also be a function that returns this object. The page at url would be given as argument, right after having been parsed through cheerio.

  • parse allow the user to resolve signin with a preparsed body. The choice of the strategy for the parsing is one of : raw, json or cheerio. cheerio being the default.

  • validate is a predicate taking three arguments statusCode, parsedBody and fullResponse. If it is false, LOGIN_FAILED is thrown, otherwise the signin resolves with parsedBody value.

  • requestOpts allows to pass eventual options to the signin's requestFactory. It could be useful for pages using latin1 encoding for instance.

updateOrCreate

The goal of this function is create or update the given entries according to if they already exist in the cozy or not You need the full permission for the given doctype in your manifest, to be able to use this function.

Parameters:

  • entries is an array of objects with any attributes :
  • doctype (string) is the cozy doctype where the entries should be saved
  • matchingAttributes (array of strings) is the list of attributes in each entry should be used to check if an entry is already saved in the cozy
utils

This module proposes some small utils regarding connectors

Classes

BaseKonnector

The class from which all the connectors must inherit. It takes a fetch function in parameter that must return a Promise. You need at least the GET permission on io.cozy.accounts in your manifest to allow it to fetch account information for your connector.

Document

Simple Model for Documents. Allows to specify shouldSave, shouldUpdate as methods.

Has useful isEqual method

Constants

LOGIN_FAILED : String

The konnector could not login

NOT_EXISTING_DIRECTORY : String

The folder specified as folder_to_save does not exist (checked by BaseKonnector)

VENDOR_DOWN : String

The vendor's website is down

USER_ACTION_NEEDED : String

There was an unexpected error, please take a look at the logs to know what happened

FILE_DOWNLOAD_FAILED : String

There was a problem while downloading a file

SAVE_FILE_FAILED : String

There was a problem while saving a file

DISK_QUOTA_EXCEEDED : String

Could not save a file to the cozy because of disk quota exceeded

CHALLENGE_ASKED : String

It seems that the website requires a second authentification factor that we don’t support yet.

LOGIN_FAILED_TOO_MANY_ATTEMPTS : String

Temporarily blocked

USER_ACTION_NEEDED_OAUTH_OUTDATED : String

Access refresh required

USER_ACTION_NEEDED_ACCOUNT_REMOVED : String

Unavailable account

USER_ACTION_NEEDED_CHANGE_PASSWORD : String

Unavailable account

USER_ACTION_NEEDED_PERMISSIONS_CHANGED : String

Password update required

USER_ACTION_NEEDED_CGU_FORM : String

The user needs to accept a CGU form before accessing the rest of the website

Functions

mkSpec()

Declarative scraping.

Describe your items attributes and where to find/parse them instead of imperatively building them.

Heavily inspired by artoo scraping method.

scrape($, spec(s), [childSelector])object | array

Scrape a cheerio object for properties

addData

This function saves the data into the cozy blindly without check You need at least the POST permission for the given doctype in your manifest, to be able to use this function.

Parameters:

const documents = [
  {
    name: 'toto',
    height: 1.8
  },
  {
    name: 'titi',
    height: 1.7
  }
]

return addData(documents, 'io.cozy.height')

cozyClient

This is a cozy-client-js instance already initialized and ready to use

If you want to access cozy-client-js directly, this method gives you directly an instance of it, initialized according to COZY_URL and COZY_CREDENTIALS environment variable given by cozy-stack You can refer to the cozy-client-js documentation for more information.

Example :

const {cozyClient} = require('cozy-konnector-libs')

cozyClient.data.defineIndex('my.doctype', ['_id'])

hydrateAndFilter

This function filters the passed array from data already present in the cozy so that there is not duplicated data in the cozy. You need at least the GET permission for the given doctype in your manifest, to be able to use this function.

Parameters:

const documents = [
  {
    name: 'toto',
    height: 1.8
  },
  {
    name: 'titi',
    height: 1.7
  }
]

return hydrateAndFilter(documents, 'io.cozy.height', {
  keys: ['name']
}).then(filteredDocuments => addData(filteredDocuments, 'io.cozy.height'))

hydrateAndFilter~suitableCall()

Since we can use methods or basic functions for shouldSave and shouldUpdate we pass the appropriate this and arguments.

If funcOrMethod is a method, it will be called with args[0] as this and the rest as arguments Otherwise, this will be null and args will be passed as arguments.

Kind: inner method of hydrateAndFilter

linkBankOperations

linkBankOperations ( entries, doctype, fields, options = {} )

This function will soon move to a dedicated service. You should not use it. The goal of this function is to find links between bills and bank operations.

mkdirp

Creates a directory and its missing ancestors as needed.

Options :

await mkdirp('/foo') // Creates /foo
await mkdirp('/foo') // Does nothing as /foo already exists
await mkdirp('/bar/baz') // Creates /bar, then /bar/baz
await mkdirp('/foo/bar/baz') // Creates /foo/bar, then /foo/bar/baz, not /foo
await mkdirp('/') // Does nothing
await mkdirp('/qux', 'qux2/qux3', 'qux4') // Creates /qux, then /qux/qux2,
                                          // then /qux/qux2/qux3 and
                                          // finally /qux/qux2/qux3/qux4

The function will automatically add a leading slash when missing:

await mkdirp('foo', 'bar') // Creates /foo, then /foo/bar

normalizeFilename

Returns the given name, replacing characters that could be an issue when used in a filename with spaces.

Replaced characters include:

An exception will be thrown in case there is not any filename-compatible character in the given name.

Parameters:

const { normalizeFilename } = require('cozy-konnector-libs')

const filename = normalizeFilename('*foo/bar: <baz> \\"qux"\t???', '.txt')
// `filename` === `foo bar baz qux.txt`

saveBills

Encapsulate the saving of Bills : saves the files, saves the new data, and associate the files to an existing bank operation

saveBills

Combines the features of saveFiles, hydrateAndFilter, addData and linkBankOperations for a common case: bills. Will create io.cozy.bills objects. The default deduplication keys are ['date', 'amount', 'vendor']. You need the full permission on io.cozy.bills, full permission on io.cozy.files and also full permission on io.cozy.bank.operations in your manifest, to be able to use this function.

Parameters:

Example

const { BaseKonnector, saveBills } = require('cozy-konnector-libs')

module.exports = new BaseKonnector(function fetch (fields) {
  const documents = []
  // some code which fills documents
  return saveBills(documents, fields, {
    identifiers: ['vendor']
  })
})

saveFiles

The goal of this function is to save the given files in the given folder via the Cozy API. You need the full permission on io.cozy.files in your manifest to use this function.

Example

await saveFiles([{fileurl: 'https://...', filename: 'bill1.pdf'}], fields)

signin

The goal of this function is to provide an handy method to log the user in, on html form pages. On success, it resolves to a promise with a parsed body.

Errors:

It does not submit values provided through select tags, except if populated by user with formData.

Example
== basic example : ==

const $ = signin({
  url: `http://quotes.toscrape.com/login`,
  formSelector: 'form',
  formData: { username, password }
})

If the behavior of the targeted website is not standard. You can pass a validate function which will allow you to: - detect if the credentials work or not -> LOGIN_FAILED - detect if actions from the user are needed -> USER_ACTION_NEEDED - detect if the targeted website is out -> VENDOR_DOWN Example

const $ = signin({
  url: `http://quotes.toscrape.com/login`,
  formSelector: 'form',
  formData: { username, password },
  validate: (statusCode, $, fullResponse) {
   if (statusCode !== 200) return false // LOGIN_FAILED
   if ($('.cgu').length) throw new Error('USER_ACTION_NEEDED')
   if (fullResponse.request.uri.href.includes('error')) throw new Error('VENDOR_DOWN')
  }
})

Do not forget that the use of the signin function is not mandatory in a connector and won’t work if the signin page does not use html forms. Here, a simple POST request may be a lot more simple.

updateOrCreate

The goal of this function is create or update the given entries according to if they already exist in the cozy or not You need the full permission for the given doctype in your manifest, to be able to use this function.

Parameters:

utils

This module proposes some small utils regarding connectors

utils~fetchAll()

This function allows to fetch all documents for a given doctype. It is the fastest to get all documents but without filtering possibilities deprecated by the findAll method from cozyClient

Parameters:

Kind: inner method of utils

utils~queryAll()

This function allows to fetch all documents for a given doctype exceeding the 100 limit. It is slower that fetchAll because it fetches the data 100 by 100 but allows to filter the data with a selector and an index

Parameters:

const documents = await queryAll('io.cozy.bills', {vendor: 'Direct Energie'})

Kind: inner method of utils

utils~findDuplicates()

This function find duplicates in a given doctype, filtered by an optional mango selector

Parameters:

Returns an object with the following keys: toKeep: this is the list of unique documents that you should keep in db toRemove: this is the list of documents that can remove from db. If this is io.cozy.bills documents, do not forget to clean linked bank operations

const {toKeep, toRemove} = await findDuplicates('io.cozy.bills', {selector: {vendor: 'Direct Energie'}})

Kind: inner method of utils

utils~batchUpdateAttributes()

This is a shortcut to update multiple documents in one call

Parameters:

Returns a promise which resolves with all the return values of updateAttributes

await batchUpdateAttributes('io.cozy.bills', [1, 2, 3], {vendor: 'Direct Energie'})

Kind: inner method of utils

utils~batchDelete()

This is a shortcut to delete multiple documents in one call

Parameters:

Returns a promise which resolves with all the return values of updateAttributes

Example to remove all the documents for a given doctype

const documents = await fetchAll('io.cozy.marvel')
await batchDelete('io.cozy.marvel', documents)

Kind: inner method of utils

utils~getPdfText()

This function can read the content of a cozy pdf file and output its text

Parameters:

Returns a promise which resolves with an object with the following attributes: - text (string) : The full text of the pdf - 1 : The full pdfjs data for page 1 - n : The full pdfjs data for page n

Example:

const pdfText = (await getPdfText('887ABCFE87687')).text

Kind: inner method of utils

BaseKonnector

The class from which all the connectors must inherit. It takes a fetch function in parameter that must return a Promise. You need at least the GET permission on io.cozy.accounts in your manifest to allow it to fetch account information for your connector.

Kind: global class

new BaseKonnector(fetch)

Its role is twofold :

⚠️ A promise should be returned from the fetch function otherwise the konnector cannot know that asynchronous code has been called.

this.terminate('LOGIN_FAILED')
Param Type Description
fetch function Function to be run automatically after account data is fetched. This function will be binded to the current connector. If not fetch function is given. The connector will have to handle itself it’s own exection and error handling

Example

const { BaseKonnector } = require('cozy-konnector-libs')

module.exports = new BaseKonnector(function fetch () {
 // use this to access the instance of the konnector to
 // store any information that needs to be passed to
 // different stages of the konnector
 return request('http://ameli.fr')
   .then(computeReimbursements)
   .then(saveBills)
})

baseKonnector.end()

Hook called when the connector is ended

Kind: instance method of BaseKonnector

baseKonnector.fail()

Hook called when the connector fails

Kind: instance method of BaseKonnector

baseKonnector.init() ⇒ Promise

Initializes the current connector with data coming from the associated account

Kind: instance method of BaseKonnector
Returns: Promise - with the fields as an object

baseKonnector.saveAccountData(data, options) ⇒ Promise

Saves data to the account that is passed to the konnector. Use it to persist data that needs to be passed to each konnector run.

By default, the data is merged to the remote data, use options.merge = false to overwrite the data.

The data is saved under the .data attribute of the cozy account.

Kind: instance method of BaseKonnector

Param Type Description
data object Attributes to be merged
options object { merge: true

baseKonnector.getAccountData() ⇒ object

Get the data saved by saveAccountData

Kind: instance method of BaseKonnector

baseKonnector.terminate(message)

Send a special error code which is interpreted by the cozy stack to terminate the execution of the connector now

Kind: instance method of BaseKonnector

Param Type Description
message string The error code to be saved as connector result see [docs/ERROR_CODES.md]

Document

Simple Model for Documents. Allows to specify shouldSave, shouldUpdate as methods.

Has useful isEqual method

Kind: global class

document.isEqual()

Compares to another document deeply.

_id and _rev are by default ignored in the comparison.

By default, will compare dates loosely since you often compare existing documents (dates in ISO string) with documents that just have been scraped where dates are Dates.

Kind: instance method of Document

LOGIN_FAILED : String

The konnector could not login

Kind: global constant

NOT_EXISTING_DIRECTORY : String

The folder specified as folder_to_save does not exist (checked by BaseKonnector)

Kind: global constant

VENDOR_DOWN : String

The vendor’s website is down

Kind: global constant

USER_ACTION_NEEDED : String

There was an unexpected error, please take a look at the logs to know what happened

Kind: global constant

FILE_DOWNLOAD_FAILED : String

There was a problem while downloading a file

Kind: global constant

SAVE_FILE_FAILED : String

There was a problem while saving a file

Kind: global constant

DISK_QUOTA_EXCEEDED : String

Could not save a file to the cozy because of disk quota exceeded

Kind: global constant

CHALLENGE_ASKED : String

It seems that the website requires a second authentification factor that we don’t support yet.

Kind: global constant

LOGIN_FAILED_TOO_MANY_ATTEMPTS : String

Temporarily blocked

Kind: global constant

USER_ACTION_NEEDED_OAUTH_OUTDATED : String

Access refresh required

Kind: global constant

USER_ACTION_NEEDED_ACCOUNT_REMOVED : String

Unavailable account

Kind: global constant

USER_ACTION_NEEDED_CHANGE_PASSWORD : String

Unavailable account

Kind: global constant

USER_ACTION_NEEDED_PERMISSIONS_CHANGED : String

Password update required

Kind: global constant

USER_ACTION_NEEDED_CGU_FORM : String

The user needs to accept a CGU form before accessing the rest of the website

Kind: global constant

mkSpec()

Declarative scraping.

Describe your items attributes and where to find/parse them instead of imperatively building them.

Heavily inspired by artoo scraping method.

Kind: global function

scrape($, spec(s), [childSelector]) ⇒ object | array

Scrape a cheerio object for properties

Kind: global function
Returns: object | array - - Item(s) scraped

Param Type Description
$ cheerio Cheerio node which will be scraped
spec(s) object | string Options object describing what you want to scrape
[childSelector] string If passed, scrape will return an array of items

Example
scrape can be used to declaratively extract data :

const item = scrape($('#item'), {
  title: '.title',
  content: '.content'
})
const items = scrape($('#content'), {
  title: '.title',
  content: '.content'
}, '.item')

For more power, you can use objects for each retriever :

const items = scrape($('#content'), {
  title: '.title',
  content: '.content',
  link: {
    sel: 'a',
    attr: 'href'
  },
}, '.item')

Here the href attribute of the a inside .items would have been put into the link attribute of the items returned by scrape.

Available options :

⚠ Permissions

Please note that some classes require some permissions: