API

Modules

addData

Saves the data into the cozy blindly without check.

cozyClient

cozy-client-js instance already initialized and ready to use.

hydrateAndFilter

Filters the passed array from data already present in the cozy so that there is not duplicated data in the Cozy.

linkBankOperations

Finds links between bills and bank operations.

mkdirp
normalizeFilename

Returns the given name, replacing characters that could be an issue when used in a filename with spaces.

saveBills

Encapsulates the saving of Bills : saves the files, saves the new data, and associate the files to an existing bank operation

saveFiles

Saves the given files in the given folder via the Cozy API.

signin

Provides an handy method to log the user in, on HTML form pages. On success, it resolves to a promise with a parsed body.

solveCaptcha

Use every possible means to solve a captcha. At the moment, Anticaptcha web service is used if any related secret key is found in COZY_PARAMETERS environment variable.

updateOrCreate

Creates or updates the given entries according to if they already exist in the cozy or not

utils

Small utilities helping to develop konnectors.

categorization

Bank transactions categorization

Classes

BaseKonnector

The class from which all the connectors must inherit. It takes a fetch function in parameter that must return a Promise. You need at least the GET permission on io.cozy.accounts in your manifest to allow it to fetch account information for your connector.

CookieKonnector

Connector base class extending BaseKonnector which handles cookie session in a central way It also handles saving cookie session in the account and automatically restore it for the next connector run. All cozy-konnector-libs tools using request are proposed as methods of this class to force them to use the central cookie which can be saved/restored. You need at least the GET and PUT permissions on io.cozy.accounts in your manifest to allow it to save/restore cookies

Document

Simple Model for Documents. Allows to specify shouldSave, shouldUpdate as methods.

Has useful isEqual method

Constants

fs

Manifest is provided differently in developement that in production.

  • In production, the manifest has been "merged" via Webpack via the DefinePlugin

  • In development/test, we simply read the manifest from the fs

LOGIN_FAILED : String

The konnector could not login

NOT_EXISTING_DIRECTORY : String

The folder specified as folder_to_save does not exist (checked by BaseKonnector)

VENDOR_DOWN : String

The vendor's website is down

USER_ACTION_NEEDED : String

There was an unexpected error, please take a look at the logs to know what happened

FILE_DOWNLOAD_FAILED : String

There was a problem while downloading a file

SAVE_FILE_FAILED : String

There was a problem while saving a file

DISK_QUOTA_EXCEEDED : String

Could not save a file to the cozy because of disk quota exceeded

CHALLENGE_ASKED : String

It seems that the website requires a second authentification factor that we don’t support yet.

LOGIN_FAILED_TOO_MANY_ATTEMPTS : String

Temporarily blocked

USER_ACTION_NEEDED_OAUTH_OUTDATED : String

Access refresh required

USER_ACTION_NEEDED_ACCOUNT_REMOVED : String

Unavailable account

USER_ACTION_NEEDED_CHANGE_PASSWORD : String

Unavailable account

USER_ACTION_NEEDED_PERMISSIONS_CHANGED : String

Password update required

USER_ACTION_NEEDED_CGU_FORM : String

The user needs to accept a CGU form before accessing the rest of the website

CAPTCHA_RESOLUTION_FAILED : String

solveCaptcha failed to solve the captcha

Functions

mkSpec()

Declarative scraping.

Describe your items attributes and where to find/parse them instead of imperatively building them.

Heavily inspired by artoo scraping method.

scrape($, spec(s), [childSelector])object | array

Scrape a cheerio object for properties

addData

Saves the data into the cozy blindly without check.

module.exports() ⏏

Saves the data into the cozy blindly without check.

You need at least the POST permission for the given doctype in your manifest, to be able to use this function.

Parameters:

const documents = [
  {
    name: 'toto',
    height: 1.8
  },
  {
    name: 'titi',
    height: 1.7
  }
]

return addData(documents, 'io.cozy.height')

Kind: Exported function

cozyClient

cozy-client-js instance already initialized and ready to use.

module.exports ⏏

cozy-client-js instance already initialized and ready to use.

If you want to access cozy-client-js directly, this method gives you directly an instance of it, initialized according to COZY_URL and COZY_CREDENTIALS environment variable given by cozy-stack You can refer to the cozy-client-js documentation for more information.

Example :

const {cozyClient} = require('cozy-konnector-libs')

cozyClient.data.defineIndex('my.doctype', ['_id'])

Kind: Exported member

hydrateAndFilter

Filters the passed array from data already present in the cozy so that there is not duplicated data in the Cozy.

hydrateAndFilter() ⏏

Filters the passed array from data already present in the cozy so that there is not duplicated data in the Cozy.

You need at least the GET permission for the given doctype in your manifest, to be able to use this function.

Parameters:

const documents = [
  {
    name: 'toto',
    height: 1.8
  },
  {
    name: 'titi',
    height: 1.7
  }
]

return hydrateAndFilter(documents, 'io.cozy.height', {
  keys: ['name']
}).then(filteredDocuments => addData(filteredDocuments, 'io.cozy.height'))

Kind: Exported function

hydrateAndFilter~suitableCall()

Since we can use methods or basic functions for shouldSave and shouldUpdate we pass the appropriate this and arguments.

If funcOrMethod is a method, it will be called with args[0] as this and the rest as arguments Otherwise, this will be null and args will be passed as arguments.

Kind: inner method of hydrateAndFilter

linkBankOperations

Finds links between bills and bank operations.

module.exports() ⏏

Will soon move to a dedicated service. You should not use it.

Finds links between bills and bank operations.

Kind: Exported function

mkdirp

module.exports ⏏

Creates a directory and its missing ancestors as needed.

Options :

await mkdirp('/foo') // Creates /foo
await mkdirp('/foo') // Does nothing as /foo already exists
await mkdirp('/bar/baz') // Creates /bar, then /bar/baz
await mkdirp('/foo/bar/baz') // Creates /foo/bar, then /foo/bar/baz, not /foo
await mkdirp('/') // Does nothing
await mkdirp('/qux', 'qux2/qux3', 'qux4') // Creates /qux, then /qux/qux2,
                                          // then /qux/qux2/qux3 and
                                          // finally /qux/qux2/qux3/qux4

The function will automatically add a leading slash when missing:

await mkdirp('foo', 'bar') // Creates /foo, then /foo/bar

Kind: Exported member

normalizeFilename

Returns the given name, replacing characters that could be an issue when used in a filename with spaces.

normalizeFilename() ⏏

Returns the given name, replacing characters that could be an issue when used in a filename with spaces.

Replaced characters include:

An exception will be thrown in case there is not any filename-compatible character in the given name.

Parameters:

const { normalizeFilename } = require('cozy-konnector-libs')

const filename = normalizeFilename('*foo/bar: <baz> \\"qux"\t???', '.txt')
// `filename` === `foo bar baz qux.txt`

Kind: Exported function

saveBills

Encapsulates the saving of Bills : saves the files, saves the new data, and associate the files to an existing bank operation

module.exports() ⏏

Combines the features of saveFiles, hydrateAndFilter, addData and linkBankOperations for a common case: bills. Will create io.cozy.bills objects. The default deduplication keys are ['date', 'amount', 'vendor']. You need the full permission on io.cozy.bills, full permission on io.cozy.files and also full permission on io.cozy.bank.operations in your manifest, to be able to use this function.

Parameters:

Kind: Exported function
Example

const { BaseKonnector, saveBills } = require('cozy-konnector-libs')

module.exports = new BaseKonnector(function fetch (fields) {
  const documents = []
  // some code which fills documents
  return saveBills(documents, fields, {
    identifiers: ['vendor']
  })
})

saveFiles

Saves the given files in the given folder via the Cozy API.

saveFiles() ⏏

Saves the files given in the fileurl attribute of each entries

You need the full permission on io.cozy.files in your manifest to use this function.

Kind: Exported function
Example

await saveFiles([{fileurl: 'https://...', filename: 'bill1.pdf'}], fields)

signin

Provides an handy method to log the user in, on HTML form pages. On success, it resolves to a promise with a parsed body.

module.exports() ⏏

Provides an handy method to log the user in, on HTML form pages. On success, it resolves to a promise with a parsed body.

Errors:

It does not submit values provided through select tags, except if populated by user with formData.

Kind: Exported function
Example
== basic example : ==

const $ = signin({
  url: `http://quotes.toscrape.com/login`,
  formSelector: 'form',
  formData: { username, password }
})

If the behavior of the targeted website is not standard. You can pass a validate function which will allow you to: - detect if the credentials work or not -> LOGIN_FAILED - detect if actions from the user are needed -> USER_ACTION_NEEDED - detect if the targeted website is out -> VENDOR_DOWN Example

const $ = signin({
  url: `http://quotes.toscrape.com/login`,
  formSelector: 'form',
  formData: { username, password },
  validate: (statusCode, $, fullResponse) {
   if (statusCode !== 200) return false // LOGIN_FAILED
   if ($('.cgu').length) throw new Error('USER_ACTION_NEEDED')
   if (fullResponse.request.uri.href.includes('error')) throw new Error('VENDOR_DOWN')
  }
})

Do not forget that the use of the signin function is not mandatory in a connector and won’t work if the signin page does not use html forms. Here, a simple POST request may be a lot more simple.

solveCaptcha

Use every possible means to solve a captcha. At the moment, Anticaptcha web service is used if any related secret key is found in COZY_PARAMETERS environment variable.

module.exports() ⏏

Use every possible means to solve a captcha. At the moment, Anticaptcha web service is used if any related secret key is found in COZY_PARAMETERS environment variable. If you do not want to solve the captcha each time the connector is run, please also use CookieKonnector which will help you save the session.

Parameters:

Kind: Exported function
Example

const { solveCaptcha } = require('cozy-konnector-libs')

const solvedKey = await solveCaptcha({
  websiteKey: 'the key in the webpage',
  websiteURL: 'http://quotes.toscrape.com/login',
})
// now use the solveKey to submit your form

updateOrCreate

Creates or updates the given entries according to if they already exist in the cozy or not

module.exports(entries, doctype, matchingAttributes) ⇒ Promise

Creates or updates the given entries according to if they already exist in the cozy or not

You need the full permission for the given doctype in your manifest, to be able to use this function.

Kind: Exported function

Param Type Description
entries Array.<Object> Documents to save
doctype String Doctype of the documents
matchingAttributes Array.<String> attributes in each entry used to check if an entry already exists in the Cozy

utils

Small utilities helping to develop konnectors.

utils~fetchAll()

This function allows to fetch all documents for a given doctype. It is the fastest to get all documents but without filtering possibilities deprecated by the findAll method from cozyClient

Parameters:

Kind: inner method of utils

utils~queryAll()

This function allows to fetch all documents for a given doctype exceeding the 100 limit. It is slower that fetchAll because it fetches the data 100 by 100 but allows to filter the data with a selector and an index

Parameters:

const documents = await queryAll('io.cozy.bills', {vendor: 'Direct Energie'})

Kind: inner method of utils

utils~findDuplicates()

This function find duplicates in a given doctype, filtered by an optional mango selector

Parameters:

Returns an object with the following keys: toKeep: this is the list of unique documents that you should keep in db toRemove: this is the list of documents that can remove from db. If this is io.cozy.bills documents, do not forget to clean linked bank operations

const {toKeep, toRemove} = await findDuplicates('io.cozy.bills', {selector: {vendor: 'Direct Energie'}})

Kind: inner method of utils

utils~batchUpdateAttributes()

This is a shortcut to update multiple documents in one call

Parameters:

Returns a promise which resolves with all the return values of updateAttributes

await batchUpdateAttributes('io.cozy.bills', [1, 2, 3], {vendor: 'Direct Energie'})

Kind: inner method of utils

utils~batchDelete()

This is a shortcut to delete multiple documents in one call

Parameters:

Returns a promise which resolves with all the return values of updateAttributes

Example to remove all the documents for a given doctype

const documents = await fetchAll('io.cozy.marvel')
await batchDelete('io.cozy.marvel', documents)

Kind: inner method of utils

utils~getPdfText()

This function can read the content of a cozy pdf file and output its text

Parameters:

Returns a promise which resolves with an object with the following attributes: - text (string) : The full text of the pdf - 1 : The full pdfjs data for page 1 - n : The full pdfjs data for page n

Example:

const pdfText = (await getPdfText('887ABCFE87687')).text

Kind: inner method of utils

utils~formatDate()

This function convert a Date Object to a ISO date string (2018-07-31)

Parameters:

Returns a string

Example:

const date = formatFrenchDate(New Date.now())

Kind: inner method of utils

categorization

Bank transactions categorization

categorization~categorize(transactions) ⇒ Array.<Object>

Apply both global and local categorization models to an array of transactions.

The global model is a model specific to hosted Cozy instances. It is not available for self-hosted instances. It will just do nothing in that case.

The local model is based on the user manual categorizations.

Each model adds two properties to the transactions: * The global model adds cozyCategoryId and cozyCategoryProba * The local model adds localCategoryId and localCategoryProba

In the end, each transaction can have up to four different categories. An application can use these categories to show the most significant for the user. See https://github.com/cozy/cozy-doctypes/blob/master/docs/io.cozy.bank.md#categories for more informations.

Kind: inner method of categorization
Returns: Array.<Object> - The categorized transactions

Param Type Description
transactions Array.<Object> The transactions to categorize

Example

const { BaseKonnector, categorize } = require('cozy-konnector-libs')

class BankingKonnector extends BaseKonnector {
  saveTransactions() {
    const transactions = await this.fetchTransactions()
    const categorizedTransactions = await categorize(transactions)

    // Save categorizedTransactions
  }
}

BaseKonnector

The class from which all the connectors must inherit. It takes a fetch function in parameter that must return a Promise. You need at least the GET permission on io.cozy.accounts in your manifest to allow it to fetch account information for your connector.

Kind: global class

new BaseKonnector(fetch)

Its role is twofold :

⚠️ A promise should be returned from the fetch function otherwise the konnector cannot know that asynchronous code has been called.

this.terminate('LOGIN_FAILED')
Param Type Description
fetch function Function to be run automatically after account data is fetched. This function will be binded to the current connector. If not fetch function is given. The connector will have to handle itself it’s own exection and error handling

Example

const { BaseKonnector } = require('cozy-konnector-libs')

module.exports = new BaseKonnector(function fetch () {
 // use this to access the instance of the konnector to
 // store any information that needs to be passed to
 // different stages of the konnector
 return request('http://ameli.fr')
   .then(computeReimbursements)
   .then(saveBills)
})

baseKonnector.end()

Hook called when the connector is ended

Kind: instance method of BaseKonnector

baseKonnector.fail()

Hook called when the connector fails

Kind: instance method of BaseKonnector

baseKonnector.init() ⇒ Promise

Initializes the current connector with data coming from the associated account

Kind: instance method of BaseKonnector
Returns: Promise - with the fields as an object

baseKonnector.saveAccountData(data, options) ⇒ Promise

Saves data to the account that is passed to the konnector. Use it to persist data that needs to be passed to each konnector run.

By default, the data is merged to the remote data, use options.merge = false to overwrite the data.

The data is saved under the .data attribute of the cozy account.

Kind: instance method of BaseKonnector

Param Type Description
data object Attributes to be merged
options object { merge: true

baseKonnector.getAccountData() ⇒ object

Get the data saved by saveAccountData

Kind: instance method of BaseKonnector

baseKonnector.updateAccountAttributes()

Update account attributes and cache the account

Kind: instance method of BaseKonnector

baseKonnector.terminate(message)

Send a special error code which is interpreted by the cozy stack to terminate the execution of the connector now

Kind: instance method of BaseKonnector

Param Type Description
message string The error code to be saved as connector result see [docs/ERROR_CODES.md]

CookieKonnector

Connector base class extending BaseKonnector which handles cookie session in a central way It also handles saving cookie session in the account and automatically restore it for the next connector run. All cozy-konnector-libs tools using request are proposed as methods of this class to force them to use the central cookie which can be saved/restored. You need at least the GET and PUT permissions on io.cozy.accounts in your manifest to allow it to save/restore cookies

Kind: global class

new CookieKonnector(requestFactoryOptions)

Constructor

Param Type Description
requestFactoryOptions function Option object passed to requestFactory to initialize this.request. It is still possible to change this.request doing : javascript this.request = this.requestFactory(...) Please not you have to run the connector yourself doing : javascript connector.run()

Example

const { CookieKonnector } = require('cozy-konnector-libs')
class MyConnector extends CookieKonnector {
  async fetch(fields) {
     // the code of your connector
     await this.request('https://...')
  }
  async testSession() {
     const $ = await this.request('https://...')
     return $('')
  }
}
const connector = new MyKonnector({
  cheerio: true,
  json: false
})
connector.run()

cookieKonnector.init() ⇒ Promise

Initializes the current connector with data coming from the associated account and also the session

Kind: instance method of CookieKonnector
Returns: Promise - with the fields as an object

cookieKonnector.end()

Hook called when the connector is ended

Kind: instance method of CookieKonnector

cookieKonnector.requestFactory(options) ⇒ object

Calls cozy-konnector-libs requestFactory forcing this._jar as the cookie

Kind: instance method of CookieKonnector
Returns: object - - The resulting request object

Param Type Description
options object requestFactory option

cookieKonnector.resetSession() ⇒ Promise

Reset cookie session with a new empty session and save it to the associated account

Kind: instance method of CookieKonnector

cookieKonnector.initSession() ⇒ Promise

Get the cookie session from the account if any

Kind: instance method of CookieKonnector
Returns: Promise - true or false if the session in the account exists or not

cookieKonnector.saveSession() ⇒ Promise

Saves the current cookie session to the account

Kind: instance method of CookieKonnector

cookieKonnector.signin() ⇒ Promise

This is signin function from cozy-konnector-libs which is forced to use the current cookies and current request from CookieKonnector. It also automatically saves the session after signin if it is a success.

Kind: instance method of CookieKonnector

cookieKonnector.saveFiles() ⇒ Promise

This is saveFiles function from cozy-konnector-libs which is forced to use the current cookies and current request from CookieKonnector.

Kind: instance method of CookieKonnector

cookieKonnector.saveBills() ⇒ Promise

This is saveBills function from cozy-konnector-libs which is forced to use the current cookies and current request from CookieKonnector.

Kind: instance method of CookieKonnector

Document

Simple Model for Documents. Allows to specify shouldSave, shouldUpdate as methods.

Has useful isEqual method

Kind: global class

document.isEqual()

Compares to another document deeply.

_id and _rev are by default ignored in the comparison.

By default, will compare dates loosely since you often compare existing documents (dates in ISO string) with documents that just have been scraped where dates are Dates.

Kind: instance method of Document

fs

Manifest is provided differently in developement that in production.

Kind: global constant

LOGIN_FAILED : String

The konnector could not login

Kind: global constant

NOT_EXISTING_DIRECTORY : String

The folder specified as folder_to_save does not exist (checked by BaseKonnector)

Kind: global constant

VENDOR_DOWN : String

The vendor’s website is down

Kind: global constant

USER_ACTION_NEEDED : String

There was an unexpected error, please take a look at the logs to know what happened

Kind: global constant

FILE_DOWNLOAD_FAILED : String

There was a problem while downloading a file

Kind: global constant

SAVE_FILE_FAILED : String

There was a problem while saving a file

Kind: global constant

DISK_QUOTA_EXCEEDED : String

Could not save a file to the cozy because of disk quota exceeded

Kind: global constant

CHALLENGE_ASKED : String

It seems that the website requires a second authentification factor that we don’t support yet.

Kind: global constant

LOGIN_FAILED_TOO_MANY_ATTEMPTS : String

Temporarily blocked

Kind: global constant

USER_ACTION_NEEDED_OAUTH_OUTDATED : String

Access refresh required

Kind: global constant

USER_ACTION_NEEDED_ACCOUNT_REMOVED : String

Unavailable account

Kind: global constant

USER_ACTION_NEEDED_CHANGE_PASSWORD : String

Unavailable account

Kind: global constant

USER_ACTION_NEEDED_PERMISSIONS_CHANGED : String

Password update required

Kind: global constant

USER_ACTION_NEEDED_CGU_FORM : String

The user needs to accept a CGU form before accessing the rest of the website

Kind: global constant

CAPTCHA_RESOLUTION_FAILED : String

solveCaptcha failed to solve the captcha

Kind: global constant

mkSpec()

Declarative scraping.

Describe your items attributes and where to find/parse them instead of imperatively building them.

Heavily inspired by artoo scraping method.

Kind: global function

scrape($, spec(s), [childSelector]) ⇒ object | array

Scrape a cheerio object for properties

Kind: global function
Returns: object | array - - Item(s) scraped

Param Type Description
$ cheerio Cheerio node which will be scraped
spec(s) object | string Options object describing what you want to scrape
[childSelector] string If passed, scrape will return an array of items

Example
scrape can be used to declaratively extract data :

const item = scrape($('#item'), {
  title: '.title',
  content: '.content'
})
const items = scrape($('#content'), {
  title: '.title',
  content: '.content'
}, '.item')

For more power, you can use objects for each retriever :

const items = scrape($('#content'), {
  title: '.title',
  content: '.content',
  link: {
    sel: 'a',
    attr: 'href'
  },
}, '.item')

Here the href attribute of the a inside .items would have been put into the link attribute of the items returned by scrape.

Available options :

⚠ Permissions

Please note that some classes require some permissions: