API
API¶
Modules¶
- CozyBrowser
Get a javascript simulation of a real browser (jsdom)
- addData
Saves the data into the cozy blindly without check.
- cozyClient
cozy-client-js instance already initialized and ready to use.
- hydrateAndFilter
Filters the passed array from data already present in the cozy so that there is not duplicated data in the Cozy.
- linkBankOperations
Finds links between bills and bank operations.
- mkdirp
- normalizeFilename
Returns the given name, replacing characters that could be an issue when used in a filename with spaces.
- saveBills
Encapsulates the saving of Bills : saves the files, saves the new data, and associate the files to an existing bank operation
- saveFiles
Saves the given files in the given folder via the Cozy API.
- saveIdentity
Helper to set or merge io.cozy.identities See https://github.com/cozy/cozy-doctypes/blob/master/docs/io.cozy.identities.md
- signin
Provides an handy method to log the user in, on HTML form pages. On success, it resolves to a promise with a parsed body.
- solveCaptcha
Use every possible means to solve a captcha. At the moment, Anticaptcha web service is used if any related secret key is found in COZY_PARAMETERS environment variable.
- updateOrCreate
Creates or updates the given entries according to if they already exist in the cozy or not
- utils
Small utilities helping to develop konnectors.
- categorization
Bank transactions categorization
Classes¶
- BaseKonnector
The class from which all the connectors must inherit. It takes a fetch function in parameter that must return a
Promise
. You need at least theGET
permission onio.cozy.accounts
in your manifest to allow it to fetch account information for your connector.Its role is twofold :
- Make the link between account data and konnector
- Handle errors
⚠️ A promise should be returned from the
fetch
function otherwise the konnector cannot know that asynchronous code has been called.- CookieKonnector
Connector base class extending BaseKonnector which handles cookie session in a central way It also handles saving cookie session in the account and automatically restore it for the next connector run. All cozy-konnector-libs tools using request are proposed as methods of this class to force them to use the central cookie which can be saved/restored. You need at least the
GET
andPUT
permissions onio.cozy.accounts
in your manifest to allow it to save/restore cookies- Document
Simple Model for Documents. Allows to specify
shouldSave
,shouldUpdate
as methods.Has useful
isEqual
method
Constants¶
- fs
Manifest is provided differently in developement that in production.
In production, the manifest has been "merged" via Webpack via the DefinePlugin
In development/test, we simply read the manifest from the fs
- LOGIN_FAILED :
string
The konnector could not login
- NOT_EXISTING_DIRECTORY :
string
The folder specified as folder_to_save does not exist (checked by BaseKonnector)
- VENDOR_DOWN :
string
The vendor's website is down
- USER_ACTION_NEEDED :
string
There was an unexpected error, please take a look at the logs to know what happened
- FILE_DOWNLOAD_FAILED :
string
There was a problem while downloading a file
- SAVE_FILE_FAILED :
string
There was a problem while saving a file
- DISK_QUOTA_EXCEEDED :
string
Could not save a file to the cozy because of disk quota exceeded
- CHALLENGE_ASKED :
string
It seems that the website requires a second authentification factor that we don’t support yet.
- LOGIN_FAILED_TOO_MANY_ATTEMPTS :
string
Temporarily blocked
- USER_ACTION_NEEDED_OAUTH_OUTDATED :
string
Access refresh required
- USER_ACTION_NEEDED_ACCOUNT_REMOVED :
string
Unavailable account
- USER_ACTION_NEEDED_CHANGE_PASSWORD :
string
Unavailable account
- USER_ACTION_NEEDED_PERMISSIONS_CHANGED :
string
Password update required
- USER_ACTION_NEEDED_CGU_FORM :
string
The user needs to accept a CGU form before accessing the rest of the website
- CAPTCHA_RESOLUTION_FAILED :
string
solveCaptcha failed to solve the captcha
- LOGIN_FAILED_NEEDS_SECRET :
string
Additionnal information is needed to check your login details
- MAINTENANCE :
string
remote website seems to be unavailable
- TERMS_VERSION_MISMATCH :
string
User needs to accept new terms
- UNKNOWN_ERROR :
string
unkown error
- UNKNOWN_ERROR_PARTIAL_SYNC :
string
The synchronization is complete but some elements may be missing
- USER_ACTION_NEEDED_SCA_REQUIRED :
string
Renewal of authentication required
- USER_ACTION_NEEDED_TWOFA_EXPIRED :
string
Authentication renewal required
- USER_ACTION_NEEDED_WEBAUTH_REQUIRED :
string
Authentication on vendor website required
- USER_ACTION_NEEDED_WRONG_TWOFA_CODE :
string
Incorrect strong authentication code
- VENDOR_DOWN_BANK_DOWN :
string
Unavailable bank website
- VENDOR_DOWN_LINXO_DOWN :
string
Unavailable bank website
Functions¶
- attachProcessEventHandlers(prcs) ⇒
function
Attach event handlers to catch uncaught exceptions/rejections and signals. Log them as critical and exit the process accordingly. If the cleanup function has not been called, calling again the function is a no-op.
- mkSpec()
Declarative scraping.
Describe your items attributes and where to find/parse them instead of imperatively building them.
Heavily inspired by artoo scraping method.
- scrape($, specs, [childSelector]) ⇒
object
|Array
Scrape a cheerio object for properties
CozyBrowser¶
Get a javascript simulation of a real browser (jsdom)
- CozyBrowser
- defaultOptions ⇒
object
⏏
- defaultOptions ⇒
defaultOptions ⇒ object
⏏¶
Get a preconfigured jsdom browser simulator using the zombie npm package See http://zombie.js.org/ for complete documentation The connector has to import the zombie npm package itself.
Kind: Exported constant
Returns: object
- Zombie browser extended class
Param | Type | Description |
---|---|---|
options.userAgent | string |
The user agent string used by the browser |
Example
const Browser = require('cozy-konnector-libs/libs/CozyBrowser') const browser = new Browser() await browser.visit('http://quotes.toscrape.com/')
defaultOptions~addListeners()¶
Add cozy-konnector-libs specific logs to browser events
Kind: inner method of defaultOptions
addData¶
Saves the data into the cozy blindly without check.
addData() ⏏¶
Saves the data into the cozy blindly without check.
You need at least the POST
permission for the given doctype in your manifest, to be able to
use this function.
Parameters:
documents
: an array of objects corresponding to the data you want to save in the cozy
doctype
(string): the doctype where you want to save data (ex: ‘io.cozy.bills’)
options
(object): option object
+ sourceAccount
(String): id of the source account
+ sourceAccountIdentifier
(String): identifier unique to the account targetted by the connector. It is the login most of the time
const documents = [ { name: 'toto', height: 1.8 }, { name: 'titi', height: 1.7 } ] return addData(documents, 'io.cozy.height')
cozyClient¶
cozy-client-js instance already initialized and ready to use.
cozyClient ⏏¶
cozy-client-js instance already initialized and ready to use.
If you want to access cozy-client-js directly, this method gives you directly an instance of it,
initialized according to COZY_URL
and COZY_CREDENTIALS
environment variable given by cozy-stack
You can refer to the cozy-client-js documentation for more information.
Example :
const {cozyClient} = require('cozy-konnector-libs') cozyClient.data.defineIndex('my.doctype', ['_id'])
hydrateAndFilter¶
Filters the passed array from data already present in the cozy so that there is not duplicated data in the Cozy.
hydrateAndFilter() ⏏¶
Filters the passed array from data already present in the cozy so that there is not duplicated data in the Cozy.
You need at least the GET
permission for the given doctype in your manifest, to be able to
use this function.
Parameters:
documents
: an array of objects corresponding to the data you want to save in the cozy
doctype
(string): the doctype where you want to save data (ex: ‘io.cozy.bills’)
options
:
- keys
(array) : List of keys used to check that two items are the same. By default it is set to ['id']'.
-
index(optionnal) : Return value returned by
cozy.data.defineIndex, the default will correspond to all documents of the selected doctype.
-
selector(optionnal object) : Mango request to get records. Default is built from the keys
}}` to get all the records.
const documents = [ { name: 'toto', height: 1.8 }, { name: 'titi', height: 1.7 } ] return hydrateAndFilter(documents, 'io.cozy.height', { keys: ['name'] }).then(filteredDocuments => addData(filteredDocuments, 'io.cozy.height'))
hydrateAndFilter~suitableCall()¶
Since we can use methods or basic functions for
shouldSave
and shouldUpdate
we pass the
appropriate this
and arguments
.
If funcOrMethod
is a method, it will be called
with args[0] as this
and the rest as arguments
Otherwise, this
will be null and args
will be passed
as arguments
.
Kind: inner method of hydrateAndFilter
linkBankOperations¶
Finds links between bills and bank operations.
linkBankOperations() ⏏¶
Will soon move to a dedicated service. You should not use it.
Finds links between bills and bank operations.
mkdirp¶
mkdirp ⏏¶
Creates a directory and its missing ancestors as needed.
Options :
...pathComponents
: one or many path components to be joined
await mkdirp('/foo') // Creates /foo await mkdirp('/foo') // Does nothing as /foo already exists await mkdirp('/bar/baz') // Creates /bar, then /bar/baz await mkdirp('/foo/bar/baz') // Creates /foo/bar, then /foo/bar/baz, not /foo await mkdirp('/') // Does nothing await mkdirp('/qux', 'qux2/qux3', 'qux4') // Creates /qux, then /qux/qux2, // then /qux/qux2/qux3 and // finally /qux/qux2/qux3/qux4
The function will automatically add a leading slash when missing:
await mkdirp('foo', 'bar') // Creates /foo, then /foo/bar
normalizeFilename¶
Returns the given name, replacing characters that could be an issue when used in a filename with spaces.
normalizeFilename() ⏏¶
Returns the given name, replacing characters that could be an issue when used in a filename with spaces.
Replaced characters include:
- Those forbidden on one or many popular OS or filesystem:
<>:"/\|?*
- Those forbidden by the cozy-stack
\0
,\r
and\n
- Multiple spaces and/or tabs are replaced with a single space
- Leading & trailing spaces and/or tabs are removed
An exception will be thrown in case there is not any filename-compatible character in the given name.
Parameters:
basename
is whatever string you want to generate the filename fromext
is an optional file extension, with or without leading dot
const { normalizeFilename } = require('cozy-konnector-libs') const filename = normalizeFilename('*foo/bar: <baz> \\"qux"\t???', '.txt') // `filename` === `foo bar baz qux.txt`
saveBills¶
Encapsulates the saving of Bills : saves the files, saves the new data, and associate the files to an existing bank operation
saveBills() ⏏¶
Combines the features of saveFiles
, hydrateAndFilter
, addData
and linkBankOperations
for a
common case: bills.
Will create io.cozy.bills
objects. The default deduplication keys are ['date', 'amount', 'vendor']
.
You need the full permission on io.cozy.bills
, full permission on io.cozy.files
and also
full permission on io.cozy.bank.operations
in your manifest, to be able to use this function.
Parameters:
documents
is an array of objects with any attributes with some mandatory attributes :amount
(Number): the amount of the bill used to match bank operationsdate
(Date): the date of the bill also used to match bank operationsvendor
(String): the name of the vendor associated to the bill. Ex: ‘trainline’currency
(String) default: EUR: The ISO currency value (not mandatory since there is a default value.contractId
(String): Contract unique identicator used to deduplicate billscontractLabel
: (String) User label if define, must be used with contractIdmatchingCriterias
(Object): criterias that can be used by an external service to match bills with bank operations. If not specified but the ‘banksTransactionRegExp’ attribute is specified in the manifest of the connector, this value is automatically added to the bill
You can also pass attributes expected by saveFiles
: fileurl, filename, requestOptions
and more
Please take a look at io.cozy.bills doctype documentation
- fields
(object) this is the first parameter given to BaseKonnector’s constructor
- options
is passed directly to saveFiles
, hydrateAndFilter
, addData
and linkBankOperations
.
Kind: Exported function
Example
const { BaseKonnector, saveBills } = require('cozy-konnector-libs') module.exports = new BaseKonnector(function fetch (fields) { const documents = [] // some code which fills documents return saveBills(documents, fields, { identifiers: ['vendor'] }) })
saveFiles¶
Saves the given files in the given folder via the Cozy API.
saveFiles(entries, fields, options, [contract]) ⏏¶
Saves the files given in the fileurl attribute of each entries
You need the full permission on io.cozy.files
in your manifest to use this function.
Kind: Exported function
Param | Type | Description |
---|---|---|
entries | Array |
list of object describing files to save |
entries.fileurl | string |
The url of the file (can be a function returning the value). Ignored if filestream is given |
entries.fetchFile | function |
the connector can give it’s own function to fetch the file from the website, which will be run only when necessary (if the corresponding file is missing on the cozy) function returning the stream). This function must return a promise resolved as a stream |
entries.filestream | object | string |
the stream which will be directly passed to cozyClient.files.create (can also be function returning the stream) |
entries.requestOptions | object |
The options passed to request to fetch fileurl (can be a function returning the value) |
entries.filename | string |
The file name of the item written on disk. This attribute is optional and as default value, the file name will be “smartly” guessed by the function. Use this attribute if the guess is not smart enough for you, or if you use filestream (can be a function returning the value). |
entries.shouldReplaceName | string |
used to migrate filename. If saveFiles finds a file linked to this entry and this file name matches shouldReplaceName , the file is renamed to filename (can be a function returning the value) |
entries.shouldReplaceFile | function |
use this function to state if the current entry should be forced to be redownloaded and replaced. Usefull if we know the file content can change and we always want the last version. |
entries.fileAttributes | object |
ex: {created_at: new Date()} sets some additionnal file attributes passed to cozyClient.file.create |
entries.subPath | string |
A subpath to save all files, will be created if needed. |
entries.contract | object |
contract object associated to the files |
entries.contract.id | string |
id of the contract |
entries.contract.name | string |
name of the contract |
fields | object |
is the argument given to the main function of your connector by the BaseKonnector. It especially contains a folderPath which is the string path configured by the user in collect/home |
options | object |
global options |
options.timeout | number |
timestamp which can be used if your connector needs to fetch a lot of files and if the stack does not give enough time to your connector to fetch it all. It could happen that the connector is stopped right in the middle of the download of the file and the file will be broken. With the timeout option, the saveFiles function will check if the timeout has passed right after downloading each file and then will be sure to be stopped cleanly if the timeout is not too long. And since it is really fast to check that a file has already been downloaded, on the next run of the connector, it will be able to download some more files, and so on. If you want the timeout to be in 10s, do Date.now() + 10*1000 . You can try it in the previous code. |
options.contentType | number | boolean |
ex: ‘application/pdf’ used to force the contentType of documents when they are badly recognized by cozy. If “true” the content type will be recognized from the file name and forced the same way. |
options.concurrency | number |
default: 1 sets the maximum number of concurrent downloads |
options.validateFile | function |
default: do not validate if file is empty or has bad mime type |
options.validateFileContent | boolean | function |
default false. Also check the content of the file to recognize the mime type |
options.fileIdAttributes | Array |
array of strings : Describes which attributes of files will be taken as primary key for files to check if they already exist, even if they are moved. If not given, the file path will used for deduplication as before. |
options.subPath | string |
A subpath to save this file, will be created if needed. |
[contract] | object |
contract object associated to the file |
[contract.id] | string |
id of the contract |
[contract.name] | string |
name of the contract |
options.fetchFile | function |
the connector can give it’s own function to fetch the file from the website, which will be run only when necessary (if the corresponding file is missing on the cozy) function returning the stream). This function must return a promise resolved as a stream |
options.verboseFilesLog | boolean |
the connector will send saveFiles result as a warning |
Example
await saveFiles([{fileurl: 'https://...', filename: 'bill1.pdf'}], fields, { fileIdAttributes: ['fileurl'] })
saveIdentity¶
Helper to set or merge io.cozy.identities See https://github.com/cozy/cozy-doctypes/blob/master/docs/io.cozy.identities.md
saveIdentity() ⏏¶
Set or merge a io.cozy.identities
You need full permission for the doctype io.cozy.identities in your manifest, to be able to use this function. Parameters:
identity
(object): the identity to create/update as an io.cozy.identities object
accountIdentifier
(string): a string that represent the account use, if available fields.login
options
(object): options which will be given to updateOrCreate directly :
+ sourceAccount
(String): id of the source account
+ sourceAccountIdentifier
(String): identifier unique to the account targetted by the connector. It is the login most of the time
const { saveIdentity } = require('cozy-konnector-libs') const identity = { contact: { name: 'toto', email: { 'address': 'toto@example.com' } } } return saveIdentity(identity, fields.login)
signin¶
Provides an handy method to log the user in, on HTML form pages. On success, it resolves to a promise with a parsed body.
signin() ⏏¶
Provides an handy method to log the user in, on HTML form pages. On success, it resolves to a promise with a parsed body.
Errors:
- LOGIN_FAILED if the validate predicate is false
- INVALID_FORM if the element matched by
formSelector
is not a form or has noaction
attribute - UNKNOWN_PARSING_STRATEGY if
parse
is not one of the accepted values:raw
,cheerio
,json
. - VENDOR_DOWN if a request throws a RequestError, or StatusCodeError
It does not submit values provided through select
tags, except if populated
by user with formData
.
-
url
is the url to access the html form -
formSelector
is used by cheerio to uniquely identify the form in which to log in -
formData
is an object{ name: value, … }
. It is used to populate the form, in the proper inputs with the same name as the properties of this object, before submitting it. It can also be a function that returns this object. The page aturl
would be given as argument, right after having been parsed throughcheerio
. -
parse
allow the user to resolvesignin
with a preparsed body. The choice of the strategy for the parsing is one of :raw
,json
orcheerio
.cheerio
being the default. -
validate
is a predicate taking three argumentsstatusCode
,parsedBody
andfullResponse
. If it is false,LOGIN_FAILED
is thrown, otherwise the signin resolves withparsedBody
value. -
requestOpts
allows to pass eventual options to thesignin
‘srequestFactory
. It could be useful for pages usinglatin1
encoding
for instance.
Kind: Exported function
Example
== basic example : ==
const $ = signin({ url: `http://quotes.toscrape.com/login`, formSelector: 'form', formData: { username, password } })
If the behavior of the targeted website is not standard. You can pass a validate function which will allow you to: - detect if the credentials work or not -> LOGIN_FAILED - detect if actions from the user are needed -> USER_ACTION_NEEDED - detect if the targeted website is out -> VENDOR_DOWN Example
const $ = signin({ url: `http://quotes.toscrape.com/login`, formSelector: 'form', formData: { username, password }, validate: (statusCode, $, fullResponse) { if (statusCode !== 200) return false // LOGIN_FAILED if ($('.cgu').length) throw new Error('USER_ACTION_NEEDED') if (fullResponse.request.uri.href.includes('error')) throw new Error('VENDOR_DOWN') } })
Do not forget that the use of the signin function is not mandatory in a connector and won’t work if the signin page does not use html forms. Here, a simple POST request may be a lot more simple.
solveCaptcha¶
Use every possible means to solve a captcha. At the moment, Anticaptcha web service is used if any related secret key is found in COZY_PARAMETERS environment variable.
solveCaptcha() ⏏¶
Use every possible means to solve a captcha. At the moment, Anticaptcha web service is used if any related secret key is found in COZY_PARAMETERS environment variable. If you do not want to solve the captcha each time the connector is run, please also use CookieKonnector which will help you save the session.
Parameters:
params
is an array of objects with any attributes with some mandatory attributes :type
(String): (default recaptcha) type of captcha to solve. can be “recaptcha” or “image” at the momenttimeout
(Number): (default 3 minutes after now) time when the solver should stop trying to solve the captchawithFullSolution
(Boolean): (default false) Change the return to an object containing full solutionwebsiteKey
(String): the key you can find on the targeted website (for recaptcha)websiteURL
(String): The URL of the page showing the captcha (for recaptcha)body
(String): The base64 encoded image (for image captcha) Returns: Promise with the solved captcha response as a string
Kind: Exported function
Example
const { solveCaptcha } = require('cozy-konnector-libs') const solvedKey = await solveCaptcha({ websiteKey: 'the key in the webpage', websiteURL: 'http://quotes.toscrape.com/login', }) // now use the solveKey to submit your form
updateOrCreate¶
Creates or updates the given entries according to if they already exist in the cozy or not
updateOrCreate() ⏏¶
Creates or updates the given entries according to if they already exist in the cozy or not
You need the full permission for the given doctype in your manifest, to be able to use this function.
entries
(object array): Documents to save
doctype
(string): Doctype of the documents
matchingAttributes
(string array): attributes in each entry used to check if an entry already exists in the Cozy
options
(object): general option affecting metadata :
+ sourceAccount
(String): id of the source account
+ sourceAccountIdentifier
(String): identifier unique to the account targetted by the connector. It is the login most of the time
utils¶
Small utilities helping to develop konnectors.
utils~fetchAll()¶
This function allows to fetch all documents for a given doctype. It is the fastest to get all documents but without filtering possibilities deprecated by the findAll method from cozyClient
Parameters:
doctype
(string): the doctype from which you want to fetch the data
Kind: inner method of utils
utils~queryAll()¶
This function allows to fetch all documents for a given doctype exceeding the 100 limit. It is slower that fetchAll because it fetches the data 100 by 100 but allows to filter the data with a selector and an index
Parameters:
doctype
(string): the doctype from which you want to fetch the data
selector
(object): the mango query selector
index
(object): (optional) the query selector index. If not defined, the function will
create it’s own index with the keys specified in the selector
const documents = await queryAll('io.cozy.bills', {vendor: 'Direct Energie'})
Kind: inner method of utils
utils~findDuplicates()¶
This function find duplicates in a given doctype, filtered by an optional mango selector
Parameters:
doctype
(string): the doctype from which you want to fetch the data
selector
(object): (optional) the mango query selector
options
:
- keys
(array) : List of keys used to check that two items are the same.
- index
(optionnal) : Return value returned by cozy.data.defineIndex
, the default will correspond to all documents of the selected doctype.
- selector
(optionnal object) : Mango request to get records. Gets all the records by default
Returns an object with the following keys:
toKeep
: this is the list of unique documents that you should keep in db
toRemove
: this is the list of documents that can remove from db. If this is io.cozy.bills
documents, do not forget to clean linked bank operations
const {toKeep, toRemove} = await findDuplicates('io.cozy.bills', {selector: {vendor: 'Direct Energie'}})
Kind: inner method of utils
utils~batchUpdateAttributes()¶
This is a shortcut to update multiple documents in one call
Parameters:
doctype
(string): the doctype from which you want to fetch the data
ids
(array): array of ids of documents to update
transformation
(object): attributes to change with their values
options
:
- keys
(array) : List of keys used to check that two items are the same.
- index
(optionnal) : Return value returned by cozy.data.defineIndex
, the default will correspond to all documents of the selected doctype.
- selector
(optionnal object) : Mango request to get records. Gets all the records by default
Returns a promise which resolves with all the return values of updateAttributes
await batchUpdateAttributes('io.cozy.bills', [1, 2, 3], {vendor: 'Direct Energie'})
Kind: inner method of utils
utils~batchDelete()¶
This is a shortcut to delete multiple documents in one call
Parameters:
doctype
(string): the doctype from which you want to fetch the data
documents
(array): documents to delete with their ids
transformation
(object): attributes to change with their values
options
:
- keys
(array) : List of keys used to check that two items are the same.
- index
(optionnal) : Return value returned by cozy.data.defineIndex
, the default will correspond to all documents of the selected doctype.
- selector
(optionnal object) : Mango request to get records. Gets all the records by default
Returns a promise which resolves with all the return values of updateAttributes
Example to remove all the documents for a given doctype
const documents = await fetchAll('io.cozy.marvel') await batchDelete('io.cozy.marvel', documents)
Kind: inner method of utils
utils~getPdfText()¶
This function can read the content of a cozy pdf file and output its text
Parameters:
fileId
(string): the id of the file in the cozy
options
:
- pages
(array or number) : The list of page you want to interpret
You need to add pdfjs-dist package as a dependency to your connector to allow this to work
Returns a promise which resolves with an object with the following attributes:
- text
(string) : The full text of the pdf
- 1
: The full pdfjs data for page 1
- n
: The full pdfjs data for page n
Example:
const pdfText = (await getPdfText('887ABCFE87687')).text
Kind: inner method of utils
utils~formatDate()¶
This function convert a Date Object to a ISO date string (2018-07-31)
Parameters:
date
(Date): the id of the file in the cozy
Returns a string
Example:
const date = formatFrenchDate(New Date.now())
Kind: inner method of utils
categorization¶
Bank transactions categorization
- categorization
- ~createCategorizer(options) ⇒
Object
- ~categorize(transactions, options) ⇒
Array.<object>
- ~CreateCategorizerOptions
- ~createCategorizer(options) ⇒
categorization~createCategorizer(options) ⇒ Object
¶
Initialize global and local models and return an object exposing a
categorize
function that applies both models on an array of transactions
The global model is a model specific to hosted Cozy instances. It is not available for self-hosted instances. It will just do nothing in that case.
The local model is based on the user manual categorizations.
Each model adds two properties to the transactions:
The global model adds cozyCategoryId
and cozyCategoryProba
The local model adds localCategoryId
and localCategoryProba
In the end, each transaction can have up to four different categories. An application can use these categories to show the most significant for the user. See https://github.com/cozy/cozy-doctypes/blob/master/docs/io.cozy.bank.md#categories for more informations.
Kind: inner method of categorization
Returns: Object
- A method to categorize transactions and the classifiers it uses.
Param | Type | Description |
---|---|---|
options | CreateCategorizerOptions |
Options used to build the categorizer |
Example
const { BaseKonnector, createCategorizer } = require('cozy-konnector-libs') class BankingKonnector extends BaseKonnector { async saveTransactions() { const transactions = await this.fetchTransactions() const categorizer = await createCategorizer const categorizedTransactions = await categorizer.categorize(transactions) // Save categorizedTransactions } }
categorization~categorize(transactions, options) ⇒ Array.<object>
¶
Initialize global and local models and categorize the given array of transactions
Kind: inner method of categorization
Returns: Array.<object>
- the categorized transactions
See: createCategorizer for more informations about models initialization
Param | Type | Description |
---|---|---|
transactions | Array.<object> |
The transactions to categorize |
options | CreateCategorizerOptions |
Options passed to create the categorizer |
Example
const { BaseKonnector, categorize } = require('cozy-konnector-libs') class BankingKonnector extends BaseKonnector { async saveTransactions() { const transactions = await this.fetchTransactions() const categorizedTransactions = await categorize(transactions) // Save categorizedTransactions } }
categorization~CreateCategorizerOptions¶
Kind: inner typedef of categorization
Properties
Name | Type | Description |
---|---|---|
useGlobalModel | boolean |
Whether to use the globally trained model |
customTransactionFetcher | function |
A custom training transaction fetcher |
pretrainedClassifier | object |
A pretrained instance of a bayes classifier |
BaseKonnector¶
The class from which all the connectors must inherit.
It takes a fetch function in parameter that must return a Promise
.
You need at least the GET
permission on io.cozy.accounts
in your manifest to allow it to
fetch account information for your connector.
Its role is twofold :
- Make the link between account data and konnector
- Handle errors
⚠️ A promise should be returned from the fetch
function otherwise
the konnector cannot know that asynchronous code has been called.
Kind: global class
- BaseKonnector
- new BaseKonnector(fetch)
- .run()
- .main() ⇒
Promise
- .end()
- .fail()
- .readPayload() ⇒
Object
|null
- .initAttributes()
- .saveAccountData(data, options) ⇒
Promise
- .getAccountData() ⇒
object
- .updateAccountAttributes()
- .setTwoFAState(options)
- .resetTwoFAState()
- .waitForTwoFaCode(options) ⇒
Promise
- .notifySuccessfulLogin()
- .deactivateAutoSuccessfulLogin()
- .saveBills() ⇒
Promise
- .saveFiles() ⇒
Promise
- .updateOrCreate() ⇒
Promise
- .saveIdentity() ⇒
Promise
- .signin() ⇒
Promise
- .terminate(err)
- .getCozyMetadata(data)
new BaseKonnector(fetch)¶
Constructor
Param | Type | Description |
---|---|---|
fetch | function |
Function to be run automatically after account data is fetched. This function will be binded to the current connector. If not fetch function is given. The connector will have to handle itself it’s own exection and error handling |
Example
const { BaseKonnector } = require('cozy-konnector-libs') module.exports = new BaseKonnector(function fetch () { // use this to access the instance of the konnector to // store any information that needs to be passed to // different stages of the konnector return request('http://ameli.fr') .then(computeReimbursements) .then(saveBills) })
baseKonnector.run()¶
Entrypoint of the konnector
- Initializes connector attributes
- Awaits this.main
- Ensures errors are handled via this.fail
- Calls this.end when the main function succeeded
Kind: instance method of BaseKonnector
baseKonnector.main() ⇒ Promise
¶
Main runs after konnector has been initialized. Errors thrown will be automatically handled.
Kind: instance method of BaseKonnector
Returns: Promise
- - The konnector is considered successful when it resolves
baseKonnector.end()¶
Hook called when the connector has ended successfully
Kind: instance method of BaseKonnector
baseKonnector.fail()¶
Hook called when the connector fails
Kind: instance method of BaseKonnector
baseKonnector.readPayload() ⇒ Object
| null
¶
Read an eventual payload from COZY_PAYLOAD env var, wether it is a JSON string or a reference to a file containing a JSON string
Kind: instance method of BaseKonnector
Returns: Object
| null
- Promise<> result of JSON.parse from the JSON string or null if no payload
baseKonnector.initAttributes()¶
Initializes konnector attributes that will be used during its lifetime
- this._account
- this.fields
Kind: instance method of BaseKonnector
baseKonnector.saveAccountData(data, options) ⇒ Promise
¶
Saves data to the account that is passed to the konnector. Use it to persist data that needs to be passed to each konnector run.
By default, the data is merged to the remote data, use
options.merge = false
to overwrite the data.
The data is saved under the .data
attribute of the cozy
account.
Don’t forget to modify the manifest.konnector file to give the right to write on the
io.cozy.accounts
doctype. The syntax can be : "permissions": {"accounts": {"type": "io.cozy.accounts"}}
(here we juste removed the verb GET
)
Kind: instance method of BaseKonnector
Returns: Promise
- : resolved with the modified account
Param | Type | Description |
---|---|---|
data | object |
Attributes to be merged |
options | object |
{ merge: true |
baseKonnector.getAccountData() ⇒ object
¶
Get the data saved by saveAccountData
Kind: instance method of BaseKonnector
Returns: object
- the account data
baseKonnector.updateAccountAttributes()¶
Update account attributes and cache the account
Kind: instance method of BaseKonnector
baseKonnector.setTwoFAState(options)¶
Sets the 2FA state, according to the type passed. Doing so resets the twoFACode field
Typically you should not use that directly, prefer to use waitForTwoFaCode since the wait for user input will be handled for you. It is useful though for the “app” type where no user input (inside Cozy) is needed.
Kind: instance method of BaseKonnector
Param | Type | Description |
---|---|---|
options | object |
The list of options |
options.type | string |
Used by the front to show the right message (email/sms/app/app_code) |
options.retry | boolean |
Is this function call a retry ? This changes the resulting message to the user |
baseKonnector.resetTwoFAState()¶
Resets 2FA state when not needed anymore
Kind: instance method of BaseKonnector
baseKonnector.waitForTwoFaCode(options) ⇒ Promise
¶
Notices that 2FA code is needed and wait for the user to submit it. It uses the account to do the communication with the user.
Kind: instance method of BaseKonnector
Returns: Promise
- Contains twoFa code entered by user
Throws:
- Will throw
USER_ACTION_NEEDED.TWOFA_EXPIRED
if the konnector job is not run manually (we assume that not run manually means that we do not have a graphic interface to fill the required information) - Will throw
USER_ACTION_NEEDED.TWOFA_EXPIRED
if 2FA is not filled by the user soon enough
Param | Type | Description |
---|---|---|
options | object |
The list of options |
options.type | string |
(default: “email”) - Type of the expected 2FA code. The message displayed to the user will depend on it. Possible values: email, sms |
options.timeout | number |
(default 3 minutes after now) - After this date, the stop will stop waiting and and an error will be shown to the user (deprecated and alias of endTime) |
options.endTime | number |
(default 3 minutes after now) - After this timestamp, the home will stop waiting and and an error will be shown to the user |
options.heartBeat | number |
(default: 5000) - How many milliseconds between each code check |
options.retry | boolean |
(default: false) - Is it a retry. If true, an error message will be displayed to the user |
Example
const { BaseKonnector } = require('cozy-konnector-libs') module.exports = new BaseKonnector(start) async function start() { // we detect the need of a 2FA code const code = this.waitForTwoFaCode({ type: 'email' }) // send the code to the targeted site }
baseKonnector.notifySuccessfulLogin()¶
Tells Cozy-Home that we have successfully logged in.
Useful when auto-success has been deactivated.
See deactivateAutoSuccess
Kind: instance method of BaseKonnector
baseKonnector.deactivateAutoSuccessfulLogin()¶
By default, cozy-home considers that the konnector has successfully logged in when the konnector has run for more than 8s. This is problematic for 2FA since the konnector can sit idle, just waiting for the 2FA to come back.
When this method is called, cozy-home is notified and will not consider the
absence of error after 8s to be a success. Afterwards, to notify cozy-home when
the user has logged in successfully, for example, after the user has entered 2FA
codes, it is necessary to call notifySuccessfulLogin
.
Does nothing if called more than once.
Kind: instance method of BaseKonnector
baseKonnector.saveBills() ⇒ Promise
¶
This is saveBills function from cozy-konnector-libs which automatically adds sourceAccount in metadata of each entry
Kind: instance method of BaseKonnector
Returns: Promise
- resolves with entries hydrated with db data
baseKonnector.saveFiles() ⇒ Promise
¶
This is saveFiles function from cozy-konnector-libs which automatically adds sourceAccount and sourceAccountIdentifier cozyMetadatas to files
Kind: instance method of BaseKonnector
Returns: Promise
- resolves with the list of entries with file objects
baseKonnector.updateOrCreate() ⇒ Promise
¶
This is updateOrCreate function from cozy-konnector-libs which automatically adds sourceAccount in metadata of each entry
Kind: instance method of BaseKonnector
Returns: Promise
- resolves to an array of db objects
baseKonnector.saveIdentity() ⇒ Promise
¶
This is saveIdentity function from cozy-konnector-libs which automatically adds sourceAccount in metadata of each entry
Kind: instance method of BaseKonnector
Returns: Promise
- empty promise
baseKonnector.signin() ⇒ Promise
¶
This is signin function from cozy-konnector-libs which automatically adds deactivateAutoSuccessfulLogin and notifySuccessfulLogin calls
Kind: instance method of BaseKonnector
Returns: Promise
- resolve with an object containing form data
baseKonnector.terminate(err)¶
Send a special error code which is interpreted by the cozy stack to terminate the execution of the connector now
Kind: instance method of BaseKonnector
Param | Type | Description |
---|---|---|
err | string |
The error code to be saved as connector result see [docs/ERROR_CODES.md] |
Example
this.terminate('LOGIN_FAILED')
baseKonnector.getCozyMetadata(data)¶
Get cozyMetaData from the context of the connector
Kind: instance method of BaseKonnector
Param | Type | Description |
---|---|---|
data | object |
this data will be merged with cozyMetaData |
CookieKonnector¶
Connector base class extending BaseKonnector which handles cookie session in a central way
It also handles saving cookie session in the account and automatically restore it for the next
connector run.
All cozy-konnector-libs tools using request are proposed as methods of this class to force them
to use the central cookie which can be saved/restored.
You need at least the GET
and PUT
permissions on io.cozy.accounts
in your manifest to allow
it to save/restore cookies
Kind: global class
- CookieKonnector
- new CookieKonnector(requestFactoryOptions)
- .initAttributes() ⇒
Promise
- .end()
- .requestFactory(options) ⇒
object
- .resetSession() ⇒
Promise
- .initSession() ⇒
Promise
- .saveSession() ⇒
Promise
- .signin() ⇒
Promise
- .saveFiles() ⇒
Promise
- .saveBills() ⇒
Promise
new CookieKonnector(requestFactoryOptions)¶
Constructor
Param | Type | Description |
---|---|---|
requestFactoryOptions | function |
Option object passed to requestFactory to initialize this.request. It is still possible to change this.request doing : javascript this.request = this.requestFactory(...) Please not you have to run the connector yourself doing : javascript connector.run() |
Example
const { CookieKonnector } = require('cozy-konnector-libs') class MyConnector extends CookieKonnector { async fetch(fields) { // the code of your connector await this.request('https://...') } async testSession() { const $ = await this.request('https://...') return $('') } } const connector = new MyKonnector({ cheerio: true, json: false }) connector.run()
cookieKonnector.initAttributes() ⇒ Promise
¶
Initializes the current connector with data coming from the associated account and also the session
Kind: instance method of CookieKonnector
Returns: Promise
- with the fields as an object
cookieKonnector.end()¶
Hook called when the connector is ended
Kind: instance method of CookieKonnector
cookieKonnector.requestFactory(options) ⇒ object
¶
Calls cozy-konnector-libs requestFactory forcing this._jar as the cookie
Kind: instance method of CookieKonnector
Returns: object
- - The resulting request object
Param | Type | Description |
---|---|---|
options | object |
requestFactory option |
cookieKonnector.resetSession() ⇒ Promise
¶
Reset cookie session with a new empty session and save it to the associated account
Kind: instance method of CookieKonnector
Returns: Promise
- empty promise
cookieKonnector.initSession() ⇒ Promise
¶
Get the cookie session from the account if any
Kind: instance method of CookieKonnector
Returns: Promise
- true or false if the session in the account exists or not
cookieKonnector.saveSession() ⇒ Promise
¶
Saves the current cookie session to the account
Kind: instance method of CookieKonnector
Returns: Promise
- empty promise
cookieKonnector.signin() ⇒ Promise
¶
This is signin function from cozy-konnector-libs which is forced to use the current cookies and current request from CookieKonnector. It also automatically saves the session after signin if it is a success.
Kind: instance method of CookieKonnector
Returns: Promise
- resolve with an object containing form data
cookieKonnector.saveFiles() ⇒ Promise
¶
This is saveFiles function from cozy-konnector-libs which is forced to use the current cookies and current request from CookieKonnector.
Kind: instance method of CookieKonnector
Returns: Promise
- resolves with the list of entries with file objects
cookieKonnector.saveBills() ⇒ Promise
¶
This is saveBills function from cozy-konnector-libs which is forced to use the current cookies and current request from CookieKonnector.
Kind: instance method of CookieKonnector
Returns: Promise
- resolves with entries hydrated with db data
Document¶
Simple Model for Documents. Allows to specify
shouldSave
, shouldUpdate
as methods.
Has useful isEqual
method
document.isEqual()¶
Compares to another document deeply.
_id
and _rev
are by default ignored in the comparison.
By default, will compare dates loosely since you often
compare existing documents (dates in ISO string) with documents
that just have been scraped where dates are Date
s.
Kind: instance method of Document
fs¶
Manifest is provided differently in developement that in production.
-
In production, the manifest has been “merged” via Webpack via the DefinePlugin
-
In development/test, we simply read the manifest from the fs
LOGIN_FAILED : string
¶
The konnector could not login
NOT_EXISTING_DIRECTORY : string
¶
The folder specified as folder_to_save does not exist (checked by BaseKonnector)
VENDOR_DOWN : string
¶
The vendor’s website is down
USER_ACTION_NEEDED : string
¶
There was an unexpected error, please take a look at the logs to know what happened
FILE_DOWNLOAD_FAILED : string
¶
There was a problem while downloading a file
SAVE_FILE_FAILED : string
¶
There was a problem while saving a file
DISK_QUOTA_EXCEEDED : string
¶
Could not save a file to the cozy because of disk quota exceeded
CHALLENGE_ASKED : string
¶
It seems that the website requires a second authentification factor that we don’t support yet.
LOGIN_FAILED_TOO_MANY_ATTEMPTS : string
¶
Temporarily blocked
USER_ACTION_NEEDED_OAUTH_OUTDATED : string
¶
Access refresh required
USER_ACTION_NEEDED_ACCOUNT_REMOVED : string
¶
Unavailable account
USER_ACTION_NEEDED_CHANGE_PASSWORD : string
¶
Unavailable account
USER_ACTION_NEEDED_PERMISSIONS_CHANGED : string
¶
Password update required
USER_ACTION_NEEDED_CGU_FORM : string
¶
The user needs to accept a CGU form before accessing the rest of the website
CAPTCHA_RESOLUTION_FAILED : string
¶
solveCaptcha failed to solve the captcha
LOGIN_FAILED_NEEDS_SECRET : string
¶
Additionnal information is needed to check your login details
MAINTENANCE : string
¶
remote website seems to be unavailable
TERMS_VERSION_MISMATCH : string
¶
User needs to accept new terms
UNKNOWN_ERROR : string
¶
unkown error
UNKNOWN_ERROR_PARTIAL_SYNC : string
¶
The synchronization is complete but some elements may be missing
USER_ACTION_NEEDED_SCA_REQUIRED : string
¶
Renewal of authentication required
USER_ACTION_NEEDED_TWOFA_EXPIRED : string
¶
Authentication renewal required
USER_ACTION_NEEDED_WEBAUTH_REQUIRED : string
¶
Authentication on vendor website required
USER_ACTION_NEEDED_WRONG_TWOFA_CODE : string
¶
Incorrect strong authentication code
VENDOR_DOWN_BANK_DOWN : string
¶
Unavailable bank website
VENDOR_DOWN_LINXO_DOWN : string
¶
Unavailable bank website
attachProcessEventHandlers(prcs) ⇒ function
¶
Attach event handlers to catch uncaught exceptions/rejections and signals. Log them as critical and exit the process accordingly. If the cleanup function has not been called, calling again the function is a no-op.
Kind: global function
Returns: function
- When called, removes the signal handlers
Param | Type | Description |
---|---|---|
prcs | object |
Process object, default to current process |
mkSpec()¶
Declarative scraping.
Describe your items attributes and where to find/parse them instead of imperatively building them.
Heavily inspired by artoo scraping method.
scrape($, specs, [childSelector]) ⇒ object
| Array
¶
Scrape a cheerio object for properties
Kind: global function
Returns: object
| Array
- - Item(s) scraped
Param | Type | Description |
---|---|---|
$ | object |
Cheerio node which will be scraped |
specs | object | string |
Options object describing what you want to scrape |
[childSelector] | string |
If passed, scrape will return an array of items |
Example
scrape
can be used to declaratively extract data :
- For one object :
const item = scrape($('#item'), { title: '.title', content: '.content' })
- For a list of objects :
const items = scrape($('#content'), { title: '.title', content: '.content' }, '.item')
For more power, you can use object
s for each retriever :
const items = scrape($('#content'), { title: '.title', content: '.content', link: { sel: 'a', attr: 'href' }, }, '.item')
Here the href
attribute of the a
inside .item
s would have been
put into the link
attribute of the items returned by scrape
.
Available options :
sel
: the CSS selector used to target the HTML node from which data will be scrapedattr
: the HTML attribute from which to extract dataparse
: function applied to the value extracted ({ sel: '.price', parse: parseAmount }
)fn
: if you need something more complicated thanattr
, you can use this function, it receives the complete DOM node.{ sel: '.person', fn: $node => $node.attr('data-name') + $node.attr('data-firstname') }
⚠ Permissions¶
Please note that some classes require some permissions:
io.cozy.accounts
for theBaseKonnector
class (GET
only)io.cozy.files
to save filesio.cozy.bills
to save billsio.cozy.bank.operations
forlinkBankOperations