API
API¶
Modules¶
- CozyBrowser
Get a javascript simulation of a real browser (jsdom)
- addData
Saves the data into the cozy blindly without check.
- cozyClient
cozy-client-js instance already initialized and ready to use.
- hydrateAndFilter
Filters the passed array from data already present in the cozy so that there is not duplicated data in the Cozy.
- linkBankOperations
Finds links between bills and bank operations.
- mkdirp
- normalizeFilename
Returns the given name, replacing characters that could be an issue when used in a filename with spaces.
- saveBills
Encapsulates the saving of Bills : saves the files, saves the new data, and associate the files to an existing bank operation
- saveFiles
Saves the given files in the given folder via the Cozy API.
- saveIdentity
Helper to set or merge io.cozy.identities See https://github.com/cozy/cozy-doctypes/blob/master/docs/io.cozy.identities.md
- signin
Provides an handy method to log the user in, on HTML form pages. On success, it resolves to a promise with a parsed body.
- solveCaptcha
Use every possible means to solve a captcha. At the moment, Anticaptcha web service is used if any related secret key is found in COZY_PARAMETERS environment variable.
- updateOrCreate
Creates or updates the given entries according to if they already exist in the cozy or not
- utils
Small utilities helping to develop konnectors.
- categorization
Bank transactions categorization
Classes¶
- BaseKonnector
The class from which all the connectors must inherit. It takes a fetch function in parameter that must return a
Promise. You need at least theGETpermission onio.cozy.accountsin your manifest to allow it to fetch account information for your connector.Its role is twofold :
- Make the link between account data and konnector
- Handle errors
⚠️ A promise should be returned from the
fetchfunction otherwise the konnector cannot know that asynchronous code has been called.- CookieKonnector
Connector base class extending BaseKonnector which handles cookie session in a central way It also handles saving cookie session in the account and automatically restore it for the next connector run. All cozy-konnector-libs tools using request are proposed as methods of this class to force them to use the central cookie which can be saved/restored. You need at least the
GETandPUTpermissions onio.cozy.accountsin your manifest to allow it to save/restore cookies- Document
Simple Model for Documents. Allows to specify
shouldSave,shouldUpdateas methods.Has useful
isEqualmethod
Constants¶
- fs
Manifest is provided differently in developement that in production.
In production, the manifest has been "merged" via Webpack via the DefinePlugin
In development/test, we simply read the manifest from the fs
- LOGIN_FAILED :
string The konnector could not login
- NOT_EXISTING_DIRECTORY :
string The folder specified as folder_to_save does not exist (checked by BaseKonnector)
- VENDOR_DOWN :
string The vendor's website is down
- USER_ACTION_NEEDED :
string There was an unexpected error, please take a look at the logs to know what happened
- FILE_DOWNLOAD_FAILED :
string There was a problem while downloading a file
- SAVE_FILE_FAILED :
string There was a problem while saving a file
- DISK_QUOTA_EXCEEDED :
string Could not save a file to the cozy because of disk quota exceeded
- CHALLENGE_ASKED :
string It seems that the website requires a second authentification factor that we don’t support yet.
- LOGIN_FAILED_TOO_MANY_ATTEMPTS :
string Temporarily blocked
- USER_ACTION_NEEDED_OAUTH_OUTDATED :
string Access refresh required
- USER_ACTION_NEEDED_ACCOUNT_REMOVED :
string Unavailable account
- USER_ACTION_NEEDED_CHANGE_PASSWORD :
string Unavailable account
- USER_ACTION_NEEDED_PERMISSIONS_CHANGED :
string Password update required
- USER_ACTION_NEEDED_CGU_FORM :
string The user needs to accept a CGU form before accessing the rest of the website
- CAPTCHA_RESOLUTION_FAILED :
string solveCaptcha failed to solve the captcha
- LOGIN_FAILED_NEEDS_SECRET :
string Additionnal information is needed to check your login details
- MAINTENANCE :
string remote website seems to be unavailable
- TERMS_VERSION_MISMATCH :
string User needs to accept new terms
- UNKNOWN_ERROR :
string unkown error
- UNKNOWN_ERROR_PARTIAL_SYNC :
string The synchronization is complete but some elements may be missing
- USER_ACTION_NEEDED_SCA_REQUIRED :
string Renewal of authentication required
- USER_ACTION_NEEDED_TWOFA_EXPIRED :
string Authentication renewal required
- USER_ACTION_NEEDED_WEBAUTH_REQUIRED :
string Authentication on vendor website required
- USER_ACTION_NEEDED_WRONG_TWOFA_CODE :
string Incorrect strong authentication code
- VENDOR_DOWN_BANK_DOWN :
string Unavailable bank website
- VENDOR_DOWN_LINXO_DOWN :
string Unavailable bank website
Functions¶
- attachProcessEventHandlers(prcs) ⇒
function Attach event handlers to catch uncaught exceptions/rejections and signals. Log them as critical and exit the process accordingly. If the cleanup function has not been called, calling again the function is a no-op.
- mkSpec()
Declarative scraping.
Describe your items attributes and where to find/parse them instead of imperatively building them.
Heavily inspired by artoo scraping method.
- scrape($, specs, [childSelector]) ⇒
object|Array Scrape a cheerio object for properties
CozyBrowser¶
Get a javascript simulation of a real browser (jsdom)
- CozyBrowser
- defaultOptions ⇒
object⏏
- defaultOptions ⇒
defaultOptions ⇒ object ⏏¶
Get a preconfigured jsdom browser simulator using the zombie npm package See http://zombie.js.org/ for complete documentation The connector has to import the zombie npm package itself.
Kind: Exported constant
Returns: object - Zombie browser extended class
| Param | Type | Description |
|---|---|---|
| options.userAgent | string |
The user agent string used by the browser |
Example
const Browser = require('cozy-konnector-libs/libs/CozyBrowser') const browser = new Browser() await browser.visit('http://quotes.toscrape.com/')
defaultOptions~addListeners()¶
Add cozy-konnector-libs specific logs to browser events
Kind: inner method of defaultOptions
addData¶
Saves the data into the cozy blindly without check.
addData() ⏏¶
Saves the data into the cozy blindly without check.
You need at least the POST permission for the given doctype in your manifest, to be able to
use this function.
Parameters:
documents: an array of objects corresponding to the data you want to save in the cozy
doctype (string): the doctype where you want to save data (ex: ‘io.cozy.bills’)
options (object): option object
+ sourceAccount (String): id of the source account
+ sourceAccountIdentifier (String): identifier unique to the account targetted by the connector. It is the login most of the time
const documents = [ { name: 'toto', height: 1.8 }, { name: 'titi', height: 1.7 } ] return addData(documents, 'io.cozy.height')
cozyClient¶
cozy-client-js instance already initialized and ready to use.
cozyClient ⏏¶
cozy-client-js instance already initialized and ready to use.
If you want to access cozy-client-js directly, this method gives you directly an instance of it,
initialized according to COZY_URL and COZY_CREDENTIALS environment variable given by cozy-stack
You can refer to the cozy-client-js documentation for more information.
Example :
const {cozyClient} = require('cozy-konnector-libs') cozyClient.data.defineIndex('my.doctype', ['_id'])
hydrateAndFilter¶
Filters the passed array from data already present in the cozy so that there is not duplicated data in the Cozy.
hydrateAndFilter() ⏏¶
Filters the passed array from data already present in the cozy so that there is not duplicated data in the Cozy.
You need at least the GET permission for the given doctype in your manifest, to be able to
use this function.
Parameters:
documents: an array of objects corresponding to the data you want to save in the cozy
doctype (string): the doctype where you want to save data (ex: ‘io.cozy.bills’)
options :
- keys (array) : List of keys used to check that two items are the same. By default it is set to ['id']'.
-index(optionnal) : Return value returned bycozy.data.defineIndex, the default will correspond to all documents of the selected doctype.
-selector(optionnal object) : Mango request to get records. Default is built from the keys}}` to get all the records.
const documents = [ { name: 'toto', height: 1.8 }, { name: 'titi', height: 1.7 } ] return hydrateAndFilter(documents, 'io.cozy.height', { keys: ['name'] }).then(filteredDocuments => addData(filteredDocuments, 'io.cozy.height'))
hydrateAndFilter~suitableCall()¶
Since we can use methods or basic functions for
shouldSave and shouldUpdate we pass the
appropriate this and arguments.
If funcOrMethod is a method, it will be called
with args[0] as this and the rest as arguments
Otherwise, this will be null and args will be passed
as arguments.
Kind: inner method of hydrateAndFilter
linkBankOperations¶
Finds links between bills and bank operations.
linkBankOperations() ⏏¶
Will soon move to a dedicated service. You should not use it.
Finds links between bills and bank operations.
mkdirp¶
mkdirp ⏏¶
Creates a directory and its missing ancestors as needed.
Options :
...pathComponents: one or many path components to be joined
await mkdirp('/foo') // Creates /foo await mkdirp('/foo') // Does nothing as /foo already exists await mkdirp('/bar/baz') // Creates /bar, then /bar/baz await mkdirp('/foo/bar/baz') // Creates /foo/bar, then /foo/bar/baz, not /foo await mkdirp('/') // Does nothing await mkdirp('/qux', 'qux2/qux3', 'qux4') // Creates /qux, then /qux/qux2, // then /qux/qux2/qux3 and // finally /qux/qux2/qux3/qux4
The function will automatically add a leading slash when missing:
await mkdirp('foo', 'bar') // Creates /foo, then /foo/bar
normalizeFilename¶
Returns the given name, replacing characters that could be an issue when used in a filename with spaces.
normalizeFilename() ⏏¶
Returns the given name, replacing characters that could be an issue when used in a filename with spaces.
Replaced characters include:
- Those forbidden on one or many popular OS or filesystem:
<>:"/\|?* - Those forbidden by the cozy-stack
\0,\rand\n - Multiple spaces and/or tabs are replaced with a single space
- Leading & trailing spaces and/or tabs are removed
An exception will be thrown in case there is not any filename-compatible character in the given name.
Parameters:
basenameis whatever string you want to generate the filename fromextis an optional file extension, with or without leading dot
const { normalizeFilename } = require('cozy-konnector-libs') const filename = normalizeFilename('*foo/bar: <baz> \\"qux"\t???', '.txt') // `filename` === `foo bar baz qux.txt`
saveBills¶
Encapsulates the saving of Bills : saves the files, saves the new data, and associate the files to an existing bank operation
saveBills() ⏏¶
Combines the features of saveFiles, hydrateAndFilter, addData and linkBankOperations for a
common case: bills.
Will create io.cozy.bills objects. The default deduplication keys are ['date', 'amount', 'vendor'].
You need the full permission on io.cozy.bills, full permission on io.cozy.files and also
full permission on io.cozy.bank.operations in your manifest, to be able to use this function.
Parameters:
documentsis an array of objects with any attributes with some mandatory attributes :amount(Number): the amount of the bill used to match bank operationsdate(Date): the date of the bill also used to match bank operationsvendor(String): the name of the vendor associated to the bill. Ex: ‘trainline’currency(String) default: EUR: The ISO currency value (not mandatory since there is a default value.contractId(String): Contract unique identicator used to deduplicate billscontractLabel: (String) User label if define, must be used with contractIdmatchingCriterias(Object): criterias that can be used by an external service to match bills with bank operations. If not specified but the ‘banksTransactionRegExp’ attribute is specified in the manifest of the connector, this value is automatically added to the bill
You can also pass attributes expected by saveFiles : fileurl, filename, requestOptions
and more
Please take a look at io.cozy.bills doctype documentation
- fields (object) this is the first parameter given to BaseKonnector’s constructor
- options is passed directly to saveFiles, hydrateAndFilter, addData and linkBankOperations.
Kind: Exported function
Example
const { BaseKonnector, saveBills } = require('cozy-konnector-libs') module.exports = new BaseKonnector(function fetch (fields) { const documents = [] // some code which fills documents return saveBills(documents, fields, { identifiers: ['vendor'] }) })
saveFiles¶
Saves the given files in the given folder via the Cozy API.
saveFiles(entries, fields, options, [contract]) ⏏¶
Saves the files given in the fileurl attribute of each entries
You need the full permission on io.cozy.files in your manifest to use this function.
Kind: Exported function
| Param | Type | Description |
|---|---|---|
| entries | Array |
list of object describing files to save |
| entries.fileurl | string |
The url of the file (can be a function returning the value). Ignored if filestream is given |
| entries.fetchFile | function |
the connector can give it’s own function to fetch the file from the website, which will be run only when necessary (if the corresponding file is missing on the cozy) function returning the stream). This function must return a promise resolved as a stream |
| entries.filestream | object | string |
the stream which will be directly passed to cozyClient.files.create (can also be function returning the stream) |
| entries.requestOptions | object |
The options passed to request to fetch fileurl (can be a function returning the value) |
| entries.filename | string |
The file name of the item written on disk. This attribute is optional and as default value, the file name will be “smartly” guessed by the function. Use this attribute if the guess is not smart enough for you, or if you use filestream (can be a function returning the value). |
| entries.shouldReplaceName | string |
used to migrate filename. If saveFiles finds a file linked to this entry and this file name matches shouldReplaceName, the file is renamed to filename (can be a function returning the value) |
| entries.shouldReplaceFile | function |
use this function to state if the current entry should be forced to be redownloaded and replaced. Usefull if we know the file content can change and we always want the last version. |
| entries.fileAttributes | object |
ex: {created_at: new Date()} sets some additionnal file attributes passed to cozyClient.file.create |
| entries.subPath | string |
A subpath to save all files, will be created if needed. |
| entries.contract | object |
contract object associated to the files |
| entries.contract.id | string |
id of the contract |
| entries.contract.name | string |
name of the contract |
| fields | object |
is the argument given to the main function of your connector by the BaseKonnector. It especially contains a folderPath which is the string path configured by the user in collect/home |
| options | object |
global options |
| options.timeout | number |
timestamp which can be used if your connector needs to fetch a lot of files and if the stack does not give enough time to your connector to fetch it all. It could happen that the connector is stopped right in the middle of the download of the file and the file will be broken. With the timeout option, the saveFiles function will check if the timeout has passed right after downloading each file and then will be sure to be stopped cleanly if the timeout is not too long. And since it is really fast to check that a file has already been downloaded, on the next run of the connector, it will be able to download some more files, and so on. If you want the timeout to be in 10s, do Date.now() + 10*1000. You can try it in the previous code. |
| options.contentType | number | boolean |
ex: ‘application/pdf’ used to force the contentType of documents when they are badly recognized by cozy. If “true” the content type will be recognized from the file name and forced the same way. |
| options.concurrency | number |
default: 1 sets the maximum number of concurrent downloads |
| options.validateFile | function |
default: do not validate if file is empty or has bad mime type |
| options.validateFileContent | boolean | function |
default false. Also check the content of the file to recognize the mime type |
| options.fileIdAttributes | Array |
array of strings : Describes which attributes of files will be taken as primary key for files to check if they already exist, even if they are moved. If not given, the file path will used for deduplication as before. |
| options.subPath | string |
A subpath to save this file, will be created if needed. |
| [contract] | object |
contract object associated to the file |
| [contract.id] | string |
id of the contract |
| [contract.name] | string |
name of the contract |
| options.fetchFile | function |
the connector can give it’s own function to fetch the file from the website, which will be run only when necessary (if the corresponding file is missing on the cozy) function returning the stream). This function must return a promise resolved as a stream |
| options.verboseFilesLog | boolean |
the connector will send saveFiles result as a warning |
Example
await saveFiles([{fileurl: 'https://...', filename: 'bill1.pdf'}], fields, { fileIdAttributes: ['fileurl'] })
saveIdentity¶
Helper to set or merge io.cozy.identities See https://github.com/cozy/cozy-doctypes/blob/master/docs/io.cozy.identities.md
saveIdentity() ⏏¶
Set or merge a io.cozy.identities
You need full permission for the doctype io.cozy.identities in your manifest, to be able to use this function. Parameters:
identity (object): the identity to create/update as an io.cozy.identities object
accountIdentifier (string): a string that represent the account use, if available fields.login
options (object): options which will be given to updateOrCreate directly :
+ sourceAccount (String): id of the source account
+ sourceAccountIdentifier (String): identifier unique to the account targetted by the connector. It is the login most of the time
const { saveIdentity } = require('cozy-konnector-libs') const identity = { contact: { name: 'toto', email: { 'address': 'toto@example.com' } } } return saveIdentity(identity, fields.login)
signin¶
Provides an handy method to log the user in, on HTML form pages. On success, it resolves to a promise with a parsed body.
signin() ⏏¶
Provides an handy method to log the user in, on HTML form pages. On success, it resolves to a promise with a parsed body.
Errors:
- LOGIN_FAILED if the validate predicate is false
- INVALID_FORM if the element matched by
formSelectoris not a form or has noactionattribute - UNKNOWN_PARSING_STRATEGY if
parseis not one of the accepted values:raw,cheerio,json. - VENDOR_DOWN if a request throws a RequestError, or StatusCodeError
It does not submit values provided through select tags, except if populated
by user with formData.
-
urlis the url to access the html form -
formSelectoris used by cheerio to uniquely identify the form in which to log in -
formDatais an object{ name: value, … }. It is used to populate the form, in the proper inputs with the same name as the properties of this object, before submitting it. It can also be a function that returns this object. The page aturlwould be given as argument, right after having been parsed throughcheerio. -
parseallow the user to resolvesigninwith a preparsed body. The choice of the strategy for the parsing is one of :raw,jsonorcheerio.cheeriobeing the default. -
validateis a predicate taking three argumentsstatusCode,parsedBodyandfullResponse. If it is false,LOGIN_FAILEDis thrown, otherwise the signin resolves withparsedBodyvalue. -
requestOptsallows to pass eventual options to thesignin‘srequestFactory. It could be useful for pages usinglatin1encodingfor instance.
Kind: Exported function
Example
== basic example : ==
const $ = signin({ url: `http://quotes.toscrape.com/login`, formSelector: 'form', formData: { username, password } })
If the behavior of the targeted website is not standard. You can pass a validate function which will allow you to: - detect if the credentials work or not -> LOGIN_FAILED - detect if actions from the user are needed -> USER_ACTION_NEEDED - detect if the targeted website is out -> VENDOR_DOWN Example
const $ = signin({ url: `http://quotes.toscrape.com/login`, formSelector: 'form', formData: { username, password }, validate: (statusCode, $, fullResponse) { if (statusCode !== 200) return false // LOGIN_FAILED if ($('.cgu').length) throw new Error('USER_ACTION_NEEDED') if (fullResponse.request.uri.href.includes('error')) throw new Error('VENDOR_DOWN') } })
Do not forget that the use of the signin function is not mandatory in a connector and won’t work if the signin page does not use html forms. Here, a simple POST request may be a lot more simple.
solveCaptcha¶
Use every possible means to solve a captcha. At the moment, Anticaptcha web service is used if any related secret key is found in COZY_PARAMETERS environment variable.
solveCaptcha() ⏏¶
Use every possible means to solve a captcha. At the moment, Anticaptcha web service is used if any related secret key is found in COZY_PARAMETERS environment variable. If you do not want to solve the captcha each time the connector is run, please also use CookieKonnector which will help you save the session.
Parameters:
paramsis an array of objects with any attributes with some mandatory attributes :type(String): (default recaptcha) type of captcha to solve. can be “recaptcha” or “image” at the momenttimeout(Number): (default 3 minutes after now) time when the solver should stop trying to solve the captchawithFullSolution(Boolean): (default false) Change the return to an object containing full solutionwebsiteKey(String): the key you can find on the targeted website (for recaptcha)websiteURL(String): The URL of the page showing the captcha (for recaptcha)body(String): The base64 encoded image (for image captcha) Returns: Promise with the solved captcha response as a string
Kind: Exported function
Example
const { solveCaptcha } = require('cozy-konnector-libs') const solvedKey = await solveCaptcha({ websiteKey: 'the key in the webpage', websiteURL: 'http://quotes.toscrape.com/login', }) // now use the solveKey to submit your form
updateOrCreate¶
Creates or updates the given entries according to if they already exist in the cozy or not
updateOrCreate() ⏏¶
Creates or updates the given entries according to if they already exist in the cozy or not
You need the full permission for the given doctype in your manifest, to be able to use this function.
entries (object array): Documents to save
doctype (string): Doctype of the documents
matchingAttributes (string array): attributes in each entry used to check if an entry already exists in the Cozy
options (object): general option affecting metadata :
+ sourceAccount (String): id of the source account
+ sourceAccountIdentifier (String): identifier unique to the account targetted by the connector. It is the login most of the time
utils¶
Small utilities helping to develop konnectors.
utils~fetchAll()¶
This function allows to fetch all documents for a given doctype. It is the fastest to get all documents but without filtering possibilities deprecated by the findAll method from cozyClient
Parameters:
doctype (string): the doctype from which you want to fetch the data
Kind: inner method of utils
utils~queryAll()¶
This function allows to fetch all documents for a given doctype exceeding the 100 limit. It is slower that fetchAll because it fetches the data 100 by 100 but allows to filter the data with a selector and an index
Parameters:
doctype (string): the doctype from which you want to fetch the data
selector (object): the mango query selector
index (object): (optional) the query selector index. If not defined, the function will
create it’s own index with the keys specified in the selector
const documents = await queryAll('io.cozy.bills', {vendor: 'Direct Energie'})
Kind: inner method of utils
utils~findDuplicates()¶
This function find duplicates in a given doctype, filtered by an optional mango selector
Parameters:
doctype (string): the doctype from which you want to fetch the data
selector (object): (optional) the mango query selector
options :
- keys (array) : List of keys used to check that two items are the same.
- index (optionnal) : Return value returned by cozy.data.defineIndex, the default will correspond to all documents of the selected doctype.
- selector (optionnal object) : Mango request to get records. Gets all the records by default
Returns an object with the following keys:
toKeep: this is the list of unique documents that you should keep in db
toRemove: this is the list of documents that can remove from db. If this is io.cozy.bills
documents, do not forget to clean linked bank operations
const {toKeep, toRemove} = await findDuplicates('io.cozy.bills', {selector: {vendor: 'Direct Energie'}})
Kind: inner method of utils
utils~batchUpdateAttributes()¶
This is a shortcut to update multiple documents in one call
Parameters:
doctype (string): the doctype from which you want to fetch the data
ids (array): array of ids of documents to update
transformation (object): attributes to change with their values
options :
- keys (array) : List of keys used to check that two items are the same.
- index (optionnal) : Return value returned by cozy.data.defineIndex, the default will correspond to all documents of the selected doctype.
- selector (optionnal object) : Mango request to get records. Gets all the records by default
Returns a promise which resolves with all the return values of updateAttributes
await batchUpdateAttributes('io.cozy.bills', [1, 2, 3], {vendor: 'Direct Energie'})
Kind: inner method of utils
utils~batchDelete()¶
This is a shortcut to delete multiple documents in one call
Parameters:
doctype (string): the doctype from which you want to fetch the data
documents (array): documents to delete with their ids
transformation (object): attributes to change with their values
options :
- keys (array) : List of keys used to check that two items are the same.
- index (optionnal) : Return value returned by cozy.data.defineIndex, the default will correspond to all documents of the selected doctype.
- selector (optionnal object) : Mango request to get records. Gets all the records by default
Returns a promise which resolves with all the return values of updateAttributes
Example to remove all the documents for a given doctype
const documents = await fetchAll('io.cozy.marvel') await batchDelete('io.cozy.marvel', documents)
Kind: inner method of utils
utils~getPdfText()¶
This function can read the content of a cozy pdf file and output its text
Parameters:
fileId (string): the id of the file in the cozy
options :
- pages (array or number) : The list of page you want to interpret
You need to add pdfjs-dist package as a dependency to your connector to allow this to work
Returns a promise which resolves with an object with the following attributes:
- text (string) : The full text of the pdf
- 1 : The full pdfjs data for page 1
- n : The full pdfjs data for page n
Example:
const pdfText = (await getPdfText('887ABCFE87687')).text
Kind: inner method of utils
utils~formatDate()¶
This function convert a Date Object to a ISO date string (2018-07-31)
Parameters:
date (Date): the id of the file in the cozy
Returns a string
Example:
const date = formatFrenchDate(New Date.now())
Kind: inner method of utils
categorization¶
Bank transactions categorization
- categorization
- ~createCategorizer(options) ⇒
Object - ~categorize(transactions, options) ⇒
Array.<object> - ~CreateCategorizerOptions
- ~createCategorizer(options) ⇒
categorization~createCategorizer(options) ⇒ Object¶
Initialize global and local models and return an object exposing a
categorize function that applies both models on an array of transactions
The global model is a model specific to hosted Cozy instances. It is not available for self-hosted instances. It will just do nothing in that case.
The local model is based on the user manual categorizations.
Each model adds two properties to the transactions:
The global model adds cozyCategoryId and cozyCategoryProba
The local model adds localCategoryId and localCategoryProba
In the end, each transaction can have up to four different categories. An application can use these categories to show the most significant for the user. See https://github.com/cozy/cozy-doctypes/blob/master/docs/io.cozy.bank.md#categories for more informations.
Kind: inner method of categorization
Returns: Object - A method to categorize transactions and the classifiers it uses.
| Param | Type | Description |
|---|---|---|
| options | CreateCategorizerOptions |
Options used to build the categorizer |
Example
const { BaseKonnector, createCategorizer } = require('cozy-konnector-libs') class BankingKonnector extends BaseKonnector { async saveTransactions() { const transactions = await this.fetchTransactions() const categorizer = await createCategorizer const categorizedTransactions = await categorizer.categorize(transactions) // Save categorizedTransactions } }
categorization~categorize(transactions, options) ⇒ Array.<object>¶
Initialize global and local models and categorize the given array of transactions
Kind: inner method of categorization
Returns: Array.<object> - the categorized transactions
See: createCategorizer for more informations about models initialization
| Param | Type | Description |
|---|---|---|
| transactions | Array.<object> |
The transactions to categorize |
| options | CreateCategorizerOptions |
Options passed to create the categorizer |
Example
const { BaseKonnector, categorize } = require('cozy-konnector-libs') class BankingKonnector extends BaseKonnector { async saveTransactions() { const transactions = await this.fetchTransactions() const categorizedTransactions = await categorize(transactions) // Save categorizedTransactions } }
categorization~CreateCategorizerOptions¶
Kind: inner typedef of categorization
Properties
| Name | Type | Description |
|---|---|---|
| useGlobalModel | boolean |
Whether to use the globally trained model |
| customTransactionFetcher | function |
A custom training transaction fetcher |
| pretrainedClassifier | object |
A pretrained instance of a bayes classifier |
BaseKonnector¶
The class from which all the connectors must inherit.
It takes a fetch function in parameter that must return a Promise.
You need at least the GET permission on io.cozy.accounts in your manifest to allow it to
fetch account information for your connector.
Its role is twofold :
- Make the link between account data and konnector
- Handle errors
⚠️ A promise should be returned from the fetch function otherwise
the konnector cannot know that asynchronous code has been called.
Kind: global class
- BaseKonnector
- new BaseKonnector(fetch)
- .run()
- .main() ⇒
Promise - .end()
- .fail()
- .readPayload() ⇒
Object|null - .initAttributes()
- .saveAccountData(data, options) ⇒
Promise - .getAccountData() ⇒
object - .updateAccountAttributes()
- .setTwoFAState(options)
- .resetTwoFAState()
- .waitForTwoFaCode(options) ⇒
Promise - .notifySuccessfulLogin()
- .deactivateAutoSuccessfulLogin()
- .saveBills() ⇒
Promise - .saveFiles() ⇒
Promise - .updateOrCreate() ⇒
Promise - .saveIdentity() ⇒
Promise - .signin() ⇒
Promise - .terminate(err)
- .getCozyMetadata(data)
new BaseKonnector(fetch)¶
Constructor
| Param | Type | Description |
|---|---|---|
| fetch | function |
Function to be run automatically after account data is fetched. This function will be binded to the current connector. If not fetch function is given. The connector will have to handle itself it’s own exection and error handling |
Example
const { BaseKonnector } = require('cozy-konnector-libs') module.exports = new BaseKonnector(function fetch () { // use this to access the instance of the konnector to // store any information that needs to be passed to // different stages of the konnector return request('http://ameli.fr') .then(computeReimbursements) .then(saveBills) })
baseKonnector.run()¶
Entrypoint of the konnector
- Initializes connector attributes
- Awaits this.main
- Ensures errors are handled via this.fail
- Calls this.end when the main function succeeded
Kind: instance method of BaseKonnector
baseKonnector.main() ⇒ Promise¶
Main runs after konnector has been initialized. Errors thrown will be automatically handled.
Kind: instance method of BaseKonnector
Returns: Promise - - The konnector is considered successful when it resolves
baseKonnector.end()¶
Hook called when the connector has ended successfully
Kind: instance method of BaseKonnector
baseKonnector.fail()¶
Hook called when the connector fails
Kind: instance method of BaseKonnector
baseKonnector.readPayload() ⇒ Object | null¶
Read an eventual payload from COZY_PAYLOAD env var, wether it is a JSON string or a reference to a file containing a JSON string
Kind: instance method of BaseKonnector
Returns: Object | null - Promise<> result of JSON.parse from the JSON string or null if no payload
baseKonnector.initAttributes()¶
Initializes konnector attributes that will be used during its lifetime
- this._account
- this.fields
Kind: instance method of BaseKonnector
baseKonnector.saveAccountData(data, options) ⇒ Promise¶
Saves data to the account that is passed to the konnector. Use it to persist data that needs to be passed to each konnector run.
By default, the data is merged to the remote data, use
options.merge = false to overwrite the data.
The data is saved under the .data attribute of the cozy
account.
Don’t forget to modify the manifest.konnector file to give the right to write on the
io.cozy.accounts doctype. The syntax can be : "permissions": {"accounts": {"type": "io.cozy.accounts"}} (here we juste removed the verb GET)
Kind: instance method of BaseKonnector
Returns: Promise - : resolved with the modified account
| Param | Type | Description |
|---|---|---|
| data | object |
Attributes to be merged |
| options | object |
{ merge: true |
baseKonnector.getAccountData() ⇒ object¶
Get the data saved by saveAccountData
Kind: instance method of BaseKonnector
Returns: object - the account data
baseKonnector.updateAccountAttributes()¶
Update account attributes and cache the account
Kind: instance method of BaseKonnector
baseKonnector.setTwoFAState(options)¶
Sets the 2FA state, according to the type passed. Doing so resets the twoFACode field
Typically you should not use that directly, prefer to use waitForTwoFaCode since the wait for user input will be handled for you. It is useful though for the “app” type where no user input (inside Cozy) is needed.
Kind: instance method of BaseKonnector
| Param | Type | Description |
|---|---|---|
| options | object |
The list of options |
| options.type | string |
Used by the front to show the right message (email/sms/app/app_code) |
| options.retry | boolean |
Is this function call a retry ? This changes the resulting message to the user |
baseKonnector.resetTwoFAState()¶
Resets 2FA state when not needed anymore
Kind: instance method of BaseKonnector
baseKonnector.waitForTwoFaCode(options) ⇒ Promise¶
Notices that 2FA code is needed and wait for the user to submit it. It uses the account to do the communication with the user.
Kind: instance method of BaseKonnector
Returns: Promise - Contains twoFa code entered by user
Throws:
- Will throw
USER_ACTION_NEEDED.TWOFA_EXPIREDif the konnector job is not run manually (we assume that not run manually means that we do not have a graphic interface to fill the required information) - Will throw
USER_ACTION_NEEDED.TWOFA_EXPIREDif 2FA is not filled by the user soon enough
| Param | Type | Description |
|---|---|---|
| options | object |
The list of options |
| options.type | string |
(default: “email”) - Type of the expected 2FA code. The message displayed to the user will depend on it. Possible values: email, sms |
| options.timeout | number |
(default 3 minutes after now) - After this date, the stop will stop waiting and and an error will be shown to the user (deprecated and alias of endTime) |
| options.endTime | number |
(default 3 minutes after now) - After this timestamp, the home will stop waiting and and an error will be shown to the user |
| options.heartBeat | number |
(default: 5000) - How many milliseconds between each code check |
| options.retry | boolean |
(default: false) - Is it a retry. If true, an error message will be displayed to the user |
Example
const { BaseKonnector } = require('cozy-konnector-libs') module.exports = new BaseKonnector(start) async function start() { // we detect the need of a 2FA code const code = this.waitForTwoFaCode({ type: 'email' }) // send the code to the targeted site }
baseKonnector.notifySuccessfulLogin()¶
Tells Cozy-Home that we have successfully logged in.
Useful when auto-success has been deactivated.
See deactivateAutoSuccess
Kind: instance method of BaseKonnector
baseKonnector.deactivateAutoSuccessfulLogin()¶
By default, cozy-home considers that the konnector has successfully logged in when the konnector has run for more than 8s. This is problematic for 2FA since the konnector can sit idle, just waiting for the 2FA to come back.
When this method is called, cozy-home is notified and will not consider the
absence of error after 8s to be a success. Afterwards, to notify cozy-home when
the user has logged in successfully, for example, after the user has entered 2FA
codes, it is necessary to call notifySuccessfulLogin.
Does nothing if called more than once.
Kind: instance method of BaseKonnector
baseKonnector.saveBills() ⇒ Promise¶
This is saveBills function from cozy-konnector-libs which automatically adds sourceAccount in metadata of each entry
Kind: instance method of BaseKonnector
Returns: Promise - resolves with entries hydrated with db data
baseKonnector.saveFiles() ⇒ Promise¶
This is saveFiles function from cozy-konnector-libs which automatically adds sourceAccount and sourceAccountIdentifier cozyMetadatas to files
Kind: instance method of BaseKonnector
Returns: Promise - resolves with the list of entries with file objects
baseKonnector.updateOrCreate() ⇒ Promise¶
This is updateOrCreate function from cozy-konnector-libs which automatically adds sourceAccount in metadata of each entry
Kind: instance method of BaseKonnector
Returns: Promise - resolves to an array of db objects
baseKonnector.saveIdentity() ⇒ Promise¶
This is saveIdentity function from cozy-konnector-libs which automatically adds sourceAccount in metadata of each entry
Kind: instance method of BaseKonnector
Returns: Promise - empty promise
baseKonnector.signin() ⇒ Promise¶
This is signin function from cozy-konnector-libs which automatically adds deactivateAutoSuccessfulLogin and notifySuccessfulLogin calls
Kind: instance method of BaseKonnector
Returns: Promise - resolve with an object containing form data
baseKonnector.terminate(err)¶
Send a special error code which is interpreted by the cozy stack to terminate the execution of the connector now
Kind: instance method of BaseKonnector
| Param | Type | Description |
|---|---|---|
| err | string |
The error code to be saved as connector result see [docs/ERROR_CODES.md] |
Example
this.terminate('LOGIN_FAILED')
baseKonnector.getCozyMetadata(data)¶
Get cozyMetaData from the context of the connector
Kind: instance method of BaseKonnector
| Param | Type | Description |
|---|---|---|
| data | object |
this data will be merged with cozyMetaData |
CookieKonnector¶
Connector base class extending BaseKonnector which handles cookie session in a central way
It also handles saving cookie session in the account and automatically restore it for the next
connector run.
All cozy-konnector-libs tools using request are proposed as methods of this class to force them
to use the central cookie which can be saved/restored.
You need at least the GET and PUT permissions on io.cozy.accounts in your manifest to allow
it to save/restore cookies
Kind: global class
- CookieKonnector
- new CookieKonnector(requestFactoryOptions)
- .initAttributes() ⇒
Promise - .end()
- .requestFactory(options) ⇒
object - .resetSession() ⇒
Promise - .initSession() ⇒
Promise - .saveSession() ⇒
Promise - .signin() ⇒
Promise - .saveFiles() ⇒
Promise - .saveBills() ⇒
Promise
new CookieKonnector(requestFactoryOptions)¶
Constructor
| Param | Type | Description |
|---|---|---|
| requestFactoryOptions | function |
Option object passed to requestFactory to initialize this.request. It is still possible to change this.request doing : javascript this.request = this.requestFactory(...) Please not you have to run the connector yourself doing : javascript connector.run() |
Example
const { CookieKonnector } = require('cozy-konnector-libs') class MyConnector extends CookieKonnector { async fetch(fields) { // the code of your connector await this.request('https://...') } async testSession() { const $ = await this.request('https://...') return $('') } } const connector = new MyKonnector({ cheerio: true, json: false }) connector.run()
cookieKonnector.initAttributes() ⇒ Promise¶
Initializes the current connector with data coming from the associated account and also the session
Kind: instance method of CookieKonnector
Returns: Promise - with the fields as an object
cookieKonnector.end()¶
Hook called when the connector is ended
Kind: instance method of CookieKonnector
cookieKonnector.requestFactory(options) ⇒ object¶
Calls cozy-konnector-libs requestFactory forcing this._jar as the cookie
Kind: instance method of CookieKonnector
Returns: object - - The resulting request object
| Param | Type | Description |
|---|---|---|
| options | object |
requestFactory option |
cookieKonnector.resetSession() ⇒ Promise¶
Reset cookie session with a new empty session and save it to the associated account
Kind: instance method of CookieKonnector
Returns: Promise - empty promise
cookieKonnector.initSession() ⇒ Promise¶
Get the cookie session from the account if any
Kind: instance method of CookieKonnector
Returns: Promise - true or false if the session in the account exists or not
cookieKonnector.saveSession() ⇒ Promise¶
Saves the current cookie session to the account
Kind: instance method of CookieKonnector
Returns: Promise - empty promise
cookieKonnector.signin() ⇒ Promise¶
This is signin function from cozy-konnector-libs which is forced to use the current cookies and current request from CookieKonnector. It also automatically saves the session after signin if it is a success.
Kind: instance method of CookieKonnector
Returns: Promise - resolve with an object containing form data
cookieKonnector.saveFiles() ⇒ Promise¶
This is saveFiles function from cozy-konnector-libs which is forced to use the current cookies and current request from CookieKonnector.
Kind: instance method of CookieKonnector
Returns: Promise - resolves with the list of entries with file objects
cookieKonnector.saveBills() ⇒ Promise¶
This is saveBills function from cozy-konnector-libs which is forced to use the current cookies and current request from CookieKonnector.
Kind: instance method of CookieKonnector
Returns: Promise - resolves with entries hydrated with db data
Document¶
Simple Model for Documents. Allows to specify
shouldSave, shouldUpdate as methods.
Has useful isEqual method
document.isEqual()¶
Compares to another document deeply.
_id and _rev are by default ignored in the comparison.
By default, will compare dates loosely since you often
compare existing documents (dates in ISO string) with documents
that just have been scraped where dates are Dates.
Kind: instance method of Document
fs¶
Manifest is provided differently in developement that in production.
-
In production, the manifest has been “merged” via Webpack via the DefinePlugin
-
In development/test, we simply read the manifest from the fs
LOGIN_FAILED : string¶
The konnector could not login
NOT_EXISTING_DIRECTORY : string¶
The folder specified as folder_to_save does not exist (checked by BaseKonnector)
VENDOR_DOWN : string¶
The vendor’s website is down
USER_ACTION_NEEDED : string¶
There was an unexpected error, please take a look at the logs to know what happened
FILE_DOWNLOAD_FAILED : string¶
There was a problem while downloading a file
SAVE_FILE_FAILED : string¶
There was a problem while saving a file
DISK_QUOTA_EXCEEDED : string¶
Could not save a file to the cozy because of disk quota exceeded
CHALLENGE_ASKED : string¶
It seems that the website requires a second authentification factor that we don’t support yet.
LOGIN_FAILED_TOO_MANY_ATTEMPTS : string¶
Temporarily blocked
USER_ACTION_NEEDED_OAUTH_OUTDATED : string¶
Access refresh required
USER_ACTION_NEEDED_ACCOUNT_REMOVED : string¶
Unavailable account
USER_ACTION_NEEDED_CHANGE_PASSWORD : string¶
Unavailable account
USER_ACTION_NEEDED_PERMISSIONS_CHANGED : string¶
Password update required
USER_ACTION_NEEDED_CGU_FORM : string¶
The user needs to accept a CGU form before accessing the rest of the website
CAPTCHA_RESOLUTION_FAILED : string¶
solveCaptcha failed to solve the captcha
LOGIN_FAILED_NEEDS_SECRET : string¶
Additionnal information is needed to check your login details
MAINTENANCE : string¶
remote website seems to be unavailable
TERMS_VERSION_MISMATCH : string¶
User needs to accept new terms
UNKNOWN_ERROR : string¶
unkown error
UNKNOWN_ERROR_PARTIAL_SYNC : string¶
The synchronization is complete but some elements may be missing
USER_ACTION_NEEDED_SCA_REQUIRED : string¶
Renewal of authentication required
USER_ACTION_NEEDED_TWOFA_EXPIRED : string¶
Authentication renewal required
USER_ACTION_NEEDED_WEBAUTH_REQUIRED : string¶
Authentication on vendor website required
USER_ACTION_NEEDED_WRONG_TWOFA_CODE : string¶
Incorrect strong authentication code
VENDOR_DOWN_BANK_DOWN : string¶
Unavailable bank website
VENDOR_DOWN_LINXO_DOWN : string¶
Unavailable bank website
attachProcessEventHandlers(prcs) ⇒ function¶
Attach event handlers to catch uncaught exceptions/rejections and signals. Log them as critical and exit the process accordingly. If the cleanup function has not been called, calling again the function is a no-op.
Kind: global function
Returns: function - When called, removes the signal handlers
| Param | Type | Description |
|---|---|---|
| prcs | object |
Process object, default to current process |
mkSpec()¶
Declarative scraping.
Describe your items attributes and where to find/parse them instead of imperatively building them.
Heavily inspired by artoo scraping method.
scrape($, specs, [childSelector]) ⇒ object | Array¶
Scrape a cheerio object for properties
Kind: global function
Returns: object | Array - - Item(s) scraped
| Param | Type | Description |
|---|---|---|
| $ | object |
Cheerio node which will be scraped |
| specs | object | string |
Options object describing what you want to scrape |
| [childSelector] | string |
If passed, scrape will return an array of items |
Example
scrape can be used to declaratively extract data :
- For one object :
const item = scrape($('#item'), {
title: '.title',
content: '.content'
})
- For a list of objects :
const items = scrape($('#content'), {
title: '.title',
content: '.content'
}, '.item')
For more power, you can use objects for each retriever :
const items = scrape($('#content'), {
title: '.title',
content: '.content',
link: {
sel: 'a',
attr: 'href'
},
}, '.item')
Here the href attribute of the a inside .items would have been
put into the link attribute of the items returned by scrape.
Available options :
sel: the CSS selector used to target the HTML node from which data will be scrapedattr: the HTML attribute from which to extract dataparse: function applied to the value extracted ({ sel: '.price', parse: parseAmount })fn: if you need something more complicated thanattr, you can use this function, it receives the complete DOM node.{ sel: '.person', fn: $node => $node.attr('data-name') + $node.attr('data-firstname') }
⚠ Permissions¶
Please note that some classes require some permissions:
io.cozy.accountsfor theBaseKonnectorclass (GETonly)io.cozy.filesto save filesio.cozy.billsto save billsio.cozy.bank.operationsforlinkBankOperations