[github/shaarli/Shaarli.git] / doc / md / dev / Development.md

# Development

Please read [Contributing to Shaarli](https://github.com/shaarli/Shaarli/tree/master/CONTRIBUTING.md)

## Guidelines


- [Unit tests](Unit-tests)
- Javascript linting - Shaarli uses [Airbnb JavaScript Style Guide](https://github.com/airbnb/javascript).
Run `make eslint` to check JS style.
- [GnuPG signature](GnuPG-signature) for tags/releases


## Third-party libraries

CSS:

- Yahoo UI [CSS Reset](http://yuilibrary.com/yui/docs/cssreset/) - standardize cross-browser rendering

Javascript:

- [Awesomeplete](https://leaverou.github.io/awesomplete/) ([GitHub](https://github.com/LeaVerou/awesomplete)) - autocompletion in input forms
- [bLazy](http://dinbror.dk/blazy/) ([GitHub](https://github.com/dinbror/blazy)) - lazy loading for thumbnails
- [qr.js](http://neocotic.com/qr.js/) ([GitHub](https://github.com/neocotic/qr.js)) - QR code generation

PHP (managed through [`composer.json`](https://github.com/shaarli/Shaarli/blob/master/composer.json)):

- [RainTPL](https://github.com/rainphp/raintpl) - HTML templating for PHP
- [`shaarli/netscape-bookmark-parser`](https://packagist.org/packages/shaarli/netscape-bookmark-parser) - Import bookmarks from Netscape files
- [`erusev/parsedown`](https://packagist.org/packages/erusev/parsedown) - Parse MarkDown syntax for the MarkDown plugin
- [`slim/slim`](https://packagist.org/packages/slim/slim) - Handle routes and middleware for the REST API
- [`ArthurHoaro/web-thumbnailer`](https://github.com/ArthurHoaro/web-thumbnailer) - PHP library which will retrieve a thumbnail for any given URL
- [`pubsubhubbub/publisher`](https://github.com/pubsubhubbub/php-publisher) - A PubSubHubbub publisher module for PHP.
- [`gettext/gettext`](https://github.com/php-gettext/Gettext) - PHP library to collect and manipulate gettext (.po, .mo, .php, .json, etc)


## Security

- The password is salted, hashed and stored in the data subdirectory, in a PHP file, and protected by htaccess. Even if the webserver does not support htaccess, the hash is not readable by URL. Even if the .php file is stolen, the password cannot deduced from the hash. The salt prevents rainbow-tables attacks.
- Directories are protected using `.htaccess` files
- Forms are protected against [XSRF](http://en.wikipedia.org/wiki/Cross-site_request_forgery):
    - Forms which act on data (save,delete…) contain a token generated by the server.
    - Any posted form which does not contain a valid token is rejected.
    - Any token can only be used once.
    - Tokens are attached to the session and cannot be reused in another session.
- Sessions automatically expire after 60 minutes.
- Sessions are protected against hijacking: the session ID cannot be used from a different IP address.
- Links are stored as an associative array which is serialized, compressed (with deflate), base64-encoded and saved as a comment in a `.php` file - even if the server does not support `.htaccess` files, the data file will still not be readable by URL.
- Bruteforce protection: Successful and failed login attempts are logged - IP bans are enforced after a configurable amount of failures. Logs can also be used consumed by [fail2ban](../Server-configuration.md#fail2ban)
- A pop-up notification is shown when a new release is available.

## Link structure

Every link available through the `LinkDB` object is represented as an array
containing the following fields:

  * `id` (integer): Unique identifier.
  * `title` (string): Title of the link.
  * `url` (string): URL of the link. Used for displayable links (without redirector, url encoding, etc.).
           Can be absolute or relative for Notes.
  * `real_url` (string): Real destination URL, can be redirected, encoded, etc.
  * `shorturl` (string): Permalink small hash.
  * `description` (string): Link text description.
  * `private` (boolean): whether the link is private or not.
  * `tags` (string): all link tags separated by a single space
  * `thumbnail` (string|boolean): relative path of the thumbnail cache file, or false if there isn't any.
  * `created` (DateTime): link creation date time.
  * `updated` (DateTime): last modification date time.

Small hashes are used to make a link to an entry in Shaarli. They are unique: the date of the item (eg. `20110923_150523`) is hashed with CRC32, then converted to base64 and some characters are replaced. They are always 6 characters longs and use only `A-Z a-z 0-9 - _` and `@`.


## Directory structure

Here is the directory structure of Shaarli and the purpose of the different files:

```bash
	index.php        # Main program
	application/     # Shaarli classes
		├── LinkDB.php

        ...

		└── Utils.php
	tests/           # Shaarli unitary & functional tests
		├── LinkDBTest.php

        ...

		├── utils    # utilities to ease testing
		│   └── ReferenceLinkDB.php
		└── UtilsTest.php
	assets/
	    ├── common/                # Assets shared by multiple themes
	        ├── ...
        ├── default/               # Assets for the default template, before compilation
            ├── fonts/                  # Font files
            ├── img/                    # Images used by the default theme
            ├── js/                     # JavaScript files in ES6 syntax
            ├── scss/                   # SASS files
        └── vintage/               # Assets for the vintage template, before compilation
            └── ...
    COPYING          # Shaarli license
    inc/             # static assets and 3rd party libraries
        └── rain.tpl.class.php     # RainTPL templating library
    images/          # Images and icons used in Shaarli
    data/            # data storage: bookmark database, configuration, logs, banlist...
        ├── config.json.php        # Shaarli configuration (login, password, timezone, title...)
        ├── datastore.php          # Your link database (compressed).
        ├── ipban.php              # IP address ban system data
        ├── lastupdatecheck.txt    # Update check timestamp file
        └── log.txt                # login/IPban log.
    tpl/             # RainTPL templates for Shaarli. They are used to build the pages.
        ├── default/               # Default Shaarli theme
            ├── fonts/                  # Font files
            ├── img/                    # Images
            ├── js/                     # JavaScript files compiled by Babel and compatible with all browsers
            ├── css/                    # CSS files compiled with SASS
        └── vintage/               # Legacy Shaarli theme
            └── ...
    cache/           # thumbnails cache
                     # This directory is automatically created. You can erase it anytime you want.
    tmp/             # Temporary directory for compiled RainTPL templates.
                     # This directory is automatically created. You can erase it anytime you want.
    vendor/          # Third-party dependencies. This directory is created by Composer
```

Shaarli needs read access to:

- the root index.php file
- the `application/`, `plugins/` and `inc/` directories (recursively)

Shaarli needs read/write access to the `cache/`, `data/`, `pagecache/`, and `tmp/` directories


## Automation

A [`Makefile`](https://github.com/shaarli/Shaarli/blob/master/Makefile) is available to perform project-related operations:

- [Static analysis](#Static-analysis) - check that the code is compliant to PHP conventions
- [Unit tests](#Unit-tests) - ensure there are no regressions introduced by new commits
- Documentation - generate a local HTML copy of the markdown documentation

### Continuous Integration

[Travis CI](http://docs.travis-ci.com/) is a Continuous Integration build server, that runs a build:

- each time a commit is merged to the mainline (`master` branch)
- each time a Pull Request is submitted or updated

After all jobs have finished, Travis returns the results to GitHub:

- a status icon represents the result for the `master` branch: [![](https://api.travis-ci.org/shaarli/Shaarli.svg)](https://travis-ci.org/shaarli/Shaarli)
- Pull Requests are updated with the Travis build result.

See [`.travis.yml`](https://github.com/shaarli/Shaarli/blob/master/.travis.yml).


### Documentation

[mkdocs](https://www.mkdocs.org/) is used to convert markdown documentation to HTML pages. The [public documentation](https://shaarli.readthedocs.io/en/master/) website is rendered and hosted by [readthedocs.org](https://readthedocs.org/). A copy of the documentation is also included in prebuilt [release archives](https://github.com/shaarli/Shaarli/releases) (`doc/html/` path in your Shaarli installation). To generate the HTML documentation locally, install a recent version of Python `setuptools` and run    `make doc`.


## Static analysis

Patches should try to stick to the [PHP Standard Recommendations](http://www.php-fig.org/psr/) (PSR), and must follow:

- [PSR-1](http://www.php-fig.org/psr/psr-1/) - Basic Coding Standard
- [PSR-2](http://www.php-fig.org/psr/psr-2/) - Coding Style Guide
- [PSR-12](http://www.php-fig.org/psr/psr-12/) - Extended Coding Style  Guide

These are enforced on pull requests using our Continuous Integration tools.

**Work in progress:** Static analysis is currently being discussed here: in [#95 - Fix coding style (static analysis)](https://github.com/shaarli/Shaarli/issues/95), [#130 - Continuous Integration tools & features](https://github.com/shaarli/Shaarli/issues/130)

Static analysis tools can be installed with Composer, and used through Shaarli's [Makefile](https://github.com/shaarli/Shaarli/blob/master/Makefile).

For an overview of the available features, see:

- [Code quality: Makefile to run static code checkers](https://github.com/shaarli/Shaarli/pull/124) (#124)
- [Run PHPCS against different coding standards](https://github.com/shaarli/Shaarli/pull/276) (#276)
Commit	Line	Data
91a21c27	1	# Development
	2
	3	Please read [Contributing to Shaarli](https://github.com/shaarli/Shaarli/tree/master/CONTRIBUTING.md)
	4
	5	## Guidelines
	6
	7
	8	- [Unit tests](Unit-tests)
2f4df753	9	- Javascript linting - Shaarli uses [Airbnb JavaScript Style Guide](https://github.com/airbnb/javascript).
91a21c27	10	Run `make eslint` to check JS style.
	11	- [GnuPG signature](GnuPG-signature) for tags/releases
	12
	13
	14	## Third-party libraries
	15
	16	CSS:
	17
	18	- Yahoo UI [CSS Reset](http://yuilibrary.com/yui/docs/cssreset/) - standardize cross-browser rendering
	19
	20	Javascript:
	21
	22	- [Awesomeplete](https://leaverou.github.io/awesomplete/) ([GitHub](https://github.com/LeaVerou/awesomplete)) - autocompletion in input forms
	23	- [bLazy](http://dinbror.dk/blazy/) ([GitHub](https://github.com/dinbror/blazy)) - lazy loading for thumbnails
	24	- [qr.js](http://neocotic.com/qr.js/) ([GitHub](https://github.com/neocotic/qr.js)) - QR code generation
	25
	26	PHP (managed through [`composer.json`](https://github.com/shaarli/Shaarli/blob/master/composer.json)):
	27
	28	- [RainTPL](https://github.com/rainphp/raintpl) - HTML templating for PHP
	29	- [`shaarli/netscape-bookmark-parser`](https://packagist.org/packages/shaarli/netscape-bookmark-parser) - Import bookmarks from Netscape files
	30	- [`erusev/parsedown`](https://packagist.org/packages/erusev/parsedown) - Parse MarkDown syntax for the MarkDown plugin
	31	- [`slim/slim`](https://packagist.org/packages/slim/slim) - Handle routes and middleware for the REST API
	32	- [`ArthurHoaro/web-thumbnailer`](https://github.com/ArthurHoaro/web-thumbnailer) - PHP library which will retrieve a thumbnail for any given URL
	33	- [`pubsubhubbub/publisher`](https://github.com/pubsubhubbub/php-publisher) - A PubSubHubbub publisher module for PHP.
	34	- [`gettext/gettext`](https://github.com/php-gettext/Gettext) - PHP library to collect and manipulate gettext (.po, .mo, .php, .json, etc)
	35
	36
	37	## Security
	38
	39	- The password is salted, hashed and stored in the data subdirectory, in a PHP file, and protected by htaccess. Even if the webserver does not support htaccess, the hash is not readable by URL. Even if the .php file is stolen, the password cannot deduced from the hash. The salt prevents rainbow-tables attacks.
	40	- Directories are protected using `.htaccess` files
	41	- Forms are protected against [XSRF](http://en.wikipedia.org/wiki/Cross-site_request_forgery):
	42	- Forms which act on data (save,delete…) contain a token generated by the server.
	43	- Any posted form which does not contain a valid token is rejected.
	44	- Any token can only be used once.
	45	- Tokens are attached to the session and cannot be reused in another session.
	46	- Sessions automatically expire after 60 minutes.
	47	- Sessions are protected against hijacking: the session ID cannot be used from a different IP address.
	48	- Links are stored as an associative array which is serialized, compressed (with deflate), base64-encoded and saved as a comment in a `.php` file - even if the server does not support `.htaccess` files, the data file will still not be readable by URL.
	49	- Bruteforce protection: Successful and failed login attempts are logged - IP bans are enforced after a configurable amount of failures. Logs can also be used consumed by [fail2ban](../Server-configuration.md#fail2ban)
	50	- A pop-up notification is shown when a new release is available.
	51
	52	## Link structure
	53
2f4df753	54	Every link available through the `LinkDB` object is represented as an array
91a21c27	55	containing the following fields:
	56
	57	* `id` (integer): Unique identifier.
	58	* `title` (string): Title of the link.
2f4df753	59	* `url` (string): URL of the link. Used for displayable links (without redirector, url encoding, etc.).
91a21c27	60	Can be absolute or relative for Notes.
	61	* `real_url` (string): Real destination URL, can be redirected, encoded, etc.
	62	* `shorturl` (string): Permalink small hash.
	63	* `description` (string): Link text description.
	64	* `private` (boolean): whether the link is private or not.
	65	* `tags` (string): all link tags separated by a single space
	66	* `thumbnail` (string\|boolean): relative path of the thumbnail cache file, or false if there isn't any.
	67	* `created` (DateTime): link creation date time.
	68	* `updated` (DateTime): last modification date time.
2f4df753	69
91a21c27	70	Small hashes are used to make a link to an entry in Shaarli. They are unique: the date of the item (eg. `20110923_150523`) is hashed with CRC32, then converted to base64 and some characters are replaced. They are always 6 characters longs and use only `A-Z a-z 0-9 - _` and `@`.
	71
	72
	73	## Directory structure
	74
	75	Here is the directory structure of Shaarli and the purpose of the different files:
	76
	77	```bash
	78	index.php # Main program
	79	application/ # Shaarli classes
	80	├── LinkDB.php
	81
	82	...
	83
	84	└── Utils.php
	85	tests/ # Shaarli unitary & functional tests
	86	├── LinkDBTest.php
	87
	88	...
	89
	90	├── utils # utilities to ease testing
	91	│ └── ReferenceLinkDB.php
	92	└── UtilsTest.php
	93	assets/
	94	├── common/ # Assets shared by multiple themes
	95	├── ...
	96	├── default/ # Assets for the default template, before compilation
	97	├── fonts/ # Font files
	98	├── img/ # Images used by the default theme
	99	├── js/ # JavaScript files in ES6 syntax
	100	├── scss/ # SASS files
	101	└── vintage/ # Assets for the vintage template, before compilation
	102	└── ...
	103	COPYING # Shaarli license
	104	inc/ # static assets and 3rd party libraries
	105	└── rain.tpl.class.php # RainTPL templating library
	106	images/ # Images and icons used in Shaarli
	107	data/ # data storage: bookmark database, configuration, logs, banlist...
	108	├── config.json.php # Shaarli configuration (login, password, timezone, title...)
	109	├── datastore.php # Your link database (compressed).
	110	├── ipban.php # IP address ban system data
	111	├── lastupdatecheck.txt # Update check timestamp file
	112	└── log.txt # login/IPban log.
	113	tpl/ # RainTPL templates for Shaarli. They are used to build the pages.
	114	├── default/ # Default Shaarli theme
	115	├── fonts/ # Font files
	116	├── img/ # Images
	117	├── js/ # JavaScript files compiled by Babel and compatible with all browsers
	118	├── css/ # CSS files compiled with SASS
	119	└── vintage/ # Legacy Shaarli theme
	120	└── ...
	121	cache/ # thumbnails cache
	122	# This directory is automatically created. You can erase it anytime you want.
	123	tmp/ # Temporary directory for compiled RainTPL templates.
	124	# This directory is automatically created. You can erase it anytime you want.
	125	vendor/ # Third-party dependencies. This directory is created by Composer
	126	```
	127
	128	Shaarli needs read access to:
	129
	130	- the root index.php file
	131	- the `application/`, `plugins/` and `inc/` directories (recursively)
	132
	133	Shaarli needs read/write access to the `cache/`, `data/`, `pagecache/`, and `tmp/` directories
134
135
136	## Automation
137
138	A [`Makefile`](https://github.com/shaarli/Shaarli/blob/master/Makefile) is available to perform project-related operations:
139
140	- [Static analysis](#Static-analysis) - check that the code is compliant to PHP conventions
141	- [Unit tests](#Unit-tests) - ensure there are no regressions introduced by new commits
142	- Documentation - generate a local HTML copy of the markdown documentation
143
144	### Continuous Integration
145
146	[Travis CI](http://docs.travis-ci.com/) is a Continuous Integration build server, that runs a build:
147
148	- each time a commit is merged to the mainline (`master` branch)
149	- each time a Pull Request is submitted or updated
150
151	After all jobs have finished, Travis returns the results to GitHub:
152
153	- a status icon represents the result for the `master` branch: [![](https://api.travis-ci.org/shaarli/Shaarli.svg)](https://travis-ci.org/shaarli/Shaarli)
154	- Pull Requests are updated with the Travis build result.
155
156	See [`.travis.yml`](https://github.com/shaarli/Shaarli/blob/master/.travis.yml).
157
158
159	### Documentation
160
161	[mkdocs](https://www.mkdocs.org/) is used to convert markdown documentation to HTML pages. The [public documentation](https://shaarli.readthedocs.io/en/master/) website is rendered and hosted by [readthedocs.org](https://readthedocs.org/). A copy of the documentation is also included in prebuilt [release archives](https://github.com/shaarli/Shaarli/releases) (`doc/html/` path in your Shaarli installation). To generate the HTML documentation locally, install a recent version of Python `setuptools` and run `make doc`.
162
163
164	## Static analysis
165
2f4df753	166	Patches should try to stick to the [PHP Standard Recommendations](http://www.php-fig.org/psr/) (PSR), and must follow:
91a21c27	167
	168	- [PSR-1](http://www.php-fig.org/psr/psr-1/) - Basic Coding Standard
	169	- [PSR-2](http://www.php-fig.org/psr/psr-2/) - Coding Style Guide
2f4df753	170	- [PSR-12](http://www.php-fig.org/psr/psr-12/) - Extended Coding Style Guide
91a21c27	171
2f4df753	172	These are enforced on pull requests using our Continuous Integration tools.
91a21c27	173
	174	Work in progress: Static analysis is currently being discussed here: in [#95 - Fix coding style (static analysis)](https://github.com/shaarli/Shaarli/issues/95), [#130 - Continuous Integration tools & features](https://github.com/shaarli/Shaarli/issues/130)
	175
	176	Static analysis tools can be installed with Composer, and used through Shaarli's [Makefile](https://github.com/shaarli/Shaarli/blob/master/Makefile).
	177
	178	For an overview of the available features, see:
	179
	180	- [Code quality: Makefile to run static code checkers](https://github.com/shaarli/Shaarli/pull/124) (#124)
	181	- [Run PHPCS against different coding standards](https://github.com/shaarli/Shaarli/pull/276) (#276)