From 91a21c272960889afd4eaa431a3d29b7785b6efc Mon Sep 17 00:00:00 2001 From: nodiscc Date: Sat, 16 May 2020 12:54:51 +0200 Subject: **General rewording, proof-reading, deduplication, shortening, reordering, simplification, cleanup/formatting/standardization** - standardize page names, rework documentation structure, update TOC - use same example paths everywhere - level 1 titles on all pages - fix broken links - .md suffix on all page links (works both from readthedocs and github repository views) **Server:** A full and concise installation guide with examples is a frequent request. The documentation should provide such a guide for basic installation needs, while explaining alternative/advanced configuration at the end. Links to reference guides and documentation should be used more frequently to avoid recommending an outdated or excessively complex configuration. - server: move most server-related info to server-configuration.md, cleanup/shorten - server: update list of php dependencies/libraries, link to composer.json - server: installation: support 3 install methods (from release zip, from sources, using docker) - server: installation: use rsync instead of mv as mv results will change depending of taget directory already existing or not - server: add example/basic usage of certbot - server, upgrade, installation: update file permissions setup, use sudo for upgrade operations in webserver document root - server: apache: add comments to configuration, fix and factorize file permissions setup, set cache-control header, deny access to dotfiles, add missing apache config steps, add http->https redirect example - server: nginx: refactor nginx configuration, add comments, DO log access to denied/protected files - server: add links to MDN for x-forwarded-* http headers explanation, cleanup/clarify robots.txt and crawlers section - server: bump file upload size limit to 100MB we have reports of bookmark exports weighing +40MB - i have a 13MB one here - server: simplify phpinfo documentation - server: move backup and restore information to dedicated page - docker: move all docker docs to Docker.md, simplify/ docker setup, add docker-compose.yml example, replace docker-101 with docker cheatsheet - troubleshooting: move all troubleshooting documentation to troubleshooting.md **Usage:** - index: add getting started section on index page - features/usage: move all usage-related documentation to usage.md, add links from the main feature list to corresponding usage docs, clarify/reword features list - shaarli configuration: add note about configuring from web interface **Removed:** - remove obsolete/orphan images - remove obsolete shaarchiver example - remove outdated "decode datastore content" snippet **Development:** - development: move development-related docs (static analysis, CI, unit tests, 3rd party libs, link structure/directory, guidelines, security....) to dev/ directory - development: Merge several pages to development.md - **Breaking change?:** remove mentions of 'stable' branch, switch to new branch/release model (master=latest commit, release=latest tag) - **Breaking change?:** refer to base sharing unit as "Shaare" everywhere (TODO: reflect changes in the code?) doc: update featues list/link to usage.md for details - development: directory structure: add note about required file permissions - .travis-ci.yml: add comments - .htaccess: add comment --- doc/md/dev/Development.md | 179 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 179 insertions(+) create mode 100644 doc/md/dev/Development.md (limited to 'doc/md/dev/Development.md') diff --git a/doc/md/dev/Development.md b/doc/md/dev/Development.md new file mode 100644 index 00000000..5c085e03 --- /dev/null +++ b/doc/md/dev/Development.md @@ -0,0 +1,179 @@ +# Development + +Please read [Contributing to Shaarli](https://github.com/shaarli/Shaarli/tree/master/CONTRIBUTING.md) + +## Guidelines + + +- [Unit tests](Unit-tests) +- Javascript linting - Shaarli uses [Airbnb JavaScript Style Guide](https://github.com/airbnb/javascript). +Run `make eslint` to check JS style. +- [GnuPG signature](GnuPG-signature) for tags/releases + + +## Third-party libraries + +CSS: + +- Yahoo UI [CSS Reset](http://yuilibrary.com/yui/docs/cssreset/) - standardize cross-browser rendering + +Javascript: + +- [Awesomeplete](https://leaverou.github.io/awesomplete/) ([GitHub](https://github.com/LeaVerou/awesomplete)) - autocompletion in input forms +- [bLazy](http://dinbror.dk/blazy/) ([GitHub](https://github.com/dinbror/blazy)) - lazy loading for thumbnails +- [qr.js](http://neocotic.com/qr.js/) ([GitHub](https://github.com/neocotic/qr.js)) - QR code generation + +PHP (managed through [`composer.json`](https://github.com/shaarli/Shaarli/blob/master/composer.json)): + +- [RainTPL](https://github.com/rainphp/raintpl) - HTML templating for PHP +- [`shaarli/netscape-bookmark-parser`](https://packagist.org/packages/shaarli/netscape-bookmark-parser) - Import bookmarks from Netscape files +- [`erusev/parsedown`](https://packagist.org/packages/erusev/parsedown) - Parse MarkDown syntax for the MarkDown plugin +- [`slim/slim`](https://packagist.org/packages/slim/slim) - Handle routes and middleware for the REST API +- [`ArthurHoaro/web-thumbnailer`](https://github.com/ArthurHoaro/web-thumbnailer) - PHP library which will retrieve a thumbnail for any given URL +- [`pubsubhubbub/publisher`](https://github.com/pubsubhubbub/php-publisher) - A PubSubHubbub publisher module for PHP. +- [`gettext/gettext`](https://github.com/php-gettext/Gettext) - PHP library to collect and manipulate gettext (.po, .mo, .php, .json, etc) + + +## Security + +- The password is salted, hashed and stored in the data subdirectory, in a PHP file, and protected by htaccess. Even if the webserver does not support htaccess, the hash is not readable by URL. Even if the .php file is stolen, the password cannot deduced from the hash. The salt prevents rainbow-tables attacks. +- Directories are protected using `.htaccess` files +- Forms are protected against [XSRF](http://en.wikipedia.org/wiki/Cross-site_request_forgery): + - Forms which act on data (save,delete…) contain a token generated by the server. + - Any posted form which does not contain a valid token is rejected. + - Any token can only be used once. + - Tokens are attached to the session and cannot be reused in another session. +- Sessions automatically expire after 60 minutes. +- Sessions are protected against hijacking: the session ID cannot be used from a different IP address. +- Links are stored as an associative array which is serialized, compressed (with deflate), base64-encoded and saved as a comment in a `.php` file - even if the server does not support `.htaccess` files, the data file will still not be readable by URL. +- Bruteforce protection: Successful and failed login attempts are logged - IP bans are enforced after a configurable amount of failures. Logs can also be used consumed by [fail2ban](../Server-configuration.md#fail2ban) +- A pop-up notification is shown when a new release is available. + +## Link structure + +Every link available through the `LinkDB` object is represented as an array +containing the following fields: + + * `id` (integer): Unique identifier. + * `title` (string): Title of the link. + * `url` (string): URL of the link. Used for displayable links (without redirector, url encoding, etc.). + Can be absolute or relative for Notes. + * `real_url` (string): Real destination URL, can be redirected, encoded, etc. + * `shorturl` (string): Permalink small hash. + * `description` (string): Link text description. + * `private` (boolean): whether the link is private or not. + * `tags` (string): all link tags separated by a single space + * `thumbnail` (string|boolean): relative path of the thumbnail cache file, or false if there isn't any. + * `created` (DateTime): link creation date time. + * `updated` (DateTime): last modification date time. + +Small hashes are used to make a link to an entry in Shaarli. They are unique: the date of the item (eg. `20110923_150523`) is hashed with CRC32, then converted to base64 and some characters are replaced. They are always 6 characters longs and use only `A-Z a-z 0-9 - _` and `@`. + + +## Directory structure + +Here is the directory structure of Shaarli and the purpose of the different files: + +```bash + index.php # Main program + application/ # Shaarli classes + ├── LinkDB.php + + ... + + └── Utils.php + tests/ # Shaarli unitary & functional tests + ├── LinkDBTest.php + + ... + + ├── utils # utilities to ease testing + │ └── ReferenceLinkDB.php + └── UtilsTest.php + assets/ + ├── common/ # Assets shared by multiple themes + ├── ... + ├── default/ # Assets for the default template, before compilation + ├── fonts/ # Font files + ├── img/ # Images used by the default theme + ├── js/ # JavaScript files in ES6 syntax + ├── scss/ # SASS files + └── vintage/ # Assets for the vintage template, before compilation + └── ... + COPYING # Shaarli license + inc/ # static assets and 3rd party libraries + └── rain.tpl.class.php # RainTPL templating library + images/ # Images and icons used in Shaarli + data/ # data storage: bookmark database, configuration, logs, banlist... + ├── config.json.php # Shaarli configuration (login, password, timezone, title...) + ├── datastore.php # Your link database (compressed). + ├── ipban.php # IP address ban system data + ├── lastupdatecheck.txt # Update check timestamp file + └── log.txt # login/IPban log. + tpl/ # RainTPL templates for Shaarli. They are used to build the pages. + ├── default/ # Default Shaarli theme + ├── fonts/ # Font files + ├── img/ # Images + ├── js/ # JavaScript files compiled by Babel and compatible with all browsers + ├── css/ # CSS files compiled with SASS + └── vintage/ # Legacy Shaarli theme + └── ... + cache/ # thumbnails cache + # This directory is automatically created. You can erase it anytime you want. + tmp/ # Temporary directory for compiled RainTPL templates. + # This directory is automatically created. You can erase it anytime you want. + vendor/ # Third-party dependencies. This directory is created by Composer +``` + +Shaarli needs read access to: + +- the root index.php file +- the `application/`, `plugins/` and `inc/` directories (recursively) + +Shaarli needs read/write access to the `cache/`, `data/`, `pagecache/`, and `tmp/` directories + + +## Automation + +A [`Makefile`](https://github.com/shaarli/Shaarli/blob/master/Makefile) is available to perform project-related operations: + +- [Static analysis](#Static-analysis) - check that the code is compliant to PHP conventions +- [Unit tests](#Unit-tests) - ensure there are no regressions introduced by new commits +- Documentation - generate a local HTML copy of the markdown documentation + +### Continuous Integration + +[Travis CI](http://docs.travis-ci.com/) is a Continuous Integration build server, that runs a build: + +- each time a commit is merged to the mainline (`master` branch) +- each time a Pull Request is submitted or updated + +After all jobs have finished, Travis returns the results to GitHub: + +- a status icon represents the result for the `master` branch: [![](https://api.travis-ci.org/shaarli/Shaarli.svg)](https://travis-ci.org/shaarli/Shaarli) +- Pull Requests are updated with the Travis build result. + +See [`.travis.yml`](https://github.com/shaarli/Shaarli/blob/master/.travis.yml). + + +### Documentation + +[mkdocs](https://www.mkdocs.org/) is used to convert markdown documentation to HTML pages. The [public documentation](https://shaarli.readthedocs.io/en/master/) website is rendered and hosted by [readthedocs.org](https://readthedocs.org/). A copy of the documentation is also included in prebuilt [release archives](https://github.com/shaarli/Shaarli/releases) (`doc/html/` path in your Shaarli installation). To generate the HTML documentation locally, install a recent version of Python `setuptools` and run `make doc`. + + +## Static analysis + +Patches should try to stick to the [PHP Standard Recommendations](http://www.php-fig.org/psr/) (PSR), especially: + +- [PSR-1](http://www.php-fig.org/psr/psr-1/) - Basic Coding Standard +- [PSR-2](http://www.php-fig.org/psr/psr-2/) - Coding Style Guide + + +**Work in progress:** Static analysis is currently being discussed here: in [#95 - Fix coding style (static analysis)](https://github.com/shaarli/Shaarli/issues/95), [#130 - Continuous Integration tools & features](https://github.com/shaarli/Shaarli/issues/130) + +Static analysis tools can be installed with Composer, and used through Shaarli's [Makefile](https://github.com/shaarli/Shaarli/blob/master/Makefile). + +For an overview of the available features, see: + +- [Code quality: Makefile to run static code checkers](https://github.com/shaarli/Shaarli/pull/124) (#124) +- [Run PHPCS against different coding standards](https://github.com/shaarli/Shaarli/pull/276) (#276) -- cgit v1.2.3