aboutsummaryrefslogtreecommitdiffhomepage
diff options
context:
space:
mode:
authorChocobozzz <florian.bigard@gmail.com>2017-11-27 16:06:13 +0100
committerChocobozzz <florian.bigard@gmail.com>2017-11-27 19:43:01 +0100
commitfcaf1e0aa84213a1b1f1b1a44a3276eae35ebe70 (patch)
tree23157ef819c5dba040a576b8dae0dee4eba6e85f
parent4e50b6a1c9a3eb261e04ede73241648e6edf21d6 (diff)
downloadPeerTube-fcaf1e0aa84213a1b1f1b1a44a3276eae35ebe70.tar.gz
PeerTube-fcaf1e0aa84213a1b1f1b1a44a3276eae35ebe70.tar.zst
PeerTube-fcaf1e0aa84213a1b1f1b1a44a3276eae35ebe70.zip
Update readme, architecture
-rw-r--r--ARCHITECTURE.md89
-rw-r--r--FAQ.md7
-rw-r--r--README.md60
-rw-r--r--support/doc/server/code.md6
4 files changed, 37 insertions, 125 deletions
diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md
index ebcffd6cb..ac62fa4b7 100644
--- a/ARCHITECTURE.md
+++ b/ARCHITECTURE.md
@@ -1,83 +1,26 @@
1# Protocol (WIP, help welcome) 1# Architecture
2 2
3## Vocabulary 3## Vocabulary
4 4
5 - **Network:** several servers communicating each others with this software compose a network 5 - **Fediverse:** several servers following each others
6 - **Pod:** a server of the network (inspired from Diaspora, no really signification) 6 - **Instance:** a server which runs PeerTube in the fediverse
7 - **Friend:** a pod that communicates with yours 7 - **Origin instance:** the instance on which the video was uploaded and which is seeding (WebSeed protocol) the video
8 - **Origin pod:** the pod on which the video was uploaded and which is seeding (throught WebSeed protocol) the video 8 - **Following:** the action of a PeerTube instance which will follow another instance (subscribe to its videos)
9 - **Make friend:** the action of a server which will join a network (and so become friend with all pods that compose this network)
10 9
11## Base 10## Base
12 11
13### The first run: join a network and make friends
14 * The server generates a RSA key
15 * If the server administrator wants to join a network, he just has to make an http request to the API
16 * The server joins a network by checking entrypoints (server urls of the targeted network) in the configuration file
17 * If the config file doesn't specify other pods, the network will be composed by this only pod
18 * If the config file specifies one or more pods, the server will ask them the list of the pods in the network.
19 The server will add in its friends list all pods that are in > 50% of all other pods friends list + the pods that are asked for the list. For example if there are the following pods in a network with their following friends list:
20
21 http://pod1.com
22 - http://pod2.com
23 - http://pod3.com
24 - http://pod4.com
25 - http://pod5.com
26
27 http://pod2.com
28 - http://pod3.com
29 - http://pod5.com
30
31 http://pod3.com
32 - http://pod5.com
33
34 It will add: `http://pod1.com`, `http://pod2.com` and `http://pod3.com` because it asks to them the list of their friends. Then, it will add all pods that are in >= 50% of pods friends list so: `http://pod5.com`.
35 * When the friends list is added, the server will present itself to all these friends ("make friend" operation) with the following informations: its **public RSA key** and its **URL**
36 * Then, the pods will slowly share their videos database
37 * All the friends have the initial score of x points which represents the reliability of this friend
38 * If the score reaches 0 the friend is revoked (blacklisted for the future?) and its videos are deleted
39
40### Communications 12### Communications
41 * All the communications between pods are signed, encrypted with a RSA key and a symetric encryption in a POST request (body) 13 * All the communications between the instances are signed with [Linked Data Signatures](https://w3c-dvcg.github.io/ld-signatures/) with the private key of the account that made the action
42 * All the requests are retried if they failed 14 * We use the [ActivityPub](https://www.w3.org/TR/activitypub/) protocol (only server-server for now). Object models could be found in [shared/models/activitypub directory](https://github.com/Chocobozzz/PeerTube/tree/develop/shared/models/activitypub).
43 * The requests are made at regular intervals to all pods of the network with a limit of parallel requests: if there are 1000 pods in the networks, the server won't make more than 10 requests in parallel 15 * All the requests are retried several times if they fail
44 * If there are no informations to send (no video addition/removal), no requests are made
45 * The requests are grouped: for example if the interval is 1 hour and a user$ upload 2 videos there will be only 1 request that contains the 2 videos informations
46 * The requests in the group are ordered: if a user add a video and remove it, we need to transmit these informations to the other pods in the same order
47 * The requests are grouped with a limit: if a user uploads 100 videos at a time, the information could be propagated in a few hours to do not make too big requests
48 * If a grouped request fails the score is decreased by x points
49 * If a grouped request is a success the score is increased by x points
50 * The maximum of points would be defined 16 * The maximum of points would be defined
51 * A pod which receives a request checks if the signature corresponds to the pod it has in its database. Then, it decrypts the body (or ignores it if the signature is not valid) and process the requests in the same order
52
53### Actions on a pod
54 * A pod is a websocket tracker which is responsible for all the video uploaded in it
55 * A pod has an administrator that can add/remove users, make friends and quit friends
56 * A pod has different user accounts that can upload videos
57 * All pods have an index of all videos of the network (name, origin pod url, small description, uploader username, magnet Uri, thumbnail name, created date and the thumbnail file). For example, a test with 1000000 videos (3 tags each) with alphanum characters and the following lengths: name = 50, author = 50, podHost = 25, description = 250, videoExtension = 4, remoteId = 50, infoHash = 50 and tag = 10 has a PostgreSQL size of ~ 2GB with all the useful indexes. To this, we add 1 000 000 thumbnails of 5-15 KB so 15GB maximum
58 17
59 table_name | row_estimate | index | toast | table 18### Instance
60 pod | 983416 | 140 MB | 83 MB | 57 MB 19 * An instance has a websocket tracker which is responsible for all the video uploaded in it
61 author | 1e+06 | 229 MB | 140 MB | 89 MB 20 * An instance has an administrator that can follow other instances
62 tag | 2.96758e+06 | 309 MB | 182 MB | 127 MB 21 * An instance can be configured to follow back automatically
63 video | 1e+06 | 723 MB | 263 MB | 460 MB 22 * An instance can blacklist other instances (only used in "follow back" mode)
64 video_tag | 3e+06 | 316 MB | 212 MB | 104 MB 23 * An instance cannot choose which other instance follow it, but it can decide to **reject all** followers
65 24 * After having uploaded a video, the instance seeds it (WebSeed protocol)
66 25 * If a user wants to watch a video, he asks its instance the magnet URI and the frontend adds the torrent (with WebTorrent), creates the HTML5 video player and streams the file into it
67 * After having uploaded a video, the server seeds it (WebSeed protocol), adds the meta data in its database and makes a secure request to all of its friends
68 * If a user wants to watch a video, he asks its pod the magnetUri and the frontend adds the torrent (with WebTorrent), creates the HTML5 video tag and streams the file into it
69 * A user watching a video seeds it too (BitTorrent) so another user who is watching the same video can get the data from the origin server and the user 1 (etc) 26 * A user watching a video seeds it too (BitTorrent) so another user who is watching the same video can get the data from the origin server and the user 1 (etc)
70
71## Ideas
72
73 * A video could have more information (detailed description etc) that are not sent on other pods. The user who wants to see these informations has to ask its pod:
74 user asks its pod -> user pod asks origin video pod -> origin video pod responds with the informations -> user pod responds to the user (and puts in cache the informations ?). We could extend this scheme with other informations
75 * Redondance: if the origin pod is down, the video is not accessible anymore (no tracker/seeds). We could imagine a redondance between pods that keep seeding the video
76 * Server could transcode the video to lower qualities (cost in CPU and disk space)
77 * Add subtitles to videos
78 * Avoid stocking friends URL schemes (http/https)
79
80## Debate
81
82 * Is an ex-friend should be blacklisted for the future?
83 * Handle API breaks in a network. If a major update breaks the API: we need the entire network update to the same major version. We could specify a limit date (2, 3 weeks after the release?) after which all the updated pod would switch to the new version of the API. The non updated pod will then be ejected of the network because would not implement the new API
diff --git a/FAQ.md b/FAQ.md
index 7d4c1b9b7..7e2b3339c 100644
--- a/FAQ.md
+++ b/FAQ.md
@@ -15,7 +15,7 @@ It is a BitTorrent extension that allow a server to seed a file through HTTP. It
15Not really. Reverse proxies like Nginx handle very well requests of static files. In my tests it can send chunks at 10MB/s without consuming more than 5% of CPU on a very small VPS. 15Not really. Reverse proxies like Nginx handle very well requests of static files. In my tests it can send chunks at 10MB/s without consuming more than 5% of CPU on a very small VPS.
16 16
17 17
18## An index of all videos of the network won't be too large for small servers? 18## An index of all videos of servers you follow won't be too large for small servers?
19 19
20No, 1000000 videos will represent around 2GB on PostgreSQL. It is acceptable for a video platform. 20No, 1000000 videos will represent around 2GB on PostgreSQL. It is acceptable for a video platform.
21 21
@@ -27,5 +27,6 @@ WEBM, MP4 or OGV videos.
27 27
28## I want to change my host or move to HTTPS, how can I do that? 28## I want to change my host or move to HTTPS, how can I do that?
29 29
30If you already have friends you need to quit them. 30If you already have followers, you can't.
31Then, update your configuration and run `NODE_ENV=production npm run update-host` to update the torrent files. 31
32If you don't: update your configuration and run `NODE_ENV=production npm run update-host` to update the torrent files.
diff --git a/README.md b/README.md
index 35d53ee09..68b5dc519 100644
--- a/README.md
+++ b/README.md
@@ -77,30 +77,17 @@ Thanks to [WebTorrent](https://github.com/feross/webtorrent), we can make P2P (t
77 77
78- [X] Frontend 78- [X] Frontend
79 - [X] Angular frontend 79 - [X] Angular frontend
80- [X] Join a network 80- [X] Join the fediverse
81 - [X] Generate a RSA key 81 - [X] Follow other instances
82 - [X] Ask for the friend list of other pods and make friend with them 82 - [X] Unfollow an instance
83 - [X] Get the list of the videos owned by a pod when making friend with it 83 - [X] Get for the followers/following list
84 - [X] Post the list of its own videos when making friend with another pod
85- [X] Quit a network
86- [X] Upload a video 84- [X] Upload a video
87 - [X] Seed the video 85 - [X] Seed the video
88 - [X] Send the meta data to all other friends 86 - [X] Send the meta data with ActivityPub to followers
89- [X] Remove the video 87- [X] Remove the video
90- [X] List the videos 88- [X] List the videos
91- [X] Search a video name (local index) 89- [X] View the video in an HTML5 player with WebTorrent
92- [X] View the video in an HTML5 page with WebTorrent
93- [X] Manage admin account
94 - [X] Connection
95 - [X] Account rights (upload...)
96- [X] Make the network auto sufficient (eject bad pods etc)
97- [X] Validate the prototype (test PeerTube in a real world)
98- [ ] Manage inter pod API breaks
99- [ ] Add "DDOS" security (check if a pod don't send too many requests for example)
100- [X] Admin panel 90- [X] Admin panel
101 - [X] Stats
102 - [X] Friends list
103 - [X] Manage users (create/remove)
104- [X] OpenGraph tags 91- [X] OpenGraph tags
105- [X] OEmbed 92- [X] OEmbed
106- [X] Update video 93- [X] Update video
@@ -109,16 +96,17 @@ Thanks to [WebTorrent](https://github.com/feross/webtorrent), we can make P2P (t
109- [X] Transcoding to different definitions 96- [X] Transcoding to different definitions
110- [X] Download file/torrent 97- [X] Download file/torrent
111- [X] User video bytes quota 98- [X] User video bytes quota
112- [X] User channels 99- [X] User video channels
113- [X] NSFW warnings/settings 100- [X] NSFW warnings/settings
114- [X] Video description in markdown 101- [X] Video description in markdown
115- [X] User roles (administrator, moderator) 102- [X] User roles (administrator, moderator)
116- [X] User registration 103- [X] User registration
117- [X] Video privacy settings (public, unlisted or private) 104- [X] Video privacy settings (public, unlisted or private)
118- [X] Signaling a video to the admin origin pod 105- [X] Signaling a video to the admin origin PeerTube instance
119- [ ] Videos comments 106- [ ] Videos comments
120- [ ] User playlist 107- [ ] User playlist
121- [ ] User subscriptions (by tags, author...) 108- [ ] User subscriptions (by tags, author...)
109- [ ] Add "DDOS" security
122 110
123 111
124## Installation 112## Installation
@@ -194,7 +182,7 @@ The following commands will upgrade the source (according to your current branch
194 182
195### Development 183### Development
196 184
197In this mode, the server will run requests between pods more quickly, the video durations are limited to a few seconds. 185In this mode, the server will run requests between instances more quickly, the video durations are limited to a few seconds.
198 186
199To develop on the server-side (server files are automatically compiled when we modify them and the server restarts automatically too): 187To develop on the server-side (server files are automatically compiled when we modify them and the server restarts automatically too):
200 188
@@ -217,7 +205,7 @@ The API will listen on `localhost:9000` and the frontend on `localhost:3000` (wi
217 $ npm run clean:server:test 205 $ npm run clean:server:test
218 $ npm run play 206 $ npm run play
219 207
220Then you will get access to the three nodes at `http://localhost:900{1,2,3}` with the `root` as username and `test{1,2,3}` for the password. If you call "make friends" on `http://localhost:9002`, the pod 2 and 3 will become friends. Then if you call "make friends" on `http://localhost:9001` it will become friend with the pod 2 and 3 (check the configuration files). Then the pod will communicate with each others. If you add a video on the pod 3 you'll can see it on the pod 1 and 2 :) 208Then you will get access to the three nodes at `http://localhost:900{1,2,3}` with the `root` as username and `test{1,2,3}` for the password.
221 209
222### Other commands 210### Other commands
223 211
@@ -225,10 +213,6 @@ To print all available command run:
225 213
226 $ npm run help 214 $ npm run help
227 215
228## Dockerfile
229
230You can test it inside Docker with the [PeerTube-Docker repository](https://github.com/Chocobozzz/PeerTube-Docker). Moreover it can help you to check how to create an environment with the required dependencies for PeerTube on a GNU/Linux distribution.
231
232## Contributing 216## Contributing
233 217
234See the [contributing guide](https://github.com/Chocobozzz/PeerTube/blob/master/.github/CONTRIBUTING.md). 218See the [contributing guide](https://github.com/Chocobozzz/PeerTube/blob/master/.github/CONTRIBUTING.md).
@@ -245,18 +229,11 @@ See [ARCHITECTURE.md](https://github.com/Chocobozzz/PeerTube/blob/master/ARCHITE
245### Backend 229### Backend
246 230
247 * The backend is a REST API 231 * The backend is a REST API
248 * Servers communicate with each others through it 232 * Servers communicate with each others with [Activity Pub](https://www.w3.org/TR/activitypub/)
249 * A network is composed by servers that communicate between them 233 * Each server has its own users who query it (search videos, where the torrent URI of this specific video is...)
250 * Each server of a network has a list of all other servers of this network 234 * If a user upload a video, the server seeds it and sends the video information (name, short description, torrent URI...) its followers
251 * When a new installed server wants to join a network, it just has to get the servers list through a server that is already in the network and tell "Hi I'm new in the network, communicate with me and share me your servers list please". Then the server will "make friend" with each server of this list
252 * Each server has its own users who query it (search videos, where the torrent URI of this specific video is...)
253 * If a user upload a video, the server seeds it and sends the video information (name, short description, torrent URI...) to each server of the network
254 * Each server has a RSA key to encrypt and sign communications with other servers
255 * A server is a tracker responsible for all the videos uploaded in it 235 * A server is a tracker responsible for all the videos uploaded in it
256 * Even if nobody watches a video, it is seeded by the server (through [WebSeed protocol](http://www.bittorrent.org/beps/bep_0019.html)) where the video was uploaded 236 * Even if nobody watches a video, it is seeded by the server (through [WebSeed protocol](http://www.bittorrent.org/beps/bep_0019.html)) where the video was uploaded
257 * A network can live and evolve by expelling bad pod (with too many downtime for example)
258
259See the ARCHITECTURE.md for more information. Do not hesitate to give your opinion :)
260 237
261Here are some simple schemes: 238Here are some simple schemes:
262 239
@@ -268,13 +245,4 @@ Here are some simple schemes:
268 245
269<img src="https://lutim.cpy.re/pqKm3Q5S.png" alt="Watch a P2P video" /> 246<img src="https://lutim.cpy.re/pqKm3Q5S.png" alt="Watch a P2P video" />
270 247
271<img src="https://lutim.cpy.re/wWVuczBz.png" alt="Join a network" />
272
273<img src="https://lutim.cpy.re/AMo3uP0D.png" alt="Many networks" />
274
275</p> 248</p>
276
277### Frontend
278
279There already is a frontend (Angular) but the backend is a REST API so anybody can build a frontend (Web application, desktop application...).
280The backend uses BitTorrent protocol, so users could use their favorite BitTorrent client to download/play the video with its torrent URI.
diff --git a/support/doc/server/code.md b/support/doc/server/code.md
index b941304ce..f0ed62acc 100644
--- a/support/doc/server/code.md
+++ b/support/doc/server/code.md
@@ -28,7 +28,7 @@ All other server files are in the [server](https://github.com/Chocobozzz/PeerTub
28 |__ initializers -> functions used at the server startup (installer, database, constants...) 28 |__ initializers -> functions used at the server startup (installer, database, constants...)
29 |__ lib -> library function (WebTorrent, OAuth2, friends logic...) 29 |__ lib -> library function (WebTorrent, OAuth2, friends logic...)
30 |__ middlewares -> middlewares for controllers (requests validators, requests pagination...) 30 |__ middlewares -> middlewares for controllers (requests validators, requests pagination...)
31 |__ models -> Sequelize models for each SQL tables (videos, users, pods...) 31 |__ models -> Sequelize models for each SQL tables (videos, users, accounts...)
32 |__ tests -> API tests and real world simulations (to test the decentralized feature...) 32 |__ tests -> API tests and real world simulations (to test the decentralized feature...)
33 33
34 34
@@ -44,11 +44,11 @@ Uses [JavaScript Standard Style](http://standardjs.com/).
44 * Run PostgreSQL and create the database `peertube_dev`. 44 * Run PostgreSQL and create the database `peertube_dev`.
45 * Run `npm run dev:server` to run the server, watch server files modifications and restart it automatically. The server (API + client) listen on `localhost:9000`. 45 * Run `npm run dev:server` to run the server, watch server files modifications and restart it automatically. The server (API + client) listen on `localhost:9000`.
46 46
47The `NODE_ENV=test` is set to speed up communications between pods (see [constants.ts](https://github.com/Chocobozzz/PeerTube/blob/master/server/initializers/constants.ts)). 47The `NODE_ENV=test` is set to speed up communications between instances (see [constants.ts](https://github.com/Chocobozzz/PeerTube/blob/master/server/initializers/constants.ts)).
48 48
49`npm run help` gives you all available commands. 49`npm run help` gives you all available commands.
50 50
51If you want to test the decentralization feature, you can easily run 3 pods by running `npm run play`. The pods password are `test1`, `test2` and `test3`. 51If you want to test the decentralization feature, you can easily run 3 instances by running `npm run play`. The instances password are `test1`, `test2` and `test3`.
52 52
53 53
54## Architecture 54## Architecture