From 3a4992633ee62d5edfbb484d9c6bcb3cf158489d Mon Sep 17 00:00:00 2001 From: Chocobozzz Date: Mon, 31 Jul 2023 14:34:36 +0200 Subject: Migrate server to ESM Sorry for the very big commit that may lead to git log issues and merge conflicts, but it's a major step forward: * Server can be faster at startup because imports() are async and we can easily lazy import big modules * Angular doesn't seem to support ES import (with .js extension), so we had to correctly organize peertube into a monorepo: * Use yarn workspace feature * Use typescript reference projects for dependencies * Shared projects have been moved into "packages", each one is now a node module (with a dedicated package.json/tsconfig.json) * server/tools have been moved into apps/ and is now a dedicated app bundled and published on NPM so users don't have to build peertube cli tools manually * server/tests have been moved into packages/ so we don't compile them every time we want to run the server * Use isolatedModule option: * Had to move from const enum to const (https://www.typescriptlang.org/docs/handbook/enums.html#objects-vs-enums) * Had to explictely specify "type" imports when used in decorators * Prefer tsx (that uses esbuild under the hood) instead of ts-node to load typescript files (tests with mocha or scripts): * To reduce test complexity as esbuild doesn't support decorator metadata, we only test server files that do not import server models * We still build tests files into js files for a faster CI * Remove unmaintained peertube CLI import script * Removed some barrels to speed up execution (less imports) --- support/doc/development/lib.md | 2 +- support/doc/development/localization.md | 2 +- support/doc/development/monitoring.md | 4 +- support/doc/development/server.md | 66 ++++++++++++++++----------------- support/doc/development/tests.md | 15 +++----- 5 files changed, 43 insertions(+), 46 deletions(-) (limited to 'support/doc/development') diff --git a/support/doc/development/lib.md b/support/doc/development/lib.md index 25fe3068e..1ea09f2bc 100644 --- a/support/doc/development/lib.md +++ b/support/doc/development/lib.md @@ -18,7 +18,7 @@ The complete types package is generated via: ``` npm run generate-types-package 4.x.x -cd packages/types/dist +cd packages/types-generator/dist npm publish --access=public ``` diff --git a/support/doc/development/localization.md b/support/doc/development/localization.md index a38ed6f55..4aca9b18b 100644 --- a/support/doc/development/localization.md +++ b/support/doc/development/localization.md @@ -26,7 +26,7 @@ Nothing to do here, Github will automatically send a webhook to Weblate that wil ## Support a new language - * Add it to [/shared/models/i18n/i18n.ts](/shared/models/i18n/i18n.ts) + * Add it to [/packages/models/i18n/i18n.ts](/packages/models/i18n/i18n.ts) * Add it to [/scripts/build/client.sh](/scripts/build/client.sh) * Add it to [/client/angular.json](/client/angular.json) * Add it to [/scripts/i18n/update.sh](/scripts/i18n/update.sh) diff --git a/support/doc/development/monitoring.md b/support/doc/development/monitoring.md index 93fd1403e..023be9e92 100644 --- a/support/doc/development/monitoring.md +++ b/support/doc/development/monitoring.md @@ -13,11 +13,11 @@ npm run build -- --analyze-bundle && npm run client-report To benchmark the REST API and save result in `benchmark.json`: ``` -node dist/scripts/benchmark.js -o benchmark.json +npm run benchmark-server -- -o benchmark.json ``` You can also grep on a specific test: ``` -node dist/scripts/benchmark.js --grep homepage +npm run benchmark-server -- --grep homepage ``` diff --git a/support/doc/development/server.md b/support/doc/development/server.md index 7a9fa571f..5c83af704 100644 --- a/support/doc/development/server.md +++ b/support/doc/development/server.md @@ -1,11 +1,11 @@ -# Server code + # Server code ## Database model typing Sequelize models contain optional fields corresponding to table joins. For example, `VideoModel` has a `VideoChannel?: VideoChannelModel` field. It can be filled if the SQL query joined with the `videoChannel` table or empty if not. It can be difficult in TypeScript to understand if a function argument expects associations to be filled or not. -To improve clarity and reduce bugs, PeerTube defines multiple versions of a database model depending on its associations in `server/types/models/`. +To improve clarity and reduce bugs, PeerTube defines multiple versions of a database model depending on its associations in `server/server/types/models/`. These models start with `M` and by default do not include any association. `MVideo` for example corresponds to `VideoModel` without any association, where `VideoChannel` attribute doesn't exist. On the other hand, `MVideoWithChannel` is a `MVideo` that has a `VideoChannel` field. This way, a function that accepts `video: MVideoWithChannel` argument expects a video with channel populated. Main PeerTube code should never use `...Model` (`VideoModel`) database type, but always `M...` instead (`MVideo`, `MVideoChannel` etc). ## Add a new feature walkthrough @@ -16,67 +16,67 @@ Some of these may be optional (for example your new endpoint may not need to sen * Configuration: - Add you new configuration key in `config/default.yaml` and `config/production.yaml` - If you configuration needs to be different in dev or tests environments, also update `config/dev.yaml` and `config/test.yaml` - - Load your configuration in `server/initializers/config.ts` - - Check new configuration keys are set in `server/initializers/checker-before-init.ts` - - You can also ensure configuration consistency in `server/initializers/checker-after-init.ts` + - Load your configuration in `server/server/initializers/config.ts` + - Check new configuration keys are set in `server/server/initializers/checker-before-init.ts` + - You can also ensure configuration consistency in `server/server/initializers/checker-after-init.ts` - If you want your configuration to be available in the client: - + Add your field in `shared/models/server/server-config.model.ts` - + Update `server/lib/server-config-manager.ts` to include your new configuration + + Add your field in `packages/models/server/server/server-config.model.ts` + + Update `server/server/lib/server-config-manager.ts` to include your new configuration - If you want your configuration to be updatable by the web admin in the client: - + Add your field in `shared/models/server/custom-config.model.ts` - + Add the configuration to the config object in the `server/controllers/api/config.ts` controller + + Add your field in `packages/models/server/server/custom-config.model.ts` + + Add the configuration to the config object in the `server/server/controllers/api/config.ts` controller * Controllers: - Create the controller file and fill it with your REST API routes - Import and use your controller in the parent controller * Middlewares: - - Create your validator middleware in `server/middlewares/validators` that will be used by your controllers - - Add your new middleware file `server/middlewares/validators/index.ts` so it's easier to import - - Create the entry in `server/types/express.d.ts` to attach the database model loaded by your middleware to the express response + - Create your validator middleware in `server/server/middlewares/validators` that will be used by your controllers + - Add your new middleware file `server/server/middlewares/validators/index.ts` so it's easier to import + - Create the entry in `server/server/types/express.d.ts` to attach the database model loaded by your middleware to the express response * Validators: - - Create your validators that will be used by your middlewares in `server/helpers/custom-validators` + - Create your validators that will be used by your middlewares in `server/server/helpers/custom-validators` * Typescript models: - - Create the API models (request parameters or response) in `shared/models` + - Create the API models (request parameters or response) in `packages/models` - Add your models in `index.ts` of current directory to facilitate the imports * Sequelize model (BDD): - If you need to create a new table: - + Create the Sequelize model in `server/models/`: + + Create the Sequelize model in `server/server/models/`: * Create the `@Column` * Add some indexes if you need * Create static methods to load a specific from the database `loadBy...` * Create static methods to load a list of models from the database `listBy...` * Create the instance method `toFormattedJSON` that creates the JSON to send to the REST API from the model - + Add your new Sequelize model to `server/initializers/database.ts` - + Create a new file in `server/types` to define multiple versions of your Sequelize model depending on database associations - + Add this new file to `server/types/*/index.ts` to facilitate the imports + + Add your new Sequelize model to `server/server/initializers/database.ts` + + Create a new file in `server/server/types` to define multiple versions of your Sequelize model depending on database associations + + Add this new file to `server/server/types/*/index.ts` to facilitate the imports + Create database migrations: - * Create the migration file in `server/initializers/migrations` using raw SQL (copy the same SQL query as at PeerTube startup) - * Update `LAST_MIGRATION_VERSION` in `server/initializers/constants.ts` + * Create the migration file in `server/server/initializers/migrations` using raw SQL (copy the same SQL query as at PeerTube startup) + * Update `LAST_MIGRATION_VERSION` in `server/server/initializers/constants.ts` - If updating database schema (adding/removing/renaming a column): - + Update the sequelize models in `server/models/` + + Update the sequelize models in `server/server/models/` + Add migrations: * Create the migration file in `initializers/migrations` using Sequelize Query Interface (`.addColumn`, `.dropTable`, `.changeColumn`) - * Update `LAST_MIGRATION_VERSION` in `server/initializers/constants.ts` + * Update `LAST_MIGRATION_VERSION` in `server/server/initializers/constants.ts` * Notifications: - - Create the new notification model in `shared/models/users/user-notification.model.ts` - - Create the notification logic in `server/lib/notifier/shared`: + - Create the new notification model in `packages/models/users/user-notification.model.ts` + - Create the notification logic in `server/server/lib/notifier/shared`: + Email subject has a common prefix (defined by the admin in PeerTube configuration) - - Add your notification to `server/lib/notifier/notifier.ts` - - Create the email template in `server/lib/emails`: + - Add your notification to `server/server/lib/notifier/notifier.ts` + - Create the email template in `server/server/lib/emails`: + A text version is automatically generated from the HTML + The template usually extends `../common/grettings` that already says "Hi" and "Cheers". You just have to write the title and the content blocks that will be inserted in the appropriate places in the HTML template - If you need to associate a new table with `userNotification`: + Associate the new table in `UserNotificationModel` (don't forget the index) - + Add the object property in the API model definition (`shared/models/users/user-notification.model.ts`) + + Add the object property in the API model definition (`packages/models/users/user-notification.model.ts`) + Add the object in `UserNotificationModel.toFormattedJSON` + Handle this new notification type in client (`UserNotificationsComponent`) + Handle the new object property in client model (`UserNotification`) * Tests: - - Create your command class in `shared/server-commands/` that will wrap HTTP requests to your new endpoint + - Create your command class in `packages/server-commands/` that will wrap HTTP requests to your new endpoint - Add your command file in `index.ts` of current directory - - Instantiate your command class in `shared/server-commands/server/server.ts` - - Create your test file in `server/tests/api/check-params` to test middleware validators/authentification/user rights (offensive tests) - - Add it to `server/tests/api/check-params/index.ts` - - Create your test file in `server/tests/api` to test your new endpoints + - Instantiate your command class in `packages/server-commands/server/server/server.ts` + - Create your test file in `server/server/tests/api/check-params` to test middleware validators/authentification/user rights (offensive tests) + - Add it to `server/server/tests/api/check-params/index.ts` + - Create your test file in `server/server/tests/api` to test your new endpoints - Add it to `index.ts` of current directory - - Add your notification test in `server/tests/api/notifications` + - Add your notification test in `server/server/tests/api/notifications` * Update REST API documentation in `support/doc/api/openapi.yaml` diff --git a/support/doc/development/tests.md b/support/doc/development/tests.md index 1c2589c8a..2e4c6ff6a 100644 --- a/support/doc/development/tests.md +++ b/support/doc/development/tests.md @@ -8,7 +8,7 @@ Prepare PostgreSQL user so PeerTube can delete/create the test databases: sudo -u postgres createuser you_username --createdb --superuser ``` -Prepare databases: +Prepare the databases: ```bash npm run clean:server:test @@ -45,22 +45,19 @@ sudo apt-get install parallel libimage-exiftool-perl ### Test -To run all test suites: +To run all test suites (can be long!): ```bash npm run test # See scripts/test.sh to run a particular suite ``` -Most of tests can be run using: +To run a specific test: ```bash -TS_NODE_TRANSPILE_ONLY=true npm run mocha -- --timeout 30000 --exit -r ts-node/register -r tsconfig-paths/register --bail server/tests/api/videos/video-transcoder.ts -``` - -`server/tests/api/activitypub` tests will need different options: +npm run mocha -- --exit --bail packages/tests/src/your-test.ts -``` -TS_NODE_FILES=true mocha -- --timeout 30000 --exit -r ts-node/register -r tsconfig-paths/register --bail server/tests/api/activitypub/security.ts +# For example +npm run mocha -- --exit --bail packages/tests/src/api/videos/single-server.ts ``` ### Configuration -- cgit v1.2.3