]> git.immae.eu Git - github/Chocobozzz/PeerTube.git/commit
Resumable video uploads (#3933)
authorkontrollanten <6680299+kontrollanten@users.noreply.github.com>
Mon, 10 May 2021 09:13:41 +0000 (11:13 +0200)
committerGitHub <noreply@github.com>
Mon, 10 May 2021 09:13:41 +0000 (11:13 +0200)
commitf6d6e7f861189a4446f406efb775a29688764b48
treec3dda9958c3f189d4c39e8743c738d8c1fef4c2d
parentd29ced1a8582d99b776f664475a157adcf555d98
Resumable video uploads (#3933)

* WIP: resumable video uploads

relates to #324

* fix review comments

* video upload: error handling

* fix audio upload

* fixes after self review

* Update server/controllers/api/videos/index.ts

Co-authored-by: Rigel Kent <par@rigelk.eu>
* Update server/middlewares/validators/videos/videos.ts

Co-authored-by: Rigel Kent <par@rigelk.eu>
* Update server/controllers/api/videos/index.ts

Co-authored-by: Rigel Kent <par@rigelk.eu>
* update after code review

* refactor upload route

- restore multipart upload route
- move resumable to dedicated upload-resumable route
- move checks to middleware
- do not leak internal fs structure in response

* fix yarn.lock upon rebase

* factorize addVideo for reuse in both endpoints

* add resumable upload API to openapi spec

* add initial test and test helper for resumable upload

* typings for videoAddResumable middleware

* avoid including aws and google packages via node-uploadx, by only including uploadx/core

* rename ex-isAudioBg to more explicit name mentioning it is a preview file for audio

* add video-upload-tmp-folder-cleaner job

* stronger typing of video upload middleware

* reduce dependency to @uploadx/core

* add audio upload test

* refactor resumable uploads cleanup from job to scheduler

* refactor resumable uploads scheduler to compare to last execution time

* make resumable upload validator to always cleanup on failure

* move legacy upload request building outside of uploadVideo test helper

* filter upload-resumable middlewares down to POST, PUT, DELETE

also begin to type metadata

* merge add duration functions

* stronger typings and documentation for uploadx behaviour, move init validator up

* refactor(client/video-edit): options > uploadxOptions

* refactor(client/video-edit): remove obsolete else

* scheduler/remove-dangling-resum: rename tag

* refactor(server/video): add UploadVideoFiles type

* refactor(mw/validators): restructure eslint disable

* refactor(mw/validators/videos): rename import

* refactor(client/vid-upload): rename html elem id

* refactor(sched/remove-dangl): move fn to method

* refactor(mw/async): add method typing

* refactor(mw/vali/video): double quote > single

* refactor(server/upload-resum): express use > all

* proper http methud enum server/middlewares/async.ts

* properly type http methods

* factorize common video upload validation steps

* add check for maximum partially uploaded file size

* fix audioBg use

* fix extname(filename) in addVideo

* document parameters for uploadx's resumable protocol

* clear META files in scheduler

* last audio refactor before cramming preview in the initial POST form data

* refactor as mulitpart/form-data initial post request

this allows preview/thumbnail uploads alongside the initial request,
and cleans up the upload form

* Add more tests for resumable uploads

* Refactor remove dangling resumable uploads

* Prepare changelog

* Add more resumable upload tests

* Remove user quota check for resumable uploads

* Fix upload error handler

* Update nginx template for upload-resumable

* Cleanup comment

* Remove unused express methods

* Prefer to use got instead of raw http

* Don't retry on error 500

Co-authored-by: Rigel Kent <par@rigelk.eu>
Co-authored-by: Rigel Kent <sendmemail@rigelk.eu>
Co-authored-by: Chocobozzz <me@florianbigard.com>
46 files changed:
client/package.json
client/src/app/+my-account/my-account-settings/my-account-settings.component.ts
client/src/app/+my-library/+my-video-channels/my-video-channel-update.component.ts
client/src/app/+videos/+video-edit/video-add-components/uploaderx-form-data.ts [new file with mode: 0644]
client/src/app/+videos/+video-edit/video-add-components/video-upload.component.html
client/src/app/+videos/+video-edit/video-add-components/video-upload.component.scss
client/src/app/+videos/+video-edit/video-add-components/video-upload.component.ts
client/src/app/+videos/+video-edit/video-add.module.ts
client/src/app/helpers/utils.ts
client/yarn.lock
package.json
server.ts
server/controllers/api/server/debug.ts
server/controllers/api/videos/index.ts
server/helpers/custom-validators/misc.ts
server/helpers/custom-validators/videos.ts
server/helpers/express-utils.ts
server/helpers/upload.ts [new file with mode: 0644]
server/helpers/utils.ts
server/initializers/constants.ts
server/initializers/installer.ts
server/lib/moderation.ts
server/lib/schedulers/remove-dangling-resumable-uploads-scheduler.ts [new file with mode: 0644]
server/lib/video.ts
server/middlewares/async.ts
server/middlewares/validators/videos/videos.ts
server/tests/api/check-params/index.ts
server/tests/api/check-params/upload-quota.ts [new file with mode: 0644]
server/tests/api/check-params/users.ts
server/tests/api/check-params/videos.ts
server/tests/api/videos/index.ts
server/tests/api/videos/multiple-servers.ts
server/tests/api/videos/resumable-upload.ts [new file with mode: 0644]
server/tests/api/videos/single-server.ts
server/tests/api/videos/video-transcoder.ts
server/typings/express/index.d.ts
shared/core-utils/miscs/http-methods.ts [new file with mode: 0644]
shared/core-utils/miscs/index.ts
shared/extra-utils/server/debug.ts
shared/extra-utils/server/servers.ts
shared/extra-utils/videos/video-channels.ts
shared/extra-utils/videos/videos.ts
shared/models/server/debug.model.ts
support/doc/api/openapi.yaml
support/nginx/peertube
yarn.lock