Commit Graph

217 Commits (0cc253c9719b1e361d33d62adcef4c292cc98be9)

Author SHA1 Message Date
Rigel Kent 76148b27f7 refactor API errors to standard error format 2021-06-02 16:57:07 +02:00
kontrollanten f6d6e7f861
Resumable video uploads (#3933)
* WIP: resumable video uploads

relates to #324

* fix review comments

* video upload: error handling

* fix audio upload

* fixes after self review

* Update server/controllers/api/videos/index.ts

Co-authored-by: Rigel Kent <par@rigelk.eu>

* Update server/middlewares/validators/videos/videos.ts

Co-authored-by: Rigel Kent <par@rigelk.eu>

* Update server/controllers/api/videos/index.ts

Co-authored-by: Rigel Kent <par@rigelk.eu>

* update after code review

* refactor upload route

- restore multipart upload route
- move resumable to dedicated upload-resumable route
- move checks to middleware
- do not leak internal fs structure in response

* fix yarn.lock upon rebase

* factorize addVideo for reuse in both endpoints

* add resumable upload API to openapi spec

* add initial test and test helper for resumable upload

* typings for videoAddResumable middleware

* avoid including aws and google packages via node-uploadx, by only including uploadx/core

* rename ex-isAudioBg to more explicit name mentioning it is a preview file for audio

* add video-upload-tmp-folder-cleaner job

* stronger typing of video upload middleware

* reduce dependency to @uploadx/core

* add audio upload test

* refactor resumable uploads cleanup from job to scheduler

* refactor resumable uploads scheduler to compare to last execution time

* make resumable upload validator to always cleanup on failure

* move legacy upload request building outside of uploadVideo test helper

* filter upload-resumable middlewares down to POST, PUT, DELETE

also begin to type metadata

* merge add duration functions

* stronger typings and documentation for uploadx behaviour, move init validator up

* refactor(client/video-edit): options > uploadxOptions

* refactor(client/video-edit): remove obsolete else

* scheduler/remove-dangling-resum: rename tag

* refactor(server/video): add UploadVideoFiles type

* refactor(mw/validators): restructure eslint disable

* refactor(mw/validators/videos): rename import

* refactor(client/vid-upload): rename html elem id

* refactor(sched/remove-dangl): move fn to method

* refactor(mw/async): add method typing

* refactor(mw/vali/video): double quote > single

* refactor(server/upload-resum): express use > all

* proper http methud enum server/middlewares/async.ts

* properly type http methods

* factorize common video upload validation steps

* add check for maximum partially uploaded file size

* fix audioBg use

* fix extname(filename) in addVideo

* document parameters for uploadx's resumable protocol

* clear META files in scheduler

* last audio refactor before cramming preview in the initial POST form data

* refactor as mulitpart/form-data initial post request

this allows preview/thumbnail uploads alongside the initial request,
and cleans up the upload form

* Add more tests for resumable uploads

* Refactor remove dangling resumable uploads

* Prepare changelog

* Add more resumable upload tests

* Remove user quota check for resumable uploads

* Fix upload error handler

* Update nginx template for upload-resumable

* Cleanup comment

* Remove unused express methods

* Prefer to use got instead of raw http

* Don't retry on error 500

Co-authored-by: Rigel Kent <par@rigelk.eu>
Co-authored-by: Rigel Kent <sendmemail@rigelk.eu>
Co-authored-by: Chocobozzz <me@florianbigard.com>
2021-05-10 11:13:41 +02:00
Chocobozzz c9783c7b72
Update server dependencies 2021-05-05 11:08:02 +02:00
Chocobozzz a0e69167ae
Remove scripty
It breaks our tests with the latest npm release
2021-05-05 10:51:32 +02:00
Chocobozzz 5182473faf
Upgrade server dep' 2021-04-12 16:35:04 +02:00
Chocobozzz bae9303726
Also decache config 2021-04-09 09:37:46 +02:00
Chocobozzz b70025bfdc
Remove libxmljs in favour of pure JS implem 2021-04-06 10:25:21 +02:00
Chocobozzz a784bd7fde
Upgrade to latest oauth2-server 2021-03-24 18:18:41 +01:00
Chocobozzz 7992c9e1cb
Update yarn lock 2021-03-24 18:18:41 +01:00
Chocobozzz f43db2f46e
Refactor auth flow
Reimplement some node-oauth2-server methods to remove hacky code needed by our external
login workflow
2021-03-24 18:18:41 +01:00
Chocobozzz db4b15f21f
Use got instead of request 2021-03-24 18:18:40 +01:00
Chocobozzz fbd51e69f2
Upgrade server dependencies 2021-03-03 16:14:32 +01:00
Chocobozzz 4abbeff575
Add benchmark script 2021-02-22 15:54:15 +01:00
Chocobozzz 9162476fe9
Support NodeJS 14 and 15, deprecate Node 10 2021-02-04 14:31:18 +01:00
Chocobozzz ba5a8d89bb
Update server dependencies 2021-02-03 09:45:08 +01:00
Rigel Kent 2a6cf69cff
prettify SQL queries during debug (#3635)
* prettify SQL queries during debug

* Use sql-formatter

Co-authored-by: Chocobozzz <me@florianbigard.com>
2021-01-25 15:37:41 +01:00
Chocobozzz b49f22d8f9 Upgrade sequelize to v6 2020-12-09 11:41:22 +01:00
Chocobozzz fce7fe04ee
Update server dependencies 2020-11-19 09:07:20 +01:00
Chocobozzz c2ecfe82b7
Fix prod deploy 2020-11-10 10:09:11 +01:00
Chocobozzz bd54ad1953 Add live notification tests 2020-11-09 15:33:04 +01:00
Chocobozzz c6c0fa6cd8 Live streaming implementation first step 2020-11-09 15:33:04 +01:00
Kimsible 98b9464312
render html/markdown for new comment notification email (#3255)
Co-authored-by: kimsible <kimsible@users.noreply.github.com>
Co-authored-by: Rigel Kent <sendmemail@rigelk.eu>
2020-11-07 22:59:58 +01:00
Girish Ramakrishnan 313921b50f Fix transcoding errors in readonly docker containers
ffmpeg seems to create some temporary files in the cwd. When PeerTube
is run in a read-only docker container, this causes all transcoding
to fail. As a workaround, we set the cwd to the configured tmp dir.
2020-10-23 14:51:44 +02:00
Chocobozzz 50fcdebdb1
Update server dependencies 2020-08-25 13:54:59 +02:00
kimsible 0672dc769b Add unicode emoji to markdown 2020-08-14 15:03:38 +02:00
Chocobozzz faa9d434b4
Update server dependencies 2020-06-17 11:42:50 +02:00
Rigel Kent e6dfa58689 space optimizations for `node_modules` and client stats removal
- replace sharp with lighter jimp alternative
- remove stats in builds

fixes #2807
2020-06-10 14:01:49 +02:00
Chocobozzz e0ef8dab28
Use tslib in server 2020-06-09 14:52:55 +02:00
Rigel Kent df4c603dea Switch emails to pug templates and provide richer html/text-only versions 2020-05-08 15:31:51 +02:00
Chocobozzz d96994289b
Update server dependencies 2020-04-01 14:46:53 +02:00
Chocobozzz 84ebcf345c
Fix federation with pleroma 2020-03-12 14:14:11 +01:00
Chocobozzz 3637106d7a
Upgrade oauth server dependency 2020-02-28 16:32:46 +01:00
Chocobozzz bdd428a6d9
Update dependencies 2020-02-28 16:21:34 +01:00
Average Dude 002df3813d OpenAPI clients generation 2020-02-14 14:14:14 +01:00
Chocobozzz 9a11f73392
Upgrade express-rate-limit 2020-02-04 10:52:05 +01:00
Chocobozzz a15871560f
Move to eslint 2020-02-03 08:31:02 +01:00
Chocobozzz a22046d166
Upgrade server dep 2020-02-03 08:31:02 +01:00
Chocobozzz 835053fc8b
Remove xliff package 2020-02-03 08:30:55 +01:00
Chocobozzz 5e0171daaa
Fix missing dependency 2020-01-21 14:30:53 +01:00
Chocobozzz a3705089fa
Remove husky 2020-01-21 14:28:28 +01:00
Chocobozzz e92269053e
Update http signature 2020-01-07 15:56:09 +01:00
Chocobozzz 7cde3b9c2e
Update validator dependency 2020-01-07 15:56:07 +01:00
Chocobozzz 60815a7921
Update server dependencies 2020-01-07 15:56:05 +01:00
Rigel Kent 69c7f7525d
(rss) quickfix for pfeed when category is null 2019-12-04 09:16:51 +01:00
Chocobozzz 3e101afb90
Fix tests 2019-12-03 20:19:42 +01:00
Rigel Kent 16d9224a1c
(rss) add mrss properties and update pfeed
resolves #2298
2019-12-03 16:11:10 +01:00
Chocobozzz a5a254c4b8
Upgrade server deps 2019-12-03 14:14:30 +01:00
Chocobozzz d7a25329f9
Add ability to disable webtorrent
In favour of HLS
2019-11-25 10:59:43 +01:00
Chocobozzz 2be68ab979
Update jsonld library 2019-10-21 16:05:08 +02:00
Chocobozzz 464687bb96
Fix bug with tsconfig paths 2019-10-21 16:02:15 +02:00
Chocobozzz d5d9b6d7bf
Update server dependencies 2019-10-21 15:03:28 +02:00
Chocobozzz 8424c4026a Add auto follow back support for instances 2019-09-04 16:24:58 +02:00
Chocobozzz ad513607a3
Remove old JSON LD signature implementation
Only PeerTube was compatible with it, and the library has moved on
RsaSignature2018 and removed RsaSignature2017 support. We had to create
a dirty fork of the RsaSignature2017 branch, which is not ideal.

Now we use the Mastodon implementation, that most other AP
implementations that support JSONLD signatures use.
2019-08-29 16:15:41 +02:00
Chocobozzz 322adef183
Upgrade sequelize 2019-08-12 08:52:44 +02:00
Chocobozzz 557b13ae24
Lazy load avatars 2019-08-09 11:32:40 +02:00
Chocobozzz 2284f20207
Add gitlab ci support 2019-07-29 13:40:39 +02:00
Chocobozzz 523990dbe6
Upgrade server dependencies 2019-07-25 11:00:41 +02:00
Chocobozzz 1a12f66d63
Add more CLI tests 2019-06-13 11:09:38 +02:00
Chocobozzz 6091983127
Upgrade server dependencies 2019-06-06 15:39:11 +02:00
Chocobozzz 4c72c1cd41
Move concurrently in dev package 2019-05-31 11:48:28 +02:00
Chocobozzz 7e9d3f259d Move CLI dependencies in their own package.json 2019-05-27 14:56:06 +02:00
Chocobozzz da3a3ab682
Try parallel check params tests 2019-05-14 15:37:42 +02:00
Chocobozzz 47f6409bb8
Use apicache instead of our broken implementation 2019-04-25 11:27:13 +02:00
Chocobozzz 3acc508440
Upgrade sequelize 2019-04-24 16:26:21 +02:00
Chocobozzz 1735c82572
Update sequelize 2019-04-24 16:26:20 +02:00
Chocobozzz a1a216d9cb
Upgrade server dep 2019-04-02 16:42:13 +02:00
Chocobozzz 9b39106d57
findById -> findByPk 2019-02-21 14:40:10 +01:00
Chocobozzz 374c1db98c
Upgrade server dependencies 2019-02-21 14:22:39 +01:00
Chocobozzz cef534ed53 Add user notification base code 2019-01-09 11:15:15 +01:00
Chocobozzz a4255d8efc
Remove unused dependencies 2018-12-06 13:35:27 +01:00
Chocobozzz 2feebf3e6a
Add sitemap 2018-12-05 17:44:34 +01:00
Chocobozzz f9a971c671
Update dependencies 2018-12-04 10:08:45 +01:00
Rigel Kent d7ea359d36
quickfix back to recent nodemon that fixes event-stream vulnerability 2018-11-26 21:23:10 +01:00
Rigel Kent 0218a46c1a
quickfix back to older nodemon 2018-11-26 18:43:47 +01:00
Rigel Kent 81658ae047
quickfix back to safe nodemon 2018-11-26 18:31:03 +01:00
Rigel Kent 3e9e6f2f14
migrate Swagger 2.0 spec to OpenAPI 3.0.0 2018-11-15 14:10:16 +01:00
Chocobozzz f7454ca60a
Upgrade server dep 2018-11-15 10:54:37 +01:00
Chocobozzz df66d81583
Add compatibility with other Linked Signature algorithms 2018-11-14 16:32:27 +01:00
Rigel Kent 7cf3e57376 (doc) add manpage to peertube cli 2018-10-20 19:16:56 +02:00
Chocobozzz 41f2ebae4f
Add HTTP signature check before linked signature
It's faster, and will allow us to use RSA signature 2018 (with upstream
jsonld-signature module) without too much incompatibilities in the
peertube federation
2018-10-19 12:26:37 +02:00
Chocobozzz cdf4cb9eaf
Fix transcoding 2018-10-18 10:44:16 +02:00
Chocobozzz f2659ae0b0
Upgrade tslint rules 2018-09-21 09:26:53 +02:00
Chocobozzz 74921541db
Upgrade server dependencies 2018-09-21 09:26:52 +02:00
Chocobozzz 7348b1fd84
Speed up overviews route 2018-09-14 11:52:23 +02:00
Chocobozzz c48e82b5e0 Basic video redundancy implementation 2018-09-13 14:05:49 +02:00
Chocobozzz 09cf121ce8
Try to fix bcrypt on node 8.12 2018-09-13 09:20:06 +02:00
Chocobozzz bfcef50db9
Fix oauth server module 2018-08-28 10:56:09 +02:00
Chocobozzz 2db85d6b6e
Upgrade server dependencies 2018-08-27 16:35:26 +02:00
Chocobozzz 62689b942b
Correctly migrate to fs-extra 2018-08-27 16:23:34 +02:00
Chocobozzz e69219184b
Implement user blocking on server side 2018-08-08 15:22:58 +02:00
Rigel Kent aad0ec24e8 advertising PeerTube's rather simple DNT policy 2018-08-08 09:22:15 +02:00
Aurélien Bertron 5939081838 Add audit logs module 2018-07-31 15:40:29 +02:00
Rigel Kent 4bdd9473fd adding CSP, no-referrer policies and allow dns prefetching 2018-07-17 12:03:31 +02:00
Rigel Kent d00e2393d4 selective route permission to use embeds with x-frame-deny 2018-07-17 12:03:31 +02:00
Chocobozzz 3fd5104142
Painfull server dep upgrade 2018-07-17 10:16:45 +02:00
Chocobozzz f4001cf408
Handle .srt subtitles 2018-07-16 14:31:40 +02:00
Chocobozzz de17453a98
Remove kue from the dependencies and move prompt to main ones 2018-07-11 14:24:37 +02:00
Chocobozzz 2cdf27bae6 Switch to bull 2018-07-11 14:00:17 +02:00
Rigel Kent b81eb8fdc6 (feed) adding thumbnail support for RSS feed 2018-06-29 14:58:57 +02:00
Chocobozzz 8afc19a612
Add ability to choose the language 2018-06-28 15:53:12 +02:00