* WIP: resumable video uploads
relates to #324
* fix review comments
* video upload: error handling
* fix audio upload
* fixes after self review
* Update server/controllers/api/videos/index.ts
Co-authored-by: Rigel Kent <par@rigelk.eu>
* Update server/middlewares/validators/videos/videos.ts
Co-authored-by: Rigel Kent <par@rigelk.eu>
* Update server/controllers/api/videos/index.ts
Co-authored-by: Rigel Kent <par@rigelk.eu>
* update after code review
* refactor upload route
- restore multipart upload route
- move resumable to dedicated upload-resumable route
- move checks to middleware
- do not leak internal fs structure in response
* fix yarn.lock upon rebase
* factorize addVideo for reuse in both endpoints
* add resumable upload API to openapi spec
* add initial test and test helper for resumable upload
* typings for videoAddResumable middleware
* avoid including aws and google packages via node-uploadx, by only including uploadx/core
* rename ex-isAudioBg to more explicit name mentioning it is a preview file for audio
* add video-upload-tmp-folder-cleaner job
* stronger typing of video upload middleware
* reduce dependency to @uploadx/core
* add audio upload test
* refactor resumable uploads cleanup from job to scheduler
* refactor resumable uploads scheduler to compare to last execution time
* make resumable upload validator to always cleanup on failure
* move legacy upload request building outside of uploadVideo test helper
* filter upload-resumable middlewares down to POST, PUT, DELETE
also begin to type metadata
* merge add duration functions
* stronger typings and documentation for uploadx behaviour, move init validator up
* refactor(client/video-edit): options > uploadxOptions
* refactor(client/video-edit): remove obsolete else
* scheduler/remove-dangling-resum: rename tag
* refactor(server/video): add UploadVideoFiles type
* refactor(mw/validators): restructure eslint disable
* refactor(mw/validators/videos): rename import
* refactor(client/vid-upload): rename html elem id
* refactor(sched/remove-dangl): move fn to method
* refactor(mw/async): add method typing
* refactor(mw/vali/video): double quote > single
* refactor(server/upload-resum): express use > all
* proper http methud enum server/middlewares/async.ts
* properly type http methods
* factorize common video upload validation steps
* add check for maximum partially uploaded file size
* fix audioBg use
* fix extname(filename) in addVideo
* document parameters for uploadx's resumable protocol
* clear META files in scheduler
* last audio refactor before cramming preview in the initial POST form data
* refactor as mulitpart/form-data initial post request
this allows preview/thumbnail uploads alongside the initial request,
and cleans up the upload form
* Add more tests for resumable uploads
* Refactor remove dangling resumable uploads
* Prepare changelog
* Add more resumable upload tests
* Remove user quota check for resumable uploads
* Fix upload error handler
* Update nginx template for upload-resumable
* Cleanup comment
* Remove unused express methods
* Prefer to use got instead of raw http
* Don't retry on error 500
Co-authored-by: Rigel Kent <par@rigelk.eu>
Co-authored-by: Rigel Kent <sendmemail@rigelk.eu>
Co-authored-by: Chocobozzz <me@florianbigard.com>
The `proxy_ignore_client_abort` flag specifies whether nginx will
monitor possible connection close while waiting for an upstream
server response. If an error occurs while sending a response, the
connection will be closed regardless of the flag, much like if
there were no nginx at all.
fixes#3484
refactors the Nginx configuration for the following points:
- update tls version to include 1.3 by default. so far it was not included by default to make room for previous versions of Nginx, but since 2018 Debian stable has included Nginx in version 1.14.1, and tls 1.3 is available since Nginx 1.13.0.
- clearly indicate that new minimum required version.
- update outdated ssl_ciphers to remove cipher required to support android 4.4, since that version is unsupported since March 2020.
- reordered configuration in sections for easier maintenance: performance optimizations are separated from the vital application/websocket parts.
- move parts that always require manual configuration at the top: peertube host and server name, use server_name
- move peertube host to a more flexible upstream block: it allows to configure it in one place instead of 3, and is future-proof regarding load-balancing.
- simplified port 80 block: Let’s Encrypt supports 301 redirects.
- group certificate-related config together.
- remove reslover config: it defaults to /etc/resolv.conf which is more than enough.
- align values with their neighbors for easier reading
- always specify units
- always specify default values when they differ from the values set
- use ’m’ for minutes, ’M’ for megabytes
- add consensual optimizations wrt file serving:
- add timeout optimizations
- add file descriptor cache optimizations
- enable sendfile with chunk size > rate limit
- enable threading
- tcp optimizations
- point to further, more system-specific optimizations in the section description
- CDN configuration reduced to one line change
- support/docker/production/docker-compose.yml: addition of a nginx
image reusing support/nginx/peertube nginx conf to improve performance,
and lessen setup differences between the docker-compose install and the
typical production install.
- support/docker/production/docker-compose.yml: postgres 10 -> postgres
12, redis 4 -> redis 5. Postgres major updates implies manual upgrade.
- support/nginx/peertube: HTTP -> HTTPS redirection is now commented
by default, to allow its reuse in support/docker/production/docker-compose.yml.
* Remove hard-coded 8GB upload limit in client
Ideally we'd know what the specific server's configured upload limit
is before starting, but this 8GB limit is not useful if an administrator
has changed the nginx post limit on the server.
* Better docs for admins about client_max_body_size
Seems like some admins already tweak this value up or down to allow
for different maximum video upload sizes. The current codebase has no
other server-side limits that I'm aware of, and I've been routinely
uploading quite large videos to my instance.
This patch replaces the somewhat incorrect (or outdated?) 'hard limit'
comment with some advice about allocating enough space for nginx and
communicating the limit with your users.
Of course it would be better if this configuration could be unified with
PeerTube's config somehow. I'm not sure whether the best option there is
to turn off nginx's buffering here and let PeerTube handle the entire upload
(can we do this only for the video upload API endpoint?) or whether we want
PeerTube to generate nginx configs in a more automated way layer. In any case,
this patch is intended as an incremental improvement.
This goes against the recommendations (preloading should be opt-in). Putting it in the example makes it likely that people enable it without knowing what it means.
https://hstspreload.org/?domain=peertube.social#opt-in
Without gzip explicitly enabled, load times suffer from transferring
over a megabyte of plaintext javascript. With gzip enabled, the bundle
is down to about 300K, and loads much faster.
This change does not enable gzip on files that are already compressed,
so images, fonts, and videos will be sent without the CPU overhead.
- added precisions and suggestions about how to generate Let's Encrypt certificates. Users have reported their installations didn't work when the problem came from missing certificates (false positives).
- security defaults of Nginx follow the basic robustness principle "be conservative in what you send, be liberal in what you accept", which isn't enough with modern security standards, so we should be picky with the cipher suites we use, among other things. Extra comments (especially for the TLS1.3 protocol support parameter) make the requirement of a recent Nginx installation obvious, and the downgrade alternative remains clear to the system administrator.
All in all, we should aknowledge users will most often copy and paste the configuration files. Making them secure by default may force a few users to read their configuration, but on the long run we are making the fediverse more secure.
Since I've come to modify a bit the Nginx config in `support/doc/production.md`, I've merged it with the template so that they stay consistent.
Chocobozzz