Sorry for the very big commit that may lead to git log issues and merge
conflicts, but it's a major step forward:
* Server can be faster at startup because imports() are async and we can
easily lazy import big modules
* Angular doesn't seem to support ES import (with .js extension), so we
had to correctly organize peertube into a monorepo:
* Use yarn workspace feature
* Use typescript reference projects for dependencies
* Shared projects have been moved into "packages", each one is now a
node module (with a dedicated package.json/tsconfig.json)
* server/tools have been moved into apps/ and is now a dedicated app
bundled and published on NPM so users don't have to build peertube
cli tools manually
* server/tests have been moved into packages/ so we don't compile
them every time we want to run the server
* Use isolatedModule option:
* Had to move from const enum to const
(https://www.typescriptlang.org/docs/handbook/enums.html#objects-vs-enums)
* Had to explictely specify "type" imports when used in decorators
* Prefer tsx (that uses esbuild under the hood) instead of ts-node to
load typescript files (tests with mocha or scripts):
* To reduce test complexity as esbuild doesn't support decorator
metadata, we only test server files that do not import server
models
* We still build tests files into js files for a faster CI
* Remove unmaintained peertube CLI import script
* Removed some barrels to speed up execution (less imports)
* client: remove unused file
* refactor(client/my-actor-avatar): size from input
Read size from component input instead of scss, to make it possible to
use smaller avatar images when implemented.
* implement avatar miniatures
close#4560
* fix(test): max file size
* fix(search-index): normalize res acc to avatarMini
* refactor avatars to an array
* client/search: resize channel avatar to 120
* refactor(client/videos): remove unused function
* client(actor-avatar): set default size
* fix tests and avatars full result
When findOne is used only an array containting one avatar is returned.
* update migration version and version notations
* server/search: harmonize normalizing
* Cleanup avatar miniature PR
Co-authored-by: Chocobozzz <me@florianbigard.com>
* WIP: resumable video uploads
relates to #324
* fix review comments
* video upload: error handling
* fix audio upload
* fixes after self review
* Update server/controllers/api/videos/index.ts
Co-authored-by: Rigel Kent <par@rigelk.eu>
* Update server/middlewares/validators/videos/videos.ts
Co-authored-by: Rigel Kent <par@rigelk.eu>
* Update server/controllers/api/videos/index.ts
Co-authored-by: Rigel Kent <par@rigelk.eu>
* update after code review
* refactor upload route
- restore multipart upload route
- move resumable to dedicated upload-resumable route
- move checks to middleware
- do not leak internal fs structure in response
* fix yarn.lock upon rebase
* factorize addVideo for reuse in both endpoints
* add resumable upload API to openapi spec
* add initial test and test helper for resumable upload
* typings for videoAddResumable middleware
* avoid including aws and google packages via node-uploadx, by only including uploadx/core
* rename ex-isAudioBg to more explicit name mentioning it is a preview file for audio
* add video-upload-tmp-folder-cleaner job
* stronger typing of video upload middleware
* reduce dependency to @uploadx/core
* add audio upload test
* refactor resumable uploads cleanup from job to scheduler
* refactor resumable uploads scheduler to compare to last execution time
* make resumable upload validator to always cleanup on failure
* move legacy upload request building outside of uploadVideo test helper
* filter upload-resumable middlewares down to POST, PUT, DELETE
also begin to type metadata
* merge add duration functions
* stronger typings and documentation for uploadx behaviour, move init validator up
* refactor(client/video-edit): options > uploadxOptions
* refactor(client/video-edit): remove obsolete else
* scheduler/remove-dangling-resum: rename tag
* refactor(server/video): add UploadVideoFiles type
* refactor(mw/validators): restructure eslint disable
* refactor(mw/validators/videos): rename import
* refactor(client/vid-upload): rename html elem id
* refactor(sched/remove-dangl): move fn to method
* refactor(mw/async): add method typing
* refactor(mw/vali/video): double quote > single
* refactor(server/upload-resum): express use > all
* proper http methud enum server/middlewares/async.ts
* properly type http methods
* factorize common video upload validation steps
* add check for maximum partially uploaded file size
* fix audioBg use
* fix extname(filename) in addVideo
* document parameters for uploadx's resumable protocol
* clear META files in scheduler
* last audio refactor before cramming preview in the initial POST form data
* refactor as mulitpart/form-data initial post request
this allows preview/thumbnail uploads alongside the initial request,
and cleans up the upload form
* Add more tests for resumable uploads
* Refactor remove dangling resumable uploads
* Prepare changelog
* Add more resumable upload tests
* Remove user quota check for resumable uploads
* Fix upload error handler
* Update nginx template for upload-resumable
* Cleanup comment
* Remove unused express methods
* Prefer to use got instead of raw http
* Don't retry on error 500
Co-authored-by: Rigel Kent <par@rigelk.eu>
Co-authored-by: Rigel Kent <sendmemail@rigelk.eu>
Co-authored-by: Chocobozzz <me@florianbigard.com>
The `proxy_ignore_client_abort` flag specifies whether nginx will
monitor possible connection close while waiting for an upstream
server response. If an error occurs while sending a response, the
connection will be closed regardless of the flag, much like if
there were no nginx at all.
fixes#3484
refactors the Nginx configuration for the following points:
- update tls version to include 1.3 by default. so far it was not included by default to make room for previous versions of Nginx, but since 2018 Debian stable has included Nginx in version 1.14.1, and tls 1.3 is available since Nginx 1.13.0.
- clearly indicate that new minimum required version.
- update outdated ssl_ciphers to remove cipher required to support android 4.4, since that version is unsupported since March 2020.
- reordered configuration in sections for easier maintenance: performance optimizations are separated from the vital application/websocket parts.
- move parts that always require manual configuration at the top: peertube host and server name, use server_name
- move peertube host to a more flexible upstream block: it allows to configure it in one place instead of 3, and is future-proof regarding load-balancing.
- simplified port 80 block: Let’s Encrypt supports 301 redirects.
- group certificate-related config together.
- remove reslover config: it defaults to /etc/resolv.conf which is more than enough.
- align values with their neighbors for easier reading
- always specify units
- always specify default values when they differ from the values set
- use ’m’ for minutes, ’M’ for megabytes
- add consensual optimizations wrt file serving:
- add timeout optimizations
- add file descriptor cache optimizations
- enable sendfile with chunk size > rate limit
- enable threading
- tcp optimizations
- point to further, more system-specific optimizations in the section description
- CDN configuration reduced to one line change
- support/docker/production/docker-compose.yml: addition of a nginx
image reusing support/nginx/peertube nginx conf to improve performance,
and lessen setup differences between the docker-compose install and the
typical production install.
- support/docker/production/docker-compose.yml: postgres 10 -> postgres
12, redis 4 -> redis 5. Postgres major updates implies manual upgrade.
- support/nginx/peertube: HTTP -> HTTPS redirection is now commented
by default, to allow its reuse in support/docker/production/docker-compose.yml.
* Remove hard-coded 8GB upload limit in client
Ideally we'd know what the specific server's configured upload limit
is before starting, but this 8GB limit is not useful if an administrator
has changed the nginx post limit on the server.
* Better docs for admins about client_max_body_size
Seems like some admins already tweak this value up or down to allow
for different maximum video upload sizes. The current codebase has no
other server-side limits that I'm aware of, and I've been routinely
uploading quite large videos to my instance.
This patch replaces the somewhat incorrect (or outdated?) 'hard limit'
comment with some advice about allocating enough space for nginx and
communicating the limit with your users.
Of course it would be better if this configuration could be unified with
PeerTube's config somehow. I'm not sure whether the best option there is
to turn off nginx's buffering here and let PeerTube handle the entire upload
(can we do this only for the video upload API endpoint?) or whether we want
PeerTube to generate nginx configs in a more automated way layer. In any case,
this patch is intended as an incremental improvement.
Chocobozzz