Commit Graph

7924 Commits (0305db28c98fd6cf43a3c50ba92c76215e99d512)

Author SHA1 Message Date
Jelle Besseling 0305db28c9
Add support for saving video files to object storage (#4290)
* Add support for saving video files to object storage

* Add support for custom url generation on s3 stored files

Uses two config keys to support url generation that doesn't directly go
to (compatible s3). Can be used to generate urls to any cache server or
CDN.

* Upload files to s3 concurrently and delete originals afterwards

* Only publish after move to object storage is complete

* Use base url instead of url template

* Fix mistyped config field

* Add rudenmentary way to download before transcode

* Implement Chocobozzz suggestions

https://github.com/Chocobozzz/PeerTube/pull/4290#issuecomment-891670478

The remarks in question:
    Try to use objectStorage prefix instead of s3 prefix for your function/variables/config names
    Prefer to use a tree for the config: s3.streaming_playlists_bucket -> object_storage.streaming_playlists.bucket
    Use uppercase for config: S3.STREAMING_PLAYLISTS_BUCKETINFO.bucket -> OBJECT_STORAGE.STREAMING_PLAYLISTS.BUCKET (maybe BUCKET_NAME instead of BUCKET)
    I suggest to rename moveJobsRunning to pendingMovingJobs (or better, create a dedicated videoJobInfo table with a pendingMove & videoId columns so we could also use this table to track pending transcoding jobs)
    https://github.com/Chocobozzz/PeerTube/pull/4290/files#diff-3e26d41ca4bda1de8e1747af70ca2af642abcc1e9e0bfb94239ff2165acfbde5R19 uses a string instead of an integer
    I think we should store the origin object storage URL in fileUrl, without base_url injection. Instead, inject the base_url at "runtime" so admins can easily change this configuration without running a script to update DB URLs

* Import correct function

* Support multipart upload

* Remove import of node 15.0 module stream/promises

* Extend maximum upload job length

Using the same value as for redundancy downloading seems logical

* Use dynamic part size for really large uploads

Also adds very small part size for local testing

* Fix decreasePendingMove query

* Resolve various PR comments

* Move to object storage after optimize

* Make upload size configurable and increase default

* Prune webtorrent files that are stored in object storage

* Move files after transcoding jobs

* Fix federation

* Add video path manager

* Support move to external storage job in client

* Fix live object storage tests

Co-authored-by: Chocobozzz <me@florianbigard.com>
2021-08-17 08:26:20 +02:00
Chocobozzz f88ae8f5bc
Disable openapi generation
Until https://github.com/OpenAPITools/openapi-generator/issues/8663 is
fixed
2021-08-11 14:58:37 +02:00
Chocobozzz de34551da3
Typo 2021-08-11 14:43:03 +02:00
Chocobozzz eb8bd9ab1a
Try to fix openapi script 2021-08-11 14:38:07 +02:00
Chocobozzz 5b075bc549
Fix search index tests 2021-08-11 11:49:01 +02:00
Chocobozzz 134006b0d4
Update client dependencies 2021-08-11 11:49:01 +02:00
Chocobozzz 31aa391d13
Update server dependencies 2021-08-11 11:49:00 +02:00
Mark Van den Borre 99c05b3dcc Update FAQ.md
s/franglais/english/
2021-08-11 10:46:14 +02:00
ROBERT MCDOWELL 863c1bc6dc
fixed typo (#4312)
* Fixed typo
2021-08-11 10:45:49 +02:00
Chocobozzz 18998c45c0
Correctly handle large HLS files for redundancy 2021-08-06 15:25:45 +02:00
Chocobozzz a1c63fe1a2
Fix send rtmp command 2021-08-06 15:06:47 +02:00
Chocobozzz 7942e72e38
Fix cli build 2021-08-06 14:39:28 +02:00
Chocobozzz 679c12e69c
Improve target bitrate calculation 2021-08-06 14:13:26 +02:00
Chocobozzz c826f34a45
Limit live bitrate 2021-08-06 10:39:40 +02:00
Chocobozzz 421ff4618d
Merge branch 'release/3.3.0' into develop 2021-08-05 15:17:18 +02:00
Chocobozzz ff5f37e4f3
Fix sha fallback 2021-08-05 15:17:09 +02:00
Chocobozzz e83d06a702
Remove deprecated transcoding job names 2021-08-05 14:29:44 +02:00
Chocobozzz 45570e9397
Fix upload without files 2021-08-05 14:24:03 +02:00
Chocobozzz 27db78400c
Fix backend channel name validator consistency 2021-08-05 13:54:35 +02:00
Chocobozzz 352819ef92
Do not index remote actors 2021-08-05 13:29:55 +02:00
ROBERT MCDOWELL da948b75ca Standard output type syslog is obsolete in systemed
removing the setting
StandardOutput
StandardError
2021-08-05 10:58:05 +02:00
Chocobozzz 3e254de8be HLS v1 support 2021-08-05 10:57:49 +02:00
Chocobozzz ff4de38385
Filter videos by live in custom markup 2021-08-02 17:10:07 +02:00
Chocobozzz 3da38d6e9f
Fetch things in bulk for the homepage 2021-08-02 15:40:09 +02:00
Chocobozzz 200eaf5152
Remove thumbnail flash for autoplay 2021-08-02 11:46:11 +02:00
Chocobozzz 4a7056b283
Update translations 2021-08-02 10:49:05 +02:00
Phongpanot 90ea6ef1f1 Translated using Weblate (Thai)
Currently translated at 52.8% (952 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/th/
2021-08-02 10:41:15 +02:00
Marc Strange 7f02b833ec Translated using Weblate (Catalan)
Currently translated at 36.8% (663 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/ca/
2021-08-02 10:41:15 +02:00
Marc Strange 1fdea5f5d0 Translated using Weblate (Catalan)
Currently translated at 100.0% (118 of 118 strings)

Translation: PeerTube/player
Translate-URL: https://weblate.framasoft.org/projects/peertube/player/ca/
2021-08-02 10:41:15 +02:00
Marc Strange 6a95289c84 Translated using Weblate (Catalan)
Currently translated at 33.9% (611 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/ca/
2021-08-02 10:41:15 +02:00
Grzesiek11 4c37cbfbe2 Translated using Weblate (Polish)
Currently translated at 99.7% (1795 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/pl/
2021-08-02 10:41:15 +02:00
Hồ Nhất Duy 1becfc7cab Translated using Weblate (Vietnamese)
Currently translated at 99.8% (1798 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/vi/
2021-08-02 10:41:15 +02:00
Александр e1220d60f2 Translated using Weblate (Russian)
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/ru/
2021-08-02 10:41:15 +02:00
Laurent Ettouati 3ef9aceef9 Translated using Weblate (French (France) (fr_FR))
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/fr_FR/
2021-08-02 10:41:15 +02:00
Александр 7aa6525309 Translated using Weblate (Russian)
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/ru/
2021-08-02 10:41:15 +02:00
Simon Brosdetzko a9a6ecf8be Translated using Weblate (German)
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/de/
2021-08-02 10:41:15 +02:00
Racida S 3cfda8ea54 Translated using Weblate (Kabyle)
Currently translated at 95.5% (1720 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/kab/
2021-08-02 10:41:15 +02:00
Hồ Nhất Duy 706bc8aedc Translated using Weblate (Vietnamese)
Currently translated at 100.0% (246 of 246 strings)

Translation: PeerTube/server
Translate-URL: https://weblate.framasoft.org/projects/peertube/server/vi/
2021-08-02 10:41:15 +02:00
Hồ Nhất Duy 93a8ab2a74 Translated using Weblate (Vietnamese)
Currently translated at 99.8% (1798 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/vi/
2021-08-02 10:41:15 +02:00
Laurent Ettouati 2b7b9dd7bb Translated using Weblate (French (France) (fr_FR))
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/fr_FR/
2021-08-02 10:41:15 +02:00
framabr af85e02ab3 Translated using Weblate (Portuguese (Brazil))
Currently translated at 53.9% (971 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/pt_BR/
2021-08-02 10:41:15 +02:00
Marc Strange 8ac253b598 Translated using Weblate (Catalan)
Currently translated at 32.6% (588 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/ca/
2021-08-02 10:41:15 +02:00
Jeff Huang 2909b999fe Translated using Weblate (Chinese (Traditional))
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/zh_Hant/
2021-08-02 10:41:15 +02:00
Александр bdb7d35cf0 Translated using Weblate (Russian)
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/ru/
2021-08-02 10:41:15 +02:00
josé m 2f2b7b6b1b Translated using Weblate (Galician)
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/gl/
2021-08-02 10:41:15 +02:00
GunChleoc a082d6fd1a Translated using Weblate (Gaelic)
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/gd/
2021-08-02 10:41:15 +02:00
Berto Te 26bafbb701 Translated using Weblate (Spanish)
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/es/
2021-08-02 10:41:15 +02:00
GunChleoc b68ec72ed6 Translated using Weblate (Gaelic)
Currently translated at 98.6% (1776 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/gd/
2021-08-02 10:41:15 +02:00
Chocobozzz d26836cd95
Refactor notifier 2021-08-02 10:39:51 +02:00
Chocobozzz 2bee9db56a
Update plugin doc 2021-07-30 13:58:18 +02:00