Commit Graph

4575 Commits (0305db28c98fd6cf43a3c50ba92c76215e99d512)

Author SHA1 Message Date
Jelle Besseling 0305db28c9
Add support for saving video files to object storage (#4290)
* Add support for saving video files to object storage

* Add support for custom url generation on s3 stored files

Uses two config keys to support url generation that doesn't directly go
to (compatible s3). Can be used to generate urls to any cache server or
CDN.

* Upload files to s3 concurrently and delete originals afterwards

* Only publish after move to object storage is complete

* Use base url instead of url template

* Fix mistyped config field

* Add rudenmentary way to download before transcode

* Implement Chocobozzz suggestions

https://github.com/Chocobozzz/PeerTube/pull/4290#issuecomment-891670478

The remarks in question:
    Try to use objectStorage prefix instead of s3 prefix for your function/variables/config names
    Prefer to use a tree for the config: s3.streaming_playlists_bucket -> object_storage.streaming_playlists.bucket
    Use uppercase for config: S3.STREAMING_PLAYLISTS_BUCKETINFO.bucket -> OBJECT_STORAGE.STREAMING_PLAYLISTS.BUCKET (maybe BUCKET_NAME instead of BUCKET)
    I suggest to rename moveJobsRunning to pendingMovingJobs (or better, create a dedicated videoJobInfo table with a pendingMove & videoId columns so we could also use this table to track pending transcoding jobs)
    https://github.com/Chocobozzz/PeerTube/pull/4290/files#diff-3e26d41ca4bda1de8e1747af70ca2af642abcc1e9e0bfb94239ff2165acfbde5R19 uses a string instead of an integer
    I think we should store the origin object storage URL in fileUrl, without base_url injection. Instead, inject the base_url at "runtime" so admins can easily change this configuration without running a script to update DB URLs

* Import correct function

* Support multipart upload

* Remove import of node 15.0 module stream/promises

* Extend maximum upload job length

Using the same value as for redundancy downloading seems logical

* Use dynamic part size for really large uploads

Also adds very small part size for local testing

* Fix decreasePendingMove query

* Resolve various PR comments

* Move to object storage after optimize

* Make upload size configurable and increase default

* Prune webtorrent files that are stored in object storage

* Move files after transcoding jobs

* Fix federation

* Add video path manager

* Support move to external storage job in client

* Fix live object storage tests

Co-authored-by: Chocobozzz <me@florianbigard.com>
2021-08-17 08:26:20 +02:00
Chocobozzz 134006b0d4
Update client dependencies 2021-08-11 11:49:01 +02:00
Chocobozzz 421ff4618d
Merge branch 'release/3.3.0' into develop 2021-08-05 15:17:18 +02:00
Chocobozzz ff5f37e4f3
Fix sha fallback 2021-08-05 15:17:09 +02:00
Chocobozzz 27db78400c
Fix backend channel name validator consistency 2021-08-05 13:54:35 +02:00
Chocobozzz 3e254de8be HLS v1 support 2021-08-05 10:57:49 +02:00
Chocobozzz ff4de38385
Filter videos by live in custom markup 2021-08-02 17:10:07 +02:00
Chocobozzz 3da38d6e9f
Fetch things in bulk for the homepage 2021-08-02 15:40:09 +02:00
Chocobozzz 200eaf5152
Remove thumbnail flash for autoplay 2021-08-02 11:46:11 +02:00
Chocobozzz 4a7056b283
Update translations 2021-08-02 10:49:05 +02:00
Phongpanot 90ea6ef1f1 Translated using Weblate (Thai)
Currently translated at 52.8% (952 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/th/
2021-08-02 10:41:15 +02:00
Marc Strange 7f02b833ec Translated using Weblate (Catalan)
Currently translated at 36.8% (663 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/ca/
2021-08-02 10:41:15 +02:00
Marc Strange 1fdea5f5d0 Translated using Weblate (Catalan)
Currently translated at 100.0% (118 of 118 strings)

Translation: PeerTube/player
Translate-URL: https://weblate.framasoft.org/projects/peertube/player/ca/
2021-08-02 10:41:15 +02:00
Marc Strange 6a95289c84 Translated using Weblate (Catalan)
Currently translated at 33.9% (611 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/ca/
2021-08-02 10:41:15 +02:00
Grzesiek11 4c37cbfbe2 Translated using Weblate (Polish)
Currently translated at 99.7% (1795 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/pl/
2021-08-02 10:41:15 +02:00
Hồ Nhất Duy 1becfc7cab Translated using Weblate (Vietnamese)
Currently translated at 99.8% (1798 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/vi/
2021-08-02 10:41:15 +02:00
Александр e1220d60f2 Translated using Weblate (Russian)
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/ru/
2021-08-02 10:41:15 +02:00
Laurent Ettouati 3ef9aceef9 Translated using Weblate (French (France) (fr_FR))
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/fr_FR/
2021-08-02 10:41:15 +02:00
Александр 7aa6525309 Translated using Weblate (Russian)
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/ru/
2021-08-02 10:41:15 +02:00
Simon Brosdetzko a9a6ecf8be Translated using Weblate (German)
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/de/
2021-08-02 10:41:15 +02:00
Racida S 3cfda8ea54 Translated using Weblate (Kabyle)
Currently translated at 95.5% (1720 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/kab/
2021-08-02 10:41:15 +02:00
Hồ Nhất Duy 706bc8aedc Translated using Weblate (Vietnamese)
Currently translated at 100.0% (246 of 246 strings)

Translation: PeerTube/server
Translate-URL: https://weblate.framasoft.org/projects/peertube/server/vi/
2021-08-02 10:41:15 +02:00
Hồ Nhất Duy 93a8ab2a74 Translated using Weblate (Vietnamese)
Currently translated at 99.8% (1798 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/vi/
2021-08-02 10:41:15 +02:00
Laurent Ettouati 2b7b9dd7bb Translated using Weblate (French (France) (fr_FR))
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/fr_FR/
2021-08-02 10:41:15 +02:00
framabr af85e02ab3 Translated using Weblate (Portuguese (Brazil))
Currently translated at 53.9% (971 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/pt_BR/
2021-08-02 10:41:15 +02:00
Marc Strange 8ac253b598 Translated using Weblate (Catalan)
Currently translated at 32.6% (588 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/ca/
2021-08-02 10:41:15 +02:00
Jeff Huang 2909b999fe Translated using Weblate (Chinese (Traditional))
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/zh_Hant/
2021-08-02 10:41:15 +02:00
Александр bdb7d35cf0 Translated using Weblate (Russian)
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/ru/
2021-08-02 10:41:15 +02:00
josé m 2f2b7b6b1b Translated using Weblate (Galician)
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/gl/
2021-08-02 10:41:15 +02:00
GunChleoc a082d6fd1a Translated using Weblate (Gaelic)
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/gd/
2021-08-02 10:41:15 +02:00
Berto Te 26bafbb701 Translated using Weblate (Spanish)
Currently translated at 100.0% (1800 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/es/
2021-08-02 10:41:15 +02:00
GunChleoc b68ec72ed6 Translated using Weblate (Gaelic)
Currently translated at 98.6% (1776 of 1800 strings)

Translation: PeerTube/angular
Translate-URL: https://weblate.framasoft.org/projects/peertube/angular/gd/
2021-08-02 10:41:15 +02:00
Chocobozzz 23ca2f7bc9
Fix peertube link button 2021-07-30 11:48:21 +02:00
Chocobozzz 171efc48e6
Merge branch 'release/3.3.0' into develop 2021-07-30 11:38:19 +02:00
Chocobozzz 3ce48a0cd0
Hide schedule privacy if private does not exist 2021-07-30 11:24:38 +02:00
Chocobozzz 88d36fe4dc
Correctly fix peertube button link 2021-07-29 17:16:03 +02:00
Chocobozzz 916bf5283b
Fix empty search 2021-07-29 15:51:19 +02:00
Chocobozzz 7e8f19603b
Redirect to search page for origin instance 2021-07-29 15:35:41 +02:00
Chocobozzz af7fd04a67
Add ability to filter by host in search page 2021-07-29 15:19:22 +02:00
Chocobozzz 29837f8885
Add ability to search by host in server 2021-07-27 17:10:52 +02:00
Chocobozzz ac27887774
Merge branch 'release/3.3.0' into develop 2021-07-26 15:18:46 +02:00
Chocobozzz 84c6eadc33
Fix moderation embeds 2021-07-26 15:17:59 +02:00
Chocobozzz a00056c0c8
Fix description timestamp click 2021-07-26 15:14:17 +02:00
Chocobozzz 15a7eafb89
Refactor video links builders 2021-07-26 15:04:37 +02:00
Chocobozzz 9162fdd363
Refactor video links building 2021-07-26 14:12:50 +02:00
Chocobozzz d91714ca56
Fix privacy descriptions 2021-07-26 14:03:46 +02:00
Chocobozzz a66424978c
Fix URL copy in embed 2021-07-26 13:37:18 +02:00
Chocobozzz c4fa01f7c4
Merge branch 'release/3.3.0' into develop 2021-07-23 15:23:43 +02:00
Kimsible c1936140b9 Fix user-dropdown-indicator position on touchscreens 2021-07-23 15:23:22 +02:00
Chocobozzz 7c94e1e7b6
Update translations 2021-07-21 16:26:37 +02:00