Commit Graph

176 Commits (1d4591fd9c724c9409da31612f6d8c008f1cb5e4)

Author SHA1 Message Date
Jelle Besseling 0305db28c9
Add support for saving video files to object storage (#4290)
* Add support for saving video files to object storage

* Add support for custom url generation on s3 stored files

Uses two config keys to support url generation that doesn't directly go
to (compatible s3). Can be used to generate urls to any cache server or
CDN.

* Upload files to s3 concurrently and delete originals afterwards

* Only publish after move to object storage is complete

* Use base url instead of url template

* Fix mistyped config field

* Add rudenmentary way to download before transcode

* Implement Chocobozzz suggestions

https://github.com/Chocobozzz/PeerTube/pull/4290#issuecomment-891670478

The remarks in question:
    Try to use objectStorage prefix instead of s3 prefix for your function/variables/config names
    Prefer to use a tree for the config: s3.streaming_playlists_bucket -> object_storage.streaming_playlists.bucket
    Use uppercase for config: S3.STREAMING_PLAYLISTS_BUCKETINFO.bucket -> OBJECT_STORAGE.STREAMING_PLAYLISTS.BUCKET (maybe BUCKET_NAME instead of BUCKET)
    I suggest to rename moveJobsRunning to pendingMovingJobs (or better, create a dedicated videoJobInfo table with a pendingMove & videoId columns so we could also use this table to track pending transcoding jobs)
    https://github.com/Chocobozzz/PeerTube/pull/4290/files#diff-3e26d41ca4bda1de8e1747af70ca2af642abcc1e9e0bfb94239ff2165acfbde5R19 uses a string instead of an integer
    I think we should store the origin object storage URL in fileUrl, without base_url injection. Instead, inject the base_url at "runtime" so admins can easily change this configuration without running a script to update DB URLs

* Import correct function

* Support multipart upload

* Remove import of node 15.0 module stream/promises

* Extend maximum upload job length

Using the same value as for redundancy downloading seems logical

* Use dynamic part size for really large uploads

Also adds very small part size for local testing

* Fix decreasePendingMove query

* Resolve various PR comments

* Move to object storage after optimize

* Make upload size configurable and increase default

* Prune webtorrent files that are stored in object storage

* Move files after transcoding jobs

* Fix federation

* Add video path manager

* Support move to external storage job in client

* Fix live object storage tests

Co-authored-by: Chocobozzz <me@florianbigard.com>
2021-08-17 08:26:20 +02:00
Chocobozzz 679c12e69c
Improve target bitrate calculation 2021-08-06 14:13:26 +02:00
Chocobozzz c826f34a45
Limit live bitrate 2021-08-06 10:39:40 +02:00
Chocobozzz 28be9d1d3e
Cleanup shared models imports 2021-07-28 10:46:32 +02:00
Chocobozzz d23dd9fbfc
Introduce videos command 2021-07-20 15:27:18 +02:00
Chocobozzz 12edc1495a
Introduce comments command 2021-07-20 15:27:18 +02:00
Chocobozzz a54618880c
Introduce channels command 2021-07-20 15:27:18 +02:00
Chocobozzz e6346d59e6
Introduce playlist command 2021-07-20 15:27:18 +02:00
Chocobozzz d4a8e7a65f Support short uuid for GET video/playlist 2021-06-29 14:56:35 +02:00
Chocobozzz 37a44fc915 Add ability to search playlists 2021-06-25 14:44:01 +02:00
Chocobozzz 2b02c520e6
Cleanup shared models 2021-05-11 11:32:31 +02:00
Chocobozzz e024fd6a74
Update channel updatedAt when uploading a video 2021-05-10 09:38:11 +02:00
Chocobozzz 3e03b961b8 Add ability for plugins to specify scale filter 2021-04-09 15:53:18 +02:00
Théo Le Calvar 43f7a43ca4 add option for transcode plugins to add video filters and make all options optional 2021-04-09 15:53:18 +02:00
Théo Le Calvar 5fb7cfbac5 add support for inputOptions in trancode plugins 2021-04-09 15:53:18 +02:00
Chocobozzz 2cb03dc1f4 Add banners support 2021-04-08 10:07:53 +02:00
Chocobozzz f479685678 Agnostic actor image storage 2021-04-08 10:07:53 +02:00
Chocobozzz f66db4d5c8 Fix feeds 2021-02-18 13:38:09 +01:00
Chocobozzz 90a8bd305d Dissociate video file names and video uuid 2021-02-18 13:38:09 +01:00
Rigel Kent 3d4e112d16 add best trending strategy based on Reddit's best
inspired from https://www.reddit.com/r/changelog/comments/7spgg0/best_is_the_new_hotness/
this implementation only adds freshness, and doesn't personalize based
on subscribed communities yet.
2021-02-04 09:04:47 +01:00
Chocobozzz 1896bca09e
Support transcoding options/encoders by plugins 2021-01-28 15:55:39 +01:00
Rigel Kent 5bcbcbe338 modularize abstract video list header and implement video hotness recommendation variant 2021-01-28 15:55:34 +01:00
Rigel Kent b7085c7132 add support for 1440p (Quad HD/QHD/WQHD) videos 2021-01-13 09:14:09 +01:00
Chocobozzz a800dbf345
Live views update 2020-12-09 16:31:42 +01:00
Chocobozzz bb4ba6d94c
Add permanent live support 2020-12-03 15:21:16 +01:00
Chocobozzz 5a547f69d5 Support encoding profiles 2020-11-25 10:07:51 +01:00
Chocobozzz 0aa52e1707
Add ability to display all channel/account videos 2020-11-18 15:29:38 +01:00
Chocobozzz 0f8d00e314
Implement video comment list in admin 2020-11-13 16:38:23 +01:00
Chocobozzz 97969c4edf Add check constraints live tests 2020-11-09 15:33:04 +01:00
Chocobozzz af4ae64f6f Begin live tests 2020-11-09 15:33:04 +01:00
Chocobozzz 77e9f859c6 Add check params live tests 2020-11-09 15:33:04 +01:00
Chocobozzz b5b687550d Add ability to save live replay 2020-11-09 15:33:04 +01:00
Chocobozzz a5cf76afa3 Add watch messages if live has not started 2020-11-09 15:33:04 +01:00
Chocobozzz c6c0fa6cd8 Live streaming implementation first step 2020-11-09 15:33:04 +01:00
Chocobozzz b2c76204f9
Fix VideoStreamingPlaylist model 2020-08-27 10:08:54 +02:00
Chocobozzz 0fecf427d3
Don't use @shared in models
So other projects can use them
2020-08-27 09:58:27 +02:00
Chocobozzz 7294aab0c8 Add ability to set custom field to video form 2020-08-21 15:39:51 +02:00
Rigel Kent 02c01341f4 add ng-select for templatable select options
- create select-tags component to replace ngx-chips
- create select-options to factorize option selection in forms
- create select-channel to simplify channel selection
- refactor tags validation
2020-08-11 09:03:39 +02:00
Chocobozzz 951b582f52
Add ability to share playlists in modal 2020-08-07 13:43:48 +02:00
Chocobozzz bd45d503e5 Reorganize shared models 2020-08-07 08:28:14 +02:00
Chocobozzz 583eb04b54 Upgrade to angular 10 2020-08-07 08:28:14 +02:00
Chocobozzz d95d155988 Use 3 tables to represent abuses 2020-07-10 14:02:41 +02:00
Rigel Kent b8b3abac6a
fix change ownership typing 2020-06-28 22:45:27 +02:00
Chocobozzz 1942f11d5e Lazy load all routes 2020-06-23 16:00:49 +02:00
Chocobozzz 67ed6552b8 Reorganize client shared modules 2020-06-23 16:00:49 +02:00
Rigel Kent 1ebddadd07
predefined report reasons & improved reporter UI (#2842)
- added `startAt` and `endAt` optional timestamps to help pin down reported sections of a video
- added predefined report reasons
- added video player with report modal
2020-06-22 13:00:39 +02:00
Chocobozzz 49919ca16b
More robust max bitrate calculation 2020-06-17 09:22:56 +02:00
Chocobozzz 1ba471c55f
Add videos count in channels list 2020-06-16 14:13:01 +02:00
Chocobozzz 5fb2e2888c First implem global search 2020-06-10 14:02:41 +02:00
Rigel Kent 2158ac9034
Add server plugin filter hooks for import with torrent and url (#2621)
* Add server plugin filter hooks for import with torrent and url

* WIP: pre and post-import filter hooks

* Rebased

* Cleanup filters to accept imports

Co-authored-by: Chocobozzz <me@florianbigard.com>
2020-05-14 11:10:26 +02:00