OpenCloud
/
PeerTube
mirror of https://github.com/Chocobozzz/PeerTube
1
0
Fork 0
Code Issues Releases Wiki Activity
7,924 Commits
4 Branches
131 Tags
494 MiB
Commit Graph

37 Commits (0305db28c98fd6cf43a3c50ba92c76215e99d512)

Author SHA1 Message Date
Jelle Besseling 0305db28c9
Add support for saving video files to object storage (#4290) 
* Add support for saving video files to object storage

* Add support for custom url generation on s3 stored files

Uses two config keys to support url generation that doesn't directly go
to (compatible s3). Can be used to generate urls to any cache server or
CDN.

* Upload files to s3 concurrently and delete originals afterwards

* Only publish after move to object storage is complete

* Use base url instead of url template

* Fix mistyped config field

* Add rudenmentary way to download before transcode

* Implement Chocobozzz suggestions

https://github.com/Chocobozzz/PeerTube/pull/4290#issuecomment-891670478

The remarks in question:
    Try to use objectStorage prefix instead of s3 prefix for your function/variables/config names
    Prefer to use a tree for the config: s3.streaming_playlists_bucket -> object_storage.streaming_playlists.bucket
    Use uppercase for config: S3.STREAMING_PLAYLISTS_BUCKETINFO.bucket -> OBJECT_STORAGE.STREAMING_PLAYLISTS.BUCKET (maybe BUCKET_NAME instead of BUCKET)
    I suggest to rename moveJobsRunning to pendingMovingJobs (or better, create a dedicated videoJobInfo table with a pendingMove & videoId columns so we could also use this table to track pending transcoding jobs)
    https://github.com/Chocobozzz/PeerTube/pull/4290/files#diff-3e26d41ca4bda1de8e1747af70ca2af642abcc1e9e0bfb94239ff2165acfbde5R19 uses a string instead of an integer
    I think we should store the origin object storage URL in fileUrl, without base_url injection. Instead, inject the base_url at "runtime" so admins can easily change this configuration without running a script to update DB URLs

* Import correct function

* Support multipart upload

* Remove import of node 15.0 module stream/promises

* Extend maximum upload job length

Using the same value as for redundancy downloading seems logical

* Use dynamic part size for really large uploads

Also adds very small part size for local testing

* Fix decreasePendingMove query

* Resolve various PR comments

* Move to object storage after optimize

* Make upload size configurable and increase default

* Prune webtorrent files that are stored in object storage

* Move files after transcoding jobs

* Fix federation

* Add video path manager

* Support move to external storage job in client

* Fix live object storage tests

Co-authored-by: Chocobozzz <me@florianbigard.com>
 2021-08-17 08:26:20 +02:00
Chocobozzz 83903cb65d Generate random uuid for video files 2021-07-26 11:29:31 +02:00
Chocobozzz 4c7e60bc17
Reorganize imports 2021-07-20 15:27:19 +02:00
Chocobozzz c0e8b12e7f
Refactor requests 2021-07-20 15:27:18 +02:00
Chocobozzz 254d3579f5
Use an object to represent a server 2021-07-20 15:27:18 +02:00
Chocobozzz 89d241a79c
Shorter server command names 2021-07-20 15:27:18 +02:00
Chocobozzz d23dd9fbfc
Introduce videos command 2021-07-20 15:27:18 +02:00
Chocobozzz 6c5065a011
Introduce server commands 2021-07-20 15:27:18 +02:00
Chocobozzz 9293139fde
Introduce sql command 2021-07-20 15:27:18 +02:00
Chocobozzz 57f879a540
Introduce streaming playlists command 2021-07-20 15:27:18 +02:00
Chocobozzz bc8090411d
Introduce stats command 2021-07-20 15:27:17 +02:00
Chocobozzz dab047092b
Introduce redundancy command 2021-07-20 15:27:17 +02:00
Chocobozzz c3d29f694b
Introduce follows command 2021-07-20 15:27:17 +02:00
Chocobozzz 5e8dd6e08a
Fix lint 2021-02-01 16:05:22 +01:00
Chocobozzz 6949a1a111
Fix tests 2021-02-01 15:03:28 +01:00
Chocobozzz 7448551fe5
Fix redundancy with HLS only files 2021-02-01 11:23:12 +01:00
Rigel Kent 2d53be0267
replace numbers with typed http status codes (#3409) 2020-12-07 14:32:36 +01:00
Chocobozzz a15871560f
Move to eslint 2020-02-03 08:31:02 +01:00
Chocobozzz b764380ac2 Add ability to list redundancies 2020-01-28 11:35:26 +01:00
Chocobozzz ffc65cbd2a
Remove HLS torrents 2020-01-24 16:48:05 +01:00
Chocobozzz 97ecddae10
Filter on follows actor types in about page 2019-11-29 10:55:17 +01:00
Chocobozzz 9e3e3617dc
Fix redundancy tests 2019-08-13 10:22:54 +02:00
Chocobozzz 7243f84db0
Redundancy and search tests in parallel too 2019-05-14 15:37:43 +02:00
Chocobozzz 7c3b79768b
Use test wrapper exit function 2019-04-24 16:26:25 +02:00
Chocobozzz 94565d52bb
Shared utils -> extra-utils 
Because they need dev dependencies
 2019-04-24 16:25:52 +02:00
Chocobozzz 0b16f5f220 Add videos playlist exist tests 2019-03-18 11:17:59 +01:00
Chocobozzz 07b1a18aa6 Add playlist check param tests 2019-03-18 11:17:59 +01:00
Chocobozzz 4c280004ce Use a single file instead of segments for HLS 2019-02-11 09:13:02 +01:00
Chocobozzz 0920929696 Add hls support on server 2019-02-11 09:13:02 +01:00
Chocobozzz 2a8c5d0af1
Merge branch 'move-utils-to-shared' of https://github.com/buoyantair/PeerTube into buoyantair-move-utils-to-shared 2018-12-07 16:25:29 +01:00
Chocobozzz b9fffa297f
Create redundancy endpoint 2018-12-04 17:08:55 +01:00
buoyantair b9f234371b Merge branch 'develop' of https://github.com/Chocobozzz/PeerTube into move-utils-to-shared 2018-11-18 21:55:52 +05:30
Chocobozzz 58d515e32f
Fix images size when downloading them 2018-11-16 16:48:17 +01:00
buoyantair ae28cdf327 Merge from upstream 2018-11-16 02:37:16 +05:30
Chocobozzz 6cb3482ceb
Remove wrong redundancy test 2018-11-15 11:20:23 +01:00
buoyantair 9639bd1757 Move utils to /shared 
Move utils used by /server/tools/* & /server/tests/**/* into
/shared folder.

Issue: #1336
 2018-10-29 22:18:31 +05:30
Chocobozzz 26370ce469 Move redundancy in its own travis container 2018-10-16 16:41:36 +02:00