Commit Graph

3942 Commits (364783e3038cbdddc86822230fb1e55ebeafe689)

Author SHA1 Message Date
Chocobozzz d24d221550
Add SSRF protection 2024-08-14 15:33:57 +02:00
Chocobozzz 8fe17abb7c
Prevent headers already sent error 2024-08-14 14:21:53 +02:00
Chocobozzz e6de476ae8
Add ability to disable federation 2024-08-14 14:12:37 +02:00
Chocobozzz 38dbafcf63
Support youtube-dl executables
Also disable youtube tests in CI due to youtube blocking
2024-08-13 14:20:08 +02:00
Chocobozzz 5439a591e0
Add ability to set proxies to youtube-dl 2024-08-13 10:49:31 +02:00
Chocobozzz 766f2cb76a
Relax federation logs 2024-08-13 09:38:37 +02:00
Chocobozzz bbaf96d60d
Support max FPS configuration 2024-08-13 09:38:36 +02:00
Chocobozzz 90859f2b61
Reduce peertube types package size 2024-08-13 09:38:35 +02:00
Chocobozzz 25684e837c
Fix client player error on fast restream 2024-08-09 10:00:34 +02:00
Chocobozzz 62bf86c186
Introduce bufferStalled playback metric 2024-08-08 14:41:34 +02:00
Chocobozzz 658241d8c6
Support cover when downloading audio 2024-08-08 10:42:15 +02:00
Chocobozzz 56d48fc41b
Notify local users with internal video privacy 2024-08-07 09:26:07 +02:00
Chocobozzz 21215122a8
Better resolution label for custom video aspect 2024-08-06 16:01:38 +02:00
Chocobozzz b5fd7881a0
Fix transcoding flow 2024-08-06 14:30:08 +02:00
Chocobozzz 546265e9ae
Support fowarding query params to oembed
For now only `start` for videos and `playlistPosition` for playlists are
supported
2024-08-06 11:52:21 +02:00
Chocobozzz 5e82c1932e
Correctly set tags to replays of permanent lives 2024-08-05 16:23:44 +02:00
Chocobozzz 486183fe62
Speed up built-in db migration 2024-08-05 16:23:39 +02:00
Chocobozzz 816f346a60 Separate HLS audio and video streams
Allows:
  * The HLS player to propose an "Audio only" resolution
  * The live to output an "Audio only" resolution
  * The live to ingest and output an "Audio only" stream

 This feature is under a config for VOD videos and is enabled by default for lives

 In the future we can imagine:
  * To propose multiple audio streams for a specific video
  * To ingest an audio only VOD and just output an audio only "video"
    (the player would play the audio file and PeerTube would not
    generate additional resolutions)

This commit introduce a new way to download videos:
 * Add "/download/videos/generate/:videoId" endpoint where PeerTube can
   mux an audio only and a video only file to a mp4 container
 * The download client modal introduces a new default panel where the
   user can choose resolutions it wants to download
2024-08-05 08:31:59 +02:00
Chocobozzz f7ec8401c5
Revert webtorrent version to avoid native dep
Causes issues on some system:
 * https://github.com/Chocobozzz/PeerTube/issues/6518
 * https://github.com/Chocobozzz/PeerTube/issues/6486
 * https://github.com/Chocobozzz/PeerTube/issues/6502
2024-07-31 15:05:36 +02:00
Chocobozzz 68b7a69f99
Don't break the API 2024-07-31 13:23:28 +02:00
Chocobozzz bba6538435
Truncate chapters from youtube-dl 2024-07-31 10:13:28 +02:00
Chocobozzz 22e9d9a1fe
Fix updating hls infohash on privacy update 2024-07-31 09:26:54 +02:00
Chocobozzz 44919a8d57
Fix hidden runner jobs tab
If remote runners feature is only enabled on transcription
2024-07-31 08:56:33 +02:00
Chocobozzz 71857cc04b
More robust doesVideoExistForVideoFile 2024-07-31 08:55:56 +02:00
Chocobozzz a9d08d2646
Correctly handle invalid current password 2024-07-31 08:35:24 +02:00
Chocobozzz d0304f6712
Don't check current password param
We may have a current password that does not satisfy new password
constraints
2024-07-31 08:25:50 +02:00
Chocobozzz 4342ec8525
Comment for torrent consistency issues 2024-07-29 08:55:33 +02:00
Chocobozzz 18f13f43ec
Fix "Correctly save transaction with retries" 2024-07-24 15:20:03 +02:00
Chocobozzz 89e3951587
Correctly save transaction with retries 2024-07-23 16:38:28 +02:00
Chocobozzz 300676f62b
Don't replace caption on transcription 2024-07-12 16:20:26 +02:00
Chocobozzz e99e4aa93c
Run transcription after studio 2024-07-11 11:29:46 +02:00
Chocobozzz c5fa9fe70e
More robust updateTorrentMetadata fn 2024-07-11 08:07:30 +02:00
Chocobozzz c5de5ef1d0
Fix registering unamed settings 2024-07-10 12:12:07 +02:00
Chocobozzz 471bf5bbce
Don't block video files on transcription 2024-07-04 11:18:04 +02:00
Chocobozzz 8ab6f23a00
Don't block video update on storyboard generation 2024-07-03 15:40:10 +02:00
Chocobozzz f5104fb234
Don't transcribe/encode videos with invalid state 2024-07-03 09:09:20 +02:00
Chocobozzz 2d26eff129
Don't create pendingTranscription twice 2024-07-03 08:52:05 +02:00
Chocobozzz f81bc3d3dc
Relax client log rate limiting 2024-07-02 09:43:07 +02:00
Chocobozzz 01b3396c80
Better view handling with looped videos 2024-07-02 09:41:33 +02:00
Chocobozzz ebcf3329f5
More robust transcription error job handler 2024-07-01 14:38:19 +02:00
Chocobozzz b45ed3c05a
Do not run transcription/transcoding on lives 2024-07-01 11:15:09 +02:00
Chocobozzz 3a71086e35
Correctly handle transcription conflicts 2024-06-28 14:42:19 +02:00
Grant 4bc29f6d3f Add Express req, res to filter:api.search.videos.local.list.params 2024-06-28 09:20:39 +02:00
Chocobozzz fd4831e502
Metadata to know if the caption is auto generated 2024-06-28 08:44:59 +02:00
Chocobozzz 1bfb791e05
Integrate transcription in PeerTube 2024-06-28 08:44:58 +02:00
lutangar ef14cf4a5c
feat(transcription): groundwork
chore: fiddling around some more

chore: add ctranslate2 and timestamped

chore: add performance markers

chore: refactor test

chore: change worflow name

chore: ensure Python3

chore(duration): convert to chai/mocha syntahx

chore(transcription): add individual tests for others transcribers

chore(transcription): implement formats test of all implementations

Also compare result of other implementation to the reference implementation

chore(transcription): add more test case with other language and models size and local model

chore(test): wip ctranslate 2 adapat

chore(transcription): wip transcript file and benchmark

chore(test): clean a bit

chore(test): clean a bit

chore(test): refacto timestamed spec

chore(test): update workflow

chore(test): fix glob expansion with sh

chore(test): extract some hw info

chore(test): fix async tests

chore(benchmark): add model info

feat(transcription): allow use of a local mode in timestamped-whisper

feat(transcription): extract run and profiling info in own value object

feat(transcription): extract run concept in own class an run more bench

chore(transcription): somplify run object only a uuid is now needed and add more benchmark scenario

docs(transcription): creates own package readme

docs(transcription): add local model usage

docs(transcription): update README

fix(transcription): use fr video for better comparison

chore(transcription): make openai comparison passed

docs(timestamped): clea

chore(transcription): change transcribers transcribe method signature

Introduce whisper builtin model.

fix(transcription): activate language detection

Forbid transcript creation without a language.
Add `languageDetection` flag to an engine and some assertions.

Fix an issue in `whisper-ctranslate2` :
https://github.com/Softcatala/whisper-ctranslate2/pull/93

chore(transcription): use PeerTube time helpers instead of custom ones

Update existing time function to output an integer number of seconds and add a ms human-readable time formatter with hints of tests.

chore(transcription): use PeerTube UUID helpers

chore(transcription): enable CER evaluation

Thanks to this recent fix in Jiwer <3
https://github.com/jitsi/jiwer/issues/873

chore(jiwer): creates JiWer package

I'm not very happy with the TranscriptFileEvaluator constructor... suggestions ?

chore(JiWer): add usage in README

docs(jiwer): update JiWer readme

chore(transcription): use FunMOOC video in fixtures

chore(transcription): add proper english video fixture

chore(transcription): use os tmp directory where relevant

chore(transcription): fix jiwer cli test reference.txt

chore(transcription): move benchmark out of tests

chore(transcription): remove transcription workflow

docs(transcription): add benchmark info

fix(transcription): use ms precision in other transcribers

chore(transcription): simplify most of the tests

chore(transcription): remove slashes when building path with join

chore(transcription): make fromPath method async

chore(transcription): assert path to model is a directory for CTranslate2 transcriber

chore(transcription): ctranslate2 assertion

chore(transcription): ctranslate2 assertion

chore(transcription): add preinstall script for Python dependencies

chore(transcription): add download and unzip utils functions

chore(transcription): add download and unzip utils functions

chore(transcription): download & unzip models fixtures

chore(transcription): zip

chore(transcription): raise download file test timeout

chore(transcription): simplify download file test

chore(transcription): add transcriptions test to CI

chore(transcription): raise test preconditions timeout

chore(transcription): run preinstall scripts before running ci

chore(transcription): create dedicated tmp folder for transcriber tests

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): use short video for local model test

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): setup verbosity based on NODE_ENV value
2024-06-28 08:43:40 +02:00
Chocobozzz 43e186ef44
Add views tag to middlewares too 2024-06-27 10:51:36 +02:00
Chocobozzz 9772280e99
Upgrade to angular 18 & vite 2024-06-26 08:33:54 +02:00
Chocobozzz bc8c853a53
Fix lint 2024-06-24 09:48:57 +02:00
Chocobozzz 5412465df4
Fix build 2024-06-24 09:20:32 +02:00
Chocobozzz 209043e2d4
Fix legacy upload req timeout 2024-06-24 09:10:11 +02:00
Chocobozzz 05d84f659d
Fix loading actor involved in video 2024-06-21 16:05:40 +02:00
Chocobozzz 985e79f61e
Fix lint and tests 2024-06-21 15:28:09 +02:00
Chocobozzz a722194809
Update server dependencies 2024-06-21 14:39:50 +02:00
Chocobozzz 990266f149
Owner must not be able to approve its own comment 2024-06-21 10:27:03 +02:00
kontrollanten ab22399da0 feat(plugins/filter): add req to api.video.get.result 2024-06-21 07:01:02 +02:00
Chocobozzz 802601c241
Fix channel update federation 2024-06-20 08:48:24 +02:00
Chocobozzz 0d0a965f8a
Fix detecting account actor 2024-06-19 17:37:27 +02:00
Chocobozzz 346be1d478
Support Service AP actors 2024-06-19 17:08:59 +02:00
kontrollanten ca26687e1f
feat: config option object_storage.max_request_attempts (#6418)
* feat: config option object_storage.max_attempts

Backblaze recommends to have a high amount of attempts since they've
designed their architecture so that it will return 5xx errors to
indicate that the client should do a new attempt.

https://www.backblaze.com/blog/b2-503-500-server-error/

closes #6415

* Rephrase comment

---------

Co-authored-by: Chocobozzz <me@florianbigard.com>
2024-06-11 13:57:18 +02:00
Chocobozzz 633039c1fa
Add concurrency to delete remote files 2024-06-06 11:33:09 +02:00
Chocobozzz 585050821a
Also prune storyboards and actor images 2024-06-05 15:47:37 +02:00
Chocobozzz 2b189131fa
Add house-keeping script 2024-06-05 15:35:58 +02:00
Chocobozzz 3427330611
Support update object storage urls 2024-06-05 09:01:40 +02:00
Chocobozzz 96b9748585
Support CLI move of original video file 2024-06-04 09:08:24 +02:00
Chocobozzz 54c140c800
Support object storage in prune script
Also prune original files and user exports
2024-06-03 16:37:44 +02:00
Chocobozzz 568a1b1e85
Fix `segmentsSha256Url` in export 2024-06-03 09:44:12 +02:00
Chocobozzz 7c9f07e140
Improve `Video` AP compatibility
Compat with text/html descriptions
Compat with SPDX for licences
Compat with missing sensitive attribute
Compat with missing tag attribute
Compat with missing video file magnet URI
Compat with missing streaming playlist segmentsSha256Url
Compat with optional comments/likes/dislikes/shares URI in video object

Add more debug logs when the object is not valid
2024-06-03 09:36:29 +02:00
Chocobozzz 1e3a5b25c3
Plan to have multiple sizes in the future 2024-05-31 09:02:34 +02:00
Gergo Bogdan 3fbe849183
#5618 - added configuration fields for Thumbnails and Previews, if no config is provided, the previous, default values are configured 2024-05-31 08:57:23 +02:00
Chocobozzz 6a4db88e5c
Handle 410 HTTP response code for AP objects 2024-05-30 10:28:20 +02:00
Chocobozzz 671c6c1f96
Fix auto tags migration 2024-05-29 15:06:08 +02:00
Chocobozzz 29329d6c45 Implement auto tag on comments and videos
* Comments and videos can be automatically tagged using core rules or
   watched word lists
 * These tags can be used to automatically filter videos and comments
 * Introduce a new video comment policy where comments must be approved
   first
 * Comments may have to be approved if the user auto block them using
   core rules or watched word lists
 * Implement FEP-5624 to federate reply control policies
2024-05-29 15:03:14 +02:00
Chocobozzz b65d639695
Don't include banners in account API 2024-05-29 09:00:43 +02:00
Chocobozzz e2c354878f
Fix "reading 'isBlacklisted' of undefined" 2024-05-29 08:57:13 +02:00
Chocobozzz 4ea659d569
Handle videos with FPS < 1 2024-05-29 08:56:53 +02:00
Chocobozzz b7e07bde8c
Update search REST API doc 2024-05-28 09:05:35 +02:00
Chocobozzz 9b5c7a8dc6
Merge branch 'release/6.1.0' into develop 2024-05-17 11:36:43 +02:00
Chocobozzz 74b62dcb85
Remove invalid db value checker
fileUrl can use localhost in some specific cases for example
2024-05-17 11:18:43 +02:00
Chocobozzz cb88f25ffa
Don't count deleted comments for stats 2024-05-17 11:15:37 +02:00
Chocobozzz 9ee1f7b57c
Don't mark as ready already read notifications 2024-05-17 11:07:30 +02:00
Chocobozzz 9784e93dc7
Don't import all lodash 2024-05-17 11:04:25 +02:00
kontrollanten 1eb8fc2c06 fix(server/plugins): avoid duplicate settings
Filter settings so that the name property is unique.

closes #6356
2024-05-17 10:43:47 +02:00
kontrollanten 5dfa07adb5 fix(server/geo-ip): throttle db download 2024-05-17 09:42:25 +02:00
Chocobozzz 5dd0721bee
Merge branch 'release/6.1.0' into develop 2024-05-16 11:39:53 +02:00
Chocobozzz 3d105296e6
Fix concurrent live streams serialization issue 2024-05-16 11:38:36 +02:00
Chocobozzz e7ebcb16ac
Fix big user export file size 2024-05-16 11:18:50 +02:00
Chocobozzz 81d4d557f7
Remove internal log 2024-05-16 11:15:43 +02:00
kontrollanten 2177995056 feat(server): dont client cache HTML files
Tell the clients to not cache the HTML response.

closes #6393
2024-05-13 11:17:50 +02:00
Chocobozzz 1642c5b9e7
Protect all video related AP endpoints 2024-04-26 15:29:52 +02:00
Chocobozzz d72ef2a2b9
Fix federation when updating video privacy 2024-04-26 10:30:43 +02:00
Chocobozzz afb28272f5
Check video rights before providing AP information 2024-04-26 10:09:38 +02:00
Chocobozzz b8635c2606
Compact json-ld AP objects 2024-04-25 11:21:55 +02:00
Chocobozzz 47ae6e880d
Fix AP actor follows count 2024-04-25 10:53:53 +02:00
Chocobozzz b824480af7
Fix video source size column on db migration 2024-04-22 14:41:46 +02:00
Chocobozzz bce0f2f11b
Fix total video file size 2024-04-22 11:48:05 +02:00
Chocobozzz c67dfb979b
Support itunes:owner in podcast feed 2024-04-11 09:32:57 +02:00
Chocobozzz f84ac3ba41
Fix local video creator crash 2024-04-05 08:29:23 +02:00
Chocobozzz 5cb3e6a0b8
Use sessionId instead of IP to identify viewer
Breaking: YAML config `ip_view_expiration` is renamed `view_expiration`
Breaking: Views are taken into account after 10 seconds instead of 30
seconds (can be changed in YAML config)

Purpose of this commit is to get closer to other video platforms where
some platforms count views on play (mux, vimeo) or others use a very low
delay (instagram, tiktok)

We also want to improve the viewer identification, where we no longer
use the IP but the `sessionId` generated by the web browser. Multiple
viewers behind a NAT can now be able to be identified as independent
viewers (this method is also used by vimeo or mux)
2024-04-04 16:27:40 +02:00
kontrollanten 6f6abcabfb
feat(plugins): add peertubeHelpers.loadByIdOrUUIDWithFiles (#6302) 2024-04-04 08:20:09 +02:00