Commit Graph

570 Commits (f27f392721e140e8465922f847b53046eda717f7)

Author SHA1 Message Date
Chocobozzz d12e074621
Bumped to version v6.3.0-rc.1 2024-09-06 10:37:34 +02:00
Chocobozzz c255ea92cf
Update server dependencies 2024-08-16 14:27:49 +02:00
Chocobozzz d24d221550
Add SSRF protection 2024-08-14 15:33:57 +02:00
Chocobozzz 90859f2b61
Reduce peertube types package size 2024-08-13 09:38:35 +02:00
Chocobozzz 6d28305582
Add install-node-dependencies npm script wrapper
Will help us to change the package manager in the future
2024-08-08 09:37:46 +02:00
Chocobozzz 17cd564875
Bumped to version v6.2.1 2024-08-01 10:19:55 +02:00
Chocobozzz f7ec8401c5
Revert webtorrent version to avoid native dep
Causes issues on some system:
 * https://github.com/Chocobozzz/PeerTube/issues/6518
 * https://github.com/Chocobozzz/PeerTube/issues/6486
 * https://github.com/Chocobozzz/PeerTube/issues/6502
2024-07-31 15:05:36 +02:00
Chocobozzz 3d4d49a23e
Bumped to version v6.2.0 2024-07-16 09:00:02 +02:00
Chocobozzz fb5236f2af
Bumped to version v6.2.0-rc.1 2024-07-03 09:31:32 +02:00
Chocobozzz 9ee467b9cd
Better whisper logging 2024-07-02 09:01:54 +02:00
lutangar ef14cf4a5c
feat(transcription): groundwork
chore: fiddling around some more

chore: add ctranslate2 and timestamped

chore: add performance markers

chore: refactor test

chore: change worflow name

chore: ensure Python3

chore(duration): convert to chai/mocha syntahx

chore(transcription): add individual tests for others transcribers

chore(transcription): implement formats test of all implementations

Also compare result of other implementation to the reference implementation

chore(transcription): add more test case with other language and models size and local model

chore(test): wip ctranslate 2 adapat

chore(transcription): wip transcript file and benchmark

chore(test): clean a bit

chore(test): clean a bit

chore(test): refacto timestamed spec

chore(test): update workflow

chore(test): fix glob expansion with sh

chore(test): extract some hw info

chore(test): fix async tests

chore(benchmark): add model info

feat(transcription): allow use of a local mode in timestamped-whisper

feat(transcription): extract run and profiling info in own value object

feat(transcription): extract run concept in own class an run more bench

chore(transcription): somplify run object only a uuid is now needed and add more benchmark scenario

docs(transcription): creates own package readme

docs(transcription): add local model usage

docs(transcription): update README

fix(transcription): use fr video for better comparison

chore(transcription): make openai comparison passed

docs(timestamped): clea

chore(transcription): change transcribers transcribe method signature

Introduce whisper builtin model.

fix(transcription): activate language detection

Forbid transcript creation without a language.
Add `languageDetection` flag to an engine and some assertions.

Fix an issue in `whisper-ctranslate2` :
https://github.com/Softcatala/whisper-ctranslate2/pull/93

chore(transcription): use PeerTube time helpers instead of custom ones

Update existing time function to output an integer number of seconds and add a ms human-readable time formatter with hints of tests.

chore(transcription): use PeerTube UUID helpers

chore(transcription): enable CER evaluation

Thanks to this recent fix in Jiwer <3
https://github.com/jitsi/jiwer/issues/873

chore(jiwer): creates JiWer package

I'm not very happy with the TranscriptFileEvaluator constructor... suggestions ?

chore(JiWer): add usage in README

docs(jiwer): update JiWer readme

chore(transcription): use FunMOOC video in fixtures

chore(transcription): add proper english video fixture

chore(transcription): use os tmp directory where relevant

chore(transcription): fix jiwer cli test reference.txt

chore(transcription): move benchmark out of tests

chore(transcription): remove transcription workflow

docs(transcription): add benchmark info

fix(transcription): use ms precision in other transcribers

chore(transcription): simplify most of the tests

chore(transcription): remove slashes when building path with join

chore(transcription): make fromPath method async

chore(transcription): assert path to model is a directory for CTranslate2 transcriber

chore(transcription): ctranslate2 assertion

chore(transcription): ctranslate2 assertion

chore(transcription): add preinstall script for Python dependencies

chore(transcription): add download and unzip utils functions

chore(transcription): add download and unzip utils functions

chore(transcription): download & unzip models fixtures

chore(transcription): zip

chore(transcription): raise download file test timeout

chore(transcription): simplify download file test

chore(transcription): add transcriptions test to CI

chore(transcription): raise test preconditions timeout

chore(transcription): run preinstall scripts before running ci

chore(transcription): create dedicated tmp folder for transcriber tests

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): use short video for local model test

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): setup verbosity based on NODE_ENV value
2024-06-28 08:43:40 +02:00
Chocobozzz 2728810f60
Remove bundlewatch
CI fails, our projects generates too many chunks unfortunately
2024-06-26 14:20:45 +02:00
Chocobozzz a722194809
Update server dependencies 2024-06-21 14:39:50 +02:00
Chocobozzz 2b189131fa
Add house-keeping script 2024-06-05 15:35:58 +02:00
Chocobozzz 3427330611
Support update object storage urls 2024-06-05 09:01:40 +02:00
Chocobozzz 54c140c800
Support object storage in prune script
Also prune original files and user exports
2024-06-03 16:37:44 +02:00
Chocobozzz 29329d6c45 Implement auto tag on comments and videos
* Comments and videos can be automatically tagged using core rules or
   watched word lists
 * These tags can be used to automatically filter videos and comments
 * Introduce a new video comment policy where comments must be approved
   first
 * Comments may have to be approved if the user auto block them using
   core rules or watched word lists
 * Implement FEP-5624 to federate reply control policies
2024-05-29 15:03:14 +02:00
Chocobozzz 46b45dc51d
Bumped to version v6.1.0 2024-04-29 09:36:33 +02:00
Chocobozzz 712f7d18e6
Update superagent
To fix vulnerability of formidable
2024-04-25 11:01:10 +02:00
Chocobozzz 8e30e1f5fd
Bumped to version v6.1.0-rc.1 2024-04-16 14:42:41 +02:00
Chocobozzz c67dfb979b
Support itunes:owner in podcast feed 2024-04-11 09:32:57 +02:00
Chocobozzz 7163a1a044
Fix self link in atom feed 2024-04-05 08:50:38 +02:00
Chocobozzz d1e634acff
Bumped to version v6.0.4 2024-04-04 16:29:46 +02:00
Chocobozzz c8d1f0d9c5
Remove unused dependencies 2024-02-29 14:20:28 +01:00
Chocobozzz db06d13c67
Fix typescript tests 2024-02-23 08:33:01 +01:00
Chocobozzz 6eda5c36f4
Update server dependencies 2024-02-21 16:23:31 +01:00
Chocobozzz 8573e5a80a Implement user import/export in server 2024-02-21 13:49:08 +01:00
Chocobozzz 01e4fd067e
Bumped to version v6.0.3 2024-01-18 09:38:18 +01:00
Chocobozzz 1aab639928
Bumped to version v6.0.2 2023-12-11 09:09:56 +01:00
Chocobozzz d897a04565
Bumped to version v6.0.1 2023-11-29 10:20:14 +01:00
Chocobozzz c900788e59
Remove async-lru cache for jsonld
jsonld library has already a cache
2023-11-29 06:51:29 +01:00
Chocobozzz e3d4259e14
Bumped to version v6.0.0 2023-11-28 08:40:29 +01:00
Chocobozzz 76fc448ab4
Bumped to version v6.0.0-rc.2 2023-11-20 10:44:50 +01:00
Chocobozzz 5dd39c07a7
Bumped to version v6.0.0-rc.1 2023-11-07 08:49:09 +01:00
Chocobozzz 44e12dc809
Update bullmq dependency 2023-10-12 09:29:45 +02:00
Chocobozzz 678ae8abbd
Breaking: only support node 18
Node 16 is not supported anymore

See https://nodejs.org/fr/blog/announcements/nodejs16-eol
2023-10-11 13:59:24 +02:00
Chocobozzz 2db1f3238d
Update server dependencies 2023-10-11 11:06:34 +02:00
Chocobozzz d2640682f6
Fix atom feed with science and technology category 2023-09-01 10:09:10 +02:00
Chocobozzz 76dbe843d1
Merge branch 'release/5.2.0' into develop 2023-08-30 19:03:03 +02:00
Chocobozzz 0c302acb3c
Bumped to version v5.2.1 2023-08-30 18:35:48 +02:00
Chocobozzz 273d57023b
Update translations 2023-08-18 14:12:32 +02:00
Chocobozzz 3a4992633e
Migrate server to ESM
Sorry for the very big commit that may lead to git log issues and merge
conflicts, but it's a major step forward:

 * Server can be faster at startup because imports() are async and we can
   easily lazy import big modules
 * Angular doesn't seem to support ES import (with .js extension), so we
   had to correctly organize peertube into a monorepo:
    * Use yarn workspace feature
    * Use typescript reference projects for dependencies
    * Shared projects have been moved into "packages", each one is now a
      node module (with a dedicated package.json/tsconfig.json)
    * server/tools have been moved into apps/ and is now a dedicated app
      bundled and published on NPM so users don't have to build peertube
      cli tools manually
    * server/tests have been moved into packages/ so we don't compile
      them every time we want to run the server
 * Use isolatedModule option:
   * Had to move from const enum to const
     (https://www.typescriptlang.org/docs/handbook/enums.html#objects-vs-enums)
   * Had to explictely specify "type" imports when used in decorators
 * Prefer tsx (that uses esbuild under the hood) instead of ts-node to
   load typescript files (tests with mocha or scripts):
     * To reduce test complexity as esbuild doesn't support decorator
       metadata, we only test server files that do not import server
       models
     * We still build tests files into js files for a faster CI
 * Remove unmaintained peertube CLI import script
 * Removed some barrels to speed up execution (less imports)
2023-08-11 15:02:33 +02:00
Chocobozzz 881958d179
Correctly download files from S3
Use pre signed URLs to inject content disposition header
2023-06-29 10:18:21 +02:00
Chocobozzz 9c5cc50133
Add script to generate storyboards 2023-06-29 10:16:57 +02:00
Chocobozzz b4516abc39
Bumped to version v5.2.0 2023-06-21 09:06:13 +02:00
Chocobozzz 9a64621975
Bumped to version v5.2.0-rc.1 2023-05-30 10:18:25 +02:00
Chocobozzz 51afea5411
Update client dependencies 2023-05-25 09:24:55 +02:00
Chocobozzz 83002a8234
Update server dependencies 2023-05-22 17:11:31 +02:00
Alecks Gates cb0eda5602
Add Podcast RSS feeds (#5487)
* Initial test implementation of Podcast RSS

This is a pretty simple implementation to add support for The Podcast Namespace in RSS -- instead of affecting the existing RSS implementation, this adds a new UI option.

I attempted to retain compatibility with the rest of the RSS feed implementation as much as possible and have created a temporary fork of the "pfeed" library to support this effort.

* Update to pfeed-podcast 1.2.2

* Initial test implementation of Podcast RSS

This is a pretty simple implementation to add support for The Podcast Namespace in RSS -- instead of affecting the existing RSS implementation, this adds a new UI option.

I attempted to retain compatibility with the rest of the RSS feed implementation as much as possible and have created a temporary fork of the "pfeed" library to support this effort.

* Update to pfeed-podcast 1.2.2

* Initial test implementation of Podcast RSS

This is a pretty simple implementation to add support for The Podcast Namespace in RSS -- instead of affecting the existing RSS implementation, this adds a new UI option.

I attempted to retain compatibility with the rest of the RSS feed implementation as much as possible and have created a temporary fork of the "pfeed" library to support this effort.

* Update to pfeed-podcast 1.2.2

* Add correct feed image to RSS channel

* Prefer HLS videos for podcast RSS

Remove video/stream titles, add optional height attribute to podcast RSS

* Prefix podcast RSS images with root server URL

* Add optional video query support to include captions

* Add transcripts & person images to podcast RSS feed

* Prefer webseed/webtorrent files over HLS fragmented mp4s

* Experimentally adding podcast fields to basic config page

* Add validation for new basic config fields

* Don't include "content" in podcast feed, use full description for "description"

* Initial test implementation of Podcast RSS

This is a pretty simple implementation to add support for The Podcast Namespace in RSS -- instead of affecting the existing RSS implementation, this adds a new UI option.

I attempted to retain compatibility with the rest of the RSS feed implementation as much as possible and have created a temporary fork of the "pfeed" library to support this effort.

* Update to pfeed-podcast 1.2.2

* Add correct feed image to RSS channel

* Prefer HLS videos for podcast RSS

Remove video/stream titles, add optional height attribute to podcast RSS

* Prefix podcast RSS images with root server URL

* Add optional video query support to include captions

* Add transcripts & person images to podcast RSS feed

* Prefer webseed/webtorrent files over HLS fragmented mp4s

* Experimentally adding podcast fields to basic config page

* Add validation for new basic config fields

* Don't include "content" in podcast feed, use full description for "description"

* Add medium/socialInteract to podcast RSS feeds. Use HTML for description

* Change base production image to bullseye, install prosody in image

* Add liveItem and trackers to Podcast RSS feeds

Remove height from alternateEnclosure, replaced with title.

* Clear Podcast RSS feed cache when live streams start/end

* Upgrade to Node 16

* Refactor clearCacheRoute to use ApiCache

* Remove unnecessary type hint

* Update dockerfile to node 16, install python-is-python2

* Use new file paths for captions/playlists

* Fix legacy videos in RSS after migration to object storage

* Improve method of identifying non-fragmented mp4s in podcast RSS feeds

* Don't include fragmented MP4s in podcast RSS feeds

* Add experimental support for podcast:categories on the podcast RSS item

* Fix undefined category when no videos exist

Allows for empty feeds to exist (important for feeds that might only go live)

* Add support for podcast:locked -- user has to opt in to show their email

* Use comma for podcast:categories delimiter

* Make cache clearing async

* Fix merge, temporarily test with pfeed-podcast

* Syntax changes

* Add EXT_MIMETYPE constants for captions

* Update & fix tests, fix enclosure mimetypes, remove admin email

* Add test for podacst:socialInteract

* Add filters hooks for podcast customTags

* Remove showdown, updated to pfeed-podcast 6.1.2

* Add 'action:api.live-video.state.updated' hook

* Avoid assigning undefined category to podcast feeds

* Remove nvmrc

* Remove comment

* Remove unused podcast config

* Remove more unused podcast config

* Fix MChannelAccountDefault type hint missed in merge

* Remove extra line

* Re-add newline in config

* Fix lint errors for isEmailPublic

* Fix thumbnails in podcast feeds

* Requested changes based on review

* Provide podcast rss 2.0 only on video channels

* Misc cleanup for a less messy PR

* Lint fixes

* Remove pfeed-podcast

* Add peertube version to new hooks

* Don't use query include, remove TODO

* Remove film medium hack

* Clear podcast rss cache before video/channel update hooks

* Clear podcast rss cache before video uploaded/deleted hooks

* Refactor podcast feed cache clearing

* Set correct person name from video channel

* Styling

* Fix tests

---------

Co-authored-by: Chocobozzz <me@florianbigard.com>
2023-05-22 16:00:05 +02:00
Chocobozzz 1772b383de Add peertube runner cli 2023-05-09 08:57:34 +02:00