Commit Graph

536 Commits (16d54696294d15f8ab6ba3a6bcfac21528fec2f2)

Author SHA1 Message Date
Chocobozzz 9f1ddd2496
Change a little bit optimize-old-videos logic 2018-10-08 16:50:56 +02:00
Felix Ableitner edb4ffc7e0 Set bitrate limits for transcoding (fixes #638) (#1135)
* Set bitrate limits for transcoding (fixes #638)

* added optimization script and test, changed stuff

* fix test, improve docs

* re-add optimize-old-videos script

* added documentation

* Don't optimize videos without valid UUID, or redundancy videos

* move getUUIDFromFilename

* fix tests?

* update torrent and file size, some more fixes/improvements

* use higher bitrate for high fps video, adjust bitrates

* add test video

* don't throw error if resolution is undefined

* generate test fixture on the fly

* use random noise video for bitrate test, add promise

* shorten test video to avoid timeout

* use existing function to optimize video

* various fixes

* increase test timeout

* limit test fixture size, add link

* test fixes

* add await

* more test fixes, add -b:v parameter

* replace ffmpeg wiki link

* fix ffmpeg params

* fix unit test

* add test fixture to .gitgnore

* add video transcoding fps model

* add missing file
2018-10-08 16:26:04 +02:00
Lucas Declercq 7f2cfe3a79 Rename downloadingEnabled property to downloadEnabled 2018-10-08 14:45:22 +02:00
Chocobozzz ecf3f060ef
Fix avatar update 2018-10-08 10:37:43 +02:00
Chocobozzz 505319061e
Fix avatar update 2018-10-08 10:37:08 +02:00
Lucas Declercq 156c50af30 Add downloadingEnabled property to video model 2018-10-06 19:17:39 +02:00
Chocobozzz 6e46de095d
Add user history and resume videos 2018-10-05 11:22:38 +02:00
Chocobozzz be691a57c5
Check video exists before extending its expiration 2018-10-03 15:52:39 +02:00
Chocobozzz b88a459664
Upsert cache file on create activity 2018-10-02 14:40:03 +02:00
Chocobozzz 659edaa62c
Correctly forward comment deletion 2018-10-02 11:33:18 +02:00
Chocobozzz 792e5b8e5b
Try to improve redundancy tests 2018-10-02 09:04:19 +02:00
Chocobozzz d05be4d944
Don't build redundancy urls with "-1" as fps 2018-10-01 18:40:17 +02:00
Chocobozzz 99cc653f47
Fix redundancy with specific videos 2018-10-01 16:33:53 +02:00
Chocobozzz a3737cbf2b
Fix video fps validator 2018-10-01 16:27:47 +02:00
Chocobozzz d382f4e917
Fix refreshing external video attributes 2018-10-01 15:34:31 +02:00
Chocobozzz 601527d795
Check video channel name is unique on our instance 2018-10-01 15:20:14 +02:00
Chocobozzz cf9166cf2f
Add timeout on youtube dl to cleaup files 2018-10-01 12:00:05 +02:00
Chocobozzz e95e0463d8
Correctly delete files on failed import 2018-10-01 10:44:16 +02:00
Chocobozzz 26649b4215
Ensure video existence before duplicating it 2018-10-01 10:31:42 +02:00
Chocobozzz 46f8d69b4e
Fix redundancy with videos already duplicated with another instance 2018-10-01 09:41:48 +02:00
Chocobozzz 161b061d4e
Remove duplicated videos on unfollow/delete redundancy 2018-09-28 10:07:05 +02:00
Chocobozzz 8578e3b5e6
Fix federation with a comment reply 2018-09-27 11:05:50 +02:00
Chocobozzz 2e7cf5ae0c
Fix video files duplicated when fps is null
Null values are not considered equal in a UNIQUE index
2018-09-26 14:08:35 +02:00
Chocobozzz 4a534352ad
Fix avatar transaction retry 2018-09-26 10:15:50 +02:00
Chocobozzz cfaf819c38
Reduce video import TTL 2018-09-25 17:49:49 +02:00
Chocobozzz ebdb612458
Fix redundancy totalVideos stats 2018-09-25 17:49:48 +02:00
Chocobozzz 47d0b3ee61
Fix comment deletion with mastodon
Will only act with new comments
2018-09-24 16:24:42 +02:00
Chocobozzz e5565833f6
Improve redundancy: add 'min_lifetime' configuration 2018-09-24 13:38:39 +02:00
Chocobozzz 1cfa8d6809
Remove unnecessary actor existance check
We already checked it in the signature checker
2018-09-21 09:51:24 +02:00
Chocobozzz f201a74992
Cache user token 2018-09-20 11:45:59 +02:00
Chocobozzz 91411dba92
Limit associations fetch when loading token 2018-09-20 11:45:59 +02:00
Chocobozzz 606c946e74
Move youtubeDL upgrader in helpers/ 2018-09-19 16:24:24 +02:00
Chocobozzz f9f899b9f8
Add redundancy check interval in config 2018-09-19 16:21:30 +02:00
Chocobozzz d9bdd007d7
Put config redundancy strategies in "strategies" subkey 2018-09-19 16:12:07 +02:00
Chocobozzz 12ba460e9e
Improve AP actor checks 2018-09-19 15:47:55 +02:00
Chocobozzz e587e0ecee
Optimize activity actor load in AP processors 2018-09-19 15:22:55 +02:00
Chocobozzz d4defe07d2
Optimize video view AP processing 2018-09-19 11:41:30 +02:00
Chocobozzz 4157cdb137
Refractor videos AP functions 2018-09-19 11:16:23 +02:00
Chocobozzz 96f29c0f6d
Optimize SQL requests of videos AP endpoints 2018-09-19 10:20:38 +02:00
Chocobozzz 627621c1e8
Optimize SQL requests of watch page API endpoints 2018-09-19 09:54:37 +02:00
Chocobozzz 098eb37797
Reduce video.ts file size by moving some methods in other files 2018-09-19 09:54:35 +02:00
Chocobozzz a2377d15ee
Refractor activities sending 2018-09-14 16:51:35 +02:00
Chocobozzz 7348b1fd84
Speed up overviews route 2018-09-14 11:52:23 +02:00
Chocobozzz 3f6b6a565d
Add recently added redundancy strategy 2018-09-14 11:05:38 +02:00
Chocobozzz b36f41ca09
Add trending videos strategy 2018-09-14 09:57:21 +02:00
Chocobozzz 6f0c46be8c
Process last hour views instead of current hour 2018-09-13 14:14:15 +02:00
Chocobozzz c48e82b5e0 Basic video redundancy implementation 2018-09-13 14:05:49 +02:00
Chocobozzz 9431cabf8a
Fix video views increment 2018-09-13 10:13:25 +02:00
Chocobozzz 5cf84858d4
Add federation to ownership change 2018-09-04 10:49:53 +02:00
Josh Morel d9eaee3939 add user account email verificiation (#977)
* add user account email verificiation

includes server and client code to:

* enable verificationRequired via custom config
* send verification email with registration
* ask for verification email
* verify via email
* prevent login if not verified and required
* conditional client links to ask for new verification email

* allow login for verified=null

these are users created when verification not required
should still be able to login when verification is enabled

* refactor email verifcation pr

* change naming from verified to emailVerified
* change naming from askVerifyEmail to askSendVerifyEmail
* undo unrelated automatic prettier formatting on api/config
* use redirectService for home
* remove redundant success notification on email verified

* revert test.yaml smpt host
2018-08-31 09:18:19 +02:00
Chocobozzz 6b6168606b
Bufferize videos views in redis 2018-08-30 15:03:18 +02:00
Chocobozzz 9ee92651f6
Fix tests 2018-08-28 11:32:03 +02:00
Chocobozzz bfcef50db9
Fix oauth server module 2018-08-28 10:56:09 +02:00
Chocobozzz 62689b942b
Correctly migrate to fs-extra 2018-08-27 16:23:34 +02:00
Rigel Kent c9d5c64f98 replace fs by fs-extra to prevent EMFILE error 2018-08-27 16:03:57 +02:00
Chocobozzz 687d638c2b Fetch outbox when searching an actor 2018-08-27 09:41:54 +02:00
Chocobozzz f5b0af50c8 Search video channel handle/uri 2018-08-27 09:41:54 +02:00
Chocobozzz f37dc0dd14 Add ability to search video channels 2018-08-27 09:41:54 +02:00
Chocobozzz 240085d005 Fix tests 2018-08-27 09:41:54 +02:00
Chocobozzz 83e6519ba4 Refractor comment creation from federation 2018-08-27 09:41:54 +02:00
Chocobozzz 1297eb5db6 Add refresh video on search 2018-08-27 09:41:54 +02:00
Chocobozzz f6eebcb336 Add ability to search a video with an URL 2018-08-27 09:41:54 +02:00
Chocobozzz 22a16e36f6 Add local user subscriptions 2018-08-27 09:41:54 +02:00
Chocobozzz 8a19bee1a1 Add ability to set a name to a channel 2018-08-27 09:41:54 +02:00
Chocobozzz 965c4b22d0 Use height instead of width to represent the video resolution 2018-08-27 09:41:54 +02:00
Chocobozzz 06a05d5f47 Add subscriptions endpoints to REST API 2018-08-27 09:41:54 +02:00
Chocobozzz 4f1f6f0383
Ensure youtubedl binary exists in ydl helper 2018-08-16 09:45:51 +02:00
Chocobozzz 06215f15e0
Cleanup utils helper 2018-08-14 15:28:30 +02:00
Chocobozzz e5e7f7fe99
Add tests when getting a blacklisted video 2018-08-14 09:27:19 +02:00
Chocobozzz 26b7305a23
Add blacklist reason field 2018-08-14 09:27:18 +02:00
Chocobozzz 268eebed92
Add state and moderationComment for abuses on server side 2018-08-10 16:54:01 +02:00
Chocobozzz 141b177db0
Add ability to ban/unban users 2018-08-09 17:55:05 +02:00
Chocobozzz eacb25c436
Add reason when banning a user 2018-08-08 17:44:22 +02:00
Chocobozzz e69219184b
Implement user blocking on server side 2018-08-08 15:22:58 +02:00
Chocobozzz 541006e355 Correct webtorrent download cleanup 2018-08-08 09:30:31 +02:00
Chocobozzz 3e17515e29 Add torrent tests 2018-08-08 09:30:31 +02:00
Chocobozzz a84b8fa5cf Add import.video.torrent configuration 2018-08-08 09:30:31 +02:00
Chocobozzz 990b6a0b0c Import torrents with webtorrent 2018-08-08 09:30:31 +02:00
Chocobozzz ce33919c24 Import magnets with webtorrent 2018-08-08 09:30:31 +02:00
Chocobozzz b2977eecb8 Fix fps federation 2018-08-06 11:19:16 +02:00
Chocobozzz 590fb50690 Add tests regarding video import 2018-08-06 11:19:16 +02:00
Chocobozzz 2b86fe7274 Add job ttl 2018-08-06 11:19:16 +02:00
Chocobozzz 516df59b3b Remove ability to delete video imports
Users should remove the linked video instead
2018-08-06 11:19:16 +02:00
Chocobozzz d7f83948a1 Correctly handle video import errors 2018-08-06 11:19:16 +02:00
Chocobozzz ed31c05985 Add ability to list video imports 2018-08-06 11:19:16 +02:00
Chocobozzz 299474e827 Add ability to remove a video import 2018-08-06 11:19:16 +02:00
Chocobozzz ce32426be9 Auto update youtube-dl 2018-08-06 11:19:16 +02:00
Chocobozzz fbad87b047 Add ability to import video with youtube-dl 2018-08-06 11:19:16 +02:00
Aurélien Bertron 80e36cd9fa Add audit logs in various modules
- Videos
- Videos comments
- Users
- Videos channels
- Videos abuses
- Custom config
2018-07-31 15:40:29 +02:00
Chocobozzz 6104adc3e9
Don't send follow request if the follow is already accepted 2018-07-31 11:15:59 +02:00
Chocobozzz 25e4d6ee97
Refresh orphan actors 2018-07-31 11:04:33 +02:00
Chocobozzz 4a9e71c2b1
Increase max stalled count in job queue 2018-07-30 19:18:01 +02:00
Chocobozzz 14f2b3ad11
Graceful job queue shutdown 2018-07-30 18:49:54 +02:00
Chocobozzz 8e0fd45e14
Add more info logging 2018-07-30 17:02:40 +02:00
Chocobozzz 3ac046e2a4
Fix peertube on chrome 2018-07-26 16:46:36 +02:00
Chocobozzz 2cebd79701
Fix last commit 2018-07-26 11:02:04 +02:00
Rigel Kent c1e791bad0
expliciting type checks and predicates (server only) 2018-07-25 22:01:25 +02:00
Chocobozzz 77de223a5f
Fix publishedAt after a scheduled update 2018-07-24 15:11:28 +02:00
Chocobozzz e032aec9b9
Render CSS/title/description tags on server side 2018-07-18 10:00:37 +02:00
Chocobozzz f4001cf408
Handle .srt subtitles 2018-07-16 14:31:40 +02:00
Chocobozzz 40e87e9ecc Implement captions/subtitles 2018-07-16 11:50:08 +02:00
Rigel Kent 19f7b248d8 adding redis unix connection 2018-07-14 15:00:56 +02:00
Chocobozzz 94831479f5 Migrate to bull 2018-07-11 14:00:17 +02:00
Chocobozzz 4bbfc6c606
API: Add ability to update video channel avatar 2018-06-29 11:29:23 +02:00
Chocobozzz 1b952dd426
Reset published date on publish 2018-06-28 16:58:15 +02:00
Chocobozzz b7f1747dc7
Create webtorrent client on player load
So we don't run Angular change detection every 250ms on webtorrent
import, when https://github.com/mafintosh/speedometer/pull/7 will be
merged
2018-06-18 10:26:18 +02:00
Chocobozzz bbe0f0645c
Add ability to schedule video publication 2018-06-15 18:20:56 +02:00
Chocobozzz 2baea0c77c
Add ability for uploaders to schedule video update 2018-06-14 18:06:56 +02:00
Chocobozzz bcec136ee6
Fix lint 2018-06-13 14:55:18 +02:00
Chocobozzz 90d4bb8125
Refractor retry transaction function 2018-06-13 14:27:40 +02:00
Chocobozzz 3cd0734fd9
Improve tests when waiting pending jobs 2018-06-13 10:22:53 +02:00
Chocobozzz 2186386cca
Add concept of video state, and add ability to wait transcoding before
publishing a video
2018-06-12 20:37:51 +02:00
Chocobozzz 8eb8bc201e
Pick average video file insead of max quality 2018-06-07 17:57:11 +02:00
Chocobozzz 28be89161a
Improve create import file job
Fix federation of .ogv videos
2018-06-07 09:43:18 +02:00
Florent Fayolle 0138af9237 Add create-import-video-file-job command 2018-06-07 08:57:48 +02:00
Chocobozzz 9588d4f49b
Correctly forward video related activities 2018-05-31 10:23:56 +02:00
Chocobozzz 0c948c1659
Add ability to manually run transcoding job 2018-05-30 10:49:56 +02:00
Chocobozzz 53a613171e
Fix publishedAt value after following a new instance 2018-05-28 18:50:44 +02:00
Chocobozzz e3d5ea4f82
Fix actor outbox 2018-05-28 12:13:00 +02:00
Chocobozzz 8fffe21a7b
Refractor and optimize AP collections
Only display urls in general object, and paginate video comments, shares, likes and
dislikes
2018-05-25 16:21:16 +02:00
Chocobozzz e251f170b0
Split files in activitypub server 2018-05-25 11:33:04 +02:00
Chocobozzz 16f29007dc
Fix cc field in classic audience 2018-05-25 11:17:41 +02:00
Chocobozzz b40f057594
Handle concurrent requests in cache middleware 2018-05-23 10:03:26 +02:00
Rigel Kent 30c82f0d2e feature: db selection in redis
resolves #579
2018-05-21 17:49:56 +02:00
Chocobozzz 938d3fa0ff
Move normalize functions in helpers 2018-05-11 15:55:39 +02:00
Chocobozzz 5cf1350011
Improve AP validation for Notes 2018-05-11 15:41:54 +02:00
Chocobozzz 0f320037e6
Add ability to update a video channel 2018-05-11 15:25:51 +02:00
Chocobozzz fd4484f19e
Cache AP video route for 5 seconds 2018-05-11 15:24:17 +02:00
Chocobozzz 1d6e5dfc37
Improve video torrent AP object validator 2018-05-11 08:48:20 +02:00
Chocobozzz 71e3dfda4e
Add timeout and TTL to request jobs 2018-05-09 09:08:22 +02:00
Chocobozzz 08c1efbe32
Add video channel management 2018-04-26 16:18:01 +02:00
Chocobozzz 6b738c7a31
Video channel API routes refractor 2018-04-25 13:50:48 +02:00
Chocobozzz 9d3ef9fe05
Use ISO 639 for languages 2018-04-23 14:39:52 +02:00
Chocobozzz f55e5a7bf8
Process broadcast requests in parallel 2018-04-18 16:04:49 +02:00
Chocobozzz 5350fd8e5b
Move server follow in the job queue
It helps to track follow errors
2018-04-18 15:32:40 +02:00
Chocobozzz 4195cd2bc5
Add redis cache to feed route 2018-04-17 14:04:34 +02:00
Chocobozzz 37ddeba57f
Don't add a dot after the url in forgot password email 2018-04-15 09:29:53 +02:00
Chocobozzz 3d3441d6c7
Don't start application until all components were initialized 2018-04-04 11:04:14 +02:00
Chocobozzz c73e83da28
Truncate correctly video descriptions 2018-03-28 11:00:02 +02:00
Chocobozzz 07197db4c5
Try to refractor activities sending
There is still a need for work on this part though
2018-03-27 13:33:56 +02:00
Chocobozzz 73c0809326
Fix delete comment federation 2018-03-27 11:11:15 +02:00
Chocobozzz d5b7d9110d
Fix error logging 2018-03-26 16:04:14 +02:00
Chocobozzz 9c673970f6
Add public to activitypub announces 2018-03-23 13:27:21 +01:00
Chocobozzz 8e8234abba
Improve logging when cannot fetch remote videos 2018-03-23 09:10:49 +01:00
Chocobozzz 6be84cbcea
Improve activity pub actors implementation 2018-03-22 18:40:56 +01:00
Chocobozzz bebf2d89d3
Add option to disable starttls 2018-03-22 16:12:52 +01:00
Chocobozzz a38b297d90
Add /activity to activity create ids 2018-03-22 16:11:18 +01:00
Chocobozzz 376220e80e
Disable starttls for emails on localhost 2018-03-22 15:18:56 +01:00
Chocobozzz 05e67d6206
Add logging for emails 2018-03-22 11:47:26 +01:00
Chocobozzz bffbebbe6b
Fix issues with truncated description and utf characters 2018-03-21 15:00:58 +01:00
Chocobozzz a797728009
Fix video_share_url duplicate key in transcoding job 2018-03-19 15:22:18 +01:00
Chocobozzz 5d00a3d7aa
Update the api documentation 2018-03-19 14:20:04 +01:00
Chocobozzz f47776e265
Sanitize invalid actor description 2018-03-19 10:32:51 +01:00
Chocobozzz 2e3a0215d0
Add short description in config 2018-03-15 14:34:47 +01:00
Chocobozzz c39ea24b71
Upgrade server depencencies 2018-03-13 16:00:39 +01:00
Chocobozzz 628d28e84b
Fix typings 2018-02-27 17:42:32 +01:00
Chocobozzz 2c29ad4f3b
Fix job panel sorting in administration 2018-02-27 16:57:53 +01:00
Chocobozzz 056aa7f2b4
Fix resolution for portrait videos 2018-02-27 15:57:43 +01:00
Chocobozzz 6fdc553adb
Don't fail on upload if we cannot generate thumbnail 2018-02-27 13:46:56 +01:00
Chocobozzz ea99d15fe8
Add logs for scheduler 2018-02-27 11:08:59 +01:00
Chocobozzz b5c0e95544
Avoids easy cheating on vidoe views 2018-02-23 16:44:37 +01:00
Chocobozzz e3bb78a213
Fix lint 2018-02-23 16:08:32 +01:00
Chocobozzz 285fe7c930
Detect posting request in our own inbox 2018-02-23 15:09:12 +01:00
Chocobozzz 276d03ed1a
Unlisted videos are not displayed on Mastodon now 2018-02-21 10:07:02 +01:00
Chocobozzz 2422c46b27
Implement support field in video and video channel 2018-02-15 15:29:07 +01:00
Chocobozzz e3a682a877
Handle thumbnail update 2018-02-14 16:03:09 +01:00
Chocobozzz ac81d1a06d
Add ability to set video thumbnail/preview 2018-02-14 16:03:09 +01:00
Chocobozzz 3df456380a
Don't stuck on active jobs 2018-02-12 11:25:09 +01:00
Chocobozzz 18cfac83a1
Fix job count api 2018-02-08 09:52:57 +01:00
Chocobozzz fafc13fff1
Fix job queue api 2018-02-07 10:32:14 +01:00
Chocobozzz ba75d26859
Send an email on video abuse report 2018-02-01 11:08:29 +01:00
Chocobozzz 802dbc32a3
Fix job queue backoff 2018-01-30 15:51:55 +01:00
Chocobozzz f076daa76a
Add tests for emails 2018-01-30 15:16:24 +01:00
Chocobozzz ecb4e35f4e
Add ability to reset our password 2018-01-30 13:27:07 +01:00
Chocobozzz ba12e8b3a6
Add ability to login with email 2018-01-29 16:09:50 +01:00
Chocobozzz 46531a0abd
Add id to likes/dislikes/comments/shares collections 2018-01-29 10:52:19 +01:00
Chocobozzz 8dfd8fd7ca
Fix video announces processing 2018-01-26 17:48:41 +01:00
Chocobozzz 0a67e28bee
Use sensitive instead of nsfw in activitypub 2018-01-26 17:25:35 +01:00
Chocobozzz 4ba3b8ea1b
Don't rehost announced video activities 2018-01-26 15:49:57 +01:00
Chocobozzz 7acee6f18a
Fix announce activities 2018-01-26 13:43:34 +01:00
Chocobozzz 94a5ff8a4a
Move job queue to redis
We'll use it as cache in the future.

/!\ You'll loose your old jobs (pending jobs too) so upgrade only when
you don't have pending job anymore.
2018-01-25 18:41:17 +01:00
Chocobozzz d765fafc3f
Accept actors with url objects instead of string 2018-01-25 18:41:17 +01:00
Chocobozzz a28f353a64
Fix lint 2018-01-24 10:33:50 +01:00
Chocobozzz 1f7ab4f310
Upgrade server dependencies 2018-01-24 10:25:56 +01:00
Chocobozzz 2c897999fe
Optimize SQL query that fetch actor outbox 2018-01-18 16:43:26 +01:00
Chocobozzz 54e740594b
Avoid too many requests and fetching outbox 2018-01-18 15:42:20 +01:00
Chocobozzz f05a1c30c1
Don't show videos of remote instance after unfollow 2018-01-18 15:42:20 +01:00
Chocobozzz 7006bc6378
Fix actor followers/following counts 2018-01-15 09:47:19 +01:00
Chocobozzz 9fb3abfdac
Fix communication with mastodon 2018-01-12 15:35:30 +01:00
Chocobozzz 32b2b43c06
Update follower/following counts 2018-01-12 11:47:45 +01:00
Chocobozzz 6502c3d43e
Avoid making retried requests to dead followers 2018-01-12 10:02:11 +01:00
Chocobozzz 9a8cbd8278
Activity Pub improvements 2018-01-11 19:17:43 +01:00
Chocobozzz 4bbc373f13
Add reject processing for activitypub 2018-01-11 17:45:57 +01:00
Chocobozzz 6fd5ca1ec5
Fix actor refresh 2018-01-11 14:19:18 +01:00
Chocobozzz 7bc2917145
Add tests to handle down server 2018-01-11 11:47:44 +01:00
Chocobozzz 60650c77c8
Add scores to follows and remove bad ones 2018-01-11 09:35:50 +01:00
Chocobozzz 2ccaeeb341
Fetch remote AP objects 2018-01-10 17:19:14 +01:00
Chocobozzz 378557ef25
Fix comment creation 2018-01-08 13:15:01 +01:00
Chocobozzz 28832412d1
Fix lint 2018-01-08 10:46:41 +01:00
Chocobozzz 93ef8a9d02
Send comment to followers and parents 2018-01-08 10:15:27 +01:00