From cb0eda5602a21d1626a7face32de6153ed07b5f9 Mon Sep 17 00:00:00 2001 From: Alecks Gates Date: Mon, 22 May 2023 09:00:05 -0500 Subject: [PATCH] Add Podcast RSS feeds (#5487) * Initial test implementation of Podcast RSS This is a pretty simple implementation to add support for The Podcast Namespace in RSS -- instead of affecting the existing RSS implementation, this adds a new UI option. I attempted to retain compatibility with the rest of the RSS feed implementation as much as possible and have created a temporary fork of the "pfeed" library to support this effort. * Update to pfeed-podcast 1.2.2 * Initial test implementation of Podcast RSS This is a pretty simple implementation to add support for The Podcast Namespace in RSS -- instead of affecting the existing RSS implementation, this adds a new UI option. I attempted to retain compatibility with the rest of the RSS feed implementation as much as possible and have created a temporary fork of the "pfeed" library to support this effort. * Update to pfeed-podcast 1.2.2 * Initial test implementation of Podcast RSS This is a pretty simple implementation to add support for The Podcast Namespace in RSS -- instead of affecting the existing RSS implementation, this adds a new UI option. I attempted to retain compatibility with the rest of the RSS feed implementation as much as possible and have created a temporary fork of the "pfeed" library to support this effort. * Update to pfeed-podcast 1.2.2 * Add correct feed image to RSS channel * Prefer HLS videos for podcast RSS Remove video/stream titles, add optional height attribute to podcast RSS * Prefix podcast RSS images with root server URL * Add optional video query support to include captions * Add transcripts & person images to podcast RSS feed * Prefer webseed/webtorrent files over HLS fragmented mp4s * Experimentally adding podcast fields to basic config page * Add validation for new basic config fields * Don't include "content" in podcast feed, use full description for "description" * Initial test implementation of Podcast RSS This is a pretty simple implementation to add support for The Podcast Namespace in RSS -- instead of affecting the existing RSS implementation, this adds a new UI option. I attempted to retain compatibility with the rest of the RSS feed implementation as much as possible and have created a temporary fork of the "pfeed" library to support this effort. * Update to pfeed-podcast 1.2.2 * Add correct feed image to RSS channel * Prefer HLS videos for podcast RSS Remove video/stream titles, add optional height attribute to podcast RSS * Prefix podcast RSS images with root server URL * Add optional video query support to include captions * Add transcripts & person images to podcast RSS feed * Prefer webseed/webtorrent files over HLS fragmented mp4s * Experimentally adding podcast fields to basic config page * Add validation for new basic config fields * Don't include "content" in podcast feed, use full description for "description" * Add medium/socialInteract to podcast RSS feeds. Use HTML for description * Change base production image to bullseye, install prosody in image * Add liveItem and trackers to Podcast RSS feeds Remove height from alternateEnclosure, replaced with title. * Clear Podcast RSS feed cache when live streams start/end * Upgrade to Node 16 * Refactor clearCacheRoute to use ApiCache * Remove unnecessary type hint * Update dockerfile to node 16, install python-is-python2 * Use new file paths for captions/playlists * Fix legacy videos in RSS after migration to object storage * Improve method of identifying non-fragmented mp4s in podcast RSS feeds * Don't include fragmented MP4s in podcast RSS feeds * Add experimental support for podcast:categories on the podcast RSS item * Fix undefined category when no videos exist Allows for empty feeds to exist (important for feeds that might only go live) * Add support for podcast:locked -- user has to opt in to show their email * Use comma for podcast:categories delimiter * Make cache clearing async * Fix merge, temporarily test with pfeed-podcast * Syntax changes * Add EXT_MIMETYPE constants for captions * Update & fix tests, fix enclosure mimetypes, remove admin email * Add test for podacst:socialInteract * Add filters hooks for podcast customTags * Remove showdown, updated to pfeed-podcast 6.1.2 * Add 'action:api.live-video.state.updated' hook * Avoid assigning undefined category to podcast feeds * Remove nvmrc * Remove comment * Remove unused podcast config * Remove more unused podcast config * Fix MChannelAccountDefault type hint missed in merge * Remove extra line * Re-add newline in config * Fix lint errors for isEmailPublic * Fix thumbnails in podcast feeds * Requested changes based on review * Provide podcast rss 2.0 only on video channels * Misc cleanup for a less messy PR * Lint fixes * Remove pfeed-podcast * Add peertube version to new hooks * Don't use query include, remove TODO * Remove film medium hack * Clear podcast rss cache before video/channel update hooks * Clear podcast rss cache before video uploaded/deleted hooks * Refactor podcast feed cache clearing * Set correct person name from video channel * Styling * Fix tests --------- Co-authored-by: Chocobozzz --- .../my-account-change-email.component.ts | 5 +- .../my-account-email-preferences/index.ts | 1 + ...y-account-email-preferences.component.html | 15 + ...y-account-email-preferences.component.scss | 0 .../my-account-email-preferences.component.ts | 51 +++ .../my-account-settings.component.html | 4 +- .../src/app/+my-account/my-account.module.ts | 5 +- client/src/app/core/users/user.model.ts | 1 + .../shared/shared-main/video/video.service.ts | 11 +- package.json | 4 +- server/controllers/api/users/me.ts | 3 +- server/controllers/api/videos/update.ts | 4 +- server/controllers/feeds.ts | 389 ---------------- server/controllers/feeds/comment-feeds.ts | 96 ++++ server/controllers/feeds/index.ts | 16 + .../feeds/shared/common-feed-utils.ts | 145 ++++++ server/controllers/feeds/shared/index.ts | 2 + .../feeds/shared/video-feed-utils.ts | 66 +++ server/controllers/feeds/video-feeds.ts | 189 ++++++++ .../controllers/feeds/video-podcast-feeds.ts | 301 +++++++++++++ server/helpers/custom-validators/users.ts | 5 + server/initializers/constants.ts | 6 +- .../0775-add-user-is-email-public.ts | 25 ++ server/lib/blocklist.ts | 4 +- server/lib/client-html.ts | 6 +- .../lib/files-cache/videos-preview-cache.ts | 2 +- server/lib/internal-event-emitter.ts | 35 ++ server/lib/live/live-manager.ts | 4 + server/lib/plugins/plugin-helpers-builder.ts | 2 +- server/middlewares/cache/cache.ts | 14 +- server/middlewares/cache/shared/api-cache.ts | 45 +- server/middlewares/validators/feeds.ts | 46 ++ server/middlewares/validators/users.ts | 4 + server/models/account/account.ts | 12 +- server/models/actor/actor.ts | 8 +- server/models/user/user.ts | 6 + .../video/formatter/video-format-utils.ts | 2 +- server/models/video/thumbnail.ts | 6 +- server/models/video/video-caption.ts | 27 +- server/models/video/video-channel.ts | 27 +- server/models/video/video.ts | 53 ++- server/tests/client.ts | 4 +- server/tests/feeds/feeds.ts | 419 ++++++++++++------ .../main.js | 82 ++++ .../package.json | 19 + .../fixtures/peertube-plugin-test/main.js | 1 + server/tests/plugins/action-hooks.ts | 31 +- server/types/express.d.ts | 2 + server/types/models/account/account.ts | 7 +- server/types/models/actor/actor-follow.ts | 4 +- server/types/models/actor/actor.ts | 10 +- server/types/models/video/video-channels.ts | 7 +- server/types/models/video/video.ts | 4 +- .../plugins/server/server-hook.model.ts | 15 +- shared/models/users/user-update-me.model.ts | 1 + shared/models/users/user.model.ts | 1 + shared/models/videos/video-include.enum.ts | 3 +- shared/server-commands/feeds/feeds-command.ts | 23 + support/doc/api/openapi.yaml | 36 +- yarn.lock | 10 +- 60 files changed, 1712 insertions(+), 614 deletions(-) create mode 100644 client/src/app/+my-account/my-account-settings/my-account-email-preferences/index.ts create mode 100644 client/src/app/+my-account/my-account-settings/my-account-email-preferences/my-account-email-preferences.component.html create mode 100644 client/src/app/+my-account/my-account-settings/my-account-email-preferences/my-account-email-preferences.component.scss create mode 100644 client/src/app/+my-account/my-account-settings/my-account-email-preferences/my-account-email-preferences.component.ts delete mode 100644 server/controllers/feeds.ts create mode 100644 server/controllers/feeds/comment-feeds.ts create mode 100644 server/controllers/feeds/index.ts create mode 100644 server/controllers/feeds/shared/common-feed-utils.ts create mode 100644 server/controllers/feeds/shared/index.ts create mode 100644 server/controllers/feeds/shared/video-feed-utils.ts create mode 100644 server/controllers/feeds/video-feeds.ts create mode 100644 server/controllers/feeds/video-podcast-feeds.ts create mode 100644 server/initializers/migrations/0775-add-user-is-email-public.ts create mode 100644 server/lib/internal-event-emitter.ts create mode 100644 server/tests/fixtures/peertube-plugin-test-podcast-custom-tags/main.js create mode 100644 server/tests/fixtures/peertube-plugin-test-podcast-custom-tags/package.json diff --git a/client/src/app/+my-account/my-account-settings/my-account-change-email/my-account-change-email.component.ts b/client/src/app/+my-account/my-account-settings/my-account-change-email/my-account-change-email.component.ts index 235fbec4a..ebb7ed2da 100644 --- a/client/src/app/+my-account/my-account-settings/my-account-change-email/my-account-change-email.component.ts +++ b/client/src/app/+my-account/my-account-settings/my-account-change-email/my-account-change-email.component.ts @@ -1,7 +1,7 @@ import { forkJoin } from 'rxjs' import { tap } from 'rxjs/operators' import { Component, OnInit } from '@angular/core' -import { AuthService, ServerService, UserService } from '@app/core' +import { AuthService, Notifier, ServerService, UserService } from '@app/core' import { USER_EMAIL_VALIDATOR, USER_PASSWORD_VALIDATOR } from '@app/shared/form-validators/user-validators' import { FormReactive, FormReactiveService } from '@app/shared/shared-forms' import { HttpStatusCode, User } from '@shared/models' @@ -20,7 +20,8 @@ export class MyAccountChangeEmailComponent extends FormReactive implements OnIni protected formReactiveService: FormReactiveService, private authService: AuthService, private userService: UserService, - private serverService: ServerService + private serverService: ServerService, + private notifier: Notifier ) { super() } diff --git a/client/src/app/+my-account/my-account-settings/my-account-email-preferences/index.ts b/client/src/app/+my-account/my-account-settings/my-account-email-preferences/index.ts new file mode 100644 index 000000000..20b98e7d8 --- /dev/null +++ b/client/src/app/+my-account/my-account-settings/my-account-email-preferences/index.ts @@ -0,0 +1 @@ +export * from './my-account-email-preferences.component' diff --git a/client/src/app/+my-account/my-account-settings/my-account-email-preferences/my-account-email-preferences.component.html b/client/src/app/+my-account/my-account-settings/my-account-email-preferences/my-account-email-preferences.component.html new file mode 100644 index 000000000..c4fe52743 --- /dev/null +++ b/client/src/app/+my-account/my-account-settings/my-account-email-preferences/my-account-email-preferences.component.html @@ -0,0 +1,15 @@ +
+ +
+ + + Necessary to claim podcast RSS feeds. + + +
+ + +
diff --git a/client/src/app/+my-account/my-account-settings/my-account-email-preferences/my-account-email-preferences.component.scss b/client/src/app/+my-account/my-account-settings/my-account-email-preferences/my-account-email-preferences.component.scss new file mode 100644 index 000000000..e69de29bb diff --git a/client/src/app/+my-account/my-account-settings/my-account-email-preferences/my-account-email-preferences.component.ts b/client/src/app/+my-account/my-account-settings/my-account-email-preferences/my-account-email-preferences.component.ts new file mode 100644 index 000000000..7fd59d7c8 --- /dev/null +++ b/client/src/app/+my-account/my-account-settings/my-account-email-preferences/my-account-email-preferences.component.ts @@ -0,0 +1,51 @@ +import { Subject } from 'rxjs' +import { Component, Input, OnInit } from '@angular/core' +import { Notifier, UserService } from '@app/core' +import { FormReactive, FormReactiveService } from '@app/shared/shared-forms' +import { User, UserUpdateMe } from '@shared/models' + +@Component({ + selector: 'my-account-email-preferences', + templateUrl: './my-account-email-preferences.component.html', + styleUrls: [ './my-account-email-preferences.component.scss' ] +}) +export class MyAccountEmailPreferencesComponent extends FormReactive implements OnInit { + @Input() user: User = null + @Input() userInformationLoaded: Subject + + constructor ( + protected formReactiveService: FormReactiveService, + private userService: UserService, + private notifier: Notifier + ) { + super() + } + + ngOnInit () { + this.buildForm({ + 'email-public': null + }) + + this.userInformationLoaded.subscribe(() => { + this.form.patchValue({ 'email-public': this.user.emailPublic }) + }) + } + + updateEmailPublic () { + const details: UserUpdateMe = { + emailPublic: this.form.value['email-public'] + } + + this.userService.updateMyProfile(details) + .subscribe({ + next: () => { + if (details.emailPublic) this.notifier.success($localize`Email is now public`) + else this.notifier.success($localize`Email is now private`) + + this.user.emailPublic = details.emailPublic + }, + + error: err => console.log(err.message) + }) + } +} diff --git a/client/src/app/+my-account/my-account-settings/my-account-settings.component.html b/client/src/app/+my-account/my-account-settings/my-account-settings.component.html index 666205de6..3986354c1 100644 --- a/client/src/app/+my-account/my-account-settings/my-account-settings.component.html +++ b/client/src/app/+my-account/my-account-settings/my-account-settings.component.html @@ -68,7 +68,7 @@
- +
@@ -78,6 +78,8 @@
+ +
diff --git a/client/src/app/+my-account/my-account.module.ts b/client/src/app/+my-account/my-account.module.ts index 84b057647..673bd2837 100644 --- a/client/src/app/+my-account/my-account.module.ts +++ b/client/src/app/+my-account/my-account.module.ts @@ -22,6 +22,7 @@ import { MyAccountRoutingModule } from './my-account-routing.module' import { MyAccountChangeEmailComponent } from './my-account-settings/my-account-change-email' import { MyAccountChangePasswordComponent } from './my-account-settings/my-account-change-password/my-account-change-password.component' import { MyAccountDangerZoneComponent } from './my-account-settings/my-account-danger-zone' +import { MyAccountEmailPreferencesComponent } from './my-account-settings/my-account-email-preferences' import { MyAccountNotificationPreferencesComponent } from './my-account-settings/my-account-notification-preferences' import { MyAccountProfileComponent } from './my-account-settings/my-account-profile/my-account-profile.component' import { MyAccountSettingsComponent } from './my-account-settings/my-account-settings.component' @@ -65,7 +66,9 @@ import { MyAccountComponent } from './my-account.component' MyAccountAbusesListComponent, MyAccountServerBlocklistComponent, MyAccountNotificationsComponent, - MyAccountNotificationPreferencesComponent + MyAccountNotificationPreferencesComponent, + + MyAccountEmailPreferencesComponent ], exports: [ diff --git a/client/src/app/core/users/user.model.ts b/client/src/app/core/users/user.model.ts index 5534bca33..2d783145f 100644 --- a/client/src/app/core/users/user.model.ts +++ b/client/src/app/core/users/user.model.ts @@ -19,6 +19,7 @@ export class User implements UserServerModel { pendingEmail: string | null emailVerified: boolean + emailPublic: boolean nsfwPolicy: NSFWPolicyType adminFlags?: UserAdminFlag diff --git a/client/src/app/shared/shared-main/video/video.service.ts b/client/src/app/shared/shared-main/video/video.service.ts index 152149827..78a49567f 100644 --- a/client/src/app/shared/shared-main/video/video.service.ts +++ b/client/src/app/shared/shared-main/video/video.service.ts @@ -54,6 +54,7 @@ export type CommonVideoParams = { export class VideoService { static BASE_VIDEO_URL = environment.apiUrl + '/api/v1/videos' static BASE_FEEDS_URL = environment.apiUrl + '/feeds/videos.' + static PODCAST_FEEDS_URL = environment.apiUrl + '/feeds/podcast/videos.xml' static BASE_SUBSCRIPTION_FEEDS_URL = environment.apiUrl + '/feeds/subscriptions.' constructor ( @@ -266,7 +267,15 @@ export class VideoService { let params = this.restService.addRestGetParams(new HttpParams()) params = params.set('videoChannelId', videoChannelId.toString()) - return this.buildBaseFeedUrls(params) + const feedUrls = this.buildBaseFeedUrls(params) + + feedUrls.push({ + format: FeedFormat.RSS, + label: 'podcast rss 2.0', + url: VideoService.PODCAST_FEEDS_URL + `?videoChannelId=${videoChannelId}` + }) + + return feedUrls } getVideoSubscriptionFeedUrls (accountId: number, feedToken: string) { diff --git a/package.json b/package.json index 38ed90533..00a12c08b 100644 --- a/package.json +++ b/package.json @@ -97,7 +97,7 @@ "@opentelemetry/sdk-trace-base": "^1.3.1", "@opentelemetry/sdk-trace-node": "^1.3.1", "@opentelemetry/semantic-conventions": "^1.3.1", - "@peertube/feed": "^5.0.1", + "@peertube/feed": "^5.1.0", "@peertube/http-signature": "^1.7.0", "@uploadx/core": "^6.0.0", "async-lru": "^1.1.1", @@ -135,7 +135,7 @@ "jimp": "^0.22.4", "js-yaml": "^4.0.0", "jsonld": "~8.1.0", - "lodash": "^4.17.10", + "lodash": "^4.17.21", "lru-cache": "^7.13.0", "magnet-uri": "^6.1.0", "markdown-it": "^13.0.1", diff --git a/server/controllers/api/users/me.ts b/server/controllers/api/users/me.ts index 00f580ee9..218091d91 100644 --- a/server/controllers/api/users/me.ts +++ b/server/controllers/api/users/me.ts @@ -212,7 +212,8 @@ async function updateMe (req: express.Request, res: express.Response) { 'theme', 'noInstanceConfigWarningModal', 'noAccountSetupWarningModal', - 'noWelcomeModal' + 'noWelcomeModal', + 'emailPublic' ] for (const key of keysToUpdate) { diff --git a/server/controllers/api/videos/update.ts b/server/controllers/api/videos/update.ts index 5ab54a006..ddab428d4 100644 --- a/server/controllers/api/videos/update.ts +++ b/server/controllers/api/videos/update.ts @@ -2,10 +2,12 @@ import express from 'express' import { Transaction } from 'sequelize/types' import { changeVideoChannelShare } from '@server/lib/activitypub/share' import { addVideoJobsAfterUpdate, buildVideoThumbnailsFromReq, setVideoTags } from '@server/lib/video' +import { VideoPathManager } from '@server/lib/video-path-manager' import { setVideoPrivacy } from '@server/lib/video-privacy' import { openapiOperationDoc } from '@server/middlewares/doc' import { FilteredModelAttributes } from '@server/types' import { MVideoFullLight } from '@server/types/models' +import { forceNumber } from '@shared/core-utils' import { HttpStatusCode, VideoUpdate } from '@shared/models' import { auditLoggerFactory, getAuditIdFromRes, VideoAuditView } from '../../../helpers/audit-logger' import { resetSequelizeInstance } from '../../../helpers/database-utils' @@ -18,8 +20,6 @@ import { autoBlacklistVideoIfNeeded } from '../../../lib/video-blacklist' import { asyncMiddleware, asyncRetryTransactionMiddleware, authenticate, videosUpdateValidator } from '../../../middlewares' import { ScheduleVideoUpdateModel } from '../../../models/video/schedule-video-update' import { VideoModel } from '../../../models/video/video' -import { VideoPathManager } from '@server/lib/video-path-manager' -import { forceNumber } from '@shared/core-utils' const lTags = loggerTagsFactory('api', 'video') const auditLogger = auditLoggerFactory('videos') diff --git a/server/controllers/feeds.ts b/server/controllers/feeds.ts deleted file mode 100644 index ef810a842..000000000 --- a/server/controllers/feeds.ts +++ /dev/null @@ -1,389 +0,0 @@ -import express from 'express' -import { extname } from 'path' -import { Feed } from '@peertube/feed' -import { mdToOneLinePlainText, toSafeHtml } from '@server/helpers/markdown' -import { getServerActor } from '@server/models/application/application' -import { getCategoryLabel } from '@server/models/video/formatter/video-format-utils' -import { MAccountDefault, MChannelBannerAccountDefault, MVideoFullLight } from '@server/types/models' -import { ActorImageType, VideoInclude } from '@shared/models' -import { buildNSFWFilter } from '../helpers/express-utils' -import { CONFIG } from '../initializers/config' -import { MIMETYPES, PREVIEWS_SIZE, ROUTE_CACHE_LIFETIME, WEBSERVER } from '../initializers/constants' -import { - asyncMiddleware, - commonVideosFiltersValidator, - feedsFormatValidator, - setDefaultVideosSort, - setFeedFormatContentType, - videoCommentsFeedsValidator, - videoFeedsValidator, - videosSortValidator, - videoSubscriptionFeedsValidator -} from '../middlewares' -import { cacheRouteFactory } from '../middlewares/cache/cache' -import { VideoModel } from '../models/video/video' -import { VideoCommentModel } from '../models/video/video-comment' - -const feedsRouter = express.Router() - -const cacheRoute = cacheRouteFactory({ - headerBlacklist: [ 'Content-Type' ] -}) - -feedsRouter.get('/feeds/video-comments.:format', - feedsFormatValidator, - setFeedFormatContentType, - cacheRoute(ROUTE_CACHE_LIFETIME.FEEDS), - asyncMiddleware(videoFeedsValidator), - asyncMiddleware(videoCommentsFeedsValidator), - asyncMiddleware(generateVideoCommentsFeed) -) - -feedsRouter.get('/feeds/videos.:format', - videosSortValidator, - setDefaultVideosSort, - feedsFormatValidator, - setFeedFormatContentType, - cacheRoute(ROUTE_CACHE_LIFETIME.FEEDS), - commonVideosFiltersValidator, - asyncMiddleware(videoFeedsValidator), - asyncMiddleware(generateVideoFeed) -) - -feedsRouter.get('/feeds/subscriptions.:format', - videosSortValidator, - setDefaultVideosSort, - feedsFormatValidator, - setFeedFormatContentType, - cacheRoute(ROUTE_CACHE_LIFETIME.FEEDS), - commonVideosFiltersValidator, - asyncMiddleware(videoSubscriptionFeedsValidator), - asyncMiddleware(generateVideoFeedForSubscriptions) -) - -// --------------------------------------------------------------------------- - -export { - feedsRouter -} - -// --------------------------------------------------------------------------- - -async function generateVideoCommentsFeed (req: express.Request, res: express.Response) { - const start = 0 - const video = res.locals.videoAll - const account = res.locals.account - const videoChannel = res.locals.videoChannel - - const comments = await VideoCommentModel.listForFeed({ - start, - count: CONFIG.FEEDS.COMMENTS.COUNT, - videoId: video ? video.id : undefined, - accountId: account ? account.id : undefined, - videoChannelId: videoChannel ? videoChannel.id : undefined - }) - - const { name, description, imageUrl } = buildFeedMetadata({ video, account, videoChannel }) - - const feed = initFeed({ - name, - description, - imageUrl, - resourceType: 'video-comments', - queryString: new URL(WEBSERVER.URL + req.originalUrl).search - }) - - // Adding video items to the feed, one at a time - for (const comment of comments) { - const localLink = WEBSERVER.URL + comment.getCommentStaticPath() - - let title = comment.Video.name - const author: { name: string, link: string }[] = [] - - if (comment.Account) { - title += ` - ${comment.Account.getDisplayName()}` - author.push({ - name: comment.Account.getDisplayName(), - link: comment.Account.Actor.url - }) - } - - feed.addItem({ - title, - id: localLink, - link: localLink, - content: toSafeHtml(comment.text), - author, - date: comment.createdAt - }) - } - - // Now the feed generation is done, let's send it! - return sendFeed(feed, req, res) -} - -async function generateVideoFeed (req: express.Request, res: express.Response) { - const start = 0 - const account = res.locals.account - const videoChannel = res.locals.videoChannel - const nsfw = buildNSFWFilter(res, req.query.nsfw) - - const { name, description, imageUrl } = buildFeedMetadata({ videoChannel, account }) - - const feed = initFeed({ - name, - description, - imageUrl, - resourceType: 'videos', - queryString: new URL(WEBSERVER.URL + req.url).search - }) - - const options = { - accountId: account ? account.id : null, - videoChannelId: videoChannel ? videoChannel.id : null - } - - const server = await getServerActor() - const { data } = await VideoModel.listForApi({ - start, - count: CONFIG.FEEDS.VIDEOS.COUNT, - sort: req.query.sort, - displayOnlyForFollower: { - actorId: server.id, - orLocalVideos: true - }, - nsfw, - isLocal: req.query.isLocal, - include: req.query.include | VideoInclude.FILES, - hasFiles: true, - countVideos: false, - ...options - }) - - addVideosToFeed(feed, data) - - // Now the feed generation is done, let's send it! - return sendFeed(feed, req, res) -} - -async function generateVideoFeedForSubscriptions (req: express.Request, res: express.Response) { - const start = 0 - const account = res.locals.account - const nsfw = buildNSFWFilter(res, req.query.nsfw) - - const { name, description, imageUrl } = buildFeedMetadata({ account }) - - const feed = initFeed({ - name, - description, - imageUrl, - resourceType: 'videos', - queryString: new URL(WEBSERVER.URL + req.url).search - }) - - const { data } = await VideoModel.listForApi({ - start, - count: CONFIG.FEEDS.VIDEOS.COUNT, - sort: req.query.sort, - nsfw, - - isLocal: req.query.isLocal, - - hasFiles: true, - include: req.query.include | VideoInclude.FILES, - - countVideos: false, - - displayOnlyForFollower: { - actorId: res.locals.user.Account.Actor.id, - orLocalVideos: false - }, - user: res.locals.user - }) - - addVideosToFeed(feed, data) - - // Now the feed generation is done, let's send it! - return sendFeed(feed, req, res) -} - -function initFeed (parameters: { - name: string - description: string - imageUrl: string - resourceType?: 'videos' | 'video-comments' - queryString?: string -}) { - const webserverUrl = WEBSERVER.URL - const { name, description, resourceType, queryString, imageUrl } = parameters - - return new Feed({ - title: name, - description: mdToOneLinePlainText(description), - // updated: TODO: somehowGetLatestUpdate, // optional, default = today - id: webserverUrl, - link: webserverUrl, - image: imageUrl, - favicon: webserverUrl + '/client/assets/images/favicon.png', - copyright: `All rights reserved, unless otherwise specified in the terms specified at ${webserverUrl}/about` + - ` and potential licenses granted by each content's rightholder.`, - generator: `Toraifōsu`, // ^.~ - feedLinks: { - json: `${webserverUrl}/feeds/${resourceType}.json${queryString}`, - atom: `${webserverUrl}/feeds/${resourceType}.atom${queryString}`, - rss: `${webserverUrl}/feeds/${resourceType}.xml${queryString}` - }, - author: { - name: 'Instance admin of ' + CONFIG.INSTANCE.NAME, - email: CONFIG.ADMIN.EMAIL, - link: `${webserverUrl}/about` - } - }) -} - -function addVideosToFeed (feed: Feed, videos: VideoModel[]) { - for (const video of videos) { - const formattedVideoFiles = video.getFormattedVideoFilesJSON(false) - - const torrents = formattedVideoFiles.map(videoFile => ({ - title: video.name, - url: videoFile.torrentUrl, - size_in_bytes: videoFile.size - })) - - const videoFiles = formattedVideoFiles.map(videoFile => { - const result = { - type: MIMETYPES.VIDEO.EXT_MIMETYPE[extname(videoFile.fileUrl)], - medium: 'video', - height: videoFile.resolution.id, - fileSize: videoFile.size, - url: videoFile.fileUrl, - framerate: videoFile.fps, - duration: video.duration - } - - if (video.language) Object.assign(result, { lang: video.language }) - - return result - }) - - const categories: { value: number, label: string }[] = [] - if (video.category) { - categories.push({ - value: video.category, - label: getCategoryLabel(video.category) - }) - } - - const localLink = WEBSERVER.URL + video.getWatchStaticPath() - - feed.addItem({ - title: video.name, - id: localLink, - link: localLink, - description: mdToOneLinePlainText(video.getTruncatedDescription()), - content: toSafeHtml(video.description), - author: [ - { - name: video.VideoChannel.getDisplayName(), - link: video.VideoChannel.Actor.url - } - ], - date: video.publishedAt, - nsfw: video.nsfw, - torrents, - - // Enclosure - video: videoFiles.length !== 0 - ? { - url: videoFiles[0].url, - length: videoFiles[0].fileSize, - type: videoFiles[0].type - } - : undefined, - - // Media RSS - videos: videoFiles, - - embed: { - url: WEBSERVER.URL + video.getEmbedStaticPath(), - allowFullscreen: true - }, - player: { - url: WEBSERVER.URL + video.getWatchStaticPath() - }, - categories, - community: { - statistics: { - views: video.views - } - }, - thumbnails: [ - { - url: WEBSERVER.URL + video.getPreviewStaticPath(), - height: PREVIEWS_SIZE.height, - width: PREVIEWS_SIZE.width - } - ] - }) - } -} - -function sendFeed (feed: Feed, req: express.Request, res: express.Response) { - const format = req.params.format - - if (format === 'atom' || format === 'atom1') { - return res.send(feed.atom1()).end() - } - - if (format === 'json' || format === 'json1') { - return res.send(feed.json1()).end() - } - - if (format === 'rss' || format === 'rss2') { - return res.send(feed.rss2()).end() - } - - // We're in the ambiguous '.xml' case and we look at the format query parameter - if (req.query.format === 'atom' || req.query.format === 'atom1') { - return res.send(feed.atom1()).end() - } - - return res.send(feed.rss2()).end() -} - -function buildFeedMetadata (options: { - videoChannel?: MChannelBannerAccountDefault - account?: MAccountDefault - video?: MVideoFullLight -}) { - const { video, videoChannel, account } = options - - let imageUrl = WEBSERVER.URL + '/client/assets/images/icons/icon-96x96.png' - let name: string - let description: string - - if (videoChannel) { - name = videoChannel.getDisplayName() - description = videoChannel.description - - if (videoChannel.Actor.hasImage(ActorImageType.AVATAR)) { - imageUrl = WEBSERVER.URL + videoChannel.Actor.Avatars[0].getStaticPath() - } - } else if (account) { - name = account.getDisplayName() - description = account.description - - if (account.Actor.hasImage(ActorImageType.AVATAR)) { - imageUrl = WEBSERVER.URL + account.Actor.Avatars[0].getStaticPath() - } - } else if (video) { - name = video.name - description = video.description - } else { - name = CONFIG.INSTANCE.NAME - description = CONFIG.INSTANCE.DESCRIPTION - } - - return { name, description, imageUrl } -} diff --git a/server/controllers/feeds/comment-feeds.ts b/server/controllers/feeds/comment-feeds.ts new file mode 100644 index 000000000..bdc53b51f --- /dev/null +++ b/server/controllers/feeds/comment-feeds.ts @@ -0,0 +1,96 @@ +import express from 'express' +import { toSafeHtml } from '@server/helpers/markdown' +import { cacheRouteFactory } from '@server/middlewares' +import { CONFIG } from '../../initializers/config' +import { ROUTE_CACHE_LIFETIME, WEBSERVER } from '../../initializers/constants' +import { + asyncMiddleware, + feedsFormatValidator, + setFeedFormatContentType, + videoCommentsFeedsValidator, + videoFeedsValidator +} from '../../middlewares' +import { VideoCommentModel } from '../../models/video/video-comment' +import { buildFeedMetadata, initFeed, sendFeed } from './shared' + +const commentFeedsRouter = express.Router() + +// --------------------------------------------------------------------------- + +const { middleware: cacheRouteMiddleware } = cacheRouteFactory({ + headerBlacklist: [ 'Content-Type' ] +}) + +// --------------------------------------------------------------------------- + +commentFeedsRouter.get('/feeds/video-comments.:format', + feedsFormatValidator, + setFeedFormatContentType, + cacheRouteMiddleware(ROUTE_CACHE_LIFETIME.FEEDS), + asyncMiddleware(videoFeedsValidator), + asyncMiddleware(videoCommentsFeedsValidator), + asyncMiddleware(generateVideoCommentsFeed) +) + +// --------------------------------------------------------------------------- + +export { + commentFeedsRouter +} + +// --------------------------------------------------------------------------- + +async function generateVideoCommentsFeed (req: express.Request, res: express.Response) { + const start = 0 + const video = res.locals.videoAll + const account = res.locals.account + const videoChannel = res.locals.videoChannel + + const comments = await VideoCommentModel.listForFeed({ + start, + count: CONFIG.FEEDS.COMMENTS.COUNT, + videoId: video ? video.id : undefined, + accountId: account ? account.id : undefined, + videoChannelId: videoChannel ? videoChannel.id : undefined + }) + + const { name, description, imageUrl, link } = await buildFeedMetadata({ video, account, videoChannel }) + + const feed = initFeed({ + name, + description, + imageUrl, + isPodcast: false, + link, + resourceType: 'video-comments', + queryString: new URL(WEBSERVER.URL + req.originalUrl).search + }) + + // Adding video items to the feed, one at a time + for (const comment of comments) { + const localLink = WEBSERVER.URL + comment.getCommentStaticPath() + + let title = comment.Video.name + const author: { name: string, link: string }[] = [] + + if (comment.Account) { + title += ` - ${comment.Account.getDisplayName()}` + author.push({ + name: comment.Account.getDisplayName(), + link: comment.Account.Actor.url + }) + } + + feed.addItem({ + title, + id: localLink, + link: localLink, + content: toSafeHtml(comment.text), + author, + date: comment.createdAt + }) + } + + // Now the feed generation is done, let's send it! + return sendFeed(feed, req, res) +} diff --git a/server/controllers/feeds/index.ts b/server/controllers/feeds/index.ts new file mode 100644 index 000000000..e344a1448 --- /dev/null +++ b/server/controllers/feeds/index.ts @@ -0,0 +1,16 @@ +import express from 'express' +import { commentFeedsRouter } from './comment-feeds' +import { videoFeedsRouter } from './video-feeds' +import { videoPodcastFeedsRouter } from './video-podcast-feeds' + +const feedsRouter = express.Router() + +feedsRouter.use('/', commentFeedsRouter) +feedsRouter.use('/', videoFeedsRouter) +feedsRouter.use('/', videoPodcastFeedsRouter) + +// --------------------------------------------------------------------------- + +export { + feedsRouter +} diff --git a/server/controllers/feeds/shared/common-feed-utils.ts b/server/controllers/feeds/shared/common-feed-utils.ts new file mode 100644 index 000000000..375c2814b --- /dev/null +++ b/server/controllers/feeds/shared/common-feed-utils.ts @@ -0,0 +1,145 @@ +import express from 'express' +import { Feed } from '@peertube/feed' +import { CustomTag, CustomXMLNS, Person } from '@peertube/feed/lib/typings' +import { mdToOneLinePlainText } from '@server/helpers/markdown' +import { CONFIG } from '@server/initializers/config' +import { WEBSERVER } from '@server/initializers/constants' +import { UserModel } from '@server/models/user/user' +import { MAccountDefault, MChannelBannerAccountDefault, MUser, MVideoFullLight } from '@server/types/models' +import { pick } from '@shared/core-utils' +import { ActorImageType } from '@shared/models' + +export function initFeed (parameters: { + name: string + description: string + imageUrl: string + isPodcast: boolean + link?: string + locked?: { isLocked: boolean, email: string } + author?: { + name: string + link: string + imageUrl: string + } + person?: Person[] + resourceType?: 'videos' | 'video-comments' + queryString?: string + medium?: string + stunServers?: string[] + trackers?: string[] + customXMLNS?: CustomXMLNS[] + customTags?: CustomTag[] +}) { + const webserverUrl = WEBSERVER.URL + const { name, description, link, imageUrl, isPodcast, resourceType, queryString, medium } = parameters + + return new Feed({ + title: name, + description: mdToOneLinePlainText(description), + // updated: TODO: somehowGetLatestUpdate, // optional, default = today + id: link || webserverUrl, + link: link || webserverUrl, + image: imageUrl, + favicon: webserverUrl + '/client/assets/images/favicon.png', + copyright: `All rights reserved, unless otherwise specified in the terms specified at ${webserverUrl}/about` + + ` and potential licenses granted by each content's rightholder.`, + generator: `Toraifōsu`, // ^.~ + medium: medium || 'video', + feedLinks: { + json: `${webserverUrl}/feeds/${resourceType}.json${queryString}`, + atom: `${webserverUrl}/feeds/${resourceType}.atom${queryString}`, + rss: isPodcast + ? `${webserverUrl}/feeds/podcast/videos.xml${queryString}` + : `${webserverUrl}/feeds/${resourceType}.xml${queryString}` + }, + + ...pick(parameters, [ 'stunServers', 'trackers', 'customXMLNS', 'customTags', 'author', 'person', 'locked' ]) + }) +} + +export function sendFeed (feed: Feed, req: express.Request, res: express.Response) { + const format = req.params.format + + if (format === 'atom' || format === 'atom1') { + return res.send(feed.atom1()).end() + } + + if (format === 'json' || format === 'json1') { + return res.send(feed.json1()).end() + } + + if (format === 'rss' || format === 'rss2') { + return res.send(feed.rss2()).end() + } + + // We're in the ambiguous '.xml' case and we look at the format query parameter + if (req.query.format === 'atom' || req.query.format === 'atom1') { + return res.send(feed.atom1()).end() + } + + return res.send(feed.rss2()).end() +} + +export async function buildFeedMetadata (options: { + videoChannel?: MChannelBannerAccountDefault + account?: MAccountDefault + video?: MVideoFullLight +}) { + const { video, videoChannel, account } = options + + let imageUrl = WEBSERVER.URL + '/client/assets/images/icons/icon-96x96.png' + let accountImageUrl: string + let name: string + let userName: string + let description: string + let email: string + let link: string + let accountLink: string + let user: MUser + + if (videoChannel) { + name = videoChannel.getDisplayName() + description = videoChannel.description + link = videoChannel.getClientUrl() + accountLink = videoChannel.Account.getClientUrl() + + if (videoChannel.Actor.hasImage(ActorImageType.AVATAR)) { + imageUrl = WEBSERVER.URL + videoChannel.Actor.Avatars[0].getStaticPath() + } + + if (videoChannel.Account.Actor.hasImage(ActorImageType.AVATAR)) { + accountImageUrl = WEBSERVER.URL + videoChannel.Account.Actor.Avatars[0].getStaticPath() + } + + user = await UserModel.loadById(videoChannel.Account.userId) + userName = videoChannel.Account.getDisplayName() + } else if (account) { + name = account.getDisplayName() + description = account.description + link = account.getClientUrl() + accountLink = link + + if (account.Actor.hasImage(ActorImageType.AVATAR)) { + imageUrl = WEBSERVER.URL + account.Actor.Avatars[0].getStaticPath() + accountImageUrl = imageUrl + } + + user = await UserModel.loadById(account.userId) + } else if (video) { + name = video.name + description = video.description + link = video.url + } else { + name = CONFIG.INSTANCE.NAME + description = CONFIG.INSTANCE.DESCRIPTION + link = WEBSERVER.URL + } + + // If the user is local, has a verified email address, and allows it to be publicly displayed + // Return it so the owner can prove ownership of their feed + if (user && !user.pluginAuth && user.emailVerified && user.emailPublic) { + email = user.email + } + + return { name, userName, description, imageUrl, accountImageUrl, email, link, accountLink } +} diff --git a/server/controllers/feeds/shared/index.ts b/server/controllers/feeds/shared/index.ts new file mode 100644 index 000000000..0136c8477 --- /dev/null +++ b/server/controllers/feeds/shared/index.ts @@ -0,0 +1,2 @@ +export * from './video-feed-utils' +export * from './common-feed-utils' diff --git a/server/controllers/feeds/shared/video-feed-utils.ts b/server/controllers/feeds/shared/video-feed-utils.ts new file mode 100644 index 000000000..3175cea59 --- /dev/null +++ b/server/controllers/feeds/shared/video-feed-utils.ts @@ -0,0 +1,66 @@ +import { mdToOneLinePlainText, toSafeHtml } from '@server/helpers/markdown' +import { CONFIG } from '@server/initializers/config' +import { WEBSERVER } from '@server/initializers/constants' +import { getServerActor } from '@server/models/application/application' +import { getCategoryLabel } from '@server/models/video/formatter/video-format-utils' +import { DisplayOnlyForFollowerOptions } from '@server/models/video/sql/video' +import { VideoModel } from '@server/models/video/video' +import { MThumbnail, MUserDefault } from '@server/types/models' +import { VideoInclude } from '@shared/models' + +export async function getVideosForFeeds (options: { + sort: string + nsfw: boolean + isLocal: boolean + include: VideoInclude + + accountId?: number + videoChannelId?: number + displayOnlyForFollower?: DisplayOnlyForFollowerOptions + user?: MUserDefault +}) { + const server = await getServerActor() + + const { data } = await VideoModel.listForApi({ + start: 0, + count: CONFIG.FEEDS.VIDEOS.COUNT, + displayOnlyForFollower: { + actorId: server.id, + orLocalVideos: true + }, + hasFiles: true, + countVideos: false, + + ...options + }) + + return data +} + +export function getCommonVideoFeedAttributes (video: VideoModel) { + const localLink = WEBSERVER.URL + video.getWatchStaticPath() + + const thumbnailModels: MThumbnail[] = [] + if (video.hasPreview()) thumbnailModels.push(video.getPreview()) + thumbnailModels.push(video.getMiniature()) + + return { + title: video.name, + link: localLink, + description: mdToOneLinePlainText(video.getTruncatedDescription()), + content: toSafeHtml(video.description), + + date: video.publishedAt, + nsfw: video.nsfw, + + category: video.category + ? [ { name: getCategoryLabel(video.category) } ] + : undefined, + + thumbnails: thumbnailModels.map(t => ({ + url: WEBSERVER.URL + t.getLocalStaticPath(), + width: t.width, + height: t.height + })) + } +} diff --git a/server/controllers/feeds/video-feeds.ts b/server/controllers/feeds/video-feeds.ts new file mode 100644 index 000000000..b6e0663eb --- /dev/null +++ b/server/controllers/feeds/video-feeds.ts @@ -0,0 +1,189 @@ +import express from 'express' +import { extname } from 'path' +import { Feed } from '@peertube/feed' +import { cacheRouteFactory } from '@server/middlewares' +import { VideoModel } from '@server/models/video/video' +import { VideoInclude } from '@shared/models' +import { buildNSFWFilter } from '../../helpers/express-utils' +import { MIMETYPES, ROUTE_CACHE_LIFETIME, WEBSERVER } from '../../initializers/constants' +import { + asyncMiddleware, + commonVideosFiltersValidator, + feedsFormatValidator, + setDefaultVideosSort, + setFeedFormatContentType, + videoFeedsValidator, + videosSortValidator, + videoSubscriptionFeedsValidator +} from '../../middlewares' +import { buildFeedMetadata, getCommonVideoFeedAttributes, getVideosForFeeds, initFeed, sendFeed } from './shared' + +const videoFeedsRouter = express.Router() + +const { middleware: cacheRouteMiddleware } = cacheRouteFactory({ + headerBlacklist: [ 'Content-Type' ] +}) + +// --------------------------------------------------------------------------- + +videoFeedsRouter.get('/feeds/videos.:format', + videosSortValidator, + setDefaultVideosSort, + feedsFormatValidator, + setFeedFormatContentType, + cacheRouteMiddleware(ROUTE_CACHE_LIFETIME.FEEDS), + commonVideosFiltersValidator, + asyncMiddleware(videoFeedsValidator), + asyncMiddleware(generateVideoFeed) +) + +videoFeedsRouter.get('/feeds/subscriptions.:format', + videosSortValidator, + setDefaultVideosSort, + feedsFormatValidator, + setFeedFormatContentType, + cacheRouteMiddleware(ROUTE_CACHE_LIFETIME.FEEDS), + commonVideosFiltersValidator, + asyncMiddleware(videoSubscriptionFeedsValidator), + asyncMiddleware(generateVideoFeedForSubscriptions) +) + +// --------------------------------------------------------------------------- + +export { + videoFeedsRouter +} + +// --------------------------------------------------------------------------- + +async function generateVideoFeed (req: express.Request, res: express.Response) { + const account = res.locals.account + const videoChannel = res.locals.videoChannel + + const { name, description, imageUrl, accountImageUrl, link, accountLink } = await buildFeedMetadata({ videoChannel, account }) + + const feed = initFeed({ + name, + description, + link, + isPodcast: false, + imageUrl, + author: { name, link: accountLink, imageUrl: accountImageUrl }, + resourceType: 'videos', + queryString: new URL(WEBSERVER.URL + req.url).search + }) + + const data = await getVideosForFeeds({ + sort: req.query.sort, + nsfw: buildNSFWFilter(res, req.query.nsfw), + isLocal: req.query.isLocal, + include: req.query.include | VideoInclude.FILES, + accountId: account?.id, + videoChannelId: videoChannel?.id + }) + + addVideosToFeed(feed, data) + + // Now the feed generation is done, let's send it! + return sendFeed(feed, req, res) +} + +async function generateVideoFeedForSubscriptions (req: express.Request, res: express.Response) { + const account = res.locals.account + const { name, description, imageUrl, link } = await buildFeedMetadata({ account }) + + const feed = initFeed({ + name, + description, + link, + isPodcast: false, + imageUrl, + resourceType: 'videos', + queryString: new URL(WEBSERVER.URL + req.url).search + }) + + const data = await getVideosForFeeds({ + sort: req.query.sort, + nsfw: buildNSFWFilter(res, req.query.nsfw), + isLocal: req.query.isLocal, + include: req.query.include | VideoInclude.FILES, + displayOnlyForFollower: { + actorId: res.locals.user.Account.Actor.id, + orLocalVideos: false + }, + user: res.locals.user + }) + + addVideosToFeed(feed, data) + + // Now the feed generation is done, let's send it! + return sendFeed(feed, req, res) +} + +// --------------------------------------------------------------------------- + +function addVideosToFeed (feed: Feed, videos: VideoModel[]) { + /** + * Adding video items to the feed object, one at a time + */ + for (const video of videos) { + const formattedVideoFiles = video.getFormattedAllVideoFilesJSON(false) + + const torrents = formattedVideoFiles.map(videoFile => ({ + title: video.name, + url: videoFile.torrentUrl, + size_in_bytes: videoFile.size + })) + + const videoFiles = formattedVideoFiles.map(videoFile => { + return { + type: MIMETYPES.VIDEO.EXT_MIMETYPE[extname(videoFile.fileUrl)], + medium: 'video', + height: videoFile.resolution.id, + fileSize: videoFile.size, + url: videoFile.fileUrl, + framerate: videoFile.fps, + duration: video.duration, + lang: video.language + } + }) + + feed.addItem({ + ...getCommonVideoFeedAttributes(video), + + id: WEBSERVER.URL + video.getWatchStaticPath(), + author: [ + { + name: video.VideoChannel.getDisplayName(), + link: video.VideoChannel.getClientUrl() + } + ], + torrents, + + // Enclosure + video: videoFiles.length !== 0 + ? { + url: videoFiles[0].url, + length: videoFiles[0].fileSize, + type: videoFiles[0].type + } + : undefined, + + // Media RSS + videos: videoFiles, + + embed: { + url: WEBSERVER.URL + video.getEmbedStaticPath(), + allowFullscreen: true + }, + player: { + url: WEBSERVER.URL + video.getWatchStaticPath() + }, + community: { + statistics: { + views: video.views + } + } + }) + } +} diff --git a/server/controllers/feeds/video-podcast-feeds.ts b/server/controllers/feeds/video-podcast-feeds.ts new file mode 100644 index 000000000..45d31c781 --- /dev/null +++ b/server/controllers/feeds/video-podcast-feeds.ts @@ -0,0 +1,301 @@ +import express from 'express' +import { extname } from 'path' +import { Feed } from '@peertube/feed' +import { CustomTag, CustomXMLNS, LiveItemStatus } from '@peertube/feed/lib/typings' +import { InternalEventEmitter } from '@server/lib/internal-event-emitter' +import { Hooks } from '@server/lib/plugins/hooks' +import { buildPodcastGroupsCache, cacheRouteFactory, videoFeedsPodcastSetCacheKey } from '@server/middlewares' +import { MVideo, MVideoCaptionVideo, MVideoFullLight } from '@server/types/models' +import { sortObjectComparator } from '@shared/core-utils' +import { ActorImageType, VideoFile, VideoInclude, VideoResolution, VideoState } from '@shared/models' +import { buildNSFWFilter } from '../../helpers/express-utils' +import { MIMETYPES, ROUTE_CACHE_LIFETIME, WEBSERVER } from '../../initializers/constants' +import { asyncMiddleware, setFeedPodcastContentType, videoFeedsPodcastValidator } from '../../middlewares' +import { VideoModel } from '../../models/video/video' +import { VideoCaptionModel } from '../../models/video/video-caption' +import { buildFeedMetadata, getCommonVideoFeedAttributes, getVideosForFeeds, initFeed } from './shared' + +const videoPodcastFeedsRouter = express.Router() + +// --------------------------------------------------------------------------- + +const { middleware: podcastCacheRouteMiddleware, instance: podcastApiCache } = cacheRouteFactory({ + headerBlacklist: [ 'Content-Type' ] +}) + +for (const event of ([ 'video-created', 'video-updated', 'video-deleted' ] as const)) { + InternalEventEmitter.Instance.on(event, ({ video }) => { + if (video.remote) return + + podcastApiCache.clearGroupSafe(buildPodcastGroupsCache({ channelId: video.channelId })) + }) +} + +for (const event of ([ 'channel-updated', 'channel-deleted' ] as const)) { + InternalEventEmitter.Instance.on(event, ({ channel }) => { + podcastApiCache.clearGroupSafe(buildPodcastGroupsCache({ channelId: channel.id })) + }) +} + +// --------------------------------------------------------------------------- + +videoPodcastFeedsRouter.get('/feeds/podcast/videos.xml', + setFeedPodcastContentType, + videoFeedsPodcastSetCacheKey, + podcastCacheRouteMiddleware(ROUTE_CACHE_LIFETIME.FEEDS), + asyncMiddleware(videoFeedsPodcastValidator), + asyncMiddleware(generateVideoPodcastFeed) +) + +// --------------------------------------------------------------------------- + +export { + videoPodcastFeedsRouter +} + +// --------------------------------------------------------------------------- + +async function generateVideoPodcastFeed (req: express.Request, res: express.Response) { + const videoChannel = res.locals.videoChannel + + const { name, userName, description, imageUrl, accountImageUrl, email, link, accountLink } = await buildFeedMetadata({ videoChannel }) + + const data = await getVideosForFeeds({ + sort: '-publishedAt', + nsfw: buildNSFWFilter(), + // Prevent podcast feeds from listing videos in other instances + // helps prevent duplicates when they are indexed -- only the author should control them + isLocal: true, + include: VideoInclude.FILES, + videoChannelId: videoChannel?.id + }) + + const customTags: CustomTag[] = await Hooks.wrapObject( + [], + 'filter:feed.podcast.channel.create-custom-tags.result', + { videoChannel } + ) + + const customXMLNS: CustomXMLNS[] = await Hooks.wrapObject( + [], + 'filter:feed.podcast.rss.create-custom-xmlns.result' + ) + + const feed = initFeed({ + name, + description, + link, + isPodcast: true, + imageUrl, + + locked: email + ? { isLocked: true, email } // Default to true because we have no way of offering a redirect yet + : undefined, + + person: [ { name: userName, href: accountLink, img: accountImageUrl } ], + resourceType: 'videos', + queryString: new URL(WEBSERVER.URL + req.url).search, + medium: 'video', + customXMLNS, + customTags + }) + + await addVideosToPodcastFeed(feed, data) + + // Now the feed generation is done, let's send it! + return res.send(feed.podcast()).end() +} + +type PodcastMedia = + { + type: string + length: number + bitrate: number + sources: { uri: string, contentType?: string }[] + title: string + language?: string + } | + { + sources: { uri: string }[] + type: string + title: string + } + +async function generatePodcastItem (options: { + video: VideoModel + liveItem: boolean + media: PodcastMedia[] +}) { + const { video, liveItem, media } = options + + const customTags: CustomTag[] = await Hooks.wrapObject( + [], + 'filter:feed.podcast.video.create-custom-tags.result', + { video, liveItem } + ) + + const account = video.VideoChannel.Account + + const author = { + name: account.getDisplayName(), + href: account.getClientUrl() + } + + return { + ...getCommonVideoFeedAttributes(video), + + trackers: video.getTrackerUrls(), + + author: [ author ], + person: [ + { + ...author, + + img: account.Actor.hasImage(ActorImageType.AVATAR) + ? WEBSERVER.URL + account.Actor.Avatars[0].getStaticPath() + : undefined + } + ], + + media, + + socialInteract: [ + { + uri: video.url, + protocol: 'activitypub', + accountUrl: account.getClientUrl() + } + ], + + customTags + } +} + +async function addVideosToPodcastFeed (feed: Feed, videos: VideoModel[]) { + const captionsGroup = await VideoCaptionModel.listCaptionsOfMultipleVideos(videos.map(v => v.id)) + + for (const video of videos) { + if (!video.isLive) { + await addVODPodcastItem({ feed, video, captionsGroup }) + } else if (video.isLive && video.state !== VideoState.LIVE_ENDED) { + await addLivePodcastItem({ feed, video }) + } + } +} + +async function addVODPodcastItem (options: { + feed: Feed + video: VideoModel + captionsGroup: { [ id: number ]: MVideoCaptionVideo[] } +}) { + const { feed, video, captionsGroup } = options + + const webVideos = video.getFormattedWebVideoFilesJSON(true) + .map(f => buildVODWebVideoFile(video, f)) + .sort(sortObjectComparator('bitrate', 'desc')) + + const streamingPlaylistFiles = buildVODStreamingPlaylists(video) + + // Order matters here, the first media URI will be the "default" + // So web videos are default if enabled + const media = [ ...webVideos, ...streamingPlaylistFiles ] + + const videoCaptions = buildVODCaptions(video, captionsGroup[video.id]) + const item = await generatePodcastItem({ video, liveItem: false, media }) + + feed.addPodcastItem({ ...item, subTitle: videoCaptions }) +} + +async function addLivePodcastItem (options: { + feed: Feed + video: VideoModel +}) { + const { feed, video } = options + + let status: LiveItemStatus + + switch (video.state) { + case VideoState.WAITING_FOR_LIVE: + status = LiveItemStatus.pending + break + case VideoState.PUBLISHED: + status = LiveItemStatus.live + break + } + + const item = await generatePodcastItem({ video, liveItem: true, media: buildLiveStreamingPlaylists(video) }) + + feed.addPodcastLiveItem({ ...item, status, start: video.updatedAt.toISOString() }) +} + +// --------------------------------------------------------------------------- + +function buildVODWebVideoFile (video: MVideo, videoFile: VideoFile) { + const isAudio = videoFile.resolution.id === VideoResolution.H_NOVIDEO + const type = isAudio + ? MIMETYPES.AUDIO.EXT_MIMETYPE[extname(videoFile.fileUrl)] + : MIMETYPES.VIDEO.EXT_MIMETYPE[extname(videoFile.fileUrl)] + + const sources = [ + { uri: videoFile.fileUrl }, + { uri: videoFile.torrentUrl, contentType: 'application/x-bittorrent' } + ] + + if (videoFile.magnetUri) { + sources.push({ uri: videoFile.magnetUri }) + } + + return { + type, + title: videoFile.resolution.label, + length: videoFile.size, + bitrate: videoFile.size / video.duration * 8, + language: video.language, + sources + } +} + +function buildVODStreamingPlaylists (video: MVideoFullLight) { + const hls = video.getHLSPlaylist() + if (!hls) return [] + + return [ + { + type: 'application/x-mpegURL', + title: 'HLS', + sources: [ + { uri: hls.getMasterPlaylistUrl(video) } + ], + language: video.language + } + ] +} + +function buildLiveStreamingPlaylists (video: MVideoFullLight) { + const hls = video.getHLSPlaylist() + + return [ + { + type: 'application/x-mpegURL', + title: `HLS live stream`, + sources: [ + { uri: hls.getMasterPlaylistUrl(video) } + ], + language: video.language + } + ] +} + +function buildVODCaptions (video: MVideo, videoCaptions: MVideoCaptionVideo[]) { + return videoCaptions.map(caption => { + const type = MIMETYPES.VIDEO_CAPTIONS.EXT_MIMETYPE[extname(caption.filename)] + if (!type) return null + + return { + url: caption.getFileUrl(video), + language: caption.language, + type, + rel: 'captions' + } + }).filter(c => c) +} diff --git a/server/helpers/custom-validators/users.ts b/server/helpers/custom-validators/users.ts index 9df550fc2..f02b3ba65 100644 --- a/server/helpers/custom-validators/users.ts +++ b/server/helpers/custom-validators/users.ts @@ -80,6 +80,10 @@ function isUserAutoPlayNextVideoPlaylistValid (value: any) { return isBooleanValid(value) } +function isUserEmailPublicValid (value: any) { + return isBooleanValid(value) +} + function isUserNoModal (value: any) { return isBooleanValid(value) } @@ -114,5 +118,6 @@ export { isUserAutoPlayNextVideoPlaylistValid, isUserDisplayNameValid, isUserDescriptionValid, + isUserEmailPublicValid, isUserNoModal } diff --git a/server/initializers/constants.ts b/server/initializers/constants.ts index ba522c9de..020ed68da 100644 --- a/server/initializers/constants.ts +++ b/server/initializers/constants.ts @@ -27,7 +27,7 @@ import { CONFIG, registerConfigChangedHandler } from './config' // --------------------------------------------------------------------------- -const LAST_MIGRATION_VERSION = 770 +const LAST_MIGRATION_VERSION = 775 // --------------------------------------------------------------------------- @@ -634,7 +634,8 @@ const MIMETYPES = { 'text/vtt': '.vtt', 'application/x-subrip': '.srt', 'text/plain': '.srt' - } + }, + EXT_MIMETYPE: null as { [ id: string ]: string } }, TORRENT: { MIMETYPE_EXT: { @@ -649,6 +650,7 @@ const MIMETYPES = { } MIMETYPES.AUDIO.EXT_MIMETYPE = invert(MIMETYPES.AUDIO.MIMETYPE_EXT) MIMETYPES.IMAGE.EXT_MIMETYPE = invert(MIMETYPES.IMAGE.MIMETYPE_EXT) +MIMETYPES.VIDEO_CAPTIONS.EXT_MIMETYPE = invert(MIMETYPES.VIDEO_CAPTIONS.MIMETYPE_EXT) const BINARY_CONTENT_TYPES = new Set([ 'binary/octet-stream', diff --git a/server/initializers/migrations/0775-add-user-is-email-public.ts b/server/initializers/migrations/0775-add-user-is-email-public.ts new file mode 100644 index 000000000..74dee192c --- /dev/null +++ b/server/initializers/migrations/0775-add-user-is-email-public.ts @@ -0,0 +1,25 @@ +import * as Sequelize from 'sequelize' + +async function up (utils: { + transaction: Sequelize.Transaction + queryInterface: Sequelize.QueryInterface + sequelize: Sequelize.Sequelize +}): Promise { + + const data = { + type: Sequelize.BOOLEAN, + allowNull: false, + defaultValue: false + } + + await utils.queryInterface.addColumn('user', 'emailPublic', data) +} + +function down (options) { + throw new Error('Not implemented.') +} + +export { + up, + down +} diff --git a/server/lib/blocklist.ts b/server/lib/blocklist.ts index a11b717b5..009e229ce 100644 --- a/server/lib/blocklist.ts +++ b/server/lib/blocklist.ts @@ -1,6 +1,6 @@ import { sequelizeTypescript } from '@server/initializers/database' import { getServerActor } from '@server/models/application/application' -import { MAccountBlocklist, MAccountId, MAccountServer, MServerBlocklist } from '@server/types/models' +import { MAccountBlocklist, MAccountId, MAccountHost, MServerBlocklist } from '@server/types/models' import { AccountBlocklistModel } from '../models/account/account-blocklist' import { ServerBlocklistModel } from '../models/server/server-blocklist' @@ -34,7 +34,7 @@ function removeServerFromBlocklist (serverBlock: MServerBlocklist) { }) } -async function isBlockedByServerOrAccount (targetAccount: MAccountServer, userAccount?: MAccountId) { +async function isBlockedByServerOrAccount (targetAccount: MAccountHost, userAccount?: MAccountId) { const serverAccountId = (await getServerActor()).Account.id const sourceAccounts = [ serverAccountId ] diff --git a/server/lib/client-html.ts b/server/lib/client-html.ts index 058f29f03..18b16bee1 100644 --- a/server/lib/client-html.ts +++ b/server/lib/client-html.ts @@ -27,7 +27,7 @@ import { AccountModel } from '../models/account/account' import { VideoModel } from '../models/video/video' import { VideoChannelModel } from '../models/video/video-channel' import { VideoPlaylistModel } from '../models/video/video-playlist' -import { MAccountActor, MChannelActor, MVideo, MVideoPlaylist } from '../types/models' +import { MAccountHost, MChannelHost, MVideo, MVideoPlaylist } from '../types/models' import { getActivityStreamDuration } from './activitypub/activity' import { getBiggestActorImage } from './actor-image' import { Hooks } from './plugins/hooks' @@ -260,7 +260,7 @@ class ClientHtml { } private static async getAccountOrChannelHTMLPage ( - loader: () => Promise, + loader: () => Promise, req: express.Request, res: express.Response ) { @@ -280,7 +280,7 @@ class ClientHtml { let customHtml = ClientHtml.addTitleTag(html, entity.getDisplayName()) customHtml = ClientHtml.addDescriptionTag(customHtml, description) - const url = entity.getLocalUrl() + const url = entity.getClientUrl() const originUrl = entity.Actor.url const siteName = CONFIG.INSTANCE.NAME const title = entity.getDisplayName() diff --git a/server/lib/files-cache/videos-preview-cache.ts b/server/lib/files-cache/videos-preview-cache.ts index 48d2cb52c..d19c3f4f4 100644 --- a/server/lib/files-cache/videos-preview-cache.ts +++ b/server/lib/files-cache/videos-preview-cache.ts @@ -37,7 +37,7 @@ class VideosPreviewCache extends AbstractVideoStaticFileCache { const preview = video.getPreview() const destPath = join(FILES_CACHE.PREVIEWS.DIRECTORY, preview.filename) - const remoteUrl = preview.getFileUrl(video) + const remoteUrl = preview.getOriginFileUrl(video) try { await doRequestAndSaveToFile(remoteUrl, destPath) diff --git a/server/lib/internal-event-emitter.ts b/server/lib/internal-event-emitter.ts new file mode 100644 index 000000000..08b46a5c3 --- /dev/null +++ b/server/lib/internal-event-emitter.ts @@ -0,0 +1,35 @@ +import { MChannel, MVideo } from '@server/types/models' +import { EventEmitter } from 'events' + +export interface PeerTubeInternalEvents { + 'video-created': (options: { video: MVideo }) => void + 'video-updated': (options: { video: MVideo }) => void + 'video-deleted': (options: { video: MVideo }) => void + + 'channel-created': (options: { channel: MChannel }) => void + 'channel-updated': (options: { channel: MChannel }) => void + 'channel-deleted': (options: { channel: MChannel }) => void +} + +declare interface InternalEventEmitter { + on( + event: U, listener: PeerTubeInternalEvents[U] + ): this + + emit( + event: U, ...args: Parameters + ): boolean +} + +class InternalEventEmitter extends EventEmitter { + + private static instance: InternalEventEmitter + + static get Instance () { + return this.instance || (this.instance = new this()) + } +} + +export { + InternalEventEmitter +} diff --git a/server/lib/live/live-manager.ts b/server/lib/live/live-manager.ts index 5c6e69806..acb7af274 100644 --- a/server/lib/live/live-manager.ts +++ b/server/lib/live/live-manager.ts @@ -399,6 +399,8 @@ class LiveManager { } PeerTubeSocket.Instance.sendVideoLiveNewState(video) + + Hooks.runAction('action:live.video.state.updated', { video }) } catch (err) { logger.error('Cannot save/federate live video %d.', videoId, { err, ...localLTags }) } @@ -466,6 +468,8 @@ class LiveManager { PeerTubeSocket.Instance.sendVideoLiveNewState(fullVideo) await federateVideoIfNeeded(fullVideo, false) + + Hooks.runAction('action:live.video.state.updated', { video: fullVideo }) } catch (err) { logger.error('Cannot save/federate new video state of live streaming of video %s.', videoUUID, { err, ...lTags(videoUUID) }) } diff --git a/server/lib/plugins/plugin-helpers-builder.ts b/server/lib/plugins/plugin-helpers-builder.ts index 92ef87cca..d235f52c0 100644 --- a/server/lib/plugins/plugin-helpers-builder.ts +++ b/server/lib/plugins/plugin-helpers-builder.ts @@ -133,7 +133,7 @@ function buildVideosHelpers () { const thumbnails = video.Thumbnails.map(t => ({ type: t.type, - url: t.getFileUrl(video), + url: t.getOriginFileUrl(video), path: t.getPath() })) diff --git a/server/middlewares/cache/cache.ts b/server/middlewares/cache/cache.ts index e14160ba8..6041c76c3 100644 --- a/server/middlewares/cache/cache.ts +++ b/server/middlewares/cache/cache.ts @@ -17,12 +17,22 @@ function cacheRoute (duration: string) { function cacheRouteFactory (options: APICacheOptions) { const instance = new ApiCache({ ...defaultOptions, ...options }) - return instance.buildMiddleware.bind(instance) + return { instance, middleware: instance.buildMiddleware.bind(instance) } +} + +// --------------------------------------------------------------------------- + +function buildPodcastGroupsCache (options: { + channelId: number +}) { + return 'podcast-feed-' + options.channelId } // --------------------------------------------------------------------------- export { cacheRoute, - cacheRouteFactory + cacheRouteFactory, + + buildPodcastGroupsCache } diff --git a/server/middlewares/cache/shared/api-cache.ts b/server/middlewares/cache/shared/api-cache.ts index 7c366db00..c6197b972 100644 --- a/server/middlewares/cache/shared/api-cache.ts +++ b/server/middlewares/cache/shared/api-cache.ts @@ -27,7 +27,13 @@ export class ApiCache { private readonly options: APICacheOptions private readonly timers: { [ id: string ]: NodeJS.Timeout } = {} - private readonly index: { all: string[] } = { all: [] } + private readonly index = { + groups: [] as string[], + all: [] as string[] + } + + // Cache keys per group + private groups: { [groupIndex: string]: string[] } = {} constructor (options: APICacheOptions) { this.options = { @@ -43,7 +49,7 @@ export class ApiCache { return asyncMiddleware( async (req: express.Request, res: express.Response, next: express.NextFunction) => { - const key = Redis.Instance.getPrefix() + 'api-cache-' + req.originalUrl + const key = this.getCacheKey(req) const redis = Redis.Instance.getClient() if (!Redis.Instance.isConnected()) return this.makeResponseCacheable(res, next, key, duration) @@ -62,6 +68,29 @@ export class ApiCache { ) } + clearGroupSafe (group: string) { + const run = async () => { + const cacheKeys = this.groups[group] + if (!cacheKeys) return + + for (const key of cacheKeys) { + try { + await this.clear(key) + } catch (err) { + logger.error('Cannot clear ' + key, { err }) + } + } + + delete this.groups[group] + } + + void run() + } + + private getCacheKey (req: express.Request) { + return Redis.Instance.getPrefix() + 'api-cache-' + req.originalUrl + } + private shouldCacheResponse (response: express.Response) { if (!response) return false if (this.options.excludeStatus.includes(response.statusCode)) return false @@ -69,8 +98,16 @@ export class ApiCache { return true } - private addIndexEntries (key: string) { + private addIndexEntries (key: string, res: express.Response) { this.index.all.unshift(key) + + const groups = res.locals.apicacheGroups || [] + + for (const group of groups) { + if (!this.groups[group]) this.groups[group] = [] + + this.groups[group].push(key) + } } private filterBlacklistedHeaders (headers: OutgoingHttpHeaders) { @@ -177,7 +214,7 @@ export class ApiCache { self.accumulateContent(res, content) if (res.locals.apicache.cacheable && res.locals.apicache.content) { - self.addIndexEntries(key) + self.addIndexEntries(key, res) const headers = res.locals.apicache.headers || res.getHeaders() const cacheObject = self.createCacheObject( diff --git a/server/middlewares/validators/feeds.ts b/server/middlewares/validators/feeds.ts index 0bfe89e6f..ee8615cae 100644 --- a/server/middlewares/validators/feeds.ts +++ b/server/middlewares/validators/feeds.ts @@ -3,6 +3,7 @@ import { param, query } from 'express-validator' import { HttpStatusCode } from '../../../shared/models/http/http-error-codes' import { isValidRSSFeed } from '../../helpers/custom-validators/feeds' import { exists, isIdOrUUIDValid, isIdValid, toCompleteUUID } from '../../helpers/custom-validators/misc' +import { buildPodcastGroupsCache } from '../cache' import { areValidationErrors, checkCanSeeVideo, @@ -43,6 +44,21 @@ function setFeedFormatContentType (req: express.Request, res: express.Response, acceptableContentTypes = [ 'application/xml', 'text/xml' ] } + return feedContentTypeResponse(req, res, next, acceptableContentTypes) +} + +function setFeedPodcastContentType (req: express.Request, res: express.Response, next: express.NextFunction) { + const acceptableContentTypes = [ 'application/rss+xml', 'application/xml', 'text/xml' ] + + return feedContentTypeResponse(req, res, next, acceptableContentTypes) +} + +function feedContentTypeResponse ( + req: express.Request, + res: express.Response, + next: express.NextFunction, + acceptableContentTypes: string[] +) { if (req.accepts(acceptableContentTypes)) { res.set('Content-Type', req.accepts(acceptableContentTypes) as string) } else { @@ -55,6 +71,8 @@ function setFeedFormatContentType (req: express.Request, res: express.Response, return next() } +// --------------------------------------------------------------------------- + const videoFeedsValidator = [ query('accountId') .optional() @@ -82,6 +100,31 @@ const videoFeedsValidator = [ } ] +// --------------------------------------------------------------------------- + +const videoFeedsPodcastValidator = [ + query('videoChannelId') + .custom(isIdValid), + + async (req: express.Request, res: express.Response, next: express.NextFunction) => { + if (areValidationErrors(req, res)) return + if (!await doesVideoChannelIdExist(req.query.videoChannelId, res)) return + + return next() + } +] + +const videoFeedsPodcastSetCacheKey = [ + (req: express.Request, res: express.Response, next: express.NextFunction) => { + if (req.query.videoChannelId) { + res.locals.apicacheGroups = [ buildPodcastGroupsCache({ channelId: req.query.videoChannelId }) ] + } + + return next() + } +] +// --------------------------------------------------------------------------- + const videoSubscriptionFeedsValidator = [ query('accountId') .custom(isIdValid), @@ -126,7 +169,10 @@ const videoCommentsFeedsValidator = [ export { feedsFormatValidator, setFeedFormatContentType, + setFeedPodcastContentType, videoFeedsValidator, + videoFeedsPodcastValidator, videoSubscriptionFeedsValidator, + videoFeedsPodcastSetCacheKey, videoCommentsFeedsValidator } diff --git a/server/middlewares/validators/users.ts b/server/middlewares/validators/users.ts index 7ebea048d..3d311b15b 100644 --- a/server/middlewares/validators/users.ts +++ b/server/middlewares/validators/users.ts @@ -11,6 +11,7 @@ import { isUserBlockedReasonValid, isUserDescriptionValid, isUserDisplayNameValid, + isUserEmailPublicValid, isUserNoModal, isUserNSFWPolicyValid, isUserP2PEnabledValid, @@ -213,6 +214,9 @@ const usersUpdateMeValidator = [ body('password') .optional() .custom(isUserPasswordValid), + body('emailPublic') + .optional() + .custom(isUserEmailPublicValid), body('email') .optional() .isEmail(), diff --git a/server/models/account/account.ts b/server/models/account/account.ts index ec4e8d946..396959352 100644 --- a/server/models/account/account.ts +++ b/server/models/account/account.ts @@ -28,8 +28,9 @@ import { MAccountAP, MAccountDefault, MAccountFormattable, + MAccountHost, MAccountSummaryFormattable, - MChannelActor + MChannelHost } from '../../types/models' import { ActorModel } from '../actor/actor' import { ActorFollowModel } from '../actor/actor-follow' @@ -410,10 +411,6 @@ export class AccountModel extends Model>> { .findAll(query) } - getClientUrl () { - return WEBSERVER.URL + '/accounts/' + this.Actor.getIdentifier() - } - toFormattedJSON (this: MAccountFormattable): Account { return { ...this.Actor.toFormattedJSON(), @@ -463,8 +460,9 @@ export class AccountModel extends Model>> { return this.name } - getLocalUrl (this: MAccountActor | MChannelActor) { - return WEBSERVER.URL + `/accounts/` + this.Actor.preferredUsername + // Avoid error when running this method on MAccount... | MChannel... + getClientUrl (this: MAccountHost | MChannelHost) { + return WEBSERVER.URL + '/a/' + this.Actor.getIdentifier() } isBlocked () { diff --git a/server/models/actor/actor.ts b/server/models/actor/actor.ts index 80a646c77..dccb47a10 100644 --- a/server/models/actor/actor.ts +++ b/server/models/actor/actor.ts @@ -46,8 +46,8 @@ import { MActorFormattable, MActorFull, MActorHost, + MActorHostOnly, MActorId, - MActorServer, MActorSummaryFormattable, MActorUrl, MActorWithInboxes @@ -663,15 +663,15 @@ export class ActorModel extends Model>> { return this.serverId === null } - getWebfingerUrl (this: MActorServer) { + getWebfingerUrl (this: MActorHost) { return 'acct:' + this.preferredUsername + '@' + this.getHost() } - getIdentifier () { + getIdentifier (this: MActorHost) { return this.Server ? `${this.preferredUsername}@${this.Server.host}` : this.preferredUsername } - getHost (this: MActorHost) { + getHost (this: MActorHostOnly) { return this.Server ? this.Server.host : WEBSERVER.HOST } diff --git a/server/models/user/user.ts b/server/models/user/user.ts index 735b5c171..4f6a8fce4 100644 --- a/server/models/user/user.ts +++ b/server/models/user/user.ts @@ -404,6 +404,11 @@ export class UserModel extends Model>> { @Column lastLoginDate: Date + @AllowNull(false) + @Default(false) + @Column + emailPublic: boolean + @AllowNull(true) @Default(null) @Column @@ -880,6 +885,7 @@ export class UserModel extends Model>> { theme: getThemeOrDefault(this.theme, DEFAULT_USER_THEME_NAME), pendingEmail: this.pendingEmail, + emailPublic: this.emailPublic, emailVerified: this.emailVerified, nsfwPolicy: this.nsfwPolicy, diff --git a/server/models/video/formatter/video-format-utils.ts b/server/models/video/formatter/video-format-utils.ts index 6f05dbdc8..f2001e432 100644 --- a/server/models/video/formatter/video-format-utils.ts +++ b/server/models/video/formatter/video-format-utils.ts @@ -459,7 +459,7 @@ function videoModelToActivityPubObject (video: MVideoAP): VideoObject { icon: icons.map(i => ({ type: 'Image', - url: i.getFileUrl(video), + url: i.getOriginFileUrl(video), mediaType: 'image/jpeg', width: i.width, height: i.height diff --git a/server/models/video/thumbnail.ts b/server/models/video/thumbnail.ts index f33bd3179..a4ac581e5 100644 --- a/server/models/video/thumbnail.ts +++ b/server/models/video/thumbnail.ts @@ -164,7 +164,7 @@ export class ThumbnailModel extends Model return join(directory, filename) } - getFileUrl (video: MVideo) { + getOriginFileUrl (video: MVideo) { const staticPath = ThumbnailModel.types[this.type].staticPath + this.filename if (video.isOwned()) return WEBSERVER.URL + staticPath @@ -172,6 +172,10 @@ export class ThumbnailModel extends Model return this.fileUrl } + getLocalStaticPath () { + return ThumbnailModel.types[this.type].staticPath + this.filename + } + getPath () { return ThumbnailModel.buildPath(this.type, this.filename) } diff --git a/server/models/video/video-caption.ts b/server/models/video/video-caption.ts index 2eaa77407..1fb1cae82 100644 --- a/server/models/video/video-caption.ts +++ b/server/models/video/video-caption.ts @@ -1,6 +1,6 @@ import { remove } from 'fs-extra' import { join } from 'path' -import { OrderItem, Transaction } from 'sequelize' +import { Op, OrderItem, Transaction } from 'sequelize' import { AllowNull, BeforeDestroy, @@ -166,6 +166,31 @@ export class VideoCaptionModel extends Model(query) + const result: { [ id: number ]: MVideoCaptionVideo[] } = {} + + for (const id of videoIds) { + result[id] = [] + } + + for (const caption of captions) { + result[caption.videoId].push(caption) + } + + return result + } + static getLanguageLabel (language: string) { return VIDEO_LANGUAGES[language] || 'Unknown' } diff --git a/server/models/video/video-channel.ts b/server/models/video/video-channel.ts index 0fb52827e..19dd681a7 100644 --- a/server/models/video/video-channel.ts +++ b/server/models/video/video-channel.ts @@ -1,5 +1,8 @@ import { FindOptions, Includeable, literal, Op, QueryTypes, ScopeOptions, Transaction, WhereOptions } from 'sequelize' import { + AfterCreate, + AfterDestroy, + AfterUpdate, AllowNull, BeforeDestroy, BelongsTo, @@ -18,7 +21,8 @@ import { UpdatedAt } from 'sequelize-typescript' import { CONFIG } from '@server/initializers/config' -import { MAccountActor } from '@server/types/models' +import { InternalEventEmitter } from '@server/lib/internal-event-emitter' +import { MAccountHost } from '@server/types/models' import { forceNumber, pick } from '@shared/core-utils' import { AttributesOnly } from '@shared/typescript-utils' import { ActivityPubActor } from '../../../shared/models/activitypub' @@ -36,6 +40,7 @@ import { MChannelAP, MChannelBannerAccountDefault, MChannelFormattable, + MChannelHost, MChannelSummaryFormattable } from '../../types/models/video' import { AccountModel, ScopeNames as AccountModelScopeNames, SummaryOptions as AccountSummaryOptions } from '../account/account' @@ -416,6 +421,21 @@ export class VideoChannelModel extends Model>> { }) VideoJobInfo: VideoJobInfoModel + @AfterCreate + static notifyCreate (video: MVideo) { + InternalEventEmitter.Instance.emit('video-created', { video }) + } + + @AfterUpdate + static notifyUpdate (video: MVideo) { + InternalEventEmitter.Instance.emit('video-updated', { video }) + } + + @AfterDestroy + static notifyDestroy (video: MVideo) { + InternalEventEmitter.Instance.emit('video-deleted', { video }) + } + @BeforeDestroy - static async sendDelete (instance: MVideoAccountLight, options) { + static async sendDelete (instance: MVideoAccountLight, options: { transaction: Transaction }) { if (!instance.isOwned()) return undefined // Lazy load channels @@ -1686,15 +1704,14 @@ export class VideoModel extends Model>> { const thumbnail = this.getMiniature() if (!thumbnail) return null - return join(STATIC_PATHS.THUMBNAILS, thumbnail.filename) + return thumbnail.getLocalStaticPath() } getPreviewStaticPath () { const preview = this.getPreview() if (!preview) return null - // We use a local cache, so specify our cache endpoint instead of potential remote URL - return join(LAZY_STATIC_PATHS.PREVIEWS, preview.filename) + return preview.getLocalStaticPath() } toFormattedJSON (this: MVideoFormattable, options?: VideoFormattingJSONOptions): Video { @@ -1705,17 +1722,29 @@ export class VideoModel extends Model>> { return videoModelToFormattedDetailsJSON(this) } - getFormattedVideoFilesJSON (includeMagnet = true): VideoFile[] { + getFormattedWebVideoFilesJSON (includeMagnet = true): VideoFile[] { + return videoFilesModelToFormattedJSON(this, this.VideoFiles, { includeMagnet }) + } + + getFormattedHLSVideoFilesJSON (includeMagnet = true): VideoFile[] { + let acc: VideoFile[] = [] + + for (const p of this.VideoStreamingPlaylists) { + acc = acc.concat(videoFilesModelToFormattedJSON(this, p.VideoFiles, { includeMagnet })) + } + + return acc + } + + getFormattedAllVideoFilesJSON (includeMagnet = true): VideoFile[] { let files: VideoFile[] = [] if (Array.isArray(this.VideoFiles)) { - const result = videoFilesModelToFormattedJSON(this, this.VideoFiles, { includeMagnet }) - files = files.concat(result) + files = files.concat(this.getFormattedWebVideoFilesJSON(includeMagnet)) } - for (const p of (this.VideoStreamingPlaylists || [])) { - const result = videoFilesModelToFormattedJSON(this, p.VideoFiles, { includeMagnet }) - files = files.concat(result) + if (Array.isArray(this.VideoStreamingPlaylists)) { + files = files.concat(this.getFormattedHLSVideoFilesJSON(includeMagnet)) } return files diff --git a/server/tests/client.ts b/server/tests/client.ts index 9a20c2a10..e84251561 100644 --- a/server/tests/client.ts +++ b/server/tests/client.ts @@ -172,7 +172,7 @@ describe('Test a client controllers', function () { expect(text).to.contain(``) expect(text).to.contain(``) expect(text).to.contain('') - expect(text).to.contain(``) + expect(text).to.contain(``) } async function channelPageTest (path: string) { @@ -182,7 +182,7 @@ describe('Test a client controllers', function () { expect(text).to.contain(``) expect(text).to.contain(``) expect(text).to.contain('') - expect(text).to.contain(``) + expect(text).to.contain(``) } async function watchVideoPageTest (path: string) { diff --git a/server/tests/feeds/feeds.ts b/server/tests/feeds/feeds.ts index ecd1badc1..57eefff6d 100644 --- a/server/tests/feeds/feeds.ts +++ b/server/tests/feeds/feeds.ts @@ -11,6 +11,7 @@ import { makeGetRequest, makeRawRequest, PeerTubeServer, + PluginsCommand, setAccessTokensToServers, setDefaultChannelAvatar, stopFfmpeg, @@ -26,12 +27,15 @@ const expect = chai.expect describe('Test syndication feeds', () => { let servers: PeerTubeServer[] = [] let serverHLSOnly: PeerTubeServer + let userAccessToken: string let rootAccountId: number let rootChannelId: number + let userAccountId: number let userChannelId: number let userFeedToken: string + let liveId: string before(async function () { @@ -93,7 +97,11 @@ describe('Test syndication feeds', () => { await servers[0].comments.createThread({ videoId: id, text: 'comment on unlisted video' }) } - await waitJobs(servers) + await serverHLSOnly.videos.upload({ attributes: { name: 'hls only video' } }) + + await waitJobs([ ...servers, serverHLSOnly ]) + + await servers[0].plugins.install({ path: PluginsCommand.getPluginTestPath('-podcast-custom-tags') }) }) describe('All feed', function () { @@ -108,6 +116,11 @@ describe('Test syndication feeds', () => { } }) + it('Should be well formed XML (covers Podcast endpoint)', async function () { + const podcast = await servers[0].feed.getPodcastXML({ ignoreCache: true, channelId: rootChannelId }) + expect(podcast).xml.to.be.valid() + }) + it('Should be well formed JSON (covers JSON feed 1.0 endpoint)', async function () { for (const feed of [ 'video-comments' as 'video-comments', 'videos' as 'videos' ]) { const jsonText = await servers[0].feed.getJSON({ feed, ignoreCache: true }) @@ -153,168 +166,290 @@ describe('Test syndication feeds', () => { describe('Videos feed', function () { - it('Should contain a valid enclosure (covers RSS 2.0 endpoint)', async function () { - for (const server of servers) { - const rss = await server.feed.getXML({ feed: 'videos', ignoreCache: true }) + describe('Podcast feed', function () { + + it('Should contain a valid podcast:alternateEnclosure', async function () { + // Since podcast feeds should only work on the server they originate on, + // only test the first server where the videos reside + const rss = await servers[0].feed.getPodcastXML({ ignoreCache: false, channelId: rootChannelId }) expect(XMLValidator.validate(rss)).to.be.true const parser = new XMLParser({ parseAttributeValue: true, ignoreAttributes: false }) const xmlDoc = parser.parse(rss) - const enclosure = xmlDoc.rss.channel.item[0].enclosure + const enclosure = xmlDoc.rss.channel.item.enclosure expect(enclosure).to.exist + const alternateEnclosure = xmlDoc.rss.channel.item['podcast:alternateEnclosure'] + expect(alternateEnclosure).to.exist - expect(enclosure['@_type']).to.equal('video/webm') - expect(enclosure['@_length']).to.equal(218910) - expect(enclosure['@_url']).to.contain('-720.webm') - } + expect(alternateEnclosure['@_type']).to.equal('video/webm') + expect(alternateEnclosure['@_length']).to.equal(218910) + expect(alternateEnclosure['@_lang']).to.equal('zh') + expect(alternateEnclosure['@_title']).to.equal('720p') + expect(alternateEnclosure['@_default']).to.equal(true) + + expect(alternateEnclosure['podcast:source'][0]['@_uri']).to.contain('-720.webm') + expect(alternateEnclosure['podcast:source'][0]['@_uri']).to.equal(enclosure['@_url']) + expect(alternateEnclosure['podcast:source'][1]['@_uri']).to.contain('-720.torrent') + expect(alternateEnclosure['podcast:source'][1]['@_contentType']).to.equal('application/x-bittorrent') + expect(alternateEnclosure['podcast:source'][2]['@_uri']).to.contain('magnet:?') + }) + + it('Should contain a valid podcast:alternateEnclosure with HLS only', async function () { + const rss = await serverHLSOnly.feed.getPodcastXML({ ignoreCache: false, channelId: rootChannelId }) + expect(XMLValidator.validate(rss)).to.be.true + + const parser = new XMLParser({ parseAttributeValue: true, ignoreAttributes: false }) + const xmlDoc = parser.parse(rss) + + const enclosure = xmlDoc.rss.channel.item.enclosure + const alternateEnclosure = xmlDoc.rss.channel.item['podcast:alternateEnclosure'] + expect(alternateEnclosure).to.exist + + expect(alternateEnclosure['@_type']).to.equal('application/x-mpegURL') + expect(alternateEnclosure['@_lang']).to.equal('zh') + expect(alternateEnclosure['@_title']).to.equal('HLS') + expect(alternateEnclosure['@_default']).to.equal(true) + + expect(alternateEnclosure['podcast:source']['@_uri']).to.contain('-master.m3u8') + expect(alternateEnclosure['podcast:source']['@_uri']).to.equal(enclosure['@_url']) + }) + + it('Should contain a valid podcast:socialInteract', async function () { + const rss = await servers[0].feed.getPodcastXML({ ignoreCache: false, channelId: rootChannelId }) + expect(XMLValidator.validate(rss)).to.be.true + + const parser = new XMLParser({ parseAttributeValue: true, ignoreAttributes: false }) + const xmlDoc = parser.parse(rss) + + const item = xmlDoc.rss.channel.item + const socialInteract = item['podcast:socialInteract'] + expect(socialInteract).to.exist + expect(socialInteract['@_protocol']).to.equal('activitypub') + expect(socialInteract['@_uri']).to.exist + expect(socialInteract['@_accountUrl']).to.exist + }) + + it('Should contain a valid support custom tags for plugins', async function () { + const rss = await servers[0].feed.getPodcastXML({ ignoreCache: false, channelId: userChannelId }) + expect(XMLValidator.validate(rss)).to.be.true + + const parser = new XMLParser({ parseAttributeValue: true, ignoreAttributes: false }) + const xmlDoc = parser.parse(rss) + + const fooTag = xmlDoc.rss.channel.fooTag + expect(fooTag).to.exist + expect(fooTag['@_bar']).to.equal('baz') + expect(fooTag['#text']).to.equal(42) + + const bizzBuzzItem = xmlDoc.rss.channel['biz:buzzItem'] + expect(bizzBuzzItem).to.exist + + let nestedTag = bizzBuzzItem.nestedTag + expect(nestedTag).to.exist + expect(nestedTag).to.equal('example nested tag') + + const item = xmlDoc.rss.channel.item + const fizzTag = item.fizzTag + expect(fizzTag).to.exist + expect(fizzTag['@_bar']).to.equal('baz') + expect(fizzTag['#text']).to.equal(21) + + const bizzBuzz = item['biz:buzz'] + expect(bizzBuzz).to.exist + + nestedTag = bizzBuzz.nestedTag + expect(nestedTag).to.exist + expect(nestedTag).to.equal('example nested tag') + }) + + it('Should contain a valid podcast:liveItem for live streams', async function () { + this.timeout(120000) + + const { uuid } = await servers[0].live.create({ + fields: { + name: 'live-0', + privacy: VideoPrivacy.PUBLIC, + channelId: rootChannelId, + permanentLive: false + } + }) + liveId = uuid + + const ffmpeg = await servers[0].live.sendRTMPStreamInVideo({ videoId: liveId, copyCodecs: true, fixtureName: 'video_short.mp4' }) + await servers[0].live.waitUntilPublished({ videoId: liveId }) + + const rss = await servers[0].feed.getPodcastXML({ ignoreCache: false, channelId: rootChannelId }) + expect(XMLValidator.validate(rss)).to.be.true + + const parser = new XMLParser({ parseAttributeValue: true, ignoreAttributes: false }) + const xmlDoc = parser.parse(rss) + const liveItem = xmlDoc.rss.channel['podcast:liveItem'] + expect(liveItem.title).to.equal('live-0') + expect(liveItem['@_status']).to.equal('live') + + const enclosure = liveItem.enclosure + const alternateEnclosure = liveItem['podcast:alternateEnclosure'] + expect(alternateEnclosure).to.exist + expect(alternateEnclosure['@_type']).to.equal('application/x-mpegURL') + expect(alternateEnclosure['@_title']).to.equal('HLS live stream') + expect(alternateEnclosure['@_default']).to.equal(true) + + expect(alternateEnclosure['podcast:source']['@_uri']).to.contain('/master.m3u8') + expect(alternateEnclosure['podcast:source']['@_uri']).to.equal(enclosure['@_url']) + + await stopFfmpeg(ffmpeg) + + await servers[0].live.waitUntilEnded({ videoId: liveId }) + + await waitJobs(servers) + }) }) - it('Should contain a valid \'attachments\' object (covers JSON feed 1.0 endpoint)', async function () { - for (const server of servers) { - const json = await server.feed.getJSON({ feed: 'videos', ignoreCache: true }) - const jsonObj = JSON.parse(json) - expect(jsonObj.items.length).to.be.equal(2) - expect(jsonObj.items[0].attachments).to.exist - expect(jsonObj.items[0].attachments.length).to.be.eq(1) - expect(jsonObj.items[0].attachments[0].mime_type).to.be.eq('application/x-bittorrent') - expect(jsonObj.items[0].attachments[0].size_in_bytes).to.be.eq(218910) - expect(jsonObj.items[0].attachments[0].url).to.contain('720.torrent') - } - }) + describe('JSON feed', function () { - it('Should filter by account', async function () { - { - const json = await servers[0].feed.getJSON({ feed: 'videos', query: { accountId: rootAccountId }, ignoreCache: true }) - const jsonObj = JSON.parse(json) - expect(jsonObj.items.length).to.be.equal(1) - expect(jsonObj.items[0].title).to.equal('my super name for server 1') - expect(jsonObj.items[0].author.name).to.equal('Main root channel') - } - - { - const json = await servers[0].feed.getJSON({ feed: 'videos', query: { accountId: userAccountId }, ignoreCache: true }) - const jsonObj = JSON.parse(json) - expect(jsonObj.items.length).to.be.equal(1) - expect(jsonObj.items[0].title).to.equal('user video') - expect(jsonObj.items[0].author.name).to.equal('Main john channel') - } - - for (const server of servers) { - { - const json = await server.feed.getJSON({ feed: 'videos', query: { accountName: 'root@' + servers[0].host }, ignoreCache: true }) + it('Should contain a valid \'attachments\' object', async function () { + for (const server of servers) { + const json = await server.feed.getJSON({ feed: 'videos', ignoreCache: true }) const jsonObj = JSON.parse(json) - expect(jsonObj.items.length).to.be.equal(1) - expect(jsonObj.items[0].title).to.equal('my super name for server 1') - } - - { - const json = await server.feed.getJSON({ feed: 'videos', query: { accountName: 'john@' + servers[0].host }, ignoreCache: true }) - const jsonObj = JSON.parse(json) - expect(jsonObj.items.length).to.be.equal(1) - expect(jsonObj.items[0].title).to.equal('user video') - } - } - }) - - it('Should filter by video channel', async function () { - { - const json = await servers[0].feed.getJSON({ feed: 'videos', query: { videoChannelId: rootChannelId }, ignoreCache: true }) - const jsonObj = JSON.parse(json) - expect(jsonObj.items.length).to.be.equal(1) - expect(jsonObj.items[0].title).to.equal('my super name for server 1') - expect(jsonObj.items[0].author.name).to.equal('Main root channel') - } - - { - const json = await servers[0].feed.getJSON({ feed: 'videos', query: { videoChannelId: userChannelId }, ignoreCache: true }) - const jsonObj = JSON.parse(json) - expect(jsonObj.items.length).to.be.equal(1) - expect(jsonObj.items[0].title).to.equal('user video') - expect(jsonObj.items[0].author.name).to.equal('Main john channel') - } - - for (const server of servers) { - { - const query = { videoChannelName: 'root_channel@' + servers[0].host } - const json = await server.feed.getJSON({ feed: 'videos', query, ignoreCache: true }) - const jsonObj = JSON.parse(json) - expect(jsonObj.items.length).to.be.equal(1) - expect(jsonObj.items[0].title).to.equal('my super name for server 1') - } - - { - const query = { videoChannelName: 'john_channel@' + servers[0].host } - const json = await server.feed.getJSON({ feed: 'videos', query, ignoreCache: true }) - const jsonObj = JSON.parse(json) - expect(jsonObj.items.length).to.be.equal(1) - expect(jsonObj.items[0].title).to.equal('user video') - } - } - }) - - it('Should correctly have videos feed with HLS only', async function () { - this.timeout(120000) - - await serverHLSOnly.videos.upload({ attributes: { name: 'hls only video' } }) - - await waitJobs([ serverHLSOnly ]) - - const json = await serverHLSOnly.feed.getJSON({ feed: 'videos', ignoreCache: true }) - const jsonObj = JSON.parse(json) - expect(jsonObj.items.length).to.be.equal(1) - expect(jsonObj.items[0].attachments).to.exist - expect(jsonObj.items[0].attachments.length).to.be.eq(4) - - for (let i = 0; i < 4; i++) { - expect(jsonObj.items[0].attachments[i].mime_type).to.be.eq('application/x-bittorrent') - expect(jsonObj.items[0].attachments[i].size_in_bytes).to.be.greaterThan(0) - expect(jsonObj.items[0].attachments[i].url).to.exist - } - }) - - it('Should not display waiting live videos', async function () { - const { uuid } = await servers[0].live.create({ - fields: { - name: 'live', - privacy: VideoPrivacy.PUBLIC, - channelId: rootChannelId + expect(jsonObj.items.length).to.be.equal(2) + expect(jsonObj.items[0].attachments).to.exist + expect(jsonObj.items[0].attachments.length).to.be.eq(1) + expect(jsonObj.items[0].attachments[0].mime_type).to.be.eq('application/x-bittorrent') + expect(jsonObj.items[0].attachments[0].size_in_bytes).to.be.eq(218910) + expect(jsonObj.items[0].attachments[0].url).to.contain('720.torrent') } }) - liveId = uuid - const json = await servers[0].feed.getJSON({ feed: 'videos', ignoreCache: true }) + it('Should filter by account', async function () { + { + const json = await servers[0].feed.getJSON({ feed: 'videos', query: { accountId: rootAccountId }, ignoreCache: true }) + const jsonObj = JSON.parse(json) + expect(jsonObj.items.length).to.be.equal(1) + expect(jsonObj.items[0].title).to.equal('my super name for server 1') + expect(jsonObj.items[0].author.name).to.equal('Main root channel') + } - const jsonObj = JSON.parse(json) - expect(jsonObj.items.length).to.be.equal(2) - expect(jsonObj.items[0].title).to.equal('my super name for server 1') - expect(jsonObj.items[1].title).to.equal('user video') - }) + { + const json = await servers[0].feed.getJSON({ feed: 'videos', query: { accountId: userAccountId }, ignoreCache: true }) + const jsonObj = JSON.parse(json) + expect(jsonObj.items.length).to.be.equal(1) + expect(jsonObj.items[0].title).to.equal('user video') + expect(jsonObj.items[0].author.name).to.equal('Main john channel') + } - it('Should display published live videos', async function () { - this.timeout(120000) + for (const server of servers) { + { + const json = await server.feed.getJSON({ feed: 'videos', query: { accountName: 'root@' + servers[0].host }, ignoreCache: true }) + const jsonObj = JSON.parse(json) + expect(jsonObj.items.length).to.be.equal(1) + expect(jsonObj.items[0].title).to.equal('my super name for server 1') + } - const ffmpeg = await servers[0].live.sendRTMPStreamInVideo({ videoId: liveId, copyCodecs: true, fixtureName: 'video_short.mp4' }) - await servers[0].live.waitUntilPublished({ videoId: liveId }) + { + const json = await server.feed.getJSON({ feed: 'videos', query: { accountName: 'john@' + servers[0].host }, ignoreCache: true }) + const jsonObj = JSON.parse(json) + expect(jsonObj.items.length).to.be.equal(1) + expect(jsonObj.items[0].title).to.equal('user video') + } + } + }) - const json = await servers[0].feed.getJSON({ feed: 'videos', ignoreCache: true }) + it('Should filter by video channel', async function () { + { + const json = await servers[0].feed.getJSON({ feed: 'videos', query: { videoChannelId: rootChannelId }, ignoreCache: true }) + const jsonObj = JSON.parse(json) + expect(jsonObj.items.length).to.be.equal(1) + expect(jsonObj.items[0].title).to.equal('my super name for server 1') + expect(jsonObj.items[0].author.name).to.equal('Main root channel') + } - const jsonObj = JSON.parse(json) - expect(jsonObj.items.length).to.be.equal(3) - expect(jsonObj.items[0].title).to.equal('live') - expect(jsonObj.items[1].title).to.equal('my super name for server 1') - expect(jsonObj.items[2].title).to.equal('user video') + { + const json = await servers[0].feed.getJSON({ feed: 'videos', query: { videoChannelId: userChannelId }, ignoreCache: true }) + const jsonObj = JSON.parse(json) + expect(jsonObj.items.length).to.be.equal(1) + expect(jsonObj.items[0].title).to.equal('user video') + expect(jsonObj.items[0].author.name).to.equal('Main john channel') + } - await stopFfmpeg(ffmpeg) - }) + for (const server of servers) { + { + const query = { videoChannelName: 'root_channel@' + servers[0].host } + const json = await server.feed.getJSON({ feed: 'videos', query, ignoreCache: true }) + const jsonObj = JSON.parse(json) + expect(jsonObj.items.length).to.be.equal(1) + expect(jsonObj.items[0].title).to.equal('my super name for server 1') + } - it('Should have the channel avatar as feed icon', async function () { - const json = await servers[0].feed.getJSON({ feed: 'videos', query: { videoChannelId: rootChannelId }, ignoreCache: true }) + { + const query = { videoChannelName: 'john_channel@' + servers[0].host } + const json = await server.feed.getJSON({ feed: 'videos', query, ignoreCache: true }) + const jsonObj = JSON.parse(json) + expect(jsonObj.items.length).to.be.equal(1) + expect(jsonObj.items[0].title).to.equal('user video') + } + } + }) - const jsonObj = JSON.parse(json) - const imageUrl = jsonObj.icon - expect(imageUrl).to.include('/lazy-static/avatars/') - await makeRawRequest({ url: imageUrl, expectedStatus: HttpStatusCode.OK_200 }) + it('Should correctly have videos feed with HLS only', async function () { + this.timeout(120000) + + const json = await serverHLSOnly.feed.getJSON({ feed: 'videos', ignoreCache: true }) + const jsonObj = JSON.parse(json) + expect(jsonObj.items.length).to.be.equal(1) + expect(jsonObj.items[0].attachments).to.exist + expect(jsonObj.items[0].attachments.length).to.be.eq(4) + + for (let i = 0; i < 4; i++) { + expect(jsonObj.items[0].attachments[i].mime_type).to.be.eq('application/x-bittorrent') + expect(jsonObj.items[0].attachments[i].size_in_bytes).to.be.greaterThan(0) + expect(jsonObj.items[0].attachments[i].url).to.exist + } + }) + + it('Should not display waiting live videos', async function () { + const { uuid } = await servers[0].live.create({ + fields: { + name: 'live', + privacy: VideoPrivacy.PUBLIC, + channelId: rootChannelId + } + }) + liveId = uuid + + const json = await servers[0].feed.getJSON({ feed: 'videos', ignoreCache: true }) + + const jsonObj = JSON.parse(json) + expect(jsonObj.items.length).to.be.equal(2) + expect(jsonObj.items[0].title).to.equal('my super name for server 1') + expect(jsonObj.items[1].title).to.equal('user video') + }) + + it('Should display published live videos', async function () { + this.timeout(120000) + + const ffmpeg = await servers[0].live.sendRTMPStreamInVideo({ videoId: liveId, copyCodecs: true, fixtureName: 'video_short.mp4' }) + await servers[0].live.waitUntilPublished({ videoId: liveId }) + + const json = await servers[0].feed.getJSON({ feed: 'videos', ignoreCache: true }) + + const jsonObj = JSON.parse(json) + expect(jsonObj.items.length).to.be.equal(3) + expect(jsonObj.items[0].title).to.equal('live') + expect(jsonObj.items[1].title).to.equal('my super name for server 1') + expect(jsonObj.items[2].title).to.equal('user video') + + await stopFfmpeg(ffmpeg) + }) + + it('Should have the channel avatar as feed icon', async function () { + const json = await servers[0].feed.getJSON({ feed: 'videos', query: { videoChannelId: rootChannelId }, ignoreCache: true }) + + const jsonObj = JSON.parse(json) + const imageUrl = jsonObj.icon + expect(imageUrl).to.include('/lazy-static/avatars/') + await makeRawRequest({ url: imageUrl, expectedStatus: HttpStatusCode.OK_200 }) + }) }) }) @@ -470,6 +605,8 @@ describe('Test syndication feeds', () => { }) after(async function () { + await servers[0].plugins.uninstall({ npmName: 'peertube-plugin-test-podcast-custom-tags' }) + await cleanupTests([ ...servers, serverHLSOnly ]) }) }) diff --git a/server/tests/fixtures/peertube-plugin-test-podcast-custom-tags/main.js b/server/tests/fixtures/peertube-plugin-test-podcast-custom-tags/main.js new file mode 100644 index 000000000..ada4a70fe --- /dev/null +++ b/server/tests/fixtures/peertube-plugin-test-podcast-custom-tags/main.js @@ -0,0 +1,82 @@ +async function register ({ registerHook, registerSetting, settingsManager, storageManager, peertubeHelpers }) { + registerHook({ + target: 'filter:feed.podcast.rss.create-custom-xmlns.result', + handler: (result, params) => { + return result.concat([ + { + name: "biz", + value: "https://example.com/biz-xmlns", + }, + ]) + } + }) + + registerHook({ + target: 'filter:feed.podcast.channel.create-custom-tags.result', + handler: (result, params) => { + const { videoChannel } = params + return result.concat([ + { + name: "fooTag", + attributes: { "bar": "baz" }, + value: "42", + }, + { + name: "biz:videoChannel", + attributes: { "name": videoChannel.name, "id": videoChannel.id }, + }, + { + name: "biz:buzzItem", + value: [ + { + name: "nestedTag", + value: "example nested tag", + }, + ], + }, + ]) + } + }) + + registerHook({ + target: 'filter:feed.podcast.video.create-custom-tags.result', + handler: (result, params) => { + const { video, liveItem } = params + return result.concat([ + { + name: "fizzTag", + attributes: { "bar": "baz" }, + value: "21", + }, + { + name: "biz:video", + attributes: { "name": video.name, "id": video.id, "isLive": liveItem }, + }, + { + name: "biz:buzz", + value: [ + { + name: "nestedTag", + value: "example nested tag", + }, + ], + } + ]) + } + }) +} + +async function unregister () { + return +} + +module.exports = { + register, + unregister +} + +// ############################################################################ + +function addToCount (obj) { + return Object.assign({}, obj, { count: obj.count + 1 }) +} diff --git a/server/tests/fixtures/peertube-plugin-test-podcast-custom-tags/package.json b/server/tests/fixtures/peertube-plugin-test-podcast-custom-tags/package.json new file mode 100644 index 000000000..0f5a05a79 --- /dev/null +++ b/server/tests/fixtures/peertube-plugin-test-podcast-custom-tags/package.json @@ -0,0 +1,19 @@ +{ + "name": "peertube-plugin-test-podcast-custom-tags", + "version": "0.0.1", + "description": "Plugin test custom tags in Podcast RSS feeds", + "engine": { + "peertube": ">=1.3.0" + }, + "keywords": [ + "peertube", + "plugin" + ], + "homepage": "https://github.com/Chocobozzz/PeerTube", + "author": "Chocobozzz", + "bugs": "https://github.com/Chocobozzz/PeerTube/issues", + "library": "./main.js", + "staticDirs": {}, + "css": [], + "clientScripts": [] +} diff --git a/server/tests/fixtures/peertube-plugin-test/main.js b/server/tests/fixtures/peertube-plugin-test/main.js index 36dd08d27..17032f6d9 100644 --- a/server/tests/fixtures/peertube-plugin-test/main.js +++ b/server/tests/fixtures/peertube-plugin-test/main.js @@ -14,6 +14,7 @@ async function register ({ registerHook, registerSetting, settingsManager, stora 'action:api.video-channel.deleted', 'action:api.live-video.created', + 'action:live.video.state.updated', 'action:api.video-thread.created', 'action:api.video-comment-reply.created', diff --git a/server/tests/plugins/action-hooks.ts b/server/tests/plugins/action-hooks.ts index e8d03ee0f..34b4e1891 100644 --- a/server/tests/plugins/action-hooks.ts +++ b/server/tests/plugins/action-hooks.ts @@ -9,7 +9,9 @@ import { PeerTubeServer, PluginsCommand, setAccessTokensToServers, - setDefaultVideoChannel + setDefaultVideoChannel, + stopFfmpeg, + waitJobs } from '@shared/server-commands' describe('Test plugin action hooks', function () { @@ -17,8 +19,8 @@ describe('Test plugin action hooks', function () { let videoUUID: string let threadId: number - function checkHook (hook: ServerHookName, strictCount = true) { - return servers[0].servers.waitUntilLog('Run hook ' + hook, 1, strictCount) + function checkHook (hook: ServerHookName, strictCount = true, count = 1) { + return servers[0].servers.waitUntilLog('Run hook ' + hook, count, strictCount) } before(async function () { @@ -115,6 +117,29 @@ describe('Test plugin action hooks', function () { await checkHook('action:api.live-video.created') }) + + it('Should run action:live.video.state.updated', async function () { + this.timeout(60000) + + const attributes = { + name: 'live', + privacy: VideoPrivacy.PUBLIC, + channelId: servers[0].store.channel.id + } + + const { uuid: liveVideoId } = await servers[0].live.create({ fields: attributes }) + const ffmpegCommand = await servers[0].live.sendRTMPStreamInVideo({ videoId: liveVideoId }) + await servers[0].live.waitUntilPublished({ videoId: liveVideoId }) + await waitJobs(servers) + + await checkHook('action:live.video.state.updated', true, 1) + + await stopFfmpeg(ffmpegCommand) + await servers[0].live.waitUntilEnded({ videoId: liveVideoId }) + await waitJobs(servers) + + await checkHook('action:live.video.state.updated', true, 2) + }) }) describe('Comments hooks', function () { diff --git a/server/types/express.d.ts b/server/types/express.d.ts index a8aeabb3a..510b9f94e 100644 --- a/server/types/express.d.ts +++ b/server/types/express.d.ts @@ -110,6 +110,8 @@ declare module 'express' { locals: { requestStart: number + apicacheGroups: string[] + apicache: { content: string | Buffer write: Writable['write'] diff --git a/server/types/models/account/account.ts b/server/types/models/account/account.ts index 282a2971b..d10b904ab 100644 --- a/server/types/models/account/account.ts +++ b/server/types/models/account/account.ts @@ -8,8 +8,8 @@ import { MActorDefault, MActorDefaultLight, MActorFormattable, + MActorHost, MActorId, - MActorServer, MActorSummary, MActorSummaryFormattable, MActorUrl @@ -68,10 +68,9 @@ export type MAccountActor = MAccount & Use<'Actor', MActor> -// Full actor with server -export type MAccountServer = +export type MAccountHost = MAccount & - Use<'Actor', MActorServer> + Use<'Actor', MActorHost> // ############################################################################ diff --git a/server/types/models/actor/actor-follow.ts b/server/types/models/actor/actor-follow.ts index 338158561..84042e228 100644 --- a/server/types/models/actor/actor-follow.ts +++ b/server/types/models/actor/actor-follow.ts @@ -7,7 +7,7 @@ import { MActorDefaultAccountChannel, MActorDefaultChannelId, MActorFormattable, - MActorHost, + MActorHostOnly, MActorUsername } from './actor' @@ -21,7 +21,7 @@ export type MActorFollow = Omit + Use<'ActorFollowing', MActorUsername & MActorHostOnly> // ############################################################################ diff --git a/server/types/models/actor/actor.ts b/server/types/models/actor/actor.ts index 280256bab..47e7b7091 100644 --- a/server/types/models/actor/actor.ts +++ b/server/types/models/actor/actor.ts @@ -29,7 +29,11 @@ export type MActorLight = Omit // Some association attributes -export type MActorHost = Use<'Server', MServerHost> +export type MActorHostOnly = Use<'Server', MServerHost> +export type MActorHost = + MActorLight & + Use<'Server', MServerHost> + export type MActorRedundancyAllowedOpt = PickWithOpt export type MActorDefaultLight = @@ -68,8 +72,8 @@ export type MActorChannel = export type MActorDefaultAccountChannel = MActorDefault & MActorAccount & MActorChannel -export type MActorServer = - MActor & +export type MActorServerLight = + MActorLight & Use<'Server', MServer> // ############################################################################ diff --git a/server/types/models/video/video-channels.ts b/server/types/models/video/video-channels.ts index af8c2ffe4..57e991494 100644 --- a/server/types/models/video/video-channels.ts +++ b/server/types/models/video/video-channels.ts @@ -21,6 +21,7 @@ import { MActorDefaultLight, MActorFormattable, MActorHost, + MActorHostOnly, MActorLight, MActorSummary, MActorSummaryFormattable, @@ -77,9 +78,13 @@ export type MChannelAccountLight = Use<'Account', MAccountLight> export type MChannelHost = - MChannelId & + MChannel & Use<'Actor', MActorHost> +export type MChannelHostOnly = + MChannelId & + Use<'Actor', MActorHostOnly> + // ############################################################################ // Account associations diff --git a/server/types/models/video/video.ts b/server/types/models/video/video.ts index d1af53b92..58ae7baad 100644 --- a/server/types/models/video/video.ts +++ b/server/types/models/video/video.ts @@ -13,7 +13,7 @@ import { MChannelAccountSummaryFormattable, MChannelActor, MChannelFormattable, - MChannelHost, + MChannelHostOnly, MChannelUserId } from './video-channels' import { MVideoFile, MVideoFileRedundanciesAll, MVideoFileRedundanciesOpt } from './video-file' @@ -146,7 +146,7 @@ export type MVideoWithChannelActor = export type MVideoWithHost = MVideo & - Use<'VideoChannel', MChannelHost> + Use<'VideoChannel', MChannelHostOnly> export type MVideoFullLight = MVideo & diff --git a/shared/models/plugins/server/server-hook.model.ts b/shared/models/plugins/server/server-hook.model.ts index 4c9d86079..0ec62222d 100644 --- a/shared/models/plugins/server/server-hook.model.ts +++ b/shared/models/plugins/server/server-hook.model.ts @@ -122,7 +122,17 @@ export const serverFilterHookObject = { // Filter the result of video JSON LD builder // You may also need to use filter:activity-pub.activity.context.build.result to also update JSON LD context - 'filter:activity-pub.video.json-ld.build.result': true + 'filter:activity-pub.video.json-ld.build.result': true, + + // Filter result to allow custom XMLNS definitions in podcast RSS feeds + // Peertube >= 5.2 + 'filter:feed.podcast.rss.create-custom-xmlns.result': true, + + // Filter result to allow custom tags in podcast RSS feeds + // Peertube >= 5.2 + 'filter:feed.podcast.channel.create-custom-tags.result': true, + // Peertube >= 5.2 + 'filter:feed.podcast.video.create-custom-tags.result': true } export type ServerFilterHookName = keyof typeof serverFilterHookObject @@ -154,6 +164,9 @@ export const serverActionHookObject = { // Fired when a live video is created 'action:api.live-video.created': true, + // Fired when a live video starts or ends + // Peertube >= 5.2 + 'action:live.video.state.updated': true, // Fired when a thread is created 'action:api.video-thread.created': true, diff --git a/shared/models/users/user-update-me.model.ts b/shared/models/users/user-update-me.model.ts index e664e44b5..f3cceb5f2 100644 --- a/shared/models/users/user-update-me.model.ts +++ b/shared/models/users/user-update-me.model.ts @@ -16,6 +16,7 @@ export interface UserUpdateMe { videoLanguages?: string[] email?: string + emailPublic?: boolean currentPassword?: string password?: string diff --git a/shared/models/users/user.model.ts b/shared/models/users/user.model.ts index 761a2edba..0761c1e32 100644 --- a/shared/models/users/user.model.ts +++ b/shared/models/users/user.model.ts @@ -13,6 +13,7 @@ export interface User { pendingEmail: string | null emailVerified: boolean + emailPublic: boolean nsfwPolicy: NSFWPolicyType adminFlags?: UserAdminFlag diff --git a/shared/models/videos/video-include.enum.ts b/shared/models/videos/video-include.enum.ts index 7e16b129a..32ee12e86 100644 --- a/shared/models/videos/video-include.enum.ts +++ b/shared/models/videos/video-include.enum.ts @@ -3,5 +3,6 @@ export const enum VideoInclude { NOT_PUBLISHED_STATE = 1 << 0, BLACKLISTED = 1 << 1, BLOCKED_OWNER = 1 << 2, - FILES = 1 << 3 + FILES = 1 << 3, + CAPTIONS = 1 << 4 } diff --git a/shared/server-commands/feeds/feeds-command.ts b/shared/server-commands/feeds/feeds-command.ts index 939b18dee..26763b43e 100644 --- a/shared/server-commands/feeds/feeds-command.ts +++ b/shared/server-commands/feeds/feeds-command.ts @@ -30,6 +30,29 @@ export class FeedCommand extends AbstractCommand { }) } + getPodcastXML (options: OverrideCommandOptions & { + ignoreCache: boolean + channelId: number + }) { + const { ignoreCache, channelId } = options + const path = `/feeds/podcast/videos.xml` + + const query: { [id: string]: string } = {} + + if (ignoreCache) query.v = buildUUID() + if (channelId) query.videoChannelId = channelId + '' + + return this.getRequestText({ + ...options, + + path, + query, + accept: 'application/xml', + implicitToken: false, + defaultExpectedStatus: HttpStatusCode.OK_200 + }) + } + getJSON (options: OverrideCommandOptions & { feed: FeedType ignoreCache: boolean diff --git a/support/doc/api/openapi.yaml b/support/doc/api/openapi.yaml index 4230fc827..cd50e86a6 100644 --- a/support/doc/api/openapi.yaml +++ b/support/doc/api/openapi.yaml @@ -433,7 +433,7 @@ paths: get: tags: - Video Feeds - summary: List comments on videos + summary: Comments on videos feeds operationId: getSyndicatedComments parameters: - name: format @@ -476,7 +476,7 @@ paths: schema: type: string responses: - '204': + '200': description: successful operation headers: Cache-Control: @@ -528,7 +528,7 @@ paths: get: tags: - Video Feeds - summary: List videos + summary: Common videos feeds operationId: getSyndicatedVideos parameters: - name: format @@ -573,7 +573,7 @@ paths: - $ref: '#/components/parameters/hasHLSFiles' - $ref: '#/components/parameters/hasWebtorrentFiles' responses: - '204': + '200': description: successful operation headers: Cache-Control: @@ -620,7 +620,7 @@ paths: get: tags: - Video Feeds - summary: List videos of subscriptions tied to a token + summary: Videos of subscriptions feeds operationId: getSyndicatedSubscriptionVideos parameters: - name: format @@ -657,7 +657,7 @@ paths: - $ref: '#/components/parameters/hasHLSFiles' - $ref: '#/components/parameters/hasWebtorrentFiles' responses: - '204': + '200': description: successful operation headers: Cache-Control: @@ -683,6 +683,30 @@ paths: '406': description: accept header unsupported + '/feeds/podcast/videos.xml': + get: + tags: + - Video Feeds + summary: Videos podcast feed + operationId: getVideosPodcastFeed + parameters: + - name: videoChannelId + in: query + description: 'Limit listing to a specific video channel' + required: true + schema: + type: string + responses: + '200': + description: successful operation + headers: + Cache-Control: + schema: + type: string + default: 'max-age=900' # 15 min cache + '404': + description: video channel not found + '/api/v1/accounts/{name}': get: tags: diff --git a/yarn.lock b/yarn.lock index e2258ee4a..d9fbc5e0f 100644 --- a/yarn.lock +++ b/yarn.lock @@ -1836,10 +1836,10 @@ resolved "https://registry.yarnpkg.com/@opentelemetry/semantic-conventions/-/semantic-conventions-1.9.1.tgz#ad3367684a57879392513479e0a436cb2ac46dad" integrity sha512-oPQdbFDmZvjXk5ZDoBGXG8B4tSB/qW5vQunJWQMFUBp7Xe8O1ByPANueJ+Jzg58esEBegyyxZ7LRmfJr7kFcFg== -"@peertube/feed@^5.0.1": - version "5.0.2" - resolved "https://registry.yarnpkg.com/@peertube/feed/-/feed-5.0.2.tgz#d9ae7f38f1ccc75d353a5e24ad335a982bc4df74" - integrity sha512-5c8NkeIDx6J8lOzYiaTGipich/7hTO+CzZjIHFb1SY3+c14BvNJxrFb8b/9aZ8tekIYxKspqb8hg7WcVYg4NXA== +"@peertube/feed@^5.1.0": + version "5.1.0" + resolved "https://registry.yarnpkg.com/@peertube/feed/-/feed-5.1.0.tgz#e2fec950459ebaa32ea35791c45177f8b6fa85e9" + integrity sha512-ggwIbjxh4oc1aAGYV7ZxtIpiEIGq3Rkg6FxvOSrk/EPZ76rExoIJCjKeSyd4zb/sGkyKldy+bGs1OUUVidWWTQ== dependencies: xml-js "^1.6.11" @@ -6362,7 +6362,7 @@ lodash.merge@4.6.2, lodash.merge@^4.6.2: resolved "https://registry.yarnpkg.com/lodash.merge/-/lodash.merge-4.6.2.tgz#558aa53b43b661e1925a0afdfa36a9a1085fe57a" integrity sha512-0KpjqXRVvrYyCsX1swR/XTK0va6VQkQM6MNo7PqW77ByjAhoARA8EfrP1N4+KlKj8YS0ZUCtRT/YUuhyYDujIQ== -lodash@4.17.21, lodash@>=4.17.13, lodash@^4.17.10, lodash@^4.17.14, lodash@^4.17.20, lodash@^4.17.21: +lodash@4.17.21, lodash@>=4.17.13, lodash@^4.17.14, lodash@^4.17.20, lodash@^4.17.21: version "4.17.21" resolved "https://registry.yarnpkg.com/lodash/-/lodash-4.17.21.tgz#679591c564c3bffaae8454cf0b3df370c3d6911c" integrity sha512-v2kDEe57lecTulaDIuNTPy3Ry4gLGJ6Z1O3vE1krgXZNrsQ+LFTGHVxVjcXPs17LhbZVGedAJv8XZ1tvj5FvSg==