Add Podcast RSS feeds (#5487)

* Initial test implementation of Podcast RSS

This is a pretty simple implementation to add support for The Podcast Namespace in RSS -- instead of affecting the existing RSS implementation, this adds a new UI option.

I attempted to retain compatibility with the rest of the RSS feed implementation as much as possible and have created a temporary fork of the "pfeed" library to support this effort.

* Update to pfeed-podcast 1.2.2

* Initial test implementation of Podcast RSS

This is a pretty simple implementation to add support for The Podcast Namespace in RSS -- instead of affecting the existing RSS implementation, this adds a new UI option.

I attempted to retain compatibility with the rest of the RSS feed implementation as much as possible and have created a temporary fork of the "pfeed" library to support this effort.

* Update to pfeed-podcast 1.2.2

* Initial test implementation of Podcast RSS

This is a pretty simple implementation to add support for The Podcast Namespace in RSS -- instead of affecting the existing RSS implementation, this adds a new UI option.

I attempted to retain compatibility with the rest of the RSS feed implementation as much as possible and have created a temporary fork of the "pfeed" library to support this effort.

* Update to pfeed-podcast 1.2.2

* Add correct feed image to RSS channel

* Prefer HLS videos for podcast RSS

Remove video/stream titles, add optional height attribute to podcast RSS

* Prefix podcast RSS images with root server URL

* Add optional video query support to include captions

* Add transcripts & person images to podcast RSS feed

* Prefer webseed/webtorrent files over HLS fragmented mp4s

* Experimentally adding podcast fields to basic config page

* Add validation for new basic config fields

* Don't include "content" in podcast feed, use full description for "description"

* Initial test implementation of Podcast RSS

This is a pretty simple implementation to add support for The Podcast Namespace in RSS -- instead of affecting the existing RSS implementation, this adds a new UI option.

I attempted to retain compatibility with the rest of the RSS feed implementation as much as possible and have created a temporary fork of the "pfeed" library to support this effort.

* Update to pfeed-podcast 1.2.2

* Add correct feed image to RSS channel

* Prefer HLS videos for podcast RSS

Remove video/stream titles, add optional height attribute to podcast RSS

* Prefix podcast RSS images with root server URL

* Add optional video query support to include captions

* Add transcripts & person images to podcast RSS feed

* Prefer webseed/webtorrent files over HLS fragmented mp4s

* Experimentally adding podcast fields to basic config page

* Add validation for new basic config fields

* Don't include "content" in podcast feed, use full description for "description"

* Add medium/socialInteract to podcast RSS feeds. Use HTML for description

* Change base production image to bullseye, install prosody in image

* Add liveItem and trackers to Podcast RSS feeds

Remove height from alternateEnclosure, replaced with title.

* Clear Podcast RSS feed cache when live streams start/end

* Upgrade to Node 16

* Refactor clearCacheRoute to use ApiCache

* Remove unnecessary type hint

* Update dockerfile to node 16, install python-is-python2

* Use new file paths for captions/playlists

* Fix legacy videos in RSS after migration to object storage

* Improve method of identifying non-fragmented mp4s in podcast RSS feeds

* Don't include fragmented MP4s in podcast RSS feeds

* Add experimental support for podcast:categories on the podcast RSS item

* Fix undefined category when no videos exist

Allows for empty feeds to exist (important for feeds that might only go live)

* Add support for podcast:locked -- user has to opt in to show their email

* Use comma for podcast:categories delimiter

* Make cache clearing async

* Fix merge, temporarily test with pfeed-podcast

* Syntax changes

* Add EXT_MIMETYPE constants for captions

* Update & fix tests, fix enclosure mimetypes, remove admin email

* Add test for podacst:socialInteract

* Add filters hooks for podcast customTags

* Remove showdown, updated to pfeed-podcast 6.1.2

* Add 'action:api.live-video.state.updated' hook

* Avoid assigning undefined category to podcast feeds

* Remove nvmrc

* Remove comment

* Remove unused podcast config

* Remove more unused podcast config

* Fix MChannelAccountDefault type hint missed in merge

* Remove extra line

* Re-add newline in config

* Fix lint errors for isEmailPublic

* Fix thumbnails in podcast feeds

* Requested changes based on review

* Provide podcast rss 2.0 only on video channels

* Misc cleanup for a less messy PR

* Lint fixes

* Remove pfeed-podcast

* Add peertube version to new hooks

* Don't use query include, remove TODO

* Remove film medium hack

* Clear podcast rss cache before video/channel update hooks

* Clear podcast rss cache before video uploaded/deleted hooks

* Refactor podcast feed cache clearing

* Set correct person name from video channel

* Styling

* Fix tests

---------

Co-authored-by: Chocobozzz <me@florianbigard.com>

1 files changed, 189 insertions, 0 deletions

diff --git a/server/controllers/feeds/video-feeds.ts b/server/controllers/feeds/video-feeds.ts
new file mode 100644
index 000000000..b6e0663eb
--- /dev/null
+++ b/server/controllers/feeds/video-feeds.ts
@@ -0,0 +1,189 @@
+import express from 'express'
+import { extname } from 'path'
+import { Feed } from '@peertube/feed'
+import { cacheRouteFactory } from '@server/middlewares'
+import { VideoModel } from '@server/models/video/video'
+import { VideoInclude } from '@shared/models'
+import { buildNSFWFilter } from '../../helpers/express-utils'
+import { MIMETYPES, ROUTE_CACHE_LIFETIME, WEBSERVER } from '../../initializers/constants'
+import {
+  asyncMiddleware,
+  commonVideosFiltersValidator,
+  feedsFormatValidator,
+  setDefaultVideosSort,
+  setFeedFormatContentType,
+  videoFeedsValidator,
+  videosSortValidator,
+  videoSubscriptionFeedsValidator
+} from '../../middlewares'
+import { buildFeedMetadata, getCommonVideoFeedAttributes, getVideosForFeeds, initFeed, sendFeed } from './shared'
+
+const videoFeedsRouter = express.Router()
+
+const { middleware: cacheRouteMiddleware } = cacheRouteFactory({
+  headerBlacklist: [ 'Content-Type' ]
+})
+
+// ---------------------------------------------------------------------------
+
+videoFeedsRouter.get('/feeds/videos.:format',
+  videosSortValidator,
+  setDefaultVideosSort,
+  feedsFormatValidator,
+  setFeedFormatContentType,
+  cacheRouteMiddleware(ROUTE_CACHE_LIFETIME.FEEDS),
+  commonVideosFiltersValidator,
+  asyncMiddleware(videoFeedsValidator),
+  asyncMiddleware(generateVideoFeed)
+)
+
+videoFeedsRouter.get('/feeds/subscriptions.:format',
+  videosSortValidator,
+  setDefaultVideosSort,
+  feedsFormatValidator,
+  setFeedFormatContentType,
+  cacheRouteMiddleware(ROUTE_CACHE_LIFETIME.FEEDS),
+  commonVideosFiltersValidator,
+  asyncMiddleware(videoSubscriptionFeedsValidator),
+  asyncMiddleware(generateVideoFeedForSubscriptions)
+)
+
+// ---------------------------------------------------------------------------
+
+export {
+  videoFeedsRouter
+}
+
+// ---------------------------------------------------------------------------
+
+async function generateVideoFeed (req: express.Request, res: express.Response) {
+  const account = res.locals.account
+  const videoChannel = res.locals.videoChannel
+
+  const { name, description, imageUrl, accountImageUrl, link, accountLink } = await buildFeedMetadata({ videoChannel, account })
+
+  const feed = initFeed({
+    name,
+    description,
+    link,
+    isPodcast: false,
+    imageUrl,
+    author: { name, link: accountLink, imageUrl: accountImageUrl },
+    resourceType: 'videos',
+    queryString: new URL(WEBSERVER.URL + req.url).search
+  })
+
+  const data = await getVideosForFeeds({
+    sort: req.query.sort,
+    nsfw: buildNSFWFilter(res, req.query.nsfw),
+    isLocal: req.query.isLocal,
+    include: req.query.include | VideoInclude.FILES,
+    accountId: account?.id,
+    videoChannelId: videoChannel?.id
+  })
+
+  addVideosToFeed(feed, data)
+
+  // Now the feed generation is done, let's send it!
+  return sendFeed(feed, req, res)
+}
+
+async function generateVideoFeedForSubscriptions (req: express.Request, res: express.Response) {
+  const account = res.locals.account
+  const { name, description, imageUrl, link } = await buildFeedMetadata({ account })
+
+  const feed = initFeed({
+    name,
+    description,
+    link,
+    isPodcast: false,
+    imageUrl,
+    resourceType: 'videos',
+    queryString: new URL(WEBSERVER.URL + req.url).search
+  })
+
+  const data = await getVideosForFeeds({
+    sort: req.query.sort,
+    nsfw: buildNSFWFilter(res, req.query.nsfw),
+    isLocal: req.query.isLocal,
+    include: req.query.include | VideoInclude.FILES,
+    displayOnlyForFollower: {
+      actorId: res.locals.user.Account.Actor.id,
+      orLocalVideos: false
+    },
+    user: res.locals.user
+  })
+
+  addVideosToFeed(feed, data)
+
+  // Now the feed generation is done, let's send it!
+  return sendFeed(feed, req, res)
+}
+
+// ---------------------------------------------------------------------------
+
+function addVideosToFeed (feed: Feed, videos: VideoModel[]) {
+  /**
+   * Adding video items to the feed object, one at a time
+   */
+  for (const video of videos) {
+    const formattedVideoFiles = video.getFormattedAllVideoFilesJSON(false)
+
+    const torrents = formattedVideoFiles.map(videoFile => ({
+      title: video.name,
+      url: videoFile.torrentUrl,
+      size_in_bytes: videoFile.size
+    }))
+
+    const videoFiles = formattedVideoFiles.map(videoFile => {
+      return {
+        type: MIMETYPES.VIDEO.EXT_MIMETYPE[extname(videoFile.fileUrl)],
+        medium: 'video',
+        height: videoFile.resolution.id,
+        fileSize: videoFile.size,
+        url: videoFile.fileUrl,
+        framerate: videoFile.fps,
+        duration: video.duration,
+        lang: video.language
+      }
+    })
+
+    feed.addItem({
+      ...getCommonVideoFeedAttributes(video),
+
+      id: WEBSERVER.URL + video.getWatchStaticPath(),
+      author: [
+        {
+          name: video.VideoChannel.getDisplayName(),
+          link: video.VideoChannel.getClientUrl()
+        }
+      ],
+      torrents,
+
+      // Enclosure
+      video: videoFiles.length !== 0
+        ? {
+          url: videoFiles[0].url,
+          length: videoFiles[0].fileSize,
+          type: videoFiles[0].type
+        }
+        : undefined,
+
+      // Media RSS
+      videos: videoFiles,
+
+      embed: {
+        url: WEBSERVER.URL + video.getEmbedStaticPath(),
+        allowFullscreen: true
+      },
+      player: {
+        url: WEBSERVER.URL + video.getWatchStaticPath()
+      },
+      community: {
+        statistics: {
+          views: video.views
+        }
+      }
+    })
+  }
+}