2
0
mirror of https://github.com/TeamNewPipe/NewPipeExtractor synced 2025-08-22 18:07:47 +00:00

Compare commits

...

2009 Commits
v0.19.4 ... dev

Author SHA1 Message Date
wb9688
6f51a23fa5
Merge pull request #1358 from TeamNewPipe/fix-gradle-jdoc
Fix aggregatedJavadocs task and update API link to JDK 11
2025-08-01 11:29:51 +02:00
TobiGr
b5c7d57d63 Fix aggregatedJavadocs task and update API link to JDK 11 2025-08-01 10:14:03 +02:00
Tobi
f668a0bc6a
Merge pull request #1356 from TeamNewPipe/fix-jdoc
Fix JDoc and automated doc build
2025-08-01 00:39:38 -07:00
Tobi
8476a463e8
Fix JDoc and automated doc build
See https://github.com/TeamNewPipe/NewPipeExtractor/actions/runs/16660575612/job/47156509194#step:5:43

NewPipeExtractor/NewPipeExtractor/extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamInfoItemLockupExtractor.java:29: error: self-closing element not allowed
 * This extractor is currently (2025-07) only used to extract related video streams.<br/>
2025-08-01 07:59:08 +02:00
Stypox
88ff8972e5
Release v0.24.8
v0.24.7 accidentally pointed at v0.24.6 and JitPack had already built so we had to make a new release
2025-07-31 23:38:02 +02:00
Stypox
d376c94bd5
Release v0.24.7 2025-07-31 23:28:24 +02:00
Stypox
0a7b72aec6
Merge pull request #1354 from AudricV/yt_more_kiosks_and_trending_deprecation 2025-07-31 23:24:13 +02:00
Stypox
426a227047
Merge pull request #1347 from litetex/fix-yt-channel-id-resolution 2025-07-31 22:28:07 +02:00
AudricV
2cccf48d6c
Add an UnsupportedContentInCountryException and use it in YouTube Charts
This allows client to distinguish of a generic
ContentNotSupportedException, which hasn't fully its place for its usage
in YouTube Charts.
2025-07-31 21:10:38 +02:00
AudricV
25e30b745a
[YouTube] Set running lives as new default kiosk
Also move YoutubeKioskExtractorTest.Trending at the bottom of the file
and add deprecation notice on test class too.
2025-07-31 21:10:38 +02:00
AudricV
5c7abeeab0
[YouTube] Add mocks for new kiosks' tests 2025-07-31 21:10:37 +02:00
AudricV
0687977690
[YouTube] Add tests for new kiosks 2025-07-31 21:00:41 +02:00
AudricV
9f7690823d
[YouTube] Move Trending kiosk to kiosk package, add deprecation comment
Trending is still working at the time this commit is made, it has been
just removed from the interface.

Also remove getInstance method of YoutubeTrendingLinkHandlerFactory to
make the INSTANCE static field public, for consistency with other
kiosks of the service.
2025-07-31 21:00:41 +02:00
AudricV
21c3aad320
[YouTube] Add trending music extractor
This kiosk is meant to return official music videos, but it also
returns unofficial content and autogenerated tracks, hence the
kiosk name.

Making requests with an unsupported YouTube Charts country leads to a
400 HTTP error, so for these countries a ContentNotSupportedException
is thrown by the extractor.
2025-07-31 21:00:41 +02:00
AudricV
f4203e632d
[YouTube] Add trending movies and shows trailers extractor
This kiosk also returns some videos for unsupported YouTube Charts
countries, even if there are fewer than in a supported country.
2025-07-31 21:00:41 +02:00
AudricV
e643024ff0
[YouTube] Add base class to parse trending videos' charts responses 2025-07-31 21:00:40 +02:00
AudricV
a4aeedff90
[YouTube] Add WEB_MUSIC_ANALYTICS constants, make some methods public
This is the client for YouTube Charts (charts.youtube.com).

Also change nullability of two fields and fix wrong client constant
usage in ofWebEmbeddedPlayerClient method in InnertubeClientRequestInfo.

Usages in YoutubeParsingHelper have been updated, getClientHeaders and
prepareJsonBuilder methods in this class have been made public.
2025-07-31 21:00:40 +02:00
TobiGr
6397b2ec3f Fix missing paranthesis in condition and improve null-safety 2025-07-31 11:19:33 +02:00
litetex
aa70b5a04b Create `YouTubeChannelHelperTest` 2025-07-31 11:19:17 +02:00
litetex
b05347d68d Redirection doesn't always seem to happen
Ref: https://github.com/TeamNewPipe/NewPipeExtractor/pull/1347#issuecomment-3123364807
Co-Authored-By: Audric V. <74829229+AudricV@users.noreply.github.com>
2025-07-31 11:10:33 +02:00
litetex
f91454d24f [YT] Correctly resolve redirects when resolving channel ids 2025-07-31 10:57:11 +02:00
litetex
4701c56e86 [YT] Build correct handle 2025-07-31 10:57:11 +02:00
Tobi
a5fcc7d7aa
Merge pull request #1351 from Stypox/fix-ytm-albums-playlists
[YouTube Music] Fix uploader and stream count for album/playlist info items
2025-07-29 06:09:28 -07:00
AudricV
f4b0a7d97e
[YouTube] Add trending gaming videos extractor from Gaming system channel 2025-07-28 21:45:54 +02:00
AudricV
a870e3767d
[YouTube] Add trending podcasts episodes extractor from Podcasts page 2025-07-28 21:45:54 +02:00
AudricV
d38d64708d
[YouTube] Add running lives extractor from Live system channel 2025-07-28 21:45:54 +02:00
AudricV
3d0e302230
[YouTube] Add base class to parse kiosks from WEB InnerTube client
Kiosks structure work in a very similar way to channel tabs, so
YoutubeChannelHelper is used in this abstract class.
2025-07-28 21:45:53 +02:00
Stypox
bd6e5d3fdb
Code style 2025-07-28 20:44:40 +02:00
Stypox
03a65516b0
[YouTube Music] Fix uploader and stream count for album/playlist info items
- handles autogenerated albums instead of returning an error for invalid uploader url
- extracts uploader url of playlists too using another method
- getStreamCount() now returns ITEM_COUNT_UNKNOWN because no info is included in the JSON
2025-07-28 17:43:53 +02:00
Stypox
260ba4749a
Merge pull request #1350 from Stypox/fix-search-filtered-lockup 2025-07-28 16:09:58 +02:00
Stypox
80e42aa584
[YouTube] Fix search filters messing with which lockup formats are extracted 2025-07-28 15:56:10 +02:00
Stypox
c5337943d6
Merge pull request #1348 from TobiGr/user-agent-140 2025-07-28 14:00:31 +02:00
TobiGr
0378fdee5c Fix checkstyle 2025-07-27 12:34:30 +02:00
TobiGr
e9dbcba308 Update USER_AGENT to Firefox ESR 140 2025-07-27 09:41:09 +02:00
litetex
af82c7ae0a
Merge pull request #1344 from FineFindus/fix/invalid-related-id
[YouTube] Correctly set `uploaderUrl` for lockup content type items
2025-07-26 15:28:33 +02:00
FineFindus
352fae640f
[YouTube] Correctly set uploaderUrl for lockup content type items
Fixes an issue, where the uploader URL for related items would be incorrect,
due to the `YoutubeChannelLinkHandlerFactory.getUrl` expecting the id with a
`channel/` prefix. However, the `browseId` used to extract the channel id is
missing this prefix.

Ref: https://github.com/TeamNewPipe/NewPipeExtractor/pull/1320

Fix getUploaderUrl not resolving correct url
2025-07-26 15:24:11 +02:00
Tobi
d9c777df78
Merge pull request #1342 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.13.4
Bump org.junit:junit-bom from 5.13.3 to 5.13.4
2025-07-21 03:31:47 -07:00
dependabot[bot]
2c0db03601
Bump org.junit:junit-bom from 5.13.3 to 5.13.4
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit-framework) from 5.13.3 to 5.13.4.
- [Release notes](https://github.com/junit-team/junit-framework/releases)
- [Commits](https://github.com/junit-team/junit-framework/compare/r5.13.3...r5.13.4)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-version: 5.13.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-07-21 10:00:05 +00:00
litetex
7adbc48a0a Merge pull request #1332 from litetex/fix-all-tests
Fix all tests and make everything work offline/with mocks.
Remove old mocks and generate new ones with new structure.

Remove SoundCloud "Top 50" kiosk.
2025-07-18 12:27:50 +02:00
Stypox
f3df599e8a
Merge pull request #1340 from TeamNewPipe/dependabot/gradle/protobufVersion-4.31.1 2025-07-17 17:43:22 +02:00
dependabot[bot]
3e6f1fa018
Bump protobufVersion from 4.30.2 to 4.31.1
Bumps `protobufVersion` from 4.30.2 to 4.31.1.

Updates `com.google.protobuf:protobuf-javalite` from 4.30.2 to 4.31.1

Updates `com.google.protobuf:protoc` from 4.30.2 to 4.31.1
- [Release notes](https://github.com/protocolbuffers/protobuf/releases)
- [Changelog](https://github.com/protocolbuffers/protobuf/blob/main/protobuf_release.bzl)
- [Commits](https://github.com/protocolbuffers/protobuf/compare/v4.30.2...v4.31.1)

---
updated-dependencies:
- dependency-name: com.google.protobuf:protobuf-javalite
  dependency-version: 4.31.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: com.google.protobuf:protoc
  dependency-version: 4.31.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-07-17 09:51:21 +00:00
Tobi
68b4c9acba
Merge pull request #1338 from Stypox/lockup-podcast
[YouTube] Support LOCKUP_CONTENT_TYPE_PODCAST
2025-07-16 09:14:33 -07:00
Stypox
35cc5a77b5
[YouTube] Support LOCKUP_CONTENT_TYPE_PODCAST
Also extend support for LOCKUP_CONTENT_TYPE_VIDEO
2025-07-16 14:48:06 +02:00
Stypox
a9a4181c0d
Merge pull request #1104 from AudricV/yt_continuations-playlists-shorts-ui 2025-07-16 13:44:28 +02:00
Stypox
a3e2d4887c
Fix *.proto ending up in JAR 2025-07-16 13:43:13 +02:00
Stypox
be65c5e94b
Let IDE fix warnings in build.gradle 2025-07-16 13:33:21 +02:00
AudricV
3ba3dbc4d1
[YouTube] Update channel mocks using playlists 2025-07-16 13:26:39 +02:00
AudricV
dbca2b4e1f
[YouTube] Update playlists mocks 2025-07-16 13:26:39 +02:00
AudricV
88efec6e4d
[YouTube] Enable YoutubePlaylistExtractorTest.ShortsUI.testMoreRelatedItems
As continuations are now again available for playlists with Shorts UI, more
items' tests can be enabled.
2025-07-16 13:26:39 +02:00
AudricV
9415e1bbef
[YouTube] Workaround playlists' Shorts UI
YouTube doesn't return currently a continuation, if applicable, for
Shorts UI playlists, restricting access to the 100th first items. The
reel items returned don't give also upload date, uploader info and
precise view count.

Using a continuation which requests the first page of the playlist
allows currently to get access to continuations, if applicable, and
also standard video elements instead of Shorts ones, making extraction
of upload date, uploader info and precise view count again possible.

This method is used for all playlist types, the original request is
still made, but now only returns what we need.

It requires to add a protocol buffer definition file, for which its
structure is based on reverse engineering of playlists continuations
sent by WEB InnerTube client, received from InnerTube responses.

Java classes of this file are generated for the Java Lite runtime of
Protobuf with the Protobuf Gradle plugin, as the lite version is enough
for our use cases. This plugin ships in JARs Protobuf definitions,
which should be avoided.

As Protobuf classes are parsed by Checkstyle checks and do not follow
our style rules at all, an exclusion rule has been for them.
2025-07-16 13:26:38 +02:00
AudricV
e12094f483
[YouTube] Add utility method to make InnerTube requests with custom
query parameters

This can be used to pass Google APIs standard parameters to requests.
2025-07-16 13:26:38 +02:00
Stypox
a94a6e0dcb
Merge pull request #1320 from litetex/add-support-for-LOCKUP_CONTENT_TYPE_VIDEO 2025-07-16 13:22:54 +02:00
Stypox
4d1ea24c5c
Improve comments 2025-07-16 13:22:13 +02:00
litetex
b6b395061c
Small live stream improvements and cleanup
Thanks to Stypox review

* Improved live stream related code
* Fixed Javadoc

Co-Authored-By: Stypox <stypox@pm.me>
2025-07-15 22:26:01 +02:00
litetex
64e4bafee1
Remove not needed comment 2025-07-15 22:06:02 +02:00
litetex
8e07a76c71
Do not parse duration when live stream
Also cache the streamtype
2025-07-14 20:39:12 +02:00
litetex
b15c0dccc5
Cleanup
Co-Authored-By: Audric V. <74829229+AudricV@users.noreply.github.com>
2025-07-14 20:31:18 +02:00
litetex
4d7a494757
Update to latest nanojson version of TeamNewPipe 2025-07-13 20:59:21 +02:00
litetex
182df6eebe
Handle situations where there are multiple uploaders 2025-07-13 20:59:21 +02:00
litetex
ef6a53ebb3
Prevent NPE on live streams and cache correctly 2025-07-13 20:59:21 +02:00
litetex
ba2efde599
Updated test mocks to also check for `YoutubeStreamInfoItemLockupExtractor` 2025-07-13 20:59:20 +02:00
litetex
d9ba25ce85
Update docs, add caching and removed unused code 2025-07-13 20:59:20 +02:00
litetex
708587a992
Cleanup 2025-07-13 20:59:20 +02:00
litetex
f15190f8f9
Change back to private 2025-07-13 20:59:20 +02:00
litetex
e08747f339
Add support for detecting livestreams 2025-07-13 20:59:20 +02:00
litetex
af4b1b95d8
Added fallback for videoId 2025-07-13 20:59:20 +02:00
litetex
9fb60c7ef5
Use already existing method 2025-07-13 20:59:20 +02:00
litetex
939d5e1f74
There is no formatting 2025-07-13 20:59:20 +02:00
litetex
a18e98adbb
Remove copyright from original file 2025-07-13 20:59:20 +02:00
litetex
b07b3dae7c
Basic implementation of `YoutubeStreamInfoItemLockupExtractor` 2025-07-13 20:59:20 +02:00
litetex
ed37a429d1
Use my nanojson version which has some shortcut methods 2025-07-13 20:59:20 +02:00
litetex
64938b024b
Merge pull request #1334 from TeamNewPipe/gh-funding
Create FUNDING.yml to display Liberapay as sponsoring option
2025-07-13 20:57:47 +02:00
Tobi
d79533db3e
Create FUNDING.yml to display Liberapay as sponsoring option 2025-07-13 14:20:04 +02:00
Tobi
71b335128e
Merge pull request #1308 from watermelon42/3783_Import_Soundcloud_likes
[SoundCloud] Add support for likes channel tab
2025-07-13 03:16:34 -07:00
Tobi
c0ff21af93
Merge pull request #1330 from litetex/speed-up-n-parameter-extraction
[YouTube] Speed up n parameter presence check
2025-07-12 05:10:49 -07:00
litetex
ebb53fb21c
Merge pull request #1290 from FineFindus/fix/deobfuscation-function
[YouTube] Fix signature deobfuscation function name extraction
2025-07-11 20:19:39 +02:00
litetex
249f8f83e7
Updated mocks 2025-07-11 20:19:12 +02:00
litetex
0dfaeb20fa
Added doc for related code 2025-07-11 20:12:50 +02:00
FineFindus
1915dc6904
[YouTube] Allow both types of string quotes 2025-07-11 08:27:44 +02:00
FineFindus
a847fdb330
[YouTube] Allow lowercase global array names 2025-07-11 08:27:44 +02:00
FineFindus
d914e86a82
[YouTube] extract global variable for signature deobfuscation
Extracts the global variable used in newer signature deobfuscation functions,
such as the one found in the `59b252b9` player.
2025-07-11 08:27:44 +02:00
FineFindus
235dd7cc55
[YouTube] add new signature regex
Adds a new regex to find the signature deobfuscation function, which
works for the new `20830619` player.

Closes: https://github.com/TeamNewPipe/NewPipeExtractor/issues/1287
Ref: https://github.com/TeamNewPipe/NewPipeExtractor/issues/1287#issuecomment-2764573093
2025-07-11 08:27:44 +02:00
litetex
1b8864afb8
Merge pull request #1302 from FineFindus/fix/throttling-function-regex
[YouTube] add new deobfuscation function name regex
2025-07-10 23:48:08 +02:00
litetex
12f7615e4a
Updated mocks for new deobfuscation function name regex 2025-07-10 23:46:28 +02:00
litetex
5184231479
Do a quick check if the n parameter is even present before exec regex 2025-07-10 23:37:54 +02:00
Stypox
ef1db0d716
Merge pull request #1326 from AudricV/yt_fix-view-count-members-first-only-shorts 2025-07-09 19:50:15 +02:00
AudricV
26bcfcdeaf
[YouTube] Detect members first and members only shorts
This fixes view count extraction error for such content, as the view
count is replaced by a text describing the availability of the short.

Also made YoutubeShortsLockupInfoItemExtractor package private, as it
doesn't need to be public.
2025-07-09 18:05:06 +02:00
watermelon42
5e1a1d51b6 Handle liked playlists using the SoundcloudPlaylistInfoItemExtractor 2025-07-08 18:44:51 +03:00
Tobi
22f7cd4a6f
Merge pull request #1309 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.21.1
Bump org.jsoup:jsoup from 1.19.1 to 1.21.1
2025-07-08 05:08:08 -07:00
Tobi
7cf4379ac5
Merge pull request #1293 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.13.1
Bump com.google.code.gson:gson from 2.12.1 to 2.13.1
2025-07-07 14:04:52 -07:00
dependabot[bot]
f06b01f036
Bump org.jsoup:jsoup from 1.19.1 to 1.21.1
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup) from 1.19.1 to 1.21.1.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES.md)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.19.1...jsoup-1.21.1)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-version: 1.21.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-07-07 21:00:47 +00:00
dependabot[bot]
8f28ab3fea
Bump com.google.code.gson:gson from 2.12.1 to 2.13.1
Bumps [com.google.code.gson:gson](https://github.com/google/gson) from 2.12.1 to 2.13.1.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/main/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.12.1...gson-parent-2.13.1)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-version: 2.13.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-07-07 21:00:04 +00:00
Tobi
83b12ba93a
Merge pull request #1317 from AudricV/soundcloud-drm-free-aac-160k
[Soundcloud] Add support for AAC 160k unprotected stream and update Creative Commons stream test
2025-07-07 13:35:22 -07:00
Tobi
9e0fdf969e
Merge pull request #1318 from AudricV/yt_fix-subs-count-channels-with-pronouns
[YouTube] Fix parsing subscribers count of channels with pronouns
2025-07-07 12:32:43 -07:00
Tobi
c4dcef4d51
Merge pull request #1319 from AudricV/peertube_support-video-only-streams-properly
[PeerTube] Detect video-only streams
2025-07-07 12:26:27 -07:00
Tobi
c043d0856d
Merge pull request #1316 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.13.3
Bump org.junit:junit-bom from 5.12.1 to 5.13.3
2025-07-07 12:23:46 -07:00
AudricV
3042e40059
[PeerTube] Detect video-only streams 2025-07-07 21:20:49 +02:00
Tobi
48d78205be
Merge pull request #1313 from asifebrahim/temp
Add Nullable annotations in ListExtractor
2025-07-07 11:56:00 -07:00
AudricV
8bab96afd6
[YouTube] Add a test of a channel with pronouns
This is to test subscribers count parsing in a page header when a
channel has pronouns.
2025-07-07 19:02:10 +02:00
AudricV
53757ae97d
[YouTube] Support pronouns in subs count parsing of page channel headers 2025-07-07 18:56:01 +02:00
AudricV
7dc7b06bab
[Soundcloud] Update Creative Commons stream test
The track which was used has been deleted, so replace it with another
track from the account.

The AAC 160k unprotected stream isn't present on this track, only recent
ones seem to have it.
2025-07-07 17:15:35 +02:00
AudricV
9d88d84bff
[Soundcloud] Add support for AAC 160k unprotected stream
This stream isn't returned in DRM-protected tracks, of course.
2025-07-07 17:14:24 +02:00
dependabot[bot]
7934f8a596
Bump org.junit:junit-bom from 5.12.1 to 5.13.3
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit-framework) from 5.12.1 to 5.13.3.
- [Release notes](https://github.com/junit-team/junit-framework/releases)
- [Commits](https://github.com/junit-team/junit-framework/compare/r5.12.1...r5.13.3)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-version: 5.13.3
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-07-07 12:10:15 +00:00
asifebrahim
cbbc6d2504 Added the @Nullable annotaions in ListExtractor.java 2025-07-06 18:06:22 +05:30
watermelon42
964512741d Reverted version change 2025-06-18 19:39:11 +03:00
watermelon42
44882b393c Added support for importing Soundcloud likes. 2025-06-18 16:44:58 +03:00
Stypox
8b9ccec936
Merge pull request #1305 from iscle/patch-1 2025-06-06 15:21:01 +02:00
Iscle
2dbf404507
Fix javadoc typo 2025-06-06 00:44:55 +02:00
Stypox
b90fd36783
Merge pull request #1301 from AudricV/yt_fix-init-playlists-continuations
[YouTube] Fix extraction of continuations in initial playlist responses
2025-05-27 13:20:30 +02:00
FineFindus
b97c6555c0
[YouTube] add new deobfuscation function name regex
Adds a new regex to find the name of the deobfuscation function for throttling,
which works for the `59b252b9` player.

Closes: https://github.com/TeamNewPipe/NewPipeExtractor/issues/1289
2025-05-26 22:33:36 +02:00
AudricV
fec89a2cf1
[YouTube] Fix extraction of continuations in initial playlist responses 2025-05-24 11:05:24 +02:00
Stypox
61059b1afa
Release v0.24.6 2025-05-07 12:31:22 +02:00
Stypox
37a08479cb
Merge pull request #1297 from AudricV/yt_fix-streams-sabr-npe
[YouTube] Fix crash on SABR-only player responses, do not use WEB client for stream URLs anymore
2025-05-07 12:15:08 +02:00
AudricV
f35874e76f
[YouTube] Fix crash on SABR-only player responses, do not use WEB client 2025-05-06 20:02:36 +02:00
Stypox
67f3301d9f
Merge pull request #1288 from thigg/dev
Remove usage of reflection in timeago-parser module
2025-04-08 23:34:24 +02:00
Stypox
0d82596f7a
getPattern final parameter 2025-04-08 23:25:54 +02:00
Stypox
1ec7d419a2
Use org.schabi.newpipe.timeago_generator as package name 2025-04-08 23:23:48 +02:00
thigg
e515252bb9
removed obsolete proguard rules 2025-03-29 19:10:01 +01:00
Thilo Kogge
18572da79d removed use of reflection in PatternsManager.java. For this the Code to generate the Localization files was extracted into its own submodule.
The generator now also creates a class that Holds a Map of all Localizations for the timeago-parser.
2025-03-27 19:39:33 +01:00
Tobi
0b99100dbd
Merge pull request #1282 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.19.1
Bump org.jsoup:jsoup from 1.18.3 to 1.19.1
2025-03-16 21:40:31 +01:00
dependabot[bot]
4f94950feb
Bump org.jsoup:jsoup from 1.18.3 to 1.19.1
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup) from 1.18.3 to 1.19.1.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES.md)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.18.3...jsoup-1.19.1)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-16 20:36:54 +00:00
Tobi
ab9231c340
Merge pull request #1281 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.12.0
Bump org.junit:junit-bom from 5.11.4 to 5.12.1
2025-03-15 17:10:31 +01:00
TobiGr
de4e27d055 Fix tests execution
See https://github.com/gradle/gradle/issues/32534#issuecomment-2678767645
2025-03-15 17:05:56 +01:00
dependabot[bot]
50b8c6ff9f
Bump org.junit:junit-bom from 5.11.4 to 5.12.0
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.11.4 to 5.12.0.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.11.4...r5.12.0)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-15 15:45:35 +00:00
litetex
d57a206344
Merge pull request #1065 from Isira-Seneviratne/Java_Files
Use Files methods in tests
2025-03-05 17:34:01 +01:00
litetex
d718f50e34
Fix compilation + Cleanup 2025-03-05 17:29:33 +01:00
Isira Seneviratne
4f16112d6d
Use Files methods in tests. 2025-03-05 17:23:45 +01:00
litetex
6bf0110e38
Merge pull request #1256 from TeamNewPipe/dependabot/gradle/org.mozilla-rhino-1.8.0
Bump org.mozilla:rhino from 1.7.15 to 1.8.0
2025-03-05 17:05:18 +01:00
litetex
da1e0e72b0
Consolidate Rhino Version 2025-03-05 16:20:50 +01:00
litetex
0c3ac0a308
Rework TokenStream 2025-03-05 16:07:05 +01:00
litetex
77ee25e3b6
Fix typo 2025-03-05 15:55:46 +01:00
litetex
907fc2ac52
Fix compile problems and optimize TokenStream 2025-03-05 15:45:08 +01:00
litetex
bcf24def8b
Cleanup test 2025-03-05 15:44:57 +01:00
litetex
470a719861
Fix compile problems in JavaScript class 2025-03-05 15:44:46 +01:00
litetex
b376539062
Also use rhino engine 2025-03-05 15:44:33 +01:00
litetex
2d3342bf56
Improve tests 2025-03-05 15:44:25 +01:00
litetex
aa67363553
Merge pull request #1245 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.18.3
Bump org.jsoup:jsoup from 1.17.2 to 1.18.3
2025-02-17 22:13:46 +01:00
dependabot[bot]
7b5d41a741
Bump org.mozilla:rhino from 1.7.15 to 1.8.0
Bumps [org.mozilla:rhino](https://github.com/mozilla/rhino) from 1.7.15 to 1.8.0.
- [Release notes](https://github.com/mozilla/rhino/releases)
- [Changelog](https://github.com/mozilla/rhino/blob/master/RELEASE-NOTES.md)
- [Commits](https://github.com/mozilla/rhino/commits)

---
updated-dependencies:
- dependency-name: org.mozilla:rhino
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-02-17 21:11:54 +00:00
litetex
c4056f5d5c
Fix tests
Upstream changes are described here: https://github.com/jhy/jsoup/issues/1278

They are effectively irrelevant as YT removes the ``"``/attributes anyway.
2025-02-17 22:09:44 +01:00
dependabot[bot]
7e21002838
Bump org.jsoup:jsoup from 1.17.2 to 1.18.3
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup) from 1.17.2 to 1.18.3.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES.md)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.17.2...jsoup-1.18.3)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-02-17 21:06:55 +00:00
litetex
5a786ebd41
Merge pull request #1266 from Stypox/update-gradle
Update gradle wrapper
2025-02-16 22:40:23 +01:00
litetex
d44b520a90
Improve github workflow
* Cache gradle wrapper distributions
* Don't always show gradle welcome message
* Deduplicated code
* Make it possible to select downloader and manually execute a run
2025-02-16 22:36:00 +01:00
Stypox
b0cc9d47ce
Update gradle-wrapper to 8.12.1 2025-02-16 21:56:00 +01:00
Stypox
b1ecb6882f
Merge pull request #1277 from litetex/fix-tests
Fix tests and cleanup other stuff
2025-02-15 15:23:27 +01:00
Stypox
e506ad3212
Merge pull request #1279 from litetex/replace-spotbugs-with-jsr305
Only include ``jsr305`` and not complete spotbugs
2025-02-15 15:06:59 +01:00
litetex
e4081243e7
Improve documentation
> > I think you can use YoutubeParsingHelper.getTextFromObject()
> That was used there before however I did not choose to use it again.
>
> ``runs`` returns an array with 3 items containing:
> 0: ``<artistname>``
> 1: ``•``
> 2: ``<subscriberCount>``
>
> There is 2 problems with that:
> 1. It's inefficient for extracting the subscription count
> 2. If for some reason the artist now has a name like "2k" the extraction malfunctions

See also: https://github.com/TeamNewPipe/NewPipeExtractor/pull/1277#discussion_r1955148742
2025-02-15 15:02:30 +01:00
litetex
677c048f1b
Only include `jsr305` and not complete spotbugs
Only ``javax.annotation.Nonull`` or ``javax.annotation.Nullable`` are currently in used which means that spotbugs can and should be removed.

See https://github.com/TeamNewPipe/NewPipeExtractor/issues/1278 for details
2025-02-15 14:23:33 +01:00
litetex
3ad9f17c53
Revert potentially breaking change
I expected this class to be located in ``youtube`` as it mostly contains YT specific playlist types, but this assumption was wrong.

See also: d9caa90d23 (r1955161108)
2025-02-13 21:24:05 +01:00
litetex
d9caa90d23
Fix okhttp deprecation 2025-02-12 22:29:06 +01:00
litetex
4c3918ac4c
Removed unused code 2025-02-12 22:09:03 +01:00
litetex
2f2e2df5aa
Add recorded mock data 2025-02-12 22:01:17 +01:00
litetex
daac243fc4
Use camelCase 2025-02-12 22:00:54 +01:00
litetex
69d5bf2a89
YouTube 100% Mock coverage 2025-02-12 21:56:02 +01:00
litetex
3b34b82e0f
Downloader: Don't force IOException
Use ``UncheckedIOException`` instead

This no longer forces one to always write ``throws IOException`` in tests
2025-02-12 21:46:45 +01:00
litetex
177262485a
Make all YT tests in MOCK mode use mock data 2025-02-11 22:19:53 +01:00
litetex
cf5df3fcff
Improve rate limiting
* Bandcamp also has rate-limiting (but a lot more req are allowed than with YT)
* Shorten default warm up time
* Fix typo
2025-02-11 21:50:38 +01:00
litetex
287123b0fd
Add rate limiter with default cold factor 2025-02-11 21:49:48 +01:00
litetex
4d7df1486c
Fix YT artist getSubscriberCount extraction 2025-02-11 21:35:19 +01:00
litetex
25e4a6f3cf
YT: Suggestion test is working again 2025-02-11 21:24:39 +01:00
litetex
ee6bc846ce
Re-Enable test as it's no longer broken 2025-02-11 21:17:46 +01:00
litetex
fa99003077
Fix duplicated ; 2025-02-11 21:16:49 +01:00
litetex
1966ad1e52
Use new Bandcamp autocomplete api 2025-02-11 21:16:30 +01:00
litetex
59a2f1aa18
No random errors 2025-02-11 21:10:31 +01:00
litetex
9f94a2912d
Remove recording retry/throttle as this is now handled in a more general way 2025-02-11 20:47:14 +01:00
litetex
460e0f77e6
Update test dependencies 2025-02-11 20:46:45 +01:00
litetex
351174a85f
Enforce modern TLS1.2+
This fixes the following problem:
Bandcamp/their Fastly CDN fails with 403 responses when running tests
* on Windows
* with Java 17+
* and OkHttp's APPROVED_CIPHER_SUITES (default) are used
2025-02-11 20:43:55 +01:00
litetex
12e2145185
Update gradle 2025-02-09 22:26:21 +01:00
litetex
ad5ca4eced
Backport/Sync code from NewPipe implementation 2025-02-09 22:06:41 +01:00
litetex
c892cfaafa
Fix PeerTube tests 2025-02-09 21:40:05 +01:00
litetex
e1b94c4148
Move into dedicated package 2025-02-09 20:57:21 +01:00
litetex
f1b90985da
Remove outdated comment 2025-02-09 20:45:17 +01:00
litetex
1a9df85213
MetaInfo Search is was removed or there is none active anymore
https://github.com/TeamNewPipe/NewPipeExtractor/issues/1274
2025-02-09 20:38:38 +01:00
litetex
61ef1699f8
Fix code style 2025-02-09 20:37:59 +01:00
litetex
d746d1b288
Disable irrelevant test 2025-02-09 20:37:50 +01:00
litetex
271a4656da
Add missing override annotations 2025-02-09 20:22:59 +01:00
litetex
51586c0957
Disable YT Shorts UI
https://github.com/TeamNewPipe/NewPipeExtractor/issues/1273
2025-02-09 20:21:52 +01:00
litetex
451bb1315c
YouTube channel mixes no longer exist
It looks like YT removed this around autumn 2024

Other mentions: https://www.reddit.com/r/youtube/comments/1dhuvyo/channel_mixes_disappeared/
2025-02-09 20:15:08 +01:00
litetex
ee7d8b042f
Channel has a new name 2025-02-09 20:08:49 +01:00
litetex
ddc7f3e33a
Fix unexpected error due to malformed url 2025-02-09 20:08:40 +01:00
litetex
6e82a80b26
Disabled age restricted test as it's currently not working 2025-02-09 20:08:29 +01:00
litetex
f0ed76c8c6
Implement rate-limiting 2025-02-09 19:57:04 +01:00
Stypox
9f83b385a4
Release v0.24.5 2025-02-05 10:05:59 +01:00
Stypox
fe168aba0b
Merge pull request #1272 from AudricV/yt_clients_changes_and_potokens_support
[YouTube] Refactor player clients, add support for poTokens, extract visitor data from the service and more
2025-02-05 10:02:51 +01:00
AudricV
96911ae2a4
[YouTube] Fix usage of WEB client headers for all HTML5 URLs in DASH creators
Also use TVHTML5 user agent for requests from this client in these
DASH manifests creators.
2025-02-05 09:56:53 +01:00
AudricV
c48d449853
[YouTube] Add ability to get TVHTML5 user agent used
Also make YoutubeParsingHelper.getOriginReferrerHeaders public, in
order to be used by other extractor classes and improve the name of a
parameter of YoutubeParsingHelper.getVisitorDataFromInnertube.
2025-02-05 09:56:52 +01:00
AudricV
0952431121
[YouTube] Move InnertubeClientRequestInfo creations in class' methods
This commits provides methods to get InnertubeClientRequestInfo
instances, which can be used by extractor clients to get visitor data
to pass to PoTokenProvider implementations using YoutubeParsingHelper.

Ability to create custom instances has been removed, but returned
objects can be modified. This is what YoutubeStreamHelper now uses to
set the visitorData property.
2025-02-05 09:56:52 +01:00
AudricV
4644e1744b
[YouTube] Add signatureTimestamp argument to TVHTML5 client requests
This argument, which has been forgot, is required to get valid
streaming URLs with this client.
2025-02-05 09:56:52 +01:00
AudricV
61f67854ed
[YouTube] Get visitorData for player requests if not provided and do some fixes
visitorData are get using InnertubeClientRequestInfo and
YoutubeParsingHelper.prepareJsonBuilder, which is replacing the
corresponding method in YoutubeStreamHelper, removed in this commit.

Also fix some bugs like JsonBuilder usages in some places and remove
HLS manifest filtering for the iOS client, as we still use for now
the full player response, in the case having streams requiring poTokens
after some time as a last resort is useful.
2025-02-05 09:56:52 +01:00
AudricV
d08331dbcf
[YouTube] Add ability to use the guide endpoint to get a visitorData
Some clients like TVHTML5 are not allowed on the visitor_id endpoint
(with this client, a 400 HTTP response is returned with a precondition
check failed error).

Also disable pretty printing for these requests, like we do for others.
2025-02-05 09:56:52 +01:00
AudricV
862a607fc6
[YouTube] Add IOS and ANDROID client IDs in ClientsConstants 2025-02-05 09:56:52 +01:00
AudricV
6533a333fa
[YouTube] Add ability to get a visitorData from InnerTube 2025-02-05 09:56:52 +01:00
AudricV
9e45c8001a
[YouTube] Do not send a visitorData for every InnerTube request
As YouTube is disabling ability to use a random visitor ID in a
visitorData on player requests and BotGuard challenges, it
shouldn't matter if we use a random one or not for other
request types.
2025-02-05 09:56:52 +01:00
AudricV
94541d2d94
[YouTube] Add utility data class to store client and device info
This new class, InnertubeClientRequestInfo, composed of two mutable
subclasses, ClientInfo and DeviceInfo, allows to store client and
device info in a better way, without requiring to pass more than 10
method parameters like in YoutubeStreamHelper currently.

Mutability has been added in order to allow changing some fields
easily, especially visitorData.
2025-02-05 09:56:52 +01:00
AudricV
9333d7fcdc
[YouTube] Update DASH manifest creation clients' handling
- Use POST requests with the same body as official clients do;
- Update methods checking the client streaming URLs come from:
  - Replace TVHTML5_SIMPLY_EMBEDDED_PLAYER by TVHTML5;
  - Add WEB_EMBEDDED_PLAYER.
2025-02-05 09:56:52 +01:00
AudricV
38e2b67cb7
[YouTube] Remove unused methods and constants in YoutubeParsingHelper 2025-02-05 09:56:52 +01:00
AudricV
3878696b2c
[YouTube] Add support for poTokens, refactor player clients' fetching
Also improve detection of age-restricted statuses in playability error
messages returned by the service and provide version 7 of DASH
manifests.
2025-02-05 09:56:52 +01:00
AudricV
9d2b840cfb
[YouTube] Add utility class to handle player requests fetching
This internal class, YoutubeStreamHelper, has to be public in order to
be used by subpackages of the service's root one.

It supports poTokens, HTML5 signatureTimestamp property, embed context
and multiple InnerTube clients. It is meant to replace the
corresponding methods in YoutubeParsingHelper, in order to reduce the
class' size, code duplicates and improve its readability.

It adds a new way to get age-restricted videos' streams, the only ones
which are playable in YouTube embeds, which is not very common.
2025-02-05 09:56:52 +01:00
AudricV
1df0267440
[YouTube] Make client and origin HTTP headers methods package-private
This allows their internal usage in an upcoming new class to be used on
other InnerTube clients than the WEB one.
2025-02-05 09:56:52 +01:00
AudricV
3691fc22c6
[YouTube] Add an interface and a class to fetch and provide poTokens 2025-02-05 09:56:52 +01:00
AudricV
fd82ec585b
[YouTube] Move to their own file and update clients' constants
Also update client version and device info and add TVHTML5 client and
WEB_EMBEDDED_PLAYER constants, these will be used in the future.

TVHTML5_SIMPLY_EMBED_CLIENT_VERSION has been kept in its place as
the corresponding client does not work anonymously anymore, so it will
be removed soon.
2025-02-05 09:56:52 +01:00
Stypox
186e32c9db
Merge pull request #1269 from AudricV/snd_no_drm_streams
[Soundcloud] Remove DRM-protected and downloadable formats extraction
2025-02-03 20:35:20 +01:00
AudricV
a336a8ca97
[Soundcloud] Update SoundcloudStreamExtractorTest with latest changes 2025-01-28 21:52:29 +01:00
AudricV
7607688cb0
[Soundcloud] Remove DRM-protected and downloadable formats extraction
DRM-protected streams have been added to some tracks, mostly from
major music companies. We do not support DRM streams in the extractor,
so they shouldn't be extracted and so waste time, energy and resources.

Extracting downloadable format requires login for a pretty long time,
so there is no point again to do requests to extract this stream to get
an unauthorized response.

Also send the track_authorization property returned in the track info,
like the website does and allow duplicate MP3 formats in progressive and
HLS delivery methods to be returned.
2025-01-28 21:52:28 +01:00
litetex
3a33cefbc0
Merge pull request #1246 from Isira-Seneviratne/BufferedReader-lines
Use BufferedReader#lines()
2025-01-27 22:35:03 +01:00
litetex
222d869efc
Polish: Use ternary instead of if 2025-01-27 22:32:01 +01:00
litetex
625e9f96d2
Remove public modifier from tests 2025-01-27 22:24:15 +01:00
Isira Seneviratne
e475b2971b
Restore URL check 2025-01-27 22:21:47 +01:00
Isira Seneviratne
0352659e2b
Use BufferedReader#lines() 2025-01-27 22:21:47 +01:00
litetex
d5ade940fc
Merge pull request #1265 from gechoto/update-ios-client
[YouTube] Update IOS client
2025-01-27 22:17:53 +01:00
Stypox
f7adeeff90
Fix checkstyle 2025-01-26 16:09:22 +01:00
Stypox
1ca8275419
Merge pull request #1253 from gechoto/fix-n-func
[YouTube] Fixes for n param deobfuscation function
2025-01-26 15:15:00 +01:00
gechoto
4b16b7ee41
YoutubeParsingHelper: update IOS client 2025-01-26 13:47:30 +00:00
Stypox
b74a8d0df8
Remove manually writing cardinals of regexes 2025-01-26 13:07:23 +01:00
Stypox
2b70b5f737
Improve new regex to account for more context 2025-01-26 13:07:23 +01:00
gechoto
ca9f1935d3
YoutubeThrottlingParameterUtils: make third regex longer to ensure it is a function which takes one parameter 2025-01-26 13:07:23 +01:00
gechoto
c7d598869c
YoutubeThrottlingParameterUtils: swap regex positions again 2025-01-26 13:07:23 +01:00
gechoto
14f7ef1093
YoutubeThrottlingParameterUtils: add back second regex and swap it with the third one 2025-01-26 13:07:23 +01:00
gechoto
d007e368f3
don't commit testing code kids 2025-01-26 13:07:23 +01:00
gechoto
4181625c2c
add comment to throttling parameter deobfuscation function fixup 2025-01-26 13:07:23 +01:00
gechoto
e0d39b606c
correct comment 2025-01-26 13:07:23 +01:00
gechoto
1393644f44
formatting again 2025-01-26 13:07:23 +01:00
gechoto
68465b27fe
fix formatting 2025-01-26 13:07:23 +01:00
gechoto
56595bd9d5
[YouTube] Fix extraction of n param deobfuscation function name and fixup function to prevent early return 2025-01-26 13:07:23 +01:00
Stypox
e0359ab872
Merge pull request #1263 from litetex/followup-1262
Partial followup for #1262
2025-01-26 12:31:40 +01:00
litetex
1c45d4e240
Fix incorrect iPhone model 2025-01-23 22:08:59 +01:00
litetex
cca3e3ffc0
Use more specific exception 2025-01-23 22:08:59 +01:00
litetex
ea401638f4
Merge pull request #1249 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.11.4
Bump org.junit:junit-bom from 5.11.3 to 5.11.4
2025-01-23 21:56:50 +01:00
litetex
485a77d885
Merge pull request #1212 from Isira-Seneviratne/TimeAgoParser-unused
Remove unused method in TimeAgoParser
2025-01-23 21:54:55 +01:00
litetex
17f0b0d7d3
Fix checkstyle 2025-01-23 21:49:27 +01:00
litetex
746b36e80d
Improve tests
* Dedup using parameterized tests
* Make ``now`` controllable
2025-01-23 21:48:29 +01:00
Isira Seneviratne
ae07ba3c72
Remove unused method in TimeAgoParser 2025-01-23 20:56:16 +01:00
Stypox
a7154c3087
Release v0.24.4 2025-01-21 23:14:31 +01:00
Stypox
4c720328ae
Merge pull request #1262 from Theta-Dev/fix/ios-client-vdata
[YouTube] update iOS client, add visitor data to requests
2025-01-21 23:12:58 +01:00
ThetaDev
936bf2d71b
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java
Co-authored-by: Stypox <stypox@pm.me>
2025-01-21 23:09:40 +01:00
Stypox
5b31ff20e5
iPhone 15 Pro Max instead of 16 in comments 2025-01-21 23:09:00 +01:00
ThetaDev
45645b043b
update comment 2025-01-21 13:12:51 +01:00
ThetaDev
65d888f4cf
move ProtoBuilder, add tests, fix incompatible rand call 2025-01-21 03:05:54 +01:00
ThetaDev
40059ed687
fix: update iOS client, add visitor data to YouTube requests 2025-01-19 19:59:10 +01:00
dependabot[bot]
a346f76704
Bump org.junit:junit-bom from 5.11.3 to 5.11.4
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.11.3 to 5.11.4.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.11.3...r5.11.4)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-12-17 09:49:42 +00:00
TobiGr
8e92227b2e Fix JDoc 2024-11-24 17:15:36 +01:00
Stypox
eebcc46255
Release v0.24.3 2024-11-24 17:07:47 +01:00
Stypox
9fb03f6c87
Merge pull request #1192 from TeamNewPipe/user-agent
[tests] Update user agent
2024-11-18 17:13:55 +01:00
TobiGr
e4a1a6ecd8 Fix tests 2024-11-17 21:38:47 +01:00
TobiGr
727e791602 [YouTube] Update mocks 2024-11-17 21:38:46 +01:00
TobiGr
d635d4db2a Make RecordingDownloader more resillient against ReCaptchaExceptions
Implement a max number of requests per minute to prevent hitting reate limits and triggering ReCaptchaExceptions. This slows down the RecordingDownloader significantly and can be adjusted if needed. A request ist retried once when facing a ReCaptchaException.
2024-11-17 21:38:46 +01:00
Stypox
ea1a1d1375
Merge pull request #1242 from TeamNewPipe/revert-1205-feature-branch
Revert "Refactored Identifiers"
2024-11-16 14:01:10 +01:00
Stypox
c00d0a7028
Revert "Refactored Identifiers (#1205)"
This reverts commit 0de224124be656b5792235303b7ce4dd0063165f.
2024-11-16 14:00:38 +01:00
Stypox
d3d5f2b3f0
Merge pull request #1240 from AudricV/yt_fix-playlists-items-extraction
[YouTube] Add support for new playlist items data structure
2024-11-14 16:37:35 +01:00
congyuluo
0de224124b
Refactored Identifiers (#1205)
Extractor.pageFetched -> Extractor.isPageFetched
Stream.equalStats(Stream) is renamed to Stream.areStatsEqual(Stream)
Stream.getBitrate() is renamed to Stream.getBitRate()
2024-11-13 10:01:20 +01:00
AudricV
183563cc9e
[YouTube] Add support for playlists lockupViewModels
This new data type, A/B tested or rolled out at the time the changes
are commited, is present on multiple surfaces.
2024-11-10 21:44:06 +01:00
AudricV
f52d2269fc
[YouTube] Move channel verified status check from badges to a method
This method will be used in more places such as the new playlist item
data.

Support for a new icon used in artist channels has been also added.
2024-11-10 20:18:04 +01:00
TobiGr
667c867ad8 Update user agent to Firefox ESR 128 2024-11-10 17:27:22 +01:00
Tobi
169098432b
Merge pull request #1239 from AudricV/yt_fix-shorts-thumbnails-extraction
[YouTube] Fix Shorts' thumbnails extraction in their channel tab
2024-11-06 09:42:13 +01:00
AudricV
06b2c8e2aa
[YouTube] Fix Shorts' thumbnails extraction in their channel tab
Wrong methods were used to access and extract the thumbnails' data.
This has been fixed with this commit.
2024-11-06 09:31:42 +01:00
Tobi
c343e31ed2
Merge pull request #1236 from Thompson3142/fix_scrubbing_seekbar_preview_crash
Add documentation for faulty framesets
2024-10-27 14:42:06 +01:00
TobiGr
1f26c12098 Use JDoc and inherit doc 2024-10-27 09:45:15 +01:00
Tobi
6af22e3e45
Merge pull request #1237 from AudricV/yt_more-audio-track-types-support
[YouTube] Add support for automatic dubbed and secondary audio tracks
2024-10-27 09:19:01 +01:00
AudricV
8a3350f79d
[YouTube] Add support for automatic dubbed and secondary tracks 2024-10-26 20:32:39 +02:00
Thompson3142
542867ff4d Add documentation for faulty framesets 2024-10-24 19:36:35 +02:00
Tobi
abba78cf9d
Merge pull request #1235 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.11.3
Bump org.junit:junit-bom from 5.11.2 to 5.11.3
2024-10-22 20:53:10 +02:00
dependabot[bot]
534bbc90cf
Bump org.junit:junit-bom from 5.11.2 to 5.11.3
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.11.2 to 5.11.3.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.11.2...r5.11.3)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-22 09:39:48 +00:00
Tobi
f169885dbc
Merge pull request #1219 from floriegl/fix-case-sensitive-jitpack-coordinates
Fix for JitPack case sensitive coordinates in README
2024-10-10 15:28:04 +02:00
Tobi
18c9f1fd38
Merge pull request #1233 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.11.2
Bump org.junit:junit-bom from 5.11.0 to 5.11.2
2024-10-05 20:22:02 +02:00
dependabot[bot]
fb81eaab82
Bump org.junit:junit-bom from 5.11.0 to 5.11.2
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.11.0 to 5.11.2.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.11.0...r5.11.2)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-05 18:17:38 +00:00
Tobi
5431069588
Merge pull request #1231 from AudricV/yt_fix-n-param-decode-function-extraction
[YouTube] Fix extraction of n param deobfuscation function name
2024-10-05 20:16:15 +02:00
Tobi
743a4000b8
Merge pull request #1207 from TeamNewPipe/fix/peertube-certain-domains
[PeerTube] Fix parsing ID for instances whose domain ends with a or c
2024-10-05 19:20:42 +02:00
AudricV
69ff271be1
[YouTube] Fix extraction of n param deobfuscation function name
This commit adds two new regular expressions to parse the n parameter function.

It also improves existing regular expressions by using the constant representing
multiple characters instead of adding the one or multiple characters
token manually in each regex for everything and not only function names.
2024-09-29 16:01:10 +02:00
Audric V.
eb30316a36
Merge pull request #1222 from AudricV/yt_fix-videos-channel-tab-linkhandler-serialization
[YouTube] Fix serialization of Videos channel tab when it is already fetched
2024-09-29 15:58:43 +02:00
AudricV
42c1afaf87
[YouTube] Fix serialization of Videos channel tab when already fetched
Also remove usage of Optional as fields as it is not a good practice. This
simplifies in some places channel info extraction code.
2024-09-29 15:35:21 +02:00
Audric V.
596bce294d
Merge pull request #1221 from AudricV/yt_support-new-shorts-ui-data
[YouTube] Fix extraction of Shorts in channels and remove visitor data usage
2024-09-29 14:54:07 +02:00
AudricV
f9ffdd91d5
[YouTube] Update YoutubeChannelTabExtractorTest.Shorts test class mocks 2024-09-08 17:51:08 +02:00
AudricV
34f28fc1f0
[YouTube] Remove visitorData usage for shorts continuations
It isn't required anymore and not used by extractor anymore since commit
5a6da5f43ebb0a612e6ffdd3feb90d78b3dd383e, as the wrong page ID is used as a
visitor data (the VerifiedStatus value as a string).
2024-09-08 17:41:23 +02:00
AudricV
f926fbcf35
[YouTube] Add support for shortsLockupViewModels
This new UI data type is replacing the reelItemRenderer one.
2024-09-08 17:21:40 +02:00
floriegl
36cc17c789
Update dependency coordinates due to case sensitivity in JitPack 2024-09-04 15:33:05 +02:00
TobiGr
6e3a4a6d9d [SoundCloud] Fix test: title changed 2024-08-15 12:22:37 +02:00
Tobi
70d6a06bf2
Merge pull request #1211 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.11.0
Bump org.junit:junit-bom from 5.10.3 to 5.11.0
2024-08-15 12:12:40 +02:00
dependabot[bot]
1278517492
Bump org.junit:junit-bom from 5.10.3 to 5.11.0
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.10.3 to 5.11.0.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.10.3...r5.11.0)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-08-15 09:10:12 +00:00
TobiGr
bcacfc53c5 [PeerTube] Fix parsing id for instances whose domain ends with a or c
The pattern to detect the channel ID was faulty and e.g. the ID detected for "https://kolektiva.media/video-channels/documentary_channel" was "a/video-channels" which is wrong.  A new pattern is added to distinguish between URLs and potential IDs because IDs must not start with a "/" while IDs inside an URL must.

Fixes TeamNewPipe/NewPipe#11369
2024-08-02 18:19:45 +02:00
TobiGr
6963385176 Fix issue template dir name 2024-08-02 12:50:30 +02:00
opusforlife2
5f1ba8cf7d
Add issue templates (#1204)
* Create bug report template

* Create feature request template

* Create config.yml

---------

Co-authored-by: TobiGr <tobigr@users.noreply.github.com>
2024-08-02 12:44:11 +02:00
Stypox
176da72cb4
Merge branch 'dev' 2024-07-25 18:29:07 +02:00
Stypox
530c157d4d
Release v0.24.2 2024-07-25 18:25:10 +02:00
Stypox
996eb046aa
Merge pull request #1203 from AudricV/yt_support-shows-and-pageheader-on-user-channels
[YouTube] Support shows and page header on user channels
2024-07-25 18:03:13 +02:00
AudricV
8db724943d
[YouTube] Update mocks of YoutubeChannelTabExtractorTest.Shorts test
For some reason, mocks of the continuation were not parsed. All mocks of the
test have been updated to fix the issue.
2024-07-25 17:51:43 +02:00
AudricV
76956ec95f
[YouTube] Fix VSauce test of YoutubeChannelExtractorTest test class
The channel description has been changed and some expected words have been
removed.
2024-07-25 17:51:43 +02:00
AudricV
10704dfc94
[YouTube] Fix NPE when getting channel header for videos channel tab 2024-07-25 17:51:43 +02:00
AudricV
8be64574e4
[YouTube] Support pageHeader on user channels
Also move duplicate strings into constants and add a missing default switch
case.
2024-07-25 17:51:42 +02:00
AudricV
df26badd4a
[YouTube] Add common methods to get ID, name and age gate object of channels
Also move duplicate strings into constants and support pageHeader channel
header in user channels on YoutubeChannelHelper methods.
2024-07-24 19:51:58 +02:00
AudricV
5a6da5f43e
[YouTube] Support shows in channels and provide verified status to items
Also fix naming of info items' collection methods.
2024-07-24 19:51:58 +02:00
AudricV
9d5201f40e
[YouTube] Add support for showRenderers in search results 2024-07-24 19:51:58 +02:00
AudricV
37178bd007
[YouTube] Add base implementation for show InfoItems
As there are multiple show UI elements which share a lot of common data, a base
implementation, an abstract class named YoutubeBaseShowInfoItemExtractor, has
been created to handle common cases.
2024-07-24 19:51:57 +02:00
AudricV
5879190ada
[YouTube] Move channel header's verified status code to YoutubeChannelHelper
Also throw an exception when we cannot get the verified status of a channel in
YoutubeChannelExtractor due to a missing channelHeader, if the channel has no
channelAgeGateRenderer.
2024-07-24 19:51:56 +02:00
AudricV
9fa8d4c0b4
[YouTube] Support playlists as URL navigation endpoints 2024-07-24 18:47:38 +02:00
AudricV
c99d94b615
[YouTube] Do not get twice runs array in YoutubeParsingHelper
The runs object was computed twice in getTextFromObject and getUrlFromObject
methods, leading to unneeded search costs. This has been avoided by storing the
array in method variables.
2024-07-24 18:47:30 +02:00
Stypox
2d36945b39
Merge pull request #1197 from AudricV/yt_innertube-clients-changes-for-streams
[YouTube] Workaround HTTP 403s on streaming URLs of WEB client (after some time or instantly for some JavaScript players), update clients info
2024-07-24 15:45:32 +02:00
AudricV
d73de6b12d
[YouTube] Don't provide streaming URLs which have an non-decoded n param
This param used to throttle bandwidth of streaming URLs which have this
parameter when the correct value is not provided but it is not the case
anymore, as the streaming URLs return now an HTTP response code 403 in
this case.
2024-07-23 20:48:39 +02:00
AudricV
22f818109f
[YouTube] Fix JavaScript n parameter decoding function name extraction
This commits fixes extraction of the function name decoding the n parameter for
HTML5 clients' streaming URLs for YouTube base JavaScript player 3400486c.

Two new regexes have been added to the existing ones. All regexes and what they
extract has been documented.
2024-07-23 20:43:56 +02:00
AudricV
480f5e223e
[YouTube] Update mocks 2024-07-23 20:43:54 +02:00
AudricV
986a76494c
[YouTube] Fix some YoutubeStreamExtractorDefaultTest tests
- Fix typo in folder name of DescriptionTestPewdiepie test;
- Fix constant usage of DownloaderTestImpl as download implementation for
UnlistedTest and CCLicensed tests.
2024-07-23 20:43:54 +02:00
AudricV
1c07764b4f
[YouTube] Fix YoutubeSearchExtractorTest.CrisisResources
The "blue whale" search query does not return a crisis resource panel anymore,
so it was changed to a different word, "suicide".
2024-07-23 20:43:54 +02:00
AudricV
a13510b962
[YouTube] Update clients info 2024-07-23 20:43:53 +02:00
AudricV
f4931d8bbd
[YouTube] Workaround 403s on streaming URLs of WEB client after some time
These changes work around an anti-bot token, for which its requirement is A/B
tested on the WEB client. In this test, streaming URLs of this client return
HTTP errors 403 if the token is not provided after some time.

It also allows to not fetch the JavaScript player for non-age restricted
videos, reducing data usage.

The TVHTML5 embed client is now only fetched in the case of age-restricted
videos.

The methods forceFetchAndroidClient and forceFetchIosClient of
YoutubeStreamExtractor have been removed, as they are now not needed anymore.

These changes also break the extraction of appropriate error states for private
and deleted videos and invalid video IDs.
2024-07-23 20:43:48 +02:00
Tobi
312e91048c
Merge pull request #1177 from TeamNewPipe/bandcamp-upgrade-to-https
[Bandcamp] Upgrade incoming links to HTTPS
2024-07-22 11:52:52 +02:00
Fynn Godau
f441036ed2 [Bandcamp] Upgrade incoming links to HTTPS 2024-07-22 11:48:02 +02:00
Marco Sirabella
02e14b8931
Add support for on.soundcloud.com urls (#1179)
Add support for on.soundcloud.com urls

Fixes #1178

Co-authored-by: TobiGr <tobigr@users.noreply.github.com>
2024-07-22 11:22:47 +02:00
Tobi
0e15f9ac1b
Merge pull request #1201 from TeamNewPipe/bandcamp-radio-v3
[Bandcamp] Show additional info in radio kiosk
2024-07-22 08:27:27 +02:00
Tobi
87af6bb223
Merge pull request #1200 from TeamNewPipe/bandcamp-fix-null-url-catenation
[Bandcamp] Null-safe url catenation in track playlist
2024-07-22 08:26:16 +02:00
Fynn Godau
227c6894a7 [Bandcamp] Show additional info in radio kiosk
A newer version of the radio list API delivers additional fields. Thanks
@crisp5.
2024-07-21 23:59:10 +02:00
Fynn Godau
97955e5e41 [Bandcamp] Null-safe url catenation in track playlist
Previously, if no URL was provided due to the track being a preorder or
unpaid track that doesn't have its own public page, we would catenate
the artist URL and null to a nonsensical string.

This invalid URL would also be used to try fetching a thumbnail URL.

The downstream behavior of the NewPipe client is only marginally
improved, as it now no longer throws visible exceptions due to the
missing thumbnails, but still gives an unfriendly error upon clicking
the item that has a `null` URL.
2024-07-21 23:35:57 +02:00
Tobi
4aaab63c12
Merge pull request #1199 from TeamNewPipe/bandcamp-update-artist-detection
[Bandcamp] Update artist page detection
2024-07-21 22:04:53 +02:00
Isira Seneviratne
9a29f9ee2d
Use the new URL encode/decode methods introduced in Java 10 (#1196)
* Use Java 10 URLDecoder/URLEncoder methods

* Simplify compatParseMap
2024-07-21 19:45:28 +05:30
Fynn Godau
de9fb7cb28 [Bandcamp] Update artist page detection
Bandcamp changed the way the footer is rendered. Therefore, we check for
a link to the cart page instead.
2024-07-21 15:37:21 +02:00
XiangRongLin
d39fc43282
[Youtube] Adjust throttling function extraction to changes (#1191)
* [Youtube] Adjust throttling function extraction to changes

---------

Co-authored-by: Stypox <stypox@pm.me>
2024-07-11 11:23:53 +02:00
XiangRongLin
592f1596e6
[Youtube] Adjust throttling function extraction to changes (#1191)
* [Youtube] Adjust throttling function extraction to changes

---------

Co-authored-by: Stypox <stypox@pm.me>
2024-07-11 11:20:33 +02:00
Tobi
c3c6de85bc
Merge pull request #1180 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.11.0
Bump com.google.code.gson:gson from 2.10.1 to 2.11.0
2024-06-29 11:27:13 +02:00
dependabot[bot]
383000f10d
Bump com.google.code.gson:gson from 2.10.1 to 2.11.0
Bumps [com.google.code.gson:gson](https://github.com/google/gson) from 2.10.1 to 2.11.0.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/main/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.10.1...gson-parent-2.11.0)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-06-29 09:22:41 +00:00
Tobi
2c7076930c
Merge pull request #1186 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.10.3
Bump org.junit:junit-bom from 5.10.2 to 5.10.3
2024-06-29 11:20:50 +02:00
dependabot[bot]
11a31721c5
Bump org.junit:junit-bom from 5.10.2 to 5.10.3
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.10.2 to 5.10.3.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.10.2...r5.10.3)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-06-28 09:50:44 +00:00
Tobi
90183056b5
Merge pull request #1183 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.8.6
Bump com.github.spotbugs:spotbugs-annotations from 4.8.5 to 4.8.6
2024-06-22 23:35:46 +02:00
dependabot[bot]
2efea787d2
Bump com.github.spotbugs:spotbugs-annotations from 4.8.5 to 4.8.6
Bumps [com.github.spotbugs:spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.8.5 to 4.8.6.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.8.5...4.8.6)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-06-18 09:35:16 +00:00
TobiGr
fafd471606 [PeerTube] Fix test for like count
Number changed
2024-05-08 19:25:12 +02:00
TobiGr
4f477ad72b [PeerTube] Fix testing comment content
The comment is not available anymore.
2024-05-08 19:25:12 +02:00
Tobi
964e429978
Merge pull request #1166 from TeamNewPipe/dependabot/github_actions/peaceiris/actions-gh-pages-4
Bump peaceiris/actions-gh-pages from 3 to 4
2024-05-08 19:24:40 +02:00
dependabot[bot]
10c6965a28
Bump peaceiris/actions-gh-pages from 3 to 4
Bumps [peaceiris/actions-gh-pages](https://github.com/peaceiris/actions-gh-pages) from 3 to 4.
- [Release notes](https://github.com/peaceiris/actions-gh-pages/releases)
- [Changelog](https://github.com/peaceiris/actions-gh-pages/blob/main/CHANGELOG.md)
- [Commits](https://github.com/peaceiris/actions-gh-pages/compare/v3...v4)

---
updated-dependencies:
- dependency-name: peaceiris/actions-gh-pages
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-05-08 17:16:08 +00:00
TobiGr
e54f38f5e7 [PeerTube] Fix test
UploaderName was changed by user
2024-05-08 19:13:49 +02:00
Tobi
5dd5c7a65b
Merge pull request #1173 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.8.5
Bump com.github.spotbugs:spotbugs-annotations from 4.8.3 to 4.8.5
2024-05-08 19:12:40 +02:00
Tobi
37438ff82f
Merge pull request #1174 from TeamNewPipe/dependabot/gradle/org.mozilla-rhino-1.7.15
Bump org.mozilla:rhino from 1.7.13 to 1.7.15
2024-05-08 19:02:57 +02:00
TobiGr
bba3b6c69b version 0.24.0 2024-05-08 13:01:06 +02:00
TobiGr
b40a5784ed Fix JDoc
YoutubePostLiveStreamDvrDashManifestCreator.java:90: error: unexpected end tag: </p>
2024-05-08 12:41:36 +02:00
dependabot[bot]
c6da4004e2
Bump org.mozilla:rhino from 1.7.13 to 1.7.15
Bumps [org.mozilla:rhino](https://github.com/mozilla/rhino) from 1.7.13 to 1.7.15.
- [Release notes](https://github.com/mozilla/rhino/releases)
- [Changelog](https://github.com/mozilla/rhino/blob/master/RELEASE-NOTES.md)
- [Commits](https://github.com/mozilla/rhino/commits)

---
updated-dependencies:
- dependency-name: org.mozilla:rhino
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-05-06 09:43:05 +00:00
dependabot[bot]
f26e84d39f
Bump com.github.spotbugs:spotbugs-annotations from 4.8.3 to 4.8.5
Bumps [com.github.spotbugs:spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.8.3 to 4.8.5.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.8.3...4.8.5)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-05-06 09:43:03 +00:00
Tobi
ec3e8378c6
Merge pull request #1171 from TeamNewPipe/fix/jdoc
Fix JavaDocs and add Checkstyle exception for line length in JavaDocs' links
2024-04-23 21:22:12 +02:00
TobiGr
8d2a7a5281 [PeerTube] Fix test 2024-04-23 20:02:29 +02:00
TobiGr
7c29dbc965 Fix JDoc
Add Checkstyle exception for LineLength in JDoc links.

See https://github.com/TeamNewPipe/NewPipeExtractor/actions/runs/8804403691/job/24164703883
2024-04-23 19:55:51 +02:00
Stypox
fbe9e6223a
Merge pull request #1168 from AudricV/yt_upd-cver-rm-keys-and-do-fixes
[YouTube] Update clients versions, restore access to some streams and more
2024-04-20 11:54:24 +02:00
Stypox
4e9e7cb29c
Improve assertTabsContain() to also check size 2024-04-20 11:48:36 +02:00
Stypox
9d0dd36034
[YouTube] Create constants for client names/versions 2024-04-20 11:43:54 +02:00
Stypox
d4e6d22e64
[YouTube] Improve meta info code for review 2024-04-20 11:43:08 +02:00
AudricV
74bf000473
[YouTube] Update mocks 2024-04-11 18:59:11 +02:00
AudricV
f9792cf3a9
[YouTube] Fix InteractiveTabbedHeader.testTabs test
YouTube now returns a Shorts tab for InteractiveTabbedHeader gaming channels,
which contains Shorts about the game of the topic channel but are not uploaded
on the game topic channel.

As this tab is already supported by the extractor, fetching a gaming topic
channel now returns a tab instead of none.

Channel name and channel URL of these Shorts needs to be set to null in a
separate commit, as Shorts on this tab do not have the topic channel as their
uploader.
2024-04-11 18:59:10 +02:00
AudricV
f40fc0aa4f
[YouTube] Add support for new crisis meta info action data
The new action data can return multiple contact actions instead of only one,
which will be concatenated by a new line return.

This commit fixes tests of the CrisisResources test class of
YoutubeSearchExtractorTest.
2024-04-11 18:59:09 +02:00
AudricV
2a3c6f80d2
[YouTube] Fix YoutubeStreamExtractorRelatedMixTest 2024-04-10 21:19:03 +02:00
AudricV
657b4377aa
[YouTube] Fix YoutubeStreamExtractorDefaultTest tests 2024-04-10 21:19:03 +02:00
AudricV
7bf50bf1cb
[YouTube] Update Android client player parameters
YouTube disabled the effectiveness of the parameters which were used (the
player response we get redirects to another video), but new parameters which
work around Android's client integrity checks have been found.
2024-04-10 21:19:03 +02:00
AudricV
27dc1b1f50
[YouTube] Remove usage of API keys for InnerTube requests, bump versions
The API keys are not used anymore by official clients in almost all cases
(still used by the Android app until it gets a configuration) for all requests
we made.

Clients and device OS versions have been bumped to their latest stable version
known.

Methods and fields related to API keys have been renamed or deleted if they're
no longer relevant.
2024-04-10 21:19:02 +02:00
AudricV
e380bb4bc3
[YouTube] Add missing prettyPrint query parameter to mixes continuations 2024-04-10 19:06:36 +02:00
Audric V
6c3c2e25d7
Merge pull request #1163 from AudricV/yt-fix_comments_extraction
[YouTube] Support new comments data
2024-04-10 18:19:59 +02:00
Stypox
02274d5395
[YouTube] Avoid XSS attacks in description or comments 2024-04-08 11:21:31 +02:00
Stypox
3f7b2653e3
[YouTube] Add YoutubeDescriptionHelperTest 2024-04-08 11:21:31 +02:00
Stypox
a90237816a
[YouTube] Cleanup description helper
Remove unneeded isClose field, and make constants private
2024-04-08 11:21:31 +02:00
Stypox
b80c3f5d51
[YouTube] Replace link text with accessibility label 2024-04-08 00:14:28 +02:00
Stypox
09732d6785
[YouTube] Add support for styles in attributed descriptions
Also refactor descriptions parsing.
2024-04-04 21:14:27 +02:00
AudricV
293c3e9e47
[YouTube] Support new A/B tested comments data
Also improve current comments code by removing outdated comment
renderer data.
2024-04-04 21:14:26 +02:00
Stypox
e5b30ae8c3
Merge pull request #1151 from Profpatsch/localization-return-optional
LocaleCompat.forLanguageTag: return Optional if parsing fails
2024-03-29 13:50:48 +01:00
Stypox
23fc7aa209
Throw ParsingException instead of IllegalArg 2024-03-29 13:44:42 +01:00
Stypox
fb468a23f4
Merge pull request #1142 from TeamNewPipe/peertube-v6
[PeerTube] Add support for PeerTube v6 features
2024-03-29 12:25:38 +01:00
Stypox
6589e2c15d
Merge pull request #1148 from Stypox/mediaccc-channel-tab-handler
[MediaCCC] Allow obtaining channel tab link handler
2024-03-28 13:45:05 +01:00
Tobi
ad71864b23
Merge pull request #1162 from Stypox/fix-comment-description-npe
Make getCommentText @Nonnull
2024-03-27 20:04:17 +01:00
Stypox
c57016b79b
Make getCommentText @Nonnull 2024-03-27 15:26:06 +01:00
Tobi
adcc1f17ee
Merge pull request #1160 from TeamNewPipe/fix/tests
Fix some failing unit tests and detect new account termination messages for YouTube
2024-03-20 15:17:55 +01:00
TobiGr
51ddacc81d [SoundCloud] Fix SoundcloudSearchExtractorTest.NoNextPage
Search did not return no item at all, causing a NothingFoundException. New search query yields three items on first page
2024-03-20 15:10:39 +01:00
TobiGr
8392d50ba6 Update mocks for YoutubeChannelExtractorTest.NotAvailable 2024-03-20 14:59:44 +01:00
TobiGr
aaccfecda8 [YouTube] Detect new account termination messages 2024-03-20 14:57:41 +01:00
TobiGr
73f0c63a9d [PeerTube] Fix tests for "What is PeerTube?" video 2024-03-20 14:44:06 +01:00
Tobi
896a55e319
Merge pull request #1139 from TeamNewPipe/dependabot/github_actions/actions/upload-artifact-4
Bump actions/upload-artifact from 3 to 4
2024-03-18 08:59:35 +01:00
dependabot[bot]
e58fc652e0
Bump actions/upload-artifact from 3 to 4
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3 to 4.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-03-18 07:53:24 +00:00
Tobi
e3f2c9aec7
Merge pull request #1153 from TeamNewPipe/dependabot/github_actions/actions/cache-4
Bump actions/cache from 3 to 4
2024-03-18 08:51:24 +01:00
Tobi
6b0fc14c04
Merge pull request #1156 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.10.2
Bump org.junit:junit-bom from 5.10.0 to 5.10.2
2024-03-18 08:46:13 +01:00
dependabot[bot]
d579b608e5
Bump org.junit:junit-bom from 5.10.0 to 5.10.2
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.10.0 to 5.10.2.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.10.0...r5.10.2)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-02-05 09:15:26 +00:00
TobiGr
fe47a4311f [PeerTube] Add test for segments and framesets 2024-01-29 10:22:06 +01:00
TobiGr
15e0e74b48 [PeerTube] Add support for stream frames/storyboards extraction
Implement PeerTubeStreamExtractor.getFrames()
2024-01-29 10:22:06 +01:00
dependabot[bot]
da04eded5d
Bump actions/cache from 3 to 4
Bumps [actions/cache](https://github.com/actions/cache) from 3 to 4.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-01-18 09:32:42 +00:00
Profpatsch
7408173246 LocaleCompat.forLanguageTag: return Optional if parsing fails
It’s not obvious that the function will fail in some cases and throw
an `IllegalArgumentException`.

So instead of just failing if parsing fails, return an Optional that
all callers have to decide what to do (e.g. the YoutubeExtractor can
just ignore the locale in that case, like it does with most other
fields in the json if they are unexpected).
2024-01-07 14:31:34 +01:00
Stypox
aaf3231fc7
[MediaCCC] Fix lambda link handler keeping reference to extractor
This caused problems in NewPipe, because extractors are not serializable, and well, keeping references to them is a bad idea anyway.
2023-12-30 23:23:19 +01:00
Stypox
137e924035
[MediaCCC] Add ChannelTabExtractorTest 2023-12-30 22:53:51 +01:00
Stypox
cc9ade962e
[MediaCCC] Allow obtaining channel tab extractor from scratch
i.e. without needing to pass through the conference/channel extractor
This was needed because clients (like NewPipe) might rely on link handlers to hold as little data as possible, since they might be kept around for long or passed around in system transactions, so this commit allows obtaining a standalone link handler that does not hold a JsonObject within itself.
2023-12-30 22:53:27 +01:00
Stypox
3402cdb666
Merge pull request #1147 from petlyh/youtube-albums
[YouTube] Add Albums channel tab
2023-12-30 21:58:19 +01:00
petlyh
6dc25f7b97
[YouTube] Add Albums channel tab mocks 2023-12-30 14:46:39 +01:00
petlyh
4408e2d0ac
[YouTube] Add Albums channel tab 2023-12-30 14:01:30 +01:00
TobiGr
9ab932e394 Rename testDoNotAcceptNonURLs() -> assertDoNotAcceptNonURLs() 2023-12-29 16:38:11 +01:00
Stypox
9d66debf3c
Merge pull request #1132 from TeamNewPipe/dependabot/github_actions/actions/setup-java-4
Bump actions/setup-java from 3 to 4
2023-12-29 16:20:46 +01:00
Tobi
038ebdedc4
Merge pull request #1143 from petlyh/peertube-non-urls
Avoid PeerTube accepting non-URLs
2023-12-29 12:47:30 +01:00
TobiGr
61d237de02 [PeerTube] Test onAccept(String URL) in LinkHandlerFactories for non-URLs 2023-12-29 12:45:02 +01:00
petlyh
2b2c1546d1 Avoid PeerTube accepting non-URLs 2023-12-29 12:27:39 +01:00
Tobi
1e93b1dc20
Merge pull request #1135 from Stypox/yt-emergency-info
[YouTube] Implement emergency meta info
2023-12-29 12:01:40 +01:00
dependabot[bot]
3400af99b3
Bump actions/setup-java from 3 to 4
Bumps [actions/setup-java](https://github.com/actions/setup-java) from 3 to 4.
- [Release notes](https://github.com/actions/setup-java/releases)
- [Commits](https://github.com/actions/setup-java/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/setup-java
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-29 10:57:47 +00:00
Tobi
1f8a044462
Merge pull request #1138 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.8.3
Bump com.github.spotbugs:spotbugs-annotations from 4.8.0 to 4.8.3
2023-12-29 11:56:18 +01:00
dependabot[bot]
1470aa7303
Bump com.github.spotbugs:spotbugs-annotations from 4.8.0 to 4.8.3
Bumps [com.github.spotbugs:spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.8.0 to 4.8.3.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.8.0...4.8.3)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-29 10:53:28 +00:00
TobiGr
8f9ebdcb77 [PeerTube] Fix failing PeertubeTrendingLinkHandlerFactoryTest
The factory was updated in #1144
2023-12-29 11:52:19 +01:00
Stypox
1553931027
Merge pull request #1145 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.17.2
Bump org.jsoup:jsoup from 1.16.2 to 1.17.2
2023-12-29 11:27:01 +01:00
Stypox
b2ec1b15fb
Merge pull request #1144 from dragfyre/patch-1
Update PeertubeTrendingLinkHandlerFactory.java
2023-12-29 11:21:08 +01:00
dependabot[bot]
151ee99da3
Bump org.jsoup:jsoup from 1.16.2 to 1.17.2
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup) from 1.16.2 to 1.17.2.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES.md)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.16.2...jsoup-1.17.2)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-29 09:47:00 +00:00
dragfyre
65e7bc5b95
Update PeertubeTrendingLinkHandlerFactory.java
correcting Peertube local trending api URL (per #10685 in main NewPipe repo); see https://docs.joinpeertube.org/api-rest-reference.html#tag/Video/operation/getVideos
2023-12-28 14:50:31 +07:00
Stypox
f276caf54a
Release v0.23.1 2023-12-21 21:59:21 +01:00
Stypox
fc54fb2fdb
Merge pull request #1140 from Stypox/yt-shorts-no-duration
[YouTube] Always return -1 as duration of Shorts returned inside reel items
2023-12-21 21:52:40 +01:00
Stypox
0518487d26
Fix SearchInfo's non-null MetaInfo being null when initialized or when an extraction error occurs (#1141)
The meta info might have been null either when SearchInfo is first initialized, or when extractor.getMetaInfo() throws an exception in getInfo().
This caused NewPipe to crash instead of showing a nice error in https://www.reddit.com/r/youtube/comments/184ttmw/what_exactly_about_blue_whales_has_youtube_so/.
2023-12-21 21:39:25 +01:00
Stypox
5b59a1a8c5
[YouTube] Move meta info extraction to separate file
YoutubeParsingHelper was longer than 2000 lines which caused checkstyle issues
2023-12-21 21:19:08 +01:00
Stypox
b8e12dd76c
[YouTube] Implement emergency meta info
YouTube provides that meta info panel when users search for really sensitive content like suicide (e.g. "blue whale").

It contains:
- an encouragement as title (e.g. "We are with you")
- a phone number as action
- details about how to call the phone number (e.g. availability)
- an url pointing to the website of an association

Also add a test that just checks if a meta info is properly extracted
2023-12-21 21:19:08 +01:00
Stypox
83c1737f70
[YouTube] Update shorts channel tab mocks 2023-12-21 20:54:16 +01:00
Stypox
2938067c2c
[YouTube] Shorts don't provide a duration anymore 2023-12-21 20:41:01 +01:00
Stypox
91419ec6e8
Release v0.23.0 2023-12-10 15:59:32 +01:00
Audric V
678c98f24c
Merge pull request #1127 from AudricV/yt_improvements-and-fixes
[YouTube] Make some improvements and fixes
2023-12-09 14:07:31 +01:00
AudricV
ec0194cfbf
[YouTube] Update mocks 2023-12-08 21:46:52 +01:00
AudricV
00a0f1a103
[YouTube] Add a blocking crisis resources bypass in search results test
This test only tests that search results are returned, when no content filters
are provided and crisis resources blocking search results should be returned.

Searches with blocking crisis resources and content filters should work too, as
the bypass has been implemented for them.
2023-12-08 21:46:49 +01:00
AudricV
06838d7245
[YouTube] Fix YoutubeSearchQHTest tests
As search parameters to bypass crisis resources blocking search results have
been implemented, they need to be added to search tests, in order to pass
them.
2023-12-08 21:46:49 +01:00
AudricV
642bb01388
[YouTube] Fix testSubscriberCount method of YoutubeChannelExtractorTest.CarouselHeader
The subscriber count is now lower than the expected count as some people
unsubscribed to the Sports system channel. The expected count has been so
lowered.
2023-12-08 21:46:49 +01:00
AudricV
bedc9e5bc0
[YouTube] Remove Channels channel tab in tests 2023-12-08 21:46:48 +01:00
AudricV
5fa22ae25b
[YouTube] Enable and change testRelatedItems method of YoutubePlaylistExtractorTest.LearningPlaylist
This change makes the method test whether there is no more items instead of
being disabled.
2023-12-08 21:46:48 +01:00
AudricV
29dc7625f2
[YouTube] Fix YoutubeSearchExtractorTest.Suggestion
YouTube doesn't return anymore a suggestion for the query "algorythm", but does
for the query "on board ing" ("on boarding"). This search query is now used and
had to be URL-encoded.

URL encoding in the complete YoutubeSearchExtractorTest test class uses now
extractor's Util class instead of Java's URLDecoder class directly.
2023-12-08 21:46:48 +01:00
AudricV
56ab35423e
[YouTube] Fix potential NullPointerException in YoutubeSearchExtractor.getSearchSuggestion 2023-12-08 21:46:48 +01:00
AudricV
9dc1eab28c
[YouTube] Fix expected upload dates of StreamExtractor tests
YouTube is rolling out or A/B testing a new date format returned inside player
responses, which are precise to the second instead of the day.

This commit makes the StreamExtractor tests use these more precise dates.
2023-12-08 21:46:48 +01:00
AudricV
ad3d187ac7
[YouTube] Fix testRelatedItems method of YoutubeStreamExtractorRelatedMixTest
This commit fixes the testRelatedItems test method by:
- accepting consent in the test class, in order to extract mixes in
recommendations;
- removing assertion of a music mix inside the recommendations, as YouTube
doesn't seem to return such mixes anymore, at least for the video used in the
test class.
2023-12-08 21:46:47 +01:00
AudricV
e111814401
[YouTube] Fix YoutubeStreamExtractorDefaultTest.RatingsDisabledTest
Replace the video used in this test class with another one publicly available
and update the corresponding expected test values.

The test class's mocks will be updated in a different commit.
2023-12-08 21:46:47 +01:00
AudricV
fc45941ead
[YouTube] Fix YoutubeChannelExtractorTest tests
- Change CarouselHeader test channel to Sports system one, as the Coachella one
doesn't return this channel header anymore;
- Fix InteractiveTabbedHeader test by checking whether the test's channel
description is not empty instead of containing some words, as it is changing
frequently.
2023-12-08 21:46:47 +01:00
AudricV
0bcb241c38
[YouTube] Fix expected data in YoutubeStreamExtractorRelatedMixTest
Video's title and tags have been changed by its uploader, so they have to be
updated.

Also make some package-private constants private, as they are not used outside
of the class, and remove unneeded test overrides.
2023-12-08 21:46:47 +01:00
AudricV
6ba8251be1
[YouTube] Bypass crisis resources blocking search results
These crisis resources are preventing search results to be returned. See
https://support.google.com/youtube/answer/10726080?hl=en for more info on them.

This commit changes search parameters to include the property allowing to show
search results.
2023-12-08 21:46:47 +01:00
AudricV
7dea2d0d27
[YouTube] Remove Channels channel tab support
This tab has been removed by YouTube.
2023-12-08 21:46:47 +01:00
AudricV
3782d9a02a
[YouTube] Support new A/B tested like data and avoid like count conversion from integer to long
Also make minor improvements to current like data extraction and remove
previous like count data support, as it is not returned anymore.
2023-12-08 21:46:46 +01:00
AudricV
b71ce1123f
[YouTube] Extract only search results corresponding to a search type
YouTube returns sometimes videos inside channel search results. As we only want
results corresponding to the type we requested, this commits makes
YoutubeSearchExtractor ignoring non-requested search results we get, using the
extractor LinkHandler's first content filter value.

Also remove an unneeded exception throwing declaration in
YoutubeSearchExtractor.
2023-12-08 21:46:46 +01:00
AudricV
ff8ed7247f
[YouTube] Switch to new consent cookie
Also move the documentation of the consent in its setter method in order to be
accessible publicly and improve it.
2023-12-08 21:46:46 +01:00
AudricV
ec838d7421
[YouTube] Add missing prettyPrint query parameter to some test InnerTube requests
This query parameter for which its value is set to false was not added to two
requests made in test classes of YoutubeMixPlaylistExtractorTest.

Also remove an unneeded ParsingException exception throwing declaration in a
test method.
2023-12-08 21:46:46 +01:00
AudricV
2c941794c0
[YouTube] Add utcOffsetMinutes to all InnerTube payloads
This should make returned dates consistent between timezones and countries on
which the extractor is ran.

It was previously only set on YouTube Music search continuations.
2023-12-08 21:46:46 +01:00
AudricV
d97c9e0db1
[YouTube] Improve payloads and URLs of InnerTube requests
For every InnerTube request:
- Always add a `request` object with the following properties:
  - "internalExperimentFlags" set to an empty array;
  - "useSsl" set to "true";
  - "lockedSafetyMode" set to "false".
- Use proper TODO comment to provide a way to enable restricted mode on every
request and add it on requests on which it wasn't present.

For YouTube Music:
- Remove alt query parameter, as it is not used anymore by the website;
- Add prettyPrint query parameter with false value on YouTube Music search
continuations.
2023-12-08 21:46:45 +01:00
AudricV
8a9ebcc373
[YouTube] Update InnerTube clients' version and devices' OS version and model 2023-12-08 21:46:45 +01:00
Tobi
eac850ca10
Merge pull request #1114 from FineFindus/feat/comment-author-is-owner
[YouTube] Add channel owner to comments
2023-10-25 09:51:12 +02:00
Tobi
5ab1f784e8
Merge pull request #1117 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.16.2
Bump org.jsoup:jsoup from 1.16.1 to 1.16.2
2023-10-21 19:26:36 +02:00
dependabot[bot]
9d7bcba050
Bump org.jsoup:jsoup from 1.16.1 to 1.16.2
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup) from 1.16.1 to 1.16.2.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.16.1...jsoup-1.16.2)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-20 09:13:21 +00:00
dependabot[bot]
e26065148a Bump com.github.spotbugs:spotbugs-annotations from 4.7.3 to 4.8.0
Bumps [com.github.spotbugs:spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.7.3 to 4.8.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.3...4.8.0)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-12 14:19:10 +02:00
FineFindus
34b05a0dda
feat(youtube/comments): support creator replies 2023-10-09 16:33:43 +02:00
TobiGr
0821f09114
Add missing mocks 2023-10-09 16:33:43 +02:00
FineFindus
c1784a4bdb
[YouTube] Add channel owner to comments 2023-10-09 16:33:43 +02:00
TobiGr
f9846352ea Fix wrong @Nullable annotation 2023-10-09 16:02:57 +02:00
Tobi
d6f5cba6e2
Merge pull request #1111 from FineFindus/feat/creator-reply
Add `hasCreatorReply()` to CommentsInfoItem
2023-10-09 12:45:56 +02:00
TobiGr
9d63c75623 Add missing mocks 2023-10-09 11:24:39 +02:00
TobiGr
d49f8411d7 [PeerTube] Implement CommentsInfoItemExtractor.hasCreatorReply() 2023-10-09 02:47:12 +02:00
Stypox
bb132167d5
Merge pull request #1113 from AudricV/snd_fix-non-jpg-images
[SoundCloud] Fix extraction of non-JPG images
2023-10-02 19:40:57 +02:00
AudricV
c98695fcea
[SoundCloud] Fix extraction of non-JPG images
Default image qualities were removed in image URLs with the jpg extension,
causing the addition of the image suffix to full non-JPG images URLs and so to
invalid image URLs.

Only the image quality name with its leading "-" character and the "."
character after the name is now removed and replaced by a string format
replaced itself with the image quality name for each quality.

As the image suffixes do not contain the image extension, the name of image
qualities lists has been adapted with these changes and some related comments
have been also improved.
2023-10-01 20:33:25 +02:00
AudricV
ac00459c1a
Change requirement of image extensions in ImageSuffix class' Javadoc to a possibility
Some services may provide different image formats using the same suffix,
without we know what format the service provide. Enforcing an image extension
could so lead to provide invalid image URLs, like for SoundCloud PNG images
currently.

With this documentation change, it is now clear that users of this class decide
of whether they want to include image extensions in the suffix. The previous
behavior described in the Javadoc was not enforced.
2023-09-30 21:11:09 +02:00
FineFindus
dd7b2d9798
feat(youtube/comments): support creator replies 2023-09-25 10:40:45 +02:00
Youssif Shaaban Alsager
917554acc4
[YouTube] Add support for ultralow audio formats (#1063) 2023-09-24 19:04:34 +02:00
Tobi
8b0068f8f4
Merge pull request #1110 from christian-2hu/chore/update-copyright
chore: Update copyright notices
2023-09-23 00:29:19 +02:00
Christian
fc67d49f59 Update copyright notices
Update copyright notices to comply to GPLv3 and change NewPipe to NewPipe Extractor on some notices that were not updated.
2023-09-22 19:10:15 -03:00
Stypox
289db1178a
Merge pull request #1108 from AudricV/yt_refactor-js-usage
[YouTube] Refactor JavaScript usage and fix extraction of obfuscated signature deobfuscation function
2023-09-22 10:41:57 +02:00
AudricV
6ed22099a2
[YouTube] Update stream mocks 2023-09-21 21:59:34 +02:00
AudricV
714b141ecb
[YouTube] Catch any exception when extracting something from JavaScript's base player 2023-09-21 21:59:33 +02:00
AudricV
588c6a8422
[YouTube] Quote signature deobfuscation function name and add semicolon only where needed 2023-09-21 21:59:33 +02:00
AudricV
1fa85ec6ca
[YouTube] Add tests for signature timestamp extraction and signature deobfuscation function extraction and execution 2023-09-21 21:59:33 +02:00
AudricV
a04bc320de
[YouTube] Convert signature timestamp to integer
The signature timestamp is used as a number by HTML5 clients, so it should be
used in the same way by the extractor too instead of being a string.

As the timestamp doesn't seem to exceed 5 digits, an integer is used to store
its value.
2023-09-21 21:59:32 +02:00
AudricV
7de3753a81
[YouTube] Refactor JavaScript player management API
This commit is introducing breaking changes.

For clients, everything is managed in a new class called
YoutubeJavaScriptPlayerManager:
- caching JavaScript base player code and its extracted code (functions and
variables);
- getting player signature timestamp;
- getting deobfuscated signatures of streaming URLs;
- getting streaming URLs with a throttling parameter deobfuscated, if
applicable.

The class delegates the extraction parts to external package-private classes:
- YoutubeJavaScriptExtractor, to extract and download YouTube's JavaScript base
player code: it always already present before and has been edited to mainly
remove the previous caching system and made it package-private;
- YoutubeSignatureUtils, for player signature timestamp and signature
deobfuscation function of streaming URLs, added in a recent commit;
- YoutubeThrottlingParameterUtils, which was originally
YoutubeThrottlingDecrypter, for throttling parameter of streaming URLs
deobfuscation function and checking whether this parameter is in a streaming
URL.

YoutubeJavaScriptPlayerManager caches and then runs the extracted code if it
has been executed successfully. The cache system of throttling parameters
deobfuscated values has been kept, its size can be get using the
getThrottlingParametersCacheSize method and can be cleared independently using
the clearThrottlingParametersCache method.

If an exception occurs during the extraction or the parsing of a function
property which is not related to JavaScript base player code fetching, it is
stored until caches are cleared, making subsequent failing extraction calls of
the requested function or property faster and consuming less resources, as the
result should be the same until the base player code changes.

All caches can be reset using the clearAllCaches method of
YoutubeJavaScriptPlayerManager.

Classes using JavaScript base player code and utilities directly (in the code
and its tests) have been also updated in this commit.
2023-09-21 21:59:32 +02:00
AudricV
6884d191cd
[YouTube] Add utility class around signatures and fix signature deobfuscation function extraction
The goal of this class is to decouple the extraction of signature timestamp and
signature deobfuscation function from YoutubeStreamExtractor.

The extraction of the signature deobfuscation function has been also adapted to
support the latest YouTube player versions.

This new class, YoutubeSignatureUtils, doens't store anything temporary such as
a copy of the player code, which has to be passed where required. It is not
public, as it will be used by a JavaScript player manager class in the future,
in order to handle in a better way fetching, caching and resetting cache of the
player code.
2023-09-21 21:59:26 +02:00
Tobi
3be76a6406
Merge pull request #1107 from Isira-Seneviratne/Locale_forLanguageTag
Use Locale.forLanguageTag() in tests
2023-09-18 16:48:30 +02:00
TobiGr
17790328cd Improve doc 2023-09-18 16:44:51 +02:00
Isira Seneviratne
4bc8ae7812 Use Locale.forLanguageTag() in tests 2023-09-18 08:59:13 +05:30
Tobi
90aed06a63
Merge pull request #1105 from TeamNewPipe/fix/bandcamp-streame-extractor-test
[Badcamp] Fix StreamExtractorTest
2023-09-18 01:49:04 +02:00
TobiGr
cf49f4a31c [Badcamp] Fix StreamExtractorTest
The song was renamed and the URL changed
2023-09-17 23:58:07 +02:00
Tobi
7c7ceaceab
Merge pull request #1103 from TeamNewPipe/dependabot/github_actions/actions/checkout-4
Bump actions/checkout from 3 to 4
2023-09-17 22:42:01 +02:00
dependabot[bot]
72c475d944
Bump actions/checkout from 3 to 4
Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-09-05 10:00:13 +00:00
Stypox
1f08d28ae5
Merge pull request #889 from AudricV/multiple-images-support
Multiple images support
2023-08-13 11:35:11 +02:00
AudricV
e8bfd20170
[MediaCCC] Apply changes in extractor tests
Also remove some public test methods modifiers.
2023-08-12 22:56:33 +02:00
AudricV
0292c4f3e8
[Bandcamp] Apply changes in extractor tests
Also remove some public test methods modifiers, add missing Test annotations on
old Junit 4 tests (and update them if needed), and use final in some places
where it was possible.

BandcampChannelExtractorTest.testLength has been removed as the test is always
true.
2023-08-12 22:56:32 +02:00
AudricV
2578f22054
[Bandcamp] Add utility test method to test images
This method, testImages(Collection<Image>), will use first the default image
collection test in DefaultTests and then will check that each image URL
contains f4.bcbits.com/img and ends with .jpg or .png.

To do so, a new non-instantiable final class has been added: BandcampTestUtils.
2023-08-12 22:56:32 +02:00
AudricV
ba5315c72d
[PeerTube] Apply changes in extractor tests
Also remove some public test methods modifiers, add missing Test annotations on
old Junit 4 tests (and update them if needed), and improve some code.
2023-08-12 22:56:32 +02:00
AudricV
1d72bac53d
[SoundCloud] Apply changes in extractor tests 2023-08-12 22:56:32 +02:00
AudricV
93a210394d
[YouTube] Apply changes in extractor tests
Also remove some public test methods modifiers, add missing Test annotations on
old Junit 4 tests (and update them if needed), and use final in some places
where it was possible.
2023-08-12 22:56:31 +02:00
AudricV
2c436d428c
[YouTube] Add utility test method to test images in YoutubeTestsUtils
This method, testImages(Collection<Image>), will use first the default image
collection test in DefaultTests and then will check that each image URL
contains the string yt.

The JavaDoc of the class has been also updated to reflect the changes made in
it (it is now more general).
2023-08-12 22:56:31 +02:00
AudricV
d381f3b70b
Update avatar, banners and thumbnail methods' name and apply changes in DefaultStreamExtractorTest 2023-08-12 22:56:31 +02:00
AudricV
434e885708
Add utility methods in ExtractorAsserts to check whether a collection is empty and to test image collections
Two new methods have been added in ExtractorAsserts to check if a collection is
empty:

- assertNotEmpty(String, Collection<?>), checking:
  - the non nullity of the collection;
  - its non emptiness (if that's not case, an exception will be thrown using
    the provided message).

- assertNotEmpty(Collection<?>), calling assertNotEmpty(String, Collection<?>)
  with null as the value of the string argument.

A new one has been added to this assertion class to check the contrary:
assertEmpty(Collection<?>), checking emptiness of the collection only if it is
not null.

Three new methods have been added in ExtractorAsserts as utility test methods
for image collections:

- assertContainsImageUrlInImageCollection(String, Collection<Image>), checking
that:
  - the provided URL and image collection are not null;
  - the image collection contains at least one image which has the provided
    string value as its URL (which is a string) property.

- assertContainsOnlyEquivalentImages(Collection<Image>, Collection<Image>),
  checking that:
  - both collections are not null;
  - they have the same size;
  - each image of the first collection has its equivalent in the second one.
    This means that the properties of an image in the first collection must be
    equal in an image of the second one.

- assertNotOnlyContainsEquivalentImages(Collection<Image>, Collection<Image>),
  checking that:
  - both collections are not null;
  - one of the following conditions is met:
    - they have different sizes;
    - an image of the first collection has not its equivalent in the second one.
      This means that the properties of an image in the first collection must
      be not equal in an image of the second one.

These methods will be used by services extractors tests (and default ones) to
test image collections.
2023-08-12 22:56:31 +02:00
AudricV
5158472852
Apply changes in DefaultTests and add utility method to test image lists
This new method, defaultTestImageList(List<Image), will check that the image
list is not null.

For each image, it will test that its URL is secure and its height and width
are more than or equal to their relevant unknown constants in the Image class
(HEIGHT_UNKNOWN and WIDTH_UNKNOWN).
2023-08-12 22:56:31 +02:00
AudricV
70fb3aa38e
Update BaseExtractorTests image methods' name
Also suppress unused warnings in BaseStreamExtractorTest, like it is done on
other BaseExtractorTests interfaces.
2023-08-12 22:56:30 +02:00
AudricV
e16d521b7b
[MediaCCC] Apply changes in Extractors
Also remove usage of the conference logo as the banner of a conference, as it
is a logo and not a banner.
2023-08-12 22:56:30 +02:00
AudricV
306068a63b
[MediaCCC] Apply changes in InfoItemExtractors 2023-08-12 22:56:30 +02:00
AudricV
2f40861428
[MediaCCC] Add utility methods to get image lists from conference logos and streams
These three new methods, added in MediaCCCParsingHelper,
getImageListFromImageUrl(String), getThumbnailsFromStreamItem(JsonObject) and
getThumbnailsFromLiveStreamItem(JsonObject) (the last two are based on a common
method, getThumbnailsFromObject(JsonObject, String, String)), return an empty
list if the case no image URL could be extracted.

Images returned have their height and width unknown and a resolution level
depending on the image key of the JSON API response.
2023-08-12 22:56:30 +02:00
AudricV
71cda03c4c
[Bandcamp] Apply changes in Extractors 2023-08-12 22:56:29 +02:00
AudricV
7e01eaac33
[Bandcamp] Apply changes in InfoItemExtractors 2023-08-12 22:56:29 +02:00
AudricV
4b80d737a4
[Bandcamp] Add utility methods to get multiple images
Bandcamp images work with image IDs, which provide different resolutions.

Images on Bandcamp are not always squares, and some IDs respect aspect ratios
where some others not.

The extractor will only use the ones which preserve aspect ratio and will not
provide original images, for performance and size purposes.

Because of this aspect ratio preservation constraint, only one dimension will
be known at a time.

The image IDs with their respective dimension used are:

- 10: 1200w;
- 101: 90h;
- 170: 422h;
- 171: 646h;
- 20: 1024w;
- 200: 420h;
- 201: 280h;
- 202: 140h;
- 204: 360h;
- 205: 240h;
- 206: 180h;
- 207: 120h;
- 43: 100h;
- 44: 200h.

(Where w represents the width of the image and h the height of the image)

Note that these dimensions are theoretical because if the image size is less
than the dimensions of the image ID, it will be not upscaled but kept to its
original size.

All these resolutions are stored in a private static list of ThumbnailSuffixes
in BandcampExtractorHelper, in which the methods to get mutliple images have
been added:

- getImagesFromImageUrl(String): public method to get images from an image URL;
- getImagesFromImageId(long, boolean): public method to get images from an
  image ID;
- getImagesFromImageBaseUrl(String): private utility method to get images from
  the static list of ThumbnailSuffixes from a given image base URL, containing
  the path to the image, a "a" letter if it comes from an album, its ID and an
  underscore.

Some existing methods have been also edited:

- the documentation of getImageUrl(long, boolean) has been changed to reflect
  the Bandcamp images findings;
- getThumbnailUrlFromSearchResult has been renamed to
  getImagesFromSearchResult, and a documentation has been added to this method.

The method replaceHttpWithHttps of the Utils class has been also used in
BandcampExtractorHelper instead of doing manually what the method does.
2023-08-12 22:56:29 +02:00
AudricV
4e6fb368bc
[PeerTube] Apply changes in Extractors and remove usages of default avatar picture
The default avatar picture was used when no profile picture was found, but it
was removed and split in multiple images.

Thumbnails' size is not known, as this data is not provided by the API.
2023-08-12 22:56:29 +02:00
AudricV
0a6011a50e
[PeerTube] Apply changes in InfoItemExtractors
Also lower the visibility of attributes of channels and playlists InfoItems to
private.
2023-08-12 22:56:29 +02:00
AudricV
6f8331524b
[PeerTube] Add utility method to get thumbnails of playlists and videos
This method, getThumbnailsFromPlaylistOrVideoItem, has been added in
PeertubeParsingHelper and returns the two image variants for playlists and
videos.
2023-08-12 22:56:28 +02:00
AudricV
81c0d80a54
[PeerTube] Add utility methods to get avatars and banners of accounts and channels
Four new static methods have been added in PeertubeParsingHelper to do so:
- two public methods to get the corresponding image type:
  getAvatarsFromOwnerAccountOrVideoChannelObject(String, JsonObject) and
  getBannersFromAccountOrVideoChannelObject(String, JsonObject);
- two private methods as helper methods: getImagesFromAvatarsOrBanners(String,
  JsonObject, String, String) and getImagesFromAvatarOrBannerArray(String,
  JsonArray).
2023-08-12 22:56:28 +02:00
AudricV
31da5beb51
[SoundCloud] Apply changes in Extractors 2023-08-12 22:56:28 +02:00
AudricV
a3a74cd566
[SoundCloud] Apply changes in InfoItemExtractors and return track user avatars as uploader avatars in SoundcloudStreamInfoItemExtractor 2023-08-12 22:56:28 +02:00
AudricV
7f818217d2
[SoundCloud] Add utility methods to get images from track JSON objects and image URLs
These new public and static methods, added in SoundcloudParsingHelper,
getAllImagesFromArtworkOrAvatarUrl(String) and
getAllImagesFromVisualUrl(String) (which call a common private method,
getAllImagesFromImageUrlReturned(String, List<ImageSuffix>, List<Image>)),
return an unmodifiable list of JPEG images containing almost every image
resolution provided by SoundCloud except the original size and the tiny
resolution (for artworks and avatars, as the image size is 20x20 for artworks
and 18x18 for avatars, so very close to or equal to the t20x20 resolution):

- for artworks and avatars:
  - mini: 16x16;
  - t20x20: 20x20;
  - small: 32x32;
  - badge: 47x47;
  - t50x50: 50x50;
  - t60x60: 60x60;
  - t67x67: 67x67;
  - large: 100x100;
  - t120x120: 120x120;
  - t200x200: 200x200;
  - t240x240: 240x240;
  - t250x250: 250x250;
  - t300x300: 300x300;
  - t500x500: 500x500.

- for visuals/user banners:
  - t1240x260: 1240x260;
  - t2480x520: 2480x520.

Duplicated code in two methods of SoundcloudParsingHelper
(getUsersFromApi(ChannelInfoItemsCollector, String) and
getStreamsFromApi(StreamInfoItemsCollector, String, boolean)) has been merged
into one common private method, getNextPageUrlFromResponseObject(JsonObject).
2023-08-12 22:56:28 +02:00
AudricV
266cd1f76b
[YouTube] Apply changes in YoutubeMusicSearchExtractor and split its InfoItemExtractors into separate classes
Splitting YoutubeMusicSearchExtractor's InfoItemExtractors into separate
classes (YoutubeMusicSongOrVideoInfoItemExtractor,
YoutubeMusicAlbumOrPlaylistInfoItemExtractor and
YoutubeMusicArtistInfoItemExtractor) allows to simplify
YoutubeMusicSearchExtractor,improves reading and applying changes to InfoItems
(no more losing at least quarter of a line due to indentations).

These InfoItems, in which the image changes have been applied, don't extend the
YouTube ones anymore, as most methods were overridden and the few ones that are
not don't apply in YouTube Music items responses, so it was useless to extend
them.

The code of YoutubeMusicSearchExtractor have been also improved a bit.
2023-08-12 22:56:27 +02:00
AudricV
c1981ed54f
[YouTube] Apply changes in Extractors except YoutubeMusicSearchExtractor
Also improve a bit some code related to the changes.
2023-08-12 22:56:27 +02:00
AudricV
4cc99f9ce1
[YouTube] Apply changes in InfoItemExtractors except YouTube Music ones 2023-08-12 22:56:27 +02:00
AudricV
adfad086ac
[YouTube] Add utility methods to get images from InfoItems and thumbnails arrays
Unmodifiable lists of Images are returned, parsed from a given YouTube
"thumbnails" JSON array.

These methods will be used in all YouTube extractors and InfoItems, as the
structures between content types (videos, channels, playlists, ...) are common.
2023-08-12 22:56:27 +02:00
AudricV
d56b880cae
Replace avatar and thumbnail URLs attributes and methods to List<Image> in Infos 2023-08-12 22:56:26 +02:00
AudricV
9d8098576e
Replace avatar and thumbnail URLs attributes and methods to List<Image> in Extractors 2023-08-12 22:56:26 +02:00
AudricV
0f4a5a8184
Replace avatar and thumbnail URLs attributes and methods to List<Image> in InfoItemsCollectors 2023-08-12 22:56:26 +02:00
AudricV
ca1d4a6fa4
Replace avatar and thumbnail URLs attributes and methods to List<Image> in InfoItemExtractors 2023-08-12 22:56:26 +02:00
AudricV
2f3ee8a3f2
Replace avatar and thumbnail URLs attributes and methods to List<Image> in InfoItems 2023-08-12 22:56:25 +02:00
AudricV
78ce65769f
Add an ImageSuffix class to the extractor
The goal of this utility class is to simply store suffixes which need to be
appended to image URLs, in order to get images at the suffix resolution.

This class contains four properties: the suffix (as a string), the height,
the width (as integers) and the estimated resolution level of the image
corresponding to the one represented by the suffix.
2023-08-12 22:56:25 +02:00
AudricV
d85454186a
Add an Image class to the extractor
Objects of this serializable class contains four properties: a URL (as a
string), a width, a height (represented as integers) and an estimated
resolution level, which can be constructed from a given height.

Possible resolution levels are:
- UNKNOWN: for unknown heights or heights <= 0;
- LOW: for heights > 0 & < 175;
- MEDIUM: for heights >= 175 & < 720;
- HIGH: for heights >= 720.

Getters of these properties are available and the constructor needs these four
properties.
2023-08-12 22:56:25 +02:00
Stypox
7294675aea
Merge pull request #1093 from AudricV/yt_support-shorts-ui-playlists
[YouTube] Support Shorts UI in playlists
2023-08-12 11:11:36 +02:00
Stypox
93a90b816d
Merge pull request #1094 from AudricV/yt_support-more-channel-headers
[YouTube] Support more channel headers
2023-08-12 11:08:30 +02:00
Stypox
44b664af15
[YouTube] Simplify Optional chains in channel 2023-08-12 11:02:51 +02:00
AudricV
2f7bfd3e7f
[YouTube] Add mocks of interactiveTabbedHeaderRenderer channel header test 2023-08-08 19:12:29 +02:00
AudricV
b147904571
[YouTube] Add test for interactiveTabbedHeaderRenderer channel header
This test uses the Minecraft game topic channel.
2023-08-08 19:12:28 +02:00
AudricV
1852031a0b
[YouTube] Support pageHeaderRenderer and interactiveTabbedHeaderRenderer channel headers
The addition of this support required to turn the isCarouselHeader boolean into
an enum containing all supported channel headers named HeaderType.

Also assert that the page has been fetched where needed to avoid
NullPointerExceptions when the channel page has been not fetched and remove the
getChannelHeaderJson method in YoutubeChannelExtractor, method for which its
code has been moved to its sole usage after the new headers support changes.
2023-08-08 19:12:27 +02:00
AudricV
698c710685
Do not require knowledge of uploader in default StreamInfoItems tests
This change is required as some services can return no uploader info, such as
YouTube for playlists with a Shorts UI.
2023-08-07 19:43:15 +02:00
AudricV
8237052ef5
Fix wrong assertion in assertNotEmpty
The non-null assertion was made on the exception message instead of the string
to check, causing a NullPointerException if the string to check was null.
2023-08-07 19:43:09 +02:00
AudricV
162c261577
[YouTube] Add mocks of the playlist with Shorts UI test 2023-08-07 19:07:53 +02:00
AudricV
e2f4ee47b9
[YouTube] Add a playlist with Shorts UI test
The system Shorts videos uploads playlist of the YouTube official channel has
been chosen for this test.
2023-08-07 19:06:09 +02:00
AudricV
e6f371fb94
[YouTube] Support Shorts UI in playlists
Also remove an outdated A/B test comment.
2023-08-07 19:01:08 +02:00
Stypox
7936987955
Merge pull request #1092 from Stypox/channel-tabs-improvements
Channel tabs code improvements
2023-08-06 21:44:10 +02:00
Stypox
6d2227111f
[YouTube] Assert that videos tab is ready after channel fetching 2023-08-06 21:14:57 +02:00
Stypox
ee625c325c
Inherit from DefaultListExtractorTest in channel tab tests 2023-08-06 21:14:56 +02:00
Stypox
276c293889
Rename assertTabsContain 2023-08-06 21:14:56 +02:00
Stypox
9d3761a371
[YouTube] Directly use playlist collector in channel tabs wrapper
Note that this introduces a "Raw use of parameterized class 'InfoItemsPage'" warning, but it can be ignored since the type missing would be <InfoItem>, and StreamInfoItem extends InfoItem
2023-08-06 21:13:25 +02:00
Stypox
95a3cc0a17
Merge pull request #1083 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.10.0
Bump org.junit:junit-bom from 5.9.3 to 5.10.0
2023-08-06 18:58:27 +02:00
Stypox
e34b4f1978
[YouTube] Avoid using Consumer 2023-08-06 13:02:31 +02:00
Stypox
ef67c7cd74
[YouTube] Simplify usage of channel header json 2023-08-06 13:02:31 +02:00
Stypox
a104cf3227
[YouTube] Fix docs in channel helper 2023-08-06 13:02:31 +02:00
Stypox
bb47f05f89
Merge pull request #1091 from Stypox/update-mocks
[YouTube] Update stream mocks after #1087
2023-08-06 12:58:24 +02:00
Stypox
468bcc045d
[YouTube] Update mocks after #1087 2023-08-06 12:33:04 +02:00
Stypox
35f3a4ad01
Merge pull request #1082 from AudricV/channel-tabs-and-tags-support
Add support for channel tabs and channel tags
2023-08-06 12:21:42 +02:00
AudricV
e7d64099a7
[YouTube] Update channel mocks and add channel tabs mocks 2023-08-06 12:15:06 +02:00
AudricV
684101c47d
[YouTube] Implement age-restricted channels support, link handlers and channels tabs and tags changes on tests
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV
eaf2600ce0
[SoundCloud] Implement link handlers and channels tabs and tags changes on tests
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV
0ee2072de5
[PeerTube] Implement link handlers and channels tabs and tags changes on tests
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV
d3801dd0e9
[MediaCCC] Implement link handlers and channels tabs and tags changes on tests
Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV
8baec04611
[Bandcamp] Implement link handlers and channels tabs and tags changes on tests
Tests in BandcampChannelExtractorTest and BandcampChannelLinkHandlerFactoryTest
have been also fixed.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV
e0ba29cd19
Add utility method to assert that given channel tabs are in the ones returned by a channel extractor
Only the first content filter of the ListLinkHandler instances provided is
used when collecting all channel tabs of the ListLinkHandler list, as channel
tabs implementations only use one content filter per ListLinkHandler instance.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
AudricV
18846baba7
Add tabs and tags methods in tests interfaces and annotate all methods with the Test JUnit annotation
These changes should help to detect tests as tests, when running a subset of
tests or all tests.
They should be also implemented in these interfaces' implementations (new and
existing ones).

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:15:06 +02:00
ThetaDev
c70a0e3543
Add a test for textual durations parsing using TimeAgoParser's patterns 2023-08-06 12:15:06 +02:00
AudricV
7366eab156
[YouTube] Add support for channel tabs and tags and age-restricted channels
Support of tags and videos, shorts, live, playlists and channels tabs has been
added for non-age restricted channels.

Age-restricted channels are now also supported and always returned the videos,
shorts and live tabs, accessible using system playlists. These tabs are the
only ones which can be accessed using YouTube's desktop website without being
logged-in.

The videos channel tab parameter has been updated to the one used by the
desktop website and when a channel extraction is fetched, this tab is returned
in the list of tabs as a cached one in the corresponding link handler.

Visitor data support per request has been added, as a valid visitor data is
required to fetch continuations with contents on the shorts tab. It is only
used in this case to enhance privacy.

A dedicated shorts UI elements (reelItemRenderers) extractor has been added,
YoutubeReelInfoItemExtractor. These elements do not provide the exact view
count, any uploader info (name, URL, avatar, verified status) and the upload
date.

All service's LinkHandlers are now using the singleton pattern and some code
has been also improved on the files changed.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:15:04 +02:00
AudricV
4586067934
Add utility method to parse textual durations using TimeAgoParser's patterns
This is required to parse duration of YouTube's reelItemRenderers, returned
only inside accessibility data.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:13:33 +02:00
AudricV
d4bfe791ee
[SoundCloud] Add tabs support for users
Support of tracks, playlists and albums has been added for users.

Also add the declaration of the UnsupportedOperationException exception to the
service's LinkHandlers.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:13:32 +02:00
AudricV
6f7d1f079f
[Bandcamp] Add tabs support for artists
Support of tracks and albums has been added for artists.

Also use the singleton pattern and add the declaration of the
UnsupportedOperationException exception to the service's LinkHandlers and
improved some code in the files changed.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:12:19 +02:00
AudricV
1e8474b22d
[PeerTube] Add tabs support for accounts and video channels
Support of channels and videos has been added for accounts and support of
videos and playlists has been added for video channels.

The following changes have been also done:
- collectStreamsFrom method in PeertubeParsingHelper has been renamed to
collectItemsFrom;
- PeertubeChannelInfoItemExtractor.getStreamCount method has been fixed due to
ChannelExtractor's new inheritance;
- the declaration of the UnsupportedOperationException exception thrown has
been added to the service's LinkHandlers;
- a channel tab LinkHandlerFactory has been added,
PeertubeChannelTabLinkHandlerFactory;
- all service's LinkHandlers are now using properly the singleton pattern.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:12:15 +02:00
AudricV
652c2c8408
Add a ListLinkHandler which can be used to be returned from ChannelInfo.getTabs() when a specific tab's data has already been fetched
This new ListLinkHandler, ReadyChannelTabListLinkHandler, should help saving
clients data, energy and time by helping to reduce duplicate requests.

Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:11:12 +02:00
AudricV
de823a6b68
Add an UnsupportedTabException exception class
This class makes easier for LinkHandlerFactory implementations to declare an
UnsupportedOperationException.
2023-08-06 12:11:12 +02:00
AudricV
76fb9dcdd7
Add UnsupportedOperationException to exceptions which can be thrown by getId and getUrl methods of LinkHandlerFactory and its base implementations
This change advertise to clients that channel tabs' link handler factories can
return an UnsupportedOperationException when a tab provided to them is
unsupported.
2023-08-06 12:11:12 +02:00
AudricV
946eb9bd91
Add structure of channel tags
Tags' getters and/or setters have been added in ChannelExtractor and
ChannelInfo to do so.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
2023-08-06 12:11:12 +02:00
AudricV
356a888d6c
Add structure of channel tabs
This commit introduces the following breaking changes:

- Three new classes have been added:
  - ChannelTabExtractor, class extending ListExtractor<InfoItem>, which
  extracts InfoItems from a channel tab;
  - ChannelTabInfo extending ListInfo<InfoItem>, which extracts InfoItems from
  a ChannelTabExtractor and returns them as a ChannelTabInfo;
  - ChannelTabs, an immutable class containing all supported channel tabs.
- StreamingService implementations must implement new methods returning a
channel tab LinkHandlerFactory (getChannelTabsLHFactory) and a
ChannelTabExtractor (getChannelTabExtractor);
- ChannelExtractor inherits Extractor instead of ListExtractor<StreamInfoItem>
and ChannelInfo inherits Info instead of ListInfo<StreamInfoItem>;
- ChannelExtractor and ChannelInfo have now getters and/or setters of tabs.

Co-authored-by: ThetaDev <t.testboy@gmail.com>
Co-authored-by: Stypox <stypox@pm.me>
2023-08-06 12:11:11 +02:00
Stypox
3faaf4301c
Merge pull request #1087 from AudricV/yt_js-extractor-improvements-and-fixes
[YouTube] Improve and fix YoutubeJavaScriptExtractor
2023-08-06 12:01:00 +02:00
Stypox
8fb6ba36fa
Merge pull request #1081 from TeamNewPipe/fix/sc/search-next-page
[SoundCloud] Detect whether there are any more search results
2023-08-06 11:49:35 +02:00
Stypox
2947257111
[SoundCloud] Properly calculate if results have finished 2023-08-06 11:38:22 +02:00
Stypox
485bfbca9d
[SoundCloud] Move try-catch inside getOffsetFromUrl 2023-08-06 11:35:37 +02:00
Stypox
7c70fef197
Merge pull request #1089 from TeamNewPipe/ccc
[media.ccc.de] Only extract kiosk live stream rooms if they are streaming
2023-08-06 10:12:04 +02:00
Stypox
de0a9bb797
Merge pull request #1088 from FireMasterK/pseudo-rng
Replace cryptographically secure random with regular random
2023-08-05 11:37:49 +02:00
TobiGr
340095515d Make Kiosk IDs accessible if possible 2023-08-05 03:18:40 +02:00
TobiGr
fe27d6a0ec [media.ccc.de] Only extract live streams if the conference is streaming 2023-08-05 01:53:43 +02:00
Kavin
25082d78b0
Replace SecureRandom with Random 2023-08-03 23:00:02 +01:00
TobiGr
aa6c17dc77 [SoundCloud] Deduplicate some code 2023-08-03 14:41:30 +02:00
TobiGr
2fb9922a15 [SoundCloud] Detect whether there are any more search results
Add test for this edge case.
2023-08-03 14:37:13 +02:00
AudricV
a3d160edab
[YouTube] Improve and fix YoutubeJavaScriptExtractor
- Enhance documentation;
- Fix the regular expression fallback on HTML embed watch page;
- Use HTML scripts tag search first instead of the regular expression approach,
now used as a last resort;
- Compile regular expressions only once, in order to improve the performance of
subsequent extraction calls when clearing the cache;
- Provide original exceptions when fetching or parsing pages on which the base
JavaScript's player could be found failed, allowing clients to detect network
errors when they are the cause of the failures for instance;
- Remove delegate method which was not taking a video ID and hardcoding one, as
we can provide the video ID in all cases or do not provide a video ID at worse;
- Rename and make extraction methods package-private, as they are not intended
to be used publicly.

These breaking internal changes have been applied where needed, in
YoutubeJavaScriptExtractorTest and YoutubeStreamExtractor (in which an unneeded
initStsFromPlayerJsIfNeeded call have been removed).
2023-08-02 23:05:08 +02:00
Stypox
5492343b8e
Pre-release v0.22.7 2023-08-02 20:02:29 +02:00
Tobi
9a59afbcf5
Merge pull request #1086 from AudricV/yt_no-exception-channels-without-banner
[YouTube] Don't throw an exception when there is no banner available on a channel
2023-08-02 18:56:26 +02:00
AudricV
bb1ab166bf
[YouTube] Test that no banner is returned for carouselHeaderRenders 2023-08-01 22:19:43 +02:00
AudricV
f1fa84b4e3
[YouTube] Don't throw an exception when there is no banner available on a channel
Channels may not have a banner, so no exception should be thrown if no banner
is found.
2023-08-01 12:40:20 +02:00
Tobi
39a911db9f
Merge pull request #1084 from AudricV/yt_android-403s-workaround-and-streams-tests-fixes
[YouTube] Workaround again 403 HTTP issues on the ANDROID InnerTube client and fix stream tests
2023-07-31 23:51:10 +02:00
dependabot[bot]
bda3a3fc5d
Bump org.junit:junit-bom from 5.9.3 to 5.10.0
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.9.3 to 5.10.0.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.9.3...r5.10.0)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-07-24 09:53:05 +00:00
AudricV
522c78160f
[YouTube] Update stream tests mocks 2023-07-23 19:36:28 +02:00
AudricV
7528eb2bd9
[YouTube] Fix stream tests failures
- Fix testCheckAudioStreams test of
YoutubeStreamExtractorDefaultTest.AudioTrackLanguage test class, by updating
the excepted audio track name test to use the updated English audio track name
(audio track type info has been added on the video tested);
- Fix YoutubeStreamExtractorDefaultTest.PublicBroadcasterTest test class by
using a different video from a French and German public broadcast channel, as
the channel Dinge Erklärt – Kurzgesagt is not affiliated with a public
broadcast channel anymore;
- Fix YoutubeStreamExtractorLivestreamTest test class, by updating the excepted
name of the livestream to the current one.
2023-07-23 19:19:02 +02:00
AudricV
164c8e3abb
[YouTube] Workaround again 403 HTTP issues on the Android client by using new player parameters
These parameters are the only ones currently known to bypass 403 HTTP issues
related to failure of passing Android client integrity checks, as the ones of
stories (and the base of the shorts ones) do not work anymore, which may be
related to end of this format on the service.
2023-07-22 20:22:16 +02:00
Tobi
8ebbe1b7e1
Merge pull request #706 from FireMasterK/av1-itags
Add support for AV1 itags.
2023-07-22 13:31:45 +02:00
FireMasterK
6db0d116fe Add support for AV1 itags. 2023-07-22 13:23:44 +02:00
Tobi
986a3f7747
Merge pull request #1080 from TeamNewPipe/proguard-rules
Add proguard rules to README
2023-07-07 12:52:10 +02:00
Tobi
d6132a9ed9
Merge pull request #1076 from AudricV/yt-fix-throttling-parameter-regex
[YouTube] Support multiple declarations for throttling parameter function name array
2023-07-07 12:51:26 +02:00
Tobi
87eb29a25d Add proguard rules to README 2023-07-05 08:25:23 +02:00
AudricV
4e22c5ee87
[YouTube] Support multiple declarations for throttling parameter function name array
Also moved the corresponding regex parts in static constants for easier future
modifications
2023-06-26 15:25:53 +02:00
Kavin
d961d349c3
[YouTube] Check whether player responses are valid for all InnerTube clients used (#1070)
Co-authored-by: Audric V <74829229+AudricV@users.noreply.github.com>
2023-06-18 21:54:52 +02:00
ThetaDev
ad97f08048
[YouTube] Fix parsing short relative date formats (English only) (#1068) 2023-06-18 21:41:29 +02:00
Tobi
d294ccb433
Merge pull request #1071 from TeamNewPipe/feat/ServiceList
Init services at the correct place
2023-06-17 20:50:24 +02:00
Tobi
5d112aa772
Merge pull request #1074 from TeamNewPipe/imp/media-formats
Add more audio media formats and MediaFormat.getAllForMimeType(mimeType)
2023-06-17 20:49:38 +02:00
TobiGr
0e01d90562 Do not init services while creating services array 2023-06-17 18:08:09 +02:00
TobiGr
5809904cf7 Add annotations to MediaFormat 2023-06-17 18:01:54 +02:00
TobiGr
53dfd871e2 Add more audio media formats and MediaFormat.getAllForMimeType(mimeType) 2023-06-17 18:01:29 +02:00
Stypox
5eb886243f
Merge pull request #1072 from TeamNewPipe/fix/bandcamp_test
Fix BandcampRadioStreamExtractorTest.testGetAudioStreams()
2023-06-09 17:35:35 +02:00
TobiGr
19ce06fe00 Fix BandcampRadioStreamExtractorTest.testGetAudioStreams() 2023-06-06 23:07:56 +02:00
Audric V
533121fb81
Merge pull request #1045 from Theta-Dev/fix/trending-video-tab
[YouTube] Extract trends from A/B tested "Videos" tab and fix extraction of trends name from A/B tested new title design
2023-05-19 11:22:49 +02:00
Audric V
92a0024424
Merge pull request #1052 from TeamNewPipe/peertube/fix/nested-comment-replies
[PeerTube] Fix multi level comment replies
2023-05-18 18:49:06 +02:00
Tobi
3c036a9c03
Merge pull request #1061 from ChunkyProgrammer/develop/extract-yt-playlist-descrption
Extract playlists description
2023-05-17 22:43:17 +02:00
TobiGr
c70bb83801 [Bandcamp] Implement PlaylistExtractor.getDescsription() 2023-05-15 15:23:03 +02:00
TobiGr
ca0ce00753 Add PlaylistInfoItem.getDescription() and PlaylistInfoItemExtractor.getDescription()
[PeerTube] Implement the corresponding extractor method.
TODO: add tests
2023-05-12 01:43:59 +02:00
TobiGr
b218bf69bd Implement PlaylistInfo.getDescription()
Implement PlaylistExtractor.getDescription() for PeerTube and SoundCloud.
Anotate   PlaylistExtractor.getDescription() as Nonnull
2023-05-12 00:44:10 +02:00
chunky programmer
81f29116ba switch from string to Description object 2023-05-11 00:36:57 -04:00
chunky programmer
e147867d41 Add tests 2023-05-11 00:00:34 -04:00
chunky programmer
5ab6cd7420 Extract YouTube playlist description 2023-05-11 00:00:22 -04:00
TobiGr
d358ba1c41 Improve PeertubeCommentsInfoItemExtractor constructor 2023-05-07 22:55:26 +02:00
TobiGr
aff3e795f8 [PeerTube] Fix multi level comment replies 2023-05-07 22:49:14 +02:00
ThetaDev
3673d4ae01
fix: YouTube trending name extraction 2023-05-03 21:16:35 +02:00
ThetaDev
0addb98cd7
tests: update mocks 2023-05-03 21:16:35 +02:00
ThetaDev
24eba62305
fix: extract YouTube trends from new "Videos" tab 2023-05-03 21:16:23 +02:00
Kavin
a9ca5c49e4
Merge pull request #1056 from AudricV/yt-improve-search-suggestions-extraction
[YouTube] Switch to new search suggestion domain and improve error handling
2023-05-02 20:17:48 +01:00
Stypox
00408db959
Merge pull request #1058 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.16.1
Bump org.jsoup:jsoup from 1.15.4 to 1.16.1
2023-05-01 19:59:02 +02:00
dependabot[bot]
108f8a7a17
Bump org.jsoup:jsoup from 1.15.4 to 1.16.1
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup) from 1.15.4 to 1.16.1.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.4...jsoup-1.16.1)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-05-01 09:58:32 +00:00
Tobi
11565db17f
Merge pull request #1055 from AudricV/peertube-no-description-channels
[PeerTube] Don't return "No description" when there is no description for a channel or an account
2023-04-30 23:38:10 +02:00
AudricV
cf6040ddb3
Update YoutubeSuggestionExtractorTest mocks 2023-04-30 19:53:44 +02:00
AudricV
593122342f
[YouTube] Improve YoutubeSuggestionExtractorTest
- Remove useless concatenation on the downloader path;
- Remove unneeded public test modifier;
- Update license header;
- Specify the service class tested instead of the generic class.
2023-04-30 19:53:43 +02:00
AudricV
e923fca440
[YouTube] Switch to new search suggestion domain and improve error handling
- Switch to the new domain used by YouTube for search suggestions,
suggestqueries-clients6.youtube.com, and add the xhr query parameter with the
t value, to allow getting responses without requiring trim;
- Use the Java 8 Stream API to collect search suggestions and improve invalid
response detection by checking whether the content type of the response
returned is JSON;
- Move the licence header at the top of the file.
2023-04-30 19:53:42 +02:00
AudricV
945165a3c0
[PeerTube] Don't return "No description" when there is no description for a channel or an account
When a description is missing, no description should be returned, even the ones
indicating there is no description. This behavior is represented by a null
return instead.

Also update PeertubeAccountExtractorTest to reflect these changes.
2023-04-30 18:41:38 +02:00
Stypox
d61dc27406
Merge pull request #1054 from TeamNewPipe/fix-ci-variable
Make sure CI is not vulnerable to bash word splitting
2023-04-27 18:52:07 +02:00
Stypox
51cba5a287
Make sure CI is not vulnerable to bash word splitting
See https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-an-intermediate-environment-variable
2023-04-26 18:36:25 +02:00
Stypox
f819c2fe34
Merge pull request #1053 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.3
Bump org.junit:junit-bom from 5.9.2 to 5.9.3
2023-04-26 15:42:19 +02:00
dependabot[bot]
4b92c5a34d
Bump org.junit:junit-bom from 5.9.2 to 5.9.3
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.9.2 to 5.9.3.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.9.2...r5.9.3)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-04-26 09:57:57 +00:00
Stypox
2deb023da4
Merge pull request #1050 from Theta-Dev/fix/channel-carousel-header
[YouTube] Add support for CarouselHeaderRenderer
2023-04-25 15:17:31 +02:00
ThetaDev
4aada7f91b refactor: rename carousel header channel test 2023-04-21 22:48:37 +02:00
Tobi
b2b426048f
Merge pull request #1051 from Theta-Dev/fix/ytm-client-version-rg
[YouTube] fix: set musicClientVersion regex capture group
2023-04-16 23:48:19 +02:00
ThetaDev
47aa9fed40 fix: set musicClientVersion regex capture group 2023-04-16 19:25:05 +02:00
ThetaDev
20370395c5 fix: add support for CarouselHeaderRenderer 2023-04-16 17:40:13 +02:00
Stypox
7dba6e3891
Merge pull request #1033 from petlyh/bandcamp-paywalled-content
[Bandcamp] Handle paywalled tracks
2023-04-12 13:04:26 +02:00
petlyh
e6aad117e7
[Bandcamp] Throw PaidContentException on paywalled albums 2023-04-03 19:27:09 +02:00
TobiGr
8495ad619e Bump version to 0.22.6 2023-04-02 22:35:02 +02:00
fynngodau
69705138e4
[Bandcamp] Fix extraction of related playlist items URL (#1047)
Small change in HTML structure
2023-04-02 22:24:29 +02:00
Björn Sigurbergsson
1b6fe5edd6
[YouTube] Fix ParsingException when comments are unavailable in a video (#1040)
Co-authored-by: bjs <bjs@elect-it.com>
Co-authored-by: Audric V. <74829229+AudricV@users.noreply.github.com>
Co-authored-by: Kavin <20838718+FireMasterK@users.noreply.github.com>
2023-03-30 19:58:06 +02:00
ThetaDev
8d1303e18f
Add track types to audio streams (#1041) 2023-03-28 00:02:20 +02:00
Tobi
44ae139d33
Merge pull request #1034 from AudricV/peertube-fix-commentsinfo-test
[PeerTube] Fix comments info test
2023-03-20 19:30:35 +01:00
Tobi
d5e9df5381
Merge pull request #1038 from TeamNewPipe/docs-jdk-11
Update JDK to 11 in JDoc workflow
2023-03-20 19:25:21 +01:00
Tobi
0486ad7a08
Update JDK to 11 in JDoc workflow 2023-03-06 18:45:11 +01:00
AudricV
80a6fc2c63
[PeerTube] Fix testGetCommentsFromCommentsInfo test of PeertubeCommentsExtractorTest.Default
The tested comment has been removed, so it couldn't be found in the comments
list.

This comment has been replaced by a new one from the current comments of the
video.

Also, in the parent class PeertubeCommentsExtractorTest, final has been used as
much as possible and for-each loops of lists have been replaced by their
forEach method or the Stream API, in order to simplify code.
2023-03-04 16:49:10 +01:00
petlyh
5a9b6ed2e3
[Bandcamp] Support loading additional comments (#1030) 2023-03-04 14:01:06 +01:00
Stypox
6bdd698c25
Merge pull request #1026 from AudricV/audio-streams-descriptive-and-locale-properties
Add descriptive and locale properties to audio streams
2023-03-01 11:15:46 +01:00
Stypox
19e4b216c9
Merge pull request #1032 from AudricV/yt_fix-comments-hashtags-links-extraction
[YouTube] Fix hashtags links extraction and escape HTML links
2023-03-01 10:47:37 +01:00
Stypox
b1298490c0
Merge pull request #1029 from AudricV/yt_fix-no-views-extraction-playlist-items
[YouTube] Fix partial non-extraction of "No views" string in stream items
2023-03-01 10:46:52 +01:00
petlyh
9dc1832733
[Bandcamp] Handle paywalled tracks 2023-02-28 17:51:30 +01:00
fynngodau
3fdb6ee476
Merge pull request #1031 from petlyh/bandcamp-fix-radio-comments
[Bandcamp] Show comments as disabled on radio streams
2023-02-27 20:50:12 +01:00
AudricV
bd79b921e8
[YouTube] Refactor the code to get stream items' view count
This refactoring avoids code duplication as much as possible.
2023-02-27 10:25:46 +01:00
AudricV
51f9b39953
[YouTube] Fix partial non-extraction of no views string in stream items
As the "No views" string is returned in the case there is no view on a video, a
number cannot be parsed in this case, so -1 was returned.

This string is now detected in all methods to get the view count of a stream.
2023-02-27 10:18:45 +01:00
AudricV
95b3f5e391
[MediaCCC] Test audio language property extraction 2023-02-26 19:06:18 +01:00
AudricV
30a0f8c510
[MediaCCC] Extract audio language property for single language audio tracks 2023-02-26 19:06:18 +01:00
AudricV
7f0269c4c7
[YouTube] Edit YoutubeStreamExtractorDefaultTest.AudioTrackLanguage to test audio locale property
The Hindi audio track language presence test has been changed from audio track
label to audio locale.
2023-02-26 19:06:18 +01:00
AudricV
034f82dae7
[YouTube] Test language and descriptive audio in YoutubeDashManifestCreatorsTest 2023-02-26 19:06:17 +01:00
AudricV
05e8cb39f7
[YouTube] Add language and descriptive audio properties to DASH manifests 2023-02-26 19:06:17 +01:00
AudricV
bf30d70152
[YouTube] Add descriptive audio test
This test uses video TjxC-evzxdk.

Also improve a bit YoutubeStreamExtractorDefaultTest.AudioTrackLanguage test.
2023-02-26 19:06:17 +01:00
AudricV
76b7c19c5d
[YouTube] Extract whether a track is a descriptive audio and audio locale when available
Also use audio track setters only for audio itags.
2023-02-26 19:06:17 +01:00
AudricV
3bb5eeef30
[YouTube] Add descriptive and locale audio support in ItagItem 2023-02-26 19:06:16 +01:00
AudricV
14bf3fb05b
Add ability to know the locale of an audio stream
Getting audio tracks locales by parsing their ID or their label, should not be
done by clients, but by the extractor.

This commit adds the ability to store the Locale of an AudioStream, which is
used to compare similar AudioStreams (in the equalStats method).
2023-02-26 19:06:16 +01:00
AudricV
f92426560c
Add descriptive audio properties
Also improve AudioStream's audio language documentation
2023-02-26 19:06:16 +01:00
AudricV
a63f289667
[YouTube] Update mocks of YoutubeCommentsExtractorTest.FormattingTest 2023-02-26 18:50:07 +01:00
AudricV
9483dcd9fa
[YouTube] Update mocks of YoutubeCommentsExtractorTest.RepliesTest 2023-02-26 18:43:36 +01:00
AudricV
1556adbb2d
[YouTube] Fix hashtags links extraction and escape text in attribute descriptions + HTML links
webCommandMetadata object is contained inside a commandMetadata one, so it is
not accessible from the root of the navigationEndpoint object.

The corresponding statement has been moved at the bottom of the specific
endpoints parsing, as the webCommandMetadata object is present almost
everywhere, otherwise URLs of some endpoints would have be changed, such as
uploader URLs (from channel IDs to handles).

As no ParsingException is now thrown by getUrlFromNavigationEndpoint, and so by
getTextFromObject, getUrlFromObject and getTextAtKey, the methods which were
catching ParsingExceptions thrown by these methods had to be updated.

URLs got in the HTML version of getTextFromObject are now escaped properly to
provide valid HTML to clients. This has been also done for attribute
descriptions, with the description text for this type of descriptions.

As YouTube descriptions are in HTML format (except for the fallback on the JSON
player response, which is plain text and only happens when there is no visual
metadata or a breaking change), all URLs returned are escaped, so tests which
are testing presence of URLs with escaped characters had to be updated (it was
only the case for YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing).
2023-02-26 18:43:36 +01:00
Stypox
99ab9777ad
Release 0.22.5 2023-02-26 16:33:25 +01:00
Stypox
39c2e1fda0
Merge pull request #1028 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.4
Bump org.jsoup:jsoup from 1.15.3 to 1.15.4
2023-02-26 16:27:31 +01:00
petlyh
f7a7a236fb
[Bandcamp] Show comments as disabled on radio streams 2023-02-23 18:42:43 +01:00
dependabot[bot]
f5599ff08d
Bump org.jsoup:jsoup from 1.15.3 to 1.15.4
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup) from 1.15.3 to 1.15.4.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.3...jsoup-1.15.4)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-20 10:05:26 +00:00
Stypox
7e793c11ae
Empty commit to make JitPack happy
The build for e920ab3f5f40fe1f669735d717135c83141be2dd succeeded, but for some reason it is considered as failed...
2023-02-08 22:35:26 +01:00
Stypox
e920ab3f5f
Merge pull request #1021 from TeamNewPipe/fix/yt-comment-text-hashtag
[YouTube] Fix getting the comment text if the comment contains a hashtag
2023-01-29 21:47:59 +01:00
TobiGr
3f7df9536e [YouTube] Fix getting the comment text if the comment contains a hashtag 2023-01-29 20:33:51 +01:00
Stypox
999fb7f812
Merge pull request #1024 from AudricV/snd_fix-tracks-like-count
[SoundCloud] Fix extraction of tracks like count
2023-01-29 10:52:54 +01:00
Stypox
3519d4c367
Merge pull request #1015 from AudricV/yt_fix-channel-id-rss-feeds
[YouTube] Fix channel ID extraction of YouTube channels RSS feeds
2023-01-29 10:41:38 +01:00
Stypox
9aca710e86
Merge pull request #1013 from Stypox/fix-music-mixes
[YouTube] Now music mixes can be treated as normal mixes
2023-01-29 09:48:51 +01:00
Stypox
76eeabac45
Merge pull request #1020 from TeamNewPipe/fix/yt-subscriber-count
[YouTube] Fix NPE in search when getting channel items without subscriber count
2023-01-29 09:44:22 +01:00
AudricV
676622f6df
[SoundCloud] Fix expectedLikeCountAtLeast tests of SoundcloudStreamExtractorTest test classes
As like count is now returned by the extractor, we need to assert a positive
minimum like count, which is close to the actual value, in order to avoid test
failures due to lower like counts than the ones excepted.
2023-01-29 01:08:02 +01:00
AudricV
2a24d407d5
[SoundCloud] Fix extraction of tracks like count
SoundCloud is using likes_count to return the like count of a track, like it
was the case before they switched to favoritings_count.
2023-01-29 01:00:49 +01:00
Tobi
0e4e6a9bac
Merge pull request #1022 from AudricV/yt_support-live-urls
[YouTube] Support live URLs
2023-01-28 21:52:26 +01:00
AudricV
ba24976e41
[YouTube] Add live URLs test and do minor improvements to YoutubeStreamLinkHandlerFactoryTest
- Remove unused imports;
- Replace wildcard imports by single class imports;
- Suppress "HTTP links are not secured" warnings from IDEA IDEs;
- Replace removed video jZViOEv90dI by an existing video, 9Dpqou5cI08 (the
corresponding test has been of course renamed).
2023-01-28 19:36:21 +01:00
AudricV
57f850bc2d
[YouTube] Support live URLs and do minor improvements to YoutubeStreamLinkHandlerFactory
- Move license header at the top;
- Use an unmodifiable set for the subpaths instead of a modifiable list;
- Add missing Nonnull and Nullable annotations;
- Improve exception messages.
2023-01-28 19:36:20 +01:00
AudricV
1f4ed9dce9
[YouTube] Fix channel ID extraction of YouTube channel RSS feeds
The yt:channelId element doesn't provide the channel ID anymore and is empty,
like the id element, so we need now to extract it from the channel URL provided
in two elements: author -> uri and feed -> link.

Also avoid a NullPointerException in getUrl and getName methods.
2023-01-28 11:53:33 +01:00
Tobi
c589a2c1a2
Merge pull request #1014 from TeamNewPipe/fix/yt-comments
[YouTube] Fix getting next comments pages
2023-01-27 11:14:55 +01:00
Tobi
c2b36fd405
Merge pull request #1016 from lonewolf2208/bug-fix
[YouTube] Remove topStandaloneBadge check for view count of stream items
2023-01-25 08:30:36 +01:00
TobiGr
72573932cf [YouTube] Fix NPE in search when getting channel items without subscriber count 2023-01-24 23:03:45 +01:00
TobiGr
f50b7275af [YouTube] Fix getting next comments pages 2023-01-24 22:39:08 +01:00
Kunal
9bdad40b06 Removed topStandaloneBadge 2023-01-20 02:41:21 +05:30
Stypox
5945057227
[YouTube] Add music mix test 2023-01-15 23:30:30 +01:00
Stypox
7293991832
[YouTube] Now music mixes can be treated as normal mixes
Using a playlist extractor on them would result in "Unviewable playlist" errors
2023-01-15 23:28:59 +01:00
Stypox
ff94e9f30b
Merge pull request #1009 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.10.1
Bump gson from 2.10 to 2.10.1
2023-01-11 15:35:36 +01:00
Stypox
c1040bccac
Merge pull request #794 from FireMasterK/comments-count
[YouTube] Add support to extract total comment count
2023-01-11 15:32:19 +01:00
Stypox
6ccc43e57e
Merge pull request #1010 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.2
Bump junit-bom from 5.9.1 to 5.9.2
2023-01-11 13:19:33 +01:00
dependabot[bot]
1bc44cb949
Bump junit-bom from 5.9.1 to 5.9.2
Bumps [junit-bom](https://github.com/junit-team/junit5) from 5.9.1 to 5.9.2.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.9.1...r5.9.2)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-11 09:04:12 +00:00
dependabot[bot]
f43049985e
Bump gson from 2.10 to 2.10.1
Bumps [gson](https://github.com/google/gson) from 2.10 to 2.10.1.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.10...gson-parent-2.10.1)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-09 09:05:55 +00:00
Stypox
33f9f2cf66
Merge pull request #1006 from TeamNewPipe/yt/fix-html-escape
[YouTube] Fix escaping links in YouTubeParsingHelper.getTextFromObject
2023-01-05 19:02:23 +01:00
TobiGr
56aab4d971 [YouTube] Fix escaping links in YouTubeParsingHelper.getTextFromObject 2023-01-05 00:28:12 +01:00
Kavin
22a47da8c7
Fix requested change and remove outdated comment. 2023-01-02 20:42:32 +00:00
Kavin
98a90fd9c8
Don't cache comments count and return early on page fetch if no token. 2023-01-02 20:40:48 +00:00
Kavin
2974dfaa48
Only store ajaxJson for initial page and eager fetch the initial continuation. 2023-01-02 20:40:48 +00:00
Kavin
64d24aa09e
Fix request changes. 2023-01-02 20:40:48 +00:00
Kavin
67ef4f4c30
Cleanup and remove optional. 2023-01-02 20:40:48 +00:00
FireMasterK
22f71b010c
Fix for requested changes. 2023-01-02 20:40:48 +00:00
FireMasterK
656b7c1cd9
Improve method documentation. 2023-01-02 20:40:48 +00:00
FireMasterK
981aee4092
Add support to extract total comment count. 2023-01-02 20:40:48 +00:00
Stypox
45636b0d00
Merge pull request #986 from Isira-Seneviratne/Static_maps
Use immutable Map factory methods.
2023-01-02 18:11:14 +01:00
Stypox
219c5c5be5
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java 2023-01-02 18:11:03 +01:00
Stypox
259de3cba6
Merge pull request #995 from TeamNewPipe/feat/soundcloud-playlistinfoitemextractor
[SoundCloud] Implement getUploaderUrl() and isUploaderVerified() for PlaylistInfoItemExtractor
2023-01-02 15:10:40 +01:00
Stypox
991394b53a
Merge pull request #1005 from FireMasterK/fix-escaping-xss
Fix for potential XSS attacks and formatting issues
2023-01-02 15:06:17 +01:00
Stypox
ce15f7cc50
Merge pull request #996 from TeamNewPipe/feat/peeertube-playlists
[PeerTube] Support searching for playlists and channels
2023-01-02 15:04:09 +01:00
Isira Seneviratne
d8ce08d969 Use immutable Map factory methods. 2023-01-02 07:50:31 +05:30
Kavin
01acf79436
Fix for potential XSS attacks. 2022-12-31 20:05:32 +00:00
TobiGr
292e0d8ce7 [SoundCloud] Implement getUploaderUrl() and isUploaderVerified() for PlaylistInfoItemExtractor 2022-12-31 18:46:39 +01:00
TobiGr
2a8729aeb2 Apply suggestions
Co-authored-by: Stypox <stypox@pm.me>
2022-12-31 18:24:33 +01:00
TobiGr
d75a997611 [PeerTube] Support searching for channels 2022-12-31 18:24:33 +01:00
TobiGr
dea6d8ce4c [PeerTube] Support searching for playlists 2022-12-31 18:24:33 +01:00
Stypox
95cc6aefbb
Merge pull request #994 from TeamNewPipe/fix/peertube-subtitles-exception
[PeerTube] Report Exceptions thrown while getting a stream's subtitles
2022-12-31 15:01:39 +01:00
Stypox
7b54457789
Merge pull request #941 from TeamNewPipe/feat/peertube-comment-replies
[PeerTube]  Support comment replies
2022-12-31 14:57:51 +01:00
AudricV
f45966d449
Merge pull request #910 from Isira-Seneviratne/Locale_forLanguageTag
Add compat Locale.forLanguageTag() implementation.
2022-12-24 23:53:30 +01:00
AudricV
d5437e0bc5
Merge pull request #863 from AudricV/add-content-type-and-content-length-headers-to-post-requests
Add Content-Type header to all POST requests without an empty body
2022-12-16 19:32:56 +01:00
Tobi
88e07e555d
Merge pull request #1000 from AudricV/yt-streaminfoitemextractor-improvements
[YouTube] Improve YoutubeStreamInfoItemExtractor
2022-12-11 17:02:29 +01:00
AudricV
0766b1d211
[YouTube] Improve YoutubeStreamInfoItemExtractor
- Return duration of video premieres;
- Add another non-localized method to determine whether a stream is a running
livestream;
- Return view count and upload date of videos in playlists;
- Store isPremiere result;
- Remove shorts workaround code, as it was only useful on channels and shorts
have been moved into a separated channel tab;
- Improve some other code.
2022-12-08 13:59:12 +01:00
Tobi
896d7e09eb
Merge pull request #978 from Theta-Dev/fix/search-channel-handles
[YouTube] Fix search subscriber count extraction with channel handles
2022-12-05 17:52:05 +01:00
TobiGr
cd3262745d [PeerTube] Report Exceptions thrown while getting a stream's subtitles 2022-12-03 16:11:21 +01:00
TobiGr
4e66b2287e [PeerTube] Add support for comment replies 2022-12-01 14:05:18 +01:00
Tobi
41c8dce452
Merge pull request #992 from Isira-Seneviratne/String_isBlank
Use String.isBlank().
2022-11-30 17:48:54 +01:00
Isira Seneviratne
2bca56f0df Use String.isBlank(). 2022-11-30 08:26:21 +05:30
Isira Seneviratne
3b80547976 Add code review suggestions. 2022-11-30 07:57:45 +05:30
Tobi
a1128ecbdc
Merge pull request #991 from FireMasterK/fix-compile
Fix complication error in comment test
2022-11-29 19:20:01 +01:00
ThetaDev
016623131e docs: update comment in YoutubeChannelInfoItemExtractor 2022-11-29 19:06:03 +01:00
Kavin
2e08eaad96
Fix complication error in comment test. 2022-11-29 16:07:48 +00:00
Kavin
abf08e1496
Merge pull request #990 from FireMasterK/bold-italic-strikethrough
[YouTube] Implement bold/italic/strike-through support
2022-11-29 15:59:38 +00:00
Tobi
72d5ed3318
Merge pull request #987 from FireMasterK/comments-text-description
Use Description object for comments text.
2022-11-29 16:54:56 +01:00
Kavin
57e7a6fb7c
Add mocks test. 2022-11-28 20:27:55 +00:00
Kavin
1d3d7fa5c3
Add test for formatting. 2022-11-28 20:26:37 +00:00
Kavin
52fda37915
Implement bold/italic/strike-through support. 2022-11-28 19:06:18 +00:00
Kavin
b566084cac
Use Description object for comments text. 2022-11-28 17:02:19 +00:00
Tobi
40f1ec4a54
Merge pull request #989 from Stypox/forbid-wrong-nullable-notnull-imports
Block wrong nullable/nonnull imports in checkstyle
2022-11-28 14:57:46 +01:00
Stypox
5f6101698b
Block wrong nullable/nonnull imports in checkstyle 2022-11-28 14:42:18 +01:00
Tobi
f8162b049d
Merge pull request #984 from FireMasterK/unused-dep
Remove unused autolink dependency
2022-11-28 11:28:42 +01:00
Tobi
ad7c3c2374
Merge pull request #985 from FireMasterK/checkstyle-update
Update checkstyle dependency.
2022-11-28 11:22:40 +01:00
Tobi
1da0190056
Merge pull request #980 from TeamNewPipe/fix/yt/unavailable
[YouTube] Fix extracting the detailed error message for unavailable streams
2022-11-28 10:07:34 +01:00
Stypox
60fb30f835
Merge pull request #928 from FireMasterK/comment-urls
Parse YouTube comments as HTML
2022-11-27 19:16:34 +01:00
Kavin
5abea22225
Fix throwing correct reason. 2022-11-26 21:09:08 +00:00
Kavin
7874d3c8aa
Update checkstyle dependency.
Since we now use JDK 11
2022-11-26 20:22:25 +00:00
Kavin
faf28f5c11
Remove unused dependency. 2022-11-26 20:17:25 +00:00
Kavin
a3ed947b2d
Merge pull request #983 from FireMasterK/update-countries
Update supported countries list.
2022-11-26 20:06:00 +00:00
Kavin
c043597255
Update supported countries list. 2022-11-26 19:01:33 +00:00
TobiGr
4680df0bdf Fix throwing correct reason 2022-11-23 17:03:22 +01:00
TobiGr
9de8405c9f [YouTube] Fix extracting the detailed error message of streams which are unavailable 2022-11-23 08:33:06 +01:00
Stypox
34d79bd267
[YouTube] Update mocks 2022-11-22 17:10:04 +01:00
AudricV
2ec296e674
Fix YoutubeSearchExtractorTest.MetaInfoTest
Not all the "learn more" button is uppercase anymore, that's only the case for
the first letter.
2022-11-22 16:34:54 +01:00
AudricV
3891542ca1
Use Downloader's postWithContentType and postWithContentTypeJson methods in services and extractors 2022-11-22 11:37:18 +01:00
AudricV
b2862f3cd1
Add postWithContentType and postWithContentTypeJson utility methods in Downloader
Co-authored-by: Stypox <stypox@pm.me>
2022-11-22 11:37:17 +01:00
AudricV
e9a0d3bd95
[YouTube] Send Content-Type header in all POST requests
This header was not sent partially before and was added and guessed by OkHttp. This can create issues when using other HTTP clients than OkHttp, such as Cronet.

Some code in the modified classes has been improved and / or deduplicated, and usages of the UTF_8 constant of the Utils class has been replaced by StandardCharsets.UTF_8 where possible.

Note that this header has been not added in except in YoutubeDashManifestCreatorsUtils, as an empty body is sent in the POST requests made by this class.
2022-11-22 11:37:16 +01:00
AudricV
b9e463de49
[Bandcamp] Send Content-Type header in POST requests
This header was not sent before and was added and guessed by OkHttp. This can create issues when using other HTTP clients than OkHttp, such as Cronet.

Also make use of StandardCharsets.UTF_8 when getting bytes of bodies instead of the platform default's charset, to make sure to prevent some encoding issues on some JVMs.
2022-11-22 11:35:46 +01:00
AudricV
65d6321e3d
Fix typos in Downloader.post JavaDocs
Post methods in Downloader return the result of a POST request and not the one of a GET request.
2022-11-22 11:35:46 +01:00
ThetaDev
5daabd1793 fix: #976 search subscriber count extraction with channel handles 2022-11-22 02:17:10 +01:00
Kavin
c953e23414
Merge pull request #968 from AudricV/yt-support-no-video-info-renderers-for-streams
[YouTube] Support lack of video info renderers for streams
2022-11-16 20:20:01 +00:00
Tobi
2211a24b69
Merge pull request #971 from lrusso96/patch-1
[YouTube] Improve duration parsing
2022-11-16 16:14:54 +01:00
Tobi
f6eefdc76c
Merge pull request #975 from FireMasterK/jitpack
Use JDK 11 for Jitpack gradle builds
2022-11-15 20:23:21 +01:00
Kavin
a1f22be0dd
Use JDK 11 for Jitpack gradle builds. 2022-11-15 18:43:19 +00:00
Kavin
9bdad55508
Merge pull request #973 from FireMasterK/audio-track
Add support for extracting audio track information
2022-11-15 08:19:47 +00:00
Kavin
86f06b333a
Address review. 2022-11-14 00:05:31 +00:00
Kavin
b16e6082e1
Add test for audio stream languages. 2022-11-13 23:10:44 +00:00
Kavin
30909da1df
Fix audio track similar comparison for IDs. 2022-11-13 23:08:54 +00:00
Kavin
6d59cdbe3a
Add support for extracting audio tracks. 2022-11-13 21:39:29 +00:00
Tobi
43d1c1f8b1
Merge pull request #880 from Isira-Seneviratne/Use_StandardCharsets
Use StandardCharsets.UTF_8.
2022-11-12 03:23:44 +01:00
Isira Seneviratne
e4d982c7ea Fix license. 2022-11-12 07:29:15 +05:30
Isira Seneviratne
416089146e Fix failing tests. 2022-11-12 07:29:15 +05:30
Isira Seneviratne
ff5f223d3f Update README.md 2022-11-12 07:29:15 +05:30
Isira Seneviratne
ddbce3b83d Add Utils methods for URL encoding/decoding using UTF-8. 2022-11-12 07:29:15 +05:30
Isira Seneviratne
366f5c1632 Use StandardCharsets.UTF_8. 2022-11-12 07:29:15 +05:30
Tobi
0ea16c0b73
Merge pull request #954 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.10
Bump gson from 2.9.1 to 2.10
2022-11-09 17:28:31 +01:00
dependabot[bot]
0b82fade51
Bump gson from 2.9.1 to 2.10
Bumps [gson](https://github.com/google/gson) from 2.9.1 to 2.10.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.9.1...gson-parent-2.10)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-09 16:21:16 +00:00
Luigi Russo
c9635218e2
[YouTube] Improve duration parsing 2022-11-09 09:41:29 +01:00
Isira Seneviratne
1be270768f
Merge pull request #893 from Isira-Seneviratne/Static_sets
Use immutable sets in YoutubeParsingHelper.
2022-11-09 09:07:01 +05:30
Isira Seneviratne
316d8573fa Use immutable sets in YoutubeParsingHelper. 2022-11-07 07:50:26 +05:30
AudricV
6a2c680d8f
[YouTube] Add mocks for YoutubeStreamExtractorDefaultTest.NoVisualMetadataVideoTest 2022-11-04 19:43:06 +01:00
AudricV
e66fed41d6
[YouTube] Add a StreamExtractor test for a video without visual metadata
The video "Makani’s first commercial-scale energy kite" (video ID:
An8vtD1FDqs), which has this behavior, is used for the new test,
NoVisualMetadataVideoTest, added in YoutubeStreamExtractorDefaultTest.

Tests of elements who throw an exception in this case (subscriber count, like
count, uploader avatar URL) test if the ParsingException exception is thrown by
YoutubeStreamExtractor.
2022-11-04 19:42:12 +01:00
AudricV
aa9a8ca23c
[YouTube] Make non-extraction of videoPrimaryInfoRenderer and/or videoSecondaryInfoRenderer not fatal
Also de-duplicated common code related to the obtain of these video info renderers.

This change allows extraction of videos without visual metadata.
2022-11-04 18:35:53 +01:00
AudricV
eb07d70a2c
Merge pull request #964 from AudricV/yt-support-handles-and-all-channel-usernames
[YouTube] Support handles and all channel usernames
2022-11-04 12:09:15 +01:00
AudricV
20cd8e8a4a
[YouTube] Update mocks of YoutubeChannelExtractorTest.Gronkh 2022-11-03 19:46:42 +01:00
AudricV
a34f060ba0
[YouTube] Use a handle for YoutubeChannelExtractorTest.Gronkh 2022-11-03 19:46:42 +01:00
AudricV
724f669ff7
[YouTube] Add tests for handles and user IDs with non ASCII characters support
Unneeded public modifiers in test methods of
YoutubeChannelLinkHandlerFactoryTest have been also removed.
2022-11-03 19:46:42 +01:00
AudricV
61ce041bda
[YouTube] Support handles and all custom channel names
More non-channel paths have been also added to the excluded custom name paths,
documentation and exception messages have been improved and fixed in some
places, and the licence header of YoutubeChannelLinkHandlerFactory has been
moved to its beginning and updated.
2022-11-03 19:46:42 +01:00
AudricV
ffffb04439
Merge pull request #953 from Theta-Dev/attributed-text-desc
[YouTube] Add support for attributed text description
2022-11-03 18:34:30 +01:00
ThetaDev
592e1d6386 fix: parsing attributed description with no command runs 2022-11-03 12:10:52 +01:00
AudricV
31bf7046b7
Merge pull request #944 from Theta-Dev/fix-rich-grid-renderer
[YouTube] Support richGridRenderer on channel page to fix extraction of videos in channels
2022-11-03 00:10:54 +01:00
ThetaDev
099b53cc4f
[YouTube] Add parser for attributedDescription
Also update the mock of the next InnerTube endpoint response of the
YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing test class with an
attributedDescription instead of a regular description
2022-11-02 23:11:33 +01:00
AudricV
e4c24d4c36
[YouTube] Regenerate supported channels mocks 2022-11-02 19:13:59 +01:00
Theta-Dev
20e4a35814
[YouTube] Support richGridRenderer on channel pages
YouTube is deploying a new layout on their channel pages, which uses richGridRenderer JSON objects.
2022-11-02 19:01:29 +01:00
AudricV
4cae66f1f9
Merge pull request #946 from chowder/dev
Add ability to identify short-form `StreamInfoItem`s
2022-11-01 12:19:58 +01:00
Tobi
eb40bb8458
Merge pull request #959 from FireMasterK/playlist-info-item-uploader
Add uploaderUrl and uploaderVerified to PlaylistInfoItem.
2022-10-31 13:10:33 +01:00
chowder
b1a899fd47 Fix null pointer exception 2022-10-31 11:12:23 +00:00
Stypox
a4db106a66
Merge pull request #960 from AudricV/yt-workaround-403-errors-android-client
[YouTube] Workaround 403 HTTP errors of ANDROID client streams
2022-10-30 21:53:26 +01:00
Kavin
b441910257
Mark uploaderUrl as nullable. 2022-10-30 13:36:04 +00:00
chowder
3fdc0e72cc Line breaks for long docstrings 2022-10-30 13:28:39 +00:00
Kavin
6a256d0631
Add uploader url and verified to PlaylistInfoItem. 2022-10-30 13:00:19 +00:00
Tobi
430504b4b5
Merge pull request #958 from AudricV/yt-playlists-support-new-metadata-format
[YouTube] Support new metadata format of playlists
2022-10-30 12:31:43 +01:00
Kavin
f9bd08c649 Address reviews. 2022-10-30 01:25:30 +00:00
Caleb
9282c3c13b Fix exception message for YoutubeStreamInfoItemExtractor#isShortFormContent
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
2022-10-30 01:23:15 +00:00
Caleb
c5216f7c12 Update docstring for StreamExtractor#isShortFormContent
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
2022-10-30 01:23:15 +00:00
Caleb
04795fe5d2 Use Stream API for ShortFormContent#testShortFormContent
Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
2022-10-30 01:23:15 +00:00
chowder
975b38a0b9 Regenerate test mocks 2022-10-30 01:23:15 +00:00
chowder
4cccd33f3d Implement isShortFormContent for StreamExtractor and StreamInfo 2022-10-30 01:23:15 +00:00
chowder
09544ceb23 Fix tests 2022-10-30 01:23:15 +00:00
chowder
644cc6cd76 Rename test function name 2022-10-30 01:23:15 +00:00
chowder
daf5674951 Add ability to identify short-form StreamInfoItems 2022-10-30 01:23:12 +00:00
Tobi
3d314169b9
Merge pull request #943 from TeamNewPipe/fix/sc/comments
[SoundCloud] Fix getting more comments
2022-10-29 22:19:50 +02:00
AudricV
65125a3eb4
[YouTube] Fix YoutubePlaylistExtractorTest.LearningPlaylist
The video count is now returned for this playlist, so it isn't unknown.

The testStreamCount method of this test class asserts now that the stream count
is greater than 40.
2022-10-29 18:12:10 +02:00
AudricV
7258a53225
[YouTube] Support new playlist layout
This new layout doesn't provide author thumbnails and is completely different
for metadata, so the code to get them has been refactored.

The code of learning playlists video count check has been also removed, as it
seems to be not relevant anymore (the video count seems to be returned for
these playlists with both layouts).

Finally, unneeded overrides of subchannel methods, which don't apply to the
YouTube service, have been removed.
2022-10-29 18:12:10 +02:00
AudricV
e37e8b358e
[YouTube] Update mocks that need to be updated 2022-10-29 18:10:18 +02:00
AudricV
60e97cd274
[YouTube] Workaround getting streaming URLs returning 403 HTTP response codes
Using the player parameters used to get stories seems to fix the issue, which
affects currently only certain countries such as UK.

This is a workaround and should be fixed in a better way (by changing the
InnerTube additional client used for videos or finding what is now required in
Android player requests).
2022-10-29 17:58:33 +02:00
AudricV
4bc90cd9d8
Merge pull request #957 from AudricV/fix-checkstyle-error
Fix Checkstyle error in YoutubeCommentsInfoItemExtractor
2022-10-29 17:55:33 +02:00
AudricV
c230d84df1
Fix Checkstyle error in YoutubeCommentsInfoItemExtractor 2022-10-29 13:24:19 +02:00
Tobi
9ffdd0948b
Merge pull request #945 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.3
Bump spotbugs-annotations from 4.7.2 to 4.7.3
2022-10-18 22:55:24 +02:00
dependabot[bot]
24137bb15f
Bump spotbugs-annotations from 4.7.2 to 4.7.3
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.7.2 to 4.7.3.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.2...4.7.3)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-10-17 09:20:51 +00:00
xz-dev
0ffcb32d9c
[YouTube] Add comment reply count support (#936)
Add comment reply count support for YouTube and introduce `CommentsInfoItem.UNKNOWN_REPLY_COUNT` constant

Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2022-10-15 12:40:06 +02:00
Isira Seneviratne
b90a566dd8 Add backport implementation of Locale.forLanguageTag(). 2022-10-12 09:21:39 +05:30
Isira Seneviratne
b232c29d22 Use Locale.forLanguageTag(). 2022-10-12 09:21:38 +05:30
TobiGr
4d136599bd [SoundCloud] Fix getting more comments 2022-10-11 15:44:54 +02:00
Tobi
a822e91909
Merge pull request #939 from TurtleArmyMc/fix/SoundcloudPlaylistExtractor_track_order
Fix SoundcloudPlaylistExtractor: tracks are in correct order
2022-10-10 22:28:18 +02:00
TobiGr
02810a7db7 Add a comment 2022-10-10 22:22:12 +02:00
Tobi
2d50369c77
Merge pull request #888 from Isira-Seneviratne/EnumMap
Use EnumMap in PatternsHolder.
2022-10-10 12:00:06 +02:00
Tobi
54092fc3c7
Merge pull request #930 from Isira-Seneviratne/Avoid_Comparator_NPE
Avoid possible NullPointerException in MediaCCCRecentKiosk.
2022-10-10 11:43:34 +02:00
Tobi
d7c23ef118
Merge pull request #924 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.2
Bump spotbugs-annotations from 4.7.1 to 4.7.2
2022-10-10 11:35:40 +02:00
dependabot[bot]
08339d6b0c
Bump spotbugs-annotations from 4.7.1 to 4.7.2
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.7.1 to 4.7.2.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.1...4.7.2)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-10-10 09:32:17 +00:00
Tobi
5537e84100
Merge pull request #931 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.1
Bump junit-bom from 5.9.0 to 5.9.1
2022-10-10 11:30:53 +02:00
dependabot[bot]
719664593f
Bump junit-bom from 5.9.0 to 5.9.1
Bumps [junit-bom](https://github.com/junit-team/junit5) from 5.9.0 to 5.9.1.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.9.0...r5.9.1)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-10-10 09:25:42 +00:00
TurtleArmyMc
bf70d32eb4 Fix SoundcloudPlaylistExtractor: tracks are in correct order 2022-09-30 15:26:25 -04:00
Stypox
5c710da160
Merge pull request #934 from AudricV/fix-yt-throttling-decryption-function-regex
[YouTube] Fix throttling decryption function regex
2022-09-25 13:26:08 +02:00
AudricV
8067c43837
[YouTube] Don't use a specific letter for the decryption function name pattern
Use the same possible characters for variables everywhere, in order to avoid
potential future throttling parameter decryption function name parsing issues
related to the usage of other letter(s) than b.
2022-09-24 21:49:22 +02:00
AudricV
abcee87167
[YouTube] Fix throttling parameter decryption function regex
- Quote the function name, as it may contain special regex symbols, such as
  dollar;
- Support multiple lines;
- Use what looks like the end of the function for the end of the regex (this
  part is inspired from yt-dlp throttling parameter decryption regex);
- Move the throttling function body regex into a private and static constant.
2022-09-24 21:28:09 +02:00
ThetaDev
7244be7627
[YouTube] Add JavaScript lexer to parse completely throttling decryption function (#905) 2022-09-24 19:55:46 +02:00
Isira Seneviratne
0e31c86aee Avoid possible NullPointerException in MediaCCCRecentKiosk. 2022-09-21 05:40:07 +05:30
Stypox
a99af9bb6e
Merge pull request #927 from Stypox/fix-feed-thumbnail
[YouTube] Return mqdefault thumbnails in fast feed
2022-09-19 08:52:01 +02:00
Kavin
4e14644c41
Ensure comments are parsed with URLs and timestamps. 2022-09-17 16:03:39 +05:30
Stypox
14ef430546
Merge pull request #920 from AudricV/fix-yt-trending-extraction
[YouTube] Fix extraction of the Trending kiosk with the new data model returned
2022-09-14 11:14:13 +02:00
Stypox
93ef73e2f2
[YouTube] Return mqdefault thumbnails in feed
The hqdefault thumbnails, which are used by default, have black bars at the top and at the bottom, and are thus not optimal.
2022-09-13 16:46:54 +02:00
AudricV
d8ddeb549c
[YouTube] Support new trending structure and filter recently trending section
This new structure allow us to filter easily Trending shorts and Recently
trending sections.

On the previous one, this Recently trending section is now filtered, by
checking whether sections have a title, which isn't the case for normal trends
contrary to the other ones.

This makes that the extractor returns now only the real 50 "Now" YouTube
trends.

Elements inside arrays are now extracted dynamically instead of only the ones
of the first index, using Java 8's Stream API.

The getInitialPage() method of YoutubeTrendingExtractor can now throw a
ParsingException if no selected tab (corresponding to the one of the trends
type extracted) has been found.

Finally, the licence header has been moved to the top of the file and updated.
2022-09-11 18:52:12 +02:00
AudricV
884da40f65
Merge pull request #926 from AudricV/fix-yt-like-count-extraction
[YouTube] Fix extraction of like count with the new data model
2022-09-11 12:21:38 +02:00
AudricV
58b09cf78b
Merge pull request #925 from Theta-Dev/fix-td-regex
[YouTube] Fix throttling parameter decryption on Android
2022-09-10 23:30:50 +02:00
AudricV
119b9129e2
[YouTube] Fix extraction of like count with the new data model
In the new data model currently A/B tested or deployed, the like count button
data is the same, but its path has been changed.

The extraction of the like count has been also improved, by using the multiple
accessibility data available instead of the only one which was used before.
This accessibility data has been also deprioritized, because it is language
dependent when there is no view.

Like count is always known, even for age-restricted videos, so returning -1 if
the like count extraction fails on this type of videos is not needed and hides
an extraction error: that's the reason why this return has been removed and the
exception will always be thrown, even if a video is age-restricted.
2022-09-10 23:28:38 +02:00
ThetaDev
4905f74700
[YouTube] Fix throttling parameter decryption on Android
Escape the curly brace in the regular expression used to parse the throttling
parameter decryption function to allow its compatibility on Android.
2022-09-10 17:03:39 +02:00
Isira Seneviratne
e101d58ae7 Use EnumMap in PatternsHolder. 2022-08-26 08:15:21 +05:30
Stypox
6a858368c8
Merge pull request #912 from Stypox/downgrade-rhino
Downgrade Rhino since it uses a class not available on Android
2022-08-25 10:10:27 +02:00
Stypox
fa25a7280f
Downgrade Rhino since it uses a class not available on Android
Reverts #774, see TeamNewPipe/NewPipe#8876
2022-08-25 09:49:13 +02:00
Stypox
25f0ce7305
Merge pull request #911 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.3
Bump jsoup from 1.15.2 to 1.15.3
2022-08-24 14:23:44 +02:00
Stypox
d8e10dd3ad
Merge pull request #907 from Isira-Seneviratne/Remove_EmptyString
Remove EMPTY_STRING.
2022-08-24 14:22:55 +02:00
dependabot[bot]
1d7a3a90a4
Bump jsoup from 1.15.2 to 1.15.3
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.15.2 to 1.15.3.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.2...jsoup-1.15.3)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-08-24 09:14:26 +00:00
Isira Seneviratne
41254ae12a Remove annotation. 2022-08-24 06:59:17 +05:30
Isira Seneviratne
943b7c033b Remove EMPTY_STRING. 2022-08-24 06:59:17 +05:30
litetex
20914a6035
Merge pull request #903 from TacoTheDank/updateGradle
Update gradle wrapper to 7.5.1
2022-08-23 16:21:56 +02:00
litetex
d6577e5e0b
Merge pull request #882 from litetex/fix-all-tests
Fix all tests
2022-08-23 16:19:27 +02:00
litetex
0a5ad90724
Merge pull request #901 from AudricV/remove-deprecated-code-yt-throttling-decrypter
[YouTube] Remove deprecated throttling decryption code
2022-08-22 17:13:00 +02:00
AudricV
03d9a4fe9d
Fix typos in YoutubeStreamExtractor.tryDecryptUrl 2022-08-21 21:17:03 +02:00
litetex
0beb55a232 Improve `YoutubeThrottlingDecrypterTest` 2022-08-21 20:17:05 +02:00
litetex
9e93d6b193 YoutubeThrottlingDecrypter: Check if returned string is a valid JavaScript function 2022-08-21 20:16:45 +02:00
litetex
ed0a07af4e YoutubeThrottlingDecrypter: Patch regex 2022-08-21 20:15:53 +02:00
litetex
641a447d9b Updated consent-related mock-data 2022-08-21 18:43:48 +02:00
litetex
8ff7a90f52 Improved consent cookie related constants and documentation 2022-08-21 18:41:40 +02:00
TacoTheDank
d8339d4060 Update gradle wrapper to 7.5.1 2022-08-18 15:09:35 -04:00
AudricV
cb64a480cd
[YouTube] Remove deprecated code in YoutubeThrottlingDecrypter
YoutubeThrottlingDecrypter is now a non-instantiable final class and non-static
attributes have been removed.

The static attributes of this class have been renamed, in order to respect the
naming specification used.
2022-08-16 15:01:05 +02:00
litetex
5bbea3a8f2 Fix `YoutubeMixPlaylistExtractorTest` again
Doesn't work for kurgesagt videos anymore, used music video (copyright free) instead
2022-08-14 15:12:27 +02:00
litetex
da06166065 Updated mock data of `YoutubeSearchExtractorTest$Suggestion` 2022-08-14 15:12:26 +02:00
litetex
2e36ab1578 Fixed `YoutubeSearchExtractorTest$Suggestion` 2022-08-14 15:12:26 +02:00
litetex
2a8a623643 `YoutubePlaylistLinkHandlerFactoryTest`: Use parameterized tests 2022-08-14 14:48:31 +02:00
litetex
938e69a16a Fixed `YoutubePlaylistLinkHandlerFactoryTest` 2022-08-14 14:48:30 +02:00
litetex
844de3e378 Fix `YoutubeStreamLinkHandlerFactoryTest ` and parameterized the tests
used code from #833
2022-08-14 14:48:30 +02:00
litetex
c1a72b8240 Use Android Studios code style 2022-08-14 14:48:29 +02:00
litetex
fc27b8a5b8 Use preexisting `ContentNotAvailableException` 2022-08-14 14:48:29 +02:00
Stypox
17bad6cedf Fix test about exception type: NPE, not IllegalArgment
The relevant change was made in #877
2022-08-14 14:48:28 +02:00
litetex
ecfc370685 Fixed all YTMixPlaylists
Added option to choose if you want to consent or not - currently this is done by a static variable in ``YoutubeParsingHelper`` - may not be the best long-term solution but for now the tests work again (in EU countries) 🥳
2022-08-14 14:48:27 +02:00
litetex
2b6fe294b2 Fixed `YoutubeMusicSearchExtractorTest`
The renderers and queries got changed.
2022-08-14 14:48:27 +02:00
litetex
d6586da614 Failing test works locally without any problems. Reformatted it a bit. 2022-08-14 14:48:27 +02:00
litetex
bf3ae5e679 Fixed `SoundcloudStreamLinkHandlerFactoryTest`
* Removed the unnecessary ``public``
* Use parameterized tests
* One song got removed (which caused the test failure), replaced it with another song
2022-08-14 14:48:27 +02:00
litetex
504f81036e Fixed `PeertubePlaylistExtractorTest`
Also removed the unnecessary ``public`` and reformatted the code a bit
2022-08-14 14:48:26 +02:00
litetex
e7c12258f4 Fixed `BandcampSearchExtractorTest`
Also removed the unnecessary ``public``
2022-08-14 14:48:26 +02:00
litetex
9e8724df4d Fixed `YoutubeStreamExtractorLivestreamTest`
The "Lofi Girl"-stream got interrupted by a copyright strike and had to be restarted. Because of this a new id is now used.
2022-08-14 14:48:26 +02:00
Tobi
76aad92fa5
Merge pull request #890 from AudricV/yt-clients-update-improvements-and-extraction-fixes
[YouTube] Update clients version and improve extraction of API key and client version of WEB client
2022-08-12 23:32:41 +02:00
AudricV
472f5d9e9c
[YouTube] Update mocks 2022-08-12 19:20:32 +02:00
AudricV
7bdca33a87
[YouTube] Ensure that an additional player response is the correct one
If YouTube detect that requests come from a third party client, they may
replace the real player response by another one of a video saying that this
content is not available on this app and to watch it on the latest version of
YouTube. We can detect this by checking whether the video ID of the player
response returned is the same as the one requested by the extractor.
2022-08-12 19:20:31 +02:00
AudricV
c82317e318
[YouTube] Spoof more mobile clients
Additional parameters have been added to the player requests of ANDROID and IOS
clients:

- for both clients: osName and osVersion: their respective values are:
  - for the ANDROID one: Android and 12;
  - for the IOS one: iOS and 15.6.0.19G71.
- for the ANDROID client: androidTargetSdkVersion, with the Android SDK version
  corresponding to the Android version used in the player requests of this
  client. This parameter is now required with this client to be sure to get a
  correct player response, otherwise, the one of a video saying that this
  content is not available in this app and to watch it with the latest version
  of YouTube can be returned instead;
- for the IOS client: deviceMake, with Apple as its value.

The iOS version sent in the IOS client player requests has been also updated to
the version 15.6 of the OS.

Finally, a comment about the requirement to use the signature timestamp from
the player JavaScript base file for HTML5 player requests on videos with
obfuscated URLs has been added and replaces a previous one which may be not
true.
2022-08-12 19:20:31 +02:00
AudricV
d0549a5a52
[YouTube] Update client versions and use a real version for the iOS client
The iOS version can be got easily in fact, by looking at the What's New section of the App Store' app page.
2022-08-12 19:20:31 +02:00
AudricV
d7e678aca2
[YouTube] Improve WEB client version and API key HTML extraction
Common code in WEB client version HTML extraction has been deduplicated, usage of the Java 8 Stream API has been made and initial data fallback has been used as a last resort.
This means that the client version extraction from regexes will be used before this fallback, as it doesn't contain the full client version.
This can be used as a way to fingerprint the extractor, even if it seems to be not the case.
2022-08-12 19:20:30 +02:00
AudricV
6a885ef5ab
Merge pull request #891 from Theta-Dev/fix-throttling-decrypter
[YouTube] Fix extraction of more complex nsig functions
2022-08-12 18:24:21 +02:00
AudricV
5b548340e8
[YouTube] Catch any exception in YoutubeThrottlingDecrypter.apply and improve docs
This will prevent any future extractor break due to decryption failure, like it was excepted to be the case before.

Some documentation about the throttling decryption has been also improved.
2022-08-12 17:49:36 +02:00
ThetaDev
52ded6e3d7
Handle curly braces inside strings in StringUtils.matchToClosingParenthesis
This is required to extract fully more complex YouTube nsig functions.
2022-08-12 16:32:00 +02:00
Stypox
d12003651b
Merge pull request #886 from Isira-Seneviratne/toArray_improvements
Make improvements to methods using toArray().
2022-08-06 22:34:23 +02:00
Isira Seneviratne
7daca10a06 Make improvements to methods using toArray(). 2022-08-06 05:21:12 +05:30
Stypox
2906be22af
Merge pull request #881 from Isira-Seneviratne/String_join
Use String.join() and Collectors.joining().
2022-08-04 12:09:33 +02:00
Stypox
4ddb96a86f
Merge pull request #883 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.9.1
Bump gson from 2.9.0 to 2.9.1
2022-08-04 11:20:41 +02:00
Isira Seneviratne
64771c5712 Use String.join() and Collectors.joining(). 2022-08-04 05:18:13 +05:30
Stypox
fc8b5ebbc6
Merge pull request #878 from Isira-Seneviratne/Use_Collections
Use Collections methods.
2022-08-03 22:50:41 +02:00
Stypox
4a4939d89c
Merge pull request #877 from Isira-Seneviratne/Use_Objects_requireNonNull
Use Objects.requireNonNull().
2022-08-03 22:45:35 +02:00
Stypox
c336bd58a5
Merge pull request #879 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.0
Bump junit-bom from 5.8.2 to 5.9.0
2022-08-03 21:06:20 +02:00
dependabot[bot]
325af31e5f
Bump gson from 2.9.0 to 2.9.1
Bumps [gson](https://github.com/google/gson) from 2.9.0 to 2.9.1.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.9.0...gson-parent-2.9.1)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-08-01 09:26:14 +00:00
dependabot[bot]
d905636021
Bump junit-bom from 5.8.2 to 5.9.0
Bumps [junit-bom](https://github.com/junit-team/junit5) from 5.8.2 to 5.9.0.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.8.2...r5.9.0)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-07-27 09:20:12 +00:00
Isira Seneviratne
1af6b8eedb Use Collections.singletonList(). 2022-07-27 07:35:57 +05:30
Isira Seneviratne
ff60e05c76 Use Collections.singletonMap(). 2022-07-27 07:35:57 +05:30
Isira Seneviratne
682a4263e5 Use Objects.requireNonNull(). 2022-07-27 06:55:26 +05:30
Stypox
8c5f014a6f
Merge pull request #869 from mhmdanas/add-workflow-permissions
Use minimum required permissions for GitHub workflows
2022-07-13 18:58:32 +02:00
Stypox
954a294e27
Merge pull request #870 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.2
Bump jsoup from 1.15.1 to 1.15.2
2022-07-13 17:48:06 +02:00
Mohammed Anas
f57049d2c0
Use temurin instead of adopt (#868)
See
https://blog.adoptopenjdk.net/2021/08/goodbye-adoptopenjdk-hello-adoptium/.
2022-07-10 18:38:08 +03:00
Stypox
5bd7bf20cc
Merge pull request #867 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.1
Bump spotbugs-annotations from 4.7.0 to 4.7.1
2022-07-06 14:23:44 +02:00
Stypox
5ab74b3631
Merge pull request #857 from FireMasterK/video-title
Get original untranslated title for YouTube
2022-07-06 10:26:45 +02:00
dependabot[bot]
122365005a
Bump jsoup from 1.15.1 to 1.15.2
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.15.1 to 1.15.2.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.1...jsoup-1.15.2)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-07-04 09:38:16 +00:00
mhmdanas
414186cff5 Use minimum required permissions for GitHub workflows
This reduces the attack surface if the workflows are ever compromised.
2022-07-03 23:55:28 +03:00
dependabot[bot]
e9b4be3e3c
Bump spotbugs-annotations from 4.7.0 to 4.7.1
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.7.0 to 4.7.1.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.0...4.7.1)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-06-28 09:15:07 +00:00
Stypox
5219a705ba
Merge pull request #864 from AudricV/fetch-android-client-for-ended/post-livestreams
[YouTube] Fetch the ANDROID client for ended/post livestreams
2022-06-24 14:55:20 +02:00
AudricV
090debd83b
[YouTube] Fetch the ANDROID client for ended/post livestreams
The ANDROID client was only fetched for video contents, where it can be useful on ended/post livestreams, if the n parameter of the WEB client cannot be decrypted, to avoid throttling issues (because the WEB client was only used before for ended/post livestreams).

It also provides an exclusive 48kbps M4A audio format in the adaptiveFormats array of the JSON player response, like other mobile clients (which can be also extracted from the response of the DASH manifest URL returned into the WEB client player's response, but the DASH manifest is not used by the extractor).

A note about non-fatality of fetching or parsing issues of the ANDROID and IOS clients has been added.
2022-06-21 18:53:49 +02:00
litetex
a26bcc55c4
Merge pull request #845 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.0
Bump spotbugs-annotations from 4.6.0 to 4.7.0
2022-06-19 15:36:55 +02:00
dependabot[bot]
424eb1c559
Bump spotbugs-annotations from 4.6.0 to 4.7.0
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.6.0 to 4.7.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.6.0...4.7.0)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-06-19 13:29:18 +00:00
litetex
2712c3d788
Merge pull request #847 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.1
Bump jsoup from 1.14.3 to 1.15.1
2022-06-19 15:28:43 +02:00
dependabot[bot]
281d2b9f81
Bump jsoup from 1.14.3 to 1.15.1
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.14.3 to 1.15.1.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.14.3...jsoup-1.15.1)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-06-19 13:15:40 +00:00
litetex
f775155d25
Merge pull request #846 from litetex/remove-unused-methods
Remove unused methods
2022-06-19 15:12:15 +02:00
litetex
9d625dd75f
Merge pull request #860 from gliptak/patch-1
Correct PeertubePlaylistExtractorTest unit tests
2022-06-19 15:09:27 +02:00
Gábor Lipták
28c9340d69
Correct unit tests 2022-06-18 11:18:38 -04:00
Stypox
1b51eab664
Merge pull request #859 from AudricV/delivery-methods-fixes-and-improvements
Fix extraction of some properties in ItagItems for YouTube livestreams and post-live streams and remove completely SoundCloud HLS workaround
2022-06-17 15:46:13 +02:00
AudricV
301a795ed3
[SoundCloud] Remove completely workaround for HLS streams
SoundCloud is currently removing this workaround completely, so there is no need to keep it, because it impacts the loading time (a HLS playlist was downloaded and parsed).
2022-06-16 12:12:54 +02:00
AudricV
e960a417ec
[YouTube] Fix extraction of fps, audioSampleRate and audioChannels fields for ItagItems of live streams and post live streams
These values were only set before for video streams.

A fallback for the audio channels count has been added, in order to prevent exceptions when generating DASH manifests of audio streams: the fallback value is 2, because most audio streams on YouTube have 2 audio channels.
2022-06-16 12:12:54 +02:00
Stypox
c8a77da2ab
Merge pull request #810 from TiA4f8R/delivery-methods-v2
Support delivery methods other than progressive HTTP
2022-06-02 22:44:24 +02:00
Kavin
7635aeed2c
Get original untranslated title for YouTube. 2022-06-02 09:57:52 +01:00
TiA4f8R
287d1dfd63
[SoundCloud] Use the HLS delivery method for all streams and extract only a single stream URL from HLS manifest for MP3 streams
SoundCloud broke the workaround used to get a single file from HLS manifests for Opus manifests, but it still works for MP3 ones.

The code has been adapted to prevent an unneeded request (the one to the Opus HLS manifest) and the HLS delivery method is now used for SoundCloud MP3 and Opus streams, plus the progressive one (for tracks which have a progressive stream (MP3) and for the ones which doesn't have one, it is still used by trying to get a progressive stream, using the workaround).

Streams extraction has been also moved to Java 8 Stream's API and the relevant test has been also updated.
2022-05-29 19:08:18 +02:00
Stypox
b3c620f0d8
Apply code review and Streams rework 2022-05-28 12:00:58 +02:00
Stypox
d652e05874
[MediaCCC] Fix comments about containsSimilarStream 2022-05-28 12:00:58 +02:00
Stypox
044639c32b
Solve some review comments 2022-05-28 12:00:57 +02:00
litetex
c33d392958
Fixed typo XEE → XXE (Xml eXternal Entity attack)
See also:
https://en.wikipedia.org/wiki/XML_external_entity_attack
https://owasp.org/www-community/vulnerabilities/XML_External_Entity_(XXE)_Processing
2022-05-28 12:00:57 +02:00
TiA4f8R
fffbbee7f3
[YouTube] Add missing Nonnull annotations in getCache method of YouTube DASH manifest creators 2022-05-28 12:00:56 +02:00
TiA4f8R
f7b1515290
[YouTube] Refactor DASH manifests creation
Move DASH manifests creation into a new subpackage of the YouTube package, dashmanifestcreators.
This subpackage contains:

- CreationException, exception extending Java's RuntimeException, thrown by manifest creators when something goes wrong;
- YoutubeDashManifestCreatorsUtils, class which contains all common methods and constants of all or a part of the manifest creators;
- a manifest creator has been added per delivery type of YouTube streams:
  - YoutubeProgressiveDashManifestCreator, for progressive streams;
  - YoutubeOtfDashManifestCreator, for OTF streams;
  - YoutubePostLiveStreamDvrDashManifestCreator, for post-live DVR streams (which use the live delivery method).

Every DASH manifest creator has a getCache() static method, which returns the ManifestCreatorCache instance used to cache results.

DeliveryType has been also extracted from the YouTube DASH manifest creators part of the extractor and moved to the YouTube package.

YoutubeDashManifestCreatorTest has been updated and renamed to YoutubeDashManifestCreatorsTest, and YoutubeDashManifestCreator has been removed.

Finally, several documentation and exception messages fixes and improvements have been made.
2022-05-28 12:00:56 +02:00
TiA4f8R
f17f7b9842
Apply requested changes in YoutubeParsingHelper 2022-05-28 12:00:55 +02:00
TiA4f8R
301b9fa024
Remove hashCode and equals methods overrides of Stream classes 2022-05-28 12:00:55 +02:00
TiA4f8R
2f3920c648
[YouTube] Return approxDurationMs value from YouTube's player response in ItagItems generated
This change allows to build DASH manifests using YoutubeDashManifestCreator with the real duration of streams and prevent potential cuts of the end of progressive streams, because the duration in YouTube's player response is in seconds and not milliseconds.
2022-05-28 12:00:54 +02:00
TiA4f8R
4158fc46a0
[Bandcamp] Fix regression of Opus radio streams extraction
When moving opus-lo into a constant, opus-lo was renamed to opus_lo and was only used if no MP3 stream was available (which was not the case before the changes in BandcampRadioStreamExtractor related to the addition of the support of all delivery methods), so these changes removed the ability to get Opus streams of Bandcamp radios.

This commit reverts this unwanted change.
2022-05-28 12:00:54 +02:00
TiA4f8R
54d323c2ae
Fix Checkstyle issue in YoutubeDashManifestCreator 2022-05-28 12:00:53 +02:00
Stypox
2321822844
Rename Stream's baseUrl to manifestUrl 2022-05-28 12:00:53 +02:00
Stypox
cfc13f4a6f
[YouTube] Reduce exception generation code and move several attributes of MPD documents into constants
Rename YoutubeDashManifestCreationException to CreationException.
Also use these constants in YoutubeDashManifestCreatorTest.
2022-05-28 12:00:53 +02:00
Stypox
00bbe5eb4d
[YouTube] Make exception messages smaller 2022-05-28 12:00:52 +02:00
Stypox
4da05afe11
[YouTube] Inline collectSegmentsData in YoutubeDashManifestCreator 2022-05-28 12:00:52 +02:00
Stypox
3708ab9ed5
[YouTube] Refactor YoutubeDashManifestCreator
- Remove all of the methods used to access caches and replace them with three caches getters
- Rename caches to shorter and more meaningful names
- Remove redundant @throws tags that just say "if this method fails to do what it should do", which is obvious
2022-05-28 12:00:51 +02:00
Stypox
5c83409039
[YouTube] Rewrite manifest test and rename long methods 2022-05-28 12:00:51 +02:00
Stypox
8226fd044f
[YouTube] Suppress Sonar security warning for XEE 2022-05-28 12:00:50 +02:00
Stypox
ba68b8c014
[YouTube] Secure DashManifestCreator against XEE attack
Also remove duplicate lines from javadoc comments, otherwise checkstyle complaints.
XEE prevention is inspired from https://github.com/ChuckerTeam/chucker/pull/201/files
For an explanation of XEE see https://rules.sonarsource.com/java/RSPEC-2755, though the solution reported there causes crashes on Android
2022-05-28 12:00:50 +02:00
Stypox
159d05c91b
Remove unused DashMpdParser (but kept in git history) 2022-05-28 12:00:49 +02:00
Stypox
50272db946
Apply reviews: improve comments, remove FILE, remove Stream#equals(Stream) 2022-05-28 12:00:49 +02:00
TiA4f8R
07b045f20d
[YouTube] Support the iOS client in YoutubeDashManifestCreator and decrypt again the n parameter, if present, for all clients
This commits reverts a new behavior introduced in this branch, which only applied the decryption if needed on streams from the WEB client.
Also fix rebase issues and documentations style in YoutubeDashManifestCreator.
2022-05-28 12:00:48 +02:00
TiA4f8R
436ddde29f
Use assertThrows in YoutubeDashManifestCreatorTest 2022-05-28 12:00:48 +02:00
TiA4f8R
d64d7bbd01
Move ManifestCreatorCache tests to a separate class and remove override of equals and hashCode methods in ManifestCreatorCache
These methods don't need to be overriden, as they are not excepted to be used in collections.
Also improve the toString method of this class, which contains also now clearFactor and maximumSize attributes and for each operations.
2022-05-28 12:00:48 +02:00
TiA4f8R
2fb1a412a6
Fix Checkstyle issues, revert resolution string changes for YouTube video streams and don't return the rn parameter in DASH manifests
This parameter is still used to get the initialization sequence of OTF and POST-live streams, but is not returned anymore in the manifests.
It has been removed in order to avoid fingerprinting based on the number sent (e.g. when starting to play a stream close to the end and using 123 as the request number where it should be 1) and should be added dynamically by clients in their requests.
The relevant test has been also updated.

Checkstyle issues in YoutubeDashManifestCreator have been fixed, and the changes in the resolution string returned for video streams in YoutubeStreamExtractor have been reverted, as they create issues on NewPipe right now.
2022-05-28 12:00:47 +02:00
TiA4f8R
f61e2092a1
[YouTube] Return a copy of the hardcoded ItagItem instead of returning the reference to the hardcoded one in ItagItem.getItag
To do so, a copy constructor has been added in the class.

This fixes, for instance, an issue in NewPipe, in which the ItagItem values where not the ones corresponsing to a stream but to another, when generating DASH manifests.
2022-05-28 12:00:47 +02:00
TiA4f8R
aa4c10e751
Improve documentation and adress most of the requested changes
Also fix some issues in several places, in the code and the documentation.
2022-05-28 12:00:46 +02:00
TiA4f8R
6985167e63
Add tests for YoutubeDashManifestCreator and ManifestCreatorCache
The test added in YoutubeDashManifestCreator uses a video of the Creative Commons channel, licenced under the Creative Commons Attribution licence (reuse allowed).
Also remove public keywords of tests in UtilsTest, as suggested by SonarLint, because they are not needed with Junit 5.
2022-05-28 12:00:46 +02:00
TiA4f8R
f6ec7f9a61
Update DefaultStreamExtractorTest and SoundcloudStreamExtractorTest to support changes made in Stream classes 2022-05-28 12:00:45 +02:00
TiA4f8R
7477ed0f3d
[YouTube] Add ability to generate manifests of progressive, OTF and post live streams
A new class has been added to do so: YoutubeDashManifestCreator.
It relies on a new class: ManifestCreatorCache, to cache the content, which relies on a new pair class named Pair.
Results are cached and there is a cache per delivery type, on which cache limit, clear factor, clearing and resetting can be applied to each cache and to all caches.
Look at code changes for more details.
2022-05-28 12:00:45 +02:00
TiA4f8R
a857684442
Apply changes in YoutubeStreamExtractor
Extract post live DVR streams as post live streams instead of live streams.

A new class has been in order to improve code: ItagInfo, which stores an itag, the content (URL) extracted and if its an URL or not.
A functional interface has been added in order to abstract the stream building: StreamBuilderHelper.
Also add the cver parameter added by the desktop web client on the corresponding streams (a new method has been added in YoutubeParsingHelper to check this and another for Android streams).

Some code in these classes has been also refactored/improved/optimized.
2022-05-28 12:00:44 +02:00
TiA4f8R
4330b5f7be
Add POST_LIVE_STREAM and POST_LIVE_AUDIO_STREAM stream types
This allows the extractor to determine if a content is an ended audio or video livestream.
2022-05-28 12:00:43 +02:00
TiA4f8R
881969f1da
Apply changes in all StreamExtractors except YouTube's one and fix extraction of PeerTube audio streams as video streams
Some code in these classes has been also refactored/improved/optimized.
Also fix the extraction of PeerTube audio streams as video streams, which are now returned as audio streams.
2022-05-28 12:00:43 +02:00
TiA4f8R
d5f3637fc3
[YouTube] Return more values returned inside the ItagItems of the player response and deprecate use of public audio and video fields
These fields can be now replaced by a getter and a setter.

New fields have been added and will allow the creation of DASH manifests for OTF and ended livestreams. There are:
- contentLength;
- approxDurationMs;
- targetDurationSec;
- sampleRate;
- audioChannels.
2022-05-28 12:00:42 +02:00
TiA4f8R
7c67d46e09
Move DashMpdParser to the YouTube package and fix extraction of streams
DashMpdParser is only working with YouTube streams, as it uses the ItagItem class.
Also update creation of AudioStreams and VideoStreams objects.
2022-05-28 12:00:41 +02:00
TiA4f8R
ad993b920f
Remove fetching of the DASH manifest extracted when getting information of a content with StreamInfo
DashMpdParser is only working with YouTube streams, as it uses the ItagItem class.

Also improve code and comments of StreamInfo (especially final use where possible).
2022-05-28 12:00:41 +02:00
TiA4f8R
2f061b8dbd
Add support of other delivery methods than progressive HTTP in Stream classes
Stream constructors are now private and streams can be constructed with new Builder classes per stream class. This change has been made to prevent creating and using several constructors in stream classes.

Some default cases have been also added in these Builder classes, so not everything has to be set, depending of the service and the content.
2022-05-28 12:00:27 +02:00
Stypox
1dc80957d8
Merge pull request #848 from TiA4f8R/fix_yt_music_website_extraction_eu
[YouTube] Fix extracting YouTube Music client version and key in the EU
2022-05-22 22:05:05 +02:00
TiA4f8R
c34b5e3a8b
[YouTube] Fix extraction of YouTube Music client version and API key when using YouTube Music's website in EU
Google returns now the consent page of YouTube for YouTube Music in EU, which can be also avoided by adding the ucbcb parameter to the URL with the value 1 ("?ucbcb=1").
2022-05-15 11:20:06 +02:00
litetex
2015eb374a Removed more unused methods 2022-05-09 21:05:03 +02:00
litetex
f69b0ff77b Remove unused methods 2022-05-09 20:59:25 +02:00
Stypox
23fa31a1ec
Merge pull request #843 from TiA4f8R/fix-checkstyle-issues
Remove Checkstyle suppressions file and fix Checkstyle issues
2022-05-02 22:07:27 +02:00
TiA4f8R
3c3cd78676
Remove Checkstyle suppressions file and fix Checkstyle issues introduced in 24e8399 and 8c1041d
The Checkstyle suppressions file is now replaced by // CHECKSTYLE:OFF and // CHECKSTYLE:ON comments.
2022-05-02 21:51:25 +02:00
Stypox
2e1c5c119d
Merge pull request #822 from Stypox/more-refactors
More refactors
2022-05-02 19:03:54 +02:00
Stypox
598ebb92ea
Merge pull request #839 from TeamNewPipe/bandcamp/extract-length
Bandcamp: extract stream length
2022-05-02 15:49:41 +02:00
litetex
5db4d1faf3
Merge pull request #782 from litetex/cleanup-yt-stream-extractor
Cleanup of ``YoutubeStreamExtractor`` and some related classes
2022-05-01 16:44:11 +02:00
litetex
fe30eb43a9 Cleanup `YoutubeStreamExtractor` and some related classes
* Fixed obvious sonar(lint) warnings
* Abstracted some code (get*Streams)
* Used some new lines to make code better readable
* Chopped down brace-jungle in some methods
* Use StandardCharset (Java 8 4tw)
2022-05-01 16:39:07 +02:00
Stypox
c2b5370517
Apply suggestions: improve switch and use EMPTY_STRING 2022-04-30 16:39:51 +02:00
Stypox
7c78c39230
Merge pull request #821 from litetex/cleanup-TimeAgoParser-java
Cleanup ``TimeAgoParser``
2022-04-30 16:20:09 +02:00
Stypox
ac1c22d81c
Release NewPipe Extractor 0.22.1 2022-04-28 11:05:52 +02:00
Stypox
3f5c8962ac
Merge pull request #841 from TiA4f8R/yt-respect-order-of-clients-for-streams
[YouTube] Use correct order of clients to get better streams first
2022-04-28 10:58:49 +02:00
TiA4f8R
9f9af35adb
[YouTube] Fix regression introduced in the order of streams used when adding more parameters to InnerTube requests, using the iOS client for livestreams and more 2022-04-25 20:23:04 +02:00
Fynn Godau
c38c016de5 Bandcamp: extract stream length 2022-04-24 21:24:19 +02:00
litetex
5f8f3929bd
Merge pull request #838 from Nickoriginal/update_user_agent
Update User-Agent in NPE
2022-04-24 14:52:18 +02:00
Nickoriginal
b5f3f9eb90
Update USER_AGENT to match the latest Firefox 2022-04-24 00:27:23 +03:00
litetex
9f665db4e1
Merge pull request #828 from TeamNewPipe/dependabot/github_actions/actions/setup-java-3
Bump actions/setup-java from 2 to 3
2022-04-23 00:09:44 +02:00
Stypox
52fa2d939a
Fix javadoc formatting error causing deployment to fail 2022-04-16 17:07:07 +02:00
Stypox
95a4b05acc
Release NewPipe Extractor 0.22.0 2022-04-16 16:57:58 +02:00
Tobi
b77c72fb88
Merge pull request #832 from Stypox/hotfix-decrypter
Fix YouTube throttling decrypter function parsing
2022-04-15 13:35:10 +02:00
Stypox
dcb7483dcf
Fix YouTube throttling decrypter function parsing 2022-04-15 13:10:19 +02:00
Robin
bebdc55ad4
Merge pull request #829 from TiA4f8R/yt-subtitles-age-restricted-videos-fix
[YouTube] Extract subtitles for age-restricted videos
2022-04-14 10:31:16 +02:00
TiA4f8R
ef49cd0007
[YouTube] Extract subtitles for age-restricted videos
Subtitles of age-restricted videos can be extracted since the InnerTube API migration, so there is no reason to not extract them anymore.
2022-04-11 22:09:56 +02:00
dependabot[bot]
00b51a2bf8
Bump actions/setup-java from 2 to 3
Bumps [actions/setup-java](https://github.com/actions/setup-java) from 2 to 3.
- [Release notes](https://github.com/actions/setup-java/releases)
- [Commits](https://github.com/actions/setup-java/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/setup-java
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-04-11 09:16:36 +00:00
TiA4f8R
2e92d718a2
Merge pull request #780 from TiA4f8R/yt-more-params-innertube-requests
[YouTube] Add more parameters to InnerTube requests, use the iOS client for livestreams and fix extraction of embeddable age-restricted videos and contents with a warning before playback
2022-04-09 18:15:18 +02:00
TiA4f8R
b30e341559
Add link to learn more about the issue which makes YoutubeStreamExtractorRelatedMixTest.testRelatedItems test disabled 2022-04-04 20:36:59 +02:00
TiA4f8R
70812fa611
Update YouTube stream mocks and disable YoutubeStreamExtractorRelatedMixTest
Mixes seems to be not given by YouTube anymore if you use a PENDING consent cookie value.
As mocks needs to updated, the test is always failing because of this change.
2022-04-02 19:28:49 +02:00
TiA4f8R
67288a0191
[YouTube] Fix extraction of embeddable age-restricted videos, fix extraction of contents with warnings and more
Use the TV embedded client technique to get streams of embeddable age-restricted videos.

This client doesn't provide the playerMicroFormatRenderer object in the player response, but it is still returned on the WEB player response, even for unavailable (but non-private) contents, so we need now to store it, as we are replacing the player response from the WEB client by the TV embedded one.
Otherwise, some metadata such as the unlisted property, category, the uploadDate and the publishDate properties.

The outdated code for these contents has been removed.

Add the racyCheckOk and contentCheckOk to player and next requests to the InnerTube API.
The first doesn't seem to make any difference when used anonymously, but the second one is needed to get streams of contents with a warning before they can be played.

Also apply some requested changes, fixes and improvements in YoutubeParsingHelper and YoutubeStreamExtractor.
2022-04-02 19:06:36 +02:00
TiA4f8R
11b5a222c4
Deduplicate code of getStringResultFromRegexArray methods in Utils
Also revert indentation in Utils.mixedNumberWordToLong.
2022-04-02 18:40:00 +02:00
TiA4f8R
9ca647a750
Update mocks 2022-03-27 22:10:59 +02:00
TiA4f8R
dfa4239661
Fix missing imports and Checkstyle issues 2022-03-27 22:10:57 +02:00
TiA4f8R
6d27996ac4
Improve code of getStringResultFromRegexArray methods in Utils 2022-03-27 22:10:57 +02:00
TiA4f8R
2e3da445e6
[YouTube] Add documentation about parameters added and clients versions and key
Also move the iPhone device machine id to a constant, explain how it is used and move the licence in the header of the file, and fix missing imports in YoutubeStreamExtractor (due to a rebase issue).
2022-03-27 22:10:57 +02:00
TiA4f8R
1dad3bfe8b
[YouTube] Update again hardcoded client versions and update mobile user agents
Also provide ability to get mobile user-agents used for mobile InnerTube requests and deduplicate related code.
2022-03-27 20:52:40 +02:00
TiA4f8R
3d38459cf3
[YouTube] Reduce InnerTube response sizes by adding the prettyPrint parameter with the false value
InnerTube responses return pretty printed responses, which increase responses' size for nothing.

By using the prettyPrint parameter on requests and setting its value to false, responses are not pretty printed anymore, which reduces responses size, and so data transfer and processing times.
This usage has been recently deployed by YouTube on their websites.
2022-03-27 20:52:40 +02:00
litetex
349ba8db7f
Improve tests and randomness
- Use the existing RNG inside YoutubeParsingHelper
- Deduplicated test-setup for YouTube tests
- Minor improvements
2022-03-27 20:52:38 +02:00
TiA4f8R
52376949e5
Add setSeedForVideoTests method in YoutubeStreamExtractor tests
In order to use still use mocks with the generation of random strings in player requests, we need to use YoutubeParsingHelper.setSeedForVideoTests() method in every stream test.
2022-03-27 20:51:39 +02:00
TiA4f8R
d0d91e6690
Adress requested changes 2022-03-27 20:51:39 +02:00
TiA4f8R
b6bc521f0d
[YouTube] Update client versions again 2022-03-27 20:51:38 +02:00
TiA4f8R
26f93f5bb0
[YouTube] Extract streams of livestreams from the iOS client and disabled the Android client for livestreams
The iOS client is only enabled for livestreams and the Android client is now only enabled for videos, both by default.

A way to force, or not, the fetch of both clients have been added with two new static methods in YoutubeStreamExtractor.
2022-03-27 20:51:38 +02:00
TiA4f8R
7d07924de8
[YouTube] Try to use lighter requests when extracting client version and key from YouTube and YouTube Music
This is done by fetching https://www.youtube.com/sw.js for YouTube and https://music.youtube.com/sw.js for YouTube Music.

Two new methods in Utils class have been added which allow to try to get a match of regular expressions in a string array, or a Pattern array, on a content, on a specific index or 0.
Also some code refactoring has been made in this class.
2022-03-27 20:51:38 +02:00
TiA4f8R
05b7fee23b
[YouTube] Add the cpn param to playback requests and try to spoof better the Android client
The cpn param, aka the content playback nonce param, is a parameter sent by YouTube web client in videoplayback requests, and for some of them, in the player request body. This PR adds it everywhere.

For the desktop/WEB client, some params were missing from the playbackContext object, which seemed (or not) to make YouTube throttle streams extracted from the WEB client. This PR adds them.

Fingerprinting on the WEB client basing on the client version used is not possible anymore, because the latest client version is extracted at the first time of a YouTube request on a session which require the extractor to fetch again the website (and this may come back the reCaptcha issues again unfortunately, but it seems there is no other way to get it).

For the Android client, the video id is now also sent as a query parameter, like a 12 characters string, in the t query parameter, in order to spoof better this client. Researches need to be done on this parameter, unique to each request, and how it is generated by clients.

This commit also fixes a small bug with the Android User-Agent string.

Some code improvements have been also made.
2022-03-27 20:51:38 +02:00
TiA4f8R
83f374bff1
[YouTube] Update client versions and fix a bug when using resetClientVersionAndKey method
The boolean keyAndVersionExtracted in YoutubeParsingHelper was not set to false when resetting the client version and the key, which makes the extractor uses null on the next getting of the client version or the key if the clientVersion and the key were extracted before.
Also update client versions.
2022-03-27 20:51:38 +02:00
Stypox
8c1041def6
Add @ null annotations where Android Studio suggested it
That is, basically where the overriding function was missing an annotation from the base method.

Also apply renaming of emptyDescription to EMPTY_DESCRIPTION
2022-03-26 22:07:14 +01:00
Stypox
adbbdc7a5b
[YouTube] Fix regex warning: use ' {2}' instead of ' ' 2022-03-26 22:07:14 +01:00
Stypox
24e83997b4
[Bandcamp] Add Java 8 streams 2022-03-26 22:07:12 +01:00
Stypox
349990fd48
Fix redundant escape \\ in regex in Utils 2022-03-26 22:01:30 +01:00
litetex
3bf7aa3762 Cleanup `TimeAgoParser` 2022-03-26 21:09:31 +01:00
litetex
5a18730845
Merge pull request #813 from litetex/fix-test-2022-03
Fixed tests (+ Youtube shorts in channels)
2022-03-26 21:09:10 +01:00
litetex
af82edf9dc Fix checkstyle problems 2022-03-26 20:54:20 +01:00
litetex
29408f9356 `YoutubeMusicSearchExtractorTest$Suggestion` has no mock data 2022-03-26 20:52:28 +01:00
litetex
fcee247f9f Switched to mockonly tests
* Made the reason why the tests are mockonly mandatory
2022-03-26 20:52:28 +01:00
litetex
0fceb4686b Added missing mock data 2022-03-26 20:52:27 +01:00
litetex
70b20f0d6b Updated mock data after conflicts 2022-03-26 20:52:27 +01:00
litetex
6a6c9359af Fixed kurzgesagt test - They changed their description... 2022-03-26 20:52:27 +01:00
litetex
26596215fa Refactored `MediaCCCRecentListExtractorTest` 2022-03-26 20:52:26 +01:00
litetex
53962bfd7d Disabled suggestion test for now
Has to be done once YT's backed works again...
2022-03-26 20:52:26 +01:00
litetex
66dc5e8bb8 API hardening against changes 2022-03-26 20:52:26 +01:00
litetex
5d58156cde Update mock data 2022-03-26 20:52:25 +01:00
litetex
7598b40957 Workaround for incorrect duration for "YT shorts" videos in channels
As a workaround 0 is returned as duration for such videos.
See also https://github.com/TeamNewPipe/NewPipe/issues/8034
2022-03-26 20:52:24 +01:00
litetex
164e21b5af Fixed `MediaCCCRecentKiosk`
Ignore faulty data/items (with duration <= 0)
2022-03-26 20:52:23 +01:00
litetex
49dfdae5d2 Removed useless throw declaration 2022-03-26 20:51:38 +01:00
litetex
69a58fd3d1 Don't use sysout 2022-03-26 20:51:37 +01:00
litetex
639be7adda Minimized some code 2022-03-26 20:51:37 +01:00
litetex
2bd4299563 Fixed test: Peertube - Use new comments and videos
Old comments/videos got deleted
2022-03-26 20:49:39 +01:00
litetex
606a386a98 Fix tests: What's peertube has less comments now 2022-03-26 20:49:39 +01:00
litetex
4d1a1c8fb8 Better test for `MediaCCCRecentListExtractorTest`
* Use assertAll
* Show which item is affected
2022-03-26 20:49:38 +01:00
litetex
ba43dbaa28 Fix Tests: The channel lost abos 2022-03-26 20:49:38 +01:00
litetex
ff436e5740 Fixed test: YT mix playlist 2022-03-26 20:49:38 +01:00
litetex
bb49f7d857
Merge pull request #817 from Stypox/checkstyle
Add checkstyle
2022-03-26 20:46:49 +01:00
litetex
9c07e8a664 Fix useage of wrong object 2022-03-26 20:17:50 +01:00
litetex
93d6e5c4d5 CI: Upload test reports when failure occurs 2022-03-26 19:57:17 +01:00
litetex
804e57004f Fixed new checkstyle problems from dev 2022-03-26 19:46:10 +01:00
litetex
33347ac18b Removed unused methods
``contentFilters`` and ``sortfilter`` are get inside the ``ListLinkHandler`` and not the ``ListLinkHandlerFactory``
 ``ListLinkHandlerFactory`` only passes these values through when ``fromQuery`` is called
2022-03-26 19:43:11 +01:00
litetex
ec5b54c38b Removed unused class 2022-03-26 19:43:10 +01:00
litetex
8771af7ba5 Restored original naming 2022-03-26 19:43:09 +01:00
litetex
01cfde0285 Fixed typo 2022-03-26 19:43:09 +01:00
Stypox
bdadcfa1f7 Legitimately suppress remaining checkstyle warnings 2022-03-26 19:43:08 +01:00
Stypox
740a37a2de [YouTube] Fix checkstyle issues 2022-03-26 19:42:40 +01:00
Stypox
9dc17cd1ca [Soundcloud] Fix checkstyle issues 2022-03-26 19:40:20 +01:00
Stypox
9ab32cb2e7 [Peertube] Fix checkstyle issues 2022-03-26 19:40:19 +01:00
Stypox
9f7e06c817 [MediaCCC] Fix checkstyle issues 2022-03-26 19:40:18 +01:00
Stypox
3a94839359 [Bandcamp] Fix checkstyle issues 2022-03-26 19:40:17 +01:00
Stypox
08dff33002 Use Java 8 streams in NewPipe class 2022-03-26 19:40:15 +01:00
Stypox
c2446ecff0 Use Java 8 streams and deduplicate code in MediaFormat class 2022-03-26 19:40:15 +01:00
Stypox
d79e20340c Fix checkstyle issues in root package extractor/
Note: not all issues were fixed because MediaFormat and ServiceList use a specific formatting that makes sense for them
2022-03-26 19:40:14 +01:00
Stypox
ca7c63f273 Fix remaining checkstyle issues in utils/ subpackage 2022-03-26 19:40:13 +01:00
Stypox
1d5f22e41f Fix checkstyle issues & more in JsonUtils
Also use Java 8 streams and extract duplicate code to getInstanceOf function
2022-03-26 19:40:13 +01:00
Stypox
87d2834986 Fix checkstyle issues & more in DonationLinkHelper
Also add comment about the class being unused and replace the fixLink function with Utils.stringToUrl()
2022-03-26 19:40:12 +01:00
Stypox
bd7b362040 Fix checkstyle issues & more in DashMpdParser
Also remove useless null check on ItagItem.getItag() as that function already throws an exception if there is no itag
2022-03-26 19:40:11 +01:00
Stypox
8aba2b47b0 Fix checkstyle issues in subpackages with abstract classes 2022-03-26 19:40:10 +01:00
Stypox
e4951a0623 Refactor code handling http headers in downloader.Request 2022-03-26 19:37:47 +01:00
Stypox
37690058d2 Add checkstyle to extractor gradle project
With respect to NewPipe's checkstyle.xml, checkstyle is disabled for javadoc comments. There is no need for strict rules over comments here in the extractor, as sometimes javadocs are just needed to clarify a small thing and having empty/meaningless @param or @throws is useless.
2022-03-26 19:37:46 +01:00
litetex
9284569c84
Merge pull request #774 from TeamNewPipe/dependabot/gradle/org.mozilla-rhino-1.7.14
Bump rhino from 1.7.13 to 1.7.14
2022-03-26 17:25:55 +01:00
litetex
358e619a27
Merge pull request #819 from TeamNewPipe/dependabot/github_actions/actions/cache-3
Bump actions/cache from 2 to 3
2022-03-23 21:02:18 +01:00
dependabot[bot]
a2c9ad104c
Bump actions/cache from 2 to 3
Bumps [actions/cache](https://github.com/actions/cache) from 2 to 3.
- [Release notes](https://github.com/actions/cache/releases)
- [Commits](https://github.com/actions/cache/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-21 09:12:36 +00:00
XiangRongLin
aa6b7272a4
Merge pull request #804 from Stypox/fix-yt-music-mix
[YouTube] Fix music mixes in some countries
2022-03-20 08:35:56 +01:00
Stypox
843945f2a6
Merge pull request #818 from litetex/update-license
Update the license to the latest version
2022-03-19 23:13:55 +01:00
litetex
fa76098750
Delete copyright
was replace by LICENSE-file
2022-03-19 21:44:38 +01:00
litetex
faaf4cbec3
Update the license to the latest version of https://www.gnu.org/licenses/gpl-3.0.txt 2022-03-19 17:59:27 +01:00
XiangRongLin
7f2ea133f0
Merge pull request #788 from Stypox/mix
Add MixInfoItem and extract YouTube mixes in related items
2022-03-19 16:46:01 +01:00
Stypox
09ddb6adbb
[YouTube] Add MockOnly to method testing mixes in related items 2022-03-19 10:54:38 +01:00
Stypox
8201b3b90e
[YouTube] Parse any playlist (including music mixes) in related items 2022-03-19 10:48:13 +01:00
Stypox
13f7900816
[YouTube] Add test for genre mix 2022-03-19 10:48:13 +01:00
Stypox
279f3a20fe
[YouTube] Fix mix tests with invalid video ids
Replaces mix tests based on a strange mix type RDQM{videoId} (only reference I could find is https://github.com/ytdl-org/youtube-dl/issues/26228) and with an invalid video id of 13 characters (the first two characters were QM, but even after removing QM there still wasn't a video available at that id).
Also updates mocks.
2022-03-19 10:48:13 +01:00
Stypox
d660c04838
[YouTube] Also test playlist type in playlist tests 2022-03-19 10:48:13 +01:00
Stypox
401082abe4
[YouTube] Extract playlist type in playlist extractor 2022-03-19 10:48:12 +01:00
Stypox
63ed06a710
[YouTube] Differentiate genre mixes from normal mixes
Note: genre mixes already worked, now they are just considered as such in various video id extraction and in related items
Note 2: now extracting a mix id from a *normal* youtube mix id will fail if the video id wouldn't be exactly 11 characters long
2022-03-19 10:46:31 +01:00
Stypox
f19660e7d9
[YouTube] Deduplicate code extracting video id from mix id 2022-03-19 10:46:30 +01:00
Stypox
8f9d5b858e
[YouTube] Remove useless comments about mixes 2022-03-19 10:44:06 +01:00
Stypox
34a4484c72
[YouTube] Add test for a video with a mix in related items 2022-03-19 10:44:06 +01:00
Stypox
50db871d89
[YouTube] Extract mixes from streams related items 2022-03-19 10:44:06 +01:00
Stypox
638da1756c
[Mix] Create MultiInfoItemsCollector
It is a collector that can handle many extractor types, to be used when a list contains items of different types (e.g. search). It was renamed from InfoItemsSearchCollector so that it can now be used not just for search but for any extractor needing it. It supports, streams, channels, playlists and *mixes*.
2022-03-19 10:44:06 +01:00
Stypox
53673d64c6
[Mix] Add type to playlists & playlist items, to distinguish mixes 2022-03-19 10:44:06 +01:00
Stypox
d8f2031619
Merge pull request #816 from Stypox/mock-only-extension
Add `@MockOnly` Junit 5 extension
2022-03-19 10:40:38 +01:00
litetex
cc2e4d7104
Merge pull request #815 from litetex/fix-soundcloud-id-once-and-for-all
Removed hardcoded soundcloud HARDCODED_CLIENT_ID
2022-03-17 13:54:08 +01:00
litetex
cd8088b217
Merge pull request #811 from TiA4f8R/playlists-improvements-and-yt-playlists-fixes
[YouTube] Fix the extraction of series playlists and don't return the view count as the stream count for learning playlists
2022-03-17 13:52:24 +01:00
TiA4f8R
c7757c0994
Apply requested changes 2022-03-16 20:14:08 +01:00
TiA4f8R
35e082248e
Fix YouTube and SoundCloud playlists tests 2022-03-16 19:40:30 +01:00
TiA4f8R
8b3f90eb7e
[YouTube] Fix extraction of series playlists and don't return the view count as the stream count for learning playlists
ITEM_COUNT_UNKNOWN is returned when the JSON array which contains usally the number of videos is less than 3 items.
Also apply the same type of optimizations done in other PlaylistExtractors in YoutubePlaylistExtractor.
2022-03-16 19:18:58 +01:00
TiA4f8R
58a247907e
Apply changes in all playlist extractors except YoutubePlaylistExtractor
Also fix some issues in the extractors, remove uneeded overrides, use the Java 8 Stream API where possible and replace usages of Utils.UTF_8 with StandardCharsets.UTF_8 in these classes.
2022-03-16 19:18:57 +01:00
TiA4f8R
fc6b45ee36
Implement some methods in PlaylistExtractor
This will prevent their override in each child class where the values corresponding to the methods could not be extracted.
2022-03-16 19:18:36 +01:00
Stypox
73d1fd472f
Add MockOnly junit 5 test extension 2022-03-16 19:03:08 +01:00
Stypox
ef71a5fa0f
static final instead of final static 2022-03-16 17:24:33 +01:00
Stypox
0c37c75981
Make getDownloader static & extract getDownloaderType 2022-03-16 17:22:42 +01:00
Stypox
40aa5104b1
Merge pull request #786 from XiangRongLin/throttling_resilience
[Youtube] Make throttling decryption more resilient to api change
2022-03-16 11:03:16 +01:00
litetex
ba56be8ef1 Removed hardcoded soundcloud id
It never works (long enough) so let's simply remove it...
2022-03-15 21:19:19 +01:00
XiangRongLin
e726437da3
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java
Co-authored-by: Stypox <stypox@pm.me>
2022-03-15 17:10:05 +01:00
litetex
e7aee0ca57
Merge pull request #807 from FireMasterK/no-commentsinfo-instance
Remove the need for a CommentsInfo instance in CommentsInfo.getMoreItems and fix PeertubeCommentsExtractorTest.Default test
2022-03-15 15:06:56 +01:00
litetex
d806984aa8
Merge pull request #800 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.9.0
Bump gson from 2.8.9 to 2.9.0
2022-03-14 19:47:54 +01:00
litetex
2aa5f98c26
Merge pull request #796 from FireMasterK/streaminfo-subscriber-count
Add support for extracting channel subscriber count in StreamInfo
2022-03-14 19:40:57 +01:00
Tobi
a267093e76
Merge pull request #809 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.6.0
Bump spotbugs-annotations from 4.5.3 to 4.6.0
2022-03-12 00:02:51 +01:00
dependabot[bot]
fdb603e9ac
Bump spotbugs-annotations from 4.5.3 to 4.6.0
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.5.3 to 4.6.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.5.3...4.6.0)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-08 09:16:28 +00:00
FireMasterK
60cc71e944
Remove the need for a CommentsInfo instance. 2022-03-03 11:48:41 +00:00
Stypox
3a6269bca6
Merge pull request #805 from TeamNewPipe/dependabot/github_actions/actions/checkout-3
Bump actions/checkout from 2 to 3
2022-03-02 11:09:28 +01:00
dependabot[bot]
e953794703
Bump actions/checkout from 2 to 3
Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-03-02 09:25:59 +00:00
Stypox
dd8687f9fe
[YouTube] Fix music mixes in some countries 2022-03-01 23:02:56 +01:00
FireMasterK
e6d334765d
Apply requested codestyle improvements. 2022-02-26 17:12:51 +00:00
FireMasterK
6950b362f2
Add test values for failing tests. 2022-02-24 12:55:13 +00:00
FireMasterK
5b0ec694a6
Add requested changes. 2022-02-24 12:50:51 +00:00
FireMasterK
d290d2e393
Move variable to super Extractor classes. 2022-02-24 12:50:51 +00:00
FireMasterK
ab49cb6e18
Add requested changes. 2022-02-24 12:50:51 +00:00
Kavin
4bd59f65f4
Update exception message.
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2022-02-24 12:50:51 +00:00
FireMasterK
6f3f608ab6
Add support for extracting channel subscriber count in StreamInfo. 2022-02-24 12:50:51 +00:00
Stypox
69e18c80cb
Merge pull request #799 from litetex/imporve-yt-comments-extractor
Refactored YoutubeCommentsExtractor
2022-02-23 17:58:00 +01:00
litetex
1a67ea100a Refactored code
according to review
2022-02-21 19:56:12 +01:00
litetex
f79ce1f52a Refactored YoutubeCommentsExtractor
* Use Java Streaming API
* Use StandardCharsets
* Prevented several NPEs/ArrayIndexOutOfBound
* Reformatted some code so that it's easier readable
2022-02-21 19:56:09 +01:00
TobiGr
dfe8716f5f Merge branch 'dev' 2022-02-18 20:30:38 +01:00
TobiGr
d07e16aadd Release NewPipe Extractor 0.21.14 2022-02-18 20:29:57 +01:00
TobiGr
d337e537dc [YouTube] Add a comment clarifying why the panelIdentfier can be null 2022-02-18 20:26:19 +01:00
Tobi
5c8fbeb166
Fix NullPointerException in YouTubeStreamExtractor.getStreamSegments() (#797)
Fix NullPointerException in YouTubeStreamExtractor.getStreamSegments()

Co-authored-by: litetex <40789489+litetex@users.noreply.github.com>
2022-02-16 23:39:26 +01:00
dependabot[bot]
89900431ab
Bump gson from 2.8.9 to 2.9.0
Bumps [gson](https://github.com/google/gson) from 2.8.9 to 2.9.0.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.9...gson-parent-2.9.0)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-14 09:15:09 +00:00
Tobi
0d6aebce25
[SoundCloud] Update clientId (#798) 2022-02-11 17:16:32 +01:00
dependabot[bot]
a2a41e3bf2
Bump rhino from 1.7.13 to 1.7.14
Bumps [rhino](https://github.com/mozilla/rhino) from 1.7.13 to 1.7.14.
- [Release notes](https://github.com/mozilla/rhino/releases)
- [Changelog](https://github.com/mozilla/rhino/blob/master/RELEASE-NOTES.md)
- [Commits](https://github.com/mozilla/rhino/commits)

---
updated-dependencies:
- dependency-name: org.mozilla:rhino
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-02-07 20:29:51 +00:00
XiangRongLin
545522b80f [Youtube] Make throttling decryption more resilient to api change 2022-02-01 19:57:34 +01:00
TobiGr
ebc129c392 NewPipe Extractor 0.21.13 2022-02-01 13:20:34 +01:00
TobiGr
5306f7a988 Improve code style 2022-02-01 13:18:18 +01:00
Abdu Ameen
38e9da8d44 Adding removal of the braces 2022-02-01 13:18:10 +01:00
Abdu Ameen
36ebb633a4 Changed the regex to account for
nonword characters
2022-02-01 13:18:05 +01:00
Tobi
65129e65a8
Merge pull request #785 from AbduAmeen/regex_error
Regex error when parsing the Youtube JavaScript code
2022-02-01 13:17:19 +01:00
TobiGr
047d75a4c9 Improve code style 2022-02-01 12:56:59 +01:00
Abdu Ameen
50a4b026f8 Adding removal of the braces 2022-01-31 23:52:31 -08:00
Abdu Ameen
a9174f6b9f Changed the regex to account for
nonword characters
2022-01-31 22:29:00 -08:00
dependabot[bot]
a4399fd4ad Bump spotbugs-annotations from 4.5.2 to 4.5.3
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.5.2 to 4.5.3.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.5.2...4.5.3)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-01-24 18:45:21 +01:00
Stypox
59e0aeba88
Merge pull request #768 from litetex/junit-5
JUnit 5
2022-01-24 16:56:16 +01:00
Mohammed Anas
65df39b3e3
[YouTube] parse timestamps with >3 digits correctly (#775)
* [YouTube] parse timestamps with >3 digits correctly

Fixes https://github.com/TeamNewPipe/NewPipe/issues/7530; check the
issue for details.

* Remove outdated comment
2022-01-12 10:45:06 +01:00
litetex
a579ef2651 Fixed Soundcloud tests 2022-01-07 14:27:24 +01:00
litetex
a32edce750 Removed deprecation because AndroidSDK/API < 19 2022-01-07 14:15:55 +01:00
litetex
602068937d
Code cleanup
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2022-01-06 21:04:16 +01:00
litetex
dd3f3da828 Kurzgesagt mock data is outdated 2022-01-04 17:44:12 +01:00
litetex
a8ebd2cead Update mock resources for YT Mix playlist 2022-01-04 17:39:14 +01:00
litetex
066afd8629 Restored test 2022-01-04 17:33:08 +01:00
litetex
3712a669b1 Code cleanup / improvements
* Deprecated Utils#UTF-8; see StandardCharsets
* Added more helpful methods to ``ExtractorAsserts``
* Use parameterized (cool new) tests
* Restore functionality of some tests + updated mockdata
* Other code cleanups + Sonarlint improvements
2022-01-04 17:28:31 +01:00
litetex
4291a90251 Fixed SoundCloud's search(for tests) 2021-12-27 21:42:43 +01:00
litetex
4995709871 Fixed SoundCloud's search(for tests)
Getting the initial page was not returning initial page
2021-12-27 21:18:39 +01:00
litetex
a2cbdf0991 Updated to JUnit 5 2021-12-27 21:08:08 +01:00
litetex
10f6cc7194
Merge pull request #755 from litetex/clean-up-yt-dis-likes
Cleanup: Remove old ways of getting YT dis/likes
2021-12-27 18:32:02 +01:00
litetex
148ba6641d Updated mock data 2021-12-27 17:08:14 +01:00
litetex
4d0a689cb6 Disabled dislike count in tests 2021-12-27 17:04:46 +01:00
litetex
a6961c5efa Fixed NumberFormatException when processing likes
See also #7525
2021-12-27 16:54:47 +01:00
litetex
65687f3b9b Removed YT dislike count as it no longer works 2021-12-27 16:54:46 +01:00
litetex
1e2e0029fc [StreamExtractor] Deduplicated a ton of code by using default methods 2021-12-27 16:54:45 +01:00
litetex
15b98ffdb4 Remove old ways of getting YT dis/likes
* Added additional check for averageRating (in dislikes)
2021-12-27 16:54:44 +01:00
dependabot[bot]
fe432425df Bump spotbugs-annotations from 4.5.1 to 4.5.2
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.5.1 to 4.5.2.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.5.1...4.5.2)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-12-14 10:35:32 +01:00
dependabot[bot]
36663dcb37 Bump spotbugs-annotations from 4.5.0 to 4.5.1
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.5.0 to 4.5.1.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.5.0...4.5.1)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-12-12 13:28:30 +01:00
litetex
12a6a48b15
Merge pull request #761 from litetex/fix-soundcloud-georestricted
Disabled ``SoundcloudStreamExtractorTest#SoundcloudGeoRestrictedTrack#testRelatedItems`` as it's unreliable
2021-12-11 18:06:17 +01:00
litetex
7484b6a098
Merge pull request #758 from litetex/release/v0.21.12
Release 0.21.12
2021-12-11 16:14:11 +01:00
litetex
795e6df596
Update README.md
Made usage section more abstract
2021-12-11 16:06:29 +01:00
litetex
4dd737c9a0 Updated version to v0.21.12 2021-12-11 15:34:46 +01:00
litetex
7489502cc8 Disabled `testRelatedItems` as it's unreliable 2021-12-11 15:23:11 +01:00
Tobi
1e0c409bec
Merge pull request #760 from litetex/fix-test
Fix test
2021-12-11 12:44:45 +01:00
litetex
d9b2a2042a Updated mock data for YoutubeCommentsExtractorTest 2021-12-08 22:11:45 +01:00
litetex
a3a0c1ee85 Fixed YoutubeCommentsExtractorTest
Old video was deleted / set to private
2021-12-08 22:11:34 +01:00
litetex
5c288eb8d7 Updated mock data for YoutubeChannelExtractorTest 2021-12-08 21:51:38 +01:00
litetex
75bb83a1b8 Fixed YoutubeChannelExtractorTest
Channel was restored, picked a new one
2021-12-08 21:51:15 +01:00
litetex
f276bacce5 Fixed SoundcloudStreamExtractorTest 2021-12-08 21:41:27 +01:00
litetex
7f99c1c193 Fixed YoutubeStreamExtractorLivestreamTest 2021-12-08 21:16:39 +01:00
litetex
652fdf2c36 Updated mock-data for YoutubeStreamExtractorAgeRestrictedTest 2021-12-08 21:08:25 +01:00
litetex
3c669d3cb5 Fixed YoutubeStreamExtractorAgeRestrictedTest
Looks like a lot of stuff (description, tags, name) got changed on the existing video.
Use a better suited video.
2021-12-08 21:08:03 +01:00
litetex
f735788a39 Fixed Soundcloud hardcoded client id 2021-12-08 21:00:53 +01:00
litetex
f6f2724634
Merge pull request #753 from B0pol/dis-likes
Fix YouTube likes + dislikes
2021-11-29 20:42:17 +01:00
bopol
56c8af710e Code refactoring 2021-11-21 18:16:31 +01:00
bopol
c4eca91be9 Fix YouTube likes + dislikes 2021-11-19 21:36:03 +01:00
TobiGr
5028396405 Improve names and formatting 2021-11-14 13:44:13 +01:00
dependabot[bot]
1e0c802d8d Bump spotbugs-annotations from 4.4.2 to 4.5.0
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.4.2 to 4.5.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.4.2...4.5.0)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-11-14 13:43:44 +01:00
Tobi
68d7e4e20a
Merge pull request #742 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.8.9
Bump gson from 2.8.8 to 2.8.9
2021-11-08 19:37:09 +01:00
bopol
8e4f9995bf
Merge pull request #745 from TeamNewPipe/update_SC_hardcoded_client_id
Update hardcoded SoundCloud client_id
2021-11-04 16:33:08 +01:00
opusforlife2
cb80a646d9
Update hardcoded SoundCloud client_id
Hah! I got here first, @TobiGr!
2021-11-04 15:28:26 +00:00
litetex
7e7b78f1b3
Merge pull request #743 from B0pol/peertube-shortlinks
PeerTube short links support
2021-11-04 16:10:23 +01:00
bopol
396aecef19 Make sure playlists aren't accepted by PeertubeStreamLinkHandlerFactory 2021-11-03 14:41:39 +01:00
bopol
38ad1eaac5 [PeerTube] Support /w/ short video links part 2
We also need to support it for comments
2021-11-03 13:54:37 +01:00
bopol
026751624f [PeerTube] Support /w/p/ short playlist links 2021-11-03 13:35:30 +01:00
bopol
e0b8e142fc [PeerTube] Support /a/ and /c/ short links 2021-11-03 13:26:27 +01:00
bopol
599a91c88c [PeerTube] Support /w/ short video links 2021-11-03 11:51:58 +01:00
dependabot[bot]
198d37090b
Bump gson from 2.8.8 to 2.8.9
Bumps [gson](https://github.com/google/gson) from 2.8.8 to 2.8.9.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.8...gson-parent-2.8.9)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 09:12:53 +00:00
litetex
4f60225ddc
Merge pull request #741 from mhmdanas/add-y2ube-link-support
Add support for y2u.be links
2021-10-26 18:43:36 +02:00
mhmdanas
3e8e2a1532 Add support for y2u.be links 2021-10-22 22:48:18 +03:00
Tobi
b425394e7a
Merge pull request #731 from FireMasterK/short-description
Extract Video Short Description in YouTube.
2021-10-15 23:10:00 +02:00
Tobi
4e9d9bffd0
Merge pull request #738 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.4.2
Bump spotbugs-annotations from 4.4.1 to 4.4.2
2021-10-12 12:09:36 +02:00
dependabot[bot]
1867a381a8
Bump spotbugs-annotations from 4.4.1 to 4.4.2
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.4.1 to 4.4.2.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.4.1...4.4.2)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-10-12 09:11:11 +00:00
FireMaskterK
62313962a0 Add default implementation for getShortDescription. 2021-10-04 01:42:29 +01:00
Kavin
d1add05bc1 Add requested changes.
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
Add final.

Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-10-04 01:36:51 +01:00
Tobi
0b0c39aec2
Merge pull request #734 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.14.3
Bump jsoup from 1.14.2 to 1.14.3
2021-10-02 16:49:34 +02:00
TobiGr
8dcb74724a Release NewPipe Extractor 0.21.11 2021-10-02 16:25:35 +02:00
dependabot[bot]
3ca9f51b30
Bump jsoup from 1.14.2 to 1.14.3
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.14.2 to 1.14.3.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.14.2...jsoup-1.14.3)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-10-02 08:20:38 +00:00
TobiGr
d8ec3ce370 [SoundCloud] Update client id 2021-10-01 02:51:38 +02:00
FireMasterK
94efe86c71
Extract Video Short Description in YouTube.
In Trending, and Search results.
2021-09-22 16:26:16 +01:00
Tobi
a9d214478d
Merge pull request #703 from FireMasterK/comment-replies
Add support for extracting comment replies continuation
2021-09-14 23:58:14 +02:00
TobiGr
ce8cabb9f8 Release NewPipe Exttractor 0.21.10 2021-09-13 13:59:35 +02:00
FireMasterK
6aabdc6d16
Fix for requested changes. 2021-09-12 01:15:19 +05:30
Kavin
0aad09fa22
Update JavaDocs for comment replies continuation.
Co-authored-by: Stypox <stypox@pm.me>
2021-09-12 01:05:20 +05:30
Tobi
a031289a63
Merge pull request #727 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.4.1
Bump spotbugs-annotations from 4.4.0 to 4.4.1
2021-09-10 18:25:27 +02:00
dependabot[bot]
82ae322cb8
Bump spotbugs-annotations from 4.4.0 to 4.4.1
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.4.0 to 4.4.1.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.4.0...4.4.1)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-10 09:11:27 +00:00
Tobi
62b87552f5
Merge pull request #725 from FireMasterK/related-uploader-avatar
Extract Channel Avatar in related videos.
2021-09-03 22:14:28 +02:00
FireMasterK
feb09e9997
Extract Channel Avatar in related videos. 2021-09-04 01:09:40 +05:30
Tobi
f0aa46b008
Merge pull request #724 from FireMasterK/streams-uploader-verified
Extract uploaderVerified to StreamInfo.
2021-09-03 18:18:04 +02:00
Tobi
c8037f5e00
Merge pull request #723 from FireMasterK/uploader-avatar
Extract Uploader's Avatar on YouTube and PeerTube
2021-09-03 18:13:59 +02:00
FireMasterK
857b2f39e8
Extract uploaderVerified to StreamInfo. 2021-09-03 21:27:58 +05:30
FireMasterK
1db463b55f
Add a default test for the Uploader's Avatar. 2021-09-03 21:05:36 +05:30
Kavin
db6b3b2c29
Extract uploader's avatar in peertube.
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2021-09-02 16:14:45 +05:30
FireMasterK
a0c1dcc8d8
Remove throws parsing exception. 2021-09-02 16:14:45 +05:30
FireMasterK
0ba03c552c
Improve based on review/feedback. 2021-09-02 00:35:56 +05:30
FireMasterK
b9fad4fcc8
Extract Uploader's Avatar on YouTube. 2021-09-02 00:19:00 +05:30
Stypox
68f1fa994a
Merge pull request #721 from Stypox/yt-csv-import
[YouTube] Small improvements to subscription import
2021-08-31 12:02:59 +02:00
TobiGr
b644f1d67b [SoundCloud] Update client id 2021-08-31 07:42:21 +02:00
Stypox
a5e9eeb790
[YouTube] Small improvements to subscription import 2021-08-30 15:45:45 +02:00
Tobi
9570882c73
Merge pull request #712 from magicbrothers/add-eduvid-tubus
Add invidious instance EduVid Tubus
2021-08-24 16:07:07 +02:00
Stypox
db6c729678
Merge pull request #709 from talanc/dev
[YouTube] Support csv and zip subscription import (Google Takeout)
2021-08-24 15:46:03 +02:00
talanc
94a29fd63f
[YouTube] csv and zip subscription import (Google Takeout)
csv:
Improved error messages
Exits early if it hasnt found any items in the first few lines

zip:
Now checks all CSV files instead of hard-coded paths

final qualifiers for immutable locals and parameters

Co-authored-by: litetex <40789489+litetex@users.noreply.github.com>
2021-08-24 15:42:49 +02:00
dependabot[bot]
816bc7137f Bump gson from 2.8.7 to 2.8.8
Bumps [gson](https://github.com/google/gson) from 2.8.7 to 2.8.8.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.7...gson-parent-2.8.8)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-08-23 13:04:10 +02:00
TobiGr
6d487e0dad Bump version to 0.21.9 2021-08-22 19:29:22 +02:00
Tobi
7823dcd99d
Merge pull request #715 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.4.0
Bump spotbugs-annotations from 4.2.3 to 4.4.0
2021-08-22 16:41:20 +02:00
dependabot[bot]
d75d54e1c6
Bump spotbugs-annotations from 4.2.3 to 4.4.0
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.2.3 to 4.4.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.2.3...4.4.0)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-08-22 07:12:54 +00:00
dependabot[bot]
72393995bb
Bump jsoup from 1.13.1 to 1.14.2 (#716)
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.13.1 to 1.14.2.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.13.1...jsoup-1.14.2)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-22 09:12:14 +02:00
Stypox
d8179dc35a
Merge pull request #713 from TiA4f8R/regexes-yt-signatureciphers
[YouTube] Consolidate the regular expressions used to find the cipher decryption function
2021-08-17 18:58:27 +02:00
Tobi
89bab241b9
Merge pull request #717 from XiangRongLin/fix_tests
Fix failing PeertubeStreamExtractorTest
2021-08-17 17:48:28 +02:00
XiangRongLin
61acdae4b8 Fix failing PeertubeStreamExtractorTest 2021-08-17 17:06:36 +02:00
FireMasterK
4f86b8ce43
Add verified info to ChannelInfo. (#714) 2021-08-17 13:08:49 +02:00
Stypox
52910ce970
Keep original exception when there is an error message (#710) 2021-08-13 16:13:58 +02:00
TiA4f8R
f8309d546a
[YouTube] Consolidate current regexes of the cipher decryption function
Also add more regexes
2021-08-12 18:26:43 +02:00
magicfelix
0e16091ce0
Add invidious instance EduVid Tubus 2021-08-12 10:06:41 +02:00
Tobi
b6f951edb4
Merge pull request #711 from gardenappl/dev
Fix minor typos in Javadoc
2021-08-11 18:16:34 +02:00
gardenapple
50bba32239
Fix typos in Javadoc 2021-08-11 12:01:25 +03:00
Tobi
78f2192cbb
Merge pull request #687 from TacoTheDank/gradle7
Upgrade to Gradle 7.1.1
2021-08-08 12:38:29 +02:00
TacoTheDank
4a7c1723a6 Use new 'maven-publish' plugin 2021-08-07 17:51:32 -04:00
TacoTheDank
8919a53d95 Update gradle wrapper to 7.1.1 2021-08-07 17:51:18 -04:00
TacoTheDank
cf9e678223 Declare common dependency versions in root gradle 2021-08-07 17:51:09 -04:00
FireMasterK
71b9fd0076
Faster iframe api based player extraction. (#694)
* Faster iframe api based player extraction.

Uses the IFrame API to reduce the required download to less than 1/50 of the size.

* Remove debug code.

* Extract to two methods.

* Add tests for player URL extraction.

* Add assertThat for tests.
2021-08-07 12:35:48 +02:00
Tobi
4b147863ec
Merge pull request #705 from FireMasterK/suppored-countries
Update supported countries list for YouTube.
2021-08-06 10:14:27 +02:00
FireMasterK
ee32317315
Update supported countries list for YouTube. 2021-08-05 23:48:38 +05:30
FireMasterK
e8e9e6cd00
Fix comment reply continuations. 2021-08-04 22:14:04 +05:30
FireMasterK
666b45958e
Fix test and update mocks. 2021-08-04 21:10:02 +05:30
FireMasterK
f6d054e5da
Add test for comments. 2021-08-04 21:03:00 +05:30
FireMasterK
ed84658055
Get it working. 2021-08-04 18:35:14 +05:30
FireMasterK
ea7e6526fd
Collect replies/page to CommentsInfoItem. 2021-08-04 18:09:42 +05:30
FireMasterK
4385cc1b7c
Add comment replies. 2021-08-04 17:56:04 +05:30
Stypox
8ca5e5e364
Merge pull request #702 from TeamNewPipe/master
This merges master into dev
2021-08-03 20:52:04 +02:00
Stypox
bb3815d19b
Merge pull request #701 from Stypox/v0.21.8
V0.21.8
2021-08-03 20:43:15 +02:00
Stypox
5b38b3ae97
Merge branch 'master' into v0.21.8 2021-08-03 20:38:07 +02:00
Stypox
4b5907e18c
Release version 0.21.8 2021-08-03 20:33:24 +02:00
Stypox
5a88263785
Merge pull request #604 from TiA4f8R/youtubei-api
[YouTube] Use the new internal API in NewPipe Extractor
2021-08-03 19:01:14 +02:00
TiA4f8R
79cc5c8d12
Fix tests of YoutubePlaylistExtractorTest and YoutubeMixPlaylistExtractorTest
Without removing RunWith and SuiteClasses annotations (and the corresponding imports) in YoutubePlaylistExtractorTest and YoutubeMixPlaylistExtractorTest, some mocks cannot be generated, so the CI fails because of the missing mocks. Mocks of workings tests have been also updated.
2021-08-01 17:28:01 +02:00
TiA4f8R
b74a39c176
Reformat some code and don't use the clickTrackingParams in continuations of YouTube Music search results
The clickTrackingParams of YouTube Music search results are not needed to get continuations. This commit removes their use, which may improve privacy.
2021-08-01 17:25:59 +02:00
TiA4f8R
f8197da09e
Update YouTube comments and YouTube playlist continuations mocks 2021-08-01 12:39:05 +02:00
FireMasterK
f3e4c9d689
Use the youtubei API for YouTube comments
Migrate YouTube comments to the desktop version by using the `next` endpoint of the InnerTube internal API.
With the desktop version, we are able to get the exact like count of YouTube comments (by parsing the accessibility data) (the current extraction is used as a fallback). We are also now able to get if the uploader of the comment is verified or not.

Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-08-01 12:39:05 +02:00
TiA4f8R
286d839a3b
Update mocks (except mocks of YoutubeCommentsExtractorTest) 2021-08-01 12:39:04 +02:00
TiA4f8R
693f654f02
Ignore a broken test of YoutubeMixPlaylistExtractorTest 2021-08-01 12:39:03 +02:00
TiA4f8R
d55109258d
Fix YoutubeStreamExtractorAgeRestrictedTest 2021-08-01 12:39:03 +02:00
FireMasterK
2eeb0a3403
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:

- the desktop API is fetched.

If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status

If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.

If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.

We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.

Other code changes:

- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed

Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-08-01 12:39:03 +02:00
TiA4f8R
7753556e66
Adress the last requested changes + update YoutubeCommentsExtractor mocks 2021-08-01 12:39:03 +02:00
TiA4f8R
2320aecb7c
Update mocks 2021-08-01 12:39:02 +02:00
TiA4f8R
629dcd63b2
Ignore some broken tests 2021-08-01 12:39:01 +02:00
TiA4f8R
3adac6a150
Workaround for rate limits: always use the Android mobile API 2021-08-01 12:39:01 +02:00
TiA4f8R
8aa60d7e8f
Update clients version 2021-08-01 12:39:01 +02:00
TiA4f8R
27bd797f1e
Update mocks 2021-08-01 12:39:01 +02:00
TiA4f8R
609919db59
Adress again reviews, fix some rebase issues 2021-08-01 12:39:00 +02:00
TiA4f8R
4299d806a2
Adress changes 2021-08-01 12:38:59 +02:00
TiA4f8R
1a6b8da438
Annotate YoutubeParsingHelper methods with Nonnull when needed 2021-08-01 12:38:59 +02:00
TiA4f8R
d8177b57f6
Loop in all formats to check if the stream has URLs protected by signatureCiphers 2021-08-01 12:38:59 +02:00
TiA4f8R
a6a2c6eb80
Revert the use of Collections.singletonList instead of Arrays.asList in addCookieHeader of YoutubeParsingHelper 2021-08-01 12:38:59 +02:00
TiA4f8R
81013e5a8e
Add a static method to reset the YouTube deobfuscation code and use it in tests
This method is needed for YouTube stream tests, because when all YouTube tests are ran, the signatureTimestamp is known (the sts string) so a different body than the body present in the mocks is send by the extractor instance.
As a result, running all YouTube stream tests with the MockDownloader (like the CI does) will fail if this method is not called before fetching the page of a test.
2021-08-01 12:38:58 +02:00
TiA4f8R
cc798523cd
Adress requested changes in other classes 2021-08-01 12:38:58 +02:00
TiA4f8R
accd5ddef3
Adress requested changes in YoutubeStreamExtractor 2021-08-01 12:38:58 +02:00
TiA4f8R
632772d17f
Adress requested changes in YoutubeParsingHelper 2021-08-01 12:38:58 +02:00
TiA4f8R
657f165771
Update client version and mocks 2021-08-01 12:38:44 +02:00
TiA4f8R
8c1c7281b0
Real fix of the fetch of the JS player at each the signatureTimestamp is required
The strings playerJsUrl, sts and playerCode are now static in order to don't fetch again the JavaScript player at each time the signatureTimestamp is needed.
2021-08-01 12:38:43 +02:00
TiA4f8R
e97a685989
Format the YoutubeMusicSearchExtractor class to be in the 100 characters per line limit 2021-08-01 12:38:43 +02:00
TiA4f8R
34a9ccb0fd
Adress requested changes 2021-08-01 12:38:42 +02:00
TiA4f8R
54d4551ca6
Adress requested changes in YoutubeParsingHelper and update mobile client version 2021-08-01 12:38:42 +02:00
TiA4f8R
a59c2a3577
Catch every exception instead of only IOException and ExtractionException and add a Javadoc
Catch every exception instead of only IOException and ExtractionException.
Add JavaDoc for fetchAndroidMobileJsonPlayer method of YoutubeStreamExtractor
2021-08-01 12:38:42 +02:00
TiA4f8R
6921e80ded
Try again to don't fetch at each time the JavaScript player 2021-08-01 12:38:42 +02:00
TiA4f8R
70927ddade
Update client version and mocks 2021-08-01 12:38:40 +02:00
TiA4f8R
318bc46a8c
Readd the deleted code of views because watching count of livestreams was broken
The number shown was the total number of views that a livestream has. In order to fix this bug, the previous code is readded.
2021-08-01 12:38:39 +02:00
TiA4f8R
947baec805
Fetch again the desktop player JSON only if the content is protected by signatureCiphers 2021-08-01 12:38:39 +02:00
TiA4f8R
7474049fd1
Update mocks 2021-08-01 12:38:39 +02:00
TiA4f8R
c32bc6e534
Try to don't fetch again the first page when requesting a channel continuation
Try to don't fetch again the first page of a YouTube channel when requesting a continuation of it by trying to store the channel name and the channel id into the next page using the ids field of the Page class.
2021-08-01 12:38:38 +02:00
TiA4f8R
b52732a1c3
Fix the fetch of the playerCode at each time getStsFromPlayerJs is called
storePlayerJs was called even if sts was not empty in the getStsFromPlayerJs method. This commit fixes it.
2021-08-01 12:38:37 +02:00
TiA4f8R
ae5abc0c5d
Fix a typo in YoutubeChannelExtractor 2021-08-01 12:38:37 +02:00
TiA4f8R
0f9e9b8b4b
Use the youtubei API for YouTube mixes + update the corresponding test + do some improvements
Use the youtubei API for YouTube mixes. The corresponding has been updated because the new API breaks the tests of YoutubeMixPlaylistExtractorTest.
Remove some deprecated code (the old search code with the pbj JSON) and do some other improvements.
2021-08-01 12:38:37 +02:00
TiA4f8R
14569c4aa9
Readd the return of itags 134 and 136 2021-08-01 12:38:37 +02:00
TiA4f8R
3017dde67e
Fix some typos 2021-08-01 12:38:36 +02:00
TiA4f8R
013b902535
Use the Android mobile API when there are OTF streams or the content is protected by signatureCiphers
Use the Android mobile API to get the itag 22 (720p with audio), removed when the content is protected by signatureCiphers.
Also use this API when they are OTF streams, to get the itag 17 and 36, low 3GPP quality streams but also the itag 139.
Update the web client version.
2021-08-01 12:38:36 +02:00
TiA4f8R
e7d589edbf
Use the youtubei API for YouTube videos + update client version
Update the hardcoded client version to 2.20210520.09.00
Use the player and next endpoints of the Innertube API for YouTube videos
2021-08-01 12:38:36 +02:00
TiA4f8R
f73c923f60
Don't use the youtubei.googleapis.com but the websites domains + update client version of the desktop internal API
Use again www.youtube.com and music.youtube.com domains instead of youtubei.googleapis.com domain because it spoofs more a web client of YouTube or YouTube Music and may reduce Google's detection of NewPipe Extractor users.
2021-08-01 12:38:34 +02:00
TiA4f8R
effcdaa4f2
Update mocks, reenable a test and fix a test
This commit updates mocks and reenables the test invalidId of the NotAvailable class of the YoutubePlaylistExtractorTest class beacuse with the youtubei/innertube API, it returns "Not found" and doesn't redirect to the YouTube homepage.
The expectedMetaInfo test of the MetaInfoTest class of the YoutubeSearchExtractorTest class was broken because YouTube removes the vaccine progress link of the WHO from the meta info so this commit removes it and the test is now passing.
2021-08-01 12:38:04 +02:00
TiA4f8R
4d682834c3
Fix localization and update client version 2021-08-01 12:38:03 +02:00
TiA4f8R
f46cfb0f26
Adress reviews and do some improvements
Adress changes requested in reviews.
Do some improvements, remove unused imports and format some code to be in the 100 characters line limit.
2021-08-01 12:38:03 +02:00
TiA4f8R
e075dd5a63
Update client version, fix some tests, update mocks and do some improvements
Add the origin and the referer headers with the https://www.youtube.com value for YouTube JSON POST requests.
Don't add the consent cookie header for the requests which use the youtubei/innertube API because it's uneeded.
Fix some tests and update YouTube mocks
2021-08-01 12:38:02 +02:00
TiA4f8R
b49ae547a3
Do some improvements to YoutubeStreamExtractor
Get the real name of the uploader (for autogenerated channels and music artist channels), like before the migration to the JSON pbj.
Do some other improvements, especially reformatting some code to be in the 100 characters line limit and use final where possible.
2021-08-01 12:38:01 +02:00
TiA4f8R
58ce9b04a1
Fix channel extraction when channel URL is youtube.com/c/username 2021-08-01 12:38:01 +02:00
TiA4f8R
991b2c7d73
Use lightweight requests when getting and checking YouTube API key and client version 2021-08-01 12:38:01 +02:00
TiA4f8R
9ab9c66ddf
Use the youtubei.googleapis.com domain for YouTube Music searches + change a check + update client version and mocks
Change the domain from music.youtube.com to youtubei.googleapis.com.
Use a lightweight request to check if the hardcoded YouTubeMusic keys are valid. Increase the length of the response to 500 because if the key is invalid, the length of the response returned is higher than 250 and the response when the key is valid is higher than 1500.
Format the YoutubeMusicSearchExtractor file.
Update YouTube web client version and mocks
2021-08-01 12:37:56 +02:00
TiA4f8R
77c031a88a
Use the youtubei API for YouTube trends 2021-08-01 12:36:28 +02:00
TiA4f8R
f461224b2b
Use the youtubei API for YouTube searches + update mocks
Add getSearchParameter, a new method in YoutubeSearchQueryHandlerFactory class which returns the params field for a search, or an empty string if there is no one.
Update mocks of YoutubeSearchExtractorTest.
2021-08-01 12:36:28 +02:00
TiA4f8R
a12c69da7d
Use the youtubei API for YouTube channels 2021-08-01 12:36:27 +02:00
TiA4f8R
5794eb2350
Use the youtubei API for YouTube playlists 2021-08-01 12:36:12 +02:00
Stypox
c97a19d719
Merge pull request #696 from XiangRongLin/decrypt_pattern
Expand regex to match n param decrypt function
2021-08-01 10:24:22 +02:00
Stypox
06a5219c9c
Merge pull request #699 from FireMasterK/yt-music-search
Fix YouTube music search.
2021-08-01 10:17:15 +02:00
XiangRongLin
37df225556 Remove length check from StringUtils.matchToClosingParenthesis 2021-07-31 16:05:24 +02:00
FireMasterK
f4aad8b014
Fix tests again. 2021-07-31 14:40:51 +05:30
FireMasterK
88c11db4c5
Fix isCorrectedSearch test. 2021-07-31 02:18:18 +05:30
FireMasterK
fd19c53f65
Fix search suggestions test. 2021-07-31 02:08:50 +05:30
FireMasterK
a685941bba
Fix YouTube music search. 2021-07-31 01:33:38 +05:30
XiangRongLin
852a65ff18 Add tests for StringUtils 2021-07-30 19:56:57 +02:00
XiangRongLin
48d897e6ad Add final and adjust utils class name 2021-07-29 21:33:45 +02:00
XiangRongLin
60794aea31 [YouTube] Add parenthesis matching as way to parse decrypt function
From @Stypox
2021-07-29 21:23:00 +02:00
XiangRongLin
2967d1ae6a [YouTube] Compile YoutubeThrottlingDecrypter pattern statically 2021-07-29 20:27:29 +02:00
XiangRongLin
1c78976900 [YouTube] Expand regex to match n param decrypt function
Temporary solution
2021-07-29 20:26:36 +02:00
TobiGr
b62fe7141e Fix JDoc 2 2021-07-29 12:34:04 +02:00
TobiGr
dbc4e01c17 Fix JDoc 2021-07-29 12:31:33 +02:00
TobiGr
6335823843 Merge branch 'master' into dev 2021-07-28 20:41:23 +02:00
TobiGr
7c6ff0a38f Release version 0.21.7 2021-07-28 20:38:45 +02:00
TobiGr
8bfcb0ad59 [SoundCloud] Fix hardcoded client id 2021-07-28 20:33:21 +02:00
Tobi
027dc65434 pull request #683 from XiangRongLin/yt_throttling
[YouTube] Fix buffering by decoding n parameter of stream urls
2021-07-28 20:32:38 +02:00
Tobi
14c179f343
Merge pull request #692 from TeamNewPipe/fix/sc/id
[SoundCloud] Fix hardcoded client id
2021-07-28 18:28:07 +02:00
Tobi
394c02ad06
Merge pull request #683 from XiangRongLin/yt_throttling
[YouTube] Fix buffering by decoding n parameter of stream urls
2021-07-28 18:01:57 +02:00
TobiGr
785ff4aa32 [SoundCloud] Fix hardcoded client id 2021-07-28 17:02:56 +02:00
TobiGr
79f2d74b04 Fix test
The decryption code changes over time. Only check whether the n parameter was changed in the URL
2021-07-28 16:45:31 +02:00
TobiGr
d70adfdb8f Add methods for cache control to YoutubeThrottlingDecrypter. 2021-07-27 20:32:22 +02:00
TiA4f8R
d13f531b6f Use YoutubeThrottlingDecrypter also in getAudioStreams and getVideoOnlyStreams methods of YoutubeStreamExtractor
Without this commit, the n param is only decrypted for streams extracted in getVideoStreams (so only for streams in the formats object of the player response).
2021-07-27 17:28:32 +02:00
TiA4f8R
32055147e0 Do some code improvements
Use final where possible, annotate some methods and parameters as Nonnull and format new code to be in the 100 characters limit per line.
2021-07-27 17:24:27 +02:00
TobiGr
1c30a2725e Cache nParams to prevent executing the JavaScript function for the same nParam multiple times.
Closes #689
2021-07-22 19:40:12 +02:00
XiangRongLin
3a3d1d7f2b Make YoutubeJavaScriptExtractor and JavaScript methods static
Also address review and rewrite some comments
2021-07-20 20:48:11 +02:00
XiangRongLin
a683c8d278 Delete duplicated code to load youtube javascript file 2021-07-17 19:14:57 +02:00
XiangRongLin
a02ee2e952 Rewrite youtube throttling solution and add tests 2021-07-17 19:10:09 +02:00
Tobi
ada67d136a
Merge pull request #647 from litetex/playerSeekbarPreview
Code changes to enable player thumbnail seekbar preview in NewPipe
2021-07-17 17:49:54 +02:00
litetex
0c12b396e5 Review changes 2021-07-17 16:41:12 +02:00
litetex
ecf4232ce3 Using `Collections.emptyList()` 2021-07-17 16:41:12 +02:00
litetex
0f9ed020a4 Frameset has to implement Serializable or NewPipe fails
java.lang.RuntimeException: Parcelable encountered IOException writing serializable object (name = org.schabi.newpipe.extractor.stream.StreamInfo)
...
Caused by: java.io.NotSerializableException: org.schabi.newpipe.extractor.stream.Frameset
...
2021-07-17 16:41:11 +02:00
litetex
17ccaf4b87 Update YoutubeStreamExtractor.java
To fix ``java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.String[] java.lang.String.split(java.lang.String)' on a null object reference``
2021-07-17 16:41:10 +02:00
litetex
4e41e172ea Update Frameset.java
Reformatted code
2021-07-17 16:41:10 +02:00
litetex
09b670d745 Renamed frames to previewFrames (so it's better understandable) 2021-07-17 16:41:09 +02:00
litetex
b086655d54 Reverted useless change 2021-07-17 16:41:08 +02:00
litetex
93b0c5b15f Added seekbar data 2021-07-17 16:41:08 +02:00
XiangRongLin
6956b72af7 Fix YoutubeThrottlingDecoder.replaceNParam
Previously it replaced the parameter itself not the value of the parameter.
2021-07-17 10:05:14 +02:00
XiangRongLin
a189f685dc cache player js code 2021-07-16 19:20:40 +02:00
XiangRongLin
fcdb9bdbeb add url to sout 2021-07-15 20:05:26 +02:00
XiangRongLin
80cf8b3acd Extract separate YoutubeThrottlingDecoder 2021-07-15 20:05:25 +02:00
XiangRongLin
a86a30103f [Youtube] bare bones version to solve throttling
Done by transforming the parameter "n" from videoplayback urls
https://github.com/ytdl-org/youtube-dl/issues/29326#issuecomment-865985377
2021-07-15 20:05:25 +02:00
bopol
c38a06e8dc
Merge pull request #661 from B0pol/peertube-livestreams
[PeerTube] Support livestreams
2021-07-13 15:48:13 +02:00
Tobi
b45bb411e8
Merge pull request #652 from litetex/fixYTCommentsAndAddDisabledComments
Fix yt comments and add disabled comments functionallity
2021-07-12 16:31:50 +02:00
TobiGr
6fd93cdb31 Release 0.21.6 2021-07-10 17:07:23 +02:00
litetex
fdebf3c6cd Added javadoc warnings 2021-07-07 20:41:59 +02:00
litetex
6860543b07 Added more doc 2021-07-06 21:16:31 +02:00
litetex
a59fead0d7 Fixed typos 2021-07-06 21:16:30 +02:00
litetex
688a1c316b Fixed Exception when YT comments are disabled and added `commentsDisabled` field
* Fixed code: Added missing finals (according to NewPipes Checkstyle guide)
* Fixed ``findValue`` method in ``YoutubeCommentsExtractor``
2021-07-06 21:16:29 +02:00
litetex
ca33f4f60b Improved code format (final) and added `isCommentsDisabled` method 2021-07-06 21:16:28 +02:00
Tobi
f324772254
Merge pull request #676 from TeamNewPipe/fix/consent-cookie
Fix invalid CONSENT cookie value
2021-07-04 17:41:47 +02:00
TobiGr
dbdc962a07 Update mocks 2021-07-03 13:30:26 +02:00
TobiGr
9ca6dc26ed Fix invalid CONSENT cookie value 2021-07-03 13:27:40 +02:00
Tobi
a023f08774
Merge pull request #673 from XiangRongLin/recording_downloader
Remove IPv4 addresses from response during mock recording
2021-07-03 13:25:41 +02:00
XiangRongLin
b9549ace25 [Youtube] Fix lofi girl test and ignore mix with id test 2021-06-29 21:12:20 +02:00
XiangRongLin
5982431ffb Remove IPv4 addresses from response during mock recording 2021-06-29 21:03:33 +02:00
bopol
7d7cc087e2 [PeerTube] Support livestreams 2021-06-24 11:02:30 +02:00
bopol
3966178979
Merge pull request #665 from B0pol/agerestricted
[youtube] Fix extraction of age restricted videos
2021-06-23 21:07:14 +02:00
bopol
b475f09ba7 don't do useless concatenation 2021-06-23 20:33:32 +02:00
TiA4f8R
d1c56be744
Merge pull request #668 from B0pol/invidious-instances
Update Invidious instances and support Piped links
2021-06-23 18:36:48 +02:00
bopol
b96b2a6eba Update invidious instances 2021-06-23 14:39:10 +02:00
bopol
c5e8bd368d Update Mocks 2021-06-23 12:21:57 +02:00
bopol
e9a992b0a9 [youtube] Fix extraction of age restricted videos 2021-06-23 12:06:02 +02:00
bopol
35c1bdd012
Merge pull request #655 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.2.3
Bump spotbugs-annotations from 4.0.2 to 4.2.3
2021-06-23 12:01:56 +02:00
dependabot[bot]
ad403ce73e
Bump spotbugs-annotations from 4.0.2 to 4.2.3
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.0.2 to 4.2.3.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.0.2...4.2.3)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-06-23 09:57:42 +00:00
bopol
65fd8740aa
Merge pull request #658 from TeamNewPipe/revert-651-fix/streamType
[YouTube] Fix getting stream type of live streams
2021-06-23 11:33:24 +02:00
bopol
361f142621 Use badge style instead of label 2021-06-23 11:25:59 +02:00
bopol
58f109ddec adress stypox review 2021-06-23 11:15:40 +02:00
bopol
fe29c78fe1
Merge pull request #656 from TeamNewPipe/dependabot/gradle/org.mozilla-rhino-1.7.13
Bump rhino from 1.7.12 to 1.7.13
2021-06-20 00:46:42 +02:00
bopol
e9412fdadc
Merge pull request #654 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.8.7
Bump gson from 2.8.6 to 2.8.7
2021-06-20 00:35:24 +02:00
XiangRongLin
1510dd5187
Merge pull request #660 from TeamNewPipe/fix/sc/client-id
[SoundCloud] Update HARDCODED_CLIENT_ID
2021-06-18 15:48:27 +02:00
TobiGr
1f9a99c931 [SoundCloud] Update HARDCODED_CLIENT_ID 2021-06-18 15:30:54 +02:00
Tobi
c998012c28
Revert "[YouTube] Fix getting stream type of live streams" 2021-06-18 09:20:32 +02:00
dependabot[bot]
38d0339753
Bump rhino from 1.7.12 to 1.7.13
Bumps [rhino](https://github.com/mozilla/rhino) from 1.7.12 to 1.7.13.
- [Release notes](https://github.com/mozilla/rhino/releases)
- [Changelog](https://github.com/mozilla/rhino/blob/master/RELEASE-NOTES.md)
- [Commits](https://github.com/mozilla/rhino/commits)

---
updated-dependencies:
- dependency-name: org.mozilla:rhino
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-06-17 21:49:07 +00:00
dependabot[bot]
5a46878b09
Bump gson from 2.8.6 to 2.8.7
Bumps [gson](https://github.com/google/gson) from 2.8.6 to 2.8.7.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.6...gson-parent-2.8.7)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-06-17 21:49:01 +00:00
bopol
598de2dd5f
Merge pull request #552 from FireMasterK/dependabot
Add dependabot
2021-06-17 23:48:41 +02:00
Tobi
0cb48e36bd
Merge pull request #651 from TeamNewPipe/fix/streamType
[YouTube] Fix getting stream type of live streams
2021-06-12 17:57:18 +02:00
TobiGr
6762fe3357 [YouTube] Fix getting stream type of live streams 2021-06-10 16:51:00 +02:00
Tobi
a27575021d
Merge pull request #625 from TeamNewPipe/commentsStreamPosition
Add streamPosition for comments
2021-06-08 11:03:15 +02:00
TobiGr
f7f727d19c Add constant NO_LIKE_COUNT to CommentsInfoItem 2021-06-08 10:28:44 +02:00
TobiGr
b70c0f93c7 Add streamPosition for comments
SoundCloud is the only service which supports adding comments at a specific timestamp in the stream.
2021-06-08 10:27:52 +02:00
Tobi
d4186d100b
Merge pull request #580 from TeamNewPipe/accountTerminated
Add AccountTerminatedException for terminated channels
2021-06-08 09:55:58 +02:00
TobiGr
89e0514d8b Fix Javadoc generation fix 2021-06-08 09:30:05 +02:00
TobiGr
80d3052033 Fix Javadoc generation 2021-06-08 09:26:01 +02:00
TobiGr
98825a2f01 Set version to 0.21.4 2021-06-07 23:45:12 +02:00
TobiGr
fa444c8298 Remove assertations of exception messages 2021-06-06 11:38:43 +02:00
Tobi
8c42a48673
Merge pull request #646 from Stypox/get-service
Add utility method getService() to Info object
2021-06-05 14:29:09 +02:00
Stypox
2158ca4060
Remove two unneded warning suppressions 2021-06-05 14:15:48 +02:00
Stypox
13f192704d
Add utility method getService() to Info object 2021-06-05 14:14:19 +02:00
XiangRongLin
0ad51e76fb
Merge pull request #638 from TeamNewPipe/spelling
Improve exception message thrown when not accepting an URL in LinkHandlerFactory#fromUrl(String url, String baseUrl)
2021-06-04 18:23:31 +02:00
Tobi
143fd3c81c
Merge pull request #645 from XiangRongLin/fix_test
Fix PeertubeAccountExtractorTest
2021-06-03 20:49:59 +02:00
XiangRongLin
faa503c7dd Fix PeertubeAccountExtractorTest 2021-06-03 19:54:42 +02:00
XiangRongLin
db81384ae0
Merge pull request #644 from TiA4f8R/utf-8-encoding-for-mocks-windows
Specify UTF-8 file encoding in RecordingDownloader and MockDownloader
2021-06-03 18:55:00 +02:00
TiA4f8R
ac31f3a883
Specify UTF-8 file encoding in RecordingDownloader and MockDownloader
On Windows, mocks are recorded and read with the Cp1252 encoding so it breaks the mocks on non ASCII characters for Linux OS (and so the CI).
The project is in Java 8, so we can't use FileReader(File, Charset) and FileReader(File, Charset) because these methods require Java 11. Instead of changing the Java version of the extractor, use OutputStreamWriter and FileOutputStream instead of FileWriter and InputStreamReader and FileInputStream instead of FileReader.
2021-06-03 18:38:25 +02:00
Tobi
7e4332e0cf
Update CI badge to representt the scheduled tests 2021-05-30 22:12:09 +02:00
bopol
ff11c2df2a
Merge pull request #628 from litetex/fix-broken-yt-liked-comments
Fix broken yt likes in comments
2021-05-29 11:04:11 +02:00
TobiGr
1ff56a85df Update junit from 4.13.1 to 4.13.2
https://github.com/junit-team/junit4/blob/main/doc/ReleaseNotes4.13.2.md
2021-05-28 15:09:26 +02:00
TobiGr
858e007804 Improve exception message thrown when not accepting an URL in LinkHandlerFactory#fromUrl(String url, String baseUrl) 2021-05-28 14:59:44 +02:00
Tobi
519bba70b3
Merge pull request #630 from XiangRongLin/fix_tests
Fix MetaInfoTest.expectedMetaInfo
2021-05-28 14:16:53 +02:00
litetex
289b84e68b Removed unused code 2021-05-27 21:27:44 +02:00
litetex
e81b0e2885 Tweaked code 2021-05-27 19:48:31 +02:00
TobiGr
f4cb32cef5 Release version 0.21.3 2021-05-26 22:40:56 +02:00
XiangRongLin
bf2af4f316
Merge pull request #635 from litetex/update-setup-java
Update setup java
2021-05-26 17:44:57 +02:00
litetex
8c96545e57 Merge branch 'TeamNewPipe:dev' into fix-broken-yt-liked-comments 2021-05-24 18:06:45 +02:00
litetex
12fb18c310 Updated mock data (only the affected) 2021-05-24 18:05:26 +02:00
litetex
8e08a2aac7 Improved docs 2021-05-24 18:05:26 +02:00
litetex
020acfed71 Fixed tests and added new ones 2021-05-24 18:05:26 +02:00
litetex
b310922fc0 Better exception message 2021-05-24 18:05:25 +02:00
litetex
86bb9efb5d Added likeCount back to tests 2021-05-24 18:05:25 +02:00
litetex
46654f5fca Reformatted code 2021-05-24 18:05:25 +02:00
litetex
2174685c5c Reimplemented likeCount 2021-05-24 18:05:25 +02:00
litetex
01cfb55505 Update YoutubeCommentsInfoItemExtractor.java 2021-05-24 18:05:25 +02:00
litetex
bedcd87abb Fixed occurrences where voteCount is not set (<=0 likes) 2021-05-24 18:05:25 +02:00
litetex
545c0a6f42 Fixed comment 2021-05-24 18:05:24 +02:00
Tobi
ff005122bf
Merge pull request #627 from TiA4f8R/use-snd-api-v2-everywhere
[SoundCloud] Use a lightweight request to check if the hardcoded client_id is valid, fix the extraction of mobile URLs and more
2021-05-23 22:52:40 +02:00
TiA4f8R
a00fdcbd3d
Format the indentation of the definition of the apiUrl string in SoundcloudParsingHelper.resolveFor method
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2021-05-23 20:27:45 +02:00
TiA4f8R
6b607eb38d
Update the hardcoded client_id 2021-05-23 18:08:44 +02:00
TiA4f8R
4552ea9c9f
Use the SOUNDCLOUD_API_V2_URL constant in all the SoundCloud package 2021-05-23 18:08:44 +02:00
TiA4f8R
86308d0603
Use a lightweight request to check if the client_id is valid
Request the api-v2 host with the client_id instead of checking if the streams of a SoundCloud track are not empty: if it is valid, the API returns 404, otherwise it should return 401.
2021-05-23 18:08:43 +02:00
TiA4f8R
c5c190500c
Format the SoundCloud package of the extractor and fix some warnings
Use final where possible in the package and format code to be in the 100 caracters per line limit.
Fix some warnings generated by Android Studio and do some code improvements
2021-05-23 18:08:42 +02:00
litetex
89ecc4e2e8
Update docs.yml 2021-05-23 14:34:29 +02:00
litetex
7791907480
Update ci.yml 2021-05-23 14:34:16 +02:00
Tobi
636e27333b
Merge pull request #603 from TeamNewPipe/peertube_hls
Add PeerTube HLS streams
2021-05-23 11:49:40 +02:00
Tobi
fadf677c42
Merge pull request #629 from TechComet/patch-1
[YouTube] Fix the fetch of the video info page
2021-05-23 09:41:24 +02:00
XiangRongLin
172394cd52 Update mocks 2021-05-22 20:33:20 +02:00
XiangRongLin
18c8cac40c Fix MetaInfoTest.expectedMetaInfo
Expected meta changed
2021-05-21 21:00:08 +02:00
Hassan Igbaria
e8b5364745
Update YoutubeStreamExtractor.java 2021-05-21 16:22:24 +03:00
litetex
c3b837fe3b
Deduped code; Using default values... 2021-05-20 20:22:40 +02:00
litetex
b934c7ccbb
Removed unused import 2021-05-20 20:11:35 +02:00
litetex
10cf081145
Removed likeCount and added textualVoteCount 2021-05-20 20:10:14 +02:00
TobiGr
1acc53b8db [PeerTube] Add support for HLS streams 2021-05-15 12:39:23 +02:00
Tobi
3a3ade20f4
Merge pull request #620 from TeamNewPipe/bandcamp-comments
Add Bandcamp comments (first page)
2021-05-02 13:43:53 +02:00
Fynn Godau
85a4270bd9 [Bandcamp] Fix comments test 2021-05-02 13:07:24 +02:00
TobiGr
ec50570053 Add mocks 2021-05-01 17:50:19 +02:00
TobiGr
f4404b5cc8 Add tests for more reasons 2021-05-01 17:50:19 +02:00
TobiGr
fc998589dc Detect channels which have been terminated due to copyright infringement 2021-05-01 17:50:19 +02:00
TobiGr
bb3861ddce Add AccountTerminatedException for better error handling of terminated channels 2021-05-01 17:50:19 +02:00
Tobi
284362fc15
Merge pull request #608 from TeamNewPipe/youtube-fix-segment-test
[YouTube] fix tests
2021-05-01 17:24:41 +02:00
TobiGr
dc678a9f6d Release version 0.21.2 2021-04-26 18:06:30 +02:00
Tobi
7cf8edd5ac
Merge pull request #618 from TeamNewPipe/soundcloud_id
Fix Soundcloud extraction
2021-04-26 18:03:26 +02:00
TobiGr
9c12dc5609 [SoundCloud] Fix SoundCloud ID extraction
resolveIdWithEmbedPlayer() does not work anymore because the JSON data has been extracted to an API call. For this reason, replace resolveIdWithEmbedPlayer() with resolveIdWithWidgetApi)( which performs the API call.
2021-04-26 17:58:34 +02:00
Fynn Godau
9dc7a1d5ee [Bandcamp] Add comments (first page) 2021-04-26 17:25:04 +02:00
TobiGr
8f023c1ec7 [SoundCloud] Update client id 2021-04-26 15:14:11 +02:00
Tobi
7f202db8b1
Fix Google URL detection (#617) 2021-04-26 11:49:47 +02:00
fynngodau
6db4bea8ca
Rework Bandcamp comments (#613) 2021-04-13 21:10:59 +02:00
Tobi
c14f6db14a
Merge pull request #610 from TeamNewPipe/bandcamp_related_items
Extract related bandcamp items
2021-04-13 21:05:27 +02:00
Fynn Godau
3671876721 Fix bandcamp radio stream extractor 2021-04-13 19:30:11 +02:00
Fynn Godau
90b5c00599 Extract related items #593 2021-04-13 17:56:19 +02:00
TobiGr
1fe645704f [YouTube] FIx testMoreRelatedItems in RandomQueryNoMorePages 2021-04-11 23:21:35 +02:00
TobiGr
9a2b814d8f Update mocks 2021-04-11 22:59:01 +02:00
TobiGr
50005ce937 [YouTube] Replace unavailable YouTube Stream Segments test
The stream is unavailable in many countries.
2021-04-11 22:58:48 +02:00
Tobi
82d11386df
Merge pull request #607 from TeamNewPipe/bandcamp_radio_segments
[Bandcamp] Add Radio stream segments
2021-04-11 22:30:03 +02:00
TobiGr
318cec7625 [Bandcamp] Add Radio stream segments 2021-04-11 21:19:44 +02:00
TobiGr
55e0f8e725 Merge branch 'master' into dev 2021-04-10 11:33:34 +02:00
Tobi
e747a89be2
Merge pull request #600 from TeamNewPipe/youtube-eu-cosent
[YouTube] Set EU consent cookie
2021-04-09 12:02:52 +02:00
TobiGr
33173eb3e6 Make CONSENT COOKIE constants private 2021-04-09 11:59:17 +02:00
TobiGr
b94b316558 Update mocks 2021-04-09 11:59:17 +02:00
TobiGr
4b8d4a84b6 Set the NumberGenerator for th remaining tests 2021-04-09 11:59:17 +02:00
XiangRongLin
db7ad9c27d Update Mocks 2021-04-08 16:37:29 +02:00
XiangRongLin
4833df60e5 Add method to inject Random into YoutubeParsingHelper and use in tests 2021-04-08 16:36:55 +02:00
TobiGr
682ec27737 Update mocks 2021-04-08 14:22:31 +02:00
XiangRongLin
eafad3ad6b
Merge pull request #602 from XiangRongLin/soundcloud_go
Ignore unreliable SoundcloudGoPlusTrack.testRelatedItems
2021-04-07 19:41:44 +02:00
TobiGr
080d1e318d Code improvement 2021-04-07 14:11:47 +02:00
TobiGr
4ee270fe01 Fix LiveStream Test 2021-04-07 14:11:41 +02:00
TobiGr
883f16e0ad [YouTube] Set CONSENT cookie 2021-04-07 14:11:37 +02:00
XiangRongLin
aee3838ed5 Ignore unreliable SoundcloudGoPlusTrack.testRelatedItems 2021-04-06 18:26:45 +02:00
XiangRongLin
90afd6162a
Merge pull request #599 from XiangRongLin/post_body
Add body field to Page and use it
2021-04-06 18:23:26 +02:00
XiangRongLin
7b06c696e2 Use Page.body for YoutubeChannelExtractor 2021-04-06 18:14:38 +02:00
XiangRongLin
eda1b6e199 Use Page.body for YoutubePlaylistExtractor 2021-04-06 18:14:38 +02:00
XiangRongLin
3af26a2821 Add body field to Page 2021-04-04 11:39:58 +02:00
Tobi
4e0be60ddc
Merge pull request #601 from XiangRongLin/fix_tests
Fix SoundcloudGoPlusTrack Test
2021-04-04 09:55:54 +02:00
XiangRongLin
7b36469807 Fix SoundcloudGoPlusTrack
https://soundcloud.com/martinsolveig/places#t=0 has related streams
2021-04-04 09:39:26 +02:00
Tobi
b481a228cd
Merge pull request #598 from XiangRongLin/fix_tests
Fix a test and ignore rest
2021-04-03 23:44:23 +02:00
XiangRongLin
3d70b6d46a Fix a test and ignore rest 2021-04-03 11:03:10 +02:00
XiangRongLin
1925dcf4dc
Merge pull request #597 from fynngodau/related-items
Rename getRelatedStreams to getRelatedItems and change return type
2021-04-02 19:54:22 +02:00
Fynn Godau
c877712647 Add deprecated old method calls to avoid breaking API 2021-04-01 22:32:36 +02:00
Fynn Godau
14f6f1b7c3 Generify related streams calls and rename method 2021-03-31 20:24:40 +02:00
Stypox
9dbc152d44
Merge pull request #594 from TeamNewPipe/conersions
Reduce number of type conversions
2021-03-31 14:34:46 +02:00
TobiGr
1498e1905e Reduce number of type conversions in YouTubeSearchExtractor.collectStreamsFrom by introducing new variable 2021-03-31 13:30:38 +02:00
Tobi
7ea2cd73d3
Merge pull request #589 from fynngodau/bandcamp-more-recent-pages
Load more featured pages and fix featured cover arts not loading
2021-03-31 12:15:12 +02:00
Fynn Godau
705f6c6e33 Apply review 2021-03-31 11:02:30 +02:00
XiangRongLin
564d74c250
Merge pull request #578 from TeamNewPipe/code_improvements
Code improvements
2021-03-30 18:12:22 +02:00
TobiGr
657b00ca11 Smaller code improvements 2021-03-30 10:10:53 +02:00
TobiGr
8fbd1a2bcf FIx potential NullPointerException 2021-03-30 10:07:20 +02:00
TobiGr
f2e7c7ebd6 Replace Vector with ArrayList for performance improvements
I could not figure out, why these two methods are necessary.
2021-03-30 10:07:20 +02:00
TobiGr
48fd8ca7b3 Remove redundant type declaration 2021-03-30 10:07:20 +02:00
TobiGr
22bd16fca3 Make vars private, final and static 2021-03-30 10:07:20 +02:00
TobiGr
6a0f6e846a Fix possible bug when language.id field is not present and this the languageCode an empty String. 2021-03-30 10:07:20 +02:00
TobiGr
b9282bbe94 Remove checks which are always true 2021-03-30 10:07:20 +02:00
TobiGr
46eab1ec17 Remove declarations of exceptions which are not thrown from method signatures 2021-03-30 10:07:20 +02:00
TobiGr
c3e23559d7 Remove unused imports 2021-03-30 10:07:20 +02:00
TobiGr
e4c40ae6d1 Use List.isEmpty() 2021-03-30 10:07:20 +02:00
XiangRongLin
af2183855a
Merge pull request #591 from TotalCaesar659/update-url-to-https
Update URL to HTTPS in README
2021-03-29 09:33:59 +02:00
TotalCaesar659
cf3f54f438 Update URL to HTTPS 2021-03-28 22:06:56 +03:00
Fynn Godau
d663be5a78 [Bandcamp] Fix featured albums not showing thumbnails 2021-03-28 13:12:22 +02:00
Fynn Godau
dbcf61c6f7 [Bandcamp] Load more featured pages 2021-03-28 12:53:05 +02:00
Stypox
b4dee6d08f
Merge pull request #306 from B0pol/metadata
Extract metadata for youtube, soundcloud & mediaccc
2021-03-27 08:45:47 +01:00
TobiGr
e61ceef005 Release 0.21.0 2021-03-25 21:59:15 +01:00
Tobi
e2500fb7cc
Merge pull request #583 from TeamNewPipe/peertube_playlist
Fix some tests and bugs
2021-03-25 21:53:41 +01:00
TobiGr
070a40e181 [YouTube] Mix Playlist - Fix getting name 2021-03-25 21:47:16 +01:00
TobiGr
b029779217 [media.cc.de] Disable unreliable tests for now 2021-03-25 19:44:44 +01:00
TobiGr
bbee15474d [PeerTube] Fix playlist extraction 2021-03-25 19:19:19 +01:00
Tobi
d4f83a1782
Merge pull request #575 from TiA4f8R/fix-peertube-test
Fix Peertube account subscribers extraction
2021-03-25 19:14:51 +01:00
TiA4f8R
ae283314da
Add a comment explaining subscribers count calculation 2021-03-25 17:54:46 +01:00
TiA4f8R
abb790f465
Remove println in getSubscriberCount from PeertubeAccountExtractor class 2021-03-25 17:49:22 +01:00
TiA4f8R
772de53a66
Increase the number of subscribers in PeertubeAccountExtractorTest 2021-03-25 17:49:20 +01:00
TiA4f8R
efe2b964f0
Extract real subscribers of a Peertube account
Apply fix provided in the PR and move all accounts/ strings to a constant, ACCOUNTS.
2021-03-25 17:47:44 +01:00
Tobi
12835bfae1
Merge pull request #551 from FireMasterK/ff-privacy
Change UA to privacy.resistFingerprinting.
2021-03-24 17:58:34 +01:00
Tobi
cae6cea551
Merge pull request #582 from Stypox/fix-trending
[YouTube] Fix trending getName()
2021-03-24 10:49:46 +01:00
Stypox
ce7cbbc9a0
[YouTube] Fix trending getName() 2021-03-24 09:14:09 +01:00
XiangRongLin
9009f1e277
Merge pull request #517 from XiangRongLin/flaky_test
Add MockOnlyRule to allow skipping specific tests based on downloader
2021-03-20 18:04:14 +01:00
bopol
152221c7fb
ignore a test with encoding problems 2021-03-19 10:14:43 +01:00
bopol
8806fb4e6b
address reviews 2021-03-19 10:14:43 +01:00
bopol
c47cc54908
Extract metadata for YouTube, SoundCloud & MediaCCC 2021-03-19 10:14:43 +01:00
Stypox
f71cfd489c
Merge pull request #526 from TiA4f8R/snd-hls-workaround
Support SoundCloud HLS-only tracks by using a workaround
2021-03-19 10:09:54 +01:00
TiA4f8R
379d7312fa
Don't use a regular expression to find the last segment URL and do code improvements
Apply suggestions provided in the PR and remove a redundant import.
2021-03-14 17:54:22 +01:00
TiA4f8R
0e3e420a25
Fix tests 2021-03-14 17:54:21 +01:00
TiA4f8R
d61d9d116d
Refactor getAudioStreams method of SoundcloudStreamExtractor
Split the method into private methods, in order to have a better reading.
2021-03-14 17:54:20 +01:00
TiA4f8R
a7b15b51e6
Change t to transcodingJsonObject in SoundcloudStreamExtractor 2021-03-14 17:54:19 +01:00
TiA4f8R
0438828e36
Add a test for the number of audioStreams in CreativeCommonsPlaysWellWithOthers test
It should be only two audio streams for track "Plays Well with Others, Ep 2: What Do an Army of Ants and an Online Encyclopedia Have in Common?" by Creative Commons (https://soundcloud.com/wearecc/plays-well-with-others-ep-2-what-do-an-army-of-ants-and-an-online-encyclopedia-have-in-common):
- one which is a progressive stream, in MP3 format with a bitrate of 128 kbps
- one which is an HLS stream, in OPUS format with a bitrate of 64 kbps.
2021-03-14 17:54:18 +01:00
TiA4f8R
3bd08a2880
Adress requested changes and use final where possible in SoundcloudStreamExtractor
This commit moved the HLS parsing task to a separate method, did little performance improvements and used final where possible in the SoundcloudStreamExtractor file.
2021-03-14 17:54:17 +01:00
TiA4f8R
cbacd3c0a5
Add a check to don't show MP3 128kbps stream twice and catch IOException when fetching the HLS Manifest
If a progressive stream is present in the transcodings, it's unnecessary to show twice an MP3 128kbps stream so if this is the case, the MP3 HLS stream will be not added to the audioStreams, else it will.
This commit also catch fetching errors in HLS manifests parsing and don't add the corresponding stream if an error occurs.
2021-03-14 17:54:16 +01:00
TiA4f8R
26f1b4e7dc
Support SoundCloud HLS by using a workaround
This commit tries to support SoundCloud HLS streams by parsing M3U manifests, get the last segment URL (in order to get track length) and request a segment URL equals to track's duration so it's a single URL.
2021-03-14 17:54:14 +01:00
Tobi
def745b801
Merge pull request #579 from fynngodau/bandcamp-fix-linkhandler
[Bandcamp] Fix link handler acceptance behaviour
2021-03-14 12:56:10 +01:00
Fynn Godau
2e57a8f24f [Bandcamp] Fix link handler acceptance behaviour
* Test for bandcamp footer instead of meta tag (which is not present on
  all pages)
* Accept links to /music, not just /releases
* Correctly handle uppercase URLs
2021-03-14 09:51:30 +01:00
TobiGr
70d9e389b9 [Bandcamp] Add tests for external bandcamp artists
Necessary, because the external pages tested before were converted to bandcamp.com pages. See f9d06252f2978a3e96b2da1591258aed298f90e3
2021-03-13 20:51:59 +01:00
TobiGr
f9d06252f2 [Bandcamp] Fix tests 2021-03-13 17:42:39 +01:00
Tobi
7e6f464407
Merge pull request #509 from TiA4f8R/soundcloud-improvements
Add new exceptions to be able to display different error messages in apps
2021-03-07 13:51:15 +01:00
Tobi
ac51134aed
Merge pull request #546 from XiangRongLin/unignore2
Unignore more tests
2021-03-07 13:15:24 +01:00
Tobi
021da75f24
Merge pull request #232 from fynngodau/dev
Bandcamp support
2021-03-05 21:37:57 +01:00
TobiGr
22fa131922 Merge branch 'dev' into bandcamp 2021-03-05 19:45:37 +01:00
TiA4f8R
e55284bb8f
Code style improvements
Use final where possible in YoutubeStreamExtractor and do some other code style improvements
2021-03-05 16:38:53 +01:00
TobiGr
d0a1041afd
Prevent NullPointerException and use equalsIgnorecase 2021-03-05 16:38:52 +01:00
TiA4f8R
890cbba625
Update PaidContent exception 2021-03-05 16:38:51 +01:00
TobiGr
da3cfa967d
Handle age-restricted videos 2021-03-05 16:38:50 +01:00
TiA4f8R
448b68700c
Use final in exceptions and add AgeRestrictedException
AgeRestrictedException will be thrown only if the reason message equals to "Sign in to confirm your age" and if the age limit is 18.
2021-03-05 16:38:49 +01:00
TobiGr
22e6f33f0a
Ignore SoundCloudGeoRestrictedTest until HLS stream extraction is added
The ContentNotSupportedException is thrown because no supported audio streams where extracted. However, SoundCLoud does not check, whether there are any streams available. 
This commit should be reverted in #526
2021-03-05 16:38:47 +01:00
TobiGr
21158744b1
Add expectedDescriptionIsEmpty() to DefaultStreamExtractorTest
Also check if related streams are empty if they are expected to be empty.
2021-03-05 16:38:46 +01:00
TiA4f8R
f15d7837a1
Fix SoundCloud tests 2021-03-05 16:38:45 +01:00
TiA4f8R
4e45aef2b3
Fix SoundCloud CreativeCommonsPlaysWellWithOthers test 2021-03-05 16:38:43 +01:00
TiA4f8R
59d6d3f04e
Fix YouTube tests 2021-03-05 16:38:42 +01:00
TiA4f8R
3925204658
Add tests and rename SoundcloudGoPlusException to SoundcloudGoPlusContentException 2021-03-05 16:38:40 +01:00
TiA4f8R
771bb1a2cb
Add three new exceptions for unavailable YouTube contents
These exceptions are thrown on a test with the error messages text, because YouTube returns only "UNPLAYABLE" status in most error cases.
Tests are based with English strings, so changing the lang used by
extractor will throw the generic exception (ContentNotAvailableException).
2021-03-05 16:38:39 +01:00
TiA4f8R
35325d980d
Add GeographicRestrictionException and SoundCloudGoPlusException in NewPipe Extractor to be able to display different error messages
This commit adds two new exceptions in NewPipe Extractor: GeographicRestrictionException and SoundCloudGoPlusException (which extend to ContentNotAvailableException). These exceptions allow showing different error messages to user when a content isn't available in his/her/its country (only used for now by SoundCloudStreamExtractor) or when the content is a SoundCloud Go+ track.
2021-03-05 16:38:37 +01:00
TobiGr
cb07ffa1eb Release 0.20.11 2021-03-05 15:35:11 +01:00
Tobi
bc0cda68d1
Merge pull request #570 from TiA4f8R/fix-yt-channels-playlists
Fix YouTube channels contination
2021-03-05 15:26:49 +01:00
Tobi
a3c6fceef5
Merge pull request #573 from B0pol/comments-performance
[youtube] improve comments extraction performance
2021-03-05 14:46:27 +01:00
bopol
ff5273b882
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeCommentsExtractor.java
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2021-03-05 14:39:01 +01:00
Tobi
ec1127dd6a
Merge pull request #571 from TeamNewPipe/soundcloud_client_id
[SoundCloud] Update client_id and fix extraction
2021-03-05 14:21:14 +01:00
bopol
ed850d0688 [youtube] improve comments extraction performance
- do not parse responseBody twice for continuation
instead try to get commentsTokenInside with the new pattern ("sectionListRenderer")
and try again with the old pattern ("commentSectionRenderer") on failure
- do not unescape responseBody multiple times
   -> parse responseBody less times
2021-03-05 13:37:08 +01:00
TobiGr
440a808b8a Update mocks for channels 2021-03-05 13:13:36 +01:00
Tobi
b608587e4d
Merge pull request #572 from golfinq/fixed-comments
Fixes missing comments under videos
2021-03-05 11:42:40 +01:00
golfinq
05213175cd
minor edit, made variable final 2021-03-05 02:49:15 -05:00
golfinq
8bceb57a6c
formatting fix 2021-03-05 02:46:06 -05:00
golfinq
ae48bdea4c
Updated to reflect variable name change 2021-03-05 02:40:36 -05:00
TiA4f8R
b27efdc9a4
Remove clickTrackingParams in getNextPage method of YoutubeChannelExtractor
Because it's unused with the new request.

Co-authored-by: bopol <bopol@e.email>
2021-03-04 22:50:20 +01:00
TobiGr
2b671b15ce [YouTube] Use new continuation API for channels
Co-authored-by: TiA4f8R <74829229+tia4f8r@users.noreply.github.com>
2021-03-04 21:47:16 +01:00
TobiGr
33594d4aff [SoundCloud] Fix client id extraction
SoundCloud changed their Javascript structure. That caused the client_id to be in the middle of the file (at byte ~43000 ). To have a little buffer, we now fetch the first 50000 bytes.
2021-03-04 21:32:34 +01:00
TobiGr
fb2f37d223 [SoundCloud] Update hardcoded client id 2021-03-04 21:25:41 +01:00
XiangRongLin
9256b3b848
Merge pull request #567 from XiangRongLin/playlist_continuations
Playlist continuations
2021-03-04 19:05:32 +01:00
XiangRongLin
506cc5fbe9 regenerate mocks 2021-03-04 19:01:58 +01:00
XiangRongLin
03b00ff1d6 Move toJsonArray and toJsonObject to JsonUtils 2021-03-04 18:58:51 +01:00
FireMasterK
668b080b3a Use dynamic key. 2021-03-04 08:45:14 +01:00
XiangRongLin
43b46bd408 Use new youtube continuations api for playlists
Requires sending a POST request instead of GET.
clientName and clientVersion, which were required as headers previously now need to be part of the request payload.
continuation id also needs to be part of request body.

quick and dirty solution.
2021-03-03 19:49:26 +01:00
XiangRongLin
beb05bd05c Replace test url, because playlist was deleted 2021-03-03 19:46:14 +01:00
XiangRongLin
e13e237392 Fix typo and reword some explanations 2021-02-26 17:48:03 +01:00
Tobi
5d594cfded
Merge pull request #549 from FireMasterK/gradle-update
Update gradle wrapper.
2021-02-24 15:59:28 +01:00
Tobi
965a66bc87
Merge pull request #557 from TiA4f8R/fix-encoding-errors-windows
Set extractor encoding to UTF-8
2021-02-24 15:49:58 +01:00
TiA4f8R
b2cf41496d
Set extractor encoding to UTF-8
When compiling the extractor on Windows, you get more than 100 errors with the message error: unmappable character for encoding Cp1252, especially in timeago-parser tasks. This commit fixes it by setting the encoding for all compileJava and compileTestJava tasks to UTF-8.
2021-02-23 14:38:20 +01:00
TobiGr
a1688fe953 Move BandcampExtractorHelper.getJsonData(String, String) to JsonUtils 2021-02-21 13:51:12 +01:00
TobiGr
70814dcfef Fix Utils.nonEmptyAndNullJoin
When using the index here, it the index needs to be decremented once an element is removed. To cirecumvent this, the native Collections.removeIf() method is used.
2021-02-21 12:51:47 +01:00
TobiGr
b9e8ee8450 Rename BandcampExtractorHelper.smartConcatenate(String[], String) to Utils.nonEmptyAndNullJoin(String, String[]) 2021-02-21 12:35:14 +01:00
TobiGr
c07db80ef0 Add BASE_URL and BASE_API_URL to BandcampExtractorHelper 2021-02-21 12:15:45 +01:00
TobiGr
adde4332d1 Code improvements 2021-02-20 23:58:54 +01:00
FireMasterK
42bdd8adc5
Update to fix failing mock tests. 2021-02-20 17:14:41 +05:30
FireMasterK
e8bccfaf5d
Change UA to privacy.resistFingerprinting. 2021-02-20 17:14:41 +05:30
bopol
a9aa385bb3
Merge pull request #554 from B0pol/test-verified
use DownloaderFactory instead of DownloaderTestImpl for search Channe…
2021-02-20 12:12:43 +01:00
bopol
0c831afa7b use DownloaderFactory instead of DownloaderTestImpl for search ChannelVerified 2021-02-20 11:41:30 +01:00
TobiGr
ea120a4637 remove print stacktrace
use interface
2021-02-20 00:23:45 +01:00
XiangRongLin
adf9d7d10f Add reason field to MockOnly
This enforces developers to document why a test is skipped
2021-02-19 18:54:18 +01:00
XiangRongLin
ea52030613 Add MockOnlyRule to allow skipping specific tests based on downloader 2021-02-19 18:54:17 +01:00
TobiGr
02920fafa8 Add isUploaderVerified() 2021-02-19 15:51:23 +01:00
TobiGr
91e9309486 Merge remote-tracking branch 'origin/dev' into bandcamp 2021-02-19 13:37:45 +01:00
TobiGr
fa61b864f2 Code improvements 2021-02-19 13:37:24 +01:00
TobiGr
98268e351c Move radio URL check into a function 2021-02-19 13:35:44 +01:00
FireMasterK
959d5d7db7
Add dependabot. 2021-02-19 11:52:56 +05:30
bopol
27a20e41cd
Merge pull request #550 from FireMasterK/add-more-data
Add more data.
2021-02-18 20:28:27 +01:00
FireMasterK
5333d8a98b
Add more data. 2021-02-19 00:55:03 +05:30
FireMasterK
88890d81a5
Update gradle wrapper. 2021-02-18 21:15:12 +05:30
bopol
9c1a7f7df6
Merge pull request #521 from B0pol/verified
Add uploader verified by service extraction
2021-02-18 16:39:53 +01:00
bopol
1a322ad8ed Add uploader verified by service extraction 2021-02-18 16:36:40 +01:00
bopol
db253e202b
Merge pull request #535 from TeamPiped/adaptive-parsing
Adaptive Stream parsing
2021-02-18 15:55:27 +01:00
FireMasterK
c24afa2cbb Permalink URL. 2021-02-18 20:16:53 +05:30
FireMasterK
8d54401233 Add fps field as well. 2021-02-18 20:16:53 +05:30
FireMasterK
4d096be14a Revert "Update gradle wrapper."
This reverts commit 591856e3a11998c4b8ba87b5e361e2b9556d760f.
2021-02-18 20:16:53 +05:30
FireMasterK
5b70645d85 Add data to muxed streams aswell. 2021-02-18 20:16:53 +05:30
FireMasterK
e975d33fbe Change jdoc / comments. 2021-02-18 20:16:53 +05:30
FireMasterK
f31b2a68fd use getters/setters. 2021-02-18 20:16:53 +05:30
FireMasterK
d4945ac55c Constructor cleanup. 2021-02-18 20:16:52 +05:30
FireMasterK
3f6a601be8 Fix jdoc / comments. 2021-02-18 20:16:52 +05:30
FireMasterK
a7c9905183 Fix requested changes. 2021-02-18 20:16:52 +05:30
FireMasterK
0c0f2d74bc Make fields private
Co-authored-by: bopol <bopol@e.email>
2021-02-18 20:16:52 +05:30
FireMasterK
d1054338d3 Fix error with livestreams. 2021-02-18 20:16:52 +05:30
FireMasterK
11eb4932f4 Add data to respective classes. 2021-02-18 20:16:52 +05:30
FireMasterK
525e345ed8 remove duplicate field. 2021-02-18 20:16:52 +05:30
FireMasterK
e844d2aed3 Add dash fields to the ItagItem. 2021-02-18 20:16:52 +05:30
FireMasterK
b713a7af8c Update gradle wrapper. 2021-02-18 20:16:52 +05:30
bopol
64f6b0478f
Merge pull request #545 from XiangRongLin/comment_reset
Add comment explaining YoutubeParsingHelper.resetClientVersionAndKey
2021-02-18 15:14:03 +01:00
XiangRongLin
a64579da2f Unignore some tests that work on Linux 2021-02-17 20:31:26 +01:00
XiangRongLin
b99cb8624f Unignore PublicBroadcasterTest
Tests only work on linux, mocks have to be generated on linux too
2021-02-17 20:31:26 +01:00
XiangRongLin
37366c65ca Remove encoding of '&' to '&amp;' in test and unignore it 2021-02-17 20:31:26 +01:00
XiangRongLin
c327922cc0 Add explanations to broken tests 2021-02-17 20:31:26 +01:00
XiangRongLin
26d50aca05 Add comment explaining YoutubeParsingHelper.resetClientVersionAndKey 2021-02-17 19:21:39 +01:00
TobiGr
54aa5b3042 Use propper structure in KioskExtractors
Made BandCampRadioExtractor a Kiosk which holds StreamInfoItems and not InfoItems.
2021-02-16 21:09:42 +01:00
TobiGr
54b8e54f80 Fix potential NPE 2021-02-16 20:59:13 +01:00
TobiGr
16973126a0 Use Collections.emptyList() instead of Collections.EMPTY_LIST or new ArrayList() 2021-02-16 20:25:48 +01:00
bopol
84380e4c3a
Merge pull request #536 from B0pol/watchendpoints
support /watch/, /v/ and /w/ subpaths
2021-02-16 20:23:27 +01:00
bopol
41f689b099 Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/linkHandler/YoutubeStreamLinkHandlerFactory.java
Co-authored-by: Stypox <stypox@pm.me>
2021-02-16 19:17:10 +01:00
XiangRongLin
a9594f531f
Merge pull request #543 from XiangRongLin/unignore1
Unignore and fix some tests
2021-02-16 18:00:16 +01:00
XiangRongLin
a28e0226c5
Merge pull request #519 from XiangRongLin/better_logging_error_handling
Better logging, error handling, doc for mock tests
2021-02-16 15:06:06 +01:00
XiangRongLin
05cb22c654 Unignore Gronkh.testDescription
Remove special character ü from assertion
2021-02-16 08:59:35 +01:00
XiangRongLin
d9b4c2779f Unignore YoutubeSearchQHTest.testRegularValues 2021-02-16 08:59:35 +01:00
XiangRongLin
137272b7a4 Unignore YoutubeCommentsExtractorTest.EmptyComment
Mock files were missing
2021-02-16 08:59:34 +01:00
TobiGr
5bf9fddba9 Code improvements 2021-02-15 22:45:17 +01:00
TobiGr
e062c8cb0d Merge branch 'dev' into bandcamp 2021-02-15 22:09:41 +01:00
bopol
416cf17b7a
Merge pull request #532 from B0pol/utf8
Code improvement
2021-02-15 18:56:28 +01:00
bopol
7a3d9bdb7d Description: add .equals() and .hashCode() 2021-02-15 18:51:47 +01:00
bopol
557934cb17 use EMPTY_STRING 2021-02-15 18:51:46 +01:00
bopol
9ca52ca68a use only one constant UTF-8 2021-02-15 18:47:54 +01:00
XiangRongLin
50e5718bd1
Merge pull request #518 from XiangRongLin/generate_mocks
Use and generate mocks for youtube tests
2021-02-15 06:39:41 +01:00
Tobi
d7486f251a
Merge pull request #538 from B0pol/soundcloud-georestriction
Soundcloud: fix charts extraction when georestricted
2021-02-14 23:52:27 +01:00
TobiGr
432c68cdec Update okhttp from 3.12.11 to 3.12.13
Changelog https://square.github.io/okhttp/changelog_3x/#version-31213
2021-02-14 23:44:19 +01:00
TobiGr
5eb75d72d3 Release 0.20.10 2021-02-14 22:09:17 +01:00
XiangRongLin
a88b2e6af4 Undo formatting of YoutubeStreamExtractorDefaultTest 2021-02-14 14:57:23 +01:00
bopol
8d89c82caa Soundcloud: fix charts extraction when georestricted 2021-02-13 22:28:31 +01:00
bopol
ee2012c948 rename folder to subpath 2021-02-13 22:25:46 +01:00
bopol
8db40db4de use substring instead of split 2021-02-13 22:20:05 +01:00
bopol
81317cf6bc youtube: support /w/ folder 2021-02-13 22:20:05 +01:00
bopol
f15c0fcfed support /watch/ and /v/ folders 2021-02-13 22:20:05 +01:00
XiangRongLin
971128c464 Ignore failing tests 2021-02-13 20:28:03 +01:00
XiangRongLin
92dea0806a Regenerate json files for youtube tests 2021-02-13 20:28:03 +01:00
XiangRongLin
d6b9930c18 Use DownloaderFactory in YoutubeCommentsExtractorTest.Pinned 2021-02-13 20:12:39 +01:00
XiangRongLin
b555f38fa5 Use real downloader for YoutubeSubscriptionExtractorTest
It doesn't make network requests anyway
2021-02-13 20:12:39 +01:00
XiangRongLin
f352b0c7af YoutubeMusicSearchExtractorTest can't be used with mocks
Requires further research
2021-02-13 20:00:51 +01:00
XiangRongLin
f45b5610bf Unignore YoutubeChannelLocalizationTest
With mocks time is not a concern anymore
2021-02-13 20:00:51 +01:00
XiangRongLin
19737e06ee Use DownloaderFactory in YoutubeChannelExtractorTest
KurzgesagtAdditional needed to be extracted because it ran a duplicated request.
2021-02-13 20:00:51 +01:00
XiangRongLin
84eb4b30c9 Use DownloaderFactory in YouTube tests 2021-02-13 20:00:51 +01:00
XiangRongLin
b43f3474f5 Make resetClientVersionAndKey public 2021-02-13 20:00:49 +01:00
Tobi
8d7b62914c
Merge pull request #540 from B0pol/fix-tests
fix tests, fix links in YouTube description
2021-02-13 13:20:29 +01:00
bopol
2859c7ba5c add comment explaining why .substring(23) 2021-02-13 12:10:41 +01:00
bopol
73353996d7 fix more tests 2021-02-12 23:18:38 +01:00
bopol
a931e31239 fix tests 2021-02-12 22:36:18 +01:00
TobiGr
d116680ea3 Merge branch 'master' into dev 2021-02-12 22:26:35 +01:00
TobiGr
f8d769617a Release 0.20.9 2021-02-12 22:14:23 +01:00
Tobi
c2525916ac
Merge pull request #539 from B0pol/yt-channel-continuation
[YouTube]: Fixed channel continuation
2021-02-12 22:09:31 +01:00
bopol
b0f356dd76 YT channels: fix continuation 2021-02-12 14:39:09 +01:00
bopol
fcdb50b825 make getInfo from Extractor public
In StreamInfo and CommentsInfo
2021-02-09 14:58:43 +01:00
bopol
44c54d403a
fix Invidious channels being accepted by YoutubeStreamLinkHandlerFactory (#527)
fixes https://github.com/TeamNewPipe/NewPipeExtractor/issues/524, see it for more details
2021-02-05 18:57:58 +01:00
TiA4f8R
1414a6f178
Fix extraction of YouTube chapters (#531)
YouTube changed the name of the chapters in the JSON from engagement-panel-macro-markers to engagement-panel-macro-markers-description-chapters, so extracting chapters doesn't work.
2021-02-03 22:07:34 +01:00
Tobias Groza
bfa639950e
Merge pull request #528 from Stypox/ytmusic-0
[YouTube Music] Fix extracting search item view/subscriber count when = 0
2021-01-29 23:23:23 +01:00
TobiGr
d728c4fad0 Remove unused imports
I searched for "StandardCharsets" to prevent possible bugs with Android KitKat. The import is unused and is therefor removed.
There are other usages of the StandardCharsets class in the test implementation, but those are not relevant.
2021-01-29 23:16:40 +01:00
Stypox
a64dfd7343
[YouTube Music] Fix extracting search item view/subscriber count when = 0 2021-01-29 16:44:24 +01:00
Tobias Groza
c00ee75d74
Merge pull request #525 from B0pol/rename
rename getHeartedByCreator and getPinned to isPinned and isHeartedByC…
2021-01-23 13:45:24 +01:00
bopol
2662ceca22 rename getHeartedByCreator and getPinned to isPinned and isHeartedByCreator 2021-01-23 13:06:07 +01:00
Tobias Groza
10e8c543e8
Merge pull request #523 from B0pol/pinned
Extract whether the comment is pinned
2021-01-23 12:35:13 +01:00
bopol
66e4eb2f96 extract whether the comment is pinned 2021-01-22 23:59:22 +01:00
Tobias Groza
69f155d292
Merge pull request #522 from B0pol/invidious_instances
update invidious instances list
2021-01-22 21:04:10 +01:00
bopol
67ddfefdc6 keep invidious instances list in the same order
as suggested by @TobiGr
2021-01-22 20:58:14 +01:00
bopol
53d3f7989b update invidious instances list 2021-01-22 19:30:25 +01:00
XiangRongLin
88e4c8667b Add comments to RecordingDownloader about usage 2021-01-17 19:53:20 +01:00
XiangRongLin
eecfe09f2c Check for non-existent folder in MockDownloader 2021-01-17 19:52:58 +01:00
XiangRongLin
a0288b8fd6 Echo which downloader is used during CI process 2021-01-17 19:52:28 +01:00
Tobias Groza
beb705013c
Merge pull request #516 from XiangRongLin/schedule_ci_with_real
Add if-else statement to run tests against real website on schedule
2021-01-16 12:59:05 +01:00
XiangRongLin
7a6d930e7b Add if-else statement to run tests against real website on schedule 2021-01-16 10:03:47 +01:00
TobiGr
07570554b6 Fix build and optimize imports 2021-01-15 21:55:40 +01:00
TobiGr
78c2113094 Merge remote-tracking branch 'origin/dev' into bandcamp 2021-01-15 21:49:58 +01:00
Stypox
b9ba95614b
Merge pull request #514 from XiangRongLin/test_jvm_property
Pass on gradle JVM system property to test JVM
2021-01-15 20:44:23 +01:00
XiangRongLin
a6cfe11ae9 Pass on gradle JVM system property to test JVM 2021-01-15 20:37:14 +01:00
Stypox
48a9993edd
Merge pull request #513 from XiangRongLin/ci_use_mock
Adjust CI workflow to run tests with mocks
2021-01-15 20:20:23 +01:00
Stypox
588fbcc61f
Merge pull request #512 from XiangRongLin/ignore_failing
Ignore all failing tests
2021-01-15 20:18:03 +01:00
XiangRongLin
6888e7773f Ignore all failing tests 2021-01-15 20:11:40 +01:00
XiangRongLin
e5dc62d9bd Add argument -Ddownloader=MOCK to CI job
Now MockDownloader will be used in tests.
2021-01-15 18:04:33 +01:00
Stypox
7c774c84ce
Merge pull request #505 from TeamNewPipe/meida.ccc.de_search_fix
[media.ccc.de] Fix NPE in search results if they contain a future talk
2021-01-15 14:16:59 +01:00
Stypox
c2c4d97efd
Merge pull request #482 from XiangRongLin/mock_mix_pl_test
Mock mix pl tests
2021-01-15 12:54:48 +01:00
XiangRongLin
4f81d9d689 Regenerate mock files 2021-01-15 09:51:23 +01:00
Stypox
951159f75f
Merge pull request #504 from TeamNewPipe/media.ccc.de_recent_order
[media.ccc.de] Recent kiosk: order streams by upload date
2021-01-14 23:31:09 +01:00
Tobias Groza
cc51c5f6a3
Merge pull request #507 from vkay94/add-interval-to-stream-frames
Add durationPerFrame to Frameset
2021-01-14 20:27:40 +01:00
XiangRongLin
6ca7123d5d Add comment for usage of YoutubeParsingHelper.resetClientVersionAndKey 2021-01-14 20:01:52 +01:00
vkay94
11dcfe638b Add durationPerFrame to Frameset and getFrameBoundsAt method 2021-01-14 20:01:06 +01:00
Tobias Groza
c2ff6723d7
Merge pull request #511 from B0pol/errors
Fix PeerTube tests by changing instance
2021-01-14 19:15:22 +01:00
Tobias Groza
6773ec8699
Merge pull request #510 from B0pol/heartedbycreator
comments: add heartedByUploader and extract it for YouTube
2021-01-12 19:00:04 +01:00
TobiGr
0c4836c800 [media.ccc.de] Recent kiosk: Add comment clarifying "length" and "duration" fields have the same value 2021-01-12 18:48:42 +01:00
XiangRongLin
4af50c6870 Use @link for downloader 2021-01-12 09:41:57 +01:00
XiangRongLin
3455f0f23c Add documentation to testing with mocks 2021-01-12 09:37:48 +01:00
bopol
5292cc1636 [peertube] test: remove usage of peertube.mastodon.host
It's down for months
2021-01-11 19:46:27 +01:00
bopol
f3c22da618 comments: add heartedByCreator and extract it for youtube 2021-01-11 18:48:08 +01:00
Tobias Groza
da968cf9c5
Merge pull request #508 from XiangRongLin/360_webm
[YouTube] Un-comment out 360p webm
2021-01-11 16:09:05 +01:00
XiangRongLin
63c237db41 Only create directories if they do not exist 2021-01-11 07:57:11 +01:00
XiangRongLin
e82cb1efea Remove dependency to commons-io 2021-01-10 20:50:38 +01:00
XiangRongLin
1ea6c6ce54 Prefix mock file from RecordingDownloader with "generated_mock_"
Only read those files in MockDownloader
2021-01-10 20:46:42 +01:00
XiangRongLin
f91916c017 Remove unnecessary file object creation 2021-01-10 20:39:22 +01:00
XiangRongLin
255c726f20 Add resource path base to DownloaderFactory 2021-01-10 20:27:34 +01:00
XiangRongLin
35e299759e Add method to allow resetting youtube client version and key
This is needed so that a request is made for each test class when running multiple at once. This way RecordingDownloader records all necessary requests.
This works as long as tests are run sequentially and not in parallel.
2021-01-10 20:24:50 +01:00
XiangRongLin
f447a7a450 fix import due to downloader package move 2021-01-10 19:49:14 +01:00
XiangRongLin
f8aa989c42 Add generated json and manually copy client_version.json 2021-01-10 19:38:47 +01:00
XiangRongLin
285c26eafa Adjust YoutubeMixPlaylistExtractorTest to use DownloaderFactory 2021-01-10 19:38:41 +01:00
XiangRongLin
e6e8e39def Add DownloaderFactory to return a specific downloader based on 2 variables.
If the system property 'downloader' is set that use that specific downloader. This is used from gradle by appending `-Ddownloader=ABCD to the command.
ABCD is one of DownloaderType.
The other variable is the static property `DEFAULT_DOWNLOADER` in DownloaderFactory, which can be easily changed as needed inside the IDE according to development needs`.

Normal workflow would be to first use the recording downloader and afterwards only use mocks, if the requests are always staying the same.
2021-01-10 19:36:24 +01:00
XiangRongLin
7c40fb8bf7 Add additional downloader implementations
RecordingDownloader relies on the real downloader and saves the request/response pair into a json file.
MockDownloader uses json files from above and mocks responses for specific requests.
2021-01-10 19:36:24 +01:00
XiangRongLin
1bcb9c76a7 Generate equals and hashCode for Request 2021-01-10 19:36:24 +01:00
XiangRongLin
4dad3d60d6 Move DownloaderTestImpl into downloader package 2021-01-10 19:36:21 +01:00
XiangRongLin
fe3902c669 Un-comment out 360p webm 2021-01-08 19:43:03 +01:00
TobiGr
2d93b23723 [media.ccc.de] Show conference title in search results instead of slug 2021-01-08 14:15:25 +01:00
TobiGr
52f3bd15f2 [media.ccc.de] Add conference title to "recent" kiosk
The "conference_title" field was added upstream recently (see https://github.com/voc/voctoweb/issues/495)
2021-01-08 12:04:39 +01:00
Tobias Groza
ab62464e86
Merge pull request #495 from B0pol/sepiasearch
[PeerTube] Implement sepia search
2021-01-06 11:06:36 +01:00
bopol
0bcea9c2cd add JDoc 2021-01-01 14:37:32 +01:00
bopol
0e45c25ea9
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/peertube/extractors/PeertubeStreamInfoItemExtractor.java
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2021-01-01 14:20:47 +01:00
TobiGr
fae67fbd45 Merge remote-tracking branch 'origin/master' into dev 2021-01-01 13:06:11 +01:00
Tobias Groza
c4b9183684
Merge pull request #506 from TeamNewPipe/gh-actions
Migrate to GitHub actions from Travis
2021-01-01 10:35:33 +01:00
TobiGr
4671e68304 Update links to Java SE components to Java SE 8 2020-12-31 18:05:49 +01:00
TobiGr
d63d20f943 Fix JDoc 2020-12-31 18:05:40 +01:00
FireMasterK
1ba689900e Migrate to GitHub actions from Travis. 2020-12-31 18:05:35 +01:00
TobiGr
2cbc3ccb1a [media.ccc.de] Fix live stream test if there are no current live streams 2020-12-31 17:03:23 +01:00
TobiGr
a896ec41b9 [media.ccc.de] Use Pattern for getting the ive stream id 2020-12-31 17:03:23 +01:00
TobiGr
5dc9a76e3c [media.ccc.de] Recent kiosk: order streams by upload date 2020-12-30 18:38:09 +01:00
TobiGr
edf8dd0e92 [media.ccc.de] Fix NPE in search results if they contain a future talk 2020-12-30 18:34:28 +01:00
Tobias Groza
3c8c8e7307
Merge pull request #497 from TiA4f8R/fix-player-js-url-redirect
Fix YouTube player JavaScript URL redirection
2020-12-27 18:04:40 +01:00
TobiGr
288ddab65b update version name 2020-12-27 18:02:16 +01:00
TiA4f8R
d30caeb091
Fix YouTube player JavaScript URL redirection 2020-12-27 15:55:03 +01:00
TobiGr
deb9af7bf5 [media.ccc.de] live kiosk - display info of the current talk 2020-12-27 15:52:09 +01:00
TobiGr
21c6a7884c Release 0.20.8 2020-12-27 14:50:34 +01:00
Tobias Groza
1259521d53
Merge pull request #500 from TeamNewPipe/media.ccc.de_live
[media.ccc.de] live stream kiosk
2020-12-27 14:47:55 +01:00
TobiGr
b4e1913971 [media.ccc.de] Play live streams 2020-12-27 13:25:48 +01:00
TobiGr
80f4d42226 [Draft] Add live extractor 2020-12-27 11:01:22 +01:00
TobiGr
ed9402c002 [media.ccc.de] Escape "." in id pattern 2020-12-27 11:01:10 +01:00
TobiGr
674f2227f3 make url pattern static 2020-12-27 01:27:37 +01:00
TobiGr
090dade5b3 [media.ccc.de] Add short test for recent kiosk 2020-12-27 01:03:08 +01:00
TobiGr
0cfefe222a Revert "[media.ccc.de] Get conference name from API URL"
This reverts commit d1a0686d2ffd3eedf0f503fc7c63ca8bd943d8e5.
2020-12-26 16:55:44 +01:00
TobiGr
d1a0686d2f [media.ccc.de] Get conference name from API URL 2020-12-26 13:15:03 +01:00
TobiGr
41699798fe [media.ccc.de] Use web URL for uploaderUrl instead of API URL in recent 2020-12-26 12:47:32 +01:00
TobiGr
b2154c6e87 [media.ccc.de] Fix conferences kiosk test 2020-12-25 23:55:45 +01:00
TobiGr
ed6ae329b3 [media.ccc.de] Add "recent" kiosk 2020-12-25 22:26:58 +01:00
TobiGr
b2837698f5 Release 0.20.7 2020-12-25 18:55:06 +01:00
Tobias Groza
b5e50cc9fb
Merge pull request #484 from TeamNewPipe/yt_,music_search
Fix YouTube Music search
2020-12-25 18:30:25 +01:00
TobiGr
d9e2da53c3 Replace explicit string checks whether a playlist íd is a certain YouTube Mix type with calling the dedicated methods 2020-12-25 15:00:34 +01:00
TobiGr
abee0a8df1 Fix typos in YoutubePlaylistLinkHandlerFactory 2020-12-24 10:34:21 +01:00
TobiGr
3033c0b993 [YouTube] Music Mix: Fix playlist not being accepted
Regression introduced by YouTube Mix support (#280)
2020-12-23 21:08:36 +01:00
TobiGr
b62144b49d [YouTube] Music: Fix search extraction caused by changed JSON response 2020-12-23 21:08:36 +01:00
TobiGr
c9d9bd1e24 Fix typo in ListLinkHandlerFactory#getUrl 2020-12-23 21:05:58 +01:00
bopol
3ae924a7f1 Implement sepia search 2020-12-23 15:07:55 +01:00
TobiGr
942d840624 Fix typo in AffiliateService: rename "NO_AFILIATE" to "NO_AFFILIATE" 2020-12-23 12:50:45 +01:00
TobiGr
865c42e273 Fix line break 2020-12-23 12:49:50 +01:00
Tobias Groza
c682ea0d18
Merge pull request #451 from TeamNewPipe/meta_info
Extract stream and search meta info for YouTube
2020-12-22 23:05:33 +01:00
Fynn Godau
c9e9953bb0 [Bandcamp] Fix channel link handler factory 2020-12-22 20:02:53 +01:00
TobiGr
41a8ed625d Make some vars final and add annotations to methods 2020-12-22 18:19:26 +01:00
TobiGr
bc6de14952 Extract stream and search meta info for YouTube
Add method to extract Google webcache URLs.
2020-12-22 18:19:26 +01:00
TobiGr
50903730b1 [Bandcamp] Fix accepting HTTP URLs 2020-12-22 08:53:10 +01:00
Tobias Groza
853a65a1a6
Merge pull request #491 from TeamNewPipe/dependecies
Update junit
2020-12-22 08:31:03 +01:00
TobiGr
838dc1ab54 Fix vulnerability in junit 2020-12-21 23:22:16 +01:00
Tobias Groza
7e481cf81a
Merge pull request #490 from B0pol/peertube_related
[peertube] fix related streams api url when no tags
2020-12-20 19:04:35 +01:00
bopol
8d3436565f [peertube] fix related streams api url when no tags
fixes #489
2020-12-19 23:25:37 +01:00
Tobias Groza
22a415156f
Merge pull request #479 from vkay94/stream-segments
Extract stream segments for YouTube
2020-12-19 11:12:22 +01:00
TobiGr
74b46fed2d [Bandcamp] Fix deprecation in parseDate helper function 2020-12-19 10:40:21 +01:00
TobiGr
116e921d6c Merge remote-tracking branch 'origin/dev' into bandcamp 2020-12-19 00:26:11 +01:00
Tobias Groza
e8cc302d5f
Merge pull request #483 from XiangRongLin/cleanup-mix-pl
Cleanup mix pl
2020-12-19 00:13:17 +01:00
TobiGr
df16a8646f [Bandcamp] Improve radio stream extractor test 2020-12-18 23:15:59 +01:00
XiangRongLin
cdcb66b93e Clean up stuff from mix pl
Coming from unclean merge and missing hints from IDE
2020-12-16 08:35:28 +01:00
XiangRongLin
ba8782a9ed Use lowercase string for extracting cookies
The map in a real response happens to ignore upper-/lowercase differences. Other maps used in unit test may not have that behaviour.
2020-12-16 08:32:04 +01:00
Stypox
85fa006214
Merge pull request #280 from XiangRongLin/mixPL
Extractor for youtube mix (auto-generated playlist)
2020-12-14 18:11:07 +01:00
XiangRongLin
f90f6fcf92 [YouTube] Don't escape & in getUrlFromNavigationEndpoint for playlists 2020-12-12 20:40:13 +01:00
Xiang Rong Lin
a338e4e08e [Youtube] Apply review suggestions and avoid channel mix edge case 2020-12-12 20:32:43 +01:00
Xiang Rong Lin
22d2f7e400 [Youtube] Add cookies to youtube mix request
This way youtube wont return duplicates when getting more items of the mix (but youtube can also track us)
2020-12-12 20:32:40 +01:00
Stypox
421935401f [Youtube] Add subchannel functions to mix and fix imports 2020-12-12 20:30:57 +01:00
Xiang Rong Lin
3ff8619bcc [Youtube] apply wb9688 suggestion (mix)
Channel mix adjusments and test
Don't accept youtube music mix urls as playlist
Don't override playlistData to keep getInitialPage()
Remove json constants
Indentation
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
822cf307f7 [Youtube] Add _ITEMS constants and improve code style
Move thumbnail id exctraction code to getThumbnailUrlFromId
Add test for "My mix" detection to service tests
Use ITEM_COUNT_UNKNOWN everywhere instead of -1 and add some tests
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
df38b1926c [Youtube] Add tests and take thumbnail image always from first video of mix
Also fix getThumbnailUrl for "My Mix"
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
68a3948af6 [Youtube] Fix get banner url 2020-12-12 20:30:57 +01:00
Xiang Rong Lin
ec6b99c082 [Youtube] Adjust mix extractor to new user agent
Also extract continuation now
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
0ff054acb4 [Youtube] Extract initial playlist info 2020-12-12 20:30:57 +01:00
Xiang Rong Lin
a376792a5d [Youtube] Handle case where url is in "youtube.com/playlist?list=listID" format.
This occurs when sharing a mix from the official youtube app.
2020-12-12 20:30:57 +01:00
Xiang Rong Lin
327a5730a8 [Youtube] Add some comments to mix 2020-12-12 20:30:57 +01:00
Xiang Rong Lin
d74265c846 [Youtube] Extract getThumbnailUrl into method and change getUploaderName 2020-12-12 20:30:57 +01:00
Xiang Rong Lin
0efb854d27 [Youtube] Implement mix extractor for auto-generated playlists.
-New YoutubeMixPlaylistExtractor, that extracts from a mix (auto-generated playlist).
-The url has the format of "youtube.com/watch?v=videoID&playlistID",
where playlistID always starts with "RD" and usually followed by the videoID.
-Change YoutubePlaylistLinkHandlerFactory to create a linkhandler with the given url if it is a mix.
-Change YoutubeService to return YoutubeMixPlaylistExtractor if the url is a mix.
2020-12-12 20:30:57 +01:00
vkay94
2ba27b39af Stream segments: Remove replaceAll from setPreviewUrl 2020-12-12 15:00:45 +01:00
Stypox
2b622fd2f1
Merge pull request #419 from Isira-Seneviratne/Use_Objects_methods
Use Objects methods.
2020-12-12 14:52:42 +01:00
vkay94
9a7a224a54 Add tests for stream segments 2020-12-12 10:24:55 +01:00
vkay94
d3eea4f4be Add stream segments to StreamInfo 2020-12-12 10:24:29 +01:00
Isira Seneviratne
57be1f1123 Use Objects.requireNonNull(). 2020-12-11 06:09:57 +05:30
Isira Seneviratne
4c19a88612 Use Objects.toString(). 2020-12-11 06:09:50 +05:30
Isira Seneviratne
b8f64595a2 Use Objects' static equals() and hashCode() methods. 2020-12-11 06:09:44 +05:30
TobiGr
b3835bd616 Release 0.20.6 2020-12-09 23:30:23 +01:00
Tobias Groza
6fbcdd24ee
Merge pull request #477 from vkay94/fix-comments-parsing
Fix comments parsing
2020-12-09 23:25:51 +01:00
vkay94
9dbacbc618 Fix comments parsing 2020-12-09 14:57:38 +01:00
Fynn Godau
04dd3d4d32 Rework link handlers to correctly accept external websites 2020-12-05 15:08:26 +01:00
Tobias Groza
8ade913d9b
Merge pull request #472 from TeamNewPipe/fix-media-ccc-de
Fix name of media.ccc.de service
2020-12-02 22:16:52 +01:00
TheAssassin
ba3e2302bc Fix name of media.ccc.de service 2020-12-01 23:54:27 +01:00
TobiGr
175df679e0 Release 0.20.5 2020-12-01 08:18:29 +01:00
Tobias Groza
b7a995187f
Merge pull request #463 from opusforlife2/update_invidious_URLs
Update Invidious URL List
2020-11-28 13:30:32 +01:00
Fynn Godau
be562b8436 Change tests 2020-11-27 16:48:40 +01:00
TobiGr
c91e21b37c [Bandcamp] Tests - Add finals and improve code formatting 2020-11-24 14:13:31 +01:00
TobiGr
99e7ef013e [Bandcamp] Apply small changes to code formatting and style
Make variables final when possible
Remove unused imports
Improve code formatting
2020-11-24 14:13:31 +01:00
fynngodau
8c369b0f79
Rephrase link in javadoc
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-11-24 14:01:31 +01:00
TobiGr
5a3f96b967 Release 0.20.4 2020-11-22 22:29:08 +01:00
Tobias Groza
650f0920fe
Merge pull request #465 from XiangRongLin/playlist_continuation
[YouTube] Fix playlist continuations extraction
2020-11-20 20:48:20 +01:00
TobiGr
334e1e9b53 Fix YouTube comments test 2020-11-20 19:53:53 +01:00
Fynn Godau
186936d041 Various changes regarding tests 2020-11-19 22:33:52 +01:00
Fynn Godau
6bc7e3420e Merge TNP/dev into fynngodau/dev 2020-11-19 21:32:08 +01:00
TobiGr
3582a5189f Release 0.20.3 2020-11-18 20:41:25 +01:00
XiangRongLin
5bceff0083 [YouTube] Fix extraction of next page url for the last page of playlist 2020-11-18 19:03:12 +01:00
XiangRongLin
8347e14952 [YouTube] Fix playlist continuations extraction 2020-11-17 21:04:53 +01:00
opusforlife2
2174055ef8
Update Invidious URL list in Parsing Helper 2020-11-11 15:12:31 +00:00
opusforlife2
c0afd5213a
Update Invidious URL list in Link Handler Factory 2020-11-11 14:54:16 +00:00
Tobias Groza
6701b0fe71
Merge pull request #447 from Stypox/better-deobfuscation
[YouTube] Cache deobfuscation code and more fixes
2020-11-08 00:32:36 +01:00
Stypox
89a77ae74a
[YouTube] Fix detection of ended livestreams and parse livestream upload date 2020-11-04 16:03:08 +01:00
Stypox
827f7bd137
[YouTube] Cache deobfuscation and improve requests made
Fix age restriction extraction
Automatically fixes more things
2020-11-04 16:02:12 +01:00
Stypox
b42a196f35
Merge pull request #454 from Isira-Seneviratne/Fix_date_time_issues
Fix some issues related to the date/time changes.
2020-11-04 14:48:20 +01:00
Isira Seneviratne
4fe28d7e3a Fix YouTube parse error when only a date is present. 2020-11-04 05:49:42 +05:30
Isira Seneviratne
9cf9e7e980 Call existing constructor in DateWrapper. 2020-11-04 05:49:41 +05:30
Isira Seneviratne
fe31a90cb3 Remove DateTimeFormatter.ISO_OFFSET_DATE_TIME usage. 2020-11-04 05:49:41 +05:30
Tobias Groza
b13c7e1c1e
Merge pull request #452 from Stypox/yt-import
Implement YouTube subscription import from Google takeout
2020-11-03 20:32:17 +01:00
Tobias Groza
ac50068bbe
Merge pull request #456 from B0pol/youtube_comments_test
create YouTubeCommentsLinkHandlerFactoryTest
2020-11-03 20:14:14 +01:00
bopol
345e136f6c create YouTubeCommentsLinkHandlerFactoryTest
and remove invidious test from YouTubeCommentsExtractorTest, because it was just testing if the URL is accepted, then the extractor does the same thing, we don't need to test the same thing twice
2020-11-03 19:10:10 +01:00
Stypox
501ec30152
Implement youtube subscription import from Google takeout 2020-11-02 11:04:48 +01:00
Tobias Groza
564a965810
Merge pull request #453 from bd0n4lds/fix-donation-typo
Fix typo in DonationLinkHelper and rewrote swtich statement
2020-11-02 00:15:04 +01:00
Bri@n
82746d172f Fix typo in DonationLinkHelper and rewrote swtich statement 2020-11-01 17:34:34 -05:00
Stypox
2f02c0e6a4
Merge pull request #450 from B0pol/fileutils
create FileUtils
2020-11-01 18:13:57 +01:00
bopol
f69b3ef05d create FileUtils 2020-11-01 17:15:34 +01:00
TobiGr
be9e160333 Fix build 2020-11-01 17:04:02 +01:00
Tobias Groza
8cbdec675b
Merge pull request #417 from Isira-Seneviratne/Use_Java_8_Date_Time_API
Use the Java 8 Date/Time API.
2020-11-01 16:49:06 +01:00
Isira Seneviratne
fcdfe7d939 Update README.md. 2020-11-01 06:28:54 +05:30
Isira Seneviratne
4f04cfccca Switch from Calendar to OffsetDateTime in DateWrapper. 2020-11-01 06:28:54 +05:30
Isira Seneviratne
ee3af63c04 Switch to ChronoUnit. 2020-11-01 06:28:53 +05:30
Isira Seneviratne
0526a5148d Set source and target versions to Java 8. 2020-11-01 06:28:52 +05:30
Isira Seneviratne
b2d0c098a3 Merge branch 'dev' of https://github.com/TeamNewPipe/NewPipeExtractor into dev 2020-11-01 06:27:59 +05:30
Stypox
6cc50b57e3
Merge pull request #410 from Scrxtchy/sets-track-title
Fix parsing Soundcloud tracks that contain the term 'sets'
2020-10-31 21:43:21 +01:00
Stypox
b242e1d113
Merge pull request #362 from Stypox/google-search-urls
Add support for Google search redirect url
2020-10-31 21:03:48 +01:00
Stypox
9e53cf0b56
Fix parameter reassignment and other style issues
Also remove left-behind debug statement
2020-10-31 20:58:28 +01:00
Stypox
3fe55b30ba
Add support for Google search redirect url 2020-10-31 20:58:28 +01:00
Stypox
c190a3029b
Consider protocol as base url when it is a custom one (e.g. vnd.youtube) 2020-10-31 20:58:28 +01:00
Isira Seneviratne
35c7b5640c Merge branch 'dev' of https://github.com/TeamNewPipe/NewPipeExtractor into dev 2020-10-31 05:45:02 +05:30
Stypox
a9303b24a5
Merge pull request #445 from B0pol/peertube_fix
[PeerTube] fix account and channel extractors
2020-10-30 19:35:51 +01:00
bopol
b21e59925d [PeerTube] fix account and channel extractors 2020-10-29 19:52:29 +01:00
Isira Seneviratne
61dcbbb800 Merge branch 'dev' of https://github.com/TeamNewPipe/NewPipeExtractor into dev
 Conflicts:
	extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java
2020-10-29 12:20:16 +05:30
Stypox
bcc01b99c7
Merge pull request #444 from Stypox/error-message
Remove any reference to decrypt and improve error message
2020-10-28 23:41:04 +01:00
Stypox
30ed4f2d63
Remove any reference to decrypt and improve error message 2020-10-28 20:26:00 +01:00
TobiGr
62912ee834 Merge branch 'master' into dev 2020-10-28 14:28:06 +01:00
TobiGr
94176c5d68 Merge branch 'master' into dev 2020-10-28 07:52:50 +01:00
Tobias Groza
19c0e8700d
0.20.2 2020-10-28 07:29:42 +01:00
Tobias Groza
354426fe5d
Merge pull request #441 from B0pol/fix-three-attempts
Support new YouTube JSON scheme
2020-10-28 07:24:31 +01:00
Scratch
947ce3ee10 Fix parsing Soundcloud tracks that contain the term 'sets' 2020-10-28 07:55:02 +11:00
bopol
6dc5ab4015 find playerUrl in another place when assetsPattern regex fails 2020-10-27 13:48:58 +01:00
bopol
db0ef83d6b fix youtube decryption and three attemps bug
fixes teamnewpipe/newpipe#4572 fixes #439
2020-10-27 13:40:24 +01:00
Stypox
943334ba49
Merge pull request #440 from B0pol/fix-redirect-channels
Fix redirect channels
2020-10-26 20:00:06 +01:00
bopol
c1e9857960 fix subscriber count when subscribe is disabled
fixes #305
2020-10-26 19:06:01 +01:00
bopol
a39a2cca82 fix redirect channels 2020-10-26 19:05:19 +01:00
Tobias Groza
a8665fd97f
Merge pull request #442 from B0pol/polish
Polish
2020-10-26 18:51:47 +01:00
bopol
0a12300c5e polish tests 2020-10-26 16:57:37 +01:00
bopol
01f49e8f66 polish strings 2020-10-26 16:32:39 +01:00
Tobias Groza
14c0c37d64
Merge pull request #309 from Stypox/improve-stream-tests
Improve stream extractor tests & various fixes
2020-10-25 18:49:53 +01:00
Stypox
df26f61cfc
Merge pull request #394 from kirisakow/dev
Fix TeamNewPipe/NewPipe#4213: [YouTube] Ignore leading characters in video id
2020-10-25 14:39:03 +01:00
Kiril Isakov
366ed71523
[YouTube] Ignore leading characters in video id 2020-10-25 14:34:20 +01:00
Stypox
8026304a0a
Merge pull request #409 from Bartoshr/bugfix/short-channel-urls
Support short custom YouTube channel URLs
2020-10-25 11:43:50 +01:00
Stypox
57e7994c9e
Add some missing finals, nullables and comments 2020-10-25 08:12:41 +01:00
Stypox
f11fe87688
[YouTube] Replace outdated PewDiePie video test with another one
The old video was made private, and this video (wedding) is probably never going to be removed.
2020-10-24 18:41:41 +02:00
Stypox
3c55ea9321
[PeerTube] Change age restricted video in tests
The old one wasn't available anymore
2020-10-24 18:41:41 +02:00
Stypox
880ff04a5c
[Test] Add stream dash mpd url test 2020-10-24 18:41:41 +02:00
Stypox
d0b14644bb
[YouTube/MediaCCC] Consider dates as GMT and not as local 2020-10-24 18:41:41 +02:00
Stypox
8ce711f40f
[Test] Add sub channel name, url and thumbnail tests 2020-10-24 18:41:41 +02:00
Stypox
6127826571
[Test] Add stream metadata tests 2020-10-24 18:41:41 +02:00
Stypox
a087b092b4
[Test] Improve code style and add final 2020-10-24 18:39:55 +02:00
Stypox
a4097d8d01
[MediaCCC] Return empty list of video-only streams instead of null 2020-10-24 18:39:55 +02:00
Stypox
8dc3f28618
[PeerTube] Test one channel url with api and one without 2020-10-24 18:39:55 +02:00
Stypox
68d23defba
[YouTube] Do not catch every exception on getErrorMessage
@B0pol suggestion
2020-10-24 18:39:55 +02:00
Stypox
3191bd6c70
[YouTube] Use final when possible 2020-10-24 18:39:55 +02:00
Stypox
55bc01d1ce
[SoundCloud] Use final when possible, ide refactorings 2020-10-24 18:39:55 +02:00
Stypox
06430c4749
[PeerTube] Use final when possible, ide refactorings 2020-10-24 18:39:06 +02:00
Stypox
fcb9b6f855
[MediaCCC] Use final when possible, ide refactorings
Refactorings on `throws` clause
2020-10-24 18:36:22 +02:00
Stypox
af5b8b1915
[MediaCCC] Add tests for stream and conference link handlers 2020-10-24 18:36:21 +02:00
Stypox
07a90d116a
[MediaCCC] Use regex to parse stream and conference urls 2020-10-24 18:36:21 +02:00
Stypox
0c980b2d64
[PeerTube] Improve channel and stream link handler tests 2020-10-24 18:36:21 +02:00
Stypox
d130fd79c3
[PeerTube] Prepend "accounts/" to channel id for backward compatibility 2020-10-24 18:36:21 +02:00
Stypox
492db83ccf
[MediaCCC] Return null instead of empty items collector
As per the documentation in the base getRelatedStreams()
2020-10-24 18:34:57 +02:00
Stypox
3b2a1829d4
[MediaCCC] Extract tags 2020-10-24 18:34:57 +02:00
Stypox
b461da792f
[MediaCCC] Fix link handler inconsistency providing API links 2020-10-24 18:34:34 +02:00
Stypox
aeeae87641
[PeerTube] Parse timestamp from url (previously unimplemented) 2020-10-24 18:33:42 +02:00
Stypox
7ae3cb6d07
[PeerTube] Fix link handler inconsistency providing API links 2020-10-24 18:33:11 +02:00
Stypox
4349be13af
[PeerTube] Return empty audio stream list instead of null 2020-10-24 18:25:56 +02:00
Stypox
3b2cfb4ca2
[SoundCloud] Return empty video stream list instead of null
Also replace every instance of `return new ArrayList<>();` with `return Collections.emptyList();`
2020-10-24 18:23:46 +02:00
Stypox
072bae321f
[YouTube] Fix frame extraction for livestreams
Use saved playerResponse instead of parsing json every time
2020-10-24 18:23:46 +02:00
Stypox
7cd410f3fc
[YouTube] Return 0 when there is no timestamp, not -2, as per javadoc 2020-10-24 18:23:45 +02:00
Stypox
7fb867c166
[YouTube] Fix error message obtaining when there is none 2020-10-24 18:22:44 +02:00
Stypox
1d7a86e664
[Test] Add base classes for stream extractor tests
Refactor all stream extractor tests to use new base class.

Remove check if upload date is in the past: this does not have to hold true: youtube premieres turn up in search results even though they are in the future
2020-10-24 18:22:44 +02:00
Bartosz Rumiński
29695aed0a Small field refactor 2020-10-23 16:42:30 +02:00
Bartosz Rumiński
0e67d820bc Use static regex pattern for excluded path segments 2020-10-22 20:13:29 +02:00
Bartosz Rumiński
d3f80d1538 Exlude links which are not channels 2020-10-20 20:06:06 +02:00
Tobias Groza
3058af0e43
Merge pull request #418 from Stypox/yt-shorts
[YouTube] Handle URLs for Shorts
2020-10-18 13:06:10 +02:00
Stypox
6887d59570
[YouTube] Handle urls for Shorts 2020-10-18 12:03:01 +02:00
Scratch
e945f711c1
Fix SoundCloud test artist account name (#416) 2020-10-16 23:42:46 +02:00
TobiGr
350eed6214 Release v0.20.1 2020-10-16 22:03:57 +02:00
Tobias Groza
f82eda9f7d
Merge pull request #415 from wb9688/fix-ytinitialdata
Fix parsing new ytInitialData
2020-10-16 21:59:39 +02:00
wb9688
be9a6f931c Fix parsing new ytInitialData 2020-10-16 20:27:40 +02:00
Tobias Groza
1a9bc8caf1
Merge pull request #414 from Scrxtchy/soundcloud-slash
Remove tailing slash from SoundCloud URLs
2020-10-16 09:48:44 +02:00
Scratch
6a70cb9d50 Remove tailing slash from SoundCloud URLs
Fixes #412
2020-10-16 17:17:40 +11:00
Tobias Groza
9a785f9a7e
Merge pull request #413 from Scrxtchy/soundcloud-test-update
Fix SoundCloud test artist account name
2020-10-16 08:05:44 +02:00
Scratch
538f5d3973 Fix SoundCloud test artist account name 2020-10-16 13:45:05 +11:00
Tobias Groza
527945eadb
Merge pull request #406 from Stypox/fix-decrypt
[YouTube] Fix some decryption exceptions by retrying
2020-10-15 22:03:25 +02:00
Stypox
19e862657a
[YouTube] Fix some decryption exceptions by retrying 2020-10-15 20:16:44 +02:00
Bartosz Rumiński
e3f996e014 Exlude links which are not channels 2020-10-12 20:59:56 +02:00
Bartosz Rumiński
5ab1b053d2
Update youtube channel link handler tests
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-10-12 20:11:28 +02:00
Bartosz Rumiński
7abb4b3713
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/linkHandler/YoutubeChannelLinkHandlerFactory.java
Fix typos

Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-10-12 19:57:45 +02:00
Bartosz Rumiński
9d63211a66
Fix typos
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-10-12 19:56:53 +02:00
Bartosz Rumiński
8c38a5509e Prevent attribution_link urls to be accepted by channel links handler 2020-10-09 19:07:38 +02:00
Bartosz Rumiński
4e04991762 Support short custom youtube channel urls 2020-10-09 17:06:59 +02:00
Fynn Godau
81b5e7cf3d Fix extractor 2020-10-08 17:56:03 +02:00
TobiGr
acb04eb351 Update extractor version to 0.20.0 2020-10-04 13:37:19 +02:00
Stypox
2463884aa8
Merge pull request #400 from wb9688/skip-otf
Do not list YouTube's OTF streams
2020-09-29 20:16:34 +02:00
wb9688
ebbfe7f6d4 Skip YouTube's OTF streams 2020-09-29 10:48:02 +02:00
Tobias Groza
6633f26ec5
Merge pull request #386 from mhmdanas/remove-subtitles-stream-get-url
Remove SubtitlesStream#getURL()
2020-08-26 21:32:36 +02:00
Tobias Groza
070cd92857
Merge pull request #387 from wb9688/init-safe-standard-objects
Use initSafeStandardsObjects()
2020-08-26 21:25:46 +02:00
wb9688
7657c2ed1a Use initSafeStandardsObjects() 2020-08-15 17:08:07 +02:00
mhmdanas
1a63dcb355 Remove overriden field url 2020-08-12 22:37:36 +03:00
Tobias Groza
de8edbe7a5
Merge pull request #382 from TobiGr/fix/empty_comment
[YouTube] Fix crash on empty  comment
2020-08-12 14:18:16 +02:00
mhmdanas
f0f1c009b2 Remove SubtitlesStream#getURL() 2020-08-10 19:28:56 +03:00
Fynn Godau
cfe88a74c1 Throw ContentNotSupportedException when opening radio uploader channel 2020-08-09 10:07:49 +02:00
TobiGr
0fb73301e3 [YouTube] Fix crash on empty comment
Closes #380
2020-08-05 18:25:40 +02:00
Fynn Godau
932d094d6a Upgrade featured playlist urls to https 2020-08-02 16:55:18 +02:00
Fynn Godau
c12ef3a02d Merge TNP/dev into fynngodau/dev 2020-08-02 16:43:04 +02:00
TobiGr
5ac80624a4 Fix detection of YouTube's decryption function
Closes TeamNewPipe/NewPipe#3951
2020-07-28 01:01:10 +02:00
TobiGr
b4481dfec2 Update version to 0.19.7 2020-07-26 14:24:07 +02:00
Tobias Groza
32dff1541f
Merge pull request #370 from wb9688/fix-yt-age-restricted
Fix age restricted YouTube videos
2020-07-26 14:17:19 +02:00
TobiGr
ec778200d9 Add comments to the code 2020-07-26 14:15:13 +02:00
Tobias Groza
8627d01006
Merge pull request #373 from wb9688/fix-yt-continuations
Support YouTube's new continuations
2020-07-26 13:56:33 +02:00
wb9688
667dce034c Only use fallback methods for clientVersion when clientVersion hasn't been successfully extracted yet 2020-07-26 13:14:25 +02:00
wb9688
f345f667e2 Extract YouTube's key 2020-07-26 12:01:59 +02:00
wb9688
4c987a5302 Support YouTube's new continuations for search 2020-07-26 10:01:03 +02:00
wb9688
81459e289f Fix age restricted YouTube videos 2020-07-18 19:52:54 +02:00
Tobias Groza
df28a087de
Merge pull request #337 from wb9688/next-stream
Remove getNextStream() from StreamExtractor
2020-07-18 11:55:58 +02:00
wb9688
fc3a63fec5 Update Javadoc for getRelatedStreams() 2020-07-15 18:48:37 +02:00
wb9688
ff560e907a Remove getNextStream() from StreamExtractor 2020-07-15 18:48:37 +02:00
TheAssassin
186193d366 Update version to 0.19.6 2020-07-14 17:04:03 +02:00
Tobias Groza
a70cb0283f
Merge pull request #314 from wb9688/remove-getnextpageurl
Next page stuff
2020-07-07 20:49:38 +02:00
wb9688
0a5a905bc7 Add final at more places 2020-07-07 20:45:47 +02:00
Tobias Groza
e5d23a89de
Merge pull request #366 from wb9688/yt-timestamp-start
Support start= timestamps for YouTube
2020-07-06 21:28:19 +02:00
wb9688
17ba8a57fa Clean up the code 2020-07-06 20:19:31 +02:00
wb9688
9b6fe1dea6 Throw IllegalArgumentException when Page is invalid 2020-07-06 20:19:31 +02:00
wb9688
4cc312086a Introduce Page class 2020-07-06 20:19:31 +02:00
wb9688
e3bfdba135 Remove getNextPageUrl() function from ListExtractor 2020-07-06 20:11:40 +02:00
wb9688
156a26b64b Support start= timestamps for YouTube 2020-07-06 14:08:40 +02:00
Tobias Groza
54d9e5a2f8
Merge pull request #365 from B0pol/update_inv_instances
Update inv instances
2020-07-02 23:13:07 +02:00
bopol
261471e7f4 remove invidiou.sh from tests
dead instance
2020-07-02 21:33:45 +02:00
bopol
eb3901acee update invidious instances 2020-07-02 21:31:05 +02:00
Tobias Groza
c0ceb5cb27
Merge pull request #340 from wb9688/peertube-deleted-comments
Handle isDeleted for PeerTube comments
2020-06-28 10:21:37 +02:00
Tobias Groza
f4ed7ce922
Merge pull request #361 from yausername/gradleScope
expose extractor via api scope
2020-06-27 21:45:04 +02:00
Tobias Groza
339f1d9c43
Merge pull request #349 from wb9688/okhttp
Use OkHttp for tests like in NewPipe
2020-06-27 20:08:02 +02:00
Tobias Groza
970bc7f69d
Merge pull request #359 from mauriciocolli/fix-client-id-bug
[SoundCloud] Fix concurrency issue when getting the client id
2020-06-27 17:36:24 +02:00
Tobias Groza
dbf6ab9e91
Merge pull request #355 from Redirion/patch-1
Added a comment
2020-06-27 11:47:50 +02:00
Robin
576754982e
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2020-06-27 11:35:01 +02:00
yausername
89e0e3ee2d
expose extractor via api scope 2020-06-27 03:02:43 +05:30
Mauricio Colli
7505e828cc
[SoundCloud] Update client id 2020-06-24 01:50:09 -03:00
Mauricio Colli
d2f1c0f40d
[SoundCloud] Fix concurrency issue when getting the client id 2020-06-24 01:50:08 -03:00
Robin
b63ae93495
Added a comment
and fixed a "typo" in the method parseDurationString replacing Long with Integer.

These are not the droids you are looking for wb9688 😁
2020-06-15 11:27:44 +02:00
Tobias Groza
92f6754f0f
Merge pull request #338 from wb9688/parse-duration-string-test
Add tests for parseDurationString()
2020-06-15 11:01:21 +02:00
wb9688
e02d3c37b2
Merge pull request #354 from Stypox/peertube-age-restricted
[PeerTube] Change age restricted video in tests
2020-06-14 20:09:36 +02:00
Stypox
5808439c3f
[PeerTube] Change age restricted video in tests
The old one wasn't available anymore
2020-06-14 19:57:39 +02:00
wb9688
3239aa84f2 Handle isDeleted for PeerTube comments 2020-06-14 19:27:20 +02:00
wb9688
a65e46e4b1 Use OkHttp for tests like in NewPipe 2020-06-11 15:06:08 +02:00
Tobias Groza
5f523254cb
Merge pull request #344 from B0pol/peertube_embeds
For #342: support PeerTube embeds
2020-06-06 18:53:07 +02:00
Fynn Godau
96de834b67 Bandcamp: parse date from radio info item 2020-06-04 19:36:58 +02:00
Fynn Godau
39b55b5b42 Bandcamp radio stream extractor: test uploader url 2020-06-04 19:32:19 +02:00
Fynn Godau
9a555d97e5 Remove useless bandcamp tests 2020-06-04 19:31:01 +02:00
Fynn Godau
e13f341a45 Bandcamp stream extractor: test service id 2020-06-04 19:29:37 +02:00
Fynn Godau
4dd9540782 Bandcamp license switch order 2020-06-04 19:27:39 +02:00
Fynn Godau
52103ac61f Remove "useless" comments 2020-06-04 19:27:13 +02:00
Fynn Godau
8fa81537c0 Better bandcamp stream extractor description test 2020-06-04 19:17:51 +02:00
Fynn Godau
0950a95577 Fix for testing wrong object for null or empty 2020-06-04 19:16:08 +02:00
Fynn Godau
e6ecd91929 Bandcamp: Test full date 2020-06-04 19:12:10 +02:00
Fynn Godau
27e7aad159 Bandcamp: Improve code style 2020-06-04 19:04:25 +02:00
fynngodau
ea49202f64
Don't display internal license id
Co-authored-by: bopol <bopol@e.email>
2020-06-04 18:26:58 +02:00
bopol
063f4be766 For #342: support PeerTube embeds 2020-06-04 14:26:42 +02:00
Fynn Godau
34b6928124 Don't print stacktrace before throwing new exception 2020-06-04 14:25:38 +02:00
Fynn Godau
1be20ceeec Bandcamp: parse date 2020-06-04 14:09:21 +02:00
Fynn Godau
6822fe3dc8 Bandcamp: Get Stream upload date 2020-06-04 13:03:16 +02:00
Fynn Godau
e98bff8ae9 Improved Bandcamp radio stream extractor tests 2020-06-04 12:30:53 +02:00
Fynn Godau
8f6c00f8d8 Use default test in BandcampSearchExtractorTest 2020-06-04 12:12:20 +02:00
Fynn Godau
de776561b0 Bandcamp: fix loading uploader from streams in search 2020-06-04 12:02:18 +02:00
Fynn Godau
3940138fc5 Fix Bandcamp capitalization in one more spot 2020-06-03 22:41:27 +02:00
Fynn Godau
9fa9d920a9 Bandcamp: Implement new methods required due to interface changes 2020-06-03 21:35:11 +02:00
Fynn Godau
dd955c7f0c Merge branch 'origin/dev' into dev 2020-06-03 21:13:40 +02:00
wb9688
33b1121fc7 Add tests for parseDurationString() 2020-05-30 17:20:54 +02:00
TobiGr
3aa6b93e91 Update version to 0.19.5 2020-05-30 10:45:39 +02:00
Tobias Groza
bda83fe6a5
Merge pull request #336 from wb9688/fix-time-parsing
Fix YouTube video duration parsing
2020-05-30 10:36:46 +02:00
wb9688
e39147202c Fix YouTube video duration parsing 2020-05-30 09:31:08 +02:00
Fynn Godau
89b0639faa New Bandcamp tests for channel extractor 2020-05-25 20:22:23 +02:00
Fynn Godau
e08256ebc6 Remove "url cleanup" in Bandcamp link handlers 2020-05-25 19:17:27 +02:00
Fynn Godau
692b2d06f4 Bandcamp: read info item data in place and not in advance 2020-05-25 18:53:15 +02:00
Fynn Godau
fff21caa87 Fix Bandcamp no avatar test
I own the bandcamp artist account called "NewPipeExtractorTest", so it
shouldn't break again.
2020-05-25 18:53:10 +02:00
Fynn Godau
e7b046db9d Merge remote-tracking branch 'origin/dev' into dev 2020-04-26 23:12:43 +02:00
Fynn Godau
a3e8e1c70e Bandcamp: move own methods to helper class 2020-04-26 23:05:56 +02:00
Fynn Godau
9bc7a470ab Replace fori with foreach loop 2020-04-26 23:02:14 +02:00
Fynn Godau
f71bd29d58 Bandcamp playlist extractor: add more tests 2020-04-20 23:52:12 +02:00
Fynn Godau
5009d9f53b Bandcamp tests: remove certain tests 2020-04-20 23:25:00 +02:00
Fynn Godau
b5e251c82f Bandcamp channel extractor: package-private access for getImageUrl(…) 2020-04-20 23:12:43 +02:00
Fynn Godau
8c70dab8c5 Bandcamp search query handler factory: inline variable 2020-04-20 23:11:16 +02:00
Fynn Godau
10ae3db118 Bandcamp javadoc: Replace br tags with p tags 2020-04-20 23:03:12 +02:00
Fynn Godau
9201e0d3ea Don't declare exceptions that can't actually be thrown 2020-04-20 22:46:54 +02:00
Fynn Godau
965bce00cf Bandcamp stream extractor: return NO_AGE_LIMIT 2020-04-20 22:44:45 +02:00
Fynn Godau
82099592c7 Bandcamp: Move code from SearchExtractor to InfoItemExtractors 2020-04-20 22:44:35 +02:00
Fynn Godau
00c0333059 Bandcamp: Fetch channelInfo in onFetchPage 2020-04-20 22:41:44 +02:00
Fynn Godau
433d72c26a Depend on nanojson fork in timeago-parser 2020-04-20 22:04:10 +02:00
Fynn Godau
67de0285e1 Bandcamp: Generate query JSON in a proper way 2020-04-20 22:03:29 +02:00
Fynn Godau
c133190c53 Bandcamp capitalization 2020-04-20 21:55:35 +02:00
Fynn Godau
b100b9873f Bandcmap: Use nanojson fork instead of org.json 2020-04-20 21:53:04 +02:00
Fynn Godau
9c239371f4 Change nanojson dependency to @wb9688's fork 2020-04-19 14:29:39 +02:00
Fynn Godau
5c0a03328a Merge 'origin/dev' into dev 2020-04-19 14:26:51 +02:00
Fynn Godau
a1523eb293 Update Bandcamp service to latest interface changes 2020-03-17 20:40:25 +01:00
fynngodau
b78f788017
Merge branch 'dev' into dev 2020-03-17 18:03:14 +01:00
Fynn Godau
36a316eea5
Merge branch 'dev' into dev 2020-02-08 09:43:13 +01:00
Fynn Godau
623e6a8c63
Merge branch 'dev' into dev 2020-01-17 22:37:42 +01:00
Fynn Godau
7b5702bdc2 Add bandcamp to list of supported services in README 2020-01-11 16:32:45 +01:00
Fynn Godau
ba967f1a15 Workaround enourmous load times for long bandcamp playlists
Additionally, get cover art from json instead of html
2020-01-03 13:17:04 +01:00
Fynn Godau
46e1f3922c Refuse to load bandcamp playlists without content 2020-01-03 12:32:22 +01:00
Fynn Godau
f0d36dfa7e Bandcamp radio: proper name, date shortened 2020-01-03 12:28:49 +01:00
Fynn Godau
808c8aa087 Add bandcamp weekly kiosk
Unfortunately a little messy
2020-01-03 12:28:49 +01:00
Fynn Godau
13ef11e9c8 Bandcamp featured kiosk 2020-01-03 12:28:49 +01:00
Fynn Godau
db02850d4f Bandcamp playlist extractor: support playlists with just one stream 2020-01-03 12:28:49 +01:00
Fynn Godau
13e4908b83 Bandcamp search extractor: read track count from result page 2020-01-03 12:28:49 +01:00
Fynn Godau
9b16baffb7 Bandcamp search: multiple pages 2020-01-03 12:28:49 +01:00
Fynn Godau
ce2a88e56f Add bandcamp search suggestion extractor 2020-01-03 12:28:49 +01:00
Fynn Godau
e12ddaef7f Bandcamp playlist and search extractors: better and more tests 2020-01-03 12:28:49 +01:00
Fynn Godau
655df356a2 Various improvements concerning bandcamp playlists 2020-01-03 12:28:49 +01:00
Fynn Godau
ba700bfb3e Add bandcamp playlists (albums) 2020-01-03 12:28:49 +01:00
Fynn Godau
d05b14ae48 Add channels (artists) to search results 2020-01-03 12:28:49 +01:00
Fynn Godau
794ca5eeae Bandcamp channel extractor: handel nonexistent images better 2020-01-03 12:28:49 +01:00
Fynn Godau
91c0ec7cea Bandcamp stream extractor: catch NullPointerException 2020-01-03 12:28:49 +01:00
Fynn Godau
d5cdc20be1 Bandcmap channel extractor: fix getting banner url 2020-01-03 12:28:49 +01:00
Fynn Godau
7730eb2ea1 Bandcamp channel extractor: enforce https when getting banner url 2020-01-03 12:28:49 +01:00
Fynn Godau
a42c77425d Bandcamp channel extractor (ignoring everything but tracks) 2020-01-03 12:28:49 +01:00
Fynn Godau
5281456899 Bandcamp channel link handler factory 2020-01-03 12:28:49 +01:00
Fynn Godau
c3d127ccd9 Bandcamp audio stream test and fix 2020-01-03 12:28:49 +01:00
Fynn Godau
80d67e22b7 Return mp3-128 audio stream [experimental] 2020-01-03 12:28:49 +01:00
Fynn Godau
43dc3c3d4c Return empty kiosk list instead of null 2020-01-03 12:28:49 +01:00
Fynn Godau
a579337c9a Bandcamp service with support for streams and searches 2020-01-03 12:28:49 +01:00
1249 changed files with 213050 additions and 11645 deletions

2
.github/FUNDING.yml vendored Normal file
View File

@ -0,0 +1,2 @@
liberapay: TeamNewPipe
custom: 'https://newpipe.net/donate/'

93
.github/ISSUE_TEMPLATE/bug_report.yml vendored Normal file
View File

@ -0,0 +1,93 @@
name: Bug report
description: Create a bug report to help us improve
labels: [bug]
body:
- type: markdown
attributes:
value: |
Thank you for helping to make the NewPipe Extractor better by reporting a bug. :hugs:
Please fill in as much information as possible about your bug so that we don't have to play "information ping-pong" and can help you immediately.
- type: checkboxes
id: checklist
attributes:
label: "Checklist"
options:
- label: "I am able to reproduce the bug with the latest version given here: [CLICK THIS LINK](https://github.com/TeamNewPipe/NewPipeExtractor/releases/latest)."
required: true
- label: "I am aware that this issue is being opened for the NewPipe Extractor, NOT the [app](https://github.com/TeamNewPipe/NewPipe), and my bug report will be dismissed otherwise."
required: true
- label: "I made sure that there are *no existing issues* - [open](https://github.com/TeamNewPipe/NewPipeExtractor/issues) or [closed](https://github.com/TeamNewPipe/NewPipeExtractor/issues?q=is%3Aissue+is%3Aclosed) - which I could contribute my information to."
required: true
- label: "I have taken the time to fill in all the required details. I understand that the bug report will be dismissed otherwise."
required: true
- label: "This issue contains only one bug."
required: true
- label: "I have read and understood the [contribution guidelines](https://github.com/TeamNewPipe/NewPipe/blob/dev/.github/CONTRIBUTING.md)."
required: true
- type: input
id: extractor-version
attributes:
label: Affected version
description: "In which NewPipe Extractor version did you encounter the bug?"
placeholder: "x.xx.x"
validations:
required: true
- type: textarea
id: steps-to-reproduce
attributes:
label: Steps to reproduce the bug
description: |
What did you do for the bug to show up?
If you can't cause the bug to show up again reliably (and hence don't have a proper set of steps to give us), please still try to give as many details as possible on how you think you encountered the bug.
placeholder: |
1. Init NewPipe with 'NewPipe.init(...)'
2. Create a StreamExtractor for xyz: 'StreamExtractor e = YouTube.getStreamExtractor(xyz.com)'
3. Get the description 'e.getDescription()'
validations:
required: true
- type: textarea
id: expected-behavior
attributes:
label: Expected behavior
description: |
Tell us what you expect to happen.
- type: textarea
id: actual-behavior
attributes:
label: Actual behavior
description: |
Tell us what happens with the steps given above.
- type: textarea
id: screen-media
attributes:
label: Screenshots/Screen recordings
description: |
A picture or video is worth a thousand words.
If applicable, add screenshots or a screen recording to help explain your problem.
GitHub supports uploading them directly in the text box.
If your file is too big for Github to accept, try to compress it (ZIP-file) or feel free to paste a link to an image/video hoster here instead.
- type: textarea
id: logs
attributes:
label: Logs
description: |
If your bug includes a log you think we need to see, paste it here.
- type: textarea
id: additional-information
attributes:
label: Additional information
description: |
Any other information you'd like to include, for instance that
* your cat disabled your network connection
* ...

8
.github/ISSUE_TEMPLATE/config.yml vendored Normal file
View File

@ -0,0 +1,8 @@
blank_issues_enabled: false
contact_links:
- name: 💬 Matrix
url: https://matrix.to/#/#newpipe:matrix.newpipe-ev.de
about: Chat with us via Matrix for quick Q/A
- name: 💬 IRC
url: https://web.libera.chat/#newpipe
about: Chat with us via IRC for quick Q/A

50
.github/ISSUE_TEMPLATE/enhancement.yml vendored Normal file
View File

@ -0,0 +1,50 @@
name: Feature request
description: Suggest an idea for this project
labels: [enhancement]
body:
- type: markdown
attributes:
value: |
Thank you for helping to make the NewPipe Extractor better by suggesting a feature. :hugs:
Your ideas are highly welcome! The Extractor is made for you, the downstream developers, after all.
- type: checkboxes
id: checklist
attributes:
label: "Checklist"
options:
- label: "I am aware that this issue is being opened for the NewPipe Extractor, NOT the [app](https://github.com/TeamNewPipe/NewPipe), and my feature request will be dismissed otherwise."
required: true
- label: "I made sure that there are *no existing issues* - [open](https://github.com/TeamNewPipe/NewPipeExtractor/issues) or [closed](https://github.com/TeamNewPipe/NewPipeExtractor/issues?q=is%3Aissue+is%3Aclosed) - which I could contribute my information to."
required: true
- label: "I have taken the time to fill in all the required details. I understand that the feature request will be dismissed otherwise."
required: true
- label: "This issue contains only one feature request."
required: true
- label: "I have read and understood the [contribution guidelines](https://github.com/TeamNewPipe/NewPipe/blob/dev/.github/CONTRIBUTING.md)."
required: true
- type: textarea
id: feature-description
attributes:
label: Feature description
description: |
Explain how you want the Extractor's behavior to change to suit your needs.
validations:
required: true
- type: textarea
id: why-is-the-feature-requested
attributes:
label: Why do you want this feature?
description: |
Describe any problem or limitation you come across while using the Extractor which would be solved by this feature.
validations:
required: true
- type: textarea
id: additional-information
attributes:
label: Additional information
description: Any other information you'd like to include, for instance sketches, mockups, pictures of cats, etc.

12
.github/dependabot.yml vendored Normal file
View File

@ -0,0 +1,12 @@
version: 2
updates:
# Maintain dependencies for Gradle
- package-ecosystem: "gradle"
directory: "/"
schedule:
interval: "daily"
# Maintain dependencies for GitHub Actions
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "daily"

69
.github/workflows/ci.yml vendored Normal file
View File

@ -0,0 +1,69 @@
name: CI
on:
workflow_dispatch:
inputs:
downloaderType:
description: 'Downloader Type'
type: choice
options:
- MOCK
- REAL
schedule:
# once per day
- cron: 0 0 * * *
push:
branches:
- dev
- master
pull_request:
permissions:
contents: read
jobs:
build-and-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: set up JDK 11
uses: actions/setup-java@v4
with:
java-version: '11'
distribution: 'temurin'
- name: Cache Gradle wrapper dists
uses: actions/cache@v4
with:
path: ~/.gradle/wrapper/dists
key: ${{ runner.os }}-gradle-wrapper-dists-${{ hashFiles('gradle/wrapper/**') }}
restore-keys: ${{ runner.os }}-gradle-wrapper-dists
- name: Cache Gradle dependencies
uses: actions/cache@v4
with:
path: ~/.gradle/caches
key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle') }}
restore-keys: ${{ runner.os }}-gradle
# See gradle file for difference between downloaders
- name: Build and run Tests
run: |
downloader_type="${{ github.event.inputs.downloaderType || 'MOCK' }}"
if [[ "$GITHUB_EVENT_NAME" == 'schedule' ]]; then
downloader_type="REAL"
fi
echo "Running with $downloader_type downloader"
./gradlew check \
-Dorg.gradle.welcome=never \
--stacktrace \
-Ddownloader=$downloader_type
- name: Upload test reports when failure occurs
uses: actions/upload-artifact@v4
if: failure()
with:
name: NewPipeExtractor-test-reports
path: extractor/build/reports/tests/test/**

38
.github/workflows/docs.yml vendored Normal file
View File

@ -0,0 +1,38 @@
name: Build and deploy JavaDocs
on:
push:
branches:
- master
permissions:
# The generated docs are written to the `gh-pages` branch.
contents: write
jobs:
build-and-deploy-docs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: set up JDK 11
uses: actions/setup-java@v4
with:
java-version: '11'
distribution: 'temurin'
- name: Cache Gradle dependencies
uses: actions/cache@v4
with:
path: ~/.gradle/caches
key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle') }}
restore-keys: ${{ runner.os }}-gradle
- name: Build JavaDocs
run: ./gradlew aggregatedJavadocs
- name: Deploy JavaDocs
uses: peaceiris/actions-gh-pages@v4
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: ./build/docs

View File

@ -1,14 +0,0 @@
language: java
jdk:
- openjdk8
script:
- ./gradlew check aggregatedJavadocs
deploy:
provider: pages
skip_cleanup: true
github_token: $GITHUB_TOKEN # Set in travis-ci.org dashboard
local_dir: build/docs
on:
branch: master

View File

@ -1,7 +1,7 @@
GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
@ -645,7 +645,7 @@ the "copyright" line and a pointer to where the full notice is found.
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
@ -664,11 +664,11 @@ might be different; for a GUI interface, you would use an "about box".
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU GPL, see
<http://www.gnu.org/licenses/>.
<https://www.gnu.org/licenses/>.
The GNU General Public License does not permit incorporating your program
into proprietary programs. If your program is a subroutine library, you
may consider it more useful to permit linking proprietary applications with
the library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License. But first, please read
<http://www.gnu.org/philosophy/why-not-lgpl.html>.
<https://www.gnu.org/licenses/why-not-lgpl.html>.

View File

@ -1,6 +1,6 @@
# NewPipe Extractor
[![Build Status](https://travis-ci.org/TeamNewPipe/NewPipeExtractor.svg?branch=master)](https://travis-ci.org/TeamNewPipe/NewPipeExtractor) [![JIT Pack Badge](https://jitpack.io/v/TeamNewPipe/NewPipeExtractor.svg)](https://jitpack.io/#TeamNewPipe/NewPipeExtractor) [JDoc](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/) • [Documentation](https://teamnewpipe.github.io/documentation/)
[![CI](https://github.com/TeamNewPipe/NewPipeExtractor/actions/workflows/ci.yml/badge.svg?branch=dev&event=schedule)](https://github.com/TeamNewPipe/NewPipeExtractor/actions/workflows/ci.yml) [![JIT Pack Badge](https://jitpack.io/v/teamnewpipe/NewPipeExtractor.svg)](https://jitpack.io/#teamnewpipe/NewPipeExtractor) [JDoc](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/) • [Documentation](https://teamnewpipe.github.io/documentation/)
NewPipe Extractor is a library for extracting things from streaming sites. It is a core component of [NewPipe](https://github.com/TeamNewPipe/NewPipe), but could be used independently.
@ -11,7 +11,16 @@ NewPipe Extractor is available at JitPack's Maven repo.
If you're using Gradle, you could add NewPipe Extractor as a dependency with the following steps:
1. Add `maven { url 'https://jitpack.io' }` to the `repositories` in your `build.gradle`.
2. Add `implementation 'com.github.TeamNewPipe:NewPipeExtractor:v0.19.0'`the `dependencies` in your `build.gradle`. Replace `v0.19.0` with the latest release.
2. Add `implementation 'com.github.teamnewpipe:NewPipeExtractor:INSERT_VERSION_HERE'` to the `dependencies` in your `build.gradle`. Replace `INSERT_VERSION_HERE` with the [latest release](https://github.com/TeamNewPipe/NewPipeExtractor/releases/latest).
3. If you are using tools to minimize your project, make sure to keep the files below, by e.g. adding the following lines to your proguard file:
```
## Rules for NewPipeExtractor
-keep class org.mozilla.javascript.** { *; }
-keep class org.mozilla.classfile.ClassFileWriter
-dontwarn org.mozilla.javascript.tools.**
```
**Note:** To use NewPipe Extractor in Android projects with a `minSdk` below 33, [core library desugaring](https://developer.android.com/studio/write/java8-support#library-desugaring) with the `desugar_jdk_libs_nio` artifact is required.
### Testing changes
@ -20,7 +29,7 @@ To test changes quickly you can build the library locally. A good approach would
```groovy
includeBuild('../NewPipeExtractor') {
dependencySubstitution {
substitute module('com.github.TeamNewPipe:NewPipeExtractor') with project(':extractor')
substitute module('com.github.teamnewpipe:NewPipeExtractor') with project(':extractor')
}
}
```
@ -30,7 +39,7 @@ Another approach would be to use the local Maven repository, here's a gist of ho
1. Add `mavenLocal()` in your project `repositories` list (usually as the first entry to give priority above the others).
2. It's _recommended_ that you change the `version` of this library (e.g. `LOCAL_SNAPSHOT`).
3. Run gradle's `ìnstall` task to deploy this library to your local repository (using the wrapper, present in the root of this project: `./gradlew install`)
4. Change the dependency version used in your project to match the one you chose in step 2 (`implementation 'com.github.TeamNewPipe:NewPipeExtractor:LOCAL_SNAPSHOT'`)
4. Change the dependency version used in your project to match the one you chose in step 2 (`implementation 'com.github.teamnewpipe:NewPipeExtractor:LOCAL_SNAPSHOT'`)
> Tip for Android Studio users: After you make changes and run the `install` task, use the menu option `File → "Sync with File System"` to refresh the library in your project.
@ -40,14 +49,15 @@ The following sites are currently supported:
- YouTube
- SoundCloud
- MediaCCC
- media.ccc.de
- PeerTube (no P2P)
- Bandcamp
## License
[![GNU GPLv3 Image](https://www.gnu.org/graphics/gplv3-127x51.png)](http://www.gnu.org/licenses/gpl-3.0.en.html)
[![GNU GPLv3 Image](https://www.gnu.org/graphics/gplv3-127x51.png)](https://www.gnu.org/licenses/gpl-3.0.en.html)
NewPipe is Free Software: You can use, study share and improve it at your
NewPipe Extractor is Free Software: You can use, study share and improve it at your
will. Specifically you can redistribute and/or modify it under the terms of the
[GNU General Public License](https://www.gnu.org/licenses/gpl.html) as
published by the Free Software Foundation, either version 3 of the License, or

View File

@ -1,31 +1,59 @@
allprojects {
apply plugin: 'java-library'
apply plugin: 'maven'
apply plugin: 'maven-publish'
sourceCompatibility = 1.7
targetCompatibility = 1.7
compileJava.options.encoding = 'UTF-8'
compileTestJava.options.encoding = 'UTF-8'
version 'v0.19.4'
sourceCompatibility = JavaVersion.VERSION_11
targetCompatibility = JavaVersion.VERSION_11
version 'v0.24.8'
group 'com.github.TeamNewPipe'
repositories {
jcenter()
mavenCentral()
maven { url "https://jitpack.io" }
}
afterEvaluate {
publishing {
publications {
mavenJava(MavenPublication) {
from components.java
}
}
}
}
ext {
nanojsonVersion = "e9d656ddb49a412a5a0a5d5ef20ca7ef09549996"
jsr305Version = "3.0.2"
junitVersion = "5.13.4"
checkstyleVersion = "10.4"
}
}
dependencies {
implementation project(':extractor')
api project(':extractor')
implementation project(':timeago-parser')
}
subprojects {
task sourcesJar(type: Jar, dependsOn: classes) {
classifier = 'sources'
tasks.register('sourcesJar', Jar) {
dependsOn classes
archiveClassifier.set('sources')
from sourceSets.main.allSource
}
tasks.withType(Test) {
// Protobuf files would uselessly end up in the JAR otherwise, see
// https://github.com/google/protobuf-gradle-plugin/issues/390
tasks.withType(Jar).configureEach {
exclude '**/*.proto'
includeEmptyDirs false
}
tasks.withType(Test).configureEach {
testLogging {
events "skipped", "failed"
showStandardStreams = true
@ -39,12 +67,18 @@ subprojects {
}
// https://discuss.gradle.org/t/best-approach-gradle-multi-module-project-generate-just-one-global-javadoc/18657/21
task aggregatedJavadocs(type: Javadoc, group: 'Documentation') {
destinationDir = file("$buildDir/docs/javadoc")
tasks.register('aggregatedJavadocs', Javadoc) {
destinationDir = layout.buildDirectory.file("docs/javadoc").get().asFile
title = "$project.name $version"
// options.memberLevel = JavadocMemberLevel.PRIVATE
options.links 'https://docs.oracle.com/javase/7/docs/api/'
options.links 'https://docs.oracle.com/javase/11/docs/api/'
options.encoding 'UTF-8'
// Fixes unknown tag @implNote; the other two were added precautionary
options.tags = [
"apiNote:a:API Note:",
"implSpec:a:Implementation Requirements:",
"implNote:a:Implementation Note:"
]
subprojects.each { project ->
project.tasks.withType(Javadoc).each { javadocTask ->

195
checkstyle/checkstyle.xml Normal file
View File

@ -0,0 +1,195 @@
<?xml version="1.0"?>
<!DOCTYPE module PUBLIC
"-//Checkstyle//DTD Checkstyle Configuration 1.3//EN"
"https://checkstyle.org/dtds/configuration_1_3.dtd">
<module name="Checker">
<!--
If you set the basedir property below, then all reported file
names will be relative to the specified directory. See
https://checkstyle.org/5.x/config.html#Checker
<property name="basedir" value="${basedir}"/>
-->
<property name="severity" value="error"/>
<property name="fileExtensions" value="java, properties, xml"/>
<!-- Excludes all 'module-info.java' files -->
<!-- See https://checkstyle.org/config_filefilters.html -->
<module name="BeforeExecutionExclusionFileFilter">
<property name="fileNamePattern" value="module\-info\.java$"/>
</module>
<!-- Checks that a package-info.java file exists for each package. -->
<!-- See https://checkstyle.org/config_javadoc.html#JavadocPackage -->
<!--<module name="JavadocPackage"/>-->
<!-- Checks whether files end with a new line. -->
<!-- See https://checkstyle.org/config_misc.html#NewlineAtEndOfFile -->
<module name="NewlineAtEndOfFile"/>
<!-- Checks that property files contain the same keys. -->
<!-- See https://checkstyle.org/config_misc.html#Translation -->
<module name="Translation"/>
<!-- Checks for Size Violations. -->
<!-- See https://checkstyle.org/config_sizes.html -->
<module name="FileLength"/>
<module name="LineLength">
<property name="max" value="100"/>
<property name="fileExtensions" value="java"/>
<!-- Also allow links in javadocs to be longer (the default would just cover imports) -->
<property name="ignorePattern" value="^((package|import) .*)|( *\* (@see )?&lt;a href ?\= ?&quot;.*&quot;&gt;)$"/>
</module>
<!-- Checks for whitespace -->
<!-- See https://checkstyle.org/config_whitespace.html -->
<module name="FileTabCharacter"/>
<!-- Miscellaneous other checks. -->
<!-- See https://checkstyle.org/config_misc.html -->
<module name="RegexpSingleline">
<property name="format" value="\s+$"/>
<property name="minimum" value="0"/>
<property name="maximum" value="0"/>
<property name="message" value="Line has trailing spaces."/>
</module>
<!-- Checks for Headers -->
<!-- See https://checkstyle.org/config_header.html -->
<!-- <module name="Header"> -->
<!-- <property name="headerFile" value="${checkstyle.header.file}"/> -->
<!-- <property name="fileExtensions" value="java"/> -->
<!-- </module> -->
<module name="SuppressWarningsFilter" />
<module name="SuppressWithPlainTextCommentFilter"/>
<module name="TreeWalker">
<!-- Checks for Javadoc comments. -->
<!-- See https://checkstyle.org/config_javadoc.html -->
<module name="InvalidJavadocPosition"/>
<module name="JavadocMethod">
<property name="allowMissingParamTags" value="true"/>
<property name="allowMissingReturnTag" value="true"/>
</module>
<module name="JavadocType"/>
<!--<module name="JavadocVariable"/>-->
<module name="JavadocStyle">
<property name="checkFirstSentence" value="false"/>
</module>
<!--<module name="MissingJavadocMethod"/>-->
<!-- Checks for Naming Conventions. -->
<!-- See https://checkstyle.org/config_naming.html -->
<module name="ConstantName"/>
<module name="LocalFinalVariableName"/>
<module name="LocalVariableName"/>
<module name="MemberName">
<property name="format" value="^(TAG|DEBUG|[a-z][a-zA-Z0-9]*)$"/>
</module>
<module name="MethodName"/>
<module name="PackageName"/>
<module name="ParameterName"/>
<module name="StaticVariableName"/>
<module name="TypeName"/>
<!-- Checks for imports -->
<!-- See https://checkstyle.org/config_import.html -->
<module name="AvoidStarImport"/>
<module name="IllegalImport"> <!-- defaults to sun.* packages -->
<property name="illegalClasses" value="
org.jetbrains.annotations.Nullable,
org.jetbrains.annotations.NotNull,
androidx.annotation.Nullable,
androidx.annotation.NonNull,
io.reactivex.rxjava3.annotations.NonNull,
io.reactivex.rxjava3.annotations.Nullable" />
</module>
<module name="RedundantImport"/>
<module name="UnusedImports"/>
<!-- Checks for Size Violations. -->
<!-- See https://checkstyle.org/config_sizes.html -->
<module name="MethodLength">
<property name="severity" value="warning"/>
</module>
<module name="ParameterNumber">
<property name="severity" value="warning"/>
</module>
<!-- Checks for whitespace -->
<!-- See https://checkstyle.org/config_whitespace.html -->
<module name="EmptyForIteratorPad"/>
<module name="GenericWhitespace"/>
<module name="MethodParamPad"/>
<module name="NoWhitespaceAfter"/>
<module name="NoWhitespaceBefore"/>
<module name="OperatorWrap"/>
<module name="ParenPad"/>
<module name="TypecastParenPad"/>
<module name="WhitespaceAfter"/>
<module name="WhitespaceAround"/>
<!-- Modifier Checks -->
<!-- See https://checkstyle.org/config_modifiers.html -->
<module name="ModifierOrder"/>
<module name="RedundantModifier"/>
<!-- Checks for blocks. You know, those {}'s -->
<!-- See https://checkstyle.org/config_blocks.html -->
<module name="AvoidNestedBlocks"/>
<module name="EmptyBlock"/>
<module name="LeftCurly"/>
<module name="NeedBraces"/>
<module name="RightCurly"/>
<!-- Checks for common coding problems -->
<!-- See https://checkstyle.org/config_coding.html -->
<module name="EmptyStatement"/>
<module name="EqualsHashCode">
<property name="severity" value="warning"/>
</module>
<module name="HiddenField">
<property name="ignoreConstructorParameter" value="true"/>
<property name="ignoreSetter" value="true"/>
</module>
<module name="IllegalInstantiation"/>
<module name="InnerAssignment"/>
<!--<module name="MagicNumber"/>-->
<!--<module name="MissingSwitchDefault">
<property name="severity" value="warning"/>
</module>-->
<module name="MultipleVariableDeclarations"/>
<module name="SimplifyBooleanExpression"/>
<module name="SimplifyBooleanReturn"/>
<module name="FinalLocalVariable">
<property name="tokens" value="VARIABLE_DEF,PARAMETER_DEF"/>
<property name="validateEnhancedForLoopVariable" value="true"/>
</module>
<!-- Checks for class design -->
<!-- See https://checkstyle.org/config_design.html -->
<!--<module name="DesignForExtension"/>-->
<module name="FinalClass"/>
<module name="HideUtilityClassConstructor"/>
<module name="InterfaceIsType"/>
<!--<module name="VisibilityModifier">
<property name="ignoreAnnotationCanonicalNames" value="State,ColumnInfo"/>
<property name="severity" value="warning"/>
</module>-->
<!-- Miscellaneous other checks. -->
<!-- See https://checkstyle.org/config_misc.html -->
<module name="ArrayTypeStyle"/>
<module name="FinalParameters"/>
<!--<module name="TodoComment">
<property name="format" value="(TODO:|FIXME:)"/>
<property name="severity" value="warning"/>
</module>-->
<module name="UpperEll"/>
<module name="SuppressWarningsHolder" />
</module>
</module>

View File

@ -1,15 +0,0 @@
Copyright: 2017 Christian Schabesberger <chris.schabesberger@mailbox.com>
License: GPL-3.0+
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.

View File

@ -1,11 +1,71 @@
plugins {
id "checkstyle"
id "com.google.protobuf" version "0.9.5"
}
test {
// Pass on downloader type to tests for different CI jobs. See DownloaderFactory.java and ci.yml
if (System.properties.containsKey('downloader')) {
systemProperty('downloader', System.getProperty('downloader'))
}
useJUnitPlatform()
dependsOn checkstyleMain // run checkstyle when testing
}
checkstyle {
getConfigDirectory().set(rootProject.file("checkstyle"))
ignoreFailures false
showViolations true
toolVersion checkstyleVersion
}
// Exclude Protobuf generated files from Checkstyle
checkstyleMain.exclude("org/schabi/newpipe/extractor/services/youtube/protos")
checkstyleTest {
enabled false // do not checkstyle test files
}
ext {
rhinoVersion = '1.8.0'
protobufVersion = '4.31.1'
}
dependencies {
implementation project(':timeago-parser')
implementation 'com.github.TeamNewPipe:nanojson:1d9e1aea9049fc9f85e68b43ba39fe7be1c1f751'
implementation 'org.jsoup:jsoup:1.13.1'
implementation 'org.mozilla:rhino:1.7.12'
implementation 'com.github.spotbugs:spotbugs-annotations:4.0.2'
implementation 'org.nibor.autolink:autolink:0.10.0'
implementation "com.github.TeamNewPipe:nanojson:$nanojsonVersion"
implementation 'org.jsoup:jsoup:1.21.1'
implementation "com.google.code.findbugs:jsr305:$jsr305Version"
implementation "com.google.protobuf:protobuf-javalite:$protobufVersion"
testImplementation 'junit:junit:4.13'
implementation "org.mozilla:rhino:$rhinoVersion"
implementation "org.mozilla:rhino-engine:$rhinoVersion"
checkstyle "com.puppycrawl.tools:checkstyle:$checkstyleVersion"
testImplementation platform("org.junit:junit-bom:$junitVersion")
testImplementation 'org.junit.jupiter:junit-jupiter-api'
testRuntimeOnly 'org.junit.platform:junit-platform-launcher'
testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine'
testImplementation 'org.junit.jupiter:junit-jupiter-params'
testImplementation "com.squareup.okhttp3:okhttp:4.12.0"
testImplementation 'com.google.code.gson:gson:2.13.1'
}
protobuf {
protoc {
artifact = "com.google.protobuf:protoc:$protobufVersion"
}
generateProtoTasks {
all().configureEach { task ->
task.builtins {
java {
option "lite"
}
}
}
}
}

View File

@ -10,12 +10,15 @@ import org.schabi.newpipe.extractor.localization.TimeAgoParser;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.io.IOException;
import java.util.Objects;
public abstract class Extractor {
/**
* {@link StreamingService} currently related to this extractor.<br>
* Useful for getting other things from a service (like the url handlers for cleaning/accepting/get id from urls).
* Useful for getting other things from a service (like the url handlers for
* cleaning/accepting/get id from urls).
*/
private final StreamingService service;
private final LinkHandler linkHandler;
@ -26,19 +29,18 @@ public abstract class Extractor {
private ContentCountry forcedContentCountry = null;
private boolean pageFetched = false;
// called like this to prevent checkstyle errors about "hiding a field"
private final Downloader downloader;
public Extractor(final StreamingService service, final LinkHandler linkHandler) {
if (service == null) throw new NullPointerException("service is null");
if (linkHandler == null) throw new NullPointerException("LinkHandler is null");
this.service = service;
this.linkHandler = linkHandler;
this.downloader = NewPipe.getDownloader();
if (downloader == null) throw new NullPointerException("downloader is null");
protected Extractor(final StreamingService service, final LinkHandler linkHandler) {
this.service = Objects.requireNonNull(service, "service is null");
this.linkHandler = Objects.requireNonNull(linkHandler, "LinkHandler is null");
this.downloader = Objects.requireNonNull(NewPipe.getDownloader(), "downloader is null");
}
/**
* @return The {@link LinkHandler} of the current extractor object (e.g. a ChannelExtractor should return a channel url handler).
* @return The {@link LinkHandler} of the current extractor object (e.g. a ChannelExtractor
* should return a channel url handler).
*/
@Nonnull
public LinkHandler getLinkHandler() {
@ -52,13 +54,17 @@ public abstract class Extractor {
* @throws ExtractionException if the pages content is not understood
*/
public void fetchPage() throws IOException, ExtractionException {
if (pageFetched) return;
if (pageFetched) {
return;
}
onFetchPage(downloader);
pageFetched = true;
}
protected void assertPageFetched() {
if (!pageFetched) throw new IllegalStateException("Page is not fetched. Make sure you call fetchPage()");
if (!pageFetched) {
throw new IllegalStateException("Page is not fetched. Make sure you call fetchPage()");
}
}
protected boolean isPageFetched() {
@ -68,11 +74,13 @@ public abstract class Extractor {
/**
* Fetch the current page.
*
* @param downloader the download to use
* @param downloader the downloader to use
* @throws IOException if the page can not be loaded
* @throws ExtractionException if the pages content is not understood
*/
public abstract void onFetchPage(@Nonnull Downloader downloader) throws IOException, ExtractionException;
@SuppressWarnings("HiddenField")
public abstract void onFetchPage(@Nonnull Downloader downloader)
throws IOException, ExtractionException;
@Nonnull
public String getId() throws ParsingException {
@ -120,11 +128,11 @@ public abstract class Extractor {
// Localization
//////////////////////////////////////////////////////////////////////////*/
public void forceLocalization(Localization localization) {
public void forceLocalization(final Localization localization) {
this.forcedLocalization = localization;
}
public void forceContentCountry(ContentCountry contentCountry) {
public void forceContentCountry(final ContentCountry contentCountry) {
this.forcedContentCountry = contentCountry;
}
@ -135,7 +143,8 @@ public abstract class Extractor {
@Nonnull
public ContentCountry getExtractorContentCountry() {
return forcedContentCountry == null ? getService().getContentCountry() : forcedContentCountry;
return forcedContentCountry == null ? getService().getContentCountry()
: forcedContentCountry;
}
@Nonnull

View File

@ -0,0 +1,211 @@
package org.schabi.newpipe.extractor;
import javax.annotation.Nonnull;
import java.io.Serializable;
import java.util.Objects;
/**
* Class representing images in the extractor.
*
* <p>
* An image has four properties: its URL, its height, its width and its estimated quality level.
* </p>
*
* <p>
* Depending of the services, the height, the width or both properties may be not known.
* Implementations <b>must use</b> the relevant unknown constants in this case
* ({@link #HEIGHT_UNKNOWN} and {@link #WIDTH_UNKNOWN}), to ensure properly the lack of knowledge
* of one or both of these properties to extractor clients.
* </p>
*
* <p>
* They should also respect the ranges defined in the estimated image resolution levels as much as
* possible, to ensure consistency to extractor clients.
* </p>
*/
public final class Image implements Serializable {
/**
* Constant representing that the height of an {@link Image} is unknown.
*/
public static final int HEIGHT_UNKNOWN = -1;
/**
* Constant representing that the width of an {@link Image} is unknown.
*/
public static final int WIDTH_UNKNOWN = -1;
@Nonnull
private final String url;
private final int height;
private final int width;
@Nonnull
private final ResolutionLevel estimatedResolutionLevel;
/**
* Construct an {@link Image} instance.
*
* @param url the URL to the image, which should be not null or empty
* @param height the image's height
* @param width the image's width
* @param estimatedResolutionLevel the image's estimated resolution level, which must not be
* null
* @throws NullPointerException if {@code estimatedResolutionLevel} is null
*/
public Image(@Nonnull final String url,
final int height,
final int width,
@Nonnull final ResolutionLevel estimatedResolutionLevel)
throws NullPointerException {
this.url = url;
this.height = height;
this.width = width;
this.estimatedResolutionLevel = Objects.requireNonNull(
estimatedResolutionLevel, "estimatedResolutionLevel is null");
}
/**
* Get the URL of this {@link Image}.
*
* @return the {@link Image}'s URL.
*/
@Nonnull
public String getUrl() {
return url;
}
/**
* Get the height of this {@link Image}.
*
* <p>
* If it is unknown, {@link #HEIGHT_UNKNOWN} is returned instead.
* </p>
*
* @return the {@link Image}'s height or {@link #HEIGHT_UNKNOWN}
*/
public int getHeight() {
return height;
}
/**
* Get the width of this {@link Image}.
*
* <p>
* If it is unknown, {@link #WIDTH_UNKNOWN} is returned instead.
* </p>
*
* @return the {@link Image}'s width or {@link #WIDTH_UNKNOWN}
*/
public int getWidth() {
return width;
}
/**
* Get the estimated resolution level of this image.
*
* <p>
* If it is unknown, {@link ResolutionLevel#UNKNOWN} is returned instead.
* </p>
*
* @return the estimated resolution level, which is never {@code null}
* @see ResolutionLevel
*/
@Nonnull
public ResolutionLevel getEstimatedResolutionLevel() {
return estimatedResolutionLevel;
}
/**
* Get a string representation of this {@link Image} instance.
*
* <p>
* The representation will be in the following format, where {@code url}, {@code height},
* {@code width} and {@code estimatedResolutionLevel} represent the corresponding properties:
* <br>
* <br>
* {@code Image {url=url, height='height, width=width,
* estimatedResolutionLevel=estimatedResolutionLevel}'}
* </p>
*
* @return a string representation of this {@link Image} instance
*/
@Nonnull
@Override
public String toString() {
return "Image {" + "url=" + url + ", height=" + height + ", width=" + width
+ ", estimatedResolutionLevel=" + estimatedResolutionLevel + "}";
}
/**
* The estimated resolution level of an {@link Image}.
*
* <p>
* Some services don't return the size of their images, but we may know for a specific image
* type that a service returns, according to real data, an approximation of the resolution
* level.
* </p>
*/
public enum ResolutionLevel {
/**
* The high resolution level.
*
* <p>
* This level applies to images with a height greater than or equal to 720px.
* </p>
*/
HIGH,
/**
* The medium resolution level.
*
* <p>
* This level applies to images with a height between 175px inclusive and 720px exclusive.
* </p>
*/
MEDIUM,
/**
* The low resolution level.
*
* <p>
* This level applies to images with a height between 1px inclusive and 175px exclusive.
* </p>
*/
LOW,
/**
* The unknown resolution level.
*
* <p>
* This value is returned when the extractor doesn't know what resolution level an image
* could have, for example if the extractor loops in an array of images with different
* resolution levels without knowing the height.
* </p>
*/
UNKNOWN;
/**
* Get a {@link ResolutionLevel} based from the given height.
*
* @param heightPx the height from which returning the good {@link ResolutionLevel}
* @return the {@link ResolutionLevel} corresponding to the height provided. See the
* {@link ResolutionLevel} values for details about what value is returned.
*/
public static ResolutionLevel fromHeight(final int heightPx) {
if (heightPx <= 0) {
return UNKNOWN;
}
if (heightPx < 175) {
return LOW;
}
if (heightPx < 720) {
return MEDIUM;
}
return HIGH;
}
}
}

View File

@ -1,5 +1,6 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.LinkHandler;
import java.io.Serializable;
@ -16,7 +17,8 @@ public abstract class Info implements Serializable {
*/
private final String id;
/**
* Different than the {@link #originalUrl} in the sense that it <i>may</i> be set as a cleaned url.
* Different than the {@link #originalUrl} in the sense that it <i>may</i> be set as a cleaned
* url.
*
* @see LinkHandler#getUrl()
* @see Extractor#getOriginalUrl()
@ -32,15 +34,19 @@ public abstract class Info implements Serializable {
private final List<Throwable> errors = new ArrayList<>();
public void addError(Throwable throwable) {
public void addError(final Throwable throwable) {
this.errors.add(throwable);
}
public void addAllErrors(Collection<Throwable> errors) {
this.errors.addAll(errors);
public void addAllErrors(final Collection<Throwable> throwables) {
this.errors.addAll(throwables);
}
public Info(int serviceId, String id, String url, String originalUrl, String name) {
public Info(final int serviceId,
final String id,
final String url,
final String originalUrl,
final String name) {
this.serviceId = serviceId;
this.id = id;
this.url = url;
@ -48,7 +54,7 @@ public abstract class Info implements Serializable {
this.name = name;
}
public Info(int serviceId, LinkHandler linkHandler, String name) {
public Info(final int serviceId, final LinkHandler linkHandler, final String name) {
this(serviceId,
linkHandler.getId(),
linkHandler.getUrl(),
@ -58,20 +64,31 @@ public abstract class Info implements Serializable {
@Override
public String toString() {
final String ifDifferentString = !url.equals(originalUrl) ? " (originalUrl=\"" + originalUrl + "\")" : "";
return getClass().getSimpleName() + "[url=\"" + url + "\"" + ifDifferentString + ", name=\"" + name + "\"]";
final String ifDifferentString
= url.equals(originalUrl) ? "" : " (originalUrl=\"" + originalUrl + "\")";
return getClass().getSimpleName() + "[url=\"" + url + "\"" + ifDifferentString
+ ", name=\"" + name + "\"]";
}
// if you use an api and want to handle the website url
// overriding original url is essential
public void setOriginalUrl(String url) {
originalUrl = url;
public void setOriginalUrl(final String originalUrl) {
this.originalUrl = originalUrl;
}
public int getServiceId() {
return serviceId;
}
public StreamingService getService() {
try {
return NewPipe.getService(serviceId);
} catch (final ExtractionException e) {
// this should be unreachable, as serviceId certainly refers to a valid service
throw new RuntimeException("Info object has invalid service id", e);
}
}
public String getId() {
return id;
}

View File

@ -1,35 +1,41 @@
package org.schabi.newpipe.extractor;
/*
* Created by Christian Schabesberger on 11.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* InfoItem.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* InfoItem.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <https://www.gnu.org/licenses/>.
*/
package org.schabi.newpipe.extractor;
import javax.annotation.Nonnull;
import java.io.Serializable;
import java.util.List;
public abstract class InfoItem implements Serializable {
private final InfoType infoType;
private final int serviceId;
private final String url;
private final String name;
private String thumbnailUrl;
@Nonnull
private List<Image> thumbnails = List.of();
public InfoItem(InfoType infoType, int serviceId, String url, String name) {
public InfoItem(final InfoType infoType,
final int serviceId,
final String url,
final String name) {
this.infoType = infoType;
this.serviceId = serviceId;
this.url = url;
@ -52,12 +58,13 @@ public abstract class InfoItem implements Serializable {
return name;
}
public void setThumbnailUrl(String thumbnailUrl) {
this.thumbnailUrl = thumbnailUrl;
public void setThumbnails(@Nonnull final List<Image> thumbnails) {
this.thumbnails = thumbnails;
}
public String getThumbnailUrl() {
return thumbnailUrl;
@Nonnull
public List<Image> getThumbnails() {
return thumbnails;
}
@Override

View File

@ -2,8 +2,12 @@ package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import javax.annotation.Nonnull;
import java.util.List;
public interface InfoItemExtractor {
String getName() throws ParsingException;
String getUrl() throws ParsingException;
String getThumbnailUrl() throws ParsingException;
@Nonnull
List<Image> getThumbnails() throws ParsingException;
}

View File

@ -3,46 +3,63 @@ package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.exceptions.FoundAdException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import javax.annotation.Nullable;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.List;
/*
* Created by Christian Schabesberger on 12.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* InfoItemsCollector.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* InfoItemsCollector.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public abstract class InfoItemsCollector<I extends InfoItem, E extends InfoItemExtractor> implements Collector<I, E> {
public abstract class InfoItemsCollector<I extends InfoItem, E extends InfoItemExtractor>
implements Collector<I, E> {
private final List<I> itemList = new ArrayList<>();
private final List<Throwable> errors = new ArrayList<>();
private final int serviceId;
@Nullable
private final Comparator<I> comparator;
/**
* Create a new collector with no comparator / sorting function
* @param serviceId the service id
*/
public InfoItemsCollector(final int serviceId) {
this(serviceId, null);
}
/**
* Create a new collector
* @param serviceId the service id
*/
public InfoItemsCollector(int serviceId) {
public InfoItemsCollector(final int serviceId, @Nullable final Comparator<I> comparator) {
this.serviceId = serviceId;
this.comparator = comparator;
}
@Override
public List<I> getItems() {
if (comparator != null) {
itemList.sort(comparator);
}
return Collections.unmodifiableList(itemList);
}
@ -61,7 +78,7 @@ public abstract class InfoItemsCollector<I extends InfoItem, E extends InfoItemE
* Add an error
* @param error the error
*/
protected void addError(Exception error) {
protected void addError(final Exception error) {
errors.add(error);
}
@ -69,7 +86,7 @@ public abstract class InfoItemsCollector<I extends InfoItem, E extends InfoItemE
* Add an item
* @param item the item
*/
protected void addItem(I item) {
protected void addItem(final I item) {
itemList.add(item);
}
@ -82,12 +99,12 @@ public abstract class InfoItemsCollector<I extends InfoItem, E extends InfoItemE
}
@Override
public void commit(E extractor) {
public void commit(final E extractor) {
try {
addItem(extract(extractor));
} catch (FoundAdException ae) {
} catch (final FoundAdException ae) {
// found an ad. Maybe a debug line could be placed here
} catch (ParsingException e) {
} catch (final ParsingException e) {
addError(e);
}
}

View File

@ -2,73 +2,60 @@ package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.utils.Utils;
import java.io.IOException;
import java.util.Collections;
import java.util.List;
import javax.annotation.Nonnull;
import static org.schabi.newpipe.extractor.utils.Utils.isNullOrEmpty;
import javax.annotation.Nullable;
/**
* Base class to extractors that have a list (e.g. playlists, users).
* @param <R> the info item type this list extractor provides
*/
public abstract class ListExtractor<R extends InfoItem> extends Extractor {
/**
* Constant that should be returned whenever
* a list has an unknown number of items.
*/
public static final long ITEM_COUNT_UNKNOWN = -1;
/**
* Constant that should be returned whenever a list has an
* infinite number of items. For example a YouTube mix.
*/
public static final long ITEM_COUNT_INFINITE = -2;
/**
* Constant that should be returned whenever a list
* has an unknown number of items bigger than 100.
*/
public static final long ITEM_COUNT_MORE_THAN_100 = -3;
public ListExtractor(StreamingService service, ListLinkHandler linkHandler) {
public ListExtractor(final StreamingService service, final ListLinkHandler linkHandler) {
super(service, linkHandler);
}
/**
* A {@link InfoItemsPage InfoItemsPage} corresponding to the initial page where the items are from the initial request and
* the nextPageUrl relative to it.
* A {@link InfoItemsPage InfoItemsPage} corresponding to the initial page
* where the items are from the initial request and the nextPage relative to it.
*
* @return a {@link InfoItemsPage} corresponding to the initial page
*/
@Nonnull
public abstract InfoItemsPage<R> getInitialPage() throws IOException, ExtractionException;
/**
* Returns an url that can be used to get the next page relative to the initial one.
* <p>Usually, these links will only work in the implementation itself.</p>
*
* @return an url pointing to the next page relative to the initial page
* @see #getPage(String)
*/
public abstract String getNextPageUrl() throws IOException, ExtractionException;
/**
* Get a list of items corresponding to the specific requested page.
*
* @param pageUrl any page url got from the exclusive implementation of the list extractor
* @param page any page got from the exclusive implementation of the list extractor
* @return a {@link InfoItemsPage} corresponding to the requested page
* @see #getNextPageUrl()
* @see InfoItemsPage#getNextPageUrl()
* @see InfoItemsPage#getNextPage()
*/
public abstract InfoItemsPage<R> getPage(final String pageUrl) throws IOException, ExtractionException;
public boolean hasNextPage() throws IOException, ExtractionException {
return !isNullOrEmpty(getNextPageUrl());
}
public abstract InfoItemsPage<R> getPage(Page page) throws IOException, ExtractionException;
@Nonnull
@Override
public ListLinkHandler getLinkHandler() {
return (ListLinkHandler) super.getLinkHandler();
@ -80,23 +67,27 @@ public abstract class ListExtractor<R extends InfoItem> extends Extractor {
/**
* A class that is used to wrap a list of gathered items and eventual errors, it
* also contains a field that points to the next available page ({@link #nextPageUrl}).
* also contains a field that points to the next available page ({@link #nextPage}).
* @param <T> the info item type that this page is supposed to store and provide
*/
public static class InfoItemsPage<T extends InfoItem> {
private static final InfoItemsPage<InfoItem> EMPTY =
new InfoItemsPage<>(Collections.<InfoItem>emptyList(), "", Collections.<Throwable>emptyList());
private static final InfoItemsPage<InfoItem> EMPTY = new InfoItemsPage<>(
Collections.emptyList(),
null,
Collections.emptyList()
);
/**
* A convenient method that returns a representation of an empty page.
*
* @return a type-safe page with the list of items and errors empty and the nextPageUrl set to an empty string.
* @return a type-safe page with the list of items and errors empty and the nextPage set to
* {@code null}.
*/
public static <T extends InfoItem> InfoItemsPage<T> emptyPage() {
//noinspection unchecked
return (InfoItemsPage<T>) EMPTY;
}
/**
* The current list of items of this page
*/
@ -105,40 +96,48 @@ public abstract class ListExtractor<R extends InfoItem> extends Extractor {
/**
* Url pointing to the next page relative to this one
*
* @see ListExtractor#getPage(String)
* @see ListExtractor#getPage(Page)
* @see Page
*/
private final String nextPageUrl;
@Nullable
private final Page nextPage;
/**
* Errors that happened during the extraction
*/
private final List<Throwable> errors;
public InfoItemsPage(InfoItemsCollector<T, ?> collector, String nextPageUrl) {
this(collector.getItems(), nextPageUrl, collector.getErrors());
public InfoItemsPage(final InfoItemsCollector<T, ?> collector,
@Nullable final Page nextPage) {
this(collector.getItems(), nextPage, collector.getErrors());
}
public InfoItemsPage(List<T> itemsList, String nextPageUrl, List<Throwable> errors) {
public InfoItemsPage(final List<T> itemsList,
@Nullable final Page nextPage,
final List<Throwable> errors) {
this.itemsList = itemsList;
this.nextPageUrl = nextPageUrl;
this.nextPage = nextPage;
this.errors = errors;
}
public boolean hasNextPage() {
return !isNullOrEmpty(nextPageUrl);
return Page.isValid(nextPage);
}
public List<T> getItems() {
return itemsList;
}
public String getNextPageUrl() {
return nextPageUrl;
/**
* @return the next page if available, or null otherwise
*/
@Nullable
public Page getNextPage() {
return nextPage;
}
public List<Throwable> getErrors() {
return errors;
}
}
}

View File

@ -4,27 +4,27 @@ import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import java.util.List;
import static org.schabi.newpipe.extractor.utils.Utils.isNullOrEmpty;
public abstract class ListInfo<T extends InfoItem> extends Info {
private List<T> relatedItems;
private String nextPageUrl = null;
private Page nextPage = null;
private final List<String> contentFilters;
private final String sortFilter;
public ListInfo(int serviceId,
String id,
String url,
String originalUrl,
String name,
List<String> contentFilter,
String sortFilter) {
public ListInfo(final int serviceId,
final String id,
final String url,
final String originalUrl,
final String name,
final List<String> contentFilter,
final String sortFilter) {
super(serviceId, id, url, originalUrl, name);
this.contentFilters = contentFilter;
this.sortFilter = sortFilter;
}
public ListInfo(int serviceId, ListLinkHandler listUrlIdHandler, String name) {
public ListInfo(final int serviceId,
final ListLinkHandler listUrlIdHandler,
final String name) {
super(serviceId, listUrlIdHandler, name);
this.contentFilters = listUrlIdHandler.getContentFilters();
this.sortFilter = listUrlIdHandler.getSortFilter();
@ -34,20 +34,20 @@ public abstract class ListInfo<T extends InfoItem> extends Info {
return relatedItems;
}
public void setRelatedItems(List<T> relatedItems) {
public void setRelatedItems(final List<T> relatedItems) {
this.relatedItems = relatedItems;
}
public boolean hasNextPage() {
return !isNullOrEmpty(nextPageUrl);
return Page.isValid(nextPage);
}
public String getNextPageUrl() {
return nextPageUrl;
public Page getNextPage() {
return nextPage;
}
public void setNextPageUrl(String pageUrl) {
this.nextPageUrl = pageUrl;
public void setNextPage(final Page page) {
this.nextPage = page;
}
public List<String> getContentFilters() {

View File

@ -3,115 +3,158 @@ package org.schabi.newpipe.extractor;
/*
* Created by Adam Howard on 08/11/15.
*
* Copyright (c) Christian Schabesberger <chris.schabesberger@mailbox.org>
* and Adam Howard <achdisposable1@gmail.com> 2015
* Copyright (c) 2015 Christian Schabesberger <chris.schabesberger@mailbox.org>
* and Adam Howard <achdisposable1@gmail.com>
*
* MediaFormat.java is part of NewPipe.
* MediaFormat.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.Arrays;
import java.util.List;
import java.util.function.Function;
import java.util.stream.Collectors;
/**
* Static data about various media formats support by NewPipe, eg mime type, extension
*/
@SuppressWarnings("MethodParamPad") // we want the media format table below to be aligned
public enum MediaFormat {
// @formatter:off
//video and audio combined formats
// id name suffix mime type
MPEG_4 (0x0, "MPEG-4", "mp4", "video/mp4"),
v3GPP (0x10, "3GPP", "3gp", "video/3gpp"),
WEBM (0x20, "WebM", "webm", "video/webm"),
// id name suffix mimeType
MPEG_4 (0x0, "MPEG-4", "mp4", "video/mp4"),
v3GPP (0x10, "3GPP", "3gp", "video/3gpp"),
WEBM (0x20, "WebM", "webm", "video/webm"),
// audio formats
M4A (0x100, "m4a", "m4a", "audio/mp4"),
WEBMA (0x200, "WebM", "webm", "audio/webm"),
MP3 (0x300, "MP3", "mp3", "audio/mpeg"),
OPUS (0x400, "opus", "opus", "audio/opus"),
OGG (0x500, "ogg", "ogg", "audio/ogg"),
WEBMA_OPUS (0x200, "WebM Opus", "webm", "audio/webm"),
M4A (0x100, "m4a", "m4a", "audio/mp4"),
WEBMA (0x200, "WebM", "webm", "audio/webm"),
MP3 (0x300, "MP3", "mp3", "audio/mpeg"),
MP2 (0x310, "MP2", "mp2", "audio/mpeg"),
OPUS (0x400, "opus", "opus", "audio/opus"),
OGG (0x500, "ogg", "ogg", "audio/ogg"),
WEBMA_OPUS(0x200, "WebM Opus", "webm", "audio/webm"),
AIFF (0x600, "AIFF", "aiff", "audio/aiff"),
/**
* Same as {@link MediaFormat#AIFF}, just with the shorter suffix/file extension
*/
AIF (0x600, "AIFF", "aif", "audio/aiff"),
WAV (0x700, "WAV", "wav", "audio/wav"),
FLAC (0x800, "FLAC", "flac", "audio/flac"),
ALAC (0x900, "ALAC", "alac", "audio/alac"),
// subtitles formats
VTT (0x1000, "WebVTT", "vtt", "text/vtt"),
TTML (0x2000, "Timed Text Markup Language", "ttml", "application/ttml+xml"),
TRANSCRIPT1 (0x3000, "TranScript v1", "srv1", "text/xml"),
TRANSCRIPT2 (0x4000, "TranScript v2", "srv2", "text/xml"),
TRANSCRIPT3 (0x5000, "TranScript v3", "srv3", "text/xml"),
SRT (0x6000, "SubRip file format", "srt", "text/srt");
VTT (0x1000, "WebVTT", "vtt", "text/vtt"),
TTML (0x2000, "Timed Text Markup Language", "ttml", "application/ttml+xml"),
TRANSCRIPT1(0x3000, "TranScript v1", "srv1", "text/xml"),
TRANSCRIPT2(0x4000, "TranScript v2", "srv2", "text/xml"),
TRANSCRIPT3(0x5000, "TranScript v3", "srv3", "text/xml"),
SRT (0x6000, "SubRip file format", "srt", "text/srt");
// @formatter:on
public final int id;
@Nonnull
public final String name;
@Nonnull
public final String suffix;
@Nonnull
public final String mimeType;
MediaFormat(int id, String name, String suffix, String mimeType) {
MediaFormat(final int id, @Nonnull final String name,
@Nonnull final String suffix, @Nonnull final String mimeType) {
this.id = id;
this.name = name;
this.suffix = suffix;
this.mimeType = mimeType;
}
private static <T> T getById(final int id,
final Function<MediaFormat, T> field,
final T orElse) {
return Arrays.stream(MediaFormat.values())
.filter(mediaFormat -> mediaFormat.id == id)
.map(field)
.findFirst()
.orElse(orElse);
}
/**
* Return the friendly name of the media format with the supplied id
*
* @param ident the id of the media format. Currently an arbitrary, NewPipe-specific number.
* @param id the id of the media format. Currently an arbitrary, NewPipe-specific number.
* @return the friendly name of the MediaFormat associated with this ids,
* or an empty String if none match it.
*/
public static String getNameById(int ident) {
for (MediaFormat vf : MediaFormat.values()) {
if (vf.id == ident) return vf.name;
}
return "";
@Nonnull
public static String getNameById(final int id) {
return getById(id, MediaFormat::getName, "");
}
/**
* Return the file extension of the media format with the supplied id
*
* @param ident the id of the media format. Currently an arbitrary, NewPipe-specific number.
* @param id the id of the media format. Currently an arbitrary, NewPipe-specific number.
* @return the file extension of the MediaFormat associated with this ids,
* or an empty String if none match it.
*/
public static String getSuffixById(int ident) {
for (MediaFormat vf : MediaFormat.values()) {
if (vf.id == ident) return vf.suffix;
}
return "";
@Nonnull
public static String getSuffixById(final int id) {
return getById(id, MediaFormat::getSuffix, "");
}
/**
* Return the MIME type of the media format with the supplied id
*
* @param ident the id of the media format. Currently an arbitrary, NewPipe-specific number.
* @param id the id of the media format. Currently an arbitrary, NewPipe-specific number.
* @return the MIME type of the MediaFormat associated with this ids,
* or an empty String if none match it.
*/
public static String getMimeById(int ident) {
for (MediaFormat vf : MediaFormat.values()) {
if (vf.id == ident) return vf.mimeType;
}
return "";
@Nullable
public static String getMimeById(final int id) {
return getById(id, MediaFormat::getMimeType, null);
}
/**
* Return the MediaFormat with the supplied mime type
* Return the first {@link MediaFormat} with the supplied mime type.
* There might be more formats which have the same mime type.
* To retrieve those, use {@link #getAllFromMimeType(String)}.
*
* @return MediaFormat associated with this mime type,
* or null if none match it.
*/
public static MediaFormat getFromMimeType(String mimeType) {
for (MediaFormat vf : MediaFormat.values()) {
if (vf.mimeType.equals(mimeType)) return vf;
}
return null;
@Nullable
public static MediaFormat getFromMimeType(final String mimeType) {
return Arrays.stream(MediaFormat.values())
.filter(mediaFormat -> mediaFormat.mimeType.equals(mimeType))
.findFirst()
.orElse(null);
}
/**
* Get all media formats which have the given mime type.
* @param mimeType the mime type to search for
* @return a modifiable {@link List} which contains the {@link MediaFormat}s
* that have the given mime type.
*/
@Nonnull
public static List<MediaFormat> getAllFromMimeType(final String mimeType) {
return Arrays.stream(MediaFormat.values())
.filter(mediaFormat -> mediaFormat.mimeType.equals(mimeType))
.collect(Collectors.toList());
}
/**
@ -120,18 +163,21 @@ public enum MediaFormat {
* @param id the id
* @return the id of the media format or null.
*/
public static MediaFormat getFormatById(int id) {
for (MediaFormat vf : values()) {
if (vf.id == id) return vf;
}
return null;
@Nullable
public static MediaFormat getFormatById(final int id) {
return getById(id, mediaFormat -> mediaFormat, null);
}
public static MediaFormat getFromSuffix(String suffix) {
for (MediaFormat vf : values()) {
if (vf.suffix.equals(suffix)) return vf;
}
return null;
/**
* Get the first media format that has the given suffix/file extension.
* @return the matching {@link MediaFormat} or {@code null} if no associated format is found
*/
@Nullable
public static MediaFormat getFromSuffix(final String suffix) {
return Arrays.stream(MediaFormat.values())
.filter(mediaFormat -> mediaFormat.suffix.equals(suffix))
.findFirst()
.orElse(null);
}
/**
@ -139,6 +185,7 @@ public enum MediaFormat {
*
* @return the name of the format
*/
@Nonnull
public String getName() {
return name;
}
@ -148,6 +195,7 @@ public enum MediaFormat {
*
* @return the filename extension
*/
@Nonnull
public String getSuffix() {
return suffix;
}
@ -157,6 +205,7 @@ public enum MediaFormat {
*
* @return the mime type
*/
@Nonnull
public String getMimeType() {
return mimeType;
}

View File

@ -0,0 +1,78 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.stream.Description;
import java.io.Serializable;
import java.net.URL;
import java.util.ArrayList;
import java.util.List;
import javax.annotation.Nonnull;
public class MetaInfo implements Serializable {
private String title = "";
private Description content;
private List<URL> urls = new ArrayList<>();
private List<String> urlTexts = new ArrayList<>();
public MetaInfo(@Nonnull final String title,
@Nonnull final Description content,
@Nonnull final List<URL> urls,
@Nonnull final List<String> urlTexts) {
this.title = title;
this.content = content;
this.urls = urls;
this.urlTexts = urlTexts;
}
public MetaInfo() {
}
/**
* @return Title of the info. Can be empty.
*/
@Nonnull
public String getTitle() {
return title;
}
public void setTitle(@Nonnull final String title) {
this.title = title;
}
@Nonnull
public Description getContent() {
return content;
}
public void setContent(@Nonnull final Description content) {
this.content = content;
}
@Nonnull
public List<URL> getUrls() {
return urls;
}
public void setUrls(@Nonnull final List<URL> urls) {
this.urls = urls;
}
public void addUrl(@Nonnull final URL url) {
urls.add(url);
}
@Nonnull
public List<String> getUrlTexts() {
return urlTexts;
}
public void setUrlTexts(@Nonnull final List<String> urlTexts) {
this.urlTexts = urlTexts;
}
public void addUrlText(@Nonnull final String urlText) {
urlTexts.add(urlText);
}
}

View File

@ -1,8 +1,5 @@
package org.schabi.newpipe.extractor.search;
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.InfoItemExtractor;
import org.schabi.newpipe.extractor.InfoItemsCollector;
import org.schabi.newpipe.extractor.channel.ChannelInfoItemExtractor;
import org.schabi.newpipe.extractor.channel.ChannelInfoItemsCollector;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
@ -18,25 +15,26 @@ import java.util.List;
/*
* Created by Christian Schabesberger on 12.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* InfoItemsSearchCollector.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* InfoItemsSearchCollector.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
/**
* Collector for search results
* A collector that can handle many extractor types, to be used when a list contains items of
* different types (e.g. search)
* <p>
* This collector can handle the following extractor types:
* <ul>
@ -44,15 +42,15 @@ import java.util.List;
* <li>{@link ChannelInfoItemExtractor}</li>
* <li>{@link PlaylistInfoItemExtractor}</li>
* </ul>
* Calling {@link #extract(InfoItemExtractor)} or {@link #commit(Object)} with any
* Calling {@link #extract(InfoItemExtractor)} or {@link #commit(InfoItemExtractor)} with any
* other extractor type will raise an exception.
*/
public class InfoItemsSearchCollector extends InfoItemsCollector<InfoItem, InfoItemExtractor> {
public class MultiInfoItemsCollector extends InfoItemsCollector<InfoItem, InfoItemExtractor> {
private final StreamInfoItemsCollector streamCollector;
private final ChannelInfoItemsCollector userCollector;
private final PlaylistInfoItemsCollector playlistCollector;
public InfoItemsSearchCollector(int serviceId) {
public MultiInfoItemsCollector(final int serviceId) {
super(serviceId);
streamCollector = new StreamInfoItemsCollector(serviceId);
userCollector = new ChannelInfoItemsCollector(serviceId);
@ -78,7 +76,7 @@ public class InfoItemsSearchCollector extends InfoItemsCollector<InfoItem, InfoI
}
@Override
public InfoItem extract(InfoItemExtractor extractor) throws ParsingException {
public InfoItem extract(final InfoItemExtractor extractor) throws ParsingException {
// Use the corresponding collector for each item extractor type
if (extractor instanceof StreamInfoItemExtractor) {
return streamCollector.extract((StreamInfoItemExtractor) extractor);

View File

@ -3,21 +3,21 @@ package org.schabi.newpipe.extractor;
/*
* Created by Christian Schabesberger on 23.08.15.
*
* Copyright (C) Christian Schabesberger 2015 <chris.schabesberger@mailbox.org>
* NewPipe.java is part of NewPipe.
* Copyright (C) 2015 Christian Schabesberger <chris.schabesberger@mailbox.org>
* NewPipe Extractor.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
import org.schabi.newpipe.extractor.downloader.Downloader;
@ -25,14 +25,15 @@ import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.localization.ContentCountry;
import org.schabi.newpipe.extractor.localization.Localization;
import java.util.List;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.List;
/**
* Provides access to streaming services supported by NewPipe.
*/
public class NewPipe {
public final class NewPipe {
private static Downloader downloader;
private static Localization preferredLocalization;
private static ContentCountry preferredContentCountry;
@ -40,19 +41,16 @@ public class NewPipe {
private NewPipe() {
}
public static void init(Downloader d) {
downloader = d;
preferredLocalization = Localization.DEFAULT;
preferredContentCountry = ContentCountry.DEFAULT;
public static void init(final Downloader d) {
init(d, Localization.DEFAULT);
}
public static void init(Downloader d, Localization l) {
downloader = d;
preferredLocalization = l;
preferredContentCountry = l.getCountryCode().isEmpty() ? ContentCountry.DEFAULT : new ContentCountry(l.getCountryCode());
public static void init(final Downloader d, final Localization l) {
init(d, l, l.getCountryCode().isEmpty()
? ContentCountry.DEFAULT : new ContentCountry(l.getCountryCode()));
}
public static void init(Downloader d, Localization l, ContentCountry c) {
public static void init(final Downloader d, final Localization l, final ContentCountry c) {
downloader = d;
preferredLocalization = l;
preferredContentCountry = c;
@ -70,26 +68,24 @@ public class NewPipe {
return ServiceList.all();
}
public static StreamingService getService(int serviceId) throws ExtractionException {
for (StreamingService service : ServiceList.all()) {
if (service.getServiceId() == serviceId) {
return service;
}
}
throw new ExtractionException("There's no service with the id = \"" + serviceId + "\"");
public static StreamingService getService(final int serviceId) throws ExtractionException {
return ServiceList.all().stream()
.filter(service -> service.getServiceId() == serviceId)
.findFirst()
.orElseThrow(() -> new ExtractionException(
"There's no service with the id = \"" + serviceId + "\""));
}
public static StreamingService getService(String serviceName) throws ExtractionException {
for (StreamingService service : ServiceList.all()) {
if (service.getServiceInfo().getName().equals(serviceName)) {
return service;
}
}
throw new ExtractionException("There's no service with the name = \"" + serviceName + "\"");
public static StreamingService getService(final String serviceName) throws ExtractionException {
return ServiceList.all().stream()
.filter(service -> service.getServiceInfo().getName().equals(serviceName))
.findFirst()
.orElseThrow(() -> new ExtractionException(
"There's no service with the name = \"" + serviceName + "\""));
}
public static StreamingService getServiceByUrl(String url) throws ExtractionException {
for (StreamingService service : ServiceList.all()) {
public static StreamingService getServiceByUrl(final String url) throws ExtractionException {
for (final StreamingService service : ServiceList.all()) {
if (service.getLinkTypeByUrl(url) != StreamingService.LinkType.NONE) {
return service;
}
@ -97,43 +93,25 @@ public class NewPipe {
throw new ExtractionException("No service can handle the url = \"" + url + "\"");
}
public static int getIdOfService(String serviceName) {
try {
//noinspection ConstantConditions
return getService(serviceName).getServiceId();
} catch (ExtractionException ignored) {
return -1;
}
}
public static String getNameOfService(int id) {
try {
//noinspection ConstantConditions
return getService(id).getServiceInfo().getName();
} catch (Exception e) {
System.err.println("Service id not known");
e.printStackTrace();
return "<unknown>";
}
}
/*//////////////////////////////////////////////////////////////////////////
// Localization
//////////////////////////////////////////////////////////////////////////*/
public static void setupLocalization(Localization preferredLocalization) {
setupLocalization(preferredLocalization, null);
public static void setupLocalization(final Localization thePreferredLocalization) {
setupLocalization(thePreferredLocalization, null);
}
public static void setupLocalization(Localization preferredLocalization, @Nullable ContentCountry preferredContentCountry) {
NewPipe.preferredLocalization = preferredLocalization;
public static void setupLocalization(
final Localization thePreferredLocalization,
@Nullable final ContentCountry thePreferredContentCountry) {
NewPipe.preferredLocalization = thePreferredLocalization;
if (preferredContentCountry != null) {
NewPipe.preferredContentCountry = preferredContentCountry;
if (thePreferredContentCountry != null) {
NewPipe.preferredContentCountry = thePreferredContentCountry;
} else {
NewPipe.preferredContentCountry = preferredLocalization.getCountryCode().isEmpty()
NewPipe.preferredContentCountry = thePreferredLocalization.getCountryCode().isEmpty()
? ContentCountry.DEFAULT
: new ContentCountry(preferredLocalization.getCountryCode());
: new ContentCountry(thePreferredLocalization.getCountryCode());
}
}
@ -142,7 +120,7 @@ public class NewPipe {
return preferredLocalization == null ? Localization.DEFAULT : preferredLocalization;
}
public static void setPreferredLocalization(Localization preferredLocalization) {
public static void setPreferredLocalization(final Localization preferredLocalization) {
NewPipe.preferredLocalization = preferredLocalization;
}
@ -151,7 +129,7 @@ public class NewPipe {
return preferredContentCountry == null ? ContentCountry.DEFAULT : preferredContentCountry;
}
public static void setPreferredContentCountry(ContentCountry preferredContentCountry) {
public static void setPreferredContentCountry(final ContentCountry preferredContentCountry) {
NewPipe.preferredContentCountry = preferredContentCountry;
}
}

View File

@ -0,0 +1,85 @@
package org.schabi.newpipe.extractor;
import java.io.Serializable;
import java.util.List;
import java.util.Map;
import javax.annotation.Nullable;
import static org.schabi.newpipe.extractor.utils.Utils.isNullOrEmpty;
public class Page implements Serializable {
private final String url;
private final String id;
private final List<String> ids;
private final Map<String, String> cookies;
@Nullable
private final byte[] body;
public Page(final String url,
final String id,
final List<String> ids,
final Map<String, String> cookies,
@Nullable final byte[] body) {
this.url = url;
this.id = id;
this.ids = ids;
this.cookies = cookies;
this.body = body;
}
public Page(final String url) {
this(url, null, null, null, null);
}
public Page(final String url, final String id) {
this(url, id, null, null, null);
}
public Page(final String url, final String id, final byte[] body) {
this(url, id, null, null, body);
}
public Page(final String url, final byte[] body) {
this(url, null, null, null, body);
}
public Page(final String url, final Map<String, String> cookies) {
this(url, null, null, cookies, null);
}
public Page(final List<String> ids) {
this(null, null, ids, null, null);
}
public Page(final List<String> ids, final Map<String, String> cookies) {
this(null, null, ids, cookies, null);
}
public String getUrl() {
return url;
}
public String getId() {
return id;
}
public List<String> getIds() {
return ids;
}
public Map<String, String> getCookies() {
return cookies;
}
public static boolean isValid(final Page page) {
return page != null && (!isNullOrEmpty(page.getUrl())
|| !isNullOrEmpty(page.getIds()));
}
@Nullable
public byte[] getBody() {
return body;
}
}

View File

@ -1,56 +1,52 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.services.bandcamp.BandcampService;
import org.schabi.newpipe.extractor.services.media_ccc.MediaCCCService;
import org.schabi.newpipe.extractor.services.peertube.PeertubeService;
import org.schabi.newpipe.extractor.services.soundcloud.SoundcloudService;
import org.schabi.newpipe.extractor.services.youtube.YoutubeService;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
/*
* Copyright (C) Christian Schabesberger 2018 <chris.schabesberger@mailbox.org>
* ServiceList.java is part of NewPipe.
* Copyright (C) 2018 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ServiceList.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
/**
* A list of supported services.
*/
@SuppressWarnings({"ConstantName", "InnerAssignment"}) // keep unusual names and inner assignments
public final class ServiceList {
private ServiceList() {
//no instance
// no instance
}
public static final YoutubeService YouTube;
public static final SoundcloudService SoundCloud;
public static final MediaCCCService MediaCCC;
public static final PeertubeService PeerTube;
public static final YoutubeService YouTube = new YoutubeService(0);
public static final SoundcloudService SoundCloud = new SoundcloudService(1);
public static final MediaCCCService MediaCCC = new MediaCCCService(2);
public static final PeertubeService PeerTube = new PeertubeService(3);
public static final BandcampService Bandcamp = new BandcampService(4);
/**
* When creating a new service, put this service in the end of this list,
* and give it the next free id.
*/
private static final List<StreamingService> SERVICES = Collections.unmodifiableList(
Arrays.asList(
YouTube = new YoutubeService(0),
SoundCloud = new SoundcloudService(1),
MediaCCC = new MediaCCCService(2),
PeerTube = new PeertubeService(3)
));
private static final List<StreamingService> SERVICES = List.of(
YouTube, SoundCloud, MediaCCC, PeerTube, Bandcamp);
/**
* Get all the supported services.

View File

@ -1,12 +1,18 @@
package org.schabi.newpipe.extractor;
import org.schabi.newpipe.extractor.channel.ChannelExtractor;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import org.schabi.newpipe.extractor.comments.CommentsExtractor;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.feed.FeedExtractor;
import org.schabi.newpipe.extractor.kiosk.KioskList;
import org.schabi.newpipe.extractor.linkhandler.*;
import org.schabi.newpipe.extractor.linkhandler.LinkHandler;
import org.schabi.newpipe.extractor.linkhandler.LinkHandlerFactory;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandler;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandlerFactory;
import org.schabi.newpipe.extractor.localization.ContentCountry;
import org.schabi.newpipe.extractor.localization.Localization;
import org.schabi.newpipe.extractor.localization.TimeAgoParser;
@ -16,27 +22,28 @@ import org.schabi.newpipe.extractor.search.SearchExtractor;
import org.schabi.newpipe.extractor.stream.StreamExtractor;
import org.schabi.newpipe.extractor.subscription.SubscriptionExtractor;
import org.schabi.newpipe.extractor.suggestion.SuggestionExtractor;
import org.schabi.newpipe.extractor.utils.Utils;
import javax.annotation.Nullable;
import java.util.Collections;
import java.util.List;
/*
* Copyright (C) Christian Schabesberger 2018 <chris.schabesberger@mailbox.org>
* StreamingService.java is part of NewPipe.
* Copyright (C) 2018 Christian Schabesberger <chris.schabesberger@mailbox.org>
* StreamingService.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public abstract class StreamingService {
@ -54,7 +61,7 @@ public abstract class StreamingService {
* @param name the name of the service
* @param mediaCapabilities the type of media this service can handle
*/
public ServiceInfo(String name, List<MediaCapability> mediaCapabilities) {
public ServiceInfo(final String name, final List<MediaCapability> mediaCapabilities) {
this.name = name;
this.mediaCapabilities = Collections.unmodifiableList(mediaCapabilities);
}
@ -73,8 +80,8 @@ public abstract class StreamingService {
}
/**
* LinkType will be used to determine which type of URL you are handling, and therefore which part
* of NewPipe should handle a certain URL.
* LinkType will be used to determine which type of URL you are handling, and therefore which
* part of NewPipe should handle a certain URL.
*/
public enum LinkType {
NONE,
@ -89,14 +96,15 @@ public abstract class StreamingService {
/**
* Creates a new Streaming service.
* If you Implement one do not set id within your implementation of this extractor, instead
* set the id when you put the extractor into
* <a href="https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ServiceList.html">ServiceList</a>.
* set the id when you put the extractor into {@link ServiceList}
* All other parameters can be set directly from the overriding constructor.
* @param id the number of the service to identify him within the NewPipe frontend
* @param name the name of the service
* @param capabilities the type of media this service can handle
*/
public StreamingService(int id, String name, List<ServiceInfo.MediaCapability> capabilities) {
public StreamingService(final int id,
final String name,
final List<ServiceInfo.MediaCapability> capabilities) {
this.serviceId = id;
this.serviceInfo = new ServiceInfo(name, capabilities);
}
@ -133,6 +141,14 @@ public abstract class StreamingService {
*/
public abstract ListLinkHandlerFactory getChannelLHFactory();
/**
* Must return a new instance of an implementation of ListLinkHandlerFactory for channel tabs.
* If support for channel tabs is not given null must be returned.
*
* @return an instance of a ListLinkHandlerFactory for channel tabs or null
*/
public abstract ListLinkHandlerFactory getChannelTabLHFactory();
/**
* Must return a new instance of an implementation of ListLinkHandlerFactory for playlists.
* If support for playlists is not given null must be returned.
@ -171,22 +187,21 @@ public abstract class StreamingService {
public abstract SubscriptionExtractor getSubscriptionExtractor();
/**
* This method decides which strategy will be chosen to fetch the feed. In YouTube, for example, a separate feed
* exists which is lightweight and made specifically to be used like this.
* This method decides which strategy will be chosen to fetch the feed. In YouTube, for example,
* a separate feed exists which is lightweight and made specifically to be used like this.
* <p>
* In services which there's no other way to retrieve them, null should be returned.
*
* @return a {@link FeedExtractor} instance or null.
*/
@Nullable
public FeedExtractor getFeedExtractor(String url) throws ExtractionException {
public FeedExtractor getFeedExtractor(final String url) throws ExtractionException {
return null;
}
/**
* Must create a new instance of a KioskList implementation.
* @return a new KioskList instance
* @throws ExtractionException
*/
public abstract KioskList getKioskList() throws ExtractionException;
@ -194,49 +209,61 @@ public abstract class StreamingService {
* Must create a new instance of a ChannelExtractor implementation.
* @param linkHandler is pointing to the channel which should be handled by this new instance.
* @return a new ChannelExtractor
* @throws ExtractionException
*/
public abstract ChannelExtractor getChannelExtractor(ListLinkHandler linkHandler) throws ExtractionException;
public abstract ChannelExtractor getChannelExtractor(ListLinkHandler linkHandler)
throws ExtractionException;
/**
* Must create a new instance of a ChannelTabExtractor implementation.
*
* @param linkHandler is pointing to the channel which should be handled by this new instance.
* @return a new ChannelTabExtractor
*/
public abstract ChannelTabExtractor getChannelTabExtractor(ListLinkHandler linkHandler)
throws ExtractionException;
/**
* Must crete a new instance of a PlaylistExtractor implementation.
* @param linkHandler is pointing to the playlist which should be handled by this new instance.
* @return a new PlaylistExtractor
* @throws ExtractionException
*/
public abstract PlaylistExtractor getPlaylistExtractor(ListLinkHandler linkHandler) throws ExtractionException;
public abstract PlaylistExtractor getPlaylistExtractor(ListLinkHandler linkHandler)
throws ExtractionException;
/**
* Must create a new instance of a StreamExtractor implementation.
* @param linkHandler is pointing to the stream which should be handled by this new instance.
* @return a new StreamExtractor
* @throws ExtractionException
*/
public abstract StreamExtractor getStreamExtractor(LinkHandler linkHandler) throws ExtractionException;
public abstract StreamExtractor getStreamExtractor(LinkHandler linkHandler)
throws ExtractionException;
public abstract CommentsExtractor getCommentsExtractor(ListLinkHandler linkHandler) throws ExtractionException;
public abstract CommentsExtractor getCommentsExtractor(ListLinkHandler linkHandler)
throws ExtractionException;
/*//////////////////////////////////////////////////////////////////////////
// Extractors without link handler
//////////////////////////////////////////////////////////////////////////*/
public SearchExtractor getSearchExtractor(String query,
List<String> contentFilter,
String sortFilter) throws ExtractionException {
public SearchExtractor getSearchExtractor(final String query,
final List<String> contentFilter,
final String sortFilter) throws ExtractionException {
return getSearchExtractor(getSearchQHFactory()
.fromQuery(query, contentFilter, sortFilter));
}
public ChannelExtractor getChannelExtractor(String id,
List<String> contentFilter,
String sortFilter) throws ExtractionException {
public ChannelExtractor getChannelExtractor(final String id,
final List<String> contentFilter,
final String sortFilter)
throws ExtractionException {
return getChannelExtractor(getChannelLHFactory()
.fromQuery(id, contentFilter, sortFilter));
}
public PlaylistExtractor getPlaylistExtractor(String id,
List<String> contentFilter,
String sortFilter) throws ExtractionException {
public PlaylistExtractor getPlaylistExtractor(final String id,
final List<String> contentFilter,
final String sortFilter)
throws ExtractionException {
return getPlaylistExtractor(getPlaylistLHFactory()
.fromQuery(id, contentFilter, sortFilter));
}
@ -245,28 +272,42 @@ public abstract class StreamingService {
// Short extractors overloads
//////////////////////////////////////////////////////////////////////////*/
public SearchExtractor getSearchExtractor(String query) throws ExtractionException {
public SearchExtractor getSearchExtractor(final String query) throws ExtractionException {
return getSearchExtractor(getSearchQHFactory().fromQuery(query));
}
public ChannelExtractor getChannelExtractor(String url) throws ExtractionException {
public ChannelExtractor getChannelExtractor(final String url) throws ExtractionException {
return getChannelExtractor(getChannelLHFactory().fromUrl(url));
}
public PlaylistExtractor getPlaylistExtractor(String url) throws ExtractionException {
public ChannelTabExtractor getChannelTabExtractorFromId(final String id, final String tab)
throws ExtractionException {
return getChannelTabExtractor(getChannelTabLHFactory().fromQuery(
id, Collections.singletonList(tab), ""));
}
public ChannelTabExtractor getChannelTabExtractorFromIdAndBaseUrl(final String id,
final String tab,
final String baseUrl)
throws ExtractionException {
return getChannelTabExtractor(getChannelTabLHFactory().fromQuery(
id, Collections.singletonList(tab), "", baseUrl));
}
public PlaylistExtractor getPlaylistExtractor(final String url) throws ExtractionException {
return getPlaylistExtractor(getPlaylistLHFactory().fromUrl(url));
}
public StreamExtractor getStreamExtractor(String url) throws ExtractionException {
public StreamExtractor getStreamExtractor(final String url) throws ExtractionException {
return getStreamExtractor(getStreamLHFactory().fromUrl(url));
}
public CommentsExtractor getCommentsExtractor(String url) throws ExtractionException {
ListLinkHandlerFactory llhf = getCommentsLHFactory();
if (llhf == null) {
public CommentsExtractor getCommentsExtractor(final String url) throws ExtractionException {
final ListLinkHandlerFactory listLinkHandlerFactory = getCommentsLHFactory();
if (listLinkHandlerFactory == null) {
return null;
}
return getCommentsExtractor(llhf.fromUrl(url));
return getCommentsExtractor(listLinkHandlerFactory.fromUrl(url));
}
/*//////////////////////////////////////////////////////////////////////////
@ -277,18 +318,19 @@ public abstract class StreamingService {
* Figures out where the link is pointing to (a channel, a video, a playlist, etc.)
* @param url the url on which it should be decided of which link type it is
* @return the link type of url
* @throws ParsingException
*/
public final LinkType getLinkTypeByUrl(String url) throws ParsingException {
LinkHandlerFactory sH = getStreamLHFactory();
LinkHandlerFactory cH = getChannelLHFactory();
LinkHandlerFactory pH = getPlaylistLHFactory();
public final LinkType getLinkTypeByUrl(final String url) throws ParsingException {
final String polishedUrl = Utils.followGoogleRedirectIfNeeded(url);
if (sH != null && sH.acceptUrl(url)) {
final LinkHandlerFactory sH = getStreamLHFactory();
final LinkHandlerFactory cH = getChannelLHFactory();
final LinkHandlerFactory pH = getPlaylistLHFactory();
if (sH != null && sH.acceptUrl(polishedUrl)) {
return LinkType.STREAM;
} else if (cH != null && cH.acceptUrl(url)) {
} else if (cH != null && cH.acceptUrl(polishedUrl)) {
return LinkType.CHANNEL;
} else if (pH != null && pH.acceptUrl(url)) {
} else if (pH != null && pH.acceptUrl(polishedUrl)) {
return LinkType.PLAYLIST;
} else {
return LinkType.NONE;
@ -318,7 +360,8 @@ public abstract class StreamingService {
* the user prefer (using {@link NewPipe#getPreferredLocalization()}), then it will:
* <ul>
* <li>Check if the exactly localization is supported by this service.</li>
* <li>If not, check if a less specific localization is available, using only the language code.</li>
* <li>If not, check if a less specific localization is available, using only the language
* code.</li>
* <li>Fallback to the {@link Localization#DEFAULT default} localization.</li>
* </ul>
*/
@ -331,8 +374,9 @@ public abstract class StreamingService {
}
// Fallback to the first supported language that matches the preferred language
for (Localization supportedLanguage : getSupportedLocalizations()) {
if (supportedLanguage.getLanguageCode().equals(preferredLocalization.getLanguageCode())) {
for (final Localization supportedLanguage : getSupportedLocalizations()) {
if (supportedLanguage.getLanguageCode()
.equals(preferredLocalization.getLanguageCode())) {
return supportedLanguage;
}
}
@ -341,8 +385,8 @@ public abstract class StreamingService {
}
/**
* Returns the country that should be used to fetch content in this service. It will get which country
* the user prefer (using {@link NewPipe#getPreferredContentCountry()}), then it will:
* Returns the country that should be used to fetch content in this service. It will get which
* country the user prefer (using {@link NewPipe#getPreferredContentCountry()}), then it will:
* <ul>
* <li>Check if the country is supported by this service.</li>
* <li>If not, fallback to the {@link ContentCountry#DEFAULT default} country.</li>
@ -359,14 +403,15 @@ public abstract class StreamingService {
}
/**
* Get an instance of the time ago parser using the patterns related to the passed localization.<br>
* <br>
* Just like {@link #getLocalization()}, it will also try to fallback to a less specific localization if
* the exact one is not available/supported.
* Get an instance of the time ago parser using the patterns related to the passed localization.
* <br><br>
* Just like {@link #getLocalization()}, it will also try to fallback to a less specific
* localization if the exact one is not available/supported.
*
* @throws IllegalArgumentException if the localization is not supported (parsing patterns are not present).
* @throws IllegalArgumentException if the localization is not supported (parsing patterns are
* not present).
*/
public TimeAgoParser getTimeAgoParser(Localization localization) {
public TimeAgoParser getTimeAgoParser(final Localization localization) {
final TimeAgoParser targetParser = TimeAgoPatternsManager.getTimeAgoParserFor(localization);
if (targetParser != null) {
@ -374,15 +419,18 @@ public abstract class StreamingService {
}
if (!localization.getCountryCode().isEmpty()) {
final Localization lessSpecificLocalization = new Localization(localization.getLanguageCode());
final TimeAgoParser lessSpecificParser = TimeAgoPatternsManager.getTimeAgoParserFor(lessSpecificLocalization);
final Localization lessSpecificLocalization
= new Localization(localization.getLanguageCode());
final TimeAgoParser lessSpecificParser
= TimeAgoPatternsManager.getTimeAgoParserFor(lessSpecificLocalization);
if (lessSpecificParser != null) {
return lessSpecificParser;
}
}
throw new IllegalArgumentException("Localization is not supported (\"" + localization.toString() + "\")");
throw new IllegalArgumentException(
"Localization is not supported (\"" + localization + "\")");
}
}

View File

@ -1,43 +1,58 @@
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
/*
* Created by Christian Schabesberger on 25.07.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* ChannelExtractor.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ChannelExtractor.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <https://www.gnu.org/licenses/>.
*/
public abstract class ChannelExtractor extends ListExtractor<StreamInfoItem> {
package org.schabi.newpipe.extractor.channel;
public ChannelExtractor(StreamingService service, ListLinkHandler linkHandler) {
import org.schabi.newpipe.extractor.Extractor;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import javax.annotation.Nonnull;
import java.util.List;
public abstract class ChannelExtractor extends Extractor {
public static final long UNKNOWN_SUBSCRIBER_COUNT = -1;
protected ChannelExtractor(final StreamingService service, final ListLinkHandler linkHandler) {
super(service, linkHandler);
}
public abstract String getAvatarUrl() throws ParsingException;
public abstract String getBannerUrl() throws ParsingException;
@Nonnull
public abstract List<Image> getAvatars() throws ParsingException;
@Nonnull
public abstract List<Image> getBanners() throws ParsingException;
public abstract String getFeedUrl() throws ParsingException;
public abstract long getSubscriberCount() throws ParsingException;
public abstract String getDescription() throws ParsingException;
public abstract String getParentChannelName() throws ParsingException;
public abstract String getParentChannelUrl() throws ParsingException;
public abstract String getParentChannelAvatarUrl() throws ParsingException;
@Nonnull
public abstract List<Image> getParentChannelAvatars() throws ParsingException;
public abstract boolean isVerified() throws ParsingException;
@Nonnull
public abstract List<ListLinkHandler> getTabs() throws ParsingException;
@Nonnull
public List<String> getTags() throws ParsingException {
return List.of();
}
}

View File

@ -1,59 +1,60 @@
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.ListExtractor.InfoItemsPage;
import org.schabi.newpipe.extractor.ListInfo;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import org.schabi.newpipe.extractor.utils.ExtractorHelper;
import java.io.IOException;
/*
* Created by Christian Schabesberger on 31.07.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* ChannelInfo.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ChannelInfo.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <https://www.gnu.org/licenses/>.
*/
public class ChannelInfo extends ListInfo<StreamInfoItem> {
package org.schabi.newpipe.extractor.channel;
public ChannelInfo(int serviceId, String id, String url, String originalUrl, String name, ListLinkHandler listLinkHandler) {
super(serviceId, id, url, originalUrl, name, listLinkHandler.getContentFilters(), listLinkHandler.getSortFilter());
import org.schabi.newpipe.extractor.Info;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import java.io.IOException;
import java.util.List;
import javax.annotation.Nonnull;
public class ChannelInfo extends Info {
public ChannelInfo(final int serviceId,
final String id,
final String url,
final String originalUrl,
final String name) {
super(serviceId, id, url, originalUrl, name);
}
public static ChannelInfo getInfo(String url) throws IOException, ExtractionException {
public static ChannelInfo getInfo(final String url) throws IOException, ExtractionException {
return getInfo(NewPipe.getServiceByUrl(url), url);
}
public static ChannelInfo getInfo(StreamingService service, String url) throws IOException, ExtractionException {
ChannelExtractor extractor = service.getChannelExtractor(url);
public static ChannelInfo getInfo(final StreamingService service, final String url)
throws IOException, ExtractionException {
final ChannelExtractor extractor = service.getChannelExtractor(url);
extractor.fetchPage();
return getInfo(extractor);
}
public static InfoItemsPage<StreamInfoItem> getMoreItems(StreamingService service,
String url,
String pageUrl) throws IOException, ExtractionException {
return service.getChannelExtractor(url).getPage(pageUrl);
}
public static ChannelInfo getInfo(ChannelExtractor extractor) throws IOException, ExtractionException {
public static ChannelInfo getInfo(final ChannelExtractor extractor)
throws IOException, ExtractionException {
final int serviceId = extractor.getServiceId();
final String id = extractor.getId();
@ -61,75 +62,98 @@ public class ChannelInfo extends ListInfo<StreamInfoItem> {
final String originalUrl = extractor.getOriginalUrl();
final String name = extractor.getName();
final ChannelInfo info = new ChannelInfo(serviceId, id, url, originalUrl, name, extractor.getLinkHandler());
final ChannelInfo info = new ChannelInfo(serviceId, id, url, originalUrl, name);
try {
info.setAvatarUrl(extractor.getAvatarUrl());
} catch (Exception e) {
info.setAvatars(extractor.getAvatars());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setBannerUrl(extractor.getBannerUrl());
} catch (Exception e) {
info.setBanners(extractor.getBanners());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setFeedUrl(extractor.getFeedUrl());
} catch (Exception e) {
} catch (final Exception e) {
info.addError(e);
}
final InfoItemsPage<StreamInfoItem> itemsPage = ExtractorHelper.getItemsPageOrLogError(info, extractor);
info.setRelatedItems(itemsPage.getItems());
info.setNextPageUrl(itemsPage.getNextPageUrl());
try {
info.setSubscriberCount(extractor.getSubscriberCount());
} catch (Exception e) {
} catch (final Exception e) {
info.addError(e);
}
try {
info.setDescription(extractor.getDescription());
} catch (Exception e) {
} catch (final Exception e) {
info.addError(e);
}
try {
info.setParentChannelName(extractor.getParentChannelName());
} catch (Exception e) {
} catch (final Exception e) {
info.addError(e);
}
try {
info.setParentChannelUrl(extractor.getParentChannelUrl());
} catch (Exception e) {
} catch (final Exception e) {
info.addError(e);
}
try {
info.setParentChannelAvatarUrl(extractor.getParentChannelAvatarUrl());
} catch (Exception e) {
info.setParentChannelAvatars(extractor.getParentChannelAvatars());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setVerified(extractor.isVerified());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setTabs(extractor.getTabs());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setTags(extractor.getTags());
} catch (final Exception e) {
info.addError(e);
}
return info;
}
private String avatarUrl;
private String parentChannelName;
private String parentChannelUrl;
private String parentChannelAvatarUrl;
private String bannerUrl;
private String feedUrl;
private long subscriberCount = -1;
private String description;
private String[] donationLinks;
@Nonnull
private List<Image> avatars = List.of();
@Nonnull
private List<Image> banners = List.of();
@Nonnull
private List<Image> parentChannelAvatars = List.of();
private boolean verified;
private List<ListLinkHandler> tabs = List.of();
private List<String> tags = List.of();
public String getParentChannelName() {
return parentChannelName;
}
public void setParentChannelName(String parentChannelName) {
public void setParentChannelName(final String parentChannelName) {
this.parentChannelName = parentChannelName;
}
@ -137,39 +161,42 @@ public class ChannelInfo extends ListInfo<StreamInfoItem> {
return parentChannelUrl;
}
public void setParentChannelUrl(String parentChannelUrl) {
public void setParentChannelUrl(final String parentChannelUrl) {
this.parentChannelUrl = parentChannelUrl;
}
public String getParentChannelAvatarUrl() {
return parentChannelAvatarUrl;
@Nonnull
public List<Image> getParentChannelAvatars() {
return parentChannelAvatars;
}
public void setParentChannelAvatarUrl(String parentChannelAvatarUrl) {
this.parentChannelAvatarUrl = parentChannelAvatarUrl;
public void setParentChannelAvatars(@Nonnull final List<Image> parentChannelAvatars) {
this.parentChannelAvatars = parentChannelAvatars;
}
public String getAvatarUrl() {
return avatarUrl;
@Nonnull
public List<Image> getAvatars() {
return avatars;
}
public void setAvatarUrl(String avatarUrl) {
this.avatarUrl = avatarUrl;
public void setAvatars(@Nonnull final List<Image> avatars) {
this.avatars = avatars;
}
public String getBannerUrl() {
return bannerUrl;
@Nonnull
public List<Image> getBanners() {
return banners;
}
public void setBannerUrl(String bannerUrl) {
this.bannerUrl = bannerUrl;
public void setBanners(@Nonnull final List<Image> banners) {
this.banners = banners;
}
public String getFeedUrl() {
return feedUrl;
}
public void setFeedUrl(String feedUrl) {
public void setFeedUrl(final String feedUrl) {
this.feedUrl = feedUrl;
}
@ -177,7 +204,7 @@ public class ChannelInfo extends ListInfo<StreamInfoItem> {
return subscriberCount;
}
public void setSubscriberCount(long subscriberCount) {
public void setSubscriberCount(final long subscriberCount) {
this.subscriberCount = subscriberCount;
}
@ -185,7 +212,7 @@ public class ChannelInfo extends ListInfo<StreamInfoItem> {
return description;
}
public void setDescription(String description) {
public void setDescription(final String description) {
this.description = description;
}
@ -193,7 +220,33 @@ public class ChannelInfo extends ListInfo<StreamInfoItem> {
return donationLinks;
}
public void setDonationLinks(String[] donationLinks) {
public void setDonationLinks(final String[] donationLinks) {
this.donationLinks = donationLinks;
}
public boolean isVerified() {
return verified;
}
public void setVerified(final boolean verified) {
this.verified = verified;
}
@Nonnull
public List<ListLinkHandler> getTabs() {
return tabs;
}
public void setTabs(@Nonnull final List<ListLinkHandler> tabs) {
this.tabs = tabs;
}
@Nonnull
public List<String> getTags() {
return tags;
}
public void setTags(@Nonnull final List<String> tags) {
this.tags = tags;
}
}

View File

@ -5,21 +5,21 @@ import org.schabi.newpipe.extractor.InfoItem;
/*
* Created by Christian Schabesberger on 11.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* ChannelInfoItem.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ChannelInfoItem.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public class ChannelInfoItem extends InfoItem {
@ -27,9 +27,9 @@ public class ChannelInfoItem extends InfoItem {
private String description;
private long subscriberCount = -1;
private long streamCount = -1;
private boolean verified = false;
public ChannelInfoItem(int serviceId, String url, String name) {
public ChannelInfoItem(final int serviceId, final String url, final String name) {
super(InfoType.CHANNEL, serviceId, url, name);
}
@ -37,7 +37,7 @@ public class ChannelInfoItem extends InfoItem {
return description;
}
public void setDescription(String description) {
public void setDescription(final String description) {
this.description = description;
}
@ -45,15 +45,23 @@ public class ChannelInfoItem extends InfoItem {
return subscriberCount;
}
public void setSubscriberCount(long subscriber_count) {
this.subscriberCount = subscriber_count;
public void setSubscriberCount(final long subscriberCount) {
this.subscriberCount = subscriberCount;
}
public long getStreamCount() {
return streamCount;
}
public void setStreamCount(long stream_count) {
this.streamCount = stream_count;
public void setStreamCount(final long streamCount) {
this.streamCount = streamCount;
}
public boolean isVerified() {
return verified;
}
public void setVerified(final boolean verified) {
this.verified = verified;
}
}

View File

@ -6,26 +6,29 @@ import org.schabi.newpipe.extractor.exceptions.ParsingException;
/*
* Created by Christian Schabesberger on 12.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* ChannelInfoItemExtractor.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ChannelInfoItemExtractor.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public interface ChannelInfoItemExtractor extends InfoItemExtractor {
String getDescription() throws ParsingException;
long getSubscriberCount() throws ParsingException;
long getStreamCount() throws ParsingException;
boolean isVerified() throws ParsingException;
}

View File

@ -1,64 +1,67 @@
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.InfoItemsCollector;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
/*
* Created by Christian Schabesberger on 12.02.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* ChannelInfoItemsCollector.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ChannelInfoItemsCollector.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <https://www.gnu.org/licenses/>.
*/
public class ChannelInfoItemsCollector extends InfoItemsCollector<ChannelInfoItem, ChannelInfoItemExtractor> {
public ChannelInfoItemsCollector(int serviceId) {
package org.schabi.newpipe.extractor.channel;
import org.schabi.newpipe.extractor.InfoItemsCollector;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
public final class ChannelInfoItemsCollector
extends InfoItemsCollector<ChannelInfoItem, ChannelInfoItemExtractor> {
public ChannelInfoItemsCollector(final int serviceId) {
super(serviceId);
}
@Override
public ChannelInfoItem extract(ChannelInfoItemExtractor extractor) throws ParsingException {
// important information
int serviceId = getServiceId();
String name = extractor.getName();
String url = extractor.getUrl();
ChannelInfoItem resultItem = new ChannelInfoItem(serviceId, url, name);
public ChannelInfoItem extract(final ChannelInfoItemExtractor extractor)
throws ParsingException {
final ChannelInfoItem resultItem = new ChannelInfoItem(
getServiceId(), extractor.getUrl(), extractor.getName());
// optional information
try {
resultItem.setSubscriberCount(extractor.getSubscriberCount());
} catch (Exception e) {
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setStreamCount(extractor.getStreamCount());
} catch (Exception e) {
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setThumbnailUrl(extractor.getThumbnailUrl());
} catch (Exception e) {
resultItem.setThumbnails(extractor.getThumbnails());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setDescription(extractor.getDescription());
} catch (Exception e) {
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setVerified(extractor.isVerified());
} catch (final Exception e) {
addError(e);
}
return resultItem;
}
}

View File

@ -0,0 +1,25 @@
package org.schabi.newpipe.extractor.channel.tabs;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import javax.annotation.Nonnull;
/**
* A {@link ListExtractor} of {@link InfoItem}s for tabs of channels.
*/
public abstract class ChannelTabExtractor extends ListExtractor<InfoItem> {
protected ChannelTabExtractor(@Nonnull final StreamingService service,
@Nonnull final ListLinkHandler linkHandler) {
super(service, linkHandler);
}
@Nonnull
@Override
public String getName() {
return getLinkHandler().getContentFilters().get(0);
}
}

View File

@ -0,0 +1,70 @@
package org.schabi.newpipe.extractor.channel.tabs;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.ListInfo;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.ChannelInfo;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.utils.ExtractorHelper;
import javax.annotation.Nonnull;
import java.io.IOException;
public class ChannelTabInfo extends ListInfo<InfoItem> {
public ChannelTabInfo(final int serviceId,
@Nonnull final ListLinkHandler linkHandler) {
super(serviceId, linkHandler, linkHandler.getContentFilters().get(0));
}
/**
* Get a {@link ChannelTabInfo} instance from the given service and tab handler.
*
* @param service streaming service
* @param linkHandler Channel tab handler (from {@link ChannelInfo})
* @return the extracted {@link ChannelTabInfo}
*/
@Nonnull
public static ChannelTabInfo getInfo(@Nonnull final StreamingService service,
@Nonnull final ListLinkHandler linkHandler)
throws ExtractionException, IOException {
final ChannelTabExtractor extractor = service.getChannelTabExtractor(linkHandler);
extractor.fetchPage();
return getInfo(extractor);
}
/**
* Get a {@link ChannelTabInfo} instance from a {@link ChannelTabExtractor}.
*
* @param extractor an extractor where {@code fetchPage()} was already got called on
* @return the extracted {@link ChannelTabInfo}
*/
@Nonnull
public static ChannelTabInfo getInfo(@Nonnull final ChannelTabExtractor extractor) {
final ChannelTabInfo info =
new ChannelTabInfo(extractor.getServiceId(), extractor.getLinkHandler());
try {
info.setOriginalUrl(extractor.getOriginalUrl());
} catch (final Exception e) {
info.addError(e);
}
final ListExtractor.InfoItemsPage<InfoItem> page
= ExtractorHelper.getItemsPageOrLogError(info, extractor);
info.setRelatedItems(page.getItems());
info.setNextPage(page.getNextPage());
return info;
}
public static ListExtractor.InfoItemsPage<InfoItem> getMoreItems(
@Nonnull final StreamingService service,
@Nonnull final ListLinkHandler linkHandler,
@Nonnull final Page page) throws ExtractionException, IOException {
return service.getChannelTabExtractor(linkHandler).getPage(page);
}
}

View File

@ -0,0 +1,18 @@
package org.schabi.newpipe.extractor.channel.tabs;
/**
* Constants of channel tabs supported by the extractor.
*/
public final class ChannelTabs {
public static final String VIDEOS = "videos";
public static final String TRACKS = "tracks";
public static final String SHORTS = "shorts";
public static final String LIVESTREAMS = "livestreams";
public static final String CHANNELS = "channels";
public static final String PLAYLISTS = "playlists";
public static final String ALBUMS = "albums";
public static final String LIKES = "likes";
private ChannelTabs() {
}
}

View File

@ -2,6 +2,7 @@ package org.schabi.newpipe.extractor.comments;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
@ -9,9 +10,23 @@ import javax.annotation.Nonnull;
public abstract class CommentsExtractor extends ListExtractor<CommentsInfoItem> {
public CommentsExtractor(StreamingService service, ListLinkHandler uiHandler) {
public CommentsExtractor(final StreamingService service, final ListLinkHandler uiHandler) {
super(service, uiHandler);
// TODO Auto-generated constructor stub
}
/**
* @apiNote Warning: This method is experimental and may get removed in a future release.
* @return <code>true</code> if the comments are disabled otherwise <code>false</code> (default)
*/
public boolean isCommentsDisabled() throws ExtractionException {
return false;
}
/**
* @return the total number of comments
*/
public int getCommentsCount() throws ExtractionException {
return -1;
}
@Nonnull

View File

@ -3,6 +3,7 @@ package org.schabi.newpipe.extractor.comments;
import org.schabi.newpipe.extractor.ListExtractor.InfoItemsPage;
import org.schabi.newpipe.extractor.ListInfo;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
@ -10,62 +11,116 @@ import org.schabi.newpipe.extractor.utils.ExtractorHelper;
import java.io.IOException;
public class CommentsInfo extends ListInfo<CommentsInfoItem> {
public final class CommentsInfo extends ListInfo<CommentsInfoItem> {
private CommentsInfo(int serviceId, ListLinkHandler listUrlIdHandler, String name) {
private CommentsInfo(
final int serviceId,
final ListLinkHandler listUrlIdHandler,
final String name) {
super(serviceId, listUrlIdHandler, name);
}
public static CommentsInfo getInfo(String url) throws IOException, ExtractionException {
public static CommentsInfo getInfo(final String url) throws IOException, ExtractionException {
return getInfo(NewPipe.getServiceByUrl(url), url);
}
public static CommentsInfo getInfo(StreamingService serviceByUrl, String url) throws ExtractionException, IOException {
return getInfo(serviceByUrl.getCommentsExtractor(url));
public static CommentsInfo getInfo(final StreamingService service, final String url)
throws ExtractionException, IOException {
return getInfo(service.getCommentsExtractor(url));
}
private static CommentsInfo getInfo(CommentsExtractor commentsExtractor) throws IOException, ExtractionException {
public static CommentsInfo getInfo(final CommentsExtractor commentsExtractor)
throws IOException, ExtractionException {
// for services which do not have a comments extractor
if (null == commentsExtractor) {
if (commentsExtractor == null) {
return null;
}
commentsExtractor.fetchPage();
String name = commentsExtractor.getName();
int serviceId = commentsExtractor.getServiceId();
ListLinkHandler listUrlIdHandler = commentsExtractor.getLinkHandler();
CommentsInfo commentsInfo = new CommentsInfo(serviceId, listUrlIdHandler, name);
final String name = commentsExtractor.getName();
final int serviceId = commentsExtractor.getServiceId();
final ListLinkHandler listUrlIdHandler = commentsExtractor.getLinkHandler();
final CommentsInfo commentsInfo = new CommentsInfo(serviceId, listUrlIdHandler, name);
commentsInfo.setCommentsExtractor(commentsExtractor);
InfoItemsPage<CommentsInfoItem> initialCommentsPage = ExtractorHelper.getItemsPageOrLogError(commentsInfo,
commentsExtractor);
final InfoItemsPage<CommentsInfoItem> initialCommentsPage =
ExtractorHelper.getItemsPageOrLogError(commentsInfo, commentsExtractor);
commentsInfo.setCommentsDisabled(commentsExtractor.isCommentsDisabled());
commentsInfo.setRelatedItems(initialCommentsPage.getItems());
commentsInfo.setNextPageUrl(initialCommentsPage.getNextPageUrl());
try {
commentsInfo.setCommentsCount(commentsExtractor.getCommentsCount());
} catch (final Exception e) {
commentsInfo.addError(e);
}
commentsInfo.setNextPage(initialCommentsPage.getNextPage());
return commentsInfo;
}
public static InfoItemsPage<CommentsInfoItem> getMoreItems(CommentsInfo commentsInfo, String pageUrl)
throws ExtractionException, IOException {
return getMoreItems(NewPipe.getService(commentsInfo.getServiceId()), commentsInfo, pageUrl);
public static InfoItemsPage<CommentsInfoItem> getMoreItems(
final CommentsInfo commentsInfo,
final Page page) throws ExtractionException, IOException {
return getMoreItems(NewPipe.getService(commentsInfo.getServiceId()), commentsInfo.getUrl(),
page);
}
public static InfoItemsPage<CommentsInfoItem> getMoreItems(StreamingService service, CommentsInfo commentsInfo,
String pageUrl) throws IOException, ExtractionException {
if (null == commentsInfo.getCommentsExtractor()) {
commentsInfo.setCommentsExtractor(service.getCommentsExtractor(commentsInfo.getUrl()));
commentsInfo.getCommentsExtractor().fetchPage();
}
return commentsInfo.getCommentsExtractor().getPage(pageUrl);
public static InfoItemsPage<CommentsInfoItem> getMoreItems(
final StreamingService service,
final CommentsInfo commentsInfo,
final Page page) throws IOException, ExtractionException {
return getMoreItems(service, commentsInfo.getUrl(), page);
}
public static InfoItemsPage<CommentsInfoItem> getMoreItems(
final StreamingService service,
final String url,
final Page page) throws IOException, ExtractionException {
return service.getCommentsExtractor(url).getPage(page);
}
private transient CommentsExtractor commentsExtractor;
private boolean commentsDisabled = false;
private int commentsCount;
public CommentsExtractor getCommentsExtractor() {
return commentsExtractor;
}
public void setCommentsExtractor(CommentsExtractor commentsExtractor) {
public void setCommentsExtractor(final CommentsExtractor commentsExtractor) {
this.commentsExtractor = commentsExtractor;
}
/**
* @return {@code true} if the comments are disabled otherwise {@code false} (default)
* @see CommentsExtractor#isCommentsDisabled()
*/
public boolean isCommentsDisabled() {
return commentsDisabled;
}
/**
* @param commentsDisabled {@code true} if the comments are disabled otherwise {@code false}
*/
public void setCommentsDisabled(final boolean commentsDisabled) {
this.commentsDisabled = commentsDisabled;
}
/**
* Returns the total number of comments.
*
* @return the total number of comments
*/
public int getCommentsCount() {
return commentsCount;
}
/**
* Sets the total number of comments.
*
* @param commentsCount the commentsCount to set.
*/
public void setCommentsCount(final int commentsCount) {
this.commentsCount = commentsCount;
}
}

View File

@ -1,23 +1,45 @@
package org.schabi.newpipe.extractor.comments;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.localization.DateWrapper;
import org.schabi.newpipe.extractor.stream.Description;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.List;
public class CommentsInfoItem extends InfoItem {
private String commentId;
private String commentText;
@Nonnull
private Description commentText = Description.EMPTY_DESCRIPTION;
private String uploaderName;
private String uploaderAvatarUrl;
@Nonnull
private List<Image> uploaderAvatars = List.of();
private String uploaderUrl;
private boolean uploaderVerified;
private String textualUploadDate;
@Nullable
private DateWrapper uploadDate;
private int likeCount;
private String textualLikeCount;
private boolean heartedByUploader;
private boolean pinned;
private int streamPosition;
private int replyCount;
@Nullable
private Page replies;
private boolean isChannelOwner;
private boolean creatorReply;
public CommentsInfoItem(int serviceId, String url, String name) {
public static final int NO_LIKE_COUNT = -1;
public static final int NO_STREAM_POSITION = -1;
public static final int UNKNOWN_REPLY_COUNT = -1;
public CommentsInfoItem(final int serviceId, final String url, final String name) {
super(InfoType.COMMENT, serviceId, url, name);
}
@ -25,15 +47,16 @@ public class CommentsInfoItem extends InfoItem {
return commentId;
}
public void setCommentId(String commentId) {
public void setCommentId(final String commentId) {
this.commentId = commentId;
}
public String getCommentText() {
@Nonnull
public Description getCommentText() {
return commentText;
}
public void setCommentText(String commentText) {
public void setCommentText(@Nonnull final Description commentText) {
this.commentText = commentText;
}
@ -41,23 +64,24 @@ public class CommentsInfoItem extends InfoItem {
return uploaderName;
}
public void setUploaderName(String uploaderName) {
public void setUploaderName(final String uploaderName) {
this.uploaderName = uploaderName;
}
public String getUploaderAvatarUrl() {
return uploaderAvatarUrl;
@Nonnull
public List<Image> getUploaderAvatars() {
return uploaderAvatars;
}
public void setUploaderAvatarUrl(String uploaderAvatarUrl) {
this.uploaderAvatarUrl = uploaderAvatarUrl;
public void setUploaderAvatars(@Nonnull final List<Image> uploaderAvatars) {
this.uploaderAvatars = uploaderAvatars;
}
public String getUploaderUrl() {
return uploaderUrl;
}
public void setUploaderUrl(String uploaderUrl) {
public void setUploaderUrl(final String uploaderUrl) {
this.uploaderUrl = uploaderUrl;
}
@ -65,7 +89,7 @@ public class CommentsInfoItem extends InfoItem {
return textualUploadDate;
}
public void setTextualUploadDate(String textualUploadDate) {
public void setTextualUploadDate(final String textualUploadDate) {
this.textualUploadDate = textualUploadDate;
}
@ -74,15 +98,100 @@ public class CommentsInfoItem extends InfoItem {
return uploadDate;
}
public void setUploadDate(@Nullable DateWrapper uploadDate) {
public void setUploadDate(@Nullable final DateWrapper uploadDate) {
this.uploadDate = uploadDate;
}
/**
* @return the comment's like count or {@link CommentsInfoItem#NO_LIKE_COUNT} if it is
* unavailable
*/
public int getLikeCount() {
return likeCount;
}
public void setLikeCount(int likeCount) {
public void setLikeCount(final int likeCount) {
this.likeCount = likeCount;
}
public String getTextualLikeCount() {
return textualLikeCount;
}
public void setTextualLikeCount(final String textualLikeCount) {
this.textualLikeCount = textualLikeCount;
}
public void setHeartedByUploader(final boolean isHeartedByUploader) {
this.heartedByUploader = isHeartedByUploader;
}
public boolean isHeartedByUploader() {
return this.heartedByUploader;
}
public boolean isPinned() {
return pinned;
}
public void setPinned(final boolean pinned) {
this.pinned = pinned;
}
public void setUploaderVerified(final boolean uploaderVerified) {
this.uploaderVerified = uploaderVerified;
}
public boolean isUploaderVerified() {
return uploaderVerified;
}
public void setStreamPosition(final int streamPosition) {
this.streamPosition = streamPosition;
}
/**
* Get the playback position of the stream to which this comment belongs.
* This is not supported by all services.
*
* @return the playback position in seconds or {@link #NO_STREAM_POSITION} if not available
*/
public int getStreamPosition() {
return streamPosition;
}
public void setReplyCount(final int replyCount) {
this.replyCount = replyCount;
}
public int getReplyCount() {
return replyCount;
}
public void setReplies(@Nullable final Page replies) {
this.replies = replies;
}
@Nullable
public Page getReplies() {
return this.replies;
}
public void setChannelOwner(final boolean channelOwner) {
this.isChannelOwner = channelOwner;
}
public boolean isChannelOwner() {
return isChannelOwner;
}
public void setCreatorReply(final boolean creatorReply) {
this.creatorReply = creatorReply;
}
public boolean hasCreatorReply() {
return creatorReply;
}
}

View File

@ -1,43 +1,152 @@
package org.schabi.newpipe.extractor.comments;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.InfoItemExtractor;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.localization.DateWrapper;
import org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeCommentsInfoItemExtractor;
import org.schabi.newpipe.extractor.stream.Description;
import org.schabi.newpipe.extractor.stream.StreamExtractor;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.List;
public interface CommentsInfoItemExtractor extends InfoItemExtractor {
/**
* Return the like count of the comment, or -1 if it's unavailable
* Return the like count of the comment,
* or {@link CommentsInfoItem#NO_LIKE_COUNT} if it is unavailable.
*
* <br>
* <p>
* NOTE: Currently only implemented for YT {@link
* YoutubeCommentsInfoItemExtractor#getLikeCount()}
* with limitations (only approximate like count is returned)
*
* @return the comment's like count
* or {@link CommentsInfoItem#NO_LIKE_COUNT} if it is unavailable
* @see StreamExtractor#getLikeCount()
*/
int getLikeCount() throws ParsingException;
default int getLikeCount() throws ParsingException {
return CommentsInfoItem.NO_LIKE_COUNT;
}
/**
* The unmodified like count given by the service
* <br>
* It may be language dependent
*/
default String getTextualLikeCount() throws ParsingException {
return "";
}
/**
* The text of the comment
*/
String getCommentText() throws ParsingException;
@Nonnull
default Description getCommentText() throws ParsingException {
return Description.EMPTY_DESCRIPTION;
}
/**
* The upload date given by the service, unmodified
*
* @see StreamExtractor#getTextualUploadDate()
*/
String getTextualUploadDate() throws ParsingException;
default String getTextualUploadDate() throws ParsingException {
return "";
}
/**
* The upload date wrapped with DateWrapper class
*
* @see StreamExtractor#getUploadDate()
*/
@Nullable
DateWrapper getUploadDate() throws ParsingException;
default DateWrapper getUploadDate() throws ParsingException {
return null;
}
String getCommentId() throws ParsingException;
default String getCommentId() throws ParsingException {
return "";
}
String getUploaderUrl() throws ParsingException;
default String getUploaderUrl() throws ParsingException {
return "";
}
String getUploaderName() throws ParsingException;
default String getUploaderName() throws ParsingException {
return "";
}
String getUploaderAvatarUrl() throws ParsingException;
@Nonnull
default List<Image> getUploaderAvatars() throws ParsingException {
return List.of();
}
/**
* Whether the comment has been hearted by the uploader
*/
default boolean isHeartedByUploader() throws ParsingException {
return false;
}
/**
* Whether the comment is pinned
*/
default boolean isPinned() throws ParsingException {
return false;
}
/**
* Whether the uploader is verified by the service
*/
default boolean isUploaderVerified() throws ParsingException {
return false;
}
/**
* The playback position of the stream to which this comment belongs.
*
* @see CommentsInfoItem#getStreamPosition()
*/
default int getStreamPosition() throws ParsingException {
return CommentsInfoItem.NO_STREAM_POSITION;
}
/**
* The count of comment replies.
*
* @return the count of the replies
* or {@link CommentsInfoItem#UNKNOWN_REPLY_COUNT} if replies are not supported
*/
default int getReplyCount() throws ParsingException {
return CommentsInfoItem.UNKNOWN_REPLY_COUNT;
}
/**
* The continuation page which is used to get comment replies from.
*
* @return the continuation Page for the replies, or null if replies are not supported
*/
@Nullable
default Page getReplies() throws ParsingException {
return null;
}
/**
* Whether the comment was made by the channel owner.
*/
default boolean isChannelOwner() throws ParsingException {
return false;
}
/**
* Whether the comment was replied to by the creator.
*/
default boolean hasCreatorReply() throws ParsingException {
return false;
}
}

View File

@ -1,94 +1,133 @@
package org.schabi.newpipe.extractor.comments;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.InfoItemsCollector;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import java.util.ArrayList;
import java.util.List;
import java.util.Vector;
public class CommentsInfoItemsCollector extends InfoItemsCollector<CommentsInfoItem, CommentsInfoItemExtractor> {
public final class CommentsInfoItemsCollector
extends InfoItemsCollector<CommentsInfoItem, CommentsInfoItemExtractor> {
public CommentsInfoItemsCollector(int serviceId) {
public CommentsInfoItemsCollector(final int serviceId) {
super(serviceId);
}
@Override
public CommentsInfoItem extract(CommentsInfoItemExtractor extractor) throws ParsingException {
// important information
int serviceId = getServiceId();
String url = extractor.getUrl();
String name = extractor.getName();
CommentsInfoItem resultItem = new CommentsInfoItem(serviceId, url, name);
public CommentsInfoItem extract(final CommentsInfoItemExtractor extractor)
throws ParsingException {
final CommentsInfoItem resultItem = new CommentsInfoItem(
getServiceId(), extractor.getUrl(), extractor.getName());
// optional information
try {
resultItem.setCommentId(extractor.getCommentId());
} catch (Exception e) {
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setCommentText(extractor.getCommentText());
} catch (Exception e) {
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setUploaderName(extractor.getUploaderName());
} catch (Exception e) {
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setUploaderAvatarUrl(extractor.getUploaderAvatarUrl());
} catch (Exception e) {
resultItem.setUploaderAvatars(extractor.getUploaderAvatars());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setUploaderUrl(extractor.getUploaderUrl());
} catch (Exception e) {
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setTextualUploadDate(extractor.getTextualUploadDate());
} catch (Exception e) {
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setUploadDate(extractor.getUploadDate());
} catch (Exception e) {
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setLikeCount(extractor.getLikeCount());
} catch (Exception e) {
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setThumbnailUrl(extractor.getThumbnailUrl());
} catch (Exception e) {
resultItem.setTextualLikeCount(extractor.getTextualLikeCount());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setThumbnails(extractor.getThumbnails());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setHeartedByUploader(extractor.isHeartedByUploader());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setPinned(extractor.isPinned());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setStreamPosition(extractor.getStreamPosition());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setReplyCount(extractor.getReplyCount());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setReplies(extractor.getReplies());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setChannelOwner(extractor.isChannelOwner());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setCreatorReply(extractor.hasCreatorReply());
} catch (final Exception e) {
addError(e);
}
return resultItem;
}
@Override
public void commit(CommentsInfoItemExtractor extractor) {
public void commit(final CommentsInfoItemExtractor extractor) {
try {
addItem(extract(extractor));
} catch (Exception e) {
} catch (final Exception e) {
addError(e);
}
}
public List<CommentsInfoItem> getCommentsInfoItemList() {
List<CommentsInfoItem> siiList = new Vector<>();
for (InfoItem ii : super.getItems()) {
if (ii instanceof CommentsInfoItem) {
siiList.add((CommentsInfoItem) ii);
}
}
return siiList;
return new ArrayList<>(super.getItems());
}
}

View File

@ -7,6 +7,8 @@ import org.schabi.newpipe.extractor.localization.Localization;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.io.IOException;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
@ -19,13 +21,14 @@ public abstract class Downloader {
/**
* Do a GET request to get the resource that the url is pointing to.<br>
* <br>
* This method calls {@link #get(String, Map, Localization)} with the default preferred localization. It should only be
* used when the resource that will be fetched won't be affected by the localization.
* This method calls {@link #get(String, Map, Localization)} with the default preferred
* localization. It should only be used when the resource that will be fetched won't be affected
* by the localization.
*
* @param url the URL that is pointing to the wanted resource
* @return the result of the GET request
*/
public Response get(String url) throws IOException, ReCaptchaException {
public Response get(final String url) throws IOException, ReCaptchaException {
return get(url, null, NewPipe.getPreferredLocalization());
}
@ -38,7 +41,8 @@ public abstract class Downloader {
* @param localization the source of the value of the {@code Accept-Language} header
* @return the result of the GET request
*/
public Response get(String url, @Nullable Localization localization) throws IOException, ReCaptchaException {
public Response get(final String url, final Localization localization)
throws IOException, ReCaptchaException {
return get(url, null, localization);
}
@ -50,7 +54,8 @@ public abstract class Downloader {
* Any default headers <b>should</b> be overridden by these.
* @return the result of the GET request
*/
public Response get(String url, @Nullable Map<String, List<String>> headers) throws IOException, ReCaptchaException {
public Response get(final String url, @Nullable final Map<String, List<String>> headers)
throws IOException, ReCaptchaException {
return get(url, headers, NewPipe.getPreferredLocalization());
}
@ -65,7 +70,9 @@ public abstract class Downloader {
* @param localization the source of the value of the {@code Accept-Language} header
* @return the result of the GET request
*/
public Response get(String url, @Nullable Map<String, List<String>> headers, @Nullable Localization localization)
public Response get(final String url,
@Nullable final Map<String, List<String>> headers,
final Localization localization)
throws IOException, ReCaptchaException {
return execute(Request.newBuilder()
.get(url)
@ -80,7 +87,7 @@ public abstract class Downloader {
* @param url the URL that is pointing to the wanted resource
* @return the result of the HEAD request
*/
public Response head(String url) throws IOException, ReCaptchaException {
public Response head(final String url) throws IOException, ReCaptchaException {
return head(url, null);
}
@ -92,7 +99,7 @@ public abstract class Downloader {
* Any default headers <b>should</b> be overridden by these.
* @return the result of the HEAD request
*/
public Response head(String url, @Nullable Map<String, List<String>> headers)
public Response head(final String url, @Nullable final Map<String, List<String>> headers)
throws IOException, ReCaptchaException {
return execute(Request.newBuilder()
.head(url)
@ -107,9 +114,11 @@ public abstract class Downloader {
* @param headers a list of headers that will be used in the request.
* Any default headers <b>should</b> be overridden by these.
* @param dataToSend byte array that will be sent when doing the request.
* @return the result of the GET request
* @return the result of the POST request
*/
public Response post(String url, @Nullable Map<String, List<String>> headers, @Nullable byte[] dataToSend)
public Response post(final String url,
@Nullable final Map<String, List<String>> headers,
@Nullable final byte[] dataToSend)
throws IOException, ReCaptchaException {
return post(url, headers, dataToSend, NewPipe.getPreferredLocalization());
}
@ -124,9 +133,12 @@ public abstract class Downloader {
* Any default headers <b>should</b> be overridden by these.
* @param dataToSend byte array that will be sent when doing the request.
* @param localization the source of the value of the {@code Accept-Language} header
* @return the result of the GET request
* @return the result of the POST request
*/
public Response post(String url, @Nullable Map<String, List<String>> headers, @Nullable byte[] dataToSend, @Nullable Localization localization)
public Response post(final String url,
@Nullable final Map<String, List<String>> headers,
@Nullable final byte[] dataToSend,
final Localization localization)
throws IOException, ReCaptchaException {
return execute(Request.newBuilder()
.post(url, dataToSend)
@ -135,10 +147,100 @@ public abstract class Downloader {
.build());
}
/**
* Convenient method to send a POST request using the specified value of the
* {@code Content-Type} header with a given {@link Localization}.
*
* @param url the URL that is pointing to the wanted resource
* @param headers a list of headers that will be used in the request.
* Any default headers <b>should</b> be overridden by these.
* @param dataToSend byte array that will be sent when doing the request.
* @param localization the source of the value of the {@code Accept-Language} header
* @param contentType the mime type of the body sent, which will be set as the value of the
* {@code Content-Type} header
* @return the result of the POST request
* @see #post(String, Map, byte[], Localization)
*/
public Response postWithContentType(final String url,
@Nullable final Map<String, List<String>> headers,
@Nullable final byte[] dataToSend,
final Localization localization,
final String contentType)
throws IOException, ReCaptchaException {
final Map<String, List<String>> actualHeaders = new HashMap<>();
if (headers != null) {
actualHeaders.putAll(headers);
}
actualHeaders.put("Content-Type", Collections.singletonList(contentType));
return post(url, actualHeaders, dataToSend, localization);
}
/**
* Convenient method to send a POST request using the specified value of the
* {@code Content-Type} header.
*
* @param url the URL that is pointing to the wanted resource
* @param headers a list of headers that will be used in the request.
* Any default headers <b>should</b> be overridden by these.
* @param dataToSend byte array that will be sent when doing the request.
* @param contentType the mime type of the body sent, which will be set as the value of the
* {@code Content-Type} header
* @return the result of the POST request
* @see #post(String, Map, byte[], Localization)
*/
public Response postWithContentType(final String url,
@Nullable final Map<String, List<String>> headers,
@Nullable final byte[] dataToSend,
final String contentType)
throws IOException, ReCaptchaException {
return postWithContentType(url, headers, dataToSend, NewPipe.getPreferredLocalization(),
contentType);
}
/**
* Convenient method to send a POST request the JSON mime type as the value of the
* {@code Content-Type} header with a given {@link Localization}.
*
* @param url the URL that is pointing to the wanted resource
* @param headers a list of headers that will be used in the request.
* Any default headers <b>should</b> be overridden by these.
* @param dataToSend byte array that will be sent when doing the request.
* @param localization the source of the value of the {@code Accept-Language} header
* @return the result of the POST request
* @see #post(String, Map, byte[], Localization)
*/
public Response postWithContentTypeJson(final String url,
@Nullable final Map<String, List<String>> headers,
@Nullable final byte[] dataToSend,
final Localization localization)
throws IOException, ReCaptchaException {
return postWithContentType(url, headers, dataToSend, localization, "application/json");
}
/**
* Convenient method to send a POST request the JSON mime type as the value of the
* {@code Content-Type} header.
*
* @param url the URL that is pointing to the wanted resource
* @param headers a list of headers that will be used in the request.
* Any default headers <b>should</b> be overridden by these.
* @param dataToSend byte array that will be sent when doing the request.
* @return the result of the POST request
* @see #post(String, Map, byte[], Localization)
*/
public Response postWithContentTypeJson(final String url,
@Nullable final Map<String, List<String>> headers,
@Nullable final byte[] dataToSend)
throws IOException, ReCaptchaException {
return postWithContentTypeJson(url, headers, dataToSend,
NewPipe.getPreferredLocalization());
}
/**
* Do a request using the specified {@link Request} object.
*
* @return the result of the request
*/
public abstract Response execute(@Nonnull Request request) throws IOException, ReCaptchaException;
public abstract Response execute(@Nonnull Request request)
throws IOException, ReCaptchaException;
}

View File

@ -2,42 +2,53 @@ package org.schabi.newpipe.extractor.downloader;
import org.schabi.newpipe.extractor.localization.Localization;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.*;
/**
* An object that holds request information used when {@link Downloader#execute(Request) executing} a request.
* An object that holds request information used when {@link Downloader#execute(Request) executing}
* a request.
*/
public class Request {
private final String httpMethod;
private final String url;
private final Map<String, List<String>> headers;
@Nullable private final byte[] dataToSend;
@Nullable private final Localization localization;
@Nullable
private final byte[] dataToSend;
@Nullable
private final Localization localization;
public Request(String httpMethod, String url, Map<String, List<String>> headers, @Nullable byte[] dataToSend,
@Nullable Localization localization, boolean automaticLocalizationHeader) {
if (httpMethod == null) throw new IllegalArgumentException("Request's httpMethod is null");
if (url == null) throw new IllegalArgumentException("Request's url is null");
this.httpMethod = httpMethod;
this.url = url;
public Request(final String httpMethod,
final String url,
@Nullable final Map<String, List<String>> headers,
@Nullable final byte[] dataToSend,
@Nullable final Localization localization,
final boolean automaticLocalizationHeader) {
this.httpMethod = Objects.requireNonNull(httpMethod, "Request's httpMethod is null");
this.url = Objects.requireNonNull(url, "Request's url is null");
this.dataToSend = dataToSend;
this.localization = localization;
Map<String, List<String>> headersToSet = null;
if (headers == null) headers = Collections.emptyMap();
final Map<String, List<String>> actualHeaders = new LinkedHashMap<>();
if (headers != null) {
actualHeaders.putAll(headers);
}
if (automaticLocalizationHeader && localization != null) {
headersToSet = new LinkedHashMap<>(headersFromLocalization(localization));
headersToSet.putAll(headers);
actualHeaders.putAll(getHeadersFromLocalization(localization));
}
this.headers = Collections.unmodifiableMap(headersToSet == null ? headers : headersToSet);
this.headers = Collections.unmodifiableMap(actualHeaders);
}
private Request(Builder builder) {
private Request(final Builder builder) {
this(builder.httpMethod, builder.url, builder.headers, builder.dataToSend,
builder.localization, builder.automaticLocalizationHeader);
}
@ -80,7 +91,7 @@ public class Request {
* A localization object that should be used when executing a request.<br>
* <br>
* Usually the {@code Accept-Language} will be set to this value (a helper
* method to do this easily: {@link Request#headersFromLocalization(Localization)}).
* method to do this easily: {@link Request#getHeadersFromLocalization(Localization)}).
*/
@Nullable
public Localization localization() {
@ -94,7 +105,7 @@ public class Request {
public static final class Builder {
private String httpMethod;
private String url;
private Map<String, List<String>> headers = new LinkedHashMap<>();
private final Map<String, List<String>> headers = new LinkedHashMap<>();
private byte[] dataToSend;
private Localization localization;
private boolean automaticLocalizationHeader = true;
@ -105,30 +116,29 @@ public class Request {
/**
* A http method (i.e. {@code GET, POST, HEAD}).
*/
public Builder httpMethod(String httpMethod) {
this.httpMethod = httpMethod;
public Builder httpMethod(final String httpMethodToSet) {
this.httpMethod = httpMethodToSet;
return this;
}
/**
* The URL that is pointing to the wanted resource.
*/
public Builder url(String url) {
this.url = url;
public Builder url(final String urlToSet) {
this.url = urlToSet;
return this;
}
/**
* A list of headers that will be used in the request.<br>
* Any default headers that the implementation may have, <b>should</b> be overridden by these.
* Any default headers that the implementation may have, <b>should</b> be overridden by
* these.
*/
public Builder headers(@Nullable Map<String, List<String>> headers) {
if (headers == null) {
this.headers.clear();
return this;
}
public Builder headers(@Nullable final Map<String, List<String>> headersToSet) {
this.headers.clear();
this.headers.putAll(headers);
if (headersToSet != null) {
this.headers.putAll(headersToSet);
}
return this;
}
@ -139,8 +149,8 @@ public class Request {
* The implementation should make note of some recommended headers
* (for example, {@code Content-Length} in a post request).
*/
public Builder dataToSend(byte[] dataToSend) {
this.dataToSend = dataToSend;
public Builder dataToSend(final byte[] dataToSendToSet) {
this.dataToSend = dataToSendToSet;
return this;
}
@ -148,18 +158,18 @@ public class Request {
* A localization object that should be used when executing a request.<br>
* <br>
* Usually the {@code Accept-Language} will be set to this value (a helper
* method to do this easily: {@link Request#headersFromLocalization(Localization)}).
* method to do this easily: {@link Request#getHeadersFromLocalization(Localization)}).
*/
public Builder localization(Localization localization) {
this.localization = localization;
public Builder localization(final Localization localizationToSet) {
this.localization = localizationToSet;
return this;
}
/**
* If localization headers should automatically be included in the request.
*/
public Builder automaticLocalizationHeader(boolean automaticLocalizationHeader) {
this.automaticLocalizationHeader = automaticLocalizationHeader;
public Builder automaticLocalizationHeader(final boolean automaticLocalizationHeaderToSet) {
this.automaticLocalizationHeader = automaticLocalizationHeaderToSet;
return this;
}
@ -172,22 +182,22 @@ public class Request {
// Http Methods Utils
//////////////////////////////////////////////////////////////////////////*/
public Builder get(String url) {
public Builder get(final String urlToSet) {
this.httpMethod = "GET";
this.url = url;
this.url = urlToSet;
return this;
}
public Builder head(String url) {
public Builder head(final String urlToSet) {
this.httpMethod = "HEAD";
this.url = url;
this.url = urlToSet;
return this;
}
public Builder post(String url, @Nullable byte[] dataToSend) {
public Builder post(final String urlToSet, @Nullable final byte[] dataToSendToSet) {
this.httpMethod = "POST";
this.url = url;
this.dataToSend = dataToSend;
this.url = urlToSet;
this.dataToSend = dataToSendToSet;
return this;
}
@ -195,13 +205,13 @@ public class Request {
// Additional Headers Utils
//////////////////////////////////////////////////////////////////////////*/
public Builder setHeaders(String headerName, List<String> headerValueList) {
public Builder setHeaders(final String headerName, final List<String> headerValueList) {
this.headers.remove(headerName);
this.headers.put(headerName, headerValueList);
return this;
}
public Builder addHeaders(String headerName, List<String> headerValueList) {
public Builder addHeaders(final String headerName, final List<String> headerValueList) {
@Nullable List<String> currentHeaderValueList = this.headers.get(headerName);
if (currentHeaderValueList == null) {
currentHeaderValueList = new ArrayList<>();
@ -212,11 +222,11 @@ public class Request {
return this;
}
public Builder setHeader(String headerName, String headerValue) {
public Builder setHeader(final String headerName, final String headerValue) {
return setHeaders(headerName, Collections.singletonList(headerValue));
}
public Builder addHeader(String headerName, String headerValue) {
public Builder addHeader(final String headerName, final String headerValue) {
return addHeaders(headerName, Collections.singletonList(headerValue));
}
@ -228,17 +238,43 @@ public class Request {
@SuppressWarnings("WeakerAccess")
@Nonnull
public static Map<String, List<String>> headersFromLocalization(@Nullable Localization localization) {
if (localization == null) return Collections.emptyMap();
final Map<String, List<String>> headers = new LinkedHashMap<>();
if (!localization.getCountryCode().isEmpty()) {
headers.put("Accept-Language", Collections.singletonList(localization.getLocalizationCode() +
", " + localization.getLanguageCode() + ";q=0.9"));
} else {
headers.put("Accept-Language", Collections.singletonList(localization.getLanguageCode()));
public static Map<String, List<String>> getHeadersFromLocalization(
@Nullable final Localization localization) {
if (localization == null) {
return Collections.emptyMap();
}
return headers;
final String languageCode = localization.getLanguageCode();
final List<String> languageCodeList = Collections.singletonList(
localization.getCountryCode().isEmpty() ? languageCode
: localization.getLocalizationCode() + ", " + languageCode + ";q=0.9");
return Collections.singletonMap("Accept-Language", languageCodeList);
}
/*//////////////////////////////////////////////////////////////////////////
// Generated
//////////////////////////////////////////////////////////////////////////*/
@Override
public boolean equals(final Object o) {
if (this == o) {
return true;
}
if (o == null || getClass() != o.getClass()) {
return false;
}
final Request request = (Request) o;
return httpMethod.equals(request.httpMethod)
&& url.equals(request.url)
&& headers.equals(request.headers)
&& Arrays.equals(dataToSend, request.dataToSend)
&& Objects.equals(localization, request.localization);
}
@Override
public int hashCode() {
int result = Objects.hash(httpMethod, url, headers, localization);
result = 31 * result + Arrays.hashCode(dataToSend);
return result;
}
}

View File

@ -17,11 +17,14 @@ public class Response {
private final String latestUrl;
public Response(int responseCode, String responseMessage, Map<String, List<String>> responseHeaders,
@Nullable String responseBody, @Nullable String latestUrl) {
public Response(final int responseCode,
final String responseMessage,
@Nullable final Map<String, List<String>> responseHeaders,
@Nullable final String responseBody,
@Nullable final String latestUrl) {
this.responseCode = responseCode;
this.responseMessage = responseMessage;
this.responseHeaders = responseHeaders != null ? responseHeaders : Collections.<String, List<String>>emptyMap();
this.responseHeaders = responseHeaders == null ? Collections.emptyMap() : responseHeaders;
this.responseBody = responseBody == null ? "" : responseBody;
this.latestUrl = latestUrl;
@ -60,19 +63,18 @@ public class Response {
/**
* For easy access to some header value that (usually) don't repeat itself.
* <p>For getting all the values associated to the header, use {@link #responseHeaders()} (e.g. {@code Set-Cookie}).
* <p>For getting all the values associated to the header, use {@link #responseHeaders()} (e.g.
* {@code Set-Cookie}).
*
* @param name the name of the header
* @return the first value assigned to this header
*/
@Nullable
public String getHeader(String name) {
for (Map.Entry<String, List<String>> headerEntry : responseHeaders.entrySet()) {
public String getHeader(final String name) {
for (final Map.Entry<String, List<String>> headerEntry : responseHeaders.entrySet()) {
final String key = headerEntry.getKey();
if (key != null && key.equalsIgnoreCase(name)) {
if (headerEntry.getValue().size() > 0) {
return headerEntry.getValue().get(0);
}
if (key != null && key.equalsIgnoreCase(name) && !headerEntry.getValue().isEmpty()) {
return headerEntry.getValue().get(0);
}
}

View File

@ -0,0 +1,31 @@
package org.schabi.newpipe.extractor.exceptions;
public class AccountTerminatedException extends ContentNotAvailableException {
private Reason reason = Reason.UNKNOWN;
public AccountTerminatedException(final String message) {
super(message);
}
public AccountTerminatedException(final String message, final Reason reason) {
super(message);
this.reason = reason;
}
public AccountTerminatedException(final String message, final Throwable cause) {
super(message, cause);
}
/**
* The reason for the violation. There should also be more info in the exception's message.
*/
public Reason getReason() {
return reason;
}
public enum Reason {
UNKNOWN,
VIOLATION
}
}

View File

@ -0,0 +1,11 @@
package org.schabi.newpipe.extractor.exceptions;
public class AgeRestrictedContentException extends ContentNotAvailableException {
public AgeRestrictedContentException(final String message) {
super(message);
}
public AgeRestrictedContentException(final String message, final Throwable cause) {
super(message, cause);
}
}

View File

@ -1,11 +1,11 @@
package org.schabi.newpipe.extractor.exceptions;
public class ContentNotAvailableException extends ParsingException {
public ContentNotAvailableException(String message) {
public ContentNotAvailableException(final String message) {
super(message);
}
public ContentNotAvailableException(String message, Throwable cause) {
public ContentNotAvailableException(final String message, final Throwable cause) {
super(message, cause);
}
}

View File

@ -1,11 +1,11 @@
package org.schabi.newpipe.extractor.exceptions;
public class ContentNotSupportedException extends ParsingException {
public ContentNotSupportedException(String message) {
public ContentNotSupportedException(final String message) {
super(message);
}
public ContentNotSupportedException(String message, Throwable cause) {
public ContentNotSupportedException(final String message, final Throwable cause) {
super(message, cause);
}
}

View File

@ -3,33 +3,33 @@ package org.schabi.newpipe.extractor.exceptions;
/*
* Created by Christian Schabesberger on 30.01.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* ExtractionException.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ExtractionException.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public class ExtractionException extends Exception {
public ExtractionException(String message) {
public ExtractionException(final String message) {
super(message);
}
public ExtractionException(Throwable cause) {
public ExtractionException(final Throwable cause) {
super(cause);
}
public ExtractionException(String message, Throwable cause) {
public ExtractionException(final String message, final Throwable cause) {
super(message, cause);
}
}
}

View File

@ -3,29 +3,29 @@ package org.schabi.newpipe.extractor.exceptions;
/*
* Created by Christian Schabesberger on 12.09.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* FoundAdException.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* FoundAdException.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public class FoundAdException extends ParsingException {
public FoundAdException(String message) {
public FoundAdException(final String message) {
super(message);
}
public FoundAdException(String message, Throwable cause) {
public FoundAdException(final String message, final Throwable cause) {
super(message, cause);
}
}

View File

@ -0,0 +1,11 @@
package org.schabi.newpipe.extractor.exceptions;
public class GeographicRestrictionException extends ContentNotAvailableException {
public GeographicRestrictionException(final String message) {
super(message);
}
public GeographicRestrictionException(final String message, final Throwable cause) {
super(message, cause);
}
}

View File

@ -0,0 +1,11 @@
package org.schabi.newpipe.extractor.exceptions;
public class PaidContentException extends ContentNotAvailableException {
public PaidContentException(final String message) {
super(message);
}
public PaidContentException(final String message, final Throwable cause) {
super(message, cause);
}
}

View File

@ -3,30 +3,30 @@ package org.schabi.newpipe.extractor.exceptions;
/*
* Created by Christian Schabesberger on 31.01.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* ParsingException.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ParsingException.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public class ParsingException extends ExtractionException {
public ParsingException(String message) {
public ParsingException(final String message) {
super(message);
}
public ParsingException(String message, Throwable cause) {
public ParsingException(final String message, final Throwable cause) {
super(message, cause);
}
}
}

View File

@ -0,0 +1,11 @@
package org.schabi.newpipe.extractor.exceptions;
public class PrivateContentException extends ContentNotAvailableException {
public PrivateContentException(final String message) {
super(message);
}
public PrivateContentException(final String message, final Throwable cause) {
super(message, cause);
}
}

View File

@ -3,27 +3,27 @@ package org.schabi.newpipe.extractor.exceptions;
/*
* Created by beneth <bmauduit@beneth.fr> on 07.12.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* ReCaptchaException.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* ReCaptchaException.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public class ReCaptchaException extends ExtractionException {
private String url;
private final String url;
public ReCaptchaException(String message, String url) {
public ReCaptchaException(final String message, final String url) {
super(message);
this.url = url;
}

View File

@ -0,0 +1,11 @@
package org.schabi.newpipe.extractor.exceptions;
public class SoundCloudGoPlusContentException extends ContentNotAvailableException {
public SoundCloudGoPlusContentException() {
super("This track is a SoundCloud Go+ track");
}
public SoundCloudGoPlusContentException(final Throwable cause) {
super("This track is a SoundCloud Go+ track", cause);
}
}

View File

@ -0,0 +1,20 @@
package org.schabi.newpipe.extractor.exceptions;
/**
* Exception for contents not supported in a country.
*
* <p>
* Unsupported content means content is not intentionally geographically restricted such as for
* distribution rights, for which {@link GeographicRestrictionException} should be used instead.
* </p>
*/
public class UnsupportedContentInCountryException extends ContentNotAvailableException {
public UnsupportedContentInCountryException(final String message) {
super(message);
}
public UnsupportedContentInCountryException(final String message, final Throwable cause) {
super(message, cause);
}
}

View File

@ -0,0 +1,7 @@
package org.schabi.newpipe.extractor.exceptions;
public final class UnsupportedTabException extends UnsupportedOperationException {
public UnsupportedTabException(final String unsupportedTab) {
super("Unsupported tab " + unsupportedTab);
}
}

View File

@ -0,0 +1,11 @@
package org.schabi.newpipe.extractor.exceptions;
public class YoutubeMusicPremiumContentException extends ContentNotAvailableException {
public YoutubeMusicPremiumContentException() {
super("This video is a YouTube Music Premium video");
}
public YoutubeMusicPremiumContentException(final Throwable cause) {
super("This video is a YouTube Music Premium video", cause);
}
}

View File

@ -11,7 +11,7 @@ import org.schabi.newpipe.extractor.stream.StreamInfoItem;
* YouTube is an example of a service that has this alternative available.
*/
public abstract class FeedExtractor extends ListExtractor<StreamInfoItem> {
public FeedExtractor(StreamingService service, ListLinkHandler listLinkHandler) {
public FeedExtractor(final StreamingService service, final ListLinkHandler listLinkHandler) {
super(service, listLinkHandler);
}
}

View File

@ -13,26 +13,35 @@ import java.util.List;
public class FeedInfo extends ListInfo<StreamInfoItem> {
public FeedInfo(int serviceId, String id, String url, String originalUrl, String name, List<String> contentFilter, String sortFilter) {
public FeedInfo(final int serviceId,
final String id,
final String url,
final String originalUrl,
final String name,
final List<String> contentFilter,
final String sortFilter) {
super(serviceId, id, url, originalUrl, name, contentFilter, sortFilter);
}
public static FeedInfo getInfo(String url) throws IOException, ExtractionException {
public static FeedInfo getInfo(final String url) throws IOException, ExtractionException {
return getInfo(NewPipe.getServiceByUrl(url), url);
}
public static FeedInfo getInfo(StreamingService service, String url) throws IOException, ExtractionException {
public static FeedInfo getInfo(final StreamingService service, final String url)
throws IOException, ExtractionException {
final FeedExtractor extractor = service.getFeedExtractor(url);
if (extractor == null) {
throw new IllegalArgumentException("Service \"" + service.getServiceInfo().getName() + "\" doesn't support FeedExtractor.");
throw new IllegalArgumentException("Service \"" + service.getServiceInfo().getName()
+ "\" doesn't support FeedExtractor.");
}
extractor.fetchPage();
return getInfo(extractor);
}
public static FeedInfo getInfo(FeedExtractor extractor) throws IOException, ExtractionException {
public static FeedInfo getInfo(final FeedExtractor extractor)
throws IOException, ExtractionException {
extractor.fetchPage();
final int serviceId = extractor.getServiceId();
@ -43,9 +52,10 @@ public class FeedInfo extends ListInfo<StreamInfoItem> {
final FeedInfo info = new FeedInfo(serviceId, id, url, originalUrl, name, null, null);
final InfoItemsPage<StreamInfoItem> itemsPage = ExtractorHelper.getItemsPageOrLogError(info, extractor);
final InfoItemsPage<StreamInfoItem> itemsPage
= ExtractorHelper.getItemsPageOrLogError(info, extractor);
info.setRelatedItems(itemsPage.getItems());
info.setNextPageUrl(itemsPage.getNextPageUrl());
info.setNextPage(itemsPage.getNextPage());
return info;
}

View File

@ -3,21 +3,21 @@ package org.schabi.newpipe.extractor.kiosk;
/*
* Created by Christian Schabesberger on 12.08.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* KioskExtractor.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* KioskExtractor.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
import org.schabi.newpipe.extractor.InfoItem;
@ -31,9 +31,9 @@ import javax.annotation.Nonnull;
public abstract class KioskExtractor<T extends InfoItem> extends ListExtractor<T> {
private final String id;
public KioskExtractor(StreamingService streamingService,
ListLinkHandler linkHandler,
String kioskId) {
public KioskExtractor(final StreamingService streamingService,
final ListLinkHandler linkHandler,
final String kioskId) {
super(streamingService, linkHandler);
this.id = kioskId;
}
@ -45,12 +45,11 @@ public abstract class KioskExtractor<T extends InfoItem> extends ListExtractor<T
}
/**
* Id should be the name of the kiosk, tho Id is used for identifing it in the frontend,
* Id should be the name of the kiosk, tho Id is used for identifying it in the frontend,
* so id should be kept in english.
* In order to get the name of the kiosk in the desired language we have to
* crawl if from the website.
* @return the tranlsated version of id
* @throws ParsingException
* @return the translated version of id
*/
@Nonnull
@Override

View File

@ -3,57 +3,53 @@ package org.schabi.newpipe.extractor.kiosk;
/*
* Created by Christian Schabesberger on 12.08.17.
*
* Copyright (C) Christian Schabesberger 2017 <chris.schabesberger@mailbox.org>
* KioskInfo.java is part of NewPipe.
* Copyright (C) 2017 Christian Schabesberger <chris.schabesberger@mailbox.org>
* KioskInfo.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.ListInfo;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import org.schabi.newpipe.extractor.utils.ExtractorHelper;
import java.io.IOException;
public class KioskInfo extends ListInfo<StreamInfoItem> {
private KioskInfo(int serviceId, ListLinkHandler linkHandler, String name) throws ParsingException {
public final class KioskInfo extends ListInfo<StreamInfoItem> {
private KioskInfo(final int serviceId, final ListLinkHandler linkHandler, final String name) {
super(serviceId, linkHandler, name);
}
public static ListExtractor.InfoItemsPage<StreamInfoItem> getMoreItems(StreamingService service,
String url,
String pageUrl)
public static ListExtractor.InfoItemsPage<StreamInfoItem> getMoreItems(
final StreamingService service, final String url, final Page page)
throws IOException, ExtractionException {
KioskList kl = service.getKioskList();
KioskExtractor extractor = kl.getExtractorByUrl(url, pageUrl);
return extractor.getPage(pageUrl);
return service.getKioskList().getExtractorByUrl(url, page).getPage(page);
}
public static KioskInfo getInfo(String url) throws IOException, ExtractionException {
public static KioskInfo getInfo(final String url) throws IOException, ExtractionException {
return getInfo(NewPipe.getServiceByUrl(url), url);
}
public static KioskInfo getInfo(StreamingService service, String url) throws IOException, ExtractionException {
KioskList kl = service.getKioskList();
KioskExtractor extractor = kl.getExtractorByUrl(url, null);
public static KioskInfo getInfo(final StreamingService service, final String url)
throws IOException, ExtractionException {
final KioskExtractor extractor = service.getKioskList().getExtractorByUrl(url, null);
extractor.fetchPage();
return getInfo(extractor);
}
@ -63,15 +59,16 @@ public class KioskInfo extends ListInfo<StreamInfoItem> {
*
* @param extractor an extractor where fetchPage() was already got called on.
*/
public static KioskInfo getInfo(KioskExtractor extractor) throws ExtractionException {
public static KioskInfo getInfo(final KioskExtractor extractor) throws ExtractionException {
final KioskInfo info = new KioskInfo(extractor.getServiceId(),
extractor.getLinkHandler(),
extractor.getName());
final ListExtractor.InfoItemsPage<StreamInfoItem> itemsPage = ExtractorHelper.getItemsPageOrLogError(info, extractor);
final ListExtractor.InfoItemsPage<StreamInfoItem> itemsPage
= ExtractorHelper.getItemsPageOrLogError(info, extractor);
info.setRelatedItems(itemsPage.getItems());
info.setNextPageUrl(itemsPage.getNextPageUrl());
info.setNextPage(itemsPage.getNextPage());
return info;
}

View File

@ -1,24 +1,28 @@
package org.schabi.newpipe.extractor.kiosk;
import static org.schabi.newpipe.extractor.utils.Utils.isNullOrEmpty;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import org.schabi.newpipe.extractor.localization.ContentCountry;
import org.schabi.newpipe.extractor.localization.Localization;
import javax.annotation.Nullable;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import java.util.Set;
import javax.annotation.Nullable;
public class KioskList {
public interface KioskExtractorFactory {
KioskExtractor createNewKiosk(final StreamingService streamingService,
final String url,
final String kioskId)
KioskExtractor createNewKiosk(StreamingService streamingService,
String url,
String kioskId)
throws ExtractionException, IOException;
}
@ -31,8 +35,8 @@ public class KioskList {
@Nullable
private ContentCountry forcedContentCountry;
private class KioskEntry {
public KioskEntry(KioskExtractorFactory ef, ListLinkHandlerFactory h) {
private static class KioskEntry {
KioskEntry(final KioskExtractorFactory ef, final ListLinkHandlerFactory h) {
extractorFactory = ef;
handlerFactory = h;
}
@ -41,11 +45,13 @@ public class KioskList {
final ListLinkHandlerFactory handlerFactory;
}
public KioskList(StreamingService service) {
public KioskList(final StreamingService service) {
this.service = service;
}
public void addKioskEntry(KioskExtractorFactory extractorFactory, ListLinkHandlerFactory handlerFactory, String id)
public void addKioskEntry(final KioskExtractorFactory extractorFactory,
final ListLinkHandlerFactory handlerFactory,
final String id)
throws Exception {
if (kioskList.get(id) != null) {
throw new Exception("Kiosk with type " + id + " already exists.");
@ -53,29 +59,30 @@ public class KioskList {
kioskList.put(id, new KioskEntry(extractorFactory, handlerFactory));
}
public void setDefaultKiosk(String kioskType) {
public void setDefaultKiosk(final String kioskType) {
defaultKiosk = kioskType;
}
public KioskExtractor getDefaultKioskExtractor()
throws ExtractionException, IOException {
return getDefaultKioskExtractor("");
return getDefaultKioskExtractor(null);
}
public KioskExtractor getDefaultKioskExtractor(String nextPageUrl)
public KioskExtractor getDefaultKioskExtractor(final Page nextPage)
throws ExtractionException, IOException {
return getDefaultKioskExtractor(nextPageUrl, NewPipe.getPreferredLocalization());
return getDefaultKioskExtractor(nextPage, NewPipe.getPreferredLocalization());
}
public KioskExtractor getDefaultKioskExtractor(String nextPageUrl, Localization localization)
public KioskExtractor getDefaultKioskExtractor(final Page nextPage,
final Localization localization)
throws ExtractionException, IOException {
if (defaultKiosk != null && !defaultKiosk.equals("")) {
return getExtractorById(defaultKiosk, nextPageUrl, localization);
if (!isNullOrEmpty(defaultKiosk)) {
return getExtractorById(defaultKiosk, nextPage, localization);
} else {
if (!kioskList.isEmpty()) {
final String first = kioskList.keySet().stream().findAny().orElse(null);
if (first != null) {
// if not set get any entry
Object[] keySet = kioskList.keySet().toArray();
return getExtractorById(keySet[0].toString(), nextPageUrl, localization);
return getExtractorById(first, nextPage, localization);
} else {
return null;
}
@ -86,22 +93,28 @@ public class KioskList {
return defaultKiosk;
}
public KioskExtractor getExtractorById(String kioskId, String nextPageUrl)
public KioskExtractor getExtractorById(final String kioskId, final Page nextPage)
throws ExtractionException, IOException {
return getExtractorById(kioskId, nextPageUrl, NewPipe.getPreferredLocalization());
return getExtractorById(kioskId, nextPage, NewPipe.getPreferredLocalization());
}
public KioskExtractor getExtractorById(String kioskId, String nextPageUrl, Localization localization)
public KioskExtractor getExtractorById(final String kioskId,
final Page nextPage,
final Localization localization)
throws ExtractionException, IOException {
KioskEntry ke = kioskList.get(kioskId);
final KioskEntry ke = kioskList.get(kioskId);
if (ke == null) {
throw new ExtractionException("No kiosk found with the type: " + kioskId);
} else {
final KioskExtractor kioskExtractor = ke.extractorFactory.createNewKiosk(service,
ke.handlerFactory.fromId(kioskId).getUrl(), kioskId);
if (forcedLocalization != null) kioskExtractor.forceLocalization(forcedLocalization);
if (forcedContentCountry != null) kioskExtractor.forceContentCountry(forcedContentCountry);
if (forcedLocalization != null) {
kioskExtractor.forceLocalization(forcedLocalization);
}
if (forcedContentCountry != null) {
kioskExtractor.forceContentCountry(forcedContentCountry);
}
return kioskExtractor;
}
@ -111,31 +124,33 @@ public class KioskList {
return kioskList.keySet();
}
public KioskExtractor getExtractorByUrl(String url, String nextPageUrl)
public KioskExtractor getExtractorByUrl(final String url, final Page nextPage)
throws ExtractionException, IOException {
return getExtractorByUrl(url, nextPageUrl, NewPipe.getPreferredLocalization());
return getExtractorByUrl(url, nextPage, NewPipe.getPreferredLocalization());
}
public KioskExtractor getExtractorByUrl(String url, String nextPageUrl, Localization localization)
public KioskExtractor getExtractorByUrl(final String url,
final Page nextPage,
final Localization localization)
throws ExtractionException, IOException {
for (Map.Entry<String, KioskEntry> e : kioskList.entrySet()) {
KioskEntry ke = e.getValue();
for (final Map.Entry<String, KioskEntry> e : kioskList.entrySet()) {
final KioskEntry ke = e.getValue();
if (ke.handlerFactory.acceptUrl(url)) {
return getExtractorById(ke.handlerFactory.getId(url), nextPageUrl, localization);
return getExtractorById(ke.handlerFactory.getId(url), nextPage, localization);
}
}
throw new ExtractionException("Could not find a kiosk that fits to the url: " + url);
}
public ListLinkHandlerFactory getListLinkHandlerFactoryByType(String type) {
public ListLinkHandlerFactory getListLinkHandlerFactoryByType(final String type) {
return kioskList.get(type).handlerFactory;
}
public void forceLocalization(@Nullable Localization localization) {
public void forceLocalization(@Nullable final Localization localization) {
this.forcedLocalization = localization;
}
public void forceContentCountry(@Nullable ContentCountry contentCountry) {
public void forceContentCountry(@Nullable final ContentCountry contentCountry) {
this.forcedContentCountry = contentCountry;
}
}

View File

@ -10,13 +10,13 @@ public class LinkHandler implements Serializable {
protected final String url;
protected final String id;
public LinkHandler(String originalUrl, String url, String id) {
public LinkHandler(final String originalUrl, final String url, final String id) {
this.originalUrl = originalUrl;
this.url = url;
this.id = id;
}
public LinkHandler(LinkHandler handler) {
public LinkHandler(final LinkHandler handler) {
this(handler.originalUrl, handler.url, handler.id);
}

View File

@ -1,27 +1,28 @@
package org.schabi.newpipe.extractor.linkhandler;
import org.schabi.newpipe.extractor.exceptions.FoundAdException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.utils.Utils;
import java.util.Objects;
/*
* Created by Christian Schabesberger on 26.07.16.
*
* Copyright (C) Christian Schabesberger 2016 <chris.schabesberger@mailbox.org>
* LinkHandlerFactory.java is part of NewPipe.
* Copyright (C) 2016 Christian Schabesberger <chris.schabesberger@mailbox.org>
* LinkHandlerFactory.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <http://www.gnu.org/licenses/>.
*/
public abstract class LinkHandlerFactory {
@ -30,11 +31,14 @@ public abstract class LinkHandlerFactory {
// To Override
///////////////////////////////////
public abstract String getId(String url) throws ParsingException;
public abstract String getUrl(String id) throws ParsingException;
public abstract boolean onAcceptUrl(final String url) throws ParsingException;
public abstract String getId(String url) throws ParsingException, UnsupportedOperationException;
public String getUrl(String id, String baseUrl) throws ParsingException {
public abstract String getUrl(String id) throws ParsingException, UnsupportedOperationException;
public abstract boolean onAcceptUrl(String url) throws ParsingException;
public String getUrl(final String id, final String baseUrl)
throws ParsingException, UnsupportedOperationException {
return getUrl(id);
}
@ -42,30 +46,52 @@ public abstract class LinkHandlerFactory {
// Logic
///////////////////////////////////
public LinkHandler fromUrl(String url) throws ParsingException {
if (url == null) throw new IllegalArgumentException("url can not be null");
final String baseUrl = Utils.getBaseUrl(url);
return fromUrl(url, baseUrl);
/**
* Builds a {@link LinkHandler} from a url.<br>
* Be sure to call {@link Utils#followGoogleRedirectIfNeeded(String)} on the url if overriding
* this function.
*
* @param url the url to extract path and id from
* @return a {@link LinkHandler} complete with information
*/
public LinkHandler fromUrl(final String url) throws ParsingException {
if (Utils.isNullOrEmpty(url)) {
throw new IllegalArgumentException("The url is null or empty");
}
final String polishedUrl = Utils.followGoogleRedirectIfNeeded(url);
final String baseUrl = Utils.getBaseUrl(polishedUrl);
return fromUrl(polishedUrl, baseUrl);
}
public LinkHandler fromUrl(String url, String baseUrl) throws ParsingException {
if (url == null) throw new IllegalArgumentException("url can not be null");
/**
* Builds a {@link LinkHandler} from an URL and a base URL. The URL is expected to be already
* polished from Google search redirects (otherwise how could {@code baseUrl} have been
* extracted?).<br>
* So do not call {@link Utils#followGoogleRedirectIfNeeded(String)} on the URL if overriding
* this function, since that should be done in {@link #fromUrl(String)}.
*
* @param url the URL without Google search redirects to extract id from
* @param baseUrl the base URL
* @return a {@link LinkHandler} complete with information
*/
public LinkHandler fromUrl(final String url, final String baseUrl) throws ParsingException {
Objects.requireNonNull(url, "URL cannot be null");
if (!acceptUrl(url)) {
throw new ParsingException("Malformed unacceptable url: " + url);
throw new ParsingException("URL not accepted: " + url);
}
final String id = getId(url);
return new LinkHandler(url, getUrl(id, baseUrl), id);
}
public LinkHandler fromId(String id) throws ParsingException {
if (id == null) throw new IllegalArgumentException("id can not be null");
public LinkHandler fromId(final String id) throws ParsingException {
Objects.requireNonNull(id, "ID cannot be null");
final String url = getUrl(id);
return new LinkHandler(url, url, id);
}
public LinkHandler fromId(String id, String baseUrl) throws ParsingException {
if (id == null) throw new IllegalArgumentException("id can not be null");
public LinkHandler fromId(final String id, final String baseUrl) throws ParsingException {
Objects.requireNonNull(id, "ID cannot be null");
final String url = getUrl(id, baseUrl);
return new LinkHandler(url, url, id);
}
@ -76,11 +102,6 @@ public abstract class LinkHandlerFactory {
* Return false if this service shall not allow to be called through ACTIONs.
*/
public boolean acceptUrl(final String url) throws ParsingException {
try {
return onAcceptUrl(url);
} catch (FoundAdException fe) {
throw fe;
}
return onAcceptUrl(url);
}
}

View File

@ -7,17 +7,17 @@ public class ListLinkHandler extends LinkHandler {
protected final List<String> contentFilters;
protected final String sortFilter;
public ListLinkHandler(String originalUrl,
String url,
String id,
List<String> contentFilters,
String sortFilter) {
public ListLinkHandler(final String originalUrl,
final String url,
final String id,
final List<String> contentFilters,
final String sortFilter) {
super(originalUrl, url, id);
this.contentFilters = Collections.unmodifiableList(contentFilters);
this.sortFilter = sortFilter;
}
public ListLinkHandler(ListLinkHandler handler) {
public ListLinkHandler(final ListLinkHandler handler) {
this(handler.originalUrl,
handler.url,
handler.id,
@ -25,14 +25,12 @@ public class ListLinkHandler extends LinkHandler {
handler.sortFilter);
}
public ListLinkHandler(LinkHandler handler,
List<String> contentFilters,
String sortFilter) {
public ListLinkHandler(final LinkHandler handler) {
this(handler.originalUrl,
handler.url,
handler.id,
contentFilters,
sortFilter);
Collections.emptyList(),
"");
}
public List<String> getContentFilters() {

View File

@ -5,6 +5,7 @@ import org.schabi.newpipe.extractor.utils.Utils;
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
public abstract class ListLinkHandlerFactory extends LinkHandlerFactory {
@ -12,17 +13,14 @@ public abstract class ListLinkHandlerFactory extends LinkHandlerFactory {
// To Override
///////////////////////////////////
public List<String> getContentFilter(String url) throws ParsingException {
return new ArrayList<>(0);
}
public abstract String getUrl(String id, List<String> contentFilter, String sortFilter)
throws ParsingException, UnsupportedOperationException;
public String getSortFilter(String url) throws ParsingException {
return "";
}
public abstract String getUrl(String id, List<String> contentFilter, String sortFilter) throws ParsingException;
public String getUrl(String id, List<String> contentFilter, String sortFilter, String baseUrl) throws ParsingException {
public String getUrl(final String id,
final List<String> contentFilter,
final String sortFilter,
final String baseUrl)
throws ParsingException, UnsupportedOperationException {
return getUrl(id, contentFilter, sortFilter);
}
@ -31,61 +29,62 @@ public abstract class ListLinkHandlerFactory extends LinkHandlerFactory {
///////////////////////////////////
@Override
public ListLinkHandler fromUrl(String url) throws ParsingException {
String baseUrl = Utils.getBaseUrl(url);
return fromUrl(url, baseUrl);
public ListLinkHandler fromUrl(final String url) throws ParsingException {
final String polishedUrl = Utils.followGoogleRedirectIfNeeded(url);
final String baseUrl = Utils.getBaseUrl(polishedUrl);
return fromUrl(polishedUrl, baseUrl);
}
@Override
public ListLinkHandler fromUrl(String url, String baseUrl) throws ParsingException {
if (url == null) throw new IllegalArgumentException("url may not be null");
return new ListLinkHandler(super.fromUrl(url, baseUrl), getContentFilter(url), getSortFilter(url));
public ListLinkHandler fromUrl(final String url, final String baseUrl) throws ParsingException {
Objects.requireNonNull(url, "URL may not be null");
return new ListLinkHandler(super.fromUrl(url, baseUrl));
}
@Override
public ListLinkHandler fromId(String id) throws ParsingException {
return new ListLinkHandler(super.fromId(id), new ArrayList<String>(0), "");
public ListLinkHandler fromId(final String id) throws ParsingException {
return new ListLinkHandler(super.fromId(id));
}
@Override
public ListLinkHandler fromId(String id, String baseUrl) throws ParsingException {
return new ListLinkHandler(super.fromId(id, baseUrl), new ArrayList<String>(0), "");
public ListLinkHandler fromId(final String id, final String baseUrl) throws ParsingException {
return new ListLinkHandler(super.fromId(id, baseUrl));
}
public ListLinkHandler fromQuery(String id,
List<String> contentFilters,
String sortFilter) throws ParsingException {
public ListLinkHandler fromQuery(final String id,
final List<String> contentFilters,
final String sortFilter) throws ParsingException {
final String url = getUrl(id, contentFilters, sortFilter);
return new ListLinkHandler(url, url, id, contentFilters, sortFilter);
}
public ListLinkHandler fromQuery(String id,
List<String> contentFilters,
String sortFilter, String baseUrl) throws ParsingException {
public ListLinkHandler fromQuery(final String id,
final List<String> contentFilters,
final String sortFilter,
final String baseUrl) throws ParsingException {
final String url = getUrl(id, contentFilters, sortFilter, baseUrl);
return new ListLinkHandler(url, url, id, contentFilters, sortFilter);
}
/**
* For making ListLinkHandlerFactory compatible with LinkHandlerFactory we need to override this,
* however it should not be overridden by the actual implementation.
* For making ListLinkHandlerFactory compatible with LinkHandlerFactory we need to override
* this, however it should not be overridden by the actual implementation.
*
* @param id
* @return the url coresponding to id without any filters applied
* @return the url corresponding to id without any filters applied
*/
public String getUrl(String id) throws ParsingException {
return getUrl(id, new ArrayList<String>(0), "");
public String getUrl(final String id) throws ParsingException, UnsupportedOperationException {
return getUrl(id, new ArrayList<>(0), "");
}
@Override
public String getUrl(String id, String baseUrl) throws ParsingException {
return getUrl(id, new ArrayList<String>(0), "", baseUrl);
public String getUrl(final String id, final String baseUrl) throws ParsingException {
return getUrl(id, new ArrayList<>(0), "", baseUrl);
}
/**
* Will returns content filter the corresponding extractor can handle like "channels", "videos", "music", etc.
* Will returns content filter the corresponding extractor can handle like "channels", "videos",
* "music", etc.
*
* @return filter that can be applied when building a query for getting a list
*/
@ -94,7 +93,8 @@ public abstract class ListLinkHandlerFactory extends LinkHandlerFactory {
}
/**
* Will returns sort filter the corresponding extractor can handle like "A-Z", "oldest first", "size", etc.
* Will returns sort filter the corresponding extractor can handle like "A-Z", "oldest first",
* "size", etc.
*
* @return filter that can be applied when building a query for getting a list
*/

View File

@ -0,0 +1,54 @@
package org.schabi.newpipe.extractor.linkhandler;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import javax.annotation.Nonnull;
import java.io.Serializable;
import java.util.List;
/**
* A {@link ListLinkHandler} which can be used to be returned from {@link
* org.schabi.newpipe.extractor.channel.ChannelInfo#getTabs() ChannelInfo#getTabs()} when a
* specific tab's data has already been fetched.
*
* <p>
* This class allows passing a builder for a {@link ChannelTabExtractor} that can hold references
* to variables.
* </p>
*
* <p>
* Note: a service that wishes to use this class in one of its {@link
* org.schabi.newpipe.extractor.channel.ChannelExtractor ChannelExtractor}s must also add the
* following snippet of code in the service's
* {@link StreamingService#getChannelTabExtractor(ListLinkHandler)}:
* <pre>
* if (linkHandler instanceof ReadyChannelTabListLinkHandler) {
* return ((ReadyChannelTabListLinkHandler) linkHandler).getChannelTabExtractor(this);
* }
* </pre>
*/
public class ReadyChannelTabListLinkHandler extends ListLinkHandler {
public interface ChannelTabExtractorBuilder extends Serializable {
@Nonnull
ChannelTabExtractor build(@Nonnull StreamingService service,
@Nonnull ListLinkHandler linkHandler);
}
private final ChannelTabExtractorBuilder extractorBuilder;
public ReadyChannelTabListLinkHandler(
final String url,
final String channelId,
@Nonnull final String channelTab,
@Nonnull final ChannelTabExtractorBuilder extractorBuilder) {
super(url, url, channelId, List.of(channelTab), "");
this.extractorBuilder = extractorBuilder;
}
@Nonnull
public ChannelTabExtractor getChannelTabExtractor(@Nonnull final StreamingService service) {
return extractorBuilder.build(service, new ListLinkHandler(this));
}
}

View File

@ -4,15 +4,15 @@ import java.util.List;
public class SearchQueryHandler extends ListLinkHandler {
public SearchQueryHandler(String originalUrl,
String url,
String searchString,
List<String> contentFilters,
String sortFilter) {
public SearchQueryHandler(final String originalUrl,
final String url,
final String searchString,
final List<String> contentFilters,
final String sortFilter) {
super(originalUrl, url, searchString, contentFilters, sortFilter);
}
public SearchQueryHandler(ListLinkHandler handler) {
public SearchQueryHandler(final ListLinkHandler handler) {
this(handler.originalUrl,
handler.url,
handler.id,

View File

@ -2,7 +2,7 @@ package org.schabi.newpipe.extractor.linkhandler;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
public abstract class SearchQueryHandlerFactory extends ListLinkHandlerFactory {
@ -12,9 +12,11 @@ public abstract class SearchQueryHandlerFactory extends ListLinkHandlerFactory {
///////////////////////////////////
@Override
public abstract String getUrl(String querry, List<String> contentFilter, String sortFilter) throws ParsingException;
public abstract String getUrl(String query, List<String> contentFilter, String sortFilter)
throws ParsingException, UnsupportedOperationException;
public String getSearchString(String url) {
@SuppressWarnings("unused")
public String getSearchString(final String url) {
return "";
}
@ -23,29 +25,26 @@ public abstract class SearchQueryHandlerFactory extends ListLinkHandlerFactory {
///////////////////////////////////
@Override
public String getId(String url) {
public String getId(final String url) throws ParsingException, UnsupportedOperationException {
return getSearchString(url);
}
@Override
public SearchQueryHandler fromQuery(String querry,
List<String> contentFilter,
String sortFilter) throws ParsingException {
return new SearchQueryHandler(super.fromQuery(querry, contentFilter, sortFilter));
public SearchQueryHandler fromQuery(final String query,
final List<String> contentFilter,
final String sortFilter) throws ParsingException {
return new SearchQueryHandler(super.fromQuery(query, contentFilter, sortFilter));
}
public SearchQueryHandler fromQuery(String querry) throws ParsingException {
return fromQuery(querry, new ArrayList<String>(0), "");
public SearchQueryHandler fromQuery(final String query) throws ParsingException {
return fromQuery(query, Collections.emptyList(), "");
}
/**
* It's not mandatory for NewPipe to handle the Url
*
* @param url
* @return
*/
@Override
public boolean onAcceptUrl(String url) {
public boolean onAcceptUrl(final String url) {
return false;
}
}

View File

@ -1,6 +1,7 @@
package org.schabi.newpipe.extractor.localization;
import javax.annotation.Nonnull;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.Collections;
@ -9,23 +10,26 @@ import java.util.List;
/**
* Represents a country that should be used when fetching content.
* <p>
* YouTube, for example, give different results in their feed depending on which country is selected.
* YouTube, for example, give different results in their feed depending on which country is
* selected.
* </p>
*/
public class ContentCountry implements Serializable {
public static final ContentCountry DEFAULT = new ContentCountry(Localization.DEFAULT.getCountryCode());
public static final ContentCountry DEFAULT =
new ContentCountry(Localization.DEFAULT.getCountryCode());
@Nonnull private final String countryCode;
@Nonnull
private final String countryCode;
public static List<ContentCountry> listFrom(String... countryCodeList) {
public static List<ContentCountry> listFrom(final String... countryCodeList) {
final List<ContentCountry> toReturn = new ArrayList<>();
for (String countryCode : countryCodeList) {
for (final String countryCode : countryCodeList) {
toReturn.add(new ContentCountry(countryCode));
}
return Collections.unmodifiableList(toReturn);
}
public ContentCountry(@Nonnull String countryCode) {
public ContentCountry(@Nonnull final String countryCode) {
this.countryCode = countryCode;
}
@ -40,11 +44,15 @@ public class ContentCountry implements Serializable {
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (!(o instanceof ContentCountry)) return false;
public boolean equals(final Object o) {
if (this == o) {
return true;
}
if (!(o instanceof ContentCountry)) {
return false;
}
ContentCountry that = (ContentCountry) o;
final ContentCountry that = (ContentCountry) o;
return countryCode.equals(that.countryCode);
}

View File

@ -1,37 +1,70 @@
package org.schabi.newpipe.extractor.localization;
import edu.umd.cs.findbugs.annotations.NonNull;
import javax.annotation.Nonnull;
import java.io.Serializable;
import java.time.OffsetDateTime;
import java.time.ZoneOffset;
import java.util.Calendar;
import java.util.GregorianCalendar;
/**
* A wrapper class that provides a field to describe if the date is precise or just an approximation.
* A wrapper class that provides a field to describe if the date/time is precise or just an
* approximation.
*/
public class DateWrapper implements Serializable {
@NonNull private final Calendar date;
@Nonnull
private final OffsetDateTime offsetDateTime;
private final boolean isApproximation;
public DateWrapper(@NonNull Calendar date) {
this(date, false);
/**
* @deprecated Use {@link #DateWrapper(OffsetDateTime)} instead.
*/
@Deprecated
public DateWrapper(@Nonnull final Calendar calendar) {
//noinspection deprecation
this(calendar, false);
}
public DateWrapper(@NonNull Calendar date, boolean isApproximation) {
this.date = date;
/**
* @deprecated Use {@link #DateWrapper(OffsetDateTime, boolean)} instead.
*/
@Deprecated
public DateWrapper(@Nonnull final Calendar calendar, final boolean isApproximation) {
this(OffsetDateTime.ofInstant(calendar.toInstant(), ZoneOffset.UTC), isApproximation);
}
public DateWrapper(@Nonnull final OffsetDateTime offsetDateTime) {
this(offsetDateTime, false);
}
public DateWrapper(@Nonnull final OffsetDateTime offsetDateTime,
final boolean isApproximation) {
this.offsetDateTime = offsetDateTime.withOffsetSameInstant(ZoneOffset.UTC);
this.isApproximation = isApproximation;
}
/**
* @return the wrapped date.
* @return the wrapped date/time as a {@link Calendar}.
* @deprecated use {@link #offsetDateTime()} instead.
*/
@NonNull
@Deprecated
@Nonnull
public Calendar date() {
return date;
return GregorianCalendar.from(offsetDateTime.toZonedDateTime());
}
/**
* @return if the date is considered is precise or just an approximation (e.g. service only returns an approximation
* like 2 weeks ago instead of a precise date).
* @return the wrapped date/time.
*/
@Nonnull
public OffsetDateTime offsetDateTime() {
return offsetDateTime;
}
/**
* @return if the date is considered is precise or just an approximation (e.g. service only
* returns an approximation like 2 weeks ago instead of a precise date).
*/
public boolean isApproximation() {
return isApproximation;

View File

@ -1,57 +1,67 @@
package org.schabi.newpipe.extractor.localization;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.utils.LocaleCompat;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
public class Localization implements Serializable {
public static final Localization DEFAULT = new Localization("en", "GB");
@Nonnull private final String languageCode;
@Nullable private final String countryCode;
@Nonnull
private final String languageCode;
@Nullable
private final String countryCode;
/**
* @param localizationCodeList a list of localization code, formatted like {@link #getLocalizationCode()}
* @param localizationCodeList a list of localization code, formatted like {@link
* #getLocalizationCode()}
* @throws IllegalArgumentException If any of the localizationCodeList is formatted incorrectly
* @return list of Localization objects
*/
public static List<Localization> listFrom(String... localizationCodeList) {
@Nonnull
public static List<Localization> listFrom(final String... localizationCodeList) {
final List<Localization> toReturn = new ArrayList<>();
for (String localizationCode : localizationCodeList) {
toReturn.add(fromLocalizationCode(localizationCode));
for (final String localizationCode : localizationCodeList) {
toReturn.add(fromLocalizationCode(localizationCode)
.orElseThrow(() -> new IllegalArgumentException(
"Not a localization code: " + localizationCode
)));
}
return Collections.unmodifiableList(toReturn);
}
/**
* @param localizationCode a localization code, formatted like {@link #getLocalizationCode()}
* @return A Localization, if the code was valid.
*/
public static Localization fromLocalizationCode(String localizationCode) {
final int indexSeparator = localizationCode.indexOf("-");
final String languageCode, countryCode;
if (indexSeparator != -1) {
languageCode = localizationCode.substring(0, indexSeparator);
countryCode = localizationCode.substring(indexSeparator + 1);
} else {
languageCode = localizationCode;
countryCode = null;
}
return new Localization(languageCode, countryCode);
@Nonnull
public static Optional<Localization> fromLocalizationCode(final String localizationCode) {
return LocaleCompat.forLanguageTag(localizationCode).map(Localization::fromLocale);
}
public Localization(@Nonnull String languageCode, @Nullable String countryCode) {
public Localization(@Nonnull final String languageCode, @Nullable final String countryCode) {
this.languageCode = languageCode;
this.countryCode = countryCode;
}
public Localization(@Nonnull String languageCode) {
public Localization(@Nonnull final String languageCode) {
this(languageCode, null);
}
@Nonnull
public String getLanguageCode() {
return languageCode;
}
@ -61,17 +71,15 @@ public class Localization implements Serializable {
return countryCode == null ? "" : countryCode;
}
public Locale asLocale() {
return new Locale(getLanguageCode(), getCountryCode());
}
public static Localization fromLocale(@Nonnull Locale locale) {
public static Localization fromLocale(@Nonnull final Locale locale) {
return new Localization(locale.getLanguage(), locale.getCountry());
}
/**
* Return a formatted string in the form of: {@code language-Country}, or
* just {@code language} if country is {@code null}.
*
* @return A correctly formatted localizationCode for this localization.
*/
public String getLocalizationCode() {
return languageCode + (countryCode == null ? "" : "-" + countryCode);
@ -83,20 +91,47 @@ public class Localization implements Serializable {
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (!(o instanceof Localization)) return false;
public boolean equals(final Object o) {
if (this == o) {
return true;
}
if (!(o instanceof Localization)) {
return false;
}
Localization that = (Localization) o;
final Localization that = (Localization) o;
if (!languageCode.equals(that.languageCode)) return false;
return countryCode != null ? countryCode.equals(that.countryCode) : that.countryCode == null;
return languageCode.equals(that.languageCode)
&& Objects.equals(countryCode, that.countryCode);
}
@Override
public int hashCode() {
int result = languageCode.hashCode();
result = 31 * result + (countryCode != null ? countryCode.hashCode() : 0);
result = 31 * result + Objects.hashCode(countryCode);
return result;
}
/**
* Converts a three letter language code (ISO 639-2/T) to a Locale
* because limits of Java Locale class.
*
* @param code a three letter language code
* @return the Locale corresponding
*/
public static Locale getLocaleFromThreeLetterCode(@Nonnull final String code)
throws ParsingException {
final String[] languages = Locale.getISOLanguages();
final Map<String, Locale> localeMap = new HashMap<>(languages.length);
for (final String language : languages) {
final Locale locale = new Locale(language);
localeMap.put(locale.getISO3Language(), locale);
}
if (localeMap.containsKey(code)) {
return localeMap.get(code);
} else {
throw new ParsingException(
"Could not get Locale from this three letter language code" + code);
}
}
}

View File

@ -2,21 +2,21 @@ package org.schabi.newpipe.extractor.localization;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.timeago.PatternsHolder;
import org.schabi.newpipe.extractor.timeago.TimeAgoUnit;
import org.schabi.newpipe.extractor.utils.Parser;
import java.util.Calendar;
import java.util.Collection;
import java.time.OffsetDateTime;
import java.time.ZoneOffset;
import java.time.temporal.ChronoUnit;
import java.util.Map;
import java.util.regex.Pattern;
/**
* A helper class that is meant to be used by services that need to parse upload dates in the
* format '2 days ago' or similar.
* A helper class that is meant to be used by services that need to parse durations such as
* {@code 23 seconds} and/or upload dates in the format {@code 2 days ago} or similar.
*/
public class TimeAgoParser {
private final PatternsHolder patternsHolder;
private final Calendar consistentNow;
private final OffsetDateTime now;
/**
* Creates a helper to parse upload dates in the format '2 days ago'.
@ -24,16 +24,31 @@ public class TimeAgoParser {
* Instantiate a new {@link TimeAgoParser} every time you extract a new batch of items.
* </p>
*
* @param patternsHolder An object that holds the "time ago" patterns, special cases, and the language word separator.
* @param patternsHolder An object that holds the "time ago" patterns, special cases, and the
* language word separator.
*/
public TimeAgoParser(PatternsHolder patternsHolder) {
this.patternsHolder = patternsHolder;
consistentNow = Calendar.getInstance();
public TimeAgoParser(final PatternsHolder patternsHolder) {
this(patternsHolder, OffsetDateTime.now(ZoneOffset.UTC));
}
/**
* Parses a textual date in the format '2 days ago' into a Calendar representation which is then wrapped in a
* {@link DateWrapper} object.
* Creates a helper to parse upload dates in the format '2 days ago'.
* <p>
* Instantiate a new {@link TimeAgoParser} every time you extract a new batch of items.
* </p>
*
* @param patternsHolder An object that holds the "time ago" patterns, special cases, and the
* language word separator.
* @param now The current time
*/
public TimeAgoParser(final PatternsHolder patternsHolder, final OffsetDateTime now) {
this.patternsHolder = patternsHolder;
this.now = now;
}
/**
* Parses a textual date in the format '2 days ago' into a Calendar representation which is then
* wrapped in a {@link DateWrapper} object.
* <p>
* Beginning with days ago, the date is considered as an approximation.
*
@ -41,136 +56,96 @@ public class TimeAgoParser {
* @return The parsed time (can be approximated)
* @throws ParsingException if the time unit could not be recognized
*/
public DateWrapper parse(String textualDate) throws ParsingException {
for (Map.Entry<TimeAgoUnit, Map<String, Integer>> caseUnitEntry : patternsHolder.specialCases().entrySet()) {
final TimeAgoUnit timeAgoUnit = caseUnitEntry.getKey();
for (Map.Entry<String, Integer> caseMapToAmountEntry : caseUnitEntry.getValue().entrySet()) {
public DateWrapper parse(final String textualDate) throws ParsingException {
for (final var caseUnitEntry : patternsHolder.specialCases().entrySet()) {
final ChronoUnit chronoUnit = caseUnitEntry.getKey();
for (final var caseMapToAmountEntry : caseUnitEntry.getValue().entrySet()) {
final String caseText = caseMapToAmountEntry.getKey();
final Integer caseAmount = caseMapToAmountEntry.getValue();
final int caseAmount = caseMapToAmountEntry.getValue();
if (textualDateMatches(textualDate, caseText)) {
return getResultFor(caseAmount, timeAgoUnit);
return getResultFor(caseAmount, chronoUnit);
}
}
}
int timeAgoAmount;
return getResultFor(parseTimeAgoAmount(textualDate), parseChronoUnit(textualDate));
}
private int parseTimeAgoAmount(final String textualDate) {
try {
timeAgoAmount = parseTimeAgoAmount(textualDate);
} catch (NumberFormatException e) {
return Integer.parseInt(textualDate.replaceAll("\\D+", ""));
} catch (final NumberFormatException ignored) {
// If there is no valid number in the textual date,
// assume it is 1 (as in 'a second ago').
timeAgoAmount = 1;
return 1;
}
final TimeAgoUnit timeAgoUnit = parseTimeAgoUnit(textualDate);
return getResultFor(timeAgoAmount, timeAgoUnit);
}
private int parseTimeAgoAmount(String textualDate) throws NumberFormatException {
String timeValueStr = textualDate.replaceAll("\\D+", "");
return Integer.parseInt(timeValueStr);
private ChronoUnit parseChronoUnit(final String textualDate) throws ParsingException {
return patternsHolder.asMap().entrySet().stream()
.filter(e -> e.getValue().stream()
.anyMatch(agoPhrase -> textualDateMatches(textualDate, agoPhrase)))
.map(Map.Entry::getKey)
.findFirst()
.orElseThrow(() ->
new ParsingException("Unable to parse the date: " + textualDate));
}
private TimeAgoUnit parseTimeAgoUnit(String textualDate) throws ParsingException {
for (Map.Entry<TimeAgoUnit, Collection<String>> entry : patternsHolder.asMap().entrySet()) {
final TimeAgoUnit timeAgoUnit = entry.getKey();
for (String agoPhrase : entry.getValue()) {
if (textualDateMatches(textualDate, agoPhrase)) {
return timeAgoUnit;
}
}
}
throw new ParsingException("Unable to parse the date: " + textualDate);
}
private boolean textualDateMatches(String textualDate, String agoPhrase) {
private boolean textualDateMatches(final String textualDate, final String agoPhrase) {
if (textualDate.equals(agoPhrase)) {
return true;
}
if (patternsHolder.wordSeparator().isEmpty()) {
return textualDate.toLowerCase().contains(agoPhrase.toLowerCase());
} else {
final String escapedPhrase = Pattern.quote(agoPhrase.toLowerCase());
final String escapedSeparator;
if (patternsHolder.wordSeparator().equals(" ")) {
// From JDK8 \h - Treat horizontal spaces as a normal one (non-breaking space, thin space, etc.)
escapedSeparator = "[ \\t\\xA0\\u1680\\u180e\\u2000-\\u200a\\u202f\\u205f\\u3000]";
} else {
escapedSeparator = Pattern.quote(patternsHolder.wordSeparator());
}
// (^|separator)pattern($|separator)
// Check if the pattern is surrounded by separators or start/end of the string.
final String pattern =
"(^|" + escapedSeparator + ")" + escapedPhrase + "($|" + escapedSeparator + ")";
return Parser.isMatch(pattern, textualDate.toLowerCase());
}
final String escapedPhrase = Pattern.quote(agoPhrase.toLowerCase());
final String escapedSeparator = patternsHolder.wordSeparator().equals(" ")
// From JDK8 \h - Treat horizontal spaces as a normal one
// (non-breaking space, thin space, etc.)
// Also split the string on numbers to be able to parse strings like "2wk"
? "[ \\t\\xA0\\u1680\\u180e\\u2000-\\u200a\\u202f\\u205f\\u3000\\d]"
: Pattern.quote(patternsHolder.wordSeparator());
// (^|separator)pattern($|separator)
// Check if the pattern is surrounded by separators or start/end of the string.
final String pattern =
"(^|" + escapedSeparator + ")" + escapedPhrase + "($|" + escapedSeparator + ")";
return Parser.isMatch(pattern, textualDate.toLowerCase());
}
private DateWrapper getResultFor(int timeAgoAmount, TimeAgoUnit timeAgoUnit) {
final Calendar calendarTime = getNow();
private DateWrapper getResultFor(final int timeAgoAmount, final ChronoUnit chronoUnit) {
OffsetDateTime offsetDateTime = now;
boolean isApproximation = false;
switch (timeAgoUnit) {
switch (chronoUnit) {
case SECONDS:
calendarTime.add(Calendar.SECOND, -timeAgoAmount);
break;
case MINUTES:
calendarTime.add(Calendar.MINUTE, -timeAgoAmount);
break;
case HOURS:
calendarTime.add(Calendar.HOUR_OF_DAY, -timeAgoAmount);
offsetDateTime = offsetDateTime.minus(timeAgoAmount, chronoUnit);
break;
case DAYS:
calendarTime.add(Calendar.DAY_OF_MONTH, -timeAgoAmount);
isApproximation = true;
break;
case WEEKS:
calendarTime.add(Calendar.WEEK_OF_YEAR, -timeAgoAmount);
isApproximation = true;
break;
case MONTHS:
calendarTime.add(Calendar.MONTH, -timeAgoAmount);
offsetDateTime = offsetDateTime.minus(timeAgoAmount, chronoUnit);
isApproximation = true;
break;
case YEARS:
calendarTime.add(Calendar.YEAR, -timeAgoAmount);
// Prevent `PrettyTime` from showing '12 months ago'.
calendarTime.add(Calendar.DAY_OF_MONTH, -1);
// minusDays is needed to prevent `PrettyTime` from showing '12 months ago'.
offsetDateTime = offsetDateTime.minusYears(timeAgoAmount).minusDays(1);
isApproximation = true;
break;
}
if (isApproximation) {
markApproximatedTime(calendarTime);
offsetDateTime = offsetDateTime.truncatedTo(ChronoUnit.HOURS);
}
return new DateWrapper(calendarTime, isApproximation);
}
private Calendar getNow() {
return (Calendar) consistentNow.clone();
}
/**
* Marks the time as approximated by setting minutes, seconds and milliseconds to 0.
*
* @param calendarTime Time to be marked as approximated
*/
private void markApproximatedTime(Calendar calendarTime) {
calendarTime.set(Calendar.MINUTE, 0);
calendarTime.set(Calendar.SECOND, 0);
calendarTime.set(Calendar.MILLISECOND, 0);
return new DateWrapper(offsetDateTime, isApproximation);
}
}

View File

@ -3,17 +3,23 @@ package org.schabi.newpipe.extractor.localization;
import org.schabi.newpipe.extractor.timeago.PatternsHolder;
import org.schabi.newpipe.extractor.timeago.PatternsManager;
import java.time.OffsetDateTime;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
public class TimeAgoPatternsManager {
@Nullable
private static PatternsHolder getPatternsFor(@Nonnull Localization localization) {
return PatternsManager.getPatterns(localization.getLanguageCode(), localization.getCountryCode());
public final class TimeAgoPatternsManager {
private TimeAgoPatternsManager() {
}
@Nullable
public static TimeAgoParser getTimeAgoParserFor(@Nonnull Localization localization) {
private static PatternsHolder getPatternsFor(@Nonnull final Localization localization) {
return PatternsManager.getPatterns(localization.getLanguageCode(),
localization.getCountryCode());
}
@Nullable
public static TimeAgoParser getTimeAgoParserFor(@Nonnull final Localization localization) {
final PatternsHolder holder = getPatternsFor(localization);
if (holder == null) {
@ -22,4 +28,17 @@ public class TimeAgoPatternsManager {
return new TimeAgoParser(holder);
}
@Nullable
public static TimeAgoParser getTimeAgoParserFor(
@Nonnull final Localization localization,
@Nonnull final OffsetDateTime now) {
final PatternsHolder holder = getPatternsFor(localization);
if (holder == null) {
return null;
}
return new TimeAgoParser(holder, now);
}
}

View File

@ -1,30 +1,61 @@
package org.schabi.newpipe.extractor.playlist;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.stream.Description;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import javax.annotation.Nonnull;
import java.util.Collections;
import java.util.List;
public abstract class PlaylistExtractor extends ListExtractor<StreamInfoItem> {
public PlaylistExtractor(StreamingService service, ListLinkHandler linkHandler) {
public PlaylistExtractor(final StreamingService service, final ListLinkHandler linkHandler) {
super(service, linkHandler);
}
public abstract String getThumbnailUrl() throws ParsingException;
public abstract String getBannerUrl() throws ParsingException;
public abstract String getUploaderUrl() throws ParsingException;
public abstract String getUploaderName() throws ParsingException;
public abstract String getUploaderAvatarUrl() throws ParsingException;
@Nonnull
public abstract List<Image> getUploaderAvatars() throws ParsingException;
public abstract boolean isUploaderVerified() throws ParsingException;
public abstract long getStreamCount() throws ParsingException;
@Nonnull public abstract String getSubChannelName() throws ParsingException;
@Nonnull public abstract String getSubChannelUrl() throws ParsingException;
@Nonnull public abstract String getSubChannelAvatarUrl() throws ParsingException;
@Nonnull
public abstract Description getDescription() throws ParsingException;
@Nonnull
public List<Image> getThumbnails() throws ParsingException {
return Collections.emptyList();
}
@Nonnull
public List<Image> getBanners() throws ParsingException {
return List.of();
}
@Nonnull
public String getSubChannelName() throws ParsingException {
return "";
}
@Nonnull
public String getSubChannelUrl() throws ParsingException {
return "";
}
@Nonnull
public List<Image> getSubChannelAvatars() throws ParsingException {
return List.of();
}
public PlaylistInfo.PlaylistType getPlaylistType() throws ParsingException {
return PlaylistInfo.PlaylistType.NORMAL;
}
}

View File

@ -1,12 +1,15 @@
package org.schabi.newpipe.extractor.playlist;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.ListExtractor.InfoItemsPage;
import org.schabi.newpipe.extractor.ListInfo;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.stream.Description;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import org.schabi.newpipe.extractor.utils.ExtractorHelper;
@ -14,26 +17,71 @@ import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
public class PlaylistInfo extends ListInfo<StreamInfoItem> {
import javax.annotation.Nonnull;
private PlaylistInfo(int serviceId, ListLinkHandler linkHandler, String name) throws ParsingException {
public final class PlaylistInfo extends ListInfo<StreamInfoItem> {
/**
* Mixes are handled as particular playlists in NewPipeExtractor. {@link PlaylistType#NORMAL} is
* for non-mixes, while other values are for the different types of mixes. The type of a mix
* depends on how its contents are autogenerated.
*/
public enum PlaylistType {
/**
* A normal playlist (not a mix)
*/
NORMAL,
/**
* A mix made only of streams related to a particular stream, for example YouTube mixes
*/
MIX_STREAM,
/**
* A mix made only of music streams related to a particular stream, for example YouTube
* music mixes
*/
MIX_MUSIC,
/**
* A mix made only of streams from (or related to) the same channel, for example YouTube
* channel mixes
*
* @deprecated There is currently no service that implements this.
* YouTube removed all playlists with this type around 2024-06
*/
@Deprecated
MIX_CHANNEL,
/**
* A mix made only of streams related to a particular (musical) genre, for example YouTube
* genre mixes
*/
MIX_GENRE,
}
@SuppressWarnings("RedundantThrows")
private PlaylistInfo(final int serviceId, final ListLinkHandler linkHandler, final String name)
throws ParsingException {
super(serviceId, linkHandler, name);
}
public static PlaylistInfo getInfo(String url) throws IOException, ExtractionException {
public static PlaylistInfo getInfo(final String url) throws IOException, ExtractionException {
return getInfo(NewPipe.getServiceByUrl(url), url);
}
public static PlaylistInfo getInfo(StreamingService service, String url) throws IOException, ExtractionException {
PlaylistExtractor extractor = service.getPlaylistExtractor(url);
public static PlaylistInfo getInfo(final StreamingService service, final String url)
throws IOException, ExtractionException {
final PlaylistExtractor extractor = service.getPlaylistExtractor(url);
extractor.fetchPage();
return getInfo(extractor);
}
public static InfoItemsPage<StreamInfoItem> getMoreItems(StreamingService service,
String url,
String pageUrl) throws IOException, ExtractionException {
return service.getPlaylistExtractor(url).getPage(pageUrl);
public static InfoItemsPage<StreamInfoItem> getMoreItems(final StreamingService service,
final String url,
final Page page)
throws IOException, ExtractionException {
return service.getPlaylistExtractor(url).getPage(page);
}
/**
@ -41,7 +89,8 @@ public class PlaylistInfo extends ListInfo<StreamInfoItem> {
*
* @param extractor an extractor where fetchPage() was already got called on.
*/
public static PlaylistInfo getInfo(PlaylistExtractor extractor) throws ExtractionException {
public static PlaylistInfo getInfo(final PlaylistExtractor extractor)
throws ExtractionException {
final PlaylistInfo info = new PlaylistInfo(
extractor.getServiceId(),
@ -49,105 +98,122 @@ public class PlaylistInfo extends ListInfo<StreamInfoItem> {
extractor.getName());
// collect uploader extraction failures until we are sure this is not
// just a playlist without an uploader
List<Throwable> uploaderParsingErrors = new ArrayList<Throwable>(3);
final List<Throwable> uploaderParsingErrors = new ArrayList<>();
try {
info.setOriginalUrl(extractor.getOriginalUrl());
} catch (Exception e) {
} catch (final Exception e) {
info.addError(e);
}
try {
info.setStreamCount(extractor.getStreamCount());
} catch (Exception e) {
} catch (final Exception e) {
info.addError(e);
}
try {
info.setThumbnailUrl(extractor.getThumbnailUrl());
} catch (Exception e) {
info.setDescription(extractor.getDescription());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setThumbnails(extractor.getThumbnails());
} catch (final Exception e) {
info.addError(e);
}
try {
info.setUploaderUrl(extractor.getUploaderUrl());
} catch (Exception e) {
info.setUploaderUrl("");
} catch (final Exception e) {
uploaderParsingErrors.add(e);
}
try {
info.setUploaderName(extractor.getUploaderName());
} catch (Exception e) {
info.setUploaderName("");
} catch (final Exception e) {
uploaderParsingErrors.add(e);
}
try {
info.setUploaderAvatarUrl(extractor.getUploaderAvatarUrl());
} catch (Exception e) {
info.setUploaderAvatarUrl("");
info.setUploaderAvatars(extractor.getUploaderAvatars());
} catch (final Exception e) {
uploaderParsingErrors.add(e);
}
try {
info.setSubChannelUrl(extractor.getSubChannelUrl());
} catch (Exception e) {
} catch (final Exception e) {
uploaderParsingErrors.add(e);
}
try {
info.setSubChannelName(extractor.getSubChannelName());
} catch (Exception e) {
} catch (final Exception e) {
uploaderParsingErrors.add(e);
}
try {
info.setSubChannelAvatarUrl(extractor.getSubChannelAvatarUrl());
} catch (Exception e) {
info.setSubChannelAvatars(extractor.getSubChannelAvatars());
} catch (final Exception e) {
uploaderParsingErrors.add(e);
}
try {
info.setBannerUrl(extractor.getBannerUrl());
} catch (Exception e) {
info.setBanners(extractor.getBanners());
} catch (final Exception e) {
info.addError(e);
}
// do not fail if everything but the uploader infos could be collected
if (uploaderParsingErrors.size() > 0 &&
(!info.getErrors().isEmpty() || uploaderParsingErrors.size() < 3)) {
try {
info.setPlaylistType(extractor.getPlaylistType());
} catch (final Exception e) {
info.addError(e);
}
// do not fail if everything but the uploader infos could be collected (TODO better comment)
if (!uploaderParsingErrors.isEmpty()
&& (!info.getErrors().isEmpty() || uploaderParsingErrors.size() < 3)) {
info.addAllErrors(uploaderParsingErrors);
}
final InfoItemsPage<StreamInfoItem> itemsPage = ExtractorHelper.getItemsPageOrLogError(info, extractor);
final InfoItemsPage<StreamInfoItem> itemsPage
= ExtractorHelper.getItemsPageOrLogError(info, extractor);
info.setRelatedItems(itemsPage.getItems());
info.setNextPageUrl(itemsPage.getNextPageUrl());
info.setNextPage(itemsPage.getNextPage());
return info;
}
private String thumbnailUrl;
private String bannerUrl;
private String uploaderUrl;
private String uploaderName;
private String uploaderAvatarUrl;
private String uploaderUrl = "";
private String uploaderName = "";
private String subChannelUrl;
private String subChannelName;
private String subChannelAvatarUrl;
private long streamCount = 0;
private Description description;
@Nonnull
private List<Image> banners = List.of();
@Nonnull
private List<Image> subChannelAvatars = List.of();
@Nonnull
private List<Image> thumbnails = List.of();
@Nonnull
private List<Image> uploaderAvatars = List.of();
private long streamCount;
private PlaylistType playlistType;
public String getThumbnailUrl() {
return thumbnailUrl;
@Nonnull
public List<Image> getThumbnails() {
return thumbnails;
}
public void setThumbnailUrl(String thumbnailUrl) {
this.thumbnailUrl = thumbnailUrl;
public void setThumbnails(@Nonnull final List<Image> thumbnails) {
this.thumbnails = thumbnails;
}
public String getBannerUrl() {
return bannerUrl;
@Nonnull
public List<Image> getBanners() {
return banners;
}
public void setBannerUrl(String bannerUrl) {
this.bannerUrl = bannerUrl;
public void setBanners(@Nonnull final List<Image> banners) {
this.banners = banners;
}
public String getUploaderUrl() {
return uploaderUrl;
}
public void setUploaderUrl(String uploaderUrl) {
public void setUploaderUrl(final String uploaderUrl) {
this.uploaderUrl = uploaderUrl;
}
@ -155,23 +221,24 @@ public class PlaylistInfo extends ListInfo<StreamInfoItem> {
return uploaderName;
}
public void setUploaderName(String uploaderName) {
public void setUploaderName(final String uploaderName) {
this.uploaderName = uploaderName;
}
public String getUploaderAvatarUrl() {
return uploaderAvatarUrl;
@Nonnull
public List<Image> getUploaderAvatars() {
return uploaderAvatars;
}
public void setUploaderAvatarUrl(String uploaderAvatarUrl) {
this.uploaderAvatarUrl = uploaderAvatarUrl;
public void setUploaderAvatars(@Nonnull final List<Image> uploaderAvatars) {
this.uploaderAvatars = uploaderAvatars;
}
public String getSubChannelUrl() {
return subChannelUrl;
}
public void setSubChannelUrl(String subChannelUrl) {
public void setSubChannelUrl(final String subChannelUrl) {
this.subChannelUrl = subChannelUrl;
}
@ -179,23 +246,40 @@ public class PlaylistInfo extends ListInfo<StreamInfoItem> {
return subChannelName;
}
public void setSubChannelName(String subChannelName) {
public void setSubChannelName(final String subChannelName) {
this.subChannelName = subChannelName;
}
public String getSubChannelAvatarUrl() {
return subChannelAvatarUrl;
@Nonnull
public List<Image> getSubChannelAvatars() {
return subChannelAvatars;
}
public void setSubChannelAvatarUrl(String subChannelAvatarUrl) {
this.subChannelAvatarUrl = subChannelAvatarUrl;
public void setSubChannelAvatars(@Nonnull final List<Image> subChannelAvatars) {
this.subChannelAvatars = subChannelAvatars;
}
public long getStreamCount() {
return streamCount;
}
public void setStreamCount(long streamCount) {
public void setStreamCount(final long streamCount) {
this.streamCount = streamCount;
}
public Description getDescription() {
return description;
}
public void setDescription(final Description description) {
this.description = description;
}
public PlaylistType getPlaylistType() {
return playlistType;
}
public void setPlaylistType(final PlaylistType playlistType) {
this.playlistType = playlistType;
}
}

View File

@ -1,16 +1,23 @@
package org.schabi.newpipe.extractor.playlist;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.stream.Description;
import javax.annotation.Nullable;
public class PlaylistInfoItem extends InfoItem {
private String uploaderName;
private String uploaderUrl;
private boolean uploaderVerified;
/**
* How many streams this playlist have
*/
private long streamCount = 0;
private Description description;
private PlaylistInfo.PlaylistType playlistType;
public PlaylistInfoItem(int serviceId, String url, String name) {
public PlaylistInfoItem(final int serviceId, final String url, final String name) {
super(InfoType.PLAYLIST, serviceId, url, name);
}
@ -18,15 +25,48 @@ public class PlaylistInfoItem extends InfoItem {
return uploaderName;
}
public void setUploaderName(String uploader_name) {
this.uploaderName = uploader_name;
public void setUploaderName(final String uploaderName) {
this.uploaderName = uploaderName;
}
@Nullable
public String getUploaderUrl() {
return uploaderUrl;
}
public void setUploaderUrl(@Nullable final String uploaderUrl) {
this.uploaderUrl = uploaderUrl;
}
public boolean isUploaderVerified() {
return uploaderVerified;
}
public void setUploaderVerified(final boolean uploaderVerified) {
this.uploaderVerified = uploaderVerified;
}
public long getStreamCount() {
return streamCount;
}
public void setStreamCount(long stream_count) {
this.streamCount = stream_count;
public void setStreamCount(final long streamCount) {
this.streamCount = streamCount;
}
public Description getDescription() {
return description;
}
public void setDescription(final Description description) {
this.description = description;
}
public PlaylistInfo.PlaylistType getPlaylistType() {
return playlistType;
}
public void setPlaylistType(final PlaylistInfo.PlaylistType playlistType) {
this.playlistType = playlistType;
}
}

View File

@ -2,20 +2,52 @@ package org.schabi.newpipe.extractor.playlist;
import org.schabi.newpipe.extractor.InfoItemExtractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.stream.Description;
import javax.annotation.Nonnull;
public interface PlaylistInfoItemExtractor extends InfoItemExtractor {
/**
* Get the uploader name
* @return the uploader name
* @throws ParsingException
*/
String getUploaderName() throws ParsingException;
/**
* Get the uploader url
* @return the uploader url
*/
String getUploaderUrl() throws ParsingException;
/**
* Get whether the uploader is verified
* @return whether the uploader is verified
*/
boolean isUploaderVerified() throws ParsingException;
/**
* Get the number of streams
* @return the number of streams
* @throws ParsingException
*/
long getStreamCount() throws ParsingException;
/**
* Get the description of the playlist if there is any.
* Otherwise, an {@link Description#EMPTY_DESCRIPTION EMPTY_DESCRIPTION} is returned.
* @return the playlist's description
*/
@Nonnull
default Description getDescription() throws ParsingException {
return Description.EMPTY_DESCRIPTION;
}
/**
* @return the type of this playlist, see {@link PlaylistInfo.PlaylistType} for a description
* of types. If not overridden always returns {@link PlaylistInfo.PlaylistType#NORMAL}.
*/
@Nonnull
default PlaylistInfo.PlaylistType getPlaylistType() throws ParsingException {
return PlaylistInfo.PlaylistType.NORMAL;
}
}

View File

@ -3,34 +3,52 @@ package org.schabi.newpipe.extractor.playlist;
import org.schabi.newpipe.extractor.InfoItemsCollector;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
public class PlaylistInfoItemsCollector extends InfoItemsCollector<PlaylistInfoItem, PlaylistInfoItemExtractor> {
public class PlaylistInfoItemsCollector
extends InfoItemsCollector<PlaylistInfoItem, PlaylistInfoItemExtractor> {
public PlaylistInfoItemsCollector(int serviceId) {
public PlaylistInfoItemsCollector(final int serviceId) {
super(serviceId);
}
@Override
public PlaylistInfoItem extract(PlaylistInfoItemExtractor extractor) throws ParsingException {
String name = extractor.getName();
int serviceId = getServiceId();
String url = extractor.getUrl();
PlaylistInfoItem resultItem = new PlaylistInfoItem(serviceId, url, name);
public PlaylistInfoItem extract(final PlaylistInfoItemExtractor extractor)
throws ParsingException {
final PlaylistInfoItem resultItem = new PlaylistInfoItem(
getServiceId(), extractor.getUrl(), extractor.getName());
try {
resultItem.setUploaderName(extractor.getUploaderName());
} catch (Exception e) {
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setThumbnailUrl(extractor.getThumbnailUrl());
} catch (Exception e) {
resultItem.setUploaderUrl(extractor.getUploaderUrl());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setUploaderVerified(extractor.isUploaderVerified());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setThumbnails(extractor.getThumbnails());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setStreamCount(extractor.getStreamCount());
} catch (Exception e) {
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setDescription(extractor.getDescription());
} catch (final Exception e) {
addError(e);
}
try {
resultItem.setPlaylistType(extractor.getPlaylistType());
} catch (final Exception e) {
addError(e);
}
return resultItem;

View File

@ -2,22 +2,24 @@ package org.schabi.newpipe.extractor.search;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.MetaInfo;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandler;
import javax.annotation.Nonnull;
import java.util.List;
public abstract class SearchExtractor extends ListExtractor<InfoItem> {
public static class NothingFoundException extends ExtractionException {
public NothingFoundException(String message) {
public NothingFoundException(final String message) {
super(message);
}
}
public SearchExtractor(StreamingService service, SearchQueryHandler linkHandler) {
public SearchExtractor(final StreamingService service, final SearchQueryHandler linkHandler) {
super(service, linkHandler);
}
@ -32,11 +34,11 @@ public abstract class SearchExtractor extends ListExtractor<InfoItem> {
* {@link SearchExtractor#isCorrectedSearch()} is true.
*
* @return a suggestion to another query, the corrected query, or an empty String.
* @throws ParsingException
*/
@Nonnull
public abstract String getSearchSuggestion() throws ParsingException;
@Nonnull
@Override
public SearchQueryHandler getLinkHandler() {
return (SearchQueryHandler) super.getLinkHandler();
@ -57,4 +59,14 @@ public abstract class SearchExtractor extends ListExtractor<InfoItem> {
* @return whether the results comes from a corrected query or not.
*/
public abstract boolean isCorrectedSearch() throws ParsingException;
/**
* Meta information about the search query.
* <p>
* Example: on YouTube, if you search for "Covid-19",
* there is a box with information from the WHO about Covid-19 and a link to the WHO's website.
* @return additional meta information about the search query
*/
@Nonnull
public abstract List<MetaInfo> getMetaInfo() throws ParsingException;
}

View File

@ -3,35 +3,42 @@ package org.schabi.newpipe.extractor.search;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.ListInfo;
import org.schabi.newpipe.extractor.MetaInfo;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandler;
import org.schabi.newpipe.extractor.utils.ExtractorHelper;
import java.io.IOException;
import java.util.List;
import javax.annotation.Nonnull;
public class SearchInfo extends ListInfo<InfoItem> {
private String searchString;
private final String searchString;
private String searchSuggestion;
private boolean isCorrectedSearch;
private List<MetaInfo> metaInfo = List.of();
public SearchInfo(int serviceId,
SearchQueryHandler qIHandler,
String searchString) {
public SearchInfo(final int serviceId,
final SearchQueryHandler qIHandler,
final String searchString) {
super(serviceId, qIHandler, "Search");
this.searchString = searchString;
}
public static SearchInfo getInfo(StreamingService service, SearchQueryHandler searchQuery) throws ExtractionException, IOException {
SearchExtractor extractor = service.getSearchExtractor(searchQuery);
public static SearchInfo getInfo(final StreamingService service,
final SearchQueryHandler searchQuery)
throws ExtractionException, IOException {
final SearchExtractor extractor = service.getSearchExtractor(searchQuery);
extractor.fetchPage();
return getInfo(extractor);
}
public static SearchInfo getInfo(SearchExtractor extractor) throws ExtractionException, IOException {
public static SearchInfo getInfo(final SearchExtractor extractor)
throws ExtractionException, IOException {
final SearchInfo info = new SearchInfo(
extractor.getServiceId(),
extractor.getLinkHandler(),
@ -39,33 +46,39 @@ public class SearchInfo extends ListInfo<InfoItem> {
try {
info.setOriginalUrl(extractor.getOriginalUrl());
} catch (Exception e) {
} catch (final Exception e) {
info.addError(e);
}
try {
info.setSearchSuggestion(extractor.getSearchSuggestion());
} catch (Exception e) {
} catch (final Exception e) {
info.addError(e);
}
try {
info.setIsCorrectedSearch(extractor.isCorrectedSearch());
} catch (Exception e) {
} catch (final Exception e) {
info.addError(e);
}
try {
info.setMetaInfo(extractor.getMetaInfo());
} catch (final Exception e) {
info.addError(e);
}
ListExtractor.InfoItemsPage<InfoItem> page = ExtractorHelper.getItemsPageOrLogError(info, extractor);
final ListExtractor.InfoItemsPage<InfoItem> page
= ExtractorHelper.getItemsPageOrLogError(info, extractor);
info.setRelatedItems(page.getItems());
info.setNextPageUrl(page.getNextPageUrl());
info.setNextPage(page.getNextPage());
return info;
}
public static ListExtractor.InfoItemsPage<InfoItem> getMoreItems(StreamingService service,
SearchQueryHandler query,
String pageUrl)
public static ListExtractor.InfoItemsPage<InfoItem> getMoreItems(final StreamingService service,
final SearchQueryHandler query,
final Page page)
throws IOException, ExtractionException {
return service.getSearchExtractor(query).getPage(pageUrl);
return service.getSearchExtractor(query).getPage(page);
}
// Getter
@ -81,11 +94,20 @@ public class SearchInfo extends ListInfo<InfoItem> {
return this.isCorrectedSearch;
}
public void setIsCorrectedSearch(boolean isCorrectedSearch) {
public void setIsCorrectedSearch(final boolean isCorrectedSearch) {
this.isCorrectedSearch = isCorrectedSearch;
}
public void setSearchSuggestion(String searchSuggestion) {
public void setSearchSuggestion(final String searchSuggestion) {
this.searchSuggestion = searchSuggestion;
}
@Nonnull
public List<MetaInfo> getMetaInfo() {
return metaInfo;
}
public void setMetaInfo(@Nonnull final List<MetaInfo> metaInfo) {
this.metaInfo = metaInfo;
}
}

View File

@ -0,0 +1,174 @@
// Created by Fynn Godau 2019, licensed GNU GPL version 3 or later
package org.schabi.newpipe.extractor.services.bandcamp;
import static org.schabi.newpipe.extractor.StreamingService.ServiceInfo.MediaCapability.AUDIO;
import static org.schabi.newpipe.extractor.StreamingService.ServiceInfo.MediaCapability.COMMENTS;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_URL;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampFeaturedExtractor.FEATURED_API_URL;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampFeaturedExtractor.KIOSK_FEATURED;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampRadioExtractor.KIOSK_RADIO;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampRadioExtractor.RADIO_API_URL;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.ChannelExtractor;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import org.schabi.newpipe.extractor.comments.CommentsExtractor;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.kiosk.KioskList;
import org.schabi.newpipe.extractor.linkhandler.LinkHandler;
import org.schabi.newpipe.extractor.linkhandler.LinkHandlerFactory;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandlerFactory;
import org.schabi.newpipe.extractor.linkhandler.ReadyChannelTabListLinkHandler;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandler;
import org.schabi.newpipe.extractor.linkhandler.SearchQueryHandlerFactory;
import org.schabi.newpipe.extractor.playlist.PlaylistExtractor;
import org.schabi.newpipe.extractor.search.SearchExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampChannelExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampChannelTabExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampCommentsExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampFeaturedExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampPlaylistExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampRadioExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampRadioStreamExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampSearchExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampStreamExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampSuggestionExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampChannelLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampChannelTabLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampCommentsLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampFeaturedLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampPlaylistLinkHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampSearchQueryHandlerFactory;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampStreamLinkHandlerFactory;
import org.schabi.newpipe.extractor.stream.StreamExtractor;
import org.schabi.newpipe.extractor.subscription.SubscriptionExtractor;
import org.schabi.newpipe.extractor.suggestion.SuggestionExtractor;
import java.util.Arrays;
public class BandcampService extends StreamingService {
public BandcampService(final int id) {
super(id, "Bandcamp", Arrays.asList(AUDIO, COMMENTS));
}
@Override
public String getBaseUrl() {
return BASE_URL;
}
@Override
public LinkHandlerFactory getStreamLHFactory() {
return BandcampStreamLinkHandlerFactory.getInstance();
}
@Override
public ListLinkHandlerFactory getChannelLHFactory() {
return BandcampChannelLinkHandlerFactory.getInstance();
}
@Override
public ListLinkHandlerFactory getChannelTabLHFactory() {
return BandcampChannelTabLinkHandlerFactory.getInstance();
}
@Override
public ListLinkHandlerFactory getPlaylistLHFactory() {
return BandcampPlaylistLinkHandlerFactory.getInstance();
}
@Override
public SearchQueryHandlerFactory getSearchQHFactory() {
return BandcampSearchQueryHandlerFactory.getInstance();
}
@Override
public ListLinkHandlerFactory getCommentsLHFactory() {
return BandcampCommentsLinkHandlerFactory.getInstance();
}
@Override
public SearchExtractor getSearchExtractor(final SearchQueryHandler queryHandler) {
return new BandcampSearchExtractor(this, queryHandler);
}
@Override
public SuggestionExtractor getSuggestionExtractor() {
return new BandcampSuggestionExtractor(this);
}
@Override
public SubscriptionExtractor getSubscriptionExtractor() {
return null;
}
@Override
public KioskList getKioskList() throws ExtractionException {
final KioskList kioskList = new KioskList(this);
final ListLinkHandlerFactory h = BandcampFeaturedLinkHandlerFactory.getInstance();
try {
kioskList.addKioskEntry(
(streamingService, url, kioskId) -> new BandcampFeaturedExtractor(
BandcampService.this,
h.fromUrl(FEATURED_API_URL),
kioskId
),
h,
KIOSK_FEATURED
);
kioskList.addKioskEntry(
(streamingService, url, kioskId) -> new BandcampRadioExtractor(
BandcampService.this,
h.fromUrl(RADIO_API_URL),
kioskId
),
h,
KIOSK_RADIO
);
kioskList.setDefaultKiosk(KIOSK_FEATURED);
} catch (final Exception e) {
throw new ExtractionException(e);
}
return kioskList;
}
@Override
public ChannelExtractor getChannelExtractor(final ListLinkHandler linkHandler) {
return new BandcampChannelExtractor(this, linkHandler);
}
@Override
public ChannelTabExtractor getChannelTabExtractor(final ListLinkHandler linkHandler) {
if (linkHandler instanceof ReadyChannelTabListLinkHandler) {
return ((ReadyChannelTabListLinkHandler) linkHandler).getChannelTabExtractor(this);
} else {
return new BandcampChannelTabExtractor(this, linkHandler);
}
}
@Override
public PlaylistExtractor getPlaylistExtractor(final ListLinkHandler linkHandler) {
return new BandcampPlaylistExtractor(this, linkHandler);
}
@Override
public StreamExtractor getStreamExtractor(final LinkHandler linkHandler) {
if (BandcampExtractorHelper.isRadioUrl(linkHandler.getUrl())) {
return new BandcampRadioStreamExtractor(this, linkHandler);
}
return new BandcampStreamExtractor(this, linkHandler);
}
@Override
public CommentsExtractor getCommentsExtractor(final ListLinkHandler linkHandler) {
return new BandcampCommentsExtractor(this, linkHandler);
}
}

View File

@ -0,0 +1,62 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.ListExtractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.playlist.PlaylistInfoItemExtractor;
import java.util.List;
import javax.annotation.Nonnull;
public class BandcampAlbumInfoItemExtractor implements PlaylistInfoItemExtractor {
private final JsonObject albumInfoItem;
private final String uploaderUrl;
public BandcampAlbumInfoItemExtractor(final JsonObject albumInfoItem,
final String uploaderUrl) {
this.albumInfoItem = albumInfoItem;
this.uploaderUrl = uploaderUrl;
}
@Override
public String getName() throws ParsingException {
return albumInfoItem.getString("title");
}
@Override
public String getUrl() throws ParsingException {
return BandcampExtractorHelper.getStreamUrlFromIds(
albumInfoItem.getLong("band_id"),
albumInfoItem.getLong("item_id"),
albumInfoItem.getString("item_type"));
}
@Nonnull
@Override
public List<Image> getThumbnails() throws ParsingException {
return BandcampExtractorHelper.getImagesFromImageId(albumInfoItem.getLong("art_id"), true);
}
@Override
public String getUploaderName() throws ParsingException {
return albumInfoItem.getString("band_name");
}
@Override
public String getUploaderUrl() {
return uploaderUrl;
}
@Override
public boolean isUploaderVerified() {
return false;
}
@Override
public long getStreamCount() {
return ListExtractor.ITEM_COUNT_UNKNOWN;
}
}

View File

@ -0,0 +1,192 @@
// Created by Fynn Godau 2019, licensed GNU GPL version 3 or later
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import static org.schabi.newpipe.extractor.Image.HEIGHT_UNKNOWN;
import static org.schabi.newpipe.extractor.Image.WIDTH_UNKNOWN;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getArtistDetails;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
import static org.schabi.newpipe.extractor.utils.Utils.replaceHttpWithHttps;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import org.jsoup.Jsoup;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.Image.ResolutionLevel;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.ChannelExtractor;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.exceptions.ReCaptchaException;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabs;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.linkhandler.ReadyChannelTabListLinkHandler;
import org.schabi.newpipe.extractor.services.bandcamp.linkHandler.BandcampChannelTabLinkHandlerFactory;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Objects;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import javax.annotation.Nonnull;
public class BandcampChannelExtractor extends ChannelExtractor {
private JsonObject channelInfo;
public BandcampChannelExtractor(final StreamingService service,
final ListLinkHandler linkHandler) {
super(service, linkHandler);
}
@Nonnull
@Override
public List<Image> getAvatars() {
return getImagesFromImageId(channelInfo.getLong("bio_image_id"), false);
}
@Nonnull
@Override
public List<Image> getBanners() throws ParsingException {
/*
* Mobile API does not return the header or not the correct header.
* Therefore, we need to query the website
*/
try {
final String html = getDownloader()
.get(replaceHttpWithHttps(channelInfo.getString("bandcamp_url")))
.responseBody();
return Stream.of(Jsoup.parse(html).getElementById("customHeader"))
.filter(Objects::nonNull)
.flatMap(element -> element.getElementsByTag("img").stream())
.map(element -> element.attr("src"))
.filter(url -> !url.isEmpty())
.map(url -> new Image(
replaceHttpWithHttps(url), HEIGHT_UNKNOWN, WIDTH_UNKNOWN,
ResolutionLevel.UNKNOWN))
.collect(Collectors.toUnmodifiableList());
} catch (final IOException | ReCaptchaException e) {
throw new ParsingException("Could not download artist web site", e);
}
}
/**
* Bandcamp discontinued their RSS feeds because it hadn't been used enough.
*/
@Override
public String getFeedUrl() {
return null;
}
@Override
public long getSubscriberCount() {
return -1;
}
@Override
public String getDescription() {
return channelInfo.getString("bio");
}
@Override
public String getParentChannelName() {
return null;
}
@Override
public String getParentChannelUrl() {
return null;
}
@Nonnull
@Override
public List<Image> getParentChannelAvatars() {
return List.of();
}
@Override
public boolean isVerified() throws ParsingException {
return false;
}
@Nonnull
@Override
public List<ListLinkHandler> getTabs() throws ParsingException {
final JsonArray discography = channelInfo.getArray("discography");
final TabExtractorBuilder builder = new TabExtractorBuilder(discography);
final List<ListLinkHandler> tabs = new ArrayList<>();
boolean foundTrackItem = false;
boolean foundAlbumItem = false;
for (final Object discographyItem : discography) {
if (foundTrackItem && foundAlbumItem) {
break;
}
if (!(discographyItem instanceof JsonObject)) {
continue;
}
final JsonObject discographyJsonItem = (JsonObject) discographyItem;
final String itemType = discographyJsonItem.getString("item_type");
if (!foundTrackItem && "track".equals(itemType)) {
foundTrackItem = true;
tabs.add(new ReadyChannelTabListLinkHandler(getUrl()
+ BandcampChannelTabLinkHandlerFactory.getUrlSuffix(ChannelTabs.TRACKS),
getId(),
ChannelTabs.TRACKS,
builder));
}
if (!foundAlbumItem && "album".equals(itemType)) {
foundAlbumItem = true;
tabs.add(new ReadyChannelTabListLinkHandler(getUrl()
+ BandcampChannelTabLinkHandlerFactory.getUrlSuffix(ChannelTabs.ALBUMS),
getId(),
ChannelTabs.ALBUMS,
builder));
}
}
return Collections.unmodifiableList(tabs);
}
@Override
public void onFetchPage(@Nonnull final Downloader downloader)
throws IOException, ExtractionException {
channelInfo = getArtistDetails(getId());
}
@Nonnull
@Override
public String getName() {
return channelInfo.getString("name");
}
private static final class TabExtractorBuilder
implements ReadyChannelTabListLinkHandler.ChannelTabExtractorBuilder {
private final JsonArray discography;
TabExtractorBuilder(final JsonArray discography) {
this.discography = discography;
}
@Nonnull
@Override
public ChannelTabExtractor build(@Nonnull final StreamingService service,
@Nonnull final ListLinkHandler linkHandler) {
return BandcampChannelTabExtractor.fromDiscography(service, linkHandler, discography);
}
}
}

View File

@ -0,0 +1,60 @@
// Created by Fynn Godau 2019, licensed GNU GPL version 3 or later
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import org.jsoup.nodes.Element;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.channel.ChannelInfoItemExtractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import javax.annotation.Nonnull;
import java.util.List;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromSearchResult;
public class BandcampChannelInfoItemExtractor implements ChannelInfoItemExtractor {
private final Element resultInfo;
private final Element searchResult;
public BandcampChannelInfoItemExtractor(final Element searchResult) {
this.searchResult = searchResult;
resultInfo = searchResult.getElementsByClass("result-info").first();
}
@Override
public String getName() throws ParsingException {
return resultInfo.getElementsByClass("heading").text();
}
@Override
public String getUrl() throws ParsingException {
return resultInfo.getElementsByClass("itemurl").text();
}
@Nonnull
@Override
public List<Image> getThumbnails() throws ParsingException {
return getImagesFromSearchResult(searchResult);
}
@Override
public String getDescription() {
return resultInfo.getElementsByClass("subhead").text();
}
@Override
public long getSubscriberCount() {
return -1;
}
@Override
public long getStreamCount() {
return -1;
}
@Override
public boolean isVerified() throws ParsingException {
return false;
}
}

View File

@ -0,0 +1,95 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.InfoItem;
import org.schabi.newpipe.extractor.MultiInfoItemsCollector;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabExtractor;
import org.schabi.newpipe.extractor.channel.tabs.ChannelTabs;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.streaminfoitem.BandcampDiscographStreamInfoItemExtractor;
import javax.annotation.Nonnull;
import java.io.IOException;
public class BandcampChannelTabExtractor extends ChannelTabExtractor {
private JsonArray discography;
private final String filter;
public BandcampChannelTabExtractor(final StreamingService service,
final ListLinkHandler linkHandler) {
super(service, linkHandler);
final String tab = linkHandler.getContentFilters().get(0);
switch (tab) {
case ChannelTabs.TRACKS:
filter = "track";
break;
case ChannelTabs.ALBUMS:
filter = "album";
break;
default:
throw new IllegalArgumentException("Unsupported channel tab: " + tab);
}
}
public static BandcampChannelTabExtractor fromDiscography(final StreamingService service,
final ListLinkHandler linkHandler,
final JsonArray discography) {
final BandcampChannelTabExtractor tabExtractor =
new BandcampChannelTabExtractor(service, linkHandler);
tabExtractor.discography = discography;
return tabExtractor;
}
@Override
public void onFetchPage(@Nonnull final Downloader downloader) throws ParsingException {
if (discography == null) {
discography = BandcampExtractorHelper.getArtistDetails(getId())
.getArray("discography");
}
}
@Nonnull
@Override
public InfoItemsPage<InfoItem> getInitialPage() throws IOException, ExtractionException {
final MultiInfoItemsCollector collector = new MultiInfoItemsCollector(getServiceId());
for (final Object discograph : discography) {
// A discograph is as an item appears in a discography
if (!(discograph instanceof JsonObject)) {
continue;
}
final JsonObject discographJsonObject = (JsonObject) discograph;
final String itemType = discographJsonObject.getString("item_type", "");
if (!itemType.equals(filter)) {
continue;
}
switch (itemType) {
case "track":
collector.commit(new BandcampDiscographStreamInfoItemExtractor(
discographJsonObject, getUrl()));
break;
case "album":
collector.commit(new BandcampAlbumInfoItemExtractor(
discographJsonObject, getUrl()));
break;
}
}
return new InfoItemsPage<>(collector, null);
}
@Override
public InfoItemsPage<InfoItem> getPage(final Page page) {
return null;
}
}

View File

@ -0,0 +1,136 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_API_URL;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonWriter;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.comments.CommentsExtractor;
import org.schabi.newpipe.extractor.comments.CommentsInfoItem;
import org.schabi.newpipe.extractor.comments.CommentsInfoItemsCollector;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.exceptions.ReCaptchaException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.utils.JsonUtils;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.Collections;
import java.util.List;
import javax.annotation.Nonnull;
public class BandcampCommentsExtractor extends CommentsExtractor {
private static final String REVIEWS_API_URL = BASE_API_URL + "/tralbumcollectors/2/reviews";
private Document document;
public BandcampCommentsExtractor(final StreamingService service,
final ListLinkHandler linkHandler) {
super(service, linkHandler);
}
@Override
public void onFetchPage(@Nonnull final Downloader downloader)
throws IOException, ExtractionException {
document = Jsoup.parse(downloader.get(getLinkHandler().getUrl()).responseBody());
}
@Nonnull
@Override
public InfoItemsPage<CommentsInfoItem> getInitialPage()
throws IOException, ExtractionException {
final CommentsInfoItemsCollector collector = new CommentsInfoItemsCollector(getServiceId());
final JsonObject collectorsData = JsonUtils.toJsonObject(
document.getElementById("collectors-data").attr("data-blob"));
final JsonArray reviews = collectorsData.getArray("reviews");
for (final Object review : reviews) {
collector.commit(
new BandcampCommentsInfoItemExtractor((JsonObject) review, getUrl()));
}
if (!collectorsData.getBoolean("more_reviews_available")) {
return new InfoItemsPage<>(collector, null);
}
final String trackId = getTrackId();
final String token = getNextPageToken(reviews);
return new InfoItemsPage<>(collector, new Page(List.of(trackId, token)));
}
@Override
public InfoItemsPage<CommentsInfoItem> getPage(final Page page)
throws IOException, ExtractionException {
final CommentsInfoItemsCollector collector = new CommentsInfoItemsCollector(getServiceId());
final List<String> pageIds = page.getIds();
final String trackId = pageIds.get(0);
final String token = pageIds.get(1);
final JsonObject reviewsData = fetchReviewsData(trackId, token);
final JsonArray reviews = reviewsData.getArray("results");
for (final Object review : reviews) {
collector.commit(
new BandcampCommentsInfoItemExtractor((JsonObject) review, getUrl()));
}
if (!reviewsData.getBoolean("more_available")) {
return new InfoItemsPage<>(collector, null);
}
return new InfoItemsPage<>(collector,
new Page(List.of(trackId, getNextPageToken(reviews))));
}
private JsonObject fetchReviewsData(final String trackId, final String token)
throws ParsingException {
try {
return JsonUtils.toJsonObject(getDownloader().postWithContentTypeJson(
REVIEWS_API_URL,
Collections.emptyMap(),
JsonWriter.string().object()
.value("tralbum_type", "t")
.value("tralbum_id", trackId)
.value("token", token)
.value("count", 7)
.array("exclude_fan_ids").end()
.end().done().getBytes(StandardCharsets.UTF_8)).responseBody());
} catch (final IOException | ReCaptchaException e) {
throw new ParsingException("Could not fetch reviews", e);
}
}
private String getNextPageToken(final JsonArray reviews) throws ParsingException {
return reviews.stream()
.filter(JsonObject.class::isInstance)
.map(JsonObject.class::cast)
.map(review -> review.getString("token"))
.reduce((a, b) -> b) // keep only the last element
.orElseThrow(() -> new ParsingException("Could not get token"));
}
private String getTrackId() throws ParsingException {
final JsonObject pageProperties = JsonUtils.toJsonObject(
document.selectFirst("meta[name=bc-page-properties]")
.attr("content"));
return Long.toString(pageProperties.getLong("item_id"));
}
@Override
public boolean isCommentsDisabled() throws ExtractionException {
return BandcampExtractorHelper.isRadioUrl(getUrl());
}
}

View File

@ -0,0 +1,57 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.comments.CommentsInfoItemExtractor;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.stream.Description;
import javax.annotation.Nonnull;
import java.util.List;
public class BandcampCommentsInfoItemExtractor implements CommentsInfoItemExtractor {
private final JsonObject review;
private final String url;
public BandcampCommentsInfoItemExtractor(final JsonObject review, final String url) {
this.review = review;
this.url = url;
}
@Override
public String getName() throws ParsingException {
return getCommentText().getContent();
}
@Override
public String getUrl() {
return url;
}
@Nonnull
@Override
public List<Image> getThumbnails() throws ParsingException {
return getUploaderAvatars();
}
@Nonnull
@Override
public Description getCommentText() throws ParsingException {
return new Description(review.getString("why"), Description.PLAIN_TEXT);
}
@Override
public String getUploaderName() throws ParsingException {
return review.getString("name");
}
@Nonnull
@Override
public List<Image> getUploaderAvatars() {
return getImagesFromImageId(review.getLong("image_id"), false);
}
}

View File

@ -0,0 +1,303 @@
// Created by Fynn Godau 2019, licensed GNU GPL version 3 or later
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParser;
import com.grack.nanojson.JsonParserException;
import com.grack.nanojson.JsonWriter;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Element;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.Image.ResolutionLevel;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.exceptions.ReCaptchaException;
import org.schabi.newpipe.extractor.localization.DateWrapper;
import org.schabi.newpipe.extractor.utils.ImageSuffix;
import org.schabi.newpipe.extractor.utils.Utils;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.time.DateTimeException;
import java.time.ZonedDateTime;
import java.time.format.DateTimeFormatter;
import java.util.Collections;
import java.util.List;
import java.util.Locale;
import java.util.stream.Collectors;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import static org.schabi.newpipe.extractor.Image.HEIGHT_UNKNOWN;
import static org.schabi.newpipe.extractor.Image.WIDTH_UNKNOWN;
import static org.schabi.newpipe.extractor.utils.Utils.isNullOrEmpty;
import static org.schabi.newpipe.extractor.utils.Utils.replaceHttpWithHttps;
public final class BandcampExtractorHelper {
/**
* List of image IDs which preserve aspect ratio with their theoretical dimension known.
*
* <p>
* Bandcamp images are not always squares, so images which preserve aspect ratio are only used.
* </p>
*
* <p>
* One of the direct consequences of this specificity is that only one dimension of images is
* known at time, depending of the image ID.
* </p>
*
* <p>
* Note also that dimensions are only theoretical because if the image size is less than the
* dimensions of the image ID, it will be not upscaled but kept to its original size.
* </p>
*
* <p>
* IDs come from <a href="https://gist.github.com/f2k1de/06f5fd0ae9c919a7c3693a44ee522213">the
* GitHub Gist "Bandcamp File Format Parameters" by f2k1de</a>
* </p>
*/
private static final List<ImageSuffix> IMAGE_URL_SUFFIXES_AND_RESOLUTIONS = List.of(
// ID | HEIGHT | WIDTH
new ImageSuffix("10.jpg", HEIGHT_UNKNOWN, 1200, ResolutionLevel.HIGH),
new ImageSuffix("101.jpg", 90, WIDTH_UNKNOWN, ResolutionLevel.LOW),
new ImageSuffix("170.jpg", 422, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
// 180 returns the same image aspect ratio and size as 171
new ImageSuffix("171.jpg", 646, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
new ImageSuffix("20.jpg", HEIGHT_UNKNOWN, 1024, ResolutionLevel.HIGH),
// 203 returns the same image aspect ratio and size as 200
new ImageSuffix("200.jpg", 420, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
new ImageSuffix("201.jpg", 280, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
new ImageSuffix("202.jpg", 140, WIDTH_UNKNOWN, ResolutionLevel.LOW),
new ImageSuffix("204.jpg", 360, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
new ImageSuffix("205.jpg", 240, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
new ImageSuffix("206.jpg", 180, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM),
new ImageSuffix("207.jpg", 120, WIDTH_UNKNOWN, ResolutionLevel.LOW),
new ImageSuffix("43.jpg", 100, WIDTH_UNKNOWN, ResolutionLevel.LOW),
new ImageSuffix("44.jpg", 200, WIDTH_UNKNOWN, ResolutionLevel.MEDIUM));
private static final String IMAGE_URL_APPENDIX_AND_EXTENSION_REGEX = "_\\d+\\.\\w+";
private static final String IMAGES_DOMAIN_AND_PATH = "https://f4.bcbits.com/img/";
public static final String BASE_URL = "https://bandcamp.com";
public static final String BASE_API_URL = BASE_URL + "/api";
private BandcampExtractorHelper() {
}
/**
* Translate all these parameters together to the URL of the corresponding album or track
* using the mobile API
*/
public static String getStreamUrlFromIds(final long bandId,
final long itemId,
final String itemType) throws ParsingException {
try {
final String jsonString = NewPipe.getDownloader().get(
BASE_API_URL + "/mobile/22/tralbum_details?band_id=" + bandId
+ "&tralbum_id=" + itemId + "&tralbum_type=" + itemType.charAt(0))
.responseBody();
return replaceHttpWithHttps(JsonParser.object().from(jsonString)
.getString("bandcamp_url"));
} catch (final JsonParserException | ReCaptchaException | IOException e) {
throw new ParsingException("Ids could not be translated to URL", e);
}
}
/**
* Fetch artist details from mobile endpoint.
* <a href="https://notabug.org/fynngodau/bandcampDirect/wiki/rewindBandcamp+%E2%80%93+Fetching+artist+details">
* More technical info.</a>
*/
public static JsonObject getArtistDetails(final String id) throws ParsingException {
try {
return JsonParser.object().from(NewPipe.getDownloader().postWithContentTypeJson(
BASE_API_URL + "/mobile/22/band_details",
Collections.emptyMap(),
JsonWriter.string()
.object()
.value("band_id", id)
.end()
.done()
.getBytes(StandardCharsets.UTF_8)).responseBody());
} catch (final IOException | ReCaptchaException | JsonParserException e) {
throw new ParsingException("Could not download band details", e);
}
}
/**
* Generate an image url from an image ID.
*
* <p>
* The image ID {@code 10} was chosen because it provides images wide up to 1200px (when
* the original image width is more than or equal this resolution).
* </p>
*
* <p>
* Other integer values are possible as well (e.g. 0 is a very large resolution, possibly the
* original); see {@link #IMAGE_URL_SUFFIXES_AND_RESOLUTIONS} for more details about image
* resolution IDs.
* </p>
*
* @param id the image ID
* @param isAlbum whether the image is the cover of an album or a track
* @return a URL of the image with this ID with a width up to 1200px
*/
@Nonnull
public static String getImageUrl(final long id, final boolean isAlbum) {
return IMAGES_DOMAIN_AND_PATH + (isAlbum ? 'a' : "") + id + "_10.jpg";
}
/**
* @return <code>true</code> if the given URL looks like it comes from a bandcamp custom domain
* or a <code>*.bandcamp.com</code> subdomain
*/
public static boolean isArtistDomain(final String url) throws ParsingException {
// Accept all bandcamp.com URLs
if (url.toLowerCase().matches("https?://.+\\.bandcamp\\.com(/.*)?")) {
return true;
}
// Reject non-artist bandcamp.com URLs
if (url.toLowerCase().matches("https?://bandcamp\\.com(/.*)?")) {
return false;
}
try {
// Test other URLs for whether they contain a footer that links to bandcamp
return Jsoup.parse(
NewPipe.getDownloader()
.get(Utils.replaceHttpWithHttps(url))
.responseBody()
)
.getElementsByClass("cart-wrapper")
.get(0)
.getElementsByTag("a")
.get(0)
.attr("href")
.equals("https://bandcamp.com/cart");
} catch (final NullPointerException | IndexOutOfBoundsException e) {
return false;
} catch (final IOException | ReCaptchaException e) {
throw new ParsingException("Could not determine whether URL is custom domain "
+ "(not available? network error?)");
}
}
/**
* Whether the URL points to a radio kiosk.
* @param url the URL to check
* @return true if the URL matches {@code https://bandcamp.com/?show=SHOW_ID}
*/
public static boolean isRadioUrl(final String url) {
return url.toLowerCase().matches("https?://bandcamp\\.com/\\?show=\\d+");
}
public static DateWrapper parseDate(final String textDate) throws ParsingException {
try {
final ZonedDateTime zonedDateTime = ZonedDateTime.parse(textDate,
DateTimeFormatter.ofPattern("dd MMM yyyy HH:mm:ss zzz", Locale.ENGLISH));
return new DateWrapper(zonedDateTime.toOffsetDateTime(), false);
} catch (final DateTimeException e) {
throw new ParsingException("Could not parse date '" + textDate + "'", e);
}
}
/**
* Get a list of images from a search result {@link Element}.
*
* <p>
* This method will call {@link #getImagesFromImageUrl(String)} using the first non null and
* non empty image URL found from the {@code src} attribute of {@code img} HTML elements, or an
* empty string if no valid image URL was found.
* </p>
*
* @param searchResult a search result {@link Element}
* @return an unmodifiable list of {@link Image}s, which is never null but can be empty, in the
* case where no valid image URL was found
*/
@Nonnull
public static List<Image> getImagesFromSearchResult(@Nonnull final Element searchResult) {
return getImagesFromImageUrl(searchResult.getElementsByClass("art")
.stream()
.flatMap(element -> element.getElementsByTag("img").stream())
.map(element -> element.attr("src"))
.filter(imageUrl -> !isNullOrEmpty(imageUrl))
.findFirst()
.orElse(""));
}
/**
* Get all images which have resolutions preserving aspect ratio from an image URL.
*
* <p>
* This method will remove the image ID and its extension from the end of the URL and then call
* {@link #getImagesFromImageBaseUrl(String)}.
* </p>
*
* @param imageUrl the full URL of an image provided by Bandcamp, such as in its HTML code
* @return an unmodifiable list of {@link Image}s, which is never null but can be empty, in the
* case where the image URL has been not extracted (and so is null or empty)
*/
@Nonnull
public static List<Image> getImagesFromImageUrl(@Nullable final String imageUrl) {
if (isNullOrEmpty(imageUrl)) {
return List.of();
}
return getImagesFromImageBaseUrl(
imageUrl.replaceFirst(IMAGE_URL_APPENDIX_AND_EXTENSION_REGEX, "_"));
}
/**
* Get all images which have resolutions preserving aspect ratio from an image ID.
*
* <p>
* This method will call {@link #getImagesFromImageBaseUrl(String)}.
* </p>
*
* @param id the id of an image provided by Bandcamp
* @param isAlbum whether the image is the cover of an album
* @return an unmodifiable list of {@link Image}s, which is never null but can be empty, in the
* case where the image ID has been not extracted (and so equal to 0)
*/
@Nonnull
public static List<Image> getImagesFromImageId(final long id, final boolean isAlbum) {
if (id == 0) {
return List.of();
}
return getImagesFromImageBaseUrl(IMAGES_DOMAIN_AND_PATH + (isAlbum ? 'a' : "") + id + "_");
}
/**
* Get all images resolutions preserving aspect ratio from a base image URL.
*
* <p>
* Base image URLs are images containing the image path, a {@code a} letter if it comes from an
* album, its ID and an underscore.
* </p>
*
* <p>
* Images resolutions returned are the ones of {@link #IMAGE_URL_SUFFIXES_AND_RESOLUTIONS}.
* </p>
*
* @param baseUrl the base URL of the image
* @return an unmodifiable and non-empty list of {@link Image}s
*/
@Nonnull
private static List<Image> getImagesFromImageBaseUrl(@Nonnull final String baseUrl) {
return IMAGE_URL_SUFFIXES_AND_RESOLUTIONS.stream()
.map(imageSuffix -> new Image(baseUrl + imageSuffix.getSuffix(),
imageSuffix.getHeight(), imageSuffix.getWidth(),
imageSuffix.getResolutionLevel()))
.collect(Collectors.toUnmodifiableList());
}
}

View File

@ -0,0 +1,118 @@
// Created by Fynn Godau 2019, licensed GNU GPL version 3 or later
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParser;
import com.grack.nanojson.JsonParserException;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.kiosk.KioskExtractor;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.playlist.PlaylistInfoItem;
import org.schabi.newpipe.extractor.playlist.PlaylistInfoItemsCollector;
import javax.annotation.Nonnull;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.Collections;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_API_URL;
public class BandcampFeaturedExtractor extends KioskExtractor<PlaylistInfoItem> {
public static final String KIOSK_FEATURED = "Featured";
public static final String FEATURED_API_URL = BASE_API_URL + "/mobile/24/bootstrap_data";
public static final String MORE_FEATURED_API_URL
= BASE_API_URL + "/mobile/24/feed_older_logged_out";
private JsonObject json;
public BandcampFeaturedExtractor(final StreamingService streamingService,
final ListLinkHandler listLinkHandler,
final String kioskId) {
super(streamingService, listLinkHandler, kioskId);
}
@Override
public void onFetchPage(@Nonnull final Downloader downloader)
throws IOException, ExtractionException {
try {
json = JsonParser.object().from(getDownloader().postWithContentTypeJson(
FEATURED_API_URL,
Collections.emptyMap(),
"{\"platform\":\"\",\"version\":0}".getBytes(StandardCharsets.UTF_8))
.responseBody());
} catch (final JsonParserException e) {
throw new ParsingException("Could not parse Bandcamp featured API response", e);
}
}
@Nonnull
@Override
public String getName() throws ParsingException {
return KIOSK_FEATURED;
}
@Nonnull
@Override
public InfoItemsPage<PlaylistInfoItem> getInitialPage()
throws IOException, ExtractionException {
final JsonArray featuredStories = json.getObject("feed_content")
.getObject("stories")
.getArray("featured");
return extractItems(featuredStories);
}
private InfoItemsPage<PlaylistInfoItem> extractItems(final JsonArray featuredStories) {
final PlaylistInfoItemsCollector c = new PlaylistInfoItemsCollector(getServiceId());
for (int i = 0; i < featuredStories.size(); i++) {
final JsonObject featuredStory = featuredStories.getObject(i);
if (featuredStory.isNull("album_title")) {
// Is not an album, ignore
continue;
}
c.commit(new BandcampPlaylistInfoItemFeaturedExtractor(featuredStory));
}
final JsonObject lastFeaturedStory = featuredStories.getObject(featuredStories.size() - 1);
return new InfoItemsPage<>(c, getNextPageFrom(lastFeaturedStory));
}
/**
* Next Page can be generated from metadata of last featured story
*/
private Page getNextPageFrom(final JsonObject lastFeaturedStory) {
final long lastStoryDate = lastFeaturedStory.getLong("story_date");
final long lastStoryId = lastFeaturedStory.getLong("ntid");
final String lastStoryType = lastFeaturedStory.getString("story_type");
return new Page(
MORE_FEATURED_API_URL + "?story_groups=featured"
+ ':' + lastStoryDate + ':' + lastStoryType + ':' + lastStoryId
);
}
@Override
public InfoItemsPage<PlaylistInfoItem> getPage(final Page page)
throws IOException, ExtractionException {
final JsonObject response;
try {
response = JsonParser.object().from(
getDownloader().get(page.getUrl()).responseBody()
);
} catch (final JsonParserException e) {
throw new ParsingException("Could not parse Bandcamp featured API response", e);
}
return extractItems(response.getObject("stories").getArray("featured"));
}
}

View File

@ -0,0 +1,179 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageUrl;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampStreamExtractor.getAlbumInfoJson;
import static org.schabi.newpipe.extractor.utils.JsonUtils.getJsonData;
import static org.schabi.newpipe.extractor.utils.Utils.HTTPS;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParserException;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.PaidContentException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.playlist.PlaylistExtractor;
import org.schabi.newpipe.extractor.services.bandcamp.extractors.streaminfoitem.BandcampPlaylistStreamInfoItemExtractor;
import org.schabi.newpipe.extractor.stream.Description;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import org.schabi.newpipe.extractor.stream.StreamInfoItemsCollector;
import java.io.IOException;
import java.util.Objects;
import java.util.List;
import javax.annotation.Nonnull;
public class BandcampPlaylistExtractor extends PlaylistExtractor {
/**
* An arbitrarily chosen number above which cover arts won't be fetched individually for each
* track; instead, it will be assumed that every track has the same cover art as the album,
* which is not always the case.
*/
private static final int MAXIMUM_INDIVIDUAL_COVER_ARTS = 10;
private Document document;
private JsonObject albumJson;
private JsonArray trackInfo;
private String name;
public BandcampPlaylistExtractor(final StreamingService service,
final ListLinkHandler linkHandler) {
super(service, linkHandler);
}
@Override
public void onFetchPage(@Nonnull final Downloader downloader)
throws IOException, ExtractionException {
final String html = downloader.get(getLinkHandler().getUrl()).responseBody();
document = Jsoup.parse(html);
albumJson = getAlbumInfoJson(html);
trackInfo = albumJson.getArray("trackinfo");
try {
name = getJsonData(html, "data-embed").getString("album_title");
} catch (final JsonParserException e) {
throw new ParsingException("Faulty JSON; page likely does not contain album data", e);
} catch (final ArrayIndexOutOfBoundsException e) {
throw new ParsingException("JSON does not exist", e);
}
if (trackInfo.isEmpty()) {
// Albums without trackInfo need to be purchased before they can be played
throw new PaidContentException("Album needs to be purchased");
}
}
@Nonnull
@Override
public List<Image> getThumbnails() throws ParsingException {
if (albumJson.isNull("art_id")) {
return List.of();
} else {
return getImagesFromImageId(albumJson.getLong("art_id"), true);
}
}
@Override
public String getUploaderUrl() throws ParsingException {
final String[] parts = getUrl().split("/");
// https: (/) (/) * .bandcamp.com (/) and leave out the rest
return HTTPS + parts[2] + "/";
}
@Override
public String getUploaderName() {
return albumJson.getString("artist");
}
@Nonnull
@Override
public List<Image> getUploaderAvatars() {
return getImagesFromImageUrl(document.getElementsByClass("band-photo")
.stream()
.map(element -> element.attr("src"))
.findFirst()
.orElse(""));
}
@Override
public boolean isUploaderVerified() throws ParsingException {
return false;
}
@Override
public long getStreamCount() {
return trackInfo.size();
}
@Nonnull
@Override
public Description getDescription() throws ParsingException {
final Element tInfo = document.getElementById("trackInfo");
if (tInfo == null) {
throw new ParsingException("Could not find trackInfo in document");
}
final Elements about = tInfo.getElementsByClass("tralbum-about");
final Elements credits = tInfo.getElementsByClass("tralbum-credits");
final Element license = document.getElementById("license");
if (about.isEmpty() && credits.isEmpty() && license == null) {
return Description.EMPTY_DESCRIPTION;
}
final StringBuilder sb = new StringBuilder();
if (!about.isEmpty()) {
sb.append(Objects.requireNonNull(about.first()).html());
}
if (!credits.isEmpty()) {
sb.append(Objects.requireNonNull(credits.first()).html());
}
if (license != null) {
sb.append(license.html());
}
return new Description(sb.toString(), Description.HTML);
}
@Nonnull
@Override
public InfoItemsPage<StreamInfoItem> getInitialPage() throws ExtractionException {
final StreamInfoItemsCollector collector = new StreamInfoItemsCollector(getServiceId());
for (int i = 0; i < trackInfo.size(); i++) {
final JsonObject track = trackInfo.getObject(i);
if (trackInfo.size() < MAXIMUM_INDIVIDUAL_COVER_ARTS) {
// Load cover art of every track individually
collector.commit(new BandcampPlaylistStreamInfoItemExtractor(
track, getUploaderUrl(), getService()));
} else {
// Pretend every track has the same cover art as the album
collector.commit(new BandcampPlaylistStreamInfoItemExtractor(
track, getUploaderUrl(), getThumbnails()));
}
}
return new InfoItemsPage<>(collector, null);
}
@Override
public InfoItemsPage<StreamInfoItem> getPage(final Page page) {
return null;
}
@Nonnull
@Override
public String getName() throws ParsingException {
return name;
}
}

View File

@ -0,0 +1,58 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import org.jsoup.nodes.Element;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.playlist.PlaylistInfoItemExtractor;
import javax.annotation.Nonnull;
import java.util.List;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromSearchResult;
public class BandcampPlaylistInfoItemExtractor implements PlaylistInfoItemExtractor {
private final Element searchResult;
private final Element resultInfo;
public BandcampPlaylistInfoItemExtractor(@Nonnull final Element searchResult) {
this.searchResult = searchResult;
resultInfo = searchResult.getElementsByClass("result-info").first();
}
@Override
public String getUploaderName() {
return resultInfo.getElementsByClass("subhead").text()
.split(" by")[0];
}
@Override
public String getUploaderUrl() {
return null;
}
@Override
public boolean isUploaderVerified() {
return false;
}
@Override
public long getStreamCount() {
final String length = resultInfo.getElementsByClass("length").text();
return Integer.parseInt(length.split(" track")[0]);
}
@Override
public String getName() {
return resultInfo.getElementsByClass("heading").text();
}
@Override
public String getUrl() {
return resultInfo.getElementsByClass("itemurl").text();
}
@Nonnull
@Override
public List<Image> getThumbnails() {
return getImagesFromSearchResult(searchResult);
}
}

View File

@ -0,0 +1,58 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.playlist.PlaylistInfoItemExtractor;
import org.schabi.newpipe.extractor.utils.Utils;
import javax.annotation.Nonnull;
import java.util.List;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
public class BandcampPlaylistInfoItemFeaturedExtractor implements PlaylistInfoItemExtractor {
private final JsonObject featuredStory;
public BandcampPlaylistInfoItemFeaturedExtractor(final JsonObject featuredStory) {
this.featuredStory = featuredStory;
}
@Override
public String getUploaderName() {
return featuredStory.getString("band_name");
}
@Override
public String getUploaderUrl() {
return null;
}
@Override
public boolean isUploaderVerified() {
return false;
}
@Override
public long getStreamCount() {
return featuredStory.getInt("num_streamable_tracks");
}
@Override
public String getName() {
return featuredStory.getString("album_title");
}
@Override
public String getUrl() {
return Utils.replaceHttpWithHttps(featuredStory.getString("item_url"));
}
@Nonnull
@Override
public List<Image> getThumbnails() {
return featuredStory.has("art_id")
? getImagesFromImageId(featuredStory.getLong("art_id"), true)
: getImagesFromImageId(featuredStory.getLong("item_art_id"), true);
}
}

View File

@ -0,0 +1,73 @@
// Created by Fynn Godau 2019, licensed GNU GPL version 3 or later
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParser;
import com.grack.nanojson.JsonParserException;
import org.schabi.newpipe.extractor.Page;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.kiosk.KioskExtractor;
import org.schabi.newpipe.extractor.linkhandler.ListLinkHandler;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
import org.schabi.newpipe.extractor.stream.StreamInfoItemsCollector;
import javax.annotation.Nonnull;
import java.io.IOException;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_API_URL;
public class BandcampRadioExtractor extends KioskExtractor<StreamInfoItem> {
public static final String KIOSK_RADIO = "Radio";
public static final String RADIO_API_URL = BASE_API_URL + "/bcweekly/3/list";
private JsonObject json = null;
public BandcampRadioExtractor(final StreamingService streamingService,
final ListLinkHandler linkHandler,
final String kioskId) {
super(streamingService, linkHandler, kioskId);
}
@Override
public void onFetchPage(@Nonnull final Downloader downloader)
throws IOException, ExtractionException {
try {
json = JsonParser.object().from(
getDownloader().get(RADIO_API_URL).responseBody());
} catch (final JsonParserException e) {
throw new ExtractionException("Could not parse Bandcamp Radio API response", e);
}
}
@Nonnull
@Override
public String getName() throws ParsingException {
return KIOSK_RADIO;
}
@Nonnull
@Override
public InfoItemsPage<StreamInfoItem> getInitialPage() {
final StreamInfoItemsCollector collector = new StreamInfoItemsCollector(getServiceId());
final JsonArray radioShows = json.getArray("results");
for (int i = 0; i < radioShows.size(); i++) {
final JsonObject radioShow = radioShows.getObject(i);
collector.commit(new BandcampRadioInfoItemExtractor(radioShow));
}
return new InfoItemsPage<>(collector, null);
}
@Override
public InfoItemsPage<StreamInfoItem> getPage(final Page page) {
return null;
}
}

View File

@ -0,0 +1,102 @@
// Created by Fynn Godau 2019, licensed GNU GPL version 3 or later
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import com.grack.nanojson.JsonObject;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.localization.DateWrapper;
import org.schabi.newpipe.extractor.stream.StreamInfoItemExtractor;
import org.schabi.newpipe.extractor.stream.StreamType;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.List;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_URL;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.parseDate;
public class BandcampRadioInfoItemExtractor implements StreamInfoItemExtractor {
private final JsonObject show;
public BandcampRadioInfoItemExtractor(final JsonObject radioShow) {
show = radioShow;
}
@Override
public long getDuration() {
/* Duration is only present in the more detailed information that has to be queried
separately. Therefore, over 300 queries would be needed every time the kiosk is opened if we
were to display the real value. */
//return query(show.getInt("id")).getLong("audio_duration");
return 0;
}
@Nullable
@Override
public String getShortDescription() {
return show.getString("desc");
}
@Nullable
@Override
public String getTextualUploadDate() {
return show.getString("date");
}
@Nullable
@Override
public DateWrapper getUploadDate() throws ParsingException {
return parseDate(getTextualUploadDate());
}
@Override
public String getName() throws ParsingException {
return show.getString("subtitle");
}
@Override
public String getUrl() {
return BASE_URL + "/?show=" + show.getInt("id");
}
@Nonnull
@Override
public List<Image> getThumbnails() {
return getImagesFromImageId(show.getLong("image_id"), false);
}
@Override
public StreamType getStreamType() {
return StreamType.AUDIO_STREAM;
}
@Override
public long getViewCount() {
return -1;
}
@Override
public String getUploaderName() {
// The "title" field contains the title of the series, e.g. "Bandcamp Weekly".
return show.getString("title");
}
@Override
public String getUploaderUrl() {
return "";
}
@Override
public boolean isUploaderVerified() throws ParsingException {
return false;
}
@Override
public boolean isAd() {
return false;
}
}

View File

@ -0,0 +1,194 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_API_URL;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_URL;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImageUrl;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageId;
import com.grack.nanojson.JsonArray;
import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParser;
import com.grack.nanojson.JsonParserException;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Element;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.Image.ResolutionLevel;
import org.schabi.newpipe.extractor.MediaFormat;
import org.schabi.newpipe.extractor.NewPipe;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ContentNotSupportedException;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.exceptions.ReCaptchaException;
import org.schabi.newpipe.extractor.linkhandler.LinkHandler;
import org.schabi.newpipe.extractor.playlist.PlaylistInfoItemsCollector;
import org.schabi.newpipe.extractor.stream.AudioStream;
import org.schabi.newpipe.extractor.stream.Description;
import org.schabi.newpipe.extractor.stream.StreamSegment;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
public class BandcampRadioStreamExtractor extends BandcampStreamExtractor {
private static final String OPUS_LO = "opus-lo";
private static final String MP3_128 = "mp3-128";
private JsonObject showInfo;
public BandcampRadioStreamExtractor(final StreamingService service,
final LinkHandler linkHandler) {
super(service, linkHandler);
}
static JsonObject query(final int id) throws ParsingException {
try {
return JsonParser.object().from(NewPipe.getDownloader()
.get(BASE_API_URL + "/bcweekly/1/get?id=" + id).responseBody());
} catch (final IOException | ReCaptchaException | JsonParserException e) {
throw new ParsingException("could not get show data", e);
}
}
@Override
public void onFetchPage(@Nonnull final Downloader downloader)
throws IOException, ExtractionException {
showInfo = query(Integer.parseInt(getId()));
}
@Nonnull
@Override
public String getName() throws ParsingException {
/* Select "subtitle" and not "audio_title", as the latter would cause a lot of
* items to show the same title, e.g. "Bandcamp Weekly".
*/
return showInfo.getString("subtitle");
}
@Nonnull
@Override
public String getUploaderUrl() throws ContentNotSupportedException {
throw new ContentNotSupportedException("Fan pages are not supported");
}
@Nonnull
@Override
public String getUrl() throws ParsingException {
return getLinkHandler().getUrl();
}
@Nonnull
@Override
public String getUploaderName() throws ParsingException {
return Jsoup.parse(showInfo.getString("image_caption")).getElementsByTag("a").stream()
.map(Element::text)
.findFirst()
.orElseThrow(() -> new ParsingException("Could not get uploader name"));
}
@Nullable
@Override
public String getTextualUploadDate() {
return showInfo.getString("published_date");
}
@Nonnull
@Override
public List<Image> getThumbnails() throws ParsingException {
return getImagesFromImageId(showInfo.getLong("show_image_id"), false);
}
@Nonnull
@Override
public List<Image> getUploaderAvatars() {
return Collections.singletonList(
new Image(BASE_URL + "/img/buttons/bandcamp-button-circle-whitecolor-512.png",
512, 512, ResolutionLevel.MEDIUM));
}
@Nonnull
@Override
public Description getDescription() {
return new Description(showInfo.getString("desc"), Description.PLAIN_TEXT);
}
@Override
public long getLength() {
return showInfo.getLong("audio_duration");
}
@Override
public List<AudioStream> getAudioStreams() {
final List<AudioStream> audioStreams = new ArrayList<>();
final JsonObject streams = showInfo.getObject("audio_stream");
if (streams.has(MP3_128)) {
audioStreams.add(new AudioStream.Builder()
.setId(MP3_128)
.setContent(streams.getString(MP3_128), true)
.setMediaFormat(MediaFormat.MP3)
.setAverageBitrate(128)
.build());
}
if (streams.has(OPUS_LO)) {
audioStreams.add(new AudioStream.Builder()
.setId(OPUS_LO)
.setContent(streams.getString(OPUS_LO), true)
.setMediaFormat(MediaFormat.OPUS)
.setAverageBitrate(100).build());
}
return audioStreams;
}
@Nonnull
@Override
public List<StreamSegment> getStreamSegments() throws ParsingException {
final JsonArray tracks = showInfo.getArray("tracks");
final List<StreamSegment> segments = new ArrayList<>(tracks.size());
for (final Object t : tracks) {
final JsonObject track = (JsonObject) t;
final StreamSegment segment = new StreamSegment(
track.getString("title"), track.getInt("timecode"));
// "track art" is the track's album cover
segment.setPreviewUrl(getImageUrl(track.getLong("track_art_id"), true));
segment.setChannelName(track.getString("artist"));
segments.add(segment);
}
return segments;
}
@Nonnull
@Override
public String getLicence() {
// Contrary to other Bandcamp streams, radio streams don't have a license
return "";
}
@Nonnull
@Override
public String getCategory() {
// Contrary to other Bandcamp streams, radio streams don't have categories
return "";
}
@Nonnull
@Override
public List<String> getTags() {
// Contrary to other Bandcamp streams, radio streams don't have tags
return Collections.emptyList();
}
@Override
public PlaylistInfoItemsCollector getRelatedItems() {
// Contrary to other Bandcamp streams, radio streams don't have related items
return null;
}
}

View File

@ -0,0 +1,60 @@
// Created by Fynn Godau 2021, licensed GNU GPL version 3 or later
package org.schabi.newpipe.extractor.services.bandcamp.extractors;
import org.jsoup.nodes.Element;
import org.schabi.newpipe.extractor.Image;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
import org.schabi.newpipe.extractor.playlist.PlaylistInfoItemExtractor;
import javax.annotation.Nonnull;
import java.util.List;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImagesFromImageUrl;
/**
* Extracts recommended albums from tracks' website
*/
public class BandcampRelatedPlaylistInfoItemExtractor implements PlaylistInfoItemExtractor {
private final Element relatedAlbum;
public BandcampRelatedPlaylistInfoItemExtractor(@Nonnull final Element relatedAlbum) {
this.relatedAlbum = relatedAlbum;
}
@Override
public String getName() throws ParsingException {
return relatedAlbum.getElementsByClass("release-title").text();
}
@Override
public String getUrl() throws ParsingException {
return relatedAlbum.getElementsByClass("album-link").attr("abs:href");
}
@Nonnull
@Override
public List<Image> getThumbnails() throws ParsingException {
return getImagesFromImageUrl(relatedAlbum.getElementsByClass("album-art").attr("src"));
}
@Override
public String getUploaderName() throws ParsingException {
return relatedAlbum.getElementsByClass("by-artist").text().replace("by ", "");
}
@Override
public String getUploaderUrl() throws ParsingException {
return null;
}
@Override
public boolean isUploaderVerified() throws ParsingException {
return false;
}
@Override
public long getStreamCount() throws ParsingException {
return -1;
}
}

Some files were not shown because too many files have changed in this diff Show More