* Non-Extension Index changes for 1.6
* Changelog
* Minor fixes
* Implement extension store
* Test build fix
* Docs
* Simplify fetching manga and chapters
* Use EMPTY JsonObject
* Update docs/Configuring-Suwayomi‐Server.md
Co-authored-by: Constantin Piber <59023762+cpiber@users.noreply.github.com>
* Improve Fetch Extension Store
* Fixes
* Simplify deprecated isNsfw in SourceQuery
* Simplify ContentRating in Source.kt
* Simplify isNsfw in SourceType
* No magic numbers for ContentRating, improves safety for future versions of extension api
* Fix SearchTest
* Lint
* Lint
* Optimize imports and fix unchecked cast warning
* Proper extension store queries
* Optimize import fixes
* Add ContentRatingFilter
* Improve extension store sync
* fix: re-sync (#2121)
* Lint
* Add ExtenionStores to the fetchExtensions result since its possible for the stores to change.
* Use a single version of ContentRating
* Exclude ServerConfig.extensionStores from GraphQL
* Use syncDbToPrefs in ExtensionStoreMutation
* Optimize Imports
* Update server/server-config/src/main/kotlin/suwayomi/tachidesk/server/ServerConfig.kt
Co-authored-by: Constantin Piber <59023762+cpiber@users.noreply.github.com>
* Remove replaceWith and add specific description for GQL APIs
* Include OkHttp ZSTD
* Update to latest Mihon extension lib
* Fix latest Mihon Extension Lib
* Lint
* Optimize imports
* Lint
* Review fixes
* Add a index to extesnion table store url
* Lint
---------
Co-authored-by: Constantin Piber <59023762+cpiber@users.noreply.github.com>
The returned result rows of the inserted chapters did not have the up-to-date "last_modified_at".
This caused "downloadNewChapters" to not be able to correctly detect unread chapters. it included the newly inserted ones, leading to exiting early due to having unread chapters.
Regression 811e15162bfixes#2097
* Implement SyncYomi
* Add ability to select what to sync
* Properly fix default category bug
* Add periodic sync
* Add PostgreSQL support
* Deschedule previous task
* Check if SyncYomi is enabled in syncData function
* Don't allow multiple syncs at the same time
* Convert SyncYomiSyncService to object
* Make startSync non-suspend
* Return a result from startSync
* Sync before library update
* Improvements
* Use NetworkHelper client
* Lint
* Use measureTime
* Database improvements
- Move entire sync operation into a single transaction
- Stop loading all manga to memory
* Revert "Database improvements"
This reverts commit bee8d214c3.
* Actual database improvements
* Remove runBlocking
* Remove title check
* Update updateNonFavorites function
* Update timeout code
* Improve PostgreSQL query
* Create lastSyncState variable
* Create lastSyncStatus query
* Convert lastSyncState to StateFlow
* Create lastSyncStatusChange subscription
* Replace backupRestoreStatus with backupRestoreId
* Add startDate and endDate
* Add logs for sync start and end
* Handle all errors in syncData
* Change category restore function to match Mihon's behavior
* Fix comment
* Remove duplicate BackupMangaHandler.backup call
* Remove duplicated log
* Rename subscription to syncStatusChanged
* Use same flags for restoring
* Update syncInterval config to use DurationSetting
* Update sync scheduling logic
* Reorder conditions to reduce database calls
* Prevent deleted ghost chapters from reappearing during sync
jobobby04/TachiyomiSY#1575
* Improve sync merging categories
jobobby04/TachiyomiSY#1559
* Make columns not null
* Improve H2 triggers
* Add documentation
The deletion of chapter data was done in its own transaction. Thus, when the update or insertion failed later on, the deletion was not rolled back
fixes#2031
Co-authored-by: Mitchell Syer <Syer10@users.noreply.github.com>
* feat(sync/koreader): implement reading progress synchronization
This commit introduces a comprehensive integration with KOReader Sync Server to enable two-way synchronization of reading progress.
The core logic is encapsulated in a new `KoreaderSyncService`, which handles authentication, registration, and progress pushing/pulling based on user-defined strategies (LATEST, KOSYNC, SUWAYOMI).
Key changes include:
- A new GraphQL API is added to manage the KOReader Sync connection:
- `connectKoSyncAccount` mutation provides a simplified flow that attempts to log in and, if the user doesn't exist, automatically registers them.
- `logoutKoSyncAccount` mutation to clear credentials.
- `koSyncStatus` query to check the current connection status.
- Reading progress is now synchronized at key points:
- The `fetchChapterPages` mutation pulls the latest progress from the sync server before loading the reader. It respects the configured sync strategy and updates the local database if necessary.
- The `updateChapters` and other progress-updating methods now push changes to the sync server automatically.
- OPDS chapter entries also pull the latest progress, ensuring clients receive up-to-date reading status.
- Supporting backend changes have been made:
- The `Chapter` table is extended with a `koreader_hash` column to uniquely identify documents. A database migration is included.
- New configuration options are added to `server.conf` to manage the feature (enable, server URL, credentials, strategy, etc.).
* perf(opds): defer KOReader sync to improve chapter feed performance
Removes the KOReader Sync progress-pulling logic from the `createChapterListEntry` function.
The previous implementation triggered a network request to the sync server for every single chapter when generating a list, leading to severe performance issues and slow load times on feeds with many entries.
This change reverts to the more performant approach of always linking to the chapter's metadata feed. The expensive sync operation will be handled within the metadata entry generation instead, ensuring it's only triggered on-demand for a single chapter. This restores the responsiveness of browsing chapter feeds.
* refactor(koreader): Use enums for sync settings and correct OPDS logic
Refactor Koreader Sync settings to use enums instead of raw strings for `checksumMethod` and `strategy`. This improves type safety, prevents typos, and makes the configuration handling more robust.
The changes include:
- Introducing `KoreaderSyncChecksumMethod` and `KoreaderSyncStrategy` enums.
- Updating `ServerConfig`, GraphQL types, and backup models to use these new enums.
- Refactoring `KoreaderSyncService` to work with the enum types.
Additionally, this commit fixes an issue in `OpdsEntryBuilder` where the logic for determining which sync progress to use (local vs. remote) was duplicated. The builder now correctly delegates this decision to `KoreaderSyncService.pullProgress`, which already contains the necessary strategy logic. This centralizes the logic and ensures consistent behavior.
* refactor(koreader): Improve config handling and remove redundant update
This commit combines several refactoring and cleanup tasks:
- **Koreader Sync:** The sync service is updated to use the modern `serverConfig` provider instead of the legacy `GlobalConfigManager`. This aligns it with the current configuration management approach in the project.
- **Download Provider:** A redundant `pageCount` database update is removed from `ChaptersFilesProvider`. This operation was unnecessary because the `getChapterDownloadReady` function, which is called earlier in the download process, already verifies and corrects the page count. This change eliminates a superfluous database write and fixes a related import issue.
* feat(sync/koreader)!: enhance sync strategy and add progress tolerance
This commit overhauls the KOReader synchronization feature to provide more granular control and robustness. The simple on/off toggle has been replaced with a more flexible strategy-based system.
Key changes include:
- Replaced `koreaderSyncEnabled` with a more powerful `koreaderSyncStrategy` enum.
- Introduced new sync strategies: `PROMPT`, `SILENT`, `SEND`, `RECEIVE`, and `DISABLE`, allowing for fine-grained control over the sync direction and conflict resolution.
- Added a `koreaderSyncPercentageTolerance` setting. This prevents unnecessary sync updates for minor progress differences between Suwayomi and KOReader.
- Refactored the `KoreaderSyncService` to implement the new strategies and use the configurable tolerance.
- Updated GraphQL schemas, mutations, and server configuration to remove the old setting and incorporate the new ones.
- Adjusted the backup and restore process to correctly handle the new configuration parameters.
- Modified API endpoints and internal logic to check and apply remote progress based on the selected strategy.
BREAKING CHANGE: The `koreaderSyncEnabled` setting is removed and replaced by a more granular `koreaderSyncStrategy`. The enum values for the strategy have been completely changed, making previous configurations incompatible.
* fix: remove unused imports
* feat(opds, sync): enhance Koreader sync and OPDS conflict handling
This commit introduces significant improvements to the Koreader synchronization feature, focusing on providing a better user experience for handling progress conflicts in both OPDS and GraphQL clients.
Key changes include:
- **OPDS Conflict Resolution:** When a reading progress conflict is detected, the OPDS feed for a chapter now provides two distinct "Read Online" links: one to continue from the local progress and another to continue from the synced progress from the remote device (e.g., "Continue Reading Online (Synced from KOReader)"). This empowers users to choose which progress to follow.
- **GraphQL Sync Conflict Information:** The `fetchChapterPages` GraphQL mutation now includes a `syncConflict` field in its payload. This field provides the remote device name and page number, allowing GraphQL clients to implement a user-facing prompt to resolve sync conflicts.
- **Improved Sync Strategy Handling:**
- The `connectKoSyncAccount` mutation no longer unconditionally sets the sync strategy to `PROMPT`. It now respects the user's existing setting, preventing accidental configuration changes upon re-login.
- The default `koreaderSyncStrategy` in the configuration is changed to `DISABLED`, providing a safer and more intuitive default for new users.
- **Refinements & Fixes:**
- The fallback for the remote device name is now centralized within the KoreaderSyncService, defaulting to "KOReader" if not provided.
- Renamed `KoreaderSyncStrategy.DISABLE` to `DISABLED` for consistency.
- Updated i18n strings for OPDS links to be more descriptive and user-friendly.
* refactor(kosync): rename stream page link titles for consistency
* refactor(kosync): return SettingsType in auth mutation payloads
The `connectKoSyncAccount` and `logoutKoSyncAccount` mutations modify server settings (username and userkey) but did not previously return the updated configuration. This forced client applications to manually refetch settings to avoid a stale cache.
This change modifies the payloads for both mutations to include the full `SettingsType`.
By returning the updated settings directly, GraphQL clients like Apollo Client can automatically update their cache, simplifying client-side state management and ensuring the UI always reflects the current server configuration.
Additionally, `clientMutationId` has been added to `KoSyncConnectPayload` for consistency with GraphQL practices, aligning it with the logout mutation.
Refs: #1560
* refactor(kosync): replace KoSyncConnectPayload with ConnectResult in connect method
* fix(kosync): add koreaderSyncPercentageTolerance default setting
* Export meta data
* Import meta data
* Add missing "opdsUseBinaryFileSize" setting to gql
* Export server settings
* Import server settings
* Streamline server config enum handling
* Use "restore amount" in backup import progress
* Update to exposed-migrations v3.5.0
* Update to kotlin-logging v7.0.0
* Update to exposed v0.46.0
* Update to exposed v0.47.0
* Update to exposed v0.55.0
* Update to exposed v0.56.0
* Update to exposed v0.57.0
* Validate setting values on mutation
* Handle invalid negative setting values
* Ensure at least one source is downloading at all times
* Prevent possible IllegalArgumentException
The "serverConfig.maxSourcesInParallel" value could have changed after the if-condition
* Persist page count during chapter list update
In case a downloaded chapter gets deleted during a chapter list update, the download status was tried to be preserved.
However, in case the status could be preserved, the page count was lost and thus, the chapter now was marked as downloaded with a page count of -1.
* Mark downloaded chapters without page count as not downloaded
* Prevent adding duplicated chapters into the db
it's possible that the source returns a list containing chapters with the same url
once such duplicated chapters have been added, they aren't being removed anymore as long as there is
a chapter with the same url in the fetched chapter list, even if the duplicated chapter itself
does not exist anymore on the source
* Drop duplicated chapters from database table
* Add unique constraint to chapter table
This is to completely prevent duplicated chapters from being added to the database.
Since once a duplicated chapter has been added to the database, it does not get removed anymore as long as a chapter with the same url is included in the requested source chapter list
In case the new chapters include duplicates from different scanlators, they would be included in the limit causing the auto download to potentially only download duplicated chapters while there might be more non duplicated chapters to download.
Instead, the limit should only consider unique chapters and then should include all duplicates of the chapters that should get downloaded
* Update test/server-reference file
* Properly handle re-uploaded chapters in auto download of new chapters
In case of unhandable re-uploaded chapters (different chapter numbers) they potentially would have prevented auto downloads due being considered as unread.
Additionally, they would not have been considered to get downloaded due to not having a higher chapter number than the previous latest existing chapter before the chapter list fetch.
* Add option to ignore re-uploads for auto downloads
* Extract check for manga category download inclusion
* Extract logic to get new chapter ids to download
* Simplify manga category download inclusion check
In case the DEFAULT category does not exist, someone messed with the database and it is basically corrupted
* Remove download ahead logic
Unnecessary on server side, should just be done by the client
* Rename "autoDownloadAheadLimit" to "autoDownloadNewChaptersLimit"
* Deprecate the old field
* Update Stable WebUI
* Update Stable WebUI
---------
Co-authored-by: Syer10 <syer10@users.noreply.github.com>
* Rename IncludeInUpdate class to IncludeOrExclude
Signed-off-by: Chance Zibolski <chance.zibolski@gmail.com>
* Add support for configuring which categories are downloaded automatically
If a manga has no configured categories, behavior remains the same and
the automatic download functionality will download new chapters without
consulting the category includeInDownload configuration.
Signed-off-by: Chance Zibolski <chance.zibolski@gmail.com>
---------
Signed-off-by: Chance Zibolski <chance.zibolski@gmail.com>
* add trackers support
* Cleanup Tracker Code
* Add GraphQL support for Tracking
* Fix lint and deprecation errors
* remove password from logs
* Fixes after merge
* Disable tracking for now
* More disabled
---------
Co-authored-by: Syer10 <syer10@users.noreply.github.com>
Subtracting 1 from the first chapter to download index caused an additional chapter to get downloaded (e.g. limit 4 would download 5 chapters)
Regression was introduced with 05bf4f5525
* Fix automatic chapter download for initial chapter list fetch
The initial fetch wasn't correctly detected which caused chapters to get downloaded.
Using index based numbers also caused the first chapter to not get downloaded due to it being omitted in the "subList" call which excludes the "toIndex".
* [Logging] Update logs
* Chapter fetch improvements
* Update previous date uploads
* Lint
* Fix backup inserts
* Remove extra maxSeenUploadDate
* Port downloaded over
* Make sure to set isDownloaded on all inserts
The function incorrectly exited early in case no latest read chapter was found.
This rendered disabling the setting "excludeEntryWithUnreadChapters" useless.
* Keep initial fetch date of existing chapters on a list fetch
The fetch at date should not get updated for existing chapters.
Updating this field makes it impossible to detect which chapters were actually newly fetched.
To get the last fetched timestamp of the chapters, the "chaptersLastFetchedAt" field of the manga should be used
got changed in 6d33d72663
* Get real chapter url safely
In case this causes an exception, it should not cause the whole list fetch to fail
was removed in 6d33d72663
In case download ahead is disabled, all new chapters should get downloaded.
Due to incorrectly calculating the index of the first new chapter to download, no new chapter was downloaded at all
* Switch to new Ktlint plugin
* Add ktlintCheck to PR builds
* Run formatter
* Put ktlint version in libs toml
* Fix lint
* Use Zip4Java from libs.toml
Since the number of chapters gets converted to be index based, 1 available chapter would result in 0.
Due to this, in case a manga had exactly one chapter before updating the chapters, it was incorrectly detected as the initial fetch and the new chapters did not get automatically downloaded.
* Rename "newChapters" to "updatedChapterList"
* Do not auto download new chapters of entries with unread chapters
Makes it possible to prevent unnecessary chapter downloads in case the entry hasn't yet been caught up
* Optionally limit auto new chapter downloads
* Prevent downloading new chapters for mangas not in the library
* Make server config value changes subscribable
* Make server config value changes subscribable - Update usage
* Add util functions to listen to server config value changes
* Listen to server config value changes - Auto backups
* Listen to server config value changes - Auto global update
* Listen to server config value changes - WebUI auto updates
* Listen to server config value changes - Javalin update ip and port
* Listen to server config value changes - Update socks proxy
* Listen to server config value changes - Update debug log level
* Listen to server config value changes - Update system tray icon
* Update config values one at a time
In case settings are changed in quick succession it's possible that each setting update reverts the change of the previous changed setting because the internal config hasn't been updated yet.
E.g.
1. settingA changed
2. settingB changed
3. settingA updates config file
4. settingB updates config file (internal config hasn't been updated yet with change from settingA)
5. settingA updates internal config (settingA updated)
6. settingB updates internal config (settingB updated, settingA outdated)
now settingA is unchanged because settingB reverted its change while updating the config with its new value
* Always add log interceptor to OkHttpClient
In case debug logs are disabled then the KotlinLogging log level will be set to level > debug and thus, these logs won't get logged
* Rename "maxParallelUpdateRequests" to "maxSourcesInParallel"
* Use server setting "maxSourcesInParallel" for downloads
* Listen to server config value changes - downloads
* Always use latest server settings - Browser
* Always use latest server settings - folders
* [Test] Fix type error
Gets already called by "Chapter::fetchChapterList", thus, this is unnecessary.
Additionally, "chapters.toList()" and "chapters.map()" have to be called in a transaction block, which they are not, and thus, cause an unhandled exception, breaking the mutation
* Change log level of download error
* Change type of sourceId in Downloader
Unclear why it was converted to Long since it just got converted back to String anyway when it was used in the Downloader
* Only stop downloads from source of the Downloader
The downloader just changed the state of all downloads, ignoring if they are from the source the Downloader is for or not
* Remove unnecessary DownloadManager::start calls
In case chapters were added to the queue the DownloadManager will start itself
* Extract download filtering into property
* Improve Downloader logging
* Notify clients only in case Downloader was started
In case nothing was done there is nothing to notify about
* Do not start Downloaders for failed downloads
In case there were failed chapter downloads in the queue the DownloadManager still created a Downloader and started it.
This Downloader would than immediately call "onComplete", since there is no available download, which then would refresh the Downloaders again which created an infinite loop until the failed download got removed from the queue
* Retry download in case it failed it gets re-added to the queue
In case a failed downloaded that was still in the queue was tried to get added to the queue again, nothing happened.
Instead of doing nothing, the download should get retried.
Thus, it also provides the logic to easily retry a failed download by just "adding" the chapter to the queue again.
Currently, to retry a failed download, the download has to be removed from the queue and then get re-added.
* Rename function "unqueue" to "dequeue"
* Move "dequeue" function
* Extract dequeue logic into function
* Improve DownloadManager logging
* Override "toString" of DownloadChapter
