* feat(sync/koreader): implement reading progress synchronization
This commit introduces a comprehensive integration with KOReader Sync Server to enable two-way synchronization of reading progress.
The core logic is encapsulated in a new `KoreaderSyncService`, which handles authentication, registration, and progress pushing/pulling based on user-defined strategies (LATEST, KOSYNC, SUWAYOMI).
Key changes include:
- A new GraphQL API is added to manage the KOReader Sync connection:
- `connectKoSyncAccount` mutation provides a simplified flow that attempts to log in and, if the user doesn't exist, automatically registers them.
- `logoutKoSyncAccount` mutation to clear credentials.
- `koSyncStatus` query to check the current connection status.
- Reading progress is now synchronized at key points:
- The `fetchChapterPages` mutation pulls the latest progress from the sync server before loading the reader. It respects the configured sync strategy and updates the local database if necessary.
- The `updateChapters` and other progress-updating methods now push changes to the sync server automatically.
- OPDS chapter entries also pull the latest progress, ensuring clients receive up-to-date reading status.
- Supporting backend changes have been made:
- The `Chapter` table is extended with a `koreader_hash` column to uniquely identify documents. A database migration is included.
- New configuration options are added to `server.conf` to manage the feature (enable, server URL, credentials, strategy, etc.).
* perf(opds): defer KOReader sync to improve chapter feed performance
Removes the KOReader Sync progress-pulling logic from the `createChapterListEntry` function.
The previous implementation triggered a network request to the sync server for every single chapter when generating a list, leading to severe performance issues and slow load times on feeds with many entries.
This change reverts to the more performant approach of always linking to the chapter's metadata feed. The expensive sync operation will be handled within the metadata entry generation instead, ensuring it's only triggered on-demand for a single chapter. This restores the responsiveness of browsing chapter feeds.
* refactor(koreader): Use enums for sync settings and correct OPDS logic
Refactor Koreader Sync settings to use enums instead of raw strings for `checksumMethod` and `strategy`. This improves type safety, prevents typos, and makes the configuration handling more robust.
The changes include:
- Introducing `KoreaderSyncChecksumMethod` and `KoreaderSyncStrategy` enums.
- Updating `ServerConfig`, GraphQL types, and backup models to use these new enums.
- Refactoring `KoreaderSyncService` to work with the enum types.
Additionally, this commit fixes an issue in `OpdsEntryBuilder` where the logic for determining which sync progress to use (local vs. remote) was duplicated. The builder now correctly delegates this decision to `KoreaderSyncService.pullProgress`, which already contains the necessary strategy logic. This centralizes the logic and ensures consistent behavior.
* refactor(koreader): Improve config handling and remove redundant update
This commit combines several refactoring and cleanup tasks:
- **Koreader Sync:** The sync service is updated to use the modern `serverConfig` provider instead of the legacy `GlobalConfigManager`. This aligns it with the current configuration management approach in the project.
- **Download Provider:** A redundant `pageCount` database update is removed from `ChaptersFilesProvider`. This operation was unnecessary because the `getChapterDownloadReady` function, which is called earlier in the download process, already verifies and corrects the page count. This change eliminates a superfluous database write and fixes a related import issue.
* feat(sync/koreader)!: enhance sync strategy and add progress tolerance
This commit overhauls the KOReader synchronization feature to provide more granular control and robustness. The simple on/off toggle has been replaced with a more flexible strategy-based system.
Key changes include:
- Replaced `koreaderSyncEnabled` with a more powerful `koreaderSyncStrategy` enum.
- Introduced new sync strategies: `PROMPT`, `SILENT`, `SEND`, `RECEIVE`, and `DISABLE`, allowing for fine-grained control over the sync direction and conflict resolution.
- Added a `koreaderSyncPercentageTolerance` setting. This prevents unnecessary sync updates for minor progress differences between Suwayomi and KOReader.
- Refactored the `KoreaderSyncService` to implement the new strategies and use the configurable tolerance.
- Updated GraphQL schemas, mutations, and server configuration to remove the old setting and incorporate the new ones.
- Adjusted the backup and restore process to correctly handle the new configuration parameters.
- Modified API endpoints and internal logic to check and apply remote progress based on the selected strategy.
BREAKING CHANGE: The `koreaderSyncEnabled` setting is removed and replaced by a more granular `koreaderSyncStrategy`. The enum values for the strategy have been completely changed, making previous configurations incompatible.
* fix: remove unused imports
* feat(opds, sync): enhance Koreader sync and OPDS conflict handling
This commit introduces significant improvements to the Koreader synchronization feature, focusing on providing a better user experience for handling progress conflicts in both OPDS and GraphQL clients.
Key changes include:
- **OPDS Conflict Resolution:** When a reading progress conflict is detected, the OPDS feed for a chapter now provides two distinct "Read Online" links: one to continue from the local progress and another to continue from the synced progress from the remote device (e.g., "Continue Reading Online (Synced from KOReader)"). This empowers users to choose which progress to follow.
- **GraphQL Sync Conflict Information:** The `fetchChapterPages` GraphQL mutation now includes a `syncConflict` field in its payload. This field provides the remote device name and page number, allowing GraphQL clients to implement a user-facing prompt to resolve sync conflicts.
- **Improved Sync Strategy Handling:**
- The `connectKoSyncAccount` mutation no longer unconditionally sets the sync strategy to `PROMPT`. It now respects the user's existing setting, preventing accidental configuration changes upon re-login.
- The default `koreaderSyncStrategy` in the configuration is changed to `DISABLED`, providing a safer and more intuitive default for new users.
- **Refinements & Fixes:**
- The fallback for the remote device name is now centralized within the KoreaderSyncService, defaulting to "KOReader" if not provided.
- Renamed `KoreaderSyncStrategy.DISABLE` to `DISABLED` for consistency.
- Updated i18n strings for OPDS links to be more descriptive and user-friendly.
* refactor(kosync): rename stream page link titles for consistency
* refactor(kosync): return SettingsType in auth mutation payloads
The `connectKoSyncAccount` and `logoutKoSyncAccount` mutations modify server settings (username and userkey) but did not previously return the updated configuration. This forced client applications to manually refetch settings to avoid a stale cache.
This change modifies the payloads for both mutations to include the full `SettingsType`.
By returning the updated settings directly, GraphQL clients like Apollo Client can automatically update their cache, simplifying client-side state management and ensuring the UI always reflects the current server configuration.
Additionally, `clientMutationId` has been added to `KoSyncConnectPayload` for consistency with GraphQL practices, aligning it with the logout mutation.
Refs: #1560
* refactor(kosync): replace KoSyncConnectPayload with ConnectResult in connect method
* fix(kosync): add koreaderSyncPercentageTolerance default setting
* [#1496] First conversion attempt
* [#1496] Configurable conversion
* Fix: allow nested configs (map)
* [#1496] Support explicit `none` conversion
* Use MimeUtils for provided download
* [1496] Support image conversion on load for downloaded images
* Lint
* [#1496] Support conversion on fresh download as well
Previous commit was only for already downloaded images, now also for
fresh and cached
* [#1496] Refactor: Move where conversion for download happens
* Rewrite config handling, improve custom types
* Lint
* Add format to pages mutation
* Lint
* Standardize url encode
* Lint
* Config: Allow additional conversion parameters
* Implement conversion quality parameter
* Lint
* Implement a conversion util to allow fallback readers
* Add downloadConversions to api and backup, fix updateValue issues
* Lint
* Minor cleanup
* Update libs.versions.toml
---------
Co-authored-by: Syer10 <syer10@users.noreply.github.com>
* [#1349] Stub basic cookie authentication
* [#1349] Basic login page
Also adjusts WebView header color and shadow to match WebUI. WebUI uses
a background-image gradient to change the perceived color, which was not
noticed originally.
* [#1349] Handle login post
* [#1349] Redirect to previous URL
* [#1349] Return a basic 401 for api endpoints
Instead of redirecting to a visual login page, API should just indicate
the bad state
* Use more appropriate 303 redirect
* Update server/src/main/kotlin/suwayomi/tachidesk/server/JavalinSetup.kt
Co-authored-by: Mitchell Syer <Syer10@users.noreply.github.com>
* Update server/src/main/kotlin/suwayomi/tachidesk/server/JavalinSetup.kt
Co-authored-by: Mitchell Syer <Syer10@users.noreply.github.com>
* Lint
* Transition to AuthMode enum with migration path
* Make basicAuthEnabled auto property, Lint
* ConfigManager: Make sure to re-parse the config after migration
* basicAuth{Username,Password} -> auth{Username,Password}
* Lint
* Update server settings backup model
* Update comment
* Minor cleanup
* Improve backup legacy settings fix
* Lint
* Simplify config value migration
---------
Co-authored-by: Mitchell Syer <Syer10@users.noreply.github.com>
* Export meta data
* Import meta data
* Add missing "opdsUseBinaryFileSize" setting to gql
* Export server settings
* Import server settings
* Streamline server config enum handling
* Use "restore amount" in backup import progress
The update subscription emitted the full update status, which, depending on how big the status was, took forever because the graphql subscription does not support data loader batching, causing it to run into the n+1 problem
* Optimize restoring manga chapters
* Streamline restoring manga data
* Optimize restoring manga trackers
* Simplify passing manga category restore data
* Properly prevent mangas from getting added to default category
76595233fc never actually worked...
* Extract logic to add manga to categories from gql mutation
* Optimize restoring manga categories
* Optimize restoring categories
coerceIn throws an error in case the max value is less than the min value ("Cannot coerce value to an empty range: maximum <max> is less than minimum <min>")
Regression from c8bd39b4bf
* Update to exposed-migrations v3.5.0
* Update to kotlin-logging v7.0.0
* Update to exposed v0.46.0
* Update to exposed v0.47.0
* Update to exposed v0.55.0
* Update to exposed v0.56.0
* Update to exposed v0.57.0
* Validate setting values on mutation
* Handle invalid negative setting values
* Ensure at least one source is downloading at all times
* Prevent possible IllegalArgumentException
The "serverConfig.maxSourcesInParallel" value could have changed after the if-condition
* Emit only download changes instead of full status
The download subscription emitted the full download status, which, depending on how big the queue was, took forever because the graphql subscription does not support data loader batching, causing it to run into the n+1 problem
* Rename "DownloadManager#status" to "DownloadManager#updates"
* Add initial queue to download subscription type
Adds the current queue at the time of sending the initial message.
This field is null for all following messages after the initial one
* Optionally limit and omit download updates
To prevent the n+1 dataloader issue, the max number of updates included in the download subscription can be limited.
This way, the problem will be circumvented and instead, the latest download status should be (re-)fetched via the download status query, which does not run into this problem.
* Formatting
* Keep up to 31 log files
On average one log file per day gets created, thus, increasing to 31 files will store log files for one month
* Decrease total log files size to 100mb
* Make log appender settings configurable
In case a manga gets added to the library which has not been initialized yet, it should be tried to initialize it.
Since it's not an error to have uninitialized manga in the library, this can be done in the background via the updater and the client receives the updated data via the update subscription.
In case e.g. a mutation was made which looked like this
myMutation {
mutationA { ... }
mutationB { ... }
mutationC { ... }
}
and mutation A and B succeeded while mutation C failed, the response only included the error of C and the successful mutation data response of A and B was missing
* Update test/server-reference file
* Properly handle re-uploaded chapters in auto download of new chapters
In case of unhandable re-uploaded chapters (different chapter numbers) they potentially would have prevented auto downloads due being considered as unread.
Additionally, they would not have been considered to get downloaded due to not having a higher chapter number than the previous latest existing chapter before the chapter list fetch.
* Add option to ignore re-uploads for auto downloads
* Extract check for manga category download inclusion
* Extract logic to get new chapter ids to download
* Simplify manga category download inclusion check
In case the DEFAULT category does not exist, someone messed with the database and it is basically corrupted
* Add mutation to fetch the latest track data from the tracker
* Update Track.kt
---------
Co-authored-by: Mitchell Syer <Syer10@users.noreply.github.com>
* Extract unbinding track into function
* Introduce new unbind mutation
* Add option to delete track binding on track service
---------
Co-authored-by: Mitchell Syer <Syer10@users.noreply.github.com>
Triggering the progress update on server side does not work because the client needs to get the mutation result, otherwise, the clients cache will get outdated
* Set updater running flag to false only at the end of the update
For clearing the data loader cache properly, the update status subscription requires the update to be running.
For the last completed manga update the flag was immediately set to false which prevented the dataloader cache from getting cleared, returning outdated data for the last updated manga
* Correctly clear the "MangaForIdsDataLoader" cache
The cache keys for this dataloader are lists of manga ids.
Thus, it is not possible to clear only the cached data of the provided manga id and instead each cache entry that includes the manga id has to be cleared
* Ensure that manga dataloader caches gets cleared during global update
The "StateFlow" drops value updates in case the collector is too slow, which was the case for the "UpdateSubscription".
This caused the dataloader cache to not get properly cleared because the running state of the update was already set to false.
* Remove download ahead logic
Unnecessary on server side, should just be done by the client
* Rename "autoDownloadAheadLimit" to "autoDownloadNewChaptersLimit"
* Deprecate the old field
* Update Stable WebUI
* Update Stable WebUI
---------
Co-authored-by: Syer10 <syer10@users.noreply.github.com>
* Run functions for specific webui flavor
* Set default flavor of WebUIFlavor enum
* Consider flavor of served webUI when checking for update
In case the flavor was changed and the served webui files are still for the previous flavor, the update check could incorrectly detect no update
* Skip validation during initial setup
In case initial setup is triggered because of an invalid local webUI, doing a validation again is unnecessary
* Handle changed flavor on startup
* Rename IncludeInUpdate class to IncludeOrExclude
Signed-off-by: Chance Zibolski <chance.zibolski@gmail.com>
* Add support for configuring which categories are downloaded automatically
If a manga has no configured categories, behavior remains the same and
the automatic download functionality will download new chapters without
consulting the category includeInDownload configuration.
Signed-off-by: Chance Zibolski <chance.zibolski@gmail.com>
---------
Signed-off-by: Chance Zibolski <chance.zibolski@gmail.com>
* add trackers support
* Cleanup Tracker Code
* Add GraphQL support for Tracking
* Fix lint and deprecation errors
* remove password from logs
* Fixes after merge
* Disable tracking for now
* More disabled
---------
Co-authored-by: Syer10 <syer10@users.noreply.github.com>
* Remove "updateAvailable" from webui update info
Doesn't add anything
* Extract status creation into function
* Optionally emit status immediately
Otherwise, some emissions can get lost due to the 1s sample period
* Rename "STOPPED" state to "IDLE"
* Reset webui update status
Currently, the update status never gets reset.
Thus, it will be "FINISHED" or "ERROR" until the next update gets triggered.
Due to this, the client won't know that the update result was already handled and will handle it again the next time it gets the update status.
To prevent this, the client has to be able to tell the server that it has handled the update result and that it can be resetted
* Use a new type for the webui about info query
Using the same type for this and the webui update queries/mutations caused apollo to save it as the same data in the cache, overwriting the "about info"
* Use a new type for the webui about check query
To prevent similar issues (cc3bf5f34a8afebadd306d037db1a10088ef9334) with the "update check" and the "update progress" payloads
* Throw update check error when calling it via the query
Otherwise, the error is never raised to the frontend
* Set "ERROR" state in case the update check failed on WebUI update trigger
* Reset backup status to idle in case of an exception
* Rename "performRestore" function
* Set backup status to failure on exception
Makes it possible to detect if the restore failed or not after the first status was received
* Set backup status to success on completion
Since the status is not provided over a subscription, but over a query that should be pulled, it is not really easily detectable if a restore finished or not, since both states will be indicated by "idle"
* Correctly wait for first new status when triggering backup import
The status is only "Idle" in case no backup import has ever run.
Once the first backup process finished it is either "Failure" or "Success"
* Rename "ProtoBackupImport::restore" function
* Add id to restore process
Makes it possible to differentiate between backup restore processes.
