* Implement SyncYomi
* Add ability to select what to sync
* Properly fix default category bug
* Add periodic sync
* Add PostgreSQL support
* Deschedule previous task
* Check if SyncYomi is enabled in syncData function
* Don't allow multiple syncs at the same time
* Convert SyncYomiSyncService to object
* Make startSync non-suspend
* Return a result from startSync
* Sync before library update
* Improvements
* Use NetworkHelper client
* Lint
* Use measureTime
* Database improvements
- Move entire sync operation into a single transaction
- Stop loading all manga to memory
* Revert "Database improvements"
This reverts commit bee8d214c3.
* Actual database improvements
* Remove runBlocking
* Remove title check
* Update updateNonFavorites function
* Update timeout code
* Improve PostgreSQL query
* Create lastSyncState variable
* Create lastSyncStatus query
* Convert lastSyncState to StateFlow
* Create lastSyncStatusChange subscription
* Replace backupRestoreStatus with backupRestoreId
* Add startDate and endDate
* Add logs for sync start and end
* Handle all errors in syncData
* Change category restore function to match Mihon's behavior
* Fix comment
* Remove duplicate BackupMangaHandler.backup call
* Remove duplicated log
* Rename subscription to syncStatusChanged
* Use same flags for restoring
* Update syncInterval config to use DurationSetting
* Update sync scheduling logic
* Reorder conditions to reduce database calls
* Prevent deleted ghost chapters from reappearing during sync
jobobby04/TachiyomiSY#1575
* Improve sync merging categories
jobobby04/TachiyomiSY#1559
* Make columns not null
* Improve H2 triggers
* Add documentation
Both filters were inversed. `notAll` did what `notAny` was supposed to do and vise versa
Co-authored-by: Mitchell Syer <Syer10@users.noreply.github.com>
* Revert "[#1739] Support sending floats (#1740)"
This reverts commit c1f2aae90d.
Closes#1746
* Use `DoubleFilter` for GQL interface, convert to `FloatFilter`
Closes#1739 (again)
* Basic JWT implementation
* Move JWT to UI_LOGIN mode and bring back SIMPLE_LOGIN as before
* Update server/src/main/kotlin/suwayomi/tachidesk/global/impl/util/Jwt.kt
Co-authored-by: Mitchell Syer <Syer10@users.noreply.github.com>
* Refresh: Update only access token
Co-authored-by: Mitchell Syer <Syer10@users.noreply.github.com>
* Implement JWT Audience
* Store JWT key
Generates the key on startup if not set
* Handle invalid Base64
* Make JWT expiry configurable
* Missing value parse
* Update server/src/main/kotlin/suwayomi/tachidesk/global/impl/util/Jwt.kt
Co-authored-by: Mitchell Syer <Syer10@users.noreply.github.com>
* Simplify Duration parsing
* JWT Protect Mutations
* JWT Protect Queries and Subscriptions
* JWT Protect v1 WebSockets
* WebSockets allow sending token via protocol header
* Also respect the `suwayomi-server-token` cookie
* JWT reduce default token expiry
* JWT Support cookie on WebSocket as well
* Lint
* Authenticate graphql subscription via connection_init payload
* WebView: Prefer explicit token over cookie
This hack was implemented because WebView sent `"null"` if no token was
supplied, just don't send a bad token, then we can do this properly
* WebView: Implement basic login dialog if no token supplied
---------
Co-authored-by: Mitchell Syer <Syer10@users.noreply.github.com>
Co-authored-by: schroda <50052685+schroda@users.noreply.github.com>
* feat(sync/koreader): implement reading progress synchronization
This commit introduces a comprehensive integration with KOReader Sync Server to enable two-way synchronization of reading progress.
The core logic is encapsulated in a new `KoreaderSyncService`, which handles authentication, registration, and progress pushing/pulling based on user-defined strategies (LATEST, KOSYNC, SUWAYOMI).
Key changes include:
- A new GraphQL API is added to manage the KOReader Sync connection:
- `connectKoSyncAccount` mutation provides a simplified flow that attempts to log in and, if the user doesn't exist, automatically registers them.
- `logoutKoSyncAccount` mutation to clear credentials.
- `koSyncStatus` query to check the current connection status.
- Reading progress is now synchronized at key points:
- The `fetchChapterPages` mutation pulls the latest progress from the sync server before loading the reader. It respects the configured sync strategy and updates the local database if necessary.
- The `updateChapters` and other progress-updating methods now push changes to the sync server automatically.
- OPDS chapter entries also pull the latest progress, ensuring clients receive up-to-date reading status.
- Supporting backend changes have been made:
- The `Chapter` table is extended with a `koreader_hash` column to uniquely identify documents. A database migration is included.
- New configuration options are added to `server.conf` to manage the feature (enable, server URL, credentials, strategy, etc.).
* perf(opds): defer KOReader sync to improve chapter feed performance
Removes the KOReader Sync progress-pulling logic from the `createChapterListEntry` function.
The previous implementation triggered a network request to the sync server for every single chapter when generating a list, leading to severe performance issues and slow load times on feeds with many entries.
This change reverts to the more performant approach of always linking to the chapter's metadata feed. The expensive sync operation will be handled within the metadata entry generation instead, ensuring it's only triggered on-demand for a single chapter. This restores the responsiveness of browsing chapter feeds.
* refactor(koreader): Use enums for sync settings and correct OPDS logic
Refactor Koreader Sync settings to use enums instead of raw strings for `checksumMethod` and `strategy`. This improves type safety, prevents typos, and makes the configuration handling more robust.
The changes include:
- Introducing `KoreaderSyncChecksumMethod` and `KoreaderSyncStrategy` enums.
- Updating `ServerConfig`, GraphQL types, and backup models to use these new enums.
- Refactoring `KoreaderSyncService` to work with the enum types.
Additionally, this commit fixes an issue in `OpdsEntryBuilder` where the logic for determining which sync progress to use (local vs. remote) was duplicated. The builder now correctly delegates this decision to `KoreaderSyncService.pullProgress`, which already contains the necessary strategy logic. This centralizes the logic and ensures consistent behavior.
* refactor(koreader): Improve config handling and remove redundant update
This commit combines several refactoring and cleanup tasks:
- **Koreader Sync:** The sync service is updated to use the modern `serverConfig` provider instead of the legacy `GlobalConfigManager`. This aligns it with the current configuration management approach in the project.
- **Download Provider:** A redundant `pageCount` database update is removed from `ChaptersFilesProvider`. This operation was unnecessary because the `getChapterDownloadReady` function, which is called earlier in the download process, already verifies and corrects the page count. This change eliminates a superfluous database write and fixes a related import issue.
* feat(sync/koreader)!: enhance sync strategy and add progress tolerance
This commit overhauls the KOReader synchronization feature to provide more granular control and robustness. The simple on/off toggle has been replaced with a more flexible strategy-based system.
Key changes include:
- Replaced `koreaderSyncEnabled` with a more powerful `koreaderSyncStrategy` enum.
- Introduced new sync strategies: `PROMPT`, `SILENT`, `SEND`, `RECEIVE`, and `DISABLE`, allowing for fine-grained control over the sync direction and conflict resolution.
- Added a `koreaderSyncPercentageTolerance` setting. This prevents unnecessary sync updates for minor progress differences between Suwayomi and KOReader.
- Refactored the `KoreaderSyncService` to implement the new strategies and use the configurable tolerance.
- Updated GraphQL schemas, mutations, and server configuration to remove the old setting and incorporate the new ones.
- Adjusted the backup and restore process to correctly handle the new configuration parameters.
- Modified API endpoints and internal logic to check and apply remote progress based on the selected strategy.
BREAKING CHANGE: The `koreaderSyncEnabled` setting is removed and replaced by a more granular `koreaderSyncStrategy`. The enum values for the strategy have been completely changed, making previous configurations incompatible.
* fix: remove unused imports
* feat(opds, sync): enhance Koreader sync and OPDS conflict handling
This commit introduces significant improvements to the Koreader synchronization feature, focusing on providing a better user experience for handling progress conflicts in both OPDS and GraphQL clients.
Key changes include:
- **OPDS Conflict Resolution:** When a reading progress conflict is detected, the OPDS feed for a chapter now provides two distinct "Read Online" links: one to continue from the local progress and another to continue from the synced progress from the remote device (e.g., "Continue Reading Online (Synced from KOReader)"). This empowers users to choose which progress to follow.
- **GraphQL Sync Conflict Information:** The `fetchChapterPages` GraphQL mutation now includes a `syncConflict` field in its payload. This field provides the remote device name and page number, allowing GraphQL clients to implement a user-facing prompt to resolve sync conflicts.
- **Improved Sync Strategy Handling:**
- The `connectKoSyncAccount` mutation no longer unconditionally sets the sync strategy to `PROMPT`. It now respects the user's existing setting, preventing accidental configuration changes upon re-login.
- The default `koreaderSyncStrategy` in the configuration is changed to `DISABLED`, providing a safer and more intuitive default for new users.
- **Refinements & Fixes:**
- The fallback for the remote device name is now centralized within the KoreaderSyncService, defaulting to "KOReader" if not provided.
- Renamed `KoreaderSyncStrategy.DISABLE` to `DISABLED` for consistency.
- Updated i18n strings for OPDS links to be more descriptive and user-friendly.
* refactor(kosync): rename stream page link titles for consistency
* refactor(kosync): return SettingsType in auth mutation payloads
The `connectKoSyncAccount` and `logoutKoSyncAccount` mutations modify server settings (username and userkey) but did not previously return the updated configuration. This forced client applications to manually refetch settings to avoid a stale cache.
This change modifies the payloads for both mutations to include the full `SettingsType`.
By returning the updated settings directly, GraphQL clients like Apollo Client can automatically update their cache, simplifying client-side state management and ensuring the UI always reflects the current server configuration.
Additionally, `clientMutationId` has been added to `KoSyncConnectPayload` for consistency with GraphQL practices, aligning it with the logout mutation.
Refs: #1560
* refactor(kosync): replace KoSyncConnectPayload with ConnectResult in connect method
* fix(kosync): add koreaderSyncPercentageTolerance default setting
* Export meta data
* Import meta data
* Add missing "opdsUseBinaryFileSize" setting to gql
* Export server settings
* Import server settings
* Streamline server config enum handling
* Use "restore amount" in backup import progress
* Remove existing installations with msi installer
* Remove unused x86 wxs file
* Uninstall old msi versions with different upgrade code
* Progress but error 2721 happens on install
* Remove added uninstall previous version wxs stuff
* Use revision as patch number
MSI only uninstalls previous versions in case the version number changed (it only checks the first three numbers (major, minor, patch)).
Thus, to prevent each preview install to result in it getting registered as a new "app" and for it to uninstall the old versions, we have to change the version on each release.
* Deprecate "BuildConfig.REVISION"
* Remove outdated env vars
---------
Co-authored-by: Syer10 <syer10@users.noreply.github.com>
The update subscription emitted the full update status, which, depending on how big the status was, took forever because the graphql subscription does not support data loader batching, causing it to run into the n+1 problem
* Update to exposed-migrations v3.5.0
* Update to kotlin-logging v7.0.0
* Update to exposed v0.46.0
* Update to exposed v0.47.0
* Update to exposed v0.55.0
* Update to exposed v0.56.0
* Update to exposed v0.57.0
* Emit only download changes instead of full status
The download subscription emitted the full download status, which, depending on how big the queue was, took forever because the graphql subscription does not support data loader batching, causing it to run into the n+1 problem
* Rename "DownloadManager#status" to "DownloadManager#updates"
* Add initial queue to download subscription type
Adds the current queue at the time of sending the initial message.
This field is null for all following messages after the initial one
* Optionally limit and omit download updates
To prevent the n+1 dataloader issue, the max number of updates included in the download subscription can be limited.
This way, the problem will be circumvented and instead, the latest download status should be (re-)fetched via the download status query, which does not run into this problem.
* Formatting
* Remove code duplication
* Remove unnecessary functions
* Simplify filtering for multiple values in queries
Makes it easier to filter for multiple values at ones without having to nest filters with multiple "and".
e.g.
```gql
query MyQuery {
mangas(
filter: {genre: {includesInsensitive: "action"}, and: {genre: {includesInsensitive: "adventure"}, and: { ... }}}
) {
nodes {
id
}
}
}
```
can be simplified to
```gql
query MyQuery {
mangas(
filter: {genre: {includesInsensitive: ["action", "adventure", ...]}}
) {
nodes {
id
}
}
}
```
* Add filter for matching "any" value in list
Makes it easier to filter for entries that match any value without having to nest filters with multiple "or".
e.g.
```gql
query MyQuery {
mangas(
filter: {genre: {includesInsensitiveAny: ["action", "adventure", ...]}}
) {
nodes {
id
}
}
}
```
instead of
```gql
query MyQuery {
mangas(
filter: {genre: {includesInsensitive: "action", or: {genre: includesInsensitive: "adventure", or: {...}}}}
) {
nodes {
id
}
}
}
```
* Add util function to apply "andWhere/All/Any"
* Include tracking in validation of backup
* Always return track records
Not clear why an empty list should be returned in case no trackers are logged in
* Include tracking in backup creation
* Restore tracking from backup
* Set updater running flag to false only at the end of the update
For clearing the data loader cache properly, the update status subscription requires the update to be running.
For the last completed manga update the flag was immediately set to false which prevented the dataloader cache from getting cleared, returning outdated data for the last updated manga
* Correctly clear the "MangaForIdsDataLoader" cache
The cache keys for this dataloader are lists of manga ids.
Thus, it is not possible to clear only the cached data of the provided manga id and instead each cache entry that includes the manga id has to be cleared
* Ensure that manga dataloader caches gets cleared during global update
The "StateFlow" drops value updates in case the collector is too slow, which was the case for the "UpdateSubscription".
This caused the dataloader cache to not get properly cleared because the running state of the update was already set to false.
* Run functions for specific webui flavor
* Set default flavor of WebUIFlavor enum
* Consider flavor of served webUI when checking for update
In case the flavor was changed and the served webui files are still for the previous flavor, the update check could incorrectly detect no update
* Skip validation during initial setup
In case initial setup is triggered because of an invalid local webUI, doing a validation again is unnecessary
* Handle changed flavor on startup
* add trackers support
* Cleanup Tracker Code
* Add GraphQL support for Tracking
* Fix lint and deprecation errors
* remove password from logs
* Fixes after merge
* Disable tracking for now
* More disabled
---------
Co-authored-by: Syer10 <syer10@users.noreply.github.com>
* Use a new type for the webui about info query
Using the same type for this and the webui update queries/mutations caused apollo to save it as the same data in the cache, overwriting the "about info"
* Use a new type for the webui about check query
To prevent similar issues (cc3bf5f34a8afebadd306d037db1a10088ef9334) with the "update check" and the "update progress" payloads
* Throw update check error when calling it via the query
Otherwise, the error is never raised to the frontend
* Set "ERROR" state in case the update check failed on WebUI update trigger
* Reset backup status to idle in case of an exception
* Rename "performRestore" function
* Set backup status to failure on exception
Makes it possible to detect if the restore failed or not after the first status was received
* Set backup status to success on completion
Since the status is not provided over a subscription, but over a query that should be pulled, it is not really easily detectable if a restore finished or not, since both states will be indicated by "idle"
* Correctly wait for first new status when triggering backup import
The status is only "Idle" in case no backup import has ever run.
Once the first backup process finished it is either "Failure" or "Success"
* Rename "ProtoBackupImport::restore" function
* Add id to restore process
Makes it possible to differentiate between backup restore processes.
* Address build warnings and cleanup
* Actual name of who defined the protocol
* Remove uneeded detekt supression
* GraphQL Before-After cleanup
* Lint
* Cleanup unused functions
* Fix some discrepancies with the 1.5 source api and fix lang exception
* Filter mangas based on each genre of the genre condition
Genres are stored as a comma separated string in the database.
Thus, unless the correct string was passed in the condition, no manga would match the condition.
* Query mangas based on genre filter
* Switch to new Ktlint plugin
* Add ktlintCheck to PR builds
* Run formatter
* Put ktlint version in libs toml
* Fix lint
* Use Zip4Java from libs.toml
When using cursors for pagination while sorting, the sort order was inverted (desc -> asc, asc -> desc).
However, this was then not considered when selecting results based on the cursor.
For before/after results where always selected via greater/less.
Due to inverting the sort order, this also needs to be inverted depending on the sort order (desc or asc).
Flow::stateIn has "Strong equality-based conflation" (see documentation).
Thus, it omits every value in case it's equal to the previous one.
Since the DownloadManger::getStatus function returns a status with a queue, that contains all current "DownloadChapters" by reference, the equality check was always true.
Thus, progress changes of downloads were never sent to subscribers.
Subscriber were only notified about finished downloads (size of queue changed) or downloader status changes
In case e.g. no manga exists for the passed id, the query returned null.
This makes it harder to have a "streamlined" error handling in the client, since these types of queries need a special handling.
* Provide last global update timestamp
* Provide skipped mangas in update status
* Extract update status logic into function
* Rename update "statusMap" to "mangaStatusMap"
* Provide info about categories in update status
* Add "uiName" to WebUI enum
* Add "Custom" WebUI to enum
* Rename "WebUI" enum to "WebUIFlavor"
* Add "WebUIInterface" enum
* Add query for server settings
* Add mutation for server settings
* Add mutation to reset the server settings
* Only update the config in case the value changed
In case the value of the config is already the same as the new value of the state flow, it is not necessary to update the config file
* Make server config value changes subscribable
* Make server config value changes subscribable - Update usage
* Add util functions to listen to server config value changes
* Listen to server config value changes - Auto backups
* Listen to server config value changes - Auto global update
* Listen to server config value changes - WebUI auto updates
* Listen to server config value changes - Javalin update ip and port
* Listen to server config value changes - Update socks proxy
* Listen to server config value changes - Update debug log level
* Listen to server config value changes - Update system tray icon
* Update config values one at a time
In case settings are changed in quick succession it's possible that each setting update reverts the change of the previous changed setting because the internal config hasn't been updated yet.
E.g.
1. settingA changed
2. settingB changed
3. settingA updates config file
4. settingB updates config file (internal config hasn't been updated yet with change from settingA)
5. settingA updates internal config (settingA updated)
6. settingB updates internal config (settingB updated, settingA outdated)
now settingA is unchanged because settingB reverted its change while updating the config with its new value
* Always add log interceptor to OkHttpClient
In case debug logs are disabled then the KotlinLogging log level will be set to level > debug and thus, these logs won't get logged
* Rename "maxParallelUpdateRequests" to "maxSourcesInParallel"
* Use server setting "maxSourcesInParallel" for downloads
* Listen to server config value changes - downloads
* Always use latest server settings - Browser
* Always use latest server settings - folders
* [Test] Fix type error
* Add "server" to "checkForUpdate" logic names
* Use "webUIRoot" as default path for "getLocalVersion"
* Use local version as default version for "isUpdateAvailable"
* Return the version with the webUI update check
* Update WebinterfaceManager to be async
* Add query, mutation and subscription for webUI update
* Catch error and return default error value for missing local WebUI version
