139 lines
4.6 KiB
Markdown
139 lines
4.6 KiB
Markdown
# RoyalRoad Updates — Project Plan
|
|
|
|
## Overview
|
|
|
|
A Go CLI tool that reads RoyalRoad story URLs from a PocketBase collection, fetches each
|
|
story's last-update time from RoyalRoad, and prints a sorted plain-text table showing
|
|
story names and how recently each was updated.
|
|
|
|
## Behavior
|
|
|
|
1. Authenticate to PocketBase using credentials baked in at compile time.
|
|
2. Query the `bookmarks` collection, filtering `category = "Geschichten"`.
|
|
3. Extract the `url` field from each matching record.
|
|
4. For each URL, sequentially fetch the RoyalRoad fiction page.
|
|
5. Extract the fiction name and `dateModified` from the embedded JSON-LD
|
|
`<script type="application/ld+json">` block.
|
|
6. Sort results by `dateModified` descending (most recent first).
|
|
7. Print a plain-text table with:
|
|
- `Story Name` column
|
|
- `Last Update` column showing relative time (e.g., "3 hours ago")
|
|
8. When `--json` is passed, output machine-readable JSON with actual ISO 8601 dates.
|
|
|
|
### URL Filtering Rules
|
|
|
|
- **Skip chapter URLs silently.** A chapter URL contains `/chapter/` in its path.
|
|
If a bookmark points to a specific chapter, the user is not yet caught up; no
|
|
update notification is wanted.
|
|
- **Skip non-RoyalRoad URLs silently.** Only URLs whose host ends with
|
|
`royalroad.com` (with or without `www.`) are processed.
|
|
- **Skip silently on any fetch or parse failure.** A single bad URL must not
|
|
crash the tool or clutter output.
|
|
|
|
### Error Handling
|
|
|
|
| Scenario | Behaviour |
|
|
|----------|-----------|
|
|
| PocketBase unreachable or auth fails | Exit code ≠ 0, error message on stderr |
|
|
| Zero valid RoyalRoad stories found after filtering | Exit code 0, empty table |
|
|
| Individual RoyalRoad page fails to fetch or parse | Silent skip (no output, no error) |
|
|
|
|
## Tech Stack
|
|
|
|
| Layer | Choice |
|
|
|-------|--------|
|
|
| Language | Go |
|
|
| Build orchestration | `Makefile` |
|
|
| Go compiler | Official `golang:1.22` Docker image (no local toolchain) |
|
|
| HTTP client | `net/http` (standard library) |
|
|
| JSON parsing | `encoding/json` (standard library) |
|
|
| HTML extraction | Regex/string search for `<script type="application/ld+json">` |
|
|
| Date parsing | `time.Parse(time.RFC3339, ...)` (standard library) |
|
|
| Relative-time formatting | Custom helper from `time.Time` |
|
|
|
|
**No external Go dependencies.** The project is deliberately small and uses only
|
|
the standard library.
|
|
|
|
## Compile-Time Configuration
|
|
|
|
The three PocketBase connection values are compiled into the binary via `-ldflags`:
|
|
|
|
```
|
|
PB_URL=https://pocketbase.example.com
|
|
PB_USER=username
|
|
PB_PASSWORD=password
|
|
```
|
|
|
|
These are read from a `.env` file at build time. The resulting binary runs
|
|
standalone without any environment variables present.
|
|
|
|
## Output Formats
|
|
|
|
### Default (plain-text table)
|
|
|
|
```
|
|
Story Name Last Update
|
|
----------------------------------- -----------
|
|
A Fractured Truth 3 hours ago
|
|
Mother of Learning 6 years ago
|
|
```
|
|
|
|
### `--json`
|
|
|
|
```json
|
|
[
|
|
{
|
|
"name": "A Fractured Truth",
|
|
"date": "2024-12-11T08:55:49Z"
|
|
},
|
|
{
|
|
"name": "Mother of Learning",
|
|
"date": "2023-07-06T17:47:41Z"
|
|
}
|
|
]
|
|
```
|
|
|
|
Dates are always sortable ISO 8601 / RFC3339 strings in JSON output.
|
|
|
|
## Data Flow
|
|
|
|
```
|
|
PocketBase (bookmarks collection)
|
|
└── Filter: category = "Geschichten"
|
|
└── Extract: url field
|
|
└── For each URL:
|
|
├── Skip if not royalroad.com
|
|
├── Skip if /chapter/ in path
|
|
├── Fetch fiction page (sequential, 30s timeout)
|
|
├── Extract JSON-LD script tag
|
|
├── Parse name + dateModified
|
|
└── Skip silently on any error
|
|
└── Collect valid results
|
|
└── Sort by dateModified descending
|
|
└── Print table or JSON
|
|
```
|
|
|
|
## Development Loop
|
|
|
|
`make check` runs the full pipeline:
|
|
|
|
1. `gofmt -d` — fail if any file is unformatted.
|
|
2. `go vet ./...` — static analysis.
|
|
3. `go test ./...` — unit tests plus end-to-end test:
|
|
- Compiles the binary using the local `.env` credentials.
|
|
- Runs `./royalroadupdates --json`.
|
|
- Asserts:
|
|
- exit code 0
|
|
- stderr empty
|
|
- stdout is valid JSON
|
|
- JSON array contains ≥ 3 entries
|
|
- Each entry has `name` and `date` fields
|
|
4. If any step fails: fix issues, rerun `make check`.
|
|
|
|
## Open Decisions / Future Tuning
|
|
|
|
- **Rate limiting:** Requests are sequential to be gentle on RoyalRoad. If
|
|
Cloudflare blocks the tool, add a per-request delay (e.g., `time.Sleep`).
|
|
- **Relative time precision:** Currently shows minutes/hours/days/years as
|
|
coarse buckets. This can be refined if desired.
|