royalroadupdates/PLAN.md

RoyalRoad Updates — Project Plan

Overview

A Go CLI tool that reads RoyalRoad story URLs from a PocketBase collection, fetches each story's last-update time from RoyalRoad, and prints a sorted plain-text table showing story names and how recently each was updated.

Behavior

1. Read the collection endpoint URL baked in at compile time.
2. Query the storyurls collection. All rows are valid; no filtering needed.
3. Extract the url field from each matching record.
4. For each URL, sequentially fetch the RoyalRoad fiction page.
5. Extract the fiction name and dateModified from the embedded JSON-LD <script type="application/ld+json"> block.
6. Sort results by dateModified descending (most recent first).
7. Print a plain-text table with:
   • Story Name column
   • Last Update column showing relative time (e.g., "3 hours ago")
8. When --json is passed, output machine-readable JSON with actual ISO 8601 dates.

URL Filtering Rules

• Skip chapter URLs silently. A chapter URL contains /chapter/ in its path. If a bookmark points to a specific chapter, the user is not yet caught up; no update notification is wanted.
• Skip non-RoyalRoad URLs silently. Only URLs whose host ends with royalroad.com (with or without www.) are processed.
• Skip silently on any fetch or parse failure. A single bad URL must not crash the tool or clutter output.

Error Handling

Scenario	Behaviour
PocketBase unreachable or fetch fails	Exit code ≠ 0, error message on stderr
Zero valid RoyalRoad stories found after filtering	Exit code 0, empty table
Individual RoyalRoad page fails to fetch or parse	Silent skip (no output, no error)

Tech Stack

Layer	Choice
Language	Go
Build orchestration	Makefile
Go compiler	Official golang:1.22 Docker image (no local toolchain)
HTTP client	net/http (standard library)
JSON parsing	encoding/json (standard library)
HTML extraction	Regex/string search for <script type="application/ld+json">
Date parsing	time.Parse(time.RFC3339, ...) (standard library)
Relative-time formatting	Custom helper from time.Time

No external Go dependencies. The project is deliberately small and uses only the standard library.

Compile-Time Configuration

The collection endpoint URL is compiled into the binary via -ldflags:

PBURL=https://example.com/api/collections/storyurls/records

This is read from the .env file at build time. The .env key is PB_URL; the Makefile passes its value via -ldflags "-X main.PBURL=${PB_URL}".

Output Formats

Default (plain-text table)

Story Name                           Last Update
-----------------------------------  -----------
A Fractured Truth                    3 hours ago
Mother of Learning                   6 years ago

--json

[
  {
    "name": "A Fractured Truth",
    "date": "2024-12-11T08:55:49Z"
  },
  {
    "name": "Mother of Learning",
    "date": "2023-07-06T17:47:41Z"
  }
]

Dates are always sortable ISO 8601 / RFC3339 strings in JSON output.

Data Flow

PocketBase (storyurls collection)
        └── Extract: url field
             └── For each URL:
                  ├── Skip if not royalroad.com
                  ├── Skip if /chapter/ in path
                  ├── Fetch fiction page (sequential, 30s timeout)
                  ├── Extract JSON-LD script tag
                  ├── Parse name + dateModified
                  └── Skip silently on any error
                  └── Collect valid results
                       └── Sort by dateModified descending
                            └── Print table or JSON

Development Loop

make check runs the full pipeline:

1. gofmt -d — fail if any file is unformatted.
2. go vet ./... — static analysis.
3. go test ./... — unit tests plus end-to-end test:
   • Compiles the binary using the local .env credentials.
   • Runs ./royalroadupdates --json.
   • Asserts:
     • exit code 0
     • stderr empty
     • stdout is valid JSON
     • JSON array contains ≥ 3 entries
     • Each entry has name and date fields
4. If any step fails: fix issues, rerun make check.

Open Decisions / Future Tuning

• Rate limiting: Requests are sequential to be gentle on RoyalRoad. If Cloudflare blocks the tool, add a per-request delay (e.g., time.Sleep).
• Relative time precision: Currently shows minutes/hours/days/years as coarse buckets. This can be refined if desired.