jazzman/royalroadupdates
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Erstes Commit

Browse Source
This commit is contained in:
Stefan Mühlinghaus
2026-07-12 02:49:54 +02:00
commit 6dd456ca4a
15 changed files with 920 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
.env.example Normal file
View File

@@ -0,0 +1,4 @@
PB_URL=https://www.alphabreed.com/api
PB_USER=your-username
PB_PASSWORD=your-password
PB_CATEGORY=ztvvxqod5s899fk

4
.gitignore vendored Normal file
View File

@@ -0,0 +1,4 @@
.env
royalroadupdates
.idea/
.pi-subagents

102
AGENTS.md Normal file
View File

@@ -0,0 +1,102 @@
# Agent Instructions — `royalroadupdates`
## Project Purpose
A small, self-contained Go CLI tool that checks RoyalRoad stories from a
PocketBase bookmarks collection and reports their last-update times.
## Repository Layout
```
.
├── cmd/royalroadupdates/ Entry point and integration tests
│ ├── main.go - CLI bootstrap, flag parsing (--json)
│ └── e2e_test.go - End-to-end test: compiles binary, runs it,
│ asserts JSON shape and entry count
├── internal/
│ ├── pocketbase/
│ │ └── client.go - Authenticates and queries the
│ │ /api/collections/bookmarks/records endpoint.
│ │ Accepts filter, perPage, and field params.
│ │ Returns a list of structs with URL fields.
│ ├── royalroad/
│ │ └── scraper.go - Given a URL, fetches the page, confirms it
│ │ is a RoyalRoad fiction page, extracts the
│ │ JSON-LD script block, parses name and
│ │ dateModified into a struct.
│ │ Returns zero-value + error for any failure
│ │ so the caller can skip silently.
│ ├── formatter/
│ │ ├── table.go - Renders []Story as a plain-text aligned
│ │ │ table with relative-time "Last Update"
│ │ │ column.
│ │ ├── json.go - Renders []Story as JSON (used with --json).
│ │ └── relative.go - Computes "X minutes/hours/days/months/years
│ │ ago" strings from time.Time.
│ └── app.go - Orchestrates the full pipeline:
│ 1. Load credentials (compile-time vars).
│ 2. Query PocketBase for URLs.
│ 3. Iterate URLs sequentially.
│ 4. Collect valid RoyalRoad results.
│ 5. Sort by dateModified descending.
│ 6. Dispatch to appropriate formatter.
├── .env - Real credentials (gitignored).
├── .env.example - Template showing required env vars.
├── .gitignore - Must include .env and the built binary.
├── Makefile - All build, test, and check targets.
├── go.mod - Standard Go module file.
└── go.sum - Standard Go checksum file.
```
## Code Style Rules
- **Formatting:** All Go source files must pass `gofmt`. The `make check` target
will reject any diff.
- **Imports:** Group standard library first, then third-party (none expected),
then local/internal. `goimports` ordering is acceptable.
- **Naming:** Exported identifiers use PascalCase; unexported use camelCase.
Acronyms like HTTP, URL, JSON keep all-caps when exported (`FetchURL`,
`ParseJSON`).
- **Error handling:** Always check errors explicitly. Do not swallow errors
silently unless the design calls for silent skipping (RoyalRoad per-item
failures). Return errors wrapped with `fmt.Errorf("...: %w", err)` for
context.
- **Comments:** Every exported type and function has a concise GoDoc comment.
Internal helpers may omit comments if their purpose is obvious from the name.
- **No globals:** Avoid package-level mutable state. Pass dependencies (HTTP
client, configuration) as struct fields or constructor parameters.
- **HTTP client reuse:** Create one `http.Client` with a configured timeout and
share it across requests. Do not use `http.DefaultClient` unwrapped.
## Build Rules
- **No local Go installation.** Use `make` targets which wrap
`docker run --rm -v $(pwd):/src -w /src golang:1.22`.
- **Credential injection:** The Makefile reads `.env` and passes values as
`-ldflags '-X main.PB_URL=... -X main.PB_USER=... -X main.PB_PASSWORD=...'`.
- **Output:** The compiled binary is written to `./royalroadupdates` in the
repository root.
- **Targets:**
- `make build` — compile the binary.
- `make check` — full quality loop (format, vet, test, e2e test).
- `make clean` — remove the compiled binary.
## Testing Rules
- **Unit tests:** Cover pure helpers (e.g., `relative.go`, URL filtering logic)
with table-driven tests. Use `t.Parallel()` where independent.
- **End-to-end test (`e2e_test.go`):**
- Uses the `.env` credentials and hits real services.
- Compiles the binary via `os/exec` so it tests the actual build path.
- Runs `./royalroadupdates --json`.
- Assertions:
1. Exit code 0.
2. Stderr is empty.
3. Stdout is valid JSON.
4. JSON is a non-empty array.
5. Array contains at least 3 entries.
6. Each entry has a non-empty `name` and a parseable `date`.
- Since this test hits live services, it is inherently non-deterministic in
data content but deterministic in shape. Treat network/Cloudflare flakiness
as a signal to tune delays rather than a code defect.
- **Fix-and-rerun cycle:** If `make check` fails at any step, fix the root cause
and rerun `make check` until all gates pass.

32
Makefile Normal file
View File

@@ -0,0 +1,32 @@
# Makefile for royalroadupdates — builds entirely inside Docker.
PWD := $(shell pwd)
DOCKER_RUN := docker run --rm -v $(PWD):/src -w /src golang:1.22
.PHONY: build check clean
build:
$(DOCKER_RUN) sh -c '\
set -e; \
if [ -f .env ]; then . ./.env; fi; \
if [ -z "$$PB_URL" ] || [ -z "$$PB_USER" ] || [ -z "$$PB_PASSWORD" ] || [ -z "$$PB_CATEGORY" ]; then \
echo "error: .env is missing or one of PB_URL / PB_USER / PB_PASSWORD / PB_CATEGORY is not set"; \
exit 1; \
fi; \
go build \
-ldflags "-X main.PBURL=$${PB_URL} -X main.PBUser=$${PB_USER} -X main.PBPassword=$${PB_PASSWORD} -X main.PBCategory=$${PB_CATEGORY}" \
-o royalroadupdates \
./cmd/royalroadupdates'
check:
$(DOCKER_RUN) sh -c '\
set -e; \
if [ -f .env ]; then . ./.env; fi; \
export PB_URL PB_USER PB_PASSWORD PB_CATEGORY; \
d=$$(gofmt -l .); \
if [ -n "$$d" ]; then echo "format errors:"; echo "$$d"; exit 1; fi; \
go vet ./...; \
go test ./...'
clean:
rm -f royalroadupdates

138
PLAN.md Normal file
View File

@@ -0,0 +1,138 @@
# RoyalRoad Updates — Project Plan
## Overview
A Go CLI tool that reads RoyalRoad story URLs from a PocketBase collection, fetches each
story's last-update time from RoyalRoad, and prints a sorted plain-text table showing
story names and how recently each was updated.
## Behavior
1. Authenticate to PocketBase using credentials baked in at compile time.
2. Query the `bookmarks` collection, filtering `category = "Geschichten"`.
3. Extract the `url` field from each matching record.
4. For each URL, sequentially fetch the RoyalRoad fiction page.
5. Extract the fiction name and `dateModified` from the embedded JSON-LD
`<script type="application/ld+json">` block.
6. Sort results by `dateModified` descending (most recent first).
7. Print a plain-text table with:
- `Story Name` column
- `Last Update` column showing relative time (e.g., "3 hours ago")
8. When `--json` is passed, output machine-readable JSON with actual ISO 8601 dates.
### URL Filtering Rules
- **Skip chapter URLs silently.** A chapter URL contains `/chapter/` in its path.
If a bookmark points to a specific chapter, the user is not yet caught up; no
update notification is wanted.
- **Skip non-RoyalRoad URLs silently.** Only URLs whose host ends with
`royalroad.com` (with or without `www.`) are processed.
- **Skip silently on any fetch or parse failure.** A single bad URL must not
crash the tool or clutter output.
### Error Handling
| Scenario | Behaviour |
|----------|-----------|
| PocketBase unreachable or auth fails | Exit code ≠ 0, error message on stderr |
| Zero valid RoyalRoad stories found after filtering | Exit code 0, empty table |
| Individual RoyalRoad page fails to fetch or parse | Silent skip (no output, no error) |
## Tech Stack
| Layer | Choice |
|-------|--------|
| Language | Go |
| Build orchestration | `Makefile` |
| Go compiler | Official `golang:1.22` Docker image (no local toolchain) |
| HTTP client | `net/http` (standard library) |
| JSON parsing | `encoding/json` (standard library) |
| HTML extraction | Regex/string search for `<script type="application/ld+json">` |
| Date parsing | `time.Parse(time.RFC3339, ...)` (standard library) |
| Relative-time formatting | Custom helper from `time.Time` |
**No external Go dependencies.** The project is deliberately small and uses only
the standard library.
## Compile-Time Configuration
The three PocketBase connection values are compiled into the binary via `-ldflags`:
```
PB_URL=https://pocketbase.example.com
PB_USER=username
PB_PASSWORD=password
```
These are read from a `.env` file at build time. The resulting binary runs
standalone without any environment variables present.
## Output Formats
### Default (plain-text table)
```
Story Name Last Update
----------------------------------- -----------
A Fractured Truth 3 hours ago
Mother of Learning 6 years ago
```
### `--json`
```json
[
{
"name": "A Fractured Truth",
"date": "2024-12-11T08:55:49Z"
},
{
"name": "Mother of Learning",
"date": "2023-07-06T17:47:41Z"
}
]
```
Dates are always sortable ISO 8601 / RFC3339 strings in JSON output.
## Data Flow
```
PocketBase (bookmarks collection)
└── Filter: category = "Geschichten"
└── Extract: url field
└── For each URL:
├── Skip if not royalroad.com
├── Skip if /chapter/ in path
├── Fetch fiction page (sequential, 30s timeout)
├── Extract JSON-LD script tag
├── Parse name + dateModified
└── Skip silently on any error
└── Collect valid results
└── Sort by dateModified descending
└── Print table or JSON
```
## Development Loop
`make check` runs the full pipeline:
1. `gofmt -d` — fail if any file is unformatted.
2. `go vet ./...` — static analysis.
3. `go test ./...` — unit tests plus end-to-end test:
- Compiles the binary using the local `.env` credentials.
- Runs `./royalroadupdates --json`.
- Asserts:
- exit code 0
- stderr empty
- stdout is valid JSON
- JSON array contains ≥ 3 entries
- Each entry has `name` and `date` fields
4. If any step fails: fix issues, rerun `make check`.
## Open Decisions / Future Tuning
- **Rate limiting:** Requests are sequential to be gentle on RoyalRoad. If
Cloudflare blocks the tool, add a per-request delay (e.g., `time.Sleep`).
- **Relative time precision:** Currently shows minutes/hours/days/years as
coarse buckets. This can be refined if desired.

3
go.mod Normal file
View File

@@ -0,0 +1,3 @@
module royalroadupdates
go 1.22.12

70
internal/app.go Normal file
View File

@@ -0,0 +1,70 @@
package internal
import (
"context"
"fmt"
"io"
"os"
"sort"
"royalroadupdates/internal/formatter"
"royalroadupdates/internal/pocketbase"
"royalroadupdates/internal/royalroad"
)
// Config holds the compile-time settings for the application.
type Config struct {
PBURL string
PBUser string
PBPassword string
PBCategory string
}
// App orchestrates the full pipeline.
type App struct {
pocketbase *pocketbase.Client
scraper *royalroad.Scraper
outW io.Writer
errW io.Writer
}
// New creates a new App with the given configuration.
func New(cfg Config) *App {
return &App{
pocketbase: pocketbase.New(cfg.PBURL, cfg.PBUser, cfg.PBPassword, cfg.PBCategory),
scraper: royalroad.NewScraper(),
outW: os.Stdout,
errW: os.Stderr,
}
}
// Run executes the full pipeline and outputs results.
func (a *App) Run(ctx context.Context, jsonOutput bool) error {
urls, err := a.pocketbase.GetBookmarkURLs(ctx)
if err != nil {
return fmt.Errorf("fetching bookmarks: %w", err)
}
var results []royalroad.Story
for _, u := range urls {
story, err := a.scraper.FetchStory(ctx, u)
if err != nil {
continue // silently skip
}
results = append(results, story)
}
sort.Slice(results, func(i, j int) bool {
return results[i].DateModified.After(results[j].DateModified)
})
if jsonOutput {
if err := formatter.PrintJSON(a.outW, results); err != nil {
return fmt.Errorf("encoding JSON: %w", err)
}
} else {
formatter.PrintTable(a.outW, results)
}
return nil
}

27
internal/formatter/json.go Normal file
View File

@@ -0,0 +1,27 @@
package formatter
import (
"encoding/json"
"io"
"royalroadupdates/internal/royalroad"
)
// jsonEntry is the JSON output shape.
type jsonEntry struct {
Name string `json:"name"`
Date string `json:"date"`
}
// PrintJSON writes stories as JSON to w.
func PrintJSON(w io.Writer, stories []royalroad.Story) error {
entries := make([]jsonEntry, 0, len(stories))
for _, s := range stories {
entries = append(entries, jsonEntry{
Name: s.Name,
Date: s.DateModified.UTC().Format("2006-01-02T15:04:05Z"),
})
}
enc := json.NewEncoder(w)
return enc.Encode(entries)
}

41
internal/formatter/relative.go Normal file
View File

@@ -0,0 +1,41 @@
package formatter
import (
"fmt"
"time"
)
// Relative returns a human-friendly relative time string like "3 hours ago".
func Relative(t time.Time) string {
d := time.Since(t)
if d < time.Minute {
return "just now"
}
if d < time.Hour {
m := int(d.Minutes())
return pluralize(m, "minute") + " ago"
}
if d < 24*time.Hour {
h := int(d.Hours())
return pluralize(h, "hour") + " ago"
}
if d < 30*24*time.Hour {
day := int(d.Hours() / 24)
return pluralize(day, "day") + " ago"
}
if d < 365*24*time.Hour {
month := int(d.Hours() / 24 / 30)
return pluralize(month, "month") + " ago"
}
year := int(d.Hours() / 24 / 365)
return pluralize(year, "year") + " ago"
}
func pluralize(n int, unit string) string {
if n == 1 {
return fmt.Sprintf("1 %s", unit)
}
return fmt.Sprintf("%d %ss", n, unit)
}

39
internal/formatter/relative_test.go Normal file
View File

@@ -0,0 +1,39 @@
package formatter
import (
"testing"
"time"
)
func TestRelative(t *testing.T) {
t.Parallel()
now := time.Now()
tests := []struct {
name string
t time.Time
want string
}{
{"just now", now.Add(-5 * time.Second), "just now"},
{"1 minute ago", now.Add(-1 * time.Minute), "1 minute ago"},
{"5 minutes ago", now.Add(-5 * time.Minute), "5 minutes ago"},
{"1 hour ago", now.Add(-1 * time.Hour), "1 hour ago"},
{"3 hours ago", now.Add(-3 * time.Hour), "3 hours ago"},
{"1 day ago", now.Add(-24 * time.Hour), "1 day ago"},
{"2 days ago", now.Add(-48 * time.Hour), "2 days ago"},
{"1 month ago", now.Add(-30 * 24 * time.Hour), "1 month ago"},
{"3 months ago", now.Add(-90 * 24 * time.Hour), "3 months ago"},
{"1 year ago", now.Add(-365 * 24 * time.Hour), "1 year ago"},
{"2 years ago", now.Add(-730 * 24 * time.Hour), "2 years ago"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
got := Relative(tt.t)
if got != tt.want {
t.Errorf("Relative() = %q, want %q", got, tt.want)
}
})
}
}

33
internal/formatter/table.go Normal file
View File

@@ -0,0 +1,33 @@
package formatter
import (
"fmt"
"io"
"strings"
"royalroadupdates/internal/royalroad"
)
// PrintTable writes a plain-text aligned table of stories to w.
func PrintTable(w io.Writer, stories []royalroad.Story) {
if len(stories) == 0 {
return
}
nameHeader := "Story Name"
updateHeader := "Last Update"
maxNameLen := len(nameHeader)
for _, s := range stories {
if l := len(s.Name); l > maxNameLen {
maxNameLen = l
}
}
fmt.Fprintf(w, "%-*s %s\n", maxNameLen, nameHeader, updateHeader)
fmt.Fprintf(w, "%s %s\n", strings.Repeat("-", maxNameLen), strings.Repeat("-", len(updateHeader)))
for _, s := range stories {
fmt.Fprintf(w, "%-*s %s\n", maxNameLen, s.Name, Relative(s.DateModified))
}
}

147
internal/pocketbase/client.go Normal file
View File

@@ -0,0 +1,147 @@
package pocketbase
import (
"bytes"
"context"
"encoding/json"
"fmt"
"net/http"
"net/url"
"strings"
"time"
)
// Client handles interactions with the PocketBase API.
type Client struct {
baseURL string
user string
password string
category string
http *http.Client
}
// New creates a new PocketBase client.
func New(baseURL, user, password, category string) *Client {
// Normalise: ensure baseURL does not already end with /api.
baseURL = strings.TrimSuffix(baseURL, "/api")
baseURL = strings.TrimSuffix(baseURL, "/")
return &Client{
baseURL: baseURL,
user: user,
password: password,
category: category,
http: &http.Client{
Timeout: 30 * time.Second,
},
}
}
type authRequest struct {
Identity string `json:"identity"`
Password string `json:"password"`
}
type authResponse struct {
Token string `json:"token"`
}
type record struct {
URL string `json:"url"`
}
type recordsResponse struct {
Items []record `json:"items"`
}
// GetBookmarkURLs authenticates and retrieves URLs from the bookmarks collection.
func (c *Client) GetBookmarkURLs(ctx context.Context) ([]string, error) {
token, err := c.authenticate(ctx)
if err != nil {
return nil, fmt.Errorf("pocketbase auth: %w", err)
}
urls, err := c.fetchURLs(ctx, token)
if err != nil {
return nil, fmt.Errorf("pocketbase fetch URLs: %w", err)
}
return urls, nil
}
func (c *Client) authenticate(ctx context.Context) (string, error) {
reqBody, err := json.Marshal(authRequest{
Identity: c.user,
Password: c.password,
})
if err != nil {
return "", err
}
authURL := c.baseURL + "/api/collections/users/auth-with-password"
req, err := http.NewRequestWithContext(ctx, "POST", authURL, bytes.NewBuffer(reqBody))
if err != nil {
return "", err
}
req.Header.Set("Content-Type", "application/json")
resp, err := c.http.Do(req)
if err != nil {
return "", err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return "", fmt.Errorf("unexpected status: %d", resp.StatusCode)
}
var authResp authResponse
if err := json.NewDecoder(resp.Body).Decode(&authResp); err != nil {
return "", err
}
return authResp.Token, nil
}
func (c *Client) fetchURLs(ctx context.Context, token string) ([]string, error) {
// Filter: category=<value>, perPage=200, fields=url
filter := fmt.Sprintf(`category="%s"`, c.category)
u, err := url.Parse(c.baseURL + "/api/collections/bookmarks/records")
if err != nil {
return nil, fmt.Errorf("invalid base URL: %w", err)
}
q := u.Query()
q.Set("filter", filter)
q.Set("perPage", "200")
q.Set("fields", "url")
u.RawQuery = q.Encode()
req, err := http.NewRequestWithContext(ctx, "GET", u.String(), nil)
if err != nil {
return nil, fmt.Errorf("creating request: %w", err)
}
req.Header.Set("Authorization", token)
resp, err := c.http.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("unexpected status: %d", resp.StatusCode)
}
var recs recordsResponse
if err := json.NewDecoder(resp.Body).Decode(&recs); err != nil {
return nil, err
}
var urls []string
for _, r := range recs.Items {
if r.URL != "" {
urls = append(urls, r.URL)
}
}
return urls, nil
}

65
internal/pocketbase/client_test.go Normal file
View File

@@ -0,0 +1,65 @@
package pocketbase
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
)
func TestGetBookmarkURLs_Success(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/api/collections/users/auth-with-password" {
w.WriteHeader(http.StatusOK)
json.NewEncoder(w).Encode(authResponse{Token: "test-token"})
return
}
if r.URL.Path == "/api/collections/bookmarks/records" {
if r.Header.Get("Authorization") != "test-token" {
t.Errorf("Expected Authorization header 'test-token', got '%s'", r.Header.Get("Authorization"))
}
w.WriteHeader(http.StatusOK)
json.NewEncoder(w).Encode(recordsResponse{
Items: []record{
{URL: "https://royalroad.com/fiction/1"},
{URL: "https://royalroad.com/fiction/2"},
},
})
return
}
t.Errorf("Unexpected request to %s", r.URL.Path)
}))
defer server.Close()
client := New(server.URL, "user", "pass", "Geschichten")
urls, err := client.GetBookmarkURLs(context.Background())
if err != nil {
t.Fatalf("Unexpected error: %v", err)
}
if len(urls) != 2 {
t.Errorf("Expected 2 URLs, got %d", len(urls))
}
if urls[0] != "https://royalroad.com/fiction/1" {
t.Errorf("Unexpected first URL: %s", urls[0])
}
}
func TestGetBookmarkURLs_AuthFailure(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/api/collections/users/auth-with-password" {
w.WriteHeader(http.StatusUnauthorized)
return
}
t.Errorf("Unexpected request to %s", r.URL.Path)
}))
defer server.Close()
client := New(server.URL, "user", "pass", "Geschichten")
_, err := client.GetBookmarkURLs(context.Background())
if err == nil {
t.Error("Expected error on auth failure, got nil")
}
}

121
internal/royalroad/scraper.go Normal file
View File

@@ -0,0 +1,121 @@
package royalroad
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"net/url"
"strings"
"time"
)
// Story represents the metadata extracted from a RoyalRoad fiction page.
type Story struct {
Name string
DateModified time.Time
}
// jsonLD represents the structured data found in RoyalRoad fiction pages.
type jsonLD struct {
Name string `json:"name"`
DateModified string `json:"dateModified"`
}
// Scraper handles fetching and parsing RoyalRoad fiction pages.
type Scraper struct {
client *http.Client
}
// NewScraper creates a new Scraper with a configured HTTP client.
func NewScraper() *Scraper {
return &Scraper{
client: &http.Client{
Timeout: 30 * time.Second,
},
}
}
// FetchStory extracts the story name and modification date from a RoyalRoad fiction page.
func (s *Scraper) FetchStory(ctx context.Context, rawURL string) (Story, error) {
parsedURL, err := url.Parse(rawURL)
if err != nil {
return Story{}, fmt.Errorf("invalid URL %q: %w", rawURL, err)
}
host := strings.ToLower(parsedURL.Hostname())
if host != "royalroad.com" && host != "www.royalroad.com" {
return Story{}, fmt.Errorf("URL host %q is not royalroad.com", host)
}
if strings.Contains(parsedURL.Path, "/chapter/") {
return Story{}, fmt.Errorf("URL %q points to a chapter, not a main fiction page", rawURL)
}
req, err := http.NewRequestWithContext(ctx, "GET", rawURL, nil)
if err != nil {
return Story{}, fmt.Errorf("failed to create request: %w", err)
}
req.Header.Set("User-Agent", "royalroadupdates/1.0")
resp, err := s.client.Do(req)
if err != nil {
return Story{}, fmt.Errorf("failed to fetch page: %w", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return Story{}, fmt.Errorf("unexpected response status: %s", resp.Status)
}
body, err := io.ReadAll(resp.Body)
if err != nil {
return Story{}, fmt.Errorf("failed to read body: %w", err)
}
story, err := s.parseJSONLD(body)
if err != nil {
return Story{}, fmt.Errorf("failed to parse JSON-LD: %w", err)
}
return story, nil
}
func (s *Scraper) parseJSONLD(body []byte) (Story, error) {
startTag := `<script type="application/ld+json">`
endTag := `</script>`
bodyStr := string(body)
startIndex := strings.Index(bodyStr, startTag)
if startIndex == -1 {
return Story{}, fmt.Errorf("JSON-LD script block not found")
}
startIndex += len(startTag)
endIndex := strings.Index(bodyStr[startIndex:], endTag)
if endIndex == -1 {
return Story{}, fmt.Errorf("JSON-LD closing tag not found")
}
endIndex += startIndex
jsonContent := body[startIndex:endIndex]
var ld jsonLD
if err := json.Unmarshal(jsonContent, &ld); err != nil {
return Story{}, fmt.Errorf("failed to decode JSON: %w", err)
}
if ld.Name == "" {
return Story{}, fmt.Errorf("story name missing in JSON-LD")
}
date, err := time.Parse(time.RFC3339, ld.DateModified)
if err != nil {
return Story{}, fmt.Errorf("invalid date format %q: %w", ld.DateModified, err)
}
return Story{
Name: ld.Name,
DateModified: date,
}, nil
}

94
internal/royalroad/scraper_test.go Normal file
View File

@@ -0,0 +1,94 @@
package royalroad
import (
"context"
"net/http"
"net/http/httptest"
"testing"
"time"
)
func TestParseJSONLD(t *testing.T) {
tests := []struct {
name string
body string
want Story
wantErr bool
}{
{
name: "Success",
body: `<html><script type="application/ld+json">{"name":"Test Story","dateModified":"2023-07-06T17:47:41Z"}</script></html>`,
want: Story{
Name: "Test Story",
DateModified: time.Date(2023, 7, 6, 17, 47, 41, 0, time.UTC),
},
wantErr: false,
},
{
name: "Missing Script",
body: `<html><body>No JSON-LD here</body></html>`,
wantErr: true,
},
{
name: "Malformed JSON",
body: `<script type="application/ld+json">{invalid json}</script>`,
wantErr: true,
},
{
name: "Missing Name",
body: `<script type="application/ld+json">{"dateModified":"2023-07-06T17:47:41Z"}</script>`,
wantErr: true,
},
{
name: "Invalid Date",
body: `<script type="application/ld+json">{"name":"Story","dateModified":"not-a-date"}</script>`,
wantErr: true,
},
}
s := NewScraper()
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := s.parseJSONLD([]byte(tt.body))
if (err != nil) != tt.wantErr {
t.Errorf("parseJSONLD() error = %v, wantErr %v", err, tt.wantErr)
return
}
if !tt.wantErr && got != tt.want {
t.Errorf("parseJSONLD() = %v, want %v", got, tt.want)
}
})
}
}
func TestFetchStory_NonRoyalRoadHost(t *testing.T) {
s := NewScraper()
_, err := s.FetchStory(context.Background(), "https://google.com/fiction/123")
if err == nil {
t.Error("Expected error for non-RoyalRoad host, got nil")
}
}
func TestFetchStory_ChapterURL(t *testing.T) {
s := NewScraper()
_, err := s.FetchStory(context.Background(), "https://www.royalroad.com/fiction/123/chapter/456")
if err == nil {
t.Error("Expected error for chapter URL, got nil")
}
}
func TestFetchStory_HTTPError(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusNotFound)
}))
defer ts.Close()
s := NewScraper()
// We can't easily mock the host check for the server URL without custom logic,
// but we can verify that a non-RR URL (the test server) fails at the host check
// before it even hits the server.
_, err := s.FetchStory(context.Background(), ts.URL+"/fiction/123")
if err == nil {
t.Error("Expected error for non-RoyalRoad host, got nil")
}
}